title: "AssertionError: Torch not compiled with CUDA enabled" in spite upgrading to CUDA version

body:
I figured out this is a popular question, but still I couldn't find a solution for that.

I'm trying to run a simple repo Here which uses PyTorch. Although I just upgraded my Pytorch to the latest CUDA version from pytorch.org (1.2.0), it still throws the same error. I'm on Windows 10 and use conda with python 3.7.

    raise AssertionError("Torch not compiled with CUDA enabled")
AssertionError: Torch not compiled with CUDA enabled

How to fix the problem?

Here is my conda list:

# Name                    Version                   Build  Channel
_ipyw_jlab_nb_ext_conf    0.1.0                    py37_0    anaconda
_pytorch_select           1.1.0                       cpu    anaconda
_tflow_select             2.3.0                       mkl    anaconda
absl-py                   0.7.1                    pypi_0    pypi
alabaster                 0.7.12                   py37_0    anaconda
anaconda                  2019.07                  py37_0    anaconda
anaconda-client           1.7.2                    py37_0    anaconda
anaconda-navigator        1.9.7                    py37_0    anaconda
anaconda-project          0.8.3                      py_0    anaconda
argparse                  1.4.0                    pypi_0    pypi
asn1crypto                0.24.0                   py37_0    anaconda
astor                     0.8.0                    pypi_0    pypi
astroid                   2.2.5                    py37_0    anaconda
astropy                   3.2.1            py37he774522_0    anaconda
atomicwrites              1.3.0                    py37_1    anaconda
attrs                     19.1.0                   py37_1    anaconda
babel                     2.7.0                      py_0    anaconda
backcall                  0.1.0                    py37_0    anaconda
backports                 1.0                        py_2    anaconda
backports-csv             1.0.7                    pypi_0    pypi
backports-functools-lru-cache 1.5                      pypi_0    pypi
backports.functools_lru_cache 1.5                        py_2    anaconda
backports.os              0.1.1                    py37_0    anaconda
backports.shutil_get_terminal_size 1.0.0                    py37_2    anaconda
backports.tempfile        1.0                        py_1    anaconda
backports.weakref         1.0.post1                  py_1    anaconda
beautifulsoup4            4.7.1                    py37_1    anaconda
bitarray                  0.9.3            py37he774522_0    anaconda
bkcharts                  0.2                      py37_0    anaconda
blas                      1.0                         mkl    anaconda
bleach                    3.1.0                    py37_0    anaconda
blosc                     1.16.3               h7bd577a_0    anaconda
bokeh                     1.2.0                    py37_0    anaconda
boto                      2.49.0                   py37_0    anaconda
bottleneck                1.2.1            py37h452e1ab_1    anaconda
bzip2                     1.0.8                he774522_0    anaconda
ca-certificates           2019.5.15                     0    anaconda
certifi                   2019.6.16                py37_0    anaconda
cffi                      1.12.3           py37h7a1dbc1_0    anaconda
chainer                   6.2.0                    pypi_0    pypi
chardet                   3.0.4                    py37_1    anaconda
cheroot                   6.5.5                    pypi_0    pypi
cherrypy                  18.1.2                   pypi_0    pypi
click                     7.0                      py37_0    anaconda
cloudpickle               1.2.1                      py_0    anaconda
clyent                    1.2.2                    py37_1    anaconda
colorama                  0.4.1                    py37_0    anaconda
comtypes                  1.1.7                    py37_0    anaconda
conda                     4.7.11                   py37_0    anaconda
conda-build               3.18.9                   py37_3    anaconda
conda-env                 2.6.0                         1    anaconda
conda-package-handling    1.3.11                   py37_0    anaconda
conda-verify              3.4.2                      py_1    anaconda
console_shortcut          0.1.1                         3    anaconda
constants                 0.6.0                    pypi_0    pypi
contextlib2               0.5.5                    py37_0    anaconda
cpuonly                   1.0                           0    pytorch
cryptography              2.7              py37h7a1dbc1_0    anaconda
cudatoolkit               10.0.130                      0    anaconda
curl                      7.65.2               h2a8f88b_0    anaconda
cycler                    0.10.0                   py37_0    anaconda
cython                    0.29.12          py37ha925a31_0    anaconda
cytoolz                   0.10.0           py37he774522_0    anaconda
dask                      2.1.0                      py_0    anaconda
dask-core                 2.1.0                      py_0    anaconda
decorator                 4.4.0                    py37_1    anaconda
defusedxml                0.6.0                      py_0    anaconda
distributed               2.1.0                      py_0    anaconda
docutils                  0.14                     py37_0    anaconda
entrypoints               0.3                      py37_0    anaconda
et_xmlfile                1.0.1                    py37_0    anaconda
ez-setup                  0.9                      pypi_0    pypi
fastcache                 1.1.0            py37he774522_0    anaconda
fasttext                  0.9.1                    pypi_0    pypi
feedparser                5.2.1                    pypi_0    pypi
ffmpeg                    4.1.3                h6538335_0    conda-forge
filelock                  3.0.12                     py_0    anaconda
first                     2.0.2                    pypi_0    pypi
flask                     1.1.1                      py_0    anaconda
freetype                  2.9.1                ha9979f8_1    anaconda
future                    0.17.1                   py37_0    anaconda
gast                      0.2.2                    py37_0    anaconda
get                       2019.4.13                pypi_0    pypi
get_terminal_size         1.0.0                h38e98db_0    anaconda
gevent                    1.4.0            py37he774522_0    anaconda
glob2                     0.7                        py_0    anaconda
google-pasta              0.1.7                    pypi_0    pypi
graphviz                  2.38.0                        4    anaconda
greenlet                  0.4.15           py37hfa6e2cd_0    anaconda
grpcio                    1.22.0                   pypi_0    pypi
h5py                      2.9.0            py37h5e291fa_0    anaconda
hdf5                      1.10.4               h7ebc959_0    anaconda
heapdict                  1.0.0                    py37_2    anaconda
html5lib                  1.0.1                    py37_0    anaconda
http-client               0.1.22                   pypi_0    pypi
hypothesis                4.34.0                   pypi_0    pypi
icc_rt                    2019.0.0             h0cc432a_1    anaconda
icu                       58.2                 ha66f8fd_1    anaconda
idna                      2.8                      py37_0    anaconda
imageio                   2.4.1                    pypi_0    pypi
imageio-ffmpeg            0.3.0                    pypi_0    pypi
imagesize                 1.1.0                    py37_0    anaconda
importlib_metadata        0.17                     py37_1    anaconda
imutils                   0.5.2                    pypi_0    pypi
intel-openmp              2019.0                   pypi_0    pypi
ipykernel                 5.1.1            py37h39e3cac_0    anaconda
ipython                   7.6.1            py37h39e3cac_0    anaconda
ipython_genutils          0.2.0                    py37_0    anaconda
ipywidgets                7.5.0                      py_0    anaconda
isort                     4.3.21                   py37_0    anaconda
itsdangerous              1.1.0                    py37_0    anaconda
jaraco-functools          2.0                      pypi_0    pypi
jdcal                     1.4.1                      py_0    anaconda
jedi                      0.13.3                   py37_0    anaconda
jinja2                    2.10.1                   py37_0    anaconda
joblib                    0.13.2                   py37_0    anaconda
jpeg                      9b                   hb83a4c4_2    anaconda
json5                     0.8.4                      py_0    anaconda
jsonschema                3.0.1                    py37_0    anaconda
jupyter                   1.0.0                    py37_7    anaconda
jupyter_client            5.3.1                      py_0    anaconda
jupyter_console           6.0.0                    py37_0    anaconda
jupyter_core              4.5.0                      py_0    anaconda
jupyterlab                1.0.2            py37hf63ae98_0    anaconda
jupyterlab_server         1.0.0                      py_0    anaconda
keras                     2.2.4                         0    anaconda
keras-applications        1.0.8                      py_0    anaconda
keras-base                2.2.4                    py37_0    anaconda
keras-preprocessing       1.1.0                      py_1    anaconda
keyring                   18.0.0                   py37_0    anaconda
kiwisolver                1.1.0            py37ha925a31_0    anaconda
krb5                      1.16.1               hc04afaa_7
lazy-object-proxy         1.4.1            py37he774522_0    anaconda
libarchive                3.3.3                h0643e63_5    anaconda
libcurl                   7.65.2               h2a8f88b_0    anaconda
libiconv                  1.15                 h1df5818_7    anaconda
liblief                   0.9.0                ha925a31_2    anaconda
libmklml                  2019.0.5                      0    anaconda
libpng                    1.6.37               h2a8f88b_0    anaconda
libprotobuf               3.8.0                h7bd577a_0    anaconda
libsodium                 1.0.16               h9d3ae62_0    anaconda
libssh2                   1.8.2                h7a1dbc1_0    anaconda
libtiff                   4.0.10               hb898794_2    anaconda
libxml2                   2.9.9                h464c3ec_0    anaconda
libxslt                   1.1.33               h579f668_0    anaconda
llvmlite                  0.29.0           py37ha925a31_0    anaconda
locket                    0.2.0                    py37_1    anaconda
lxml                      4.3.4            py37h1350720_0    anaconda
lz4-c                     1.8.1.2              h2fa13f4_0    anaconda
lzo                       2.10                 h6df0209_2    anaconda
m2w64-gcc-libgfortran     5.3.0                         6
m2w64-gcc-libs            5.3.0                         7
m2w64-gcc-libs-core       5.3.0                         7
m2w64-gmp                 6.1.0                         2
m2w64-libwinpthread-git   5.0.0.4634.697f757               2
make-dataset              1.0                      pypi_0    pypi
markdown                  3.1.1                    py37_0    anaconda
markupsafe                1.1.1            py37he774522_0    anaconda
matplotlib                3.1.0            py37hc8f65d3_0    anaconda
mccabe                    0.6.1                    py37_1    anaconda
menuinst                  1.4.16           py37he774522_0    anaconda
mistune                   0.8.4            py37he774522_0    anaconda
mkl                       2019.0                   pypi_0    pypi
mkl-service               2.0.2            py37he774522_0    anaconda
mkl_fft                   1.0.12           py37h14836fe_0    anaconda
mkl_random                1.0.2            py37h343c172_0    anaconda
mock                      3.0.5                    py37_0    anaconda
more-itertools            7.0.0                    py37_0    anaconda
moviepy                   1.0.0                    pypi_0    pypi
mpmath                    1.1.0                    py37_0    anaconda
msgpack-python            0.6.1            py37h74a9793_1    anaconda
msys2-conda-epoch         20160418                      1
multipledispatch          0.6.0                    py37_0    anaconda
mysqlclient               1.4.2.post1              pypi_0    pypi
navigator-updater         0.2.1                    py37_0    anaconda
nbconvert                 5.5.0                      py_0    anaconda
nbformat                  4.4.0                    py37_0    anaconda
networkx                  2.3                        py_0    anaconda
ninja                     1.9.0            py37h74a9793_0    anaconda
nltk                      3.4.4                    py37_0    anaconda
nose                      1.3.7                    py37_2    anaconda
notebook                  6.0.0                    py37_0    anaconda
numba                     0.44.1           py37hf9181ef_0    anaconda
numexpr                   2.6.9            py37hdce8814_0    anaconda
numpy                     1.16.4                   pypi_0    pypi
numpy-base                1.16.4           py37hc3f5095_0    anaconda
numpydoc                  0.9.1                      py_0    anaconda
olefile                   0.46                     py37_0    anaconda
opencv-contrib-python     4.1.0.25                 pypi_0    pypi
opencv-python             4.1.0.25                 pypi_0    pypi
openpyxl                  2.6.2                      py_0    anaconda
openssl                   1.1.1c               he774522_1    anaconda
packaging                 19.0                     py37_0    anaconda
pandas                    0.24.2           py37ha925a31_0    anaconda
pandoc                    2.2.3.2                       0    anaconda
pandocfilters             1.4.2                    py37_1    anaconda
parso                     0.5.0                      py_0    anaconda
partd                     1.0.0                      py_0    anaconda
path.py                   12.0.1                     py_0    anaconda
pathlib2                  2.3.4                    py37_0    anaconda
patsy                     0.5.1                    py37_0    anaconda
pattern                   3.6                      pypi_0    pypi
pdfminer-six              20181108                 pypi_0    pypi
pep8                      1.7.1                    py37_0    anaconda
pickleshare               0.7.5                    py37_0    anaconda
pillow                    6.1.0            py37hdc69c19_0    anaconda
pip                       19.1.1                   py37_0    anaconda
pkginfo                   1.5.0.1                  py37_0    anaconda
pluggy                    0.12.0                     py_0    anaconda
ply                       3.11                     py37_0    anaconda
portend                   2.5                      pypi_0    pypi
post                      2019.4.13                pypi_0    pypi
powershell_shortcut       0.0.1                         2    anaconda
proglog                   0.1.9                    pypi_0    pypi
prometheus_client         0.7.1                      py_0    anaconda
prompt_toolkit            2.0.9                    py37_0    anaconda
protobuf                  3.7.1                    pypi_0    pypi
psutil                    5.6.3            py37he774522_0    anaconda
public                    2019.4.13                pypi_0    pypi
py                        1.8.0                    py37_0    anaconda
py-lief                   0.9.0            py37ha925a31_2    anaconda
pybind11                  2.3.0                    pypi_0    pypi
pycodestyle               2.5.0                    py37_0    anaconda
pycosat                   0.6.3            py37hfa6e2cd_0    anaconda
pycparser                 2.19                     py37_0    anaconda
pycrypto                  2.6.1            py37hfa6e2cd_9    anaconda
pycryptodome              3.8.2                    pypi_0    pypi
pycurl                    7.43.0.3         py37h7a1dbc1_0    anaconda
pydot                     1.4.1                    pypi_0    pypi
pyflakes                  2.1.1                    py37_0    anaconda
pygments                  2.4.2                      py_0    anaconda
pylint                    2.3.1                    py37_0    anaconda
pyodbc                    4.0.26           py37ha925a31_0    anaconda
pyopenssl                 19.0.0                   py37_0    anaconda
pyparsing                 2.4.0                      py_0    anaconda
pyqt                      5.9.2            py37h6538335_2    anaconda
pyreadline                2.1                      py37_1    anaconda
pyrsistent                0.14.11          py37he774522_0    anaconda
pysocks                   1.7.0                    py37_0    anaconda
pytables                  3.5.2            py37h1da0976_1    anaconda
pytest                    5.0.1                    py37_0    anaconda
pytest-arraydiff          0.3              py37h39e3cac_0    anaconda
pytest-astropy            0.5.0                    py37_0    anaconda
pytest-doctestplus        0.3.0                    py37_0    anaconda
pytest-openfiles          0.3.2                    py37_0    anaconda
pytest-remotedata         0.3.1                    py37_0    anaconda
python                    3.7.3                h8c8aaf0_1    anaconda
python-dateutil           2.8.0                    py37_0    anaconda
python-docx               0.8.10                   pypi_0    pypi
python-graphviz           0.11.1                   pypi_0    pypi
python-libarchive-c       2.8                     py37_11    anaconda
pytorch                   1.2.0               py3.7_cpu_1  [cpuonly]  pytorch
pytube                    9.5.1                    pypi_0    pypi
pytz                      2019.1                     py_0    anaconda
pywavelets                1.0.3            py37h8c2d366_1    anaconda
pywin32                   223              py37hfa6e2cd_1    anaconda
pywinpty                  0.5.5                 py37_1000    anaconda
pyyaml                    5.1.1            py37he774522_0    anaconda
pyzmq                     18.0.0           py37ha925a31_0    anaconda
qt                        5.9.7            vc14h73c81de_0  [vc14]  anaconda
qtawesome                 0.5.7                    py37_1    anaconda
qtconsole                 4.5.1                      py_0    anaconda
qtpy                      1.8.0                      py_0    anaconda
query-string              2019.4.13                pypi_0    pypi
request                   2019.4.13                pypi_0    pypi
requests                  2.22.0                   py37_0    anaconda
rope                      0.14.0                     py_0    anaconda
ruamel_yaml               0.15.46          py37hfa6e2cd_0    anaconda
scikit-image              0.15.0           py37ha925a31_0    anaconda
scikit-learn              0.21.2           py37h6288b17_0    anaconda
scipy                     1.3.0                    pypi_0    pypi
scipy-stack               0.0.5                    pypi_0    pypi
seaborn                   0.9.0                    py37_0    anaconda
send2trash                1.5.0                    py37_0    anaconda
setuptools                41.1.0                   pypi_0    pypi
simplegeneric             0.8.1                    py37_2    anaconda
singledispatch            3.4.0.3                  py37_0    anaconda
sip                       4.19.8           py37h6538335_0    anaconda
six                       1.12.0                   py37_0    anaconda
snappy                    1.1.7                h777316e_3    anaconda
snowballstemmer           1.9.0                      py_0    anaconda
sortedcollections         1.1.2                    py37_0    anaconda
sortedcontainers          2.1.0                    py37_0    anaconda
soupsieve                 1.8                      py37_0    anaconda
sphinx                    2.1.2                      py_0    anaconda
sphinxcontrib             1.0                      py37_1    anaconda
sphinxcontrib-applehelp   1.0.1                      py_0    anaconda
sphinxcontrib-devhelp     1.0.1                      py_0    anaconda
sphinxcontrib-htmlhelp    1.0.2                      py_0    anaconda
sphinxcontrib-jsmath      1.0.1                      py_0    anaconda
sphinxcontrib-qthelp      1.0.2                      py_0    anaconda
sphinxcontrib-serializinghtml 1.1.3                      py_0    anaconda
sphinxcontrib-websupport  1.1.2                      py_0    anaconda
spyder                    3.3.6                    py37_0    anaconda
spyder-kernels            0.5.1                    py37_0    anaconda
sqlalchemy                1.3.5            py37he774522_0    anaconda
sqlite                    3.29.0               he774522_0    anaconda
statsmodels               0.10.0           py37h8c2d366_0    anaconda
summa                     1.2.0                    pypi_0    pypi
sympy                     1.4                      py37_0    anaconda
tbb                       2019.4               h74a9793_0    anaconda
tblib                     1.4.0                      py_0    anaconda
tempora                   1.14.1                   pypi_0    pypi
tensorboard               1.14.0           py37he3c9ec2_0    anaconda
tensorboardx              1.8                      pypi_0    pypi
tensorflow                1.14.0          mkl_py37h7908ca0_0    anaconda
tensorflow-base           1.14.0          mkl_py37ha978198_0    anaconda
tensorflow-estimator      1.14.0                     py_0    anaconda
tensorflow-mkl            1.14.0               h4fcabd2_0    anaconda
termcolor                 1.1.0                    pypi_0    pypi
terminado                 0.8.2                    py37_0    anaconda
testpath                  0.4.2                    py37_0    anaconda
tk                        8.6.8                hfa6e2cd_0    anaconda
toolz                     0.10.0                     py_0    anaconda
torchvision               0.4.0                  py37_cpu  [cpuonly]  pytorch
tornado                   6.0.3            py37he774522_0    anaconda
tqdm                      4.32.1                     py_0    anaconda
traitlets                 4.3.2                    py37_0    anaconda
typing                    3.6.6                    pypi_0    pypi
typing-extensions         3.6.6                    pypi_0    pypi
unicodecsv                0.14.1                   py37_0    anaconda
urllib3                   1.24.2                   py37_0    anaconda
validators                0.13.0                   pypi_0    pypi
vc                        14.1                 h0510ff6_4    anaconda
vs2015_runtime            14.15.26706          h3a45250_4    anaconda
wcwidth                   0.1.7                    py37_0    anaconda
webencodings              0.5.1                    py37_1    anaconda
werkzeug                  0.15.4                     py_0    anaconda
wheel                     0.33.4                   py37_0    anaconda
widgetsnbextension        3.5.0                    py37_0    anaconda
win_inet_pton             1.1.0                    py37_0    anaconda
win_unicode_console       0.5                      py37_0    anaconda
wincertstore              0.2                      py37_0    anaconda
winpty                    0.4.3                         4    anaconda
wrapt                     1.11.2           py37he774522_0    anaconda
xlrd                      1.2.0                    py37_0    anaconda
xlsxwriter                1.1.8                      py_0    anaconda
xlwings                   0.15.8                   py37_0    anaconda
xlwt                      1.3.0                    py37_0    anaconda
xz                        5.2.4                h2fa13f4_4    anaconda
yaml                      0.1.7                hc54c509_2    anaconda
youtube-dl                2019.8.2                 pypi_0    pypi
zc-lockfile               1.4                      pypi_0    pypi
zeromq                    4.3.1                h33f27b4_3    anaconda
zict                      1.0.0                      py_0    anaconda
zipp                      0.5.1                      py_0    anaconda
zlib                      1.2.11               h62dcd97_3    anaconda
zstd                      1.3.7                h508b16e_0    anaconda


title: How can I load and use a PyTorch (.pth.tar) model

body:
I am not very familiar with Torch, and I primarily use Tensorflow. I, however, need to use a retrained inception model that was retrained in Torch. Due to the large amount of computing resources required to retrain an inception model for my particular application, I would like to use the model that was already retrained.
This model is saved as a .pth.tar file.
I would like to be able to first load this model. So far, I have been able to figure out that I must use the following:
model = torch.load('iNat_2018_InceptionV3.pth.tar', map_location='cpu')

This seems to work, because print(model) prints out a large set of numbers and other values, which I presume are the values for the weights an biases.
After this, I need to be able to classify an image with it. I haven't been able to figure this out. How must I format the image? Should the image be converted into an array? After this, how must I pass the input data to the network?


title: How to convert RGB images to grayscale in PyTorch dataloader?

body:
I've downloaded some sample images from the MNIST dataset in .jpg format. Now I'm loading those images for testing my pre-trained model.

# transforms to apply to the data
trans = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,))])

# MNIST dataset
test_dataset = dataset.ImageFolder(root=DATA_PATH, transform=trans)

# Data loader
test_loader = DataLoader(dataset=test_dataset, batch_size=batch_size, shuffle=False)

Here DATA_PATH contains a subfolder with the sample image.

Here's my network definition

# Convolutional neural network (two convolutional layers)
class ConvNet(nn.Module):
    def __init__(self):
        super(ConvNet, self).__init__()
        self.network2D = nn.Sequential(
           nn.Conv2d(1, 32, kernel_size=5, stride=1, padding=2),
           nn.ReLU(),
           nn.MaxPool2d(kernel_size=2, stride=2),
           nn.Conv2d(32, 64, kernel_size=5, stride=1, padding=2),
           nn.ReLU(),
           nn.MaxPool2d(kernel_size=2, stride=2))
        self.network1D = nn.Sequential(
           nn.Dropout(),
           nn.Linear(7 * 7 * 64, 1000),
           nn.Linear(1000, 10))

    def forward(self, x):
        out = self.network2D(x)
        out = out.reshape(out.size(0), -1)
        out = self.network1D(out)
        return out

And this is my inference part

# Test the model
model = torch.load("mnist_weights_5.pth.tar")
model.eval()

for images, labels in test_loader:
   outputs = model(images.cuda())

When I run this code, I get the following error:

RuntimeError: Given groups=1, weight of size [32, 1, 5, 5], expected input[1, 3, 28, 28] to have 1 channels, but got 3 channels instead

I understand that the images are getting loaded as 3 channels (RGB). So how do I convert them to single channel in the dataloader?

Update:
I changed transforms to include Grayscale option

trans = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,)), transforms.Grayscale(num_output_channels=1)])

But now I get this error

TypeError: img should be PIL Image. Got <class 'torch.Tensor'>


title: Replace all nonzero values by zero and all zero values by a specific value

body: I have a 3d tensor which contains some zero and nonzero values. I want to replace all nonzero values by zero and zero values by a specific value. How can I do that?


title: pytorch data loader multiple iterations

body:
i use iris-dataset to train a simple network with pytorch.

trainset = iris.Iris(train=True)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=150,
                                          shuffle=True, num_workers=2)

dataiter = iter(trainloader)

the dataset itself has only 150 data points, and pytorch dataloader iterates jus t once over the whole dataset, because of the batch size of 150.

My question is now, is there generally any way to tell dataloader of pytorch to repeat over the dataset if it's once done with iteration?

thnaks

update

got it runnning :)
just created a sub class of dataloader and implemented my own __next__()


title: How do I use a saved model in Pytorch to predict the label of a never before seen image?

body:
I have been trying to use my pretrained model to predict the label on a never before seen image. I have trained a CNN to classify flowers of 5 types using the Kaggle flower recognition dataset. I so far have trained my model to 97% accuracy and saved the model to a directory. I now want to download any image of a flower from those types and be able to use this pretrained model to predict the label. So far this is my code: (A code review for all of this would be very helpful as this is my first ever project)

This is my CNN model that I train:

from multiprocessing import freeze_support

import torch
from torch import nn
from torch.autograd import Variable
from torch.utils.data import DataLoader, Sampler
from torchvision import datasets
from torchvision.transforms import transforms
from torch.optim import Adam

import matplotlib.pyplot as plt
import numpy as np

# Hyperparameters.
num_epochs = 20
num_classes = 5
batch_size = 100
learning_rate = 0.001
num_of_workers = 5

DATA_PATH_TRAIN = 'C:\\Users\Aeryes\PycharmProjects\simplecnn\images\\train\\'
DATA_PATH_TEST = 'C:\\Users\Aeryes\PycharmProjects\simplecnn\images\\test\\'
MODEL_STORE_PATH = 'C:\\Users\Aeryes\PycharmProjects\simplecnn\model'

trans = transforms.Compose([
    transforms.RandomHorizontalFlip(),
    transforms.Resize(32),
    transforms.CenterCrop(32),
    transforms.ToTensor(),
    transforms.Normalize((0.5, 0.5, 0.5),(0.5, 0.5, 0.5))
    ])

# Flowers dataset.
train_dataset = datasets.ImageFolder(root=DATA_PATH_TRAIN, transform=trans)
test_dataset = datasets.ImageFolder(root=DATA_PATH_TEST, transform=trans)

# Create custom random sampler class to iter over dataloader.
train_loader = DataLoader(dataset=train_dataset, batch_size=batch_size, shuffle=True, num_workers=num_of_workers)
test_loader = DataLoader(dataset=test_dataset, batch_size=batch_size, shuffle=False, num_workers=num_of_workers)

# CNN we are going to implement.
class Unit(nn.Module):
    def __init__(self, in_channels, out_channels):
        super(Unit, self).__init__()

        self.conv = nn.Conv2d(in_channels=in_channels, kernel_size=3, out_channels=out_channels, stride=1, padding=1)
        self.bn = nn.BatchNorm2d(num_features=out_channels)
        self.relu = nn.ReLU()

    def forward(self, input):
        output = self.conv(input)
        output = self.bn(output)
        output = self.relu(output)

        return output

class CNNet(nn.Module):
    def __init__(self, num_class):
        super(CNNet, self).__init__()

        # Create 14 layers of the unit with max pooling in between
        self.unit1 = Unit(in_channels=3, out_channels=32)
        self.unit2 = Unit(in_channels=32, out_channels=32)
        self.unit3 = Unit(in_channels=32, out_channels=32)

        self.pool1 = nn.MaxPool2d(kernel_size=2)

        self.unit4 = Unit(in_channels=32, out_channels=64)
        self.unit5 = Unit(in_channels=64, out_channels=64)
        self.unit6 = Unit(in_channels=64, out_channels=64)
        self.unit7 = Unit(in_channels=64, out_channels=64)

        self.pool2 = nn.MaxPool2d(kernel_size=2)

        self.unit8 = Unit(in_channels=64, out_channels=128)
        self.unit9 = Unit(in_channels=128, out_channels=128)
        self.unit10 = Unit(in_channels=128, out_channels=128)
        self.unit11 = Unit(in_channels=128, out_channels=128)

        self.pool3 = nn.MaxPool2d(kernel_size=2)

        self.unit12 = Unit(in_channels=128, out_channels=128)
        self.unit13 = Unit(in_channels=128, out_channels=128)
        self.unit14 = Unit(in_channels=128, out_channels=128)

        self.avgpool = nn.AvgPool2d(kernel_size=4)

        # Add all the units into the Sequential layer in exact order
        self.net = nn.Sequential(self.unit1, self.unit2, self.unit3, self.pool1, self.unit4, self.unit5, self.unit6
                                 , self.unit7, self.pool2, self.unit8, self.unit9, self.unit10, self.unit11, self.pool3,
                                 self.unit12, self.unit13, self.unit14, self.avgpool)

        self.fc = nn.Linear(in_features=128, out_features=num_class)

    def forward(self, input):
        output = self.net(input)
        output = output.view(-1, 128)
        output = self.fc(output)
        return output

# Check if gpu support is available
cuda_avail = torch.cuda.is_available()

# Create model, optimizer and loss function
model = CNNet(num_classes)

# if cuda is available, move the model to the GPU
if cuda_avail:
    model.cuda()

# Define the optimizer and loss function
optimizer = Adam(model.parameters(), lr=0.0001, weight_decay=0.0001)
loss_fn = nn.CrossEntropyLoss()

def save_models(epoch):
    torch.save(model.state_dict(), f"flowermodel_{epoch}.model")
    print("Checkpoint saved")

def test():
    model.eval()
    test_acc = 0.0
    for i, (images, labels) in enumerate(test_loader):

        if cuda_avail:
            images = Variable(images.cuda())
            labels = Variable(labels.cuda())

        # Predict classes using images from the test set
        outputs = model(images)
        _, prediction = torch.max(outputs.data, 1)

        test_acc += torch.sum(prediction == labels.data).float()

    # Compute the average acc and loss over all 10000 test images
    test_acc = test_acc / 4242 * 100

    return test_acc

def train(num_epoch):
    best_acc = 0.0

    for epoch in range(num_epoch):
        model.train()
        train_acc = 0.0
        train_loss = 0.0
        for i, (images, labels) in enumerate(train_loader):
            # Move images and labels to gpu if available
            if cuda_avail:
                images = Variable(images.cuda())
                labels = Variable(labels.cuda())

            # Clear all accumulated gradients
            optimizer.zero_grad()
            # Predict classes using images from the test set
            outputs = model(images)
            # Compute the loss based on the predictions and actual labels
            loss = loss_fn(outputs, labels)
            # Backpropagate the loss
            loss.backward()

            # Adjust parameters according to the computed gradients
            optimizer.step()

            train_loss += loss.cpu().data[0] * images.size(0)
            _, prediction = torch.max(outputs.data, 1)

            train_acc += torch.sum(prediction == labels.data).float()

        # Call the learning rate adjustment function
        #adjust_learning_rate(epoch)

        # Compute the average acc and loss over all 50000 training images
        train_acc = train_acc / 4242 * 100
        train_loss = train_loss / 8484

        # Evaluate on the test set
        test_acc = test()

        # Save the model if the test acc is greater than our current best
        if test_acc > best_acc:
            save_models(epoch)
            best_acc = test_acc

        # Print the metrics
        print(f"Epoch {epoch + 1}, Train Accuracy: {train_acc} , TrainLoss: {train_loss} , Test Accuracy: {test_acc}")

if __name__ == '__main__':
    freeze_support()
    train(num_epochs)

This is my image loader to view preprocessed images:

from multiprocessing import freeze_support

import torch
from torch import nn
import torchvision
from torch.autograd import Variable
from torch.utils.data import DataLoader, Sampler
from torchvision import datasets
from torchvision.transforms import transforms
from torch.optim import Adam

import matplotlib.pyplot as plt
import numpy as np
import PIL

num_classes = 5
batch_size = 100
num_of_workers = 5

DATA_PATH_TRAIN = 'C:\\Users\Aeryes\PycharmProjects\simplecnn\images\\train'
DATA_PATH_TEST = 'C:\\Users\Aeryes\PycharmProjects\simplecnn\images\\test'

trans = transforms.Compose([
    transforms.RandomHorizontalFlip(),
    transforms.Resize(32),
    transforms.CenterCrop(32),
    transforms.ToTensor(),
    transforms.Normalize((0.5, 0.5, 0.5),(0.5, 0.5, 0.5))
    ])

train_dataset = datasets.ImageFolder(root=DATA_PATH_TRAIN, transform=trans)
train_loader = DataLoader(dataset=train_dataset, batch_size=batch_size, shuffle=True, num_workers=num_of_workers)

def imshow(img):
    img = img / 2 + 0.5 # unnormalize
    #npimg = img.numpy()
    plt.imshow(np.transpose(img[0].numpy(), (1, 2, 0)))
    plt.show()

def main():
    # get some random training images
    dataiter = iter(train_loader)
    images, labels = dataiter.next()

    # show images
    imshow(images)

if __name__ == "__main__":
    main()

So far this is what I have for my classify new image file which will classify new images:

from multiprocessing import freeze_support

import torch
from torch import nn
from torch.autograd import Variable
from torch.utils.data import DataLoader, Sampler
from torchvision import datasets
from torchvision.transforms import transforms
from torch.optim import Adam

import matplotlib.pyplot as plt
import numpy as np

def classify_new_image():
    # Classify a new image using a pretrained model from the above training.

    # Location of the image we will classify.
    IMG_PATH = "C:\\Users\\Aeryes\\PycharmProjects\\simplecnn\\images\\pretrain_classify\\"

    # Pre-processing the new image using transform.
    min_img_size = 32
    trans = transforms.Compose([transforms.Resize(min_img_size),
                                         transforms.CenterCrop(32),
                                         transforms.ToTensor(),
                                         transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                                              std=[0.229, 0.224, 0.225])])

# Picture dataset.
classify_dataset = datasets.ImageFolder(root=IMG_PATH, transform=trans)
# Create custom random sampler class to iter over dataloader.
classify_loader = DataLoader(dataset=classify_dataset, batch_size=1, shuffle=True, num_workers=5)

# Check if gpu support is available
cuda_avail = torch.cuda.is_available()

model = torch.load('C:\\Users\\Aeryes\\PycharmProjects\\simplecnn\\src\\flowermodel_20.tar')['state_dict']

# if cuda is available, move the model to the GPU
if cuda_avail:
    model.cuda()

if __name__ == "__main__":
    classify_new_image()

A big issue that I am also facing is making sense of the outputs. I printed the prediction variable from the CNN models and it gave me a tensor of numbers ranging from 0-4, which I presume is the 5 classes I have in my data folders. If anyone can help me make sense of this a I would be very grateful.

Here is my direct question: How do I use my pre-trained model to predict never before seen images of flowers?


title: How to get an output dimension for each layer of the Neural Network in Pytorch?

body:
class Model(nn.Module):
  def __init__(self):
    super(Model, self).__init__()
    self.net = nn.Sequential(
      nn.Conv2d(in_channels = 3, out_channels = 16),
      nn.ReLU(),
      nn.MaxPool2d(2),
      nn.Conv2d(in_channels = 16, out_channels = 16),
      nn.ReLU(),
      Flatten(),
      nn.Linear(4096, 64),
      nn.ReLU(),
      nn.Linear(64, 10))

  def forward(self, x):
    return self.net(x)

I have created this model without a firm knowledge in Neural Network and I just fixed parameters until it worked in the training. I am not sure how to get the output dimension for each layer (e.g. output dimension after the first layer).

Is there an easy way to do this in Pytorch?


I am getting the following error.

Traceback (most recent call last):
  File "main.py", line 63, in <module>
    question_classifier = QuestionClassifier(corpus.dictionary, embeddings_index, corpus.max_sent_length, args)
  File "/net/if5/wua4nw/wasi/academic/research_with_prof_chang/projects/question_answering/duplicate_question_detection/source/question_classifier.py", line 26, in __init__
    self.embedding = EmbeddingLayer(len(dictionary), args.emsize, args.dropout)
  File "/if5/wua4nw/anaconda3/lib/python3.5/site-packages/torch/nn/modules/module.py", line 255, in __setattr__
    "cannot assign module before Module.__init__() call")
AttributeError: cannot assign module before Module.__init__() call

I have a class as follows.

class QuestionClassifier(nn.Module):

     def __init__(self, dictionary, embeddings_index, max_seq_length, args):
         self.embedding = EmbeddingLayer(len(dictionary), args.emsize, args.dropout)
         self.encoder = EncoderRNN(args.emsize, args.nhid, args.model, args.bidirection, args.nlayers, args.dropout)
         self.drop = nn.Dropout(args.dropout)

So, when I run the following line:

question_classifier = QuestionClassifier(corpus.dictionary, embeddings_index, corpus.max_sent_length, args)

I get the above mentioned error. Here, EmbeddingLayer and EncoderRNN is a class written by me which inherits nn.module like the QuestionClassifier class.

What I am doing wrong here?


title: pytorch: "multi-target not supported" error message

body:
so I want to classify some (3,50,50) pictures. First I loaded the dataset from the file without a dataloader or batches, it worked. Now, after adding both things I get that error:

RuntimeError: multi-target not supported at /pytorch/aten/src/THCUNN/generic/ClassNLLCriterion.cu:15

I found a lot of answers in the internet, mostly to use "target.squeeze(1)" but it doesn´t work for me.
My target-batch looks like following:

tensor([[1, 0],
        [1, 0],
        [1, 0],
        [1, 0],
        [1, 0],
        [1, 0],
        [1, 0],
        [1, 0]], device='cuda:0')

Shouldnt that be okay?

Here the full code (notice that Im only creating the structure of the model on which Im going to apply the full and correct dataset afterwards, because I dont have the full data yet, only 32 pictures and no labels, thats why I added "torch.tensor([1, 0]) as a placeholder for all labels):

import torch
import torch.utils.data
import torch.nn as nn
import torch.nn.functional as F
import torch.optim
from torch.autograd import Variable

import numpy as np
from PIL import Image

class Model(nn.Module):

    def __init__(self):
        super(Model, self).__init__()

        # model structur:
        self.conv1 = nn.Conv2d(3, 10, kernel_size=(5,5),  stride=(1,1))
        self.conv2 = nn.Conv2d(10, 20, kernel_size=(5,5),  stride=(1,1))            # with mapool: output = 20 * (9,9) feature-maps -> flatten
        self.fc1 = nn.Linear(20*9*9, 250)
        self.fc2 = nn.Linear(250, 100)
        self.fc3 = nn.Linear(100, 2)

    def forward(self, x):

        # conv layers
        x = F.relu(self.conv1(x))   # shape: 1, 10, 46, 46
        x = F.max_pool2d(x, 2, 2)   # shape: 1, 10, 23, 23
        x = F.relu(self.conv2(x))   # shape: 1, 20, 19, 19
        x = F.max_pool2d(x, 2, 2)   # shape: 1, 20, 9, 9

        # flatten to dense layer:
        x = x.view(-1, 20*9*9)

        # dense layers
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        output = F.log_softmax(self.fc3(x), dim=1)
        return output

class Run:

    def __init__(self, epochs, learning_rate, dropout, momentum):

        # load model
        self.model = Model().cuda()

        # hyperparameters:
        self.epochs = epochs
        self.learning_rate = learning_rate
        self.dropout = dropout

    def preporcessing(self):

        dataset_folder = "/media/theodor/hdd/Programming/BWKI/dataset/bilder/"

        dataset = []

        for i in range(0, 35):

            sample_image = Image.open(dataset_folder + str(i) + ".png")
            data = torch.from_numpy(np.array(sample_image)).type("torch.Tensor").reshape(3, 50, 50)
            target = torch.tensor([[1, 0]])

            sample = (data, target)

            dataset.append(sample)

        train_loader = torch.utils.data.DataLoader(dataset, batch_size=8)

        return train_loader

    def train(self):

        train_set = self.preporcessing()

        criterion = nn.CrossEntropyLoss()
        optimizer = torch.optim.SGD(self.model.parameters(), lr=self.learning_rate)

        for epoch in range(self.epochs):

            epoch_loss = 0
            for i, data in enumerate(train_set, 0):

                sample, target = data
                # set data as cuda varibale
                sample = Variable(sample.float().cuda())
                target = Variable(target.cuda())
                # initialize optimizer
                optimizer.zero_grad()
                # predict
                output = self.model(sample)
                # backpropagation
                print(output, target.squeeze(1))
                loss = criterion(output, target.squeeze(1))    # ERROR MESSAGE: RuntimeError: multi-target not supported at /pytorch/aten/src/THCUNN/generic/ClassNLLCriterion.cu:15
                loss.backward()
                optimizer.step()

                epoch_loss += loss.item()

            print("loss after epoch [", epoch, "|", self.epochs, "] :", epoch_loss)

    def test(self):
        pass

run = Run(10, 0.001, 0.5, 0.9)
run.train()

So I expected it to start training (of course not learning anything because the labels are wrong),
Thanks in advance!


title: Unique values in PyTorch tensor

body: I'm tying to find distinct values in a PyTorch tensor. Is there an efficient analogue of Tensorflow's unique op?


title: Difference between tensor.permute and tensor.view in PyTorch?

body:
What is the difference between tensor.permute() and tensor.view()?

They seem to do the same thing.


title: How to iterate over layers in Pytorch

body:
Let's say I have a network model object called m. Now I have no prior information about the number of layers this network has. How can create a for loop to iterate over its layer?
I am looking for something like:

Weight=[]
for layer in m._modules:
    Weight.append(layer.weight)


title: pytorch - connection between loss.backward() and optimizer.step()

body:
Where is an explicit connection between the optimizer and the loss?

How does the optimizer know where to get the gradients of the loss without a call liks this optimizer.step(loss)?

-More context-

When I minimize the loss, I didn't have to pass the gradients to the optimizer.

loss.backward() # Back Propagation
optimizer.step() # Gardient Descent


title: pytorch error: multi-target not supported in CrossEntropyLoss()

body:
I am on a project using acceleration data to predict some activities.
But I have problems on the loss calculation. I am using CrossEntropyLoss for it.

Data is used for it like below
I use the first 4 data of each rows to predict the index like the last one of each rows.

1 84 84 81 4
81 85 85 80 1
81 82 84 80 1
1 85 84 2 0
81 85 82 80 1
81 82 84 80 1
81 25 84 80 5

The error messages are like below.

minoh@minoh-VirtualBox:~/cow$ python lec5.py
Traceback (most recent call last):
  File "lec5.py", line 97, in <module>
    train(epoch)
  File "lec5.py", line 74, in train
    loss = criterion(y_pred, labels)
  File "/home/minoh/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 357, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/minoh/anaconda3/lib/python3.6/site-packages/torch/nn/modules/loss.py", line 679, in forward
    self.ignore_index, self.reduce)
  File "/home/minoh/anaconda3/lib/python3.6/site-packages/torch/nn/functional.py", line 1161, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, size_average, ignore_index, reduce)
  File "/home/minoh/anaconda3/lib/python3.6/site-packages/torch/nn/functional.py", line 1052, in nll_loss
    return torch._C._nn.nll_loss(input, target, weight, size_average, ignore_index, reduce)
RuntimeError: multi-target not supported at /opt/conda/conda-bld/pytorch_1518243271935/work/torch/lib/THNN/generic/ClassNLLCriterion.c:22

My code is based on Sung Kim's pytorch

import numpy as np
import torch
from torch.autograd import Variable
import torch.nn.functional as F
from torch.utils.data import Dataset, DataLoader
import torch.nn as nn
import torch.optim as optim
from torchvision import datasets, transforms

class CowDataset(Dataset):
    def __init__(self):
        xy_str = np.loadtxt('cow_test', delimiter = ' ', dtype = np.str)
        xy = xy_str.astype(np.float32)
        xy_int = xy_str.astype(np.int)
        self.len = xy.shape[0]
        self.x_data = torch.from_numpy(xy[:, 0:4])
        self.y_data = torch.from_numpy(xy_int[:, [4]])

    def __getitem__(self, index):
        return self.x_data[index], self.y_data[index]

    def __len__(self):
        return self.len

dataset = CowDataset()
train_loader = DataLoader(dataset = dataset, batch_size = 32, shuffle = True)

class CowTestset(Dataset):
        def __init__(self):
                xy_str = np.loadtxt('cow_test2', delimiter = ' ', dtype =np.str)
                xy = xy_str.astype(np.float32)
                xy_int = xy_str.astype(np.int)
                self.len = xy.shape[0]
                self.x_data = torch.from_numpy(xy[:, 0:4])
                self.y_data = torch.from_numpy(xy_int[:, [4]])

        def __getitem__(self, index):
                return self.x_data[index], self.y_data[index]

        def __len__(self):
                return self.len

testset = CowTestset()
test_loader = DataLoader(dataset = testset, batch_size = 32, shuffle = True)

class Model(torch.nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.l1 = torch.nn.Linear(4,5)
        self.l2 = torch.nn.Linear(5,7)
        self.l3 = torch.nn.Linear(7,6)
        self.sigmoid = torch.nn.Sigmoid()

    def forward(self, x):
        out1 = self.sigmoid(self.l1(x))
        out2 = self.sigmoid(self.l2(out1))
        y_pred = self.sigmoid(self.l3(out2))
        return y_pred

model = Model()
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(model.parameters(), lr = 0.1, momentum = 0.5)

def train(epoch):
    model.train()
    for batch_idx, (inputs, labels) in enumerate(train_loader):
        inputs, labels = Variable(inputs), Variable(labels)
        optimizer.zero_grad()
        y_pred = model(inputs)
        loss = criterion(y_pred, labels)
        loss.backward()
        optimizer.step()
        if batch_idx % 10 == 0:
            print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                epoch, batch_idx * len(data), len(train_loader.dataset),
                100. * batch_idx / len(train_loader), loss.data[0]))

def test():
    model.eval()
    test_loss = 0
    correct = 0
    for data, target in test_loader:
        data, target = Variable(data, volatile = True), Variable(target)
        print(target)
        output = model(data)
        test_loss += criterion(output, target).data[0]
        pred = output.data.max(1, keepdim = True)[1]
        correct += pred.eq(target.data.view_as(pred)).cpu().sum()
    test_loss /= len(test_loader.dataset)
    print('\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\n'.format(test_loss, correct, len(test_loader.dataset), 100.* correct / len(test_loader.dataset)))

for epoch in range(1,7):
    train(epoch)
    test()


title: PyTorch - How to get learning rate during training?

body:
While training, I'd like to know the value of learning_rate.
What should I do?

It's my code, like this:

my_optimizer = torch.optim.SGD(my_model.parameters(),
                               lr=0.001,
                               momentum=0.99,
                               weight_decay=2e-3)

Thank you.


title: k-fold cross validation using DataLoaders in PyTorch

body:
I have splitted my training dataset into 80% train and 20% validation data and created DataLoaders as shown below. However I do not want to limit my model's training. So I thought of splitting my data into K(maybe 5) folds and performing cross-validation. However I do not know how to combine the datasets to my dataloader after splitting them.

train_size = int(0.8 * len(full_dataset))
validation_size = len(full_dataset) - train_size
train_dataset, validation_dataset = random_split(full_dataset, [train_size, validation_size])

full_loader = DataLoader(full_dataset, batch_size=4,sampler = sampler_(full_dataset), pin_memory=True)
train_loader = DataLoader(train_dataset, batch_size=4, sampler = sampler_(train_dataset))
val_loader = DataLoader(validation_dataset, batch_size=1, sampler = sampler_(validation_dataset))

Thank you in advance !


title: How to convert Pytorch autograd.Variable to Numpy?

body:
The title says it all. I want to convert a PyTorch autograd.Variable to its equivalent numpy array. In their official documentation they advocated using a.numpy() to get the equivalent numpy array (for PyTorch tensor). But this gives me the following error:

  Traceback (most recent call last): File "stdin", line 1, in module
  File
  "/home/bishwajit/anaconda3/lib/python3.6/site-packages/torch/autograd/variable.py",
  line 63, in getattr    raise AttributeError(name) AttributeError:
  numpy

Is there any way I can circumvent this?


title: PyTorch transforms on TensorDataset

body:
I'm using TensorDataset to create dataset from numpy arrays.

# convert numpy arrays to pytorch tensors
X_train = torch.stack([torch.from_numpy(np.array(i)) for i in X_train])
y_train = torch.stack([torch.from_numpy(np.array(i)) for i in y_train])

# reshape into [C, H, W]
X_train = X_train.reshape((-1, 1, 28, 28)).float()

# create dataset and dataloaders
train_dataset = torch.utils.data.TensorDataset(X_train, y_train)
train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=64)

How do I apply data augmentation (transforms) to TensorDataset?

For example, using ImageFolder, I can specify transforms as one of its parameters torchvision.datasets.ImageFolder(root, transform=...).

According to this reply by one of PyTorch's team members, it's not supported by default. Is there any alternative way to do so?

Feel free to ask if more code is needed to explain the problem.


title: PyTorch get all layers of model

body:
What's the easiest way to take a pytorch model and get a list of all the layers without any nn.Sequence groupings? For example, a better way to do this?
import pretrainedmodels

def unwrap_model(model):
    for i in children(model):
        if isinstance(i, nn.Sequential): unwrap_model(i)
        else: l.append(i)

model = pretrainedmodels.__dict__['xception'](num_classes=1000, pretrained='imagenet')
l = []
unwrap_model(model)

print(l)


title: Debugging in Google Colab

body:
I am running the following code snippet in google colab in a single cell:

%debug
# Create tensors of shape (10, 3) and (10, 2).
x = torch.randn(10, 3)
y = torch.randn(10, 2)

# Build a fully connected layer.
linear = nn.Linear(3, 2)
print ('w: ', linear.weight)
print ('b: ', linear.bias)

I wish to debug a piece of code (step through it line by line) to understand what is going on. I wish to step inside the function nn.Linear.

However, when I step through, it does not enter the function at  all. Is there a way to step through nn.Linear line by line? Also, how exactly do I set a breakpoint in nn.Linear? Besides, I wish to step though the snippet line by line as well. However, as the picture shows, the step command automatically steps through and executes the print statement as well.


title: PyTorch element-wise product of vectors / matrices / tensors

body:
In PyTorch, how do I get the element-wise product of two vectors / matrices / tensors?

For googlers, this is product is also known as:

Hadamard product
Schur product
Entrywise product


title: Label Smoothing in PyTorch

body:
I'm building a ResNet-18 classification model for the Stanford Cars dataset using transfer learning. I would like to implement label smoothing to penalize overconfident predictions and improve generalization.
TensorFlow has a simple keyword argument in CrossEntropyLoss. Has anyone built a similar function for PyTorch that I could plug-and-play with?


title: PyTorch - Getting the 'TypeError: pic should be PIL Image or ndarray. Got ' error

body:
I am getting the error TypeError: pic should be PIL Image or ndarray. Got <class 'numpy.ndarray'> when I try to load a non-image dataset through the DataLoader. The versions of torch and torchvision are 1.0.1, and 0.2.2.post3, respectively. Python's version is 3.7.1 on a Windows 10 machine.

Here is the code:

class AndroDataset(Dataset):
    def __init__(self, csv_path):
        self.transform = transforms.Compose([transforms.ToTensor()])

        csv_data = pd.read_csv(csv_path)

        self.csv_path = csv_path
        self.features = []
        self.classes = []

        self.features.append(csv_data.iloc[:, :-1].values)
        self.classes.append(csv_data.iloc[:, -1].values)

    def __getitem__(self, index):
        # the error occurs here
        return self.transform(self.features[index]), self.transform(self.classes[index])

    def __len__(self):
        return len(self.features)

And I set the loader:

training_data = AndroDataset('android.csv')
train_loader = DataLoader(dataset=training_data, batch_size=batch_size, shuffle=True)

Here is the full error stack trace:

Traceback (most recent call last):
  File "C:\Program Files\JetBrains\PyCharm 2018.1.2\helpers\pydev\pydevd.py", line 1758, in <module>
    main()
  File "C:\Program Files\JetBrains\PyCharm 2018.1.2\helpers\pydev\pydevd.py", line 1752, in main
    globals = debugger.run(setup['file'], None, None, is_module)
  File "C:\Program Files\JetBrains\PyCharm 2018.1.2\helpers\pydev\pydevd.py", line 1147, in run
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "C:\Program Files\JetBrains\PyCharm 2018.1.2\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "C:/Users/talha/Documents/PyCharmProjects/DeepAndroid/deep_test_conv1d.py", line 231, in <module>
    main()
  File "C:/Users/talha/Documents/PyCharmProjects/DeepAndroid/deep_test_conv1d.py", line 149, in main
    for i, (images, labels) in enumerate(train_loader):
  File "C:\Users\talha\Documents\PyCharmProjects\DeepAndroid\venv\lib\site-packages\torch\utils\data\dataloader.py", line 615, in __next__
    batch = self.collate_fn([self.dataset[i] for i in indices])
  File "C:\Users\talha\Documents\PyCharmProjects\DeepAndroid\venv\lib\site-packages\torch\utils\data\dataloader.py", line 615, in <listcomp>
    batch = self.collate_fn([self.dataset[i] for i in indices])
  File "C:/Users/talha/Documents/PyCharmProjects/DeepAndroid/deep_test_conv1d.py", line 102, in __getitem__
    return self.transform(self.features[index]), self.transform(self.classes[index])
  File "C:\Users\talha\Documents\PyCharmProjects\DeepAndroid\venv\lib\site-packages\torchvision\transforms\transforms.py", line 60, in __call__
    img = t(img)
  File "C:\Users\talha\Documents\PyCharmProjects\DeepAndroid\venv\lib\site-packages\torchvision\transforms\transforms.py", line 91, in __call__
    return F.to_tensor(pic)
  File "C:\Users\talha\Documents\PyCharmProjects\DeepAndroid\venv\lib\site-packages\torchvision\transforms\functional.py", line 50, in to_tensor
    raise TypeError('pic should be PIL Image or ndarray. Got {}'.format(type(pic)))
TypeError: pic should be PIL Image or ndarray. Got <class 'numpy.ndarray'>


title: In Colaboratory, CUDA cannot be used for the torch

body:
The error message is as follows:
RuntimeError   Traceback (most recent call last)
<ipython-input-24-06e96beb03a5> in <module>()
     11
     12 x_test = np.array(test_features)
---> 13 x_test_cuda = torch.tensor(x_test, dtype=torch.float).cuda()
     14 test = torch.utils.data.TensorDataset(x_test_cuda)
     15 test_loader = torch.utils.data.DataLoader(test, batch_size=batch_size, shuffle=False)

/usr/local/lib/python3.6/dist-packages/torch/cuda/__init__.py in _lazy_init()
    160 class CudaError(RuntimeError):
    161     def __init__(self, code):
--> 162         msg = cudart().cudaGetErrorString(code).decode('utf-8')
    163         super(CudaError, self).__init__('{0} ({1})'.format(msg, code))
    164

RuntimeError: cuda runtime error (38) : no CUDA-capable device is detected at /pytorch/aten/src/THC/THCGeneral.cpp:51


title: Can not get pytorch working with tensorboard

body:
I"m going through this tutorial to set up pytorch (v1.3.0 through conda) with tensorboard https://pytorch.org/tutorials/intermediate/tensorboard_tutorial.html#

but on the step

from torch.utils.tensorboard import SummaryWriter

# default `log_dir` is "runs" - we'll be more specific here
writer = SummaryWriter('runs/fashion_mnist_experiment_1')

I keep getting the error

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
C:\ProgramData\Anaconda3\envs\fastai_v1\lib\site-packages\torch\utils\tensorboard\__init__.py in
      1 try:
----> 2     from tensorboard.summary.writer.record_writer import RecordWriter  # noqa F401
      3 except ImportError:

ModuleNotFoundError: No module named 'tensorboard.summary'; 'tensorboard' is not a package

During handling of the above exception, another exception occurred:

ImportError                               Traceback (most recent call last)
c:\Users\matt\Documents\code\playground\tensorboard.py in
----> 1 from torch.utils.tensorboard import SummaryWriter
      2
      3 # default `log_dir` is "runs" - we'll be more specific here
      4 writer = SummaryWriter('runs/fashion_mnist_experiment_1')

C:\ProgramData\Anaconda3\envs\fastai_v1\lib\site-packages\torch\utils\tensorboard\__init__.py in
      2     from tensorboard.summary.writer.record_writer import RecordWriter  # noqa F401
      3 except ImportError:
----> 4     raise ImportError('TensorBoard logging requires TensorBoard with Python summary writer installed. '
      5                       'This should be available in 1.14 or above.')
      6 from .writer import FileWriter, SummaryWriter  # noqa F401

ImportError: TensorBoard logging requires TensorBoard with Python summary writer installed. This should be available in 1.14 or above.

Does anyone have any suggestions?


title: How to iterate over two dataloaders simultaneously using pytorch?

body:
I am trying to implement a Siamese network that takes in two images. I load these images and create two separate dataloaders.

In my loop I want to go through both dataloaders simultaneously so that I can train the network on both images.

for i, data in enumerate(zip(dataloaders1, dataloaders2)):

    # get the inputs
    inputs1 = data[0][0].cuda(async=True);
    labels1 = data[0][1].cuda(async=True);

    inputs2 = data[1][0].cuda(async=True);
    labels2 = data[1][1].cuda(async=True);

    labels1 = labels1.view(batchSize,1)
    labels2 = labels2.view(batchSize,1)

    # zero the parameter gradients
    optimizer.zero_grad()

    # forward + backward + optimize
    outputs1 = alexnet(inputs1)
    outputs2 = alexnet(inputs2)

The return value of the dataloader is a tuple.
However, when I try to use zip to iterate over them, I get the following error:

OSError: [Errno 24] Too many open files
Exception NameError: "global name 'FileNotFoundError' is not defined" in <bound method _DataLoaderIter.__del__ of <torch.utils.data.dataloader._DataLoaderIter object at 0x7f2d3c00c190>> ignored

Shouldn't zip work on all iterable items? But it seems like here I can't use it on dataloaders.

Is there any other way to pursue this? Or am I approaching the implementation of a Siamese network incorrectly?


title: How to apply the torch.inverse() function of PyTorch to every sample in the batch?

body:
This may seem like a basic question, but I am unable to work it through.

In the forward pass of my neural network, I have an output tensor of shape 8x3x3, where 8 is my batch size. We can assume each 3x3 tensor to be a non-singular matrix. I need to find the inverse of these matrices.
The PyTorch inverse() function only works on square matrices. Since I now have 8x3x3, how do I apply this function to every matrix in the batch in a differentiable manner?

If I iterate through the samples and append the inverses to a python list, which I then convert to a PyTorch tensor, should it be a problem during backprop? (I am asking since converting PyTorch tensors to numpy to perform some operations and then back to a tensor won't compute gradients during backprop for such operations)

I also get the following error when I try to do something like that.

a = torch.arange(0,8).view(-1,2,2)
b = [m.inverse() for m in a]
c = torch.FloatTensor(b)

  TypeError: 'torch.FloatTensor' object does not support indexing


title: Is there any way I can download the pre-trained models available in PyTorch to a specific path?

body: I am referring to the models that can be found here: https://pytorch.org/docs/stable/torchvision/models.html#torchvision-models


title: TypeError: tensor is not a torch image

body:
While working through the AI course at Udacity I came across this error during the Transfer Learning section. Here is the code that seems to be causing the trouble:

import torch
from torch import nn
from torch import optim
import torch.nn.functional as F
from torchvision import datasets, transforms, models

data_dir = 'filename'

# TODO: Define transforms for the training data and testing data
train_transforms= transforms.Compose([transforms.Resize((224,224)), transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]), transforms.ToTensor()])
test_transforms= transforms.Compose([transforms.Resize((224,224)), transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]), transforms.ToTensor()])

# Pass transforms in here, then run the next cell to see how the transforms look
train_data = datasets.ImageFolder(data_dir + '/train', transform=train_transforms)
test_data = datasets.ImageFolder(data_dir + '/test', transform=test_transforms)

trainloader = torch.utils.data.DataLoader(train_data, batch_size=64, shuffle=True)
testloader = torch.utils.data.DataLoader(test_data, batch_size=32)


title: What are Torch Scripts in PyTorch?

body:
I've just found that PyTorch docs expose something that is called Torch Scripts. However, I do not know:

When they should be used?
How they should be used?
What are their benefits?


title: How to correctly give inputs to Embedding, LSTM and Linear layers in PyTorch?

body:
I need some clarity on how to correctly prepare inputs for batch-training using different components of the torch.nn module. Specifically, I'm looking to create an encoder-decoder network for a seq2seq model.

Suppose I have a module with these three layers, in order:

nn.Embedding
nn.LSTM
nn.Linear

nn.Embedding

Input: batch_size * seq_length
Output: batch_size * seq_length * embedding_dimension

I don't have any problems here, I just want to be explicit about the expected shape of the input and output.

nn.LSTM

Input: seq_length * batch_size * input_size (embedding_dimension in this case)
Output: seq_length * batch_size * hidden_size
last_hidden_state: batch_size * hidden_size
last_cell_state: batch_size * hidden_size

To use the output of the Embedding layer as input for the LSTM layer, I need to transpose axis 1 and 2.

Many examples I've found online do something like x = embeds.view(len(sentence), self.batch_size , -1), but that confuses me. How does this view ensure that elements of the same batch remain in the same batch? What happens when len(sentence) and self.batch size are of same size?

nn.Linear

Input: batch_size x input_size (hidden_size of LSTM in this case or ??)
Output: batch_size x output_size

If I only need the last_hidden_state of LSTM, then I can give it as input to nn.Linear.

But if I want to make use of Output (which contains all intermediate hidden states as well) then I need to change nn.Linear's input size to seq_length * hidden_size and to use Output as input to Linear module I need to transpose axis 1 and 2 of output and then I can view with Output_transposed(batch_size, -1).

Is my understanding here correct? How do I carry out these transpose operations in tensors (tensor.transpose(0, 1))?


title: How to install torch in python

body:
I tried pip3 install torch --no-cache-dir and, after few seconds, I got this:

Collecting torch
      Downloading https://files.pythonhosted.org/packages/24/19/4804aea17cd136f1705a5e98a00618cb8f6ccc375ad8bfa437408e09d058/torch-1.4.0-cp36-cp36m-manylinux1_x86_64.whl (753.4MB)
        100% |████████████████████████████████| 753.4MB 5.7MB/s
    Exception:
    Traceback (most recent call last):
      File "/usr/lib/python3/dist-packages/pip/basecommand.py", line 215, in main
        status = self.run(options, args)
      File "/usr/lib/python3/dist-packages/pip/commands/install.py", line 342, in run
        requirement_set.prepare_files(finder)
      File "/usr/lib/python3/dist-packages/pip/req/req_set.py", line 380, in prepare_files
        ignore_dependencies=self.ignore_dependencies))
      File "/usr/lib/python3/dist-packages/pip/req/req_set.py", line 620, in _prepare_file
        session=self.session, hashes=hashes)
      File "/usr/lib/python3/dist-packages/pip/download.py", line 821, in unpack_url
        hashes=hashes
      File "/usr/lib/python3/dist-packages/pip/download.py", line 663, in unpack_http_url
        unpack_file(from_path, location, content_type, link)
      File "/usr/lib/python3/dist-packages/pip/utils/__init__.py", line 617, in unpack_file
        flatten=not filename.endswith('.whl')
      File "/usr/lib/python3/dist-packages/pip/utils/__init__.py", line 506, in unzip_file
        data = zip.read(name)
      File "/usr/lib/python3.6/zipfile.py", line 1338, in read
        return fp.read()
      File "/usr/lib/python3.6/zipfile.py", line 858, in read
        buf += self._read1(self.MAX_N)
      File "/usr/lib/python3.6/zipfile.py", line 948, in _read1
        data = self._decompressor.decompress(data, n)
    MemoryError

What should I do now to install PyTorch?

I tried almost every method mentioned on google. I am working on Ubuntu, I tried using conda too, but I am unable to use that package outside conda.


title: PyTorch - How to deactivate dropout in evaluation mode

body:
This is the model I defined it is a simple lstm with 2 fully connect layers.

import copy
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

class mylstm(nn.Module):
    def __init__(self,input_dim, output_dim, hidden_dim,linear_dim):
        super(mylstm, self).__init__()
        self.hidden_dim=hidden_dim
        self.lstm=nn.LSTMCell(input_dim,self.hidden_dim)
        self.linear1=nn.Linear(hidden_dim,linear_dim)
        self.linear2=nn.Linear(linear_dim,output_dim)
    def forward(self, input):
        out,_=self.lstm(input)
        out=nn.Dropout(p=0.3)(out)
        out=self.linear1(out)
        out=nn.Dropout(p=0.3)(out)
        out=self.linear2(out)
        return out

x_train and x_val are float dataframe with shape (4478,30), while y_train and y_val are float df with shape (4478,10)

    x_train.head()
Out[271]:
       0       1       2       3    ...        26      27      28      29
0  1.6110  1.6100  1.6293  1.6370   ...    1.6870  1.6925  1.6950  1.6905
1  1.6100  1.6293  1.6370  1.6530   ...    1.6925  1.6950  1.6905  1.6960
2  1.6293  1.6370  1.6530  1.6537   ...    1.6950  1.6905  1.6960  1.6930
3  1.6370  1.6530  1.6537  1.6620   ...    1.6905  1.6960  1.6930  1.6955
4  1.6530  1.6537  1.6620  1.6568   ...    1.6960  1.6930  1.6955  1.7040

[5 rows x 30 columns]

x_train.shape
Out[272]: (4478, 30)

Define the varible and do one time bp, I can find out the vaildation loss is 1.4941

model=mylstm(30,10,200,100).double()
from torch import optim
optimizer=optim.RMSprop(model.parameters(), lr=0.001, alpha=0.9)
criterion=nn.L1Loss()
input_=torch.autograd.Variable(torch.from_numpy(np.array(x_train)))
target=torch.autograd.Variable(torch.from_numpy(np.array(y_train)))
input2_=torch.autograd.Variable(torch.from_numpy(np.array(x_val)))
target2=torch.autograd.Variable(torch.from_numpy(np.array(y_val)))
optimizer.zero_grad()
output=model(input_)
loss=criterion(output,target)
loss.backward()
optimizer.step()
moniter=criterion(model(input2_),target2)

moniter
Out[274]: tensor(1.4941, dtype=torch.float64, grad_fn=<L1LossBackward>)

But I called forward function again I get a different number due to randomness of dropout

moniter=criterion(model(input2_),target2)
moniter
Out[275]: tensor(1.4943, dtype=torch.float64, grad_fn=<L1LossBackward>)

what should I do that I can eliminate all the dropout in predicting phrase?

I tried eval():

moniter=criterion(model.eval()(input2_),target2)
moniter
Out[282]: tensor(1.4942, dtype=torch.float64, grad_fn=<L1LossBackward>)

moniter=criterion(model.eval()(input2_),target2)
moniter
Out[283]: tensor(1.4945, dtype=torch.float64, grad_fn=<L1LossBackward>)

And pass an addtional parameter p to control dropout:

import copy
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
class mylstm(nn.Module):
    def __init__(self,input_dim, output_dim, hidden_dim,linear_dim,p):
        super(mylstm, self).__init__()
        self.hidden_dim=hidden_dim
        self.lstm=nn.LSTMCell(input_dim,self.hidden_dim)
        self.linear1=nn.Linear(hidden_dim,linear_dim)
        self.linear2=nn.Linear(linear_dim,output_dim)
    def forward(self, input,p):
        out,_=self.lstm(input)
        out=nn.Dropout(p=p)(out)
        out=self.linear1(out)
        out=nn.Dropout(p=p)(out)
        out=self.linear2(out)
        return out

model=mylstm(30,10,200,100,0.3).double()

output=model(input_)
loss=criterion(output,target)
loss.backward()
optimizer.step()
moniter=criterion(model(input2_,0),target2)
Traceback (most recent call last):

  File "<ipython-input-286-e49b6fac918b>", line 1, in <module>
    output=model(input_)

  File "D:\Users\shan xu\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)

TypeError: forward() missing 1 required positional argument: 'p'

But neither of them worked.


title: Why do we need to explicitly call zero_grad()?

body: Why do we need to explicitly zero the gradients in PyTorch? Why can't gradients be zeroed when loss.backward() is called? What scenario is served by keeping the gradients on the graph and asking the user to explicitly zero the gradients?


title: what's the difference between torch.Tensor() vs torch.empty() in pytorch?

body: I have tried it out as below. It seems to me they're the same. What's the difference between torch.Tensor() vs torch.empty() in pytorch?


title: PyTorch custom loss function

body:
How should a custom loss function be implemented ? Using below code is causing error :

import torch
import torch.nn as nn
import torchvision
import torchvision.transforms as transforms
import numpy as np
import matplotlib.pyplot as plt
import torch.utils.data as data_utils
import torch.nn as nn
import torch.nn.functional as F

num_epochs = 20

x1 = np.array([0,0])
x2 = np.array([0,1])
x3 = np.array([1,0])
x4 = np.array([1,1])

num_epochs = 200

class cus2(torch.nn.Module):

    def __init__(self):
        super(cus2,self).__init__()

    def forward(self, outputs, labels):
        # reshape labels to give a flat vector of length batch_size*seq_len
        labels = labels.view(-1)

        # mask out 'PAD' tokens
        mask = (labels >= 0).float()

        # the number of tokens is the sum of elements in mask
        num_tokens = int(torch.sum(mask).data[0])

        # pick the values corresponding to labels and multiply by mask
        outputs = outputs[range(outputs.shape[0]), labels]*mask

        # cross entropy loss for all non 'PAD' tokens
        return -torch.sum(outputs)/num_tokens

x = torch.tensor([x1,x2,x3,x4]).float()

y = torch.tensor([0,1,1,0]).long()

train = data_utils.TensorDataset(x,y)
train_loader = data_utils.DataLoader(train , batch_size=2 , shuffle=True)

device = 'cpu'

input_size = 2
hidden_size = 100
num_classes = 2

learning_rate = .0001

class NeuralNet(nn.Module) :
    def __init__(self, input_size, hidden_size, num_classes) :
        super(NeuralNet, self).__init__()
        self.fc1 = nn.Linear(input_size , hidden_size)
        self.relu = nn.ReLU()
        self.fc2 = nn.Linear(hidden_size , num_classes)

    def forward(self, x) :
        out = self.fc1(x)
        out = self.relu(out)
        out = self.fc2(out)
        return out

for i in range(0 , 1) :

        model = NeuralNet(input_size, hidden_size, num_classes).to(device)

        criterion = nn.CrossEntropyLoss()
#         criterion = Regress_Loss()
#         criterion = cus2()
        optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

        total_step = len(train_loader)
        for epoch in range(num_epochs) :
            for i,(images , labels) in enumerate(train_loader) :
                images = images.reshape(-1 , 2).to(device)
                labels = labels.to(device)

                outputs = model(images)
                loss = criterion(outputs , labels)

                optimizer.zero_grad()
                loss.backward()
                optimizer.step()
#                 print(loss)

        outputs = model(x)

        print(outputs.data.max(1)[1])

makes perfect predictions on training data :

tensor([0, 1, 1, 0])

Using a custom loss function from https://cs230-stanford.github.io/pytorch-nlp.html#writing-a-custom-loss-function :

is implemented in above code as cus2

Un-commenting code #         criterion = cus2() to use this loss function returns :

tensor([0, 0, 0, 0])

A warning is also returned :

  UserWarning: invalid index of a 0-dim tensor. This will be an error in
  PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python
  number

I've not implemented the custom loss function correctly ?


title: `*** RuntimeError: mat1 dim 1 must match mat2 dim 0` whenever I run model(images)

body:
def __init__(self):
        super().__init__()

        self.conv = nn.Sequential(
            nn.Conv2d(1, 64, kernel_size=5, stride=2, bias=False),
            nn.BatchNorm2d(64),
            nn.ReLU(),

            nn.Conv2d(64, 64, kernel_size=3, stride=2, bias=False),
            nn.BatchNorm2d(64),
            nn.ReLU(),

            nn.Conv2d(64, 64, kernel_size=3, stride=2, bias=False),
            nn.BatchNorm2d(64),

        )

How can I deal with this error? I think the error is with self.fc, but I can't say how to fix it.


ValueError                                Traceback (most recent call last)
<ipython-input-30-33821ccddf5f> in <module>
     23         output = model(data)
     24         # calculate the batch loss
---> 25         loss = criterion(output, target)
     26         # backward pass: compute gradient of the loss with respect to model parameters
     27         loss.backward()

C:\Users\mnauf\Anaconda3\envs\federated_learning\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    487             result = self._slow_forward(*input, **kwargs)
    488         else:
--> 489             result = self.forward(*input, **kwargs)
    490         for hook in self._forward_hooks.values():
    491             hook_result = hook(self, input, result)

C:\Users\mnauf\Anaconda3\envs\federated_learning\lib\site-packages\torch\nn\modules\loss.py in forward(self, input, target)
    593                                                   self.weight,
    594                                                   pos_weight=self.pos_weight,
--> 595                                                   reduction=self.reduction)
    596
    597

C:\Users\mnauf\Anaconda3\envs\federated_learning\lib\site-packages\torch\nn\functional.py in binary_cross_entropy_with_logits(input, target, weight, size_average, reduce, reduction, pos_weight)
   2073
   2074     if not (target.size() == input.size()):
-> 2075         raise ValueError("Target size ({}) must be the same as input size ({})".format(target.size(), input.size()))
   2076
   2077     return torch.binary_cross_entropy_with_logits(input, target, weight, pos_weight, reduction_enum)

ValueError: Target size (torch.Size([16])) must be the same as input size (torch.Size([16, 1]))

I am training a CNN. Working on the Horses vs humans dataset. This is my code. I am using criterion = nn.BCEWithLogitsLoss() and optimizer = optim.RMSprop(model.parameters(), lr=0.01). My final layer is self.fc2 = nn.Linear(512, 1). Out last neuron, will output 1 for horse and 0 for human, right? or should I choose 2 neurons for output?

16 is the batch size. Since the error says ValueError: Target size (torch.Size([16])) must be the same as input size (torch.Size([16, 1])). I don't understand, where do I need to make change, to rectify the error.


title: PyTorch: Add validation error in training

body:
I am using PyTorch to train a cnn model. Here is my Network architecture:

import torch
from torch.autograd import Variable
import torch.nn as nn
import torch.nn.functional as F
import torch.nn.init as I

    class Net(nn.Module):

        def __init__(self):
            super(Net, self).__init__()

            self.conv1 = nn.Conv2d(1, 32, 5)
            self.pool = nn.MaxPool2d(2,2)
            self.conv1_bn = nn.BatchNorm2d(32)
            self.conv2 = nn.Conv2d(32, 64, 5)
            self.conv2_drop = nn.Dropout2d()
            self.conv2_bn = nn.BatchNorm2d(64)
            self.fc1 = torch.nn.Linear(53*53*64, 256)
            self.fc2 = nn.Linear(256, 136)

        def forward(self, x):

            x = F.relu(self.conv1_bn(self.pool(self.conv1(x))))
            x = F.relu(self.conv2_bn(self.pool(self.conv2_drop(self.conv2(x)))))
            x = x.view(-1, 53*53*64)
            x = F.relu(self.fc1(x))
            x = F.dropout(x, training=self.training)
            x = self.fc2(x)

            return x

Then I train the model like below:

# prepare the net for training
    net.train()

    for epoch in range(n_epochs):  # loop over the dataset multiple times

        running_loss = 0.0

        # train on batches of data, assumes you already have train_loader
        for batch_i, data in enumerate(train_loader):
            # get the input images and their corresponding labels
            images = data['image']
            key_pts = data['keypoints']

            # flatten pts
            key_pts = key_pts.view(key_pts.size(0), -1)

            # wrap them in a torch Variable
            images, key_pts = Variable(images), Variable(key_pts)

            # convert variables to floats for regression loss
            key_pts = key_pts.type(torch.FloatTensor)
            images = images.type(torch.FloatTensor)

            # forward pass to get outputs
            output_pts = net(images)

            # calculate the loss between predicted and target keypoints
            loss = criterion(output_pts, key_pts)

            # zero the parameter (weight) gradients
            optimizer.zero_grad()

            # backward pass to calculate the weight gradients
            loss.backward()

            # update the weights
            optimizer.step()

            # print loss statistics
            running_loss += loss.data[0]

I am wondering if it is possible to add the validation error in the training? I mean something like this (validation split) in Keras:

myModel.fit(trainX, trainY, epochs=50, batch_size=1, verbose=2, validation_split = 0.1)


title: Using expand_dims in pytorch

body:
I'm trying to tile a length 18 1 hot vector into a 40x40 grid.

Looking at pytorch docs, expand dims seems to be what i need.

But I cannot get it to work.  Any idea what I'm doing wrong?

one_hot = torch.zeros(18).unsqueeze(0)
one_hot[0,1] = 1.0
one_hot
tensor([[0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]])
one_hot.expand(-1,-1,40,40)
Traceback (most recent call last):
  File "<input>", line 1, in <module>
RuntimeError: The expanded size of the tensor (40) must match the existing size (18) at non-singleton dimension 3

I'm expecting a tensor of shape (1, 18, 40,40)


title: How to change the picture size in PyTorch

body:
I'm trying to convert CNN Keras model for Emotion Recognition using FER2013 dataset to PyTorch model and I have following error:

Traceback (most recent call last):
  File "VGG.py", line 112, in <module>
    transfer.keras_to_pytorch(keras_network, pytorch_network)
  File "/home/eorg/NeuralNetworks/user/Project/model/nntransfer.py", line 121, in keras_to_pytorch
    pytorch_model.load_state_dict(state_dict)
  File "/home/eorg/.local/lib/python2.7/site-packages/torch/nn/modules/module.py", line 334, in load_state_dict
    own_state[name].copy_(param)
RuntimeError: inconsistent tensor size at /b/wheel/pytorch-src/torch/lib/TH/generic/THTensorCopy.c:51

I understood that the error is related to the shape of images. In Keras the input size is defined to be 48 by 48.

And my question is how to define in PyTorch models that of my pictures are the shape of 48x48? I couldn't find such function in the documentation and examples.

Any help would be useful!


title: How do I load custom image based datasets into Pytorch for use with a CNN?

body:
I have searched for hours on the internet to find a good solution to my issue. Here is some relevant background information to help you answer my question.

This is my first ever deep learning project and I have no idea what I am doing. I know the theory but not the practical elements.

The data that I am using can be found on kaggle at this link:
(https://www.kaggle.com/alxmamaev/flowers-recognition)

I am aiming to classify flowers based on the images provided in the dataset using a CNN.

Here is some sample code I have tried to use to load data in so far, this is my best attempt but as I mentioned I am clueless and Pytorch docs didn't offer much help that I could understand at my level.
(https://pastebin.com/fNLVW1UW)

    # Loads the images for use with the CNN.
def load_images(image_size=32, batch_size=64, root="../images"):
    transform = transforms.Compose([
        transforms.Resize(32),
        transforms.ToTensor(),
        transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

    train_set = datasets.ImageFolder(root=root, train=True, transform=transform)
    train_loader = torch.utils.data.DataLoader(train_set, batch_size=batch_size, shuffle=True, num_workers=2)

    return train_loader

# Defining variables for use with the CNN.
classes = ('daisy', 'dandelion', 'rose', 'sunflower', 'tulip')
train_loader_data = load_images()

# Training samples.
n_training_samples = 3394
train_sampler = SubsetRandomSampler(np.arange(n_training_samples, dtype=np.int64))

# Validation samples.
n_val_samples = 424
val_sampler = SubsetRandomSampler(np.arange(n_training_samples, n_training_samples + n_val_samples, dtype=np.int64))

# Test samples.
n_test_samples = 424
test_sampler = SubsetRandomSampler(np.arange(n_test_samples, dtype=np.int64))

Here are my direct questions that I require answers too:

How do I fix my code to load in the dataset in an 80/10/10 split for training/test/validation?
How do i create the required labels/classes for these images which are already divided by folders in /images ?


title: Anaconda, update Pytorch to the latest version 1.5

body:
How could I update PyTorch from 1.4 -> 1.5 using Anaconda either through terminal or navigator?

Updating Anaconda with conda update --all updated some of the packages, not all, PyTorch included.

Initially, I installed PyTorch by running conda install -c pytorch pytorch

From PyTorch Github page there's the command

conda install -c pytorch magma-cuda90 # or [magma-cuda92 | magma-cuda100 | magma-cuda101 ] depending on your cuda version

but I wonder if there's going to be any kind of conflict between the already installed version and this one.

Thanks


title: PyTorch: RuntimeError: Input, output and indices must be on the current device

body:
I am running a BERT model on torch. It's a multi-class sentiment classification task with about 30,000 rows. I have already put everything on cuda, but not sure why I'm getting the following run time error. Here is my code:
for epoch in tqdm(range(1, epochs+1)):

    model.train()

    loss_train_total = 0

    progress_bar = tqdm(dataloader_train, desc='Epoch {:1d}'.format(epoch), leave=False, disable=False)
    for batch in progress_bar:

        model.zero_grad()

        batch = tuple(b.to(device) for b in batch)

        inputs = {'input_ids':      batch[0],
                  'attention_mask': batch[1],
                  'labels':         batch[2],
                 }

        outputs = model(**inputs)

        loss = outputs[0]
        loss_train_total += loss.item()
        loss.backward()

        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)

        optimizer.step()
        scheduler.step()

        progress_bar.set_postfix({'training_loss': '{:.3f}'.format(loss.item()/len(batch))})

    torch.save(model.state_dict(), f'finetuned_BERT_epoch_{epoch}.model')

    tqdm.write(f'\nEpoch {epoch}')

    loss_train_avg = loss_train_total/len(dataloader_train)
    tqdm.write(f'Training loss: {loss_train_avg}')

    val_loss, predictions, true_vals = evaluate(dataloader_validation)
    val_f1 = f1_score_func(predictions, true_vals)
    tqdm.write(f'Validation loss: {val_loss}')
    tqdm.write(f'F1 Score (Weighted): {val_f1}')

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-67-9306225bb55a> in <module>()
     17                  }
     18
---> 19         outputs = model(**inputs)
     20
     21         loss = outputs[0]

8 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
   1850         # remove once script supports set_grad_enabled
   1851         _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-> 1852     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
   1853
   1854

RuntimeError: Input, output and indices must be on the current device

Any suggestions would be appreciated. Thanks!


title: Indexing a multi-dimensional tensor with a tensor in PyTorch

body:
I have the following code:

a = torch.randint(0,10,[3,3,3,3])
b = torch.LongTensor([1,1,1,1])

I have a multi-dimensional index b and want to use it to select a single cell in a. If b wasn't a tensor, I could do:

a[1,1,1,1]

Which returns the correct cell, but:

a[b]

Doesn't work, because it just selects a[1] four times.

How can I do this? Thanks


title: How to include batch size in pytorch basic example?

body:
I am new to pytorch. The following is the basic example of using nn module to train a simple one-layer model with some random data (from here)

import torch
N, D_in, H, D_out = 64, 1000, 100, 10

x = torch.randn(N, D_in)
y = torch.randn(N, D_out)

model = torch.nn.Sequential(
    torch.nn.Linear(D_in, H),
    torch.nn.ReLU(),
    torch.nn.Linear(H, D_out),
)
loss_fn = torch.nn.MSELoss(reduction='sum')
optimizer = torch.optim.Adam(model.parameters(), lr=1e-4)
for t in range(500):
    y_pred = model(x)

    loss = loss_fn(y_pred, y)
    print(t, loss.item())

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

As far as I understand, the batch size is equal to 1 in the example, in other words, a single point (out of 64) is used to calculate gradients and update parameters. My question is: how to modify this example to train the model with the batch size greater than one?


title: Loading Torch7 trained models (.t7) in PyTorch

body:
I am using Torch7 library for implementing neural networks.  Mostly, I rely on pre-trained models. In Lua I use torch.load function to load a model saved as torch .t7 file. I am curious about switching to PyTorch( http://pytorch.org) and I read the documents. I couldn't find any information regarding the mechanisms to load a pre-trained model. The only relevant information I was able to find is this page:http://pytorch.org/docs/torch.html

But the function torch.load described in the page seems to load a file saved with pickle. If someone has additional information on loading .t7 models in PyTorch, please share it here.


title: Convert Pytorch Tensor to Numpy Array using Cuda

body:
I would like to convert a Pytorch tensor to numpy array using cuda:

this is the code line while not using cuda:

  A = self.tensor.weight.data.numpy()

How can I do the same operation using cuda?
According to this: https://discuss.pytorch.org/t/how-to-transform-variable-into-numpy/104/3
it seems:

  A = self.tensor.weight.data.cpu().numpy()


title: Having issues with neural network training. Loss not decreasing

body:
I'm largely following this project but am doing a pixel-wise classification.  I have 8 classes and 9 band imagery.  My images are gridded into 9x128x128.  My loss is not reducing and training accuracy doesn't fluctuate much.  I'm guessing I have something wrong with the model.  Any advice is much appreciated!  I get at least 91% accuracy using random forest.

My classes are extremely unbalanced so I attempted to adjust training weights based on the proportion of classes within the training data.

# get model
learning_rate = 0.0001
model = unet.UNetSmall(8)
optimizer = optim.Adam(model.parameters(), lr=learning_rate)

# set up weights based on data proportion
weights = np.array([0.79594768, 0.07181202, 0.02347426, 0.0042031, 0.00366211, 0.00764327, 0.07003923, 0.02321833])
weights = (1 - weights)/7
print('Weights of training data based on proportion of the training labels.  Not compted here')
print(weights)
print(sum(weights))
criterion = nn.CrossEntropyLoss(weight = weight)
lr_scheduler = optim.lr_scheduler.StepLR(optimizer, step_size=5, gamma=0.1)

  Weights of training data based on proportion of the training labels.
  Not compted here [0.02915033 0.13259828 0.13950368 0.1422567
  0.14233398 0.14176525
   0.13285154 0.13954024]
  1.0000000000000002

I've normalized the data using the transforms.functional.normalize function.  I calculated the mean and standard deviation of the training data and added this augmentation to my data loader.

dataset_train = data_utils.SatIn(data_path, 'TrainValTest.csv', 'train', transform=transforms.Compose([aug.ToTensorTarget(), aug.NormalizeTarget(mean=popmean, std=popstd)]))

I augmented my training data in preprocessing by rotating and flipping the imagery.  1 image grid then became 8.

I checked that my training data matched my classes and everything checked out.
Since I'm using  8 classes I chose to use CrossEntropyLoss since it has Softmax built in.

Current model

class UNetSmall(nn.Module):
    """
    Main UNet architecture
    """
    def __init__(self, num_classes=1):
        super().__init__()
        # encoding
        self.conv1 = encoding_block(9, 32)
        self.maxpool1 = nn.MaxPool2d(kernel_size=2)
        self.conv2 = encoding_block(32, 64)
        self.maxpool2 = nn.MaxPool2d(kernel_size=2)
        self.conv3 = encoding_block(64, 128)
        self.maxpool3 = nn.MaxPool2d(kernel_size=2)
        self.conv4 = encoding_block(128, 256)
        self.maxpool4 = nn.MaxPool2d(kernel_size=2)

        # center
        self.center = encoding_block(256, 512)

        # decoding
        self.decode4 = decoding_block(512, 256)
        self.decode3 = decoding_block(256, 128)
        self.decode2 = decoding_block(128, 64)
        self.decode1 = decoding_block(64, 32)

        # final
        self.final = nn.Conv2d(32, num_classes, kernel_size=1)

    def forward(self, input):

        # encoding
        conv1 = self.conv1(input)
        maxpool1 = self.maxpool1(conv1)
        conv2 = self.conv2(maxpool1)
        maxpool2 = self.maxpool2(conv2)
        conv3 = self.conv3(maxpool2)
        maxpool3 = self.maxpool3(conv3)
        conv4 = self.conv4(maxpool3)
        maxpool4 = self.maxpool4(conv4)

        # center
        center = self.center(maxpool4)

        # decoding
        decode4 = self.decode4(conv4, center)
        decode3 = self.decode3(conv3, decode4)
        decode2 = self.decode2(conv2, decode3)
        decode1 = self.decode1(conv1, decode2)
        # final
        final = nn.functional.upsample(self.final(decode1), input.size()[2:], mode='bilinear')
        return final

Training method

def train(train_loader, model, criterion, optimizer, scheduler, epoch_num):

    correct = 0
    totalcount = 0

    scheduler.step()

    # iterate over data
    for idx, data in enumerate(tqdm(train_loader, desc="training")):
        # get the inputs and wrap in Variable
        if torch.cuda.is_available():
            inputs = Variable(data['sat_img'].cuda())
            labels = Variable(data['map_img'].cuda())
        else:
            inputs = Variable(data['sat_img'])
            labels = Variable(data['map_img'])

        optimizer.zero_grad()
        outputs = model(inputs)
        loss = criterion(outputs, labels.long())
        loss.backward()
        optimizer.step()

        test = torch.max(outputs.data, 1)[1] == labels.long()
        correct += test.sum().item()
        totalcount += test.size()[0] * test.size()[1] * test.size()[2]

    print('Training Loss: {:.4f}, Accuracy: {:.2f}'.format(loss.data[0], correct/totalcount))
    return {'train_loss': loss.data[0], 'train_acc' : correct/totalcount}

Training call in epoch loop

lr_scheduler.step()
train_metrics = train(train_dataloader, model, criterion, optimizer, lr_scheduler, epoch)

Some epoch iteration output

  #### Epoch 0/19

  ---------- training: 100%|████████████████████████████████████████████████████████████████████████|
  84/84 [00:17<00:00,  5.77it/s] Training Loss: 0.8901, Accuracy: 0.83
  Current elapsed time 2m 6s

  #### Epoch 1/19

  ---------- training: 100%|████████████████████████████████████████████████████████████████████████|
  84/84 [00:17<00:00,  5.72it/s] Training Loss: 0.7922, Accuracy: 0.83
  Current elapsed time 2m 24s

  #### Epoch 2/19

  ---------- training: 100%|████████████████████████████████████████████████████████████████████████|
  84/84 [00:18<00:00,  5.44it/s] Training Loss: 0.8753, Accuracy: 0.84
  Current elapsed time 2m 42s

  #### Epoch 3/19

  ---------- training: 100%|████████████████████████████████████████████████████████████████████████|
  84/84 [00:18<00:00,  5.53it/s] Training Loss: 0.7741, Accuracy: 0.84
  Current elapsed time 3m 1s