title: Why does `conda list cudnn` have no output after `conda install pytorch torchvision cudatoolkit=10.2 -c pytorch` installation

body:
Please feel free to vote "Reopen" at the bottom of this question. The reason is that I have marked this as a duplicate although the answers there are not clear enough for this question.
As soon as the question is reopened, I can add the the following answer: that cuDNN is an integrated part of the conda pytorch cudatoolkit installer and that it is listed together with pytorch, see py3.7_cuda102_cudnn7_0, and not as a stand-alone package that you might perhaps get by using pip or conda to install it explicitly (not recommended! See all answers of How to run pytorch with NVIDIA “cuda toolkit” version instead of the official conda “cudatoolkit” version?). With conda's cudatoolkit, cuDNN is automatically installed together with CUDA just for pytorch, and as a built-in package inside a conda package it does not show up.

I have installed pytorch using conda install pytorch torchvision cudatoolkit=10.2 -c pytorch.
https://github.com/pytorch/pytorch/issues/17445#issuecomment-466838819 says that cudNN is automatically included.
Trying to check the install according to How to get the CUDA version?, I used conda list cudnn to check, and it is not installed, even though cudNN is needed to use your gpu for ML, while conda list cuda works.
Here is the output of anaconda prompt of both commands.
(base) PS C:\Users\Admin> conda list cudnn
# packages in environment at C:\Users\Admin\anaconda3:
#
# Name                    Version                   Build  Channel
(base) PS C:\Users\Admin> conda list cuda
# packages in environment at C:\Users\Admin\anaconda3:
#
# Name                    Version                   Build  Channel
cudatoolkit               10.2.89              h74a9793_1

I find a 445 MB cudnn file "cudnn64_7.dll" in C:\Users\Admin\anaconda3\Lib\site-packages\torch\lib, and there are some small files in C:\Users\Admin\anaconda3\Lib\site-packages\torch\backends\cudnn and C:\Users\Admin\anaconda3\Lib\site-packages\torch\include\ATen.
Questions:

Is cudnn installed as a part of pytorch only?
Is there a way to get conda list cudnn to work?

####
Output of just conda list mentions cudnn at
pytorch                   1.6.0           py3.7_cuda102_cudnn7_0    pytorch

Full list:
# Name                    Version                   Build  Channel
_ipyw_jlab_nb_ext_conf    0.1.0                    py37_0
alabaster                 0.7.12                   py37_0
anaconda                  2020.07                  py37_0
anaconda-client           1.7.2                    py37_0
anaconda-navigator        1.9.12                   py37_0
anaconda-project          0.8.4                      py_0
argh                      0.26.2                   py37_0
asn1crypto                1.3.0                    py37_1
astroid                   2.4.2                    py37_0
astropy                   4.0.1.post1      py37he774522_1
atomicwrites              1.4.0                      py_0
attrs                     19.3.0                     py_0
autopep8                  1.5.3                      py_0
babel                     2.8.0                      py_0
backcall                  0.2.0                      py_0
backports                 1.0                        py_2
backports.functools_lru_cache 1.6.1                      py_0
backports.shutil_get_terminal_size 1.0.0                    py37_2
backports.tempfile        1.0                        py_1
backports.weakref         1.0.post1                  py_1
bcrypt                    3.1.7            py37he774522_1
beautifulsoup4            4.9.1                    py37_0
bitarray                  1.4.0            py37he774522_0
bkcharts                  0.2                      py37_0
blas                      1.0                         mkl
bleach                    3.1.5                      py_0
blosc                     1.19.0               h7bd577a_0
bokeh                     2.1.1                    py37_0
boto                      2.49.0                   py37_0
bottleneck                1.3.2            py37h2a96729_1
brotlipy                  0.7.0           py37he774522_1000
bzip2                     1.0.8                he774522_0
ca-certificates           2020.6.24                     0
certifi                   2020.6.20                py37_0
cffi                      1.14.0           py37h7a1dbc1_0
chardet                   3.0.4                 py37_1003
click                     7.1.2                      py_0
cloudpickle               1.5.0                      py_0
clyent                    1.2.2                    py37_1
colorama                  0.4.3                      py_0
comtypes                  1.1.7                 py37_1001
conda                     4.8.3                    py37_0
conda-build               3.18.11                  py37_0
conda-env                 2.6.0                         1
conda-package-handling    1.6.1            py37h62dcd97_0
conda-verify              3.4.2                      py_1
console_shortcut          0.1.1                         4
contextlib2               0.6.0.post1                py_0
cryptography              2.9.2            py37h7a1dbc1_0
cudatoolkit               10.2.89              h74a9793_1
curl                      7.71.1               h2a8f88b_1
cycler                    0.10.0                   py37_0
cython                    0.29.21          py37ha925a31_0
cytoolz                   0.10.1           py37he774522_0
dask                      2.20.0                     py_0
dask-core                 2.20.0                     py_0
decorator                 4.4.2                      py_0
defusedxml                0.6.0                      py_0
diff-match-patch          20200713                   py_0
distributed               2.20.0                   py37_0
docutils                  0.16                     py37_1
entrypoints               0.3                      py37_0
et_xmlfile                1.0.1                   py_1001
fastcache                 1.1.0            py37he774522_0
filelock                  3.0.12                     py_0
flake8                    3.8.3                      py_0
flask                     1.1.2                      py_0
freetype                  2.10.2               hd328e21_0
fsspec                    0.7.4                      py_0
future                    0.18.2                   py37_1
get_terminal_size         1.0.0                h38e98db_0
gevent                    20.6.2           py37he774522_0
glob2                     0.7                        py_0
gmpy2                     2.0.8            py37h0964b28_3
greenlet                  0.4.16           py37he774522_0
h5py                      2.10.0           py37h5e291fa_0
hdf5                      1.10.4               h7ebc959_0
heapdict                  1.0.1                      py_0
html5lib                  1.1                        py_0
icc_rt                    2019.0.0             h0cc432a_1
icu                       58.2                 ha925a31_3
idna                      2.10                       py_0
imageio                   2.9.0                      py_0
imagesize                 1.2.0                      py_0
importlib-metadata        1.7.0                    py37_0
importlib_metadata        1.7.0                         0
intel-openmp              2019.4                      245
intervaltree              3.0.2                      py_1
ipykernel                 5.3.2            py37h5ca1d4c_0
ipython                   7.16.1           py37h5ca1d4c_0
ipython_genutils          0.2.0                    py37_0
ipywidgets                7.5.1                      py_0
isort                     4.3.21                   py37_0
itsdangerous              1.1.0                    py37_0
jdcal                     1.4.1                      py_0
jedi                      0.17.1                   py37_0
jinja2                    2.11.2                     py_0
joblib                    0.16.0                     py_0
jpeg                      9b                   hb83a4c4_2
json5                     0.9.5                      py_0
jsonschema                3.2.0                    py37_1
jupyter                   1.0.0                    py37_7
jupyter_client            6.1.6                      py_0
jupyter_console           6.1.0                      py_0
jupyter_core              4.6.3                    py37_0
jupyterlab                2.1.5                      py_0
jupyterlab_server         1.2.0                      py_0
keyring                   21.2.1                   py37_0
kiwisolver                1.2.0            py37h74a9793_0
krb5                      1.18.2               hc04afaa_0
lazy-object-proxy         1.4.3            py37he774522_0
libarchive                3.4.2                h5e25573_0
libcurl                   7.71.1               h2a8f88b_1
libiconv                  1.15                 h1df5818_7
liblief                   0.10.1               ha925a31_0
libllvm9                  9.0.1                h21ff451_0
libpng                    1.6.37               h2a8f88b_0
libsodium                 1.0.18               h62dcd97_0
libspatialindex           1.9.3                h33f27b4_0
libssh2                   1.9.0                h7a1dbc1_1
libtiff                   4.1.0                h56a325e_1
libxml2                   2.9.10               h464c3ec_1
libxslt                   1.1.34               he774522_0
llvmlite                  0.33.0           py37ha925a31_0
locket                    0.2.0                    py37_1
lxml                      4.5.2            py37h1350720_0
lz4-c                     1.9.2                h62dcd97_0
lzo                       2.10                 he774522_2
m2w64-gcc-libgfortran     5.3.0                         6
m2w64-gcc-libs            5.3.0                         7
m2w64-gcc-libs-core       5.3.0                         7
m2w64-gmp                 6.1.0                         2
m2w64-libwinpthread-git   5.0.0.4634.697f757               2
markupsafe                1.1.1            py37hfa6e2cd_1
matplotlib                3.2.2                         0
matplotlib-base           3.2.2            py37h64f37c6_0
mccabe                    0.6.1                    py37_1
menuinst                  1.4.16           py37he774522_1
mistune                   0.8.4           py37hfa6e2cd_1001
mkl                       2019.4                      245
mkl-service               2.3.0            py37hb782905_0
mkl_fft                   1.0.14           py37h6288b17_0
mkl_random                1.0.4            py37h343c172_0
mock                      4.0.2                      py_0
more-itertools            8.4.0                      py_0
mpc                       1.1.0                h7edee0f_1
mpfr                      4.0.2                h62dcd97_1
mpir                      3.0.0                hec2e145_1
mpmath                    1.1.0                    py37_0
msgpack-python            1.0.0            py37h74a9793_1
msys2-conda-epoch         20160418                      1
multipledispatch          0.6.0                    py37_0
navigator-updater         0.2.1                    py37_0
nbconvert                 5.6.1                    py37_1
nbformat                  5.0.7                      py_0
networkx                  2.4                        py_1
ninja                     1.9.0            py37h74a9793_0
nltk                      3.5                        py_0
nose                      1.3.7                 py37_1004
notebook                  6.0.3                    py37_0
numba                     0.50.1           py37h47e9c7a_0
numexpr                   2.7.1            py37h25d0782_0
numpy                     1.17.0           py37h19fb1c0_0
numpy-base                1.17.0           py37hc3f5095_0
numpydoc                  1.1.0                      py_0
olefile                   0.46                     py37_0
openpyxl                  3.0.4                      py_0
openssl                   1.1.1g               he774522_0
packaging                 20.4                       py_0
pandas                    1.0.5            py37h47e9c7a_0
pandoc                    2.10                          0
pandocfilters             1.4.2                    py37_1
paramiko                  2.7.1                      py_0
parso                     0.7.0                      py_0
partd                     1.1.0                      py_0
path                      13.1.0                   py37_0
path.py                   12.4.0                        0
pathlib2                  2.3.5                    py37_1
pathtools                 0.1.2                      py_1
patsy                     0.5.1                    py37_0
pep8                      1.7.1                    py37_0
pexpect                   4.8.0                    py37_1
pickleshare               0.7.5                 py37_1001
pillow                    7.2.0            py37hcc1f983_0
pip                       20.1.1                   py37_1
pkginfo                   1.5.0.1                  py37_0
pluggy                    0.13.1                   py37_0
ply                       3.11                     py37_0
powershell_shortcut       0.0.1                         3
prometheus_client         0.8.0                      py_0
prompt-toolkit            3.0.5                      py_0
prompt_toolkit            3.0.5                         0
psutil                    5.7.0            py37he774522_0
py                        1.9.0                      py_0
py-lief                   0.10.1           py37ha925a31_0
pybson                    0.5.9                    pypi_0    pypi
pyclustering              0.9.3.1                  pypi_0    pypi
pycodestyle               2.6.0                      py_0
pycosat                   0.6.3            py37he774522_0
pycparser                 2.20                       py_2
pycrypto                  2.6.1           py37he774522_10
pycurl                    7.43.0.5         py37h7a1dbc1_0
pydocstyle                5.0.2                      py_0
pyflakes                  2.2.0                      py_0
pygments                  2.6.1                      py_0
pylint                    2.5.3                    py37_0
pymongo                   3.9.0            py37ha925a31_0
pynacl                    1.4.0            py37h62dcd97_1
pyodbc                    4.0.30           py37ha925a31_0
pyopenssl                 19.1.0                     py_1
pyparsing                 2.4.7                      py_0
pyqt                      5.9.2            py37h6538335_2
pyreadline                2.1                      py37_1
pyrsistent                0.16.0           py37he774522_0
pysocks                   1.7.1                    py37_1
pytables                  3.6.1            py37h1da0976_0
pytest                    5.4.3                    py37_0
python                    3.7.7                h81c818b_4
python-dateutil           2.8.1                      py_0
python-jsonrpc-server     0.3.4                      py_1
python-language-server    0.34.1                   py37_0
python-libarchive-c       2.9                        py_0
pytorch                   1.6.0           py3.7_cuda102_cudnn7_0    pytorch
pytz                      2020.1                     py_0
pywavelets                1.1.1            py37he774522_0
pywin32                   227              py37he774522_1
pywin32-ctypes            0.2.0                 py37_1001
pywinpty                  0.5.7                    py37_0
pyyaml                    5.3.1            py37he774522_1
pyzmq                     19.0.1           py37ha925a31_1
qdarkstyle                2.8.1                      py_0
qt                        5.9.7            vc14h73c81de_0
qtawesome                 0.7.2                      py_0
qtconsole                 4.7.5                      py_0
qtpy                      1.9.0                      py_0
regex                     2020.6.8         py37he774522_0
requests                  2.24.0                     py_0
rope                      0.17.0                     py_0
rtree                     0.9.4            py37h21ff451_1
ruamel_yaml               0.15.87          py37he774522_1
scikit-image              0.16.2           py37h47e9c7a_0
scikit-learn              0.23.1           py37h25d0782_0
scipy                     1.5.0            py37h9439919_0
seaborn                   0.10.1                     py_0
send2trash                1.5.0                    py37_0
setuptools                49.2.0                   py37_0
simplegeneric             0.8.1                    py37_2
singledispatch            3.4.0.3                  py37_0
sip                       4.19.8           py37h6538335_0
six                       1.15.0                     py_0
snappy                    1.1.8                h33f27b4_0
snowballstemmer           2.0.0                      py_0
sortedcollections         1.2.1                      py_0
sortedcontainers          2.2.2                      py_0
soupsieve                 2.0.1                      py_0
sphinx                    3.1.2                      py_0
sphinxcontrib             1.0                      py37_1
sphinxcontrib-applehelp   1.0.2                      py_0
sphinxcontrib-devhelp     1.0.2                      py_0
sphinxcontrib-htmlhelp    1.0.3                      py_0
sphinxcontrib-jsmath      1.0.1                      py_0
sphinxcontrib-qthelp      1.0.3                      py_0
sphinxcontrib-serializinghtml 1.1.4                      py_0
sphinxcontrib-websupport  1.2.3                      py_0
spyder                    4.1.4                    py37_0
spyder-kernels            1.9.2                    py37_0
sqlalchemy                1.3.18           py37he774522_0
sqlite                    3.32.3               h2a8f88b_0
statsmodels               0.11.1           py37he774522_0
sympy                     1.6.1                    py37_0
tbb                       2020.0               h74a9793_0
tblib                     1.6.0                      py_0
terminado                 0.8.3                    py37_0
testpath                  0.4.4                      py_0
threadpoolctl             2.1.0              pyh5ca1d4c_0
tk                        8.6.10               he774522_0
toml                      0.10.1                     py_0
toolz                     0.10.0                     py_0
torchvision               0.7.0                py37_cu102    pytorch
tornado                   6.0.4            py37he774522_1
tqdm                      4.47.0                     py_0
traitlets                 4.3.3                    py37_0
typed-ast                 1.4.1            py37he774522_0
typing_extensions         3.7.4.2                    py_0
ujson                     1.35             py37hfa6e2cd_0
unicodecsv                0.14.1                   py37_0
urllib3                   1.25.9                     py_0
vc                        14.1                 h0510ff6_4
vs2015_runtime            14.16.27012          hf0eaf9b_3
watchdog                  0.10.3                   py37_0
wcwidth                   0.2.5                      py_0
webencodings              0.5.1                    py37_1
werkzeug                  1.0.1                      py_0
wheel                     0.34.2                   py37_0
widgetsnbextension        3.5.1                    py37_0
win_inet_pton             1.1.0                    py37_0
win_unicode_console       0.5                      py37_0
wincertstore              0.2                      py37_0
winpty                    0.4.3                         4
wrapt                     1.11.2           py37he774522_0
xlrd                      1.2.0                    py37_0
xlsxwriter                1.2.9                      py_0
xlwings                   0.19.5                   py37_0
xlwt                      1.3.0                    py37_0
xmltodict                 0.12.0                     py_0
xz                        5.2.5                h62dcd97_0
yaml                      0.2.5                he774522_0
yapf                      0.30.0                     py_0
zeromq                    4.3.2                ha925a31_2
zict                      2.0.0                      py_0
zipp                      3.1.0                      py_0
zlib                      1.2.11               h62dcd97_4
zope                      1.0                      py37_1
zope.event                4.4                      py37_0
zope.interface            4.7.1            py37he774522_0
zstd                      1.4.5                ha9fde0e_0


title: very low CUDA performance for PyTorch in Windows and WSL2

body:
Host Env:
CPU: Intel Core-i7-7700HQ @2.8GHz
RAM: 16GB
GPU: NVidia GeForce 1050Ti
OS: 64-bit Windows Home 2004 (20241.1005)
CUDA: 11.2
conda: 4.8.4
Python: 3.7.5
Pytorch: 1.5.1
torchtext: 0.6.0
cudatoolkit: 10.1
CuDNN:7
WSL2 Env:
OS: Ubuntu 18.04
conda: 4.8.4
Python: 3.7.5
Pytorch: 1.5.1
torchtext: 0.6.0
cudatoolkit: 10.1
CuDNN:7
Docker Env:
Docker: 19.03.13
conda: 4.8.4
Python: 3.7.5
Pytorch: 1.5.0
torchtext: 0.6.0
cudatoolkit: 10.1
CuDNN:7
I tested two Pytorch lenet5 training scripts in the 3 envs, one of which uses CPU and the other uses GPU. I am very confused that GPU performance is not good, and working in Windows is much slower than working the virtualized Linux and the docker . I can't put the screenshot. pls check the time logging:
Windows output:

training on cuda
epoch 1, loss 1.7694, train acc 0.353, test acc 0.581, time 7.8 sec
epoch 2, loss 0.4776, train acc 0.628, test acc 0.687, time 6.7 sec
epoch 3, loss 0.2609, train acc 0.711, test acc 0.717, time 6.7 sec
epoch 4, loss 0.1738, train acc 0.738, test acc 0.743, time 6.6 sec
epoch 5, loss 0.1276, train acc 0.754, test acc 0.756, time 6.7 sec

training on cpu
epoch 1, loss 1.8857, train acc 0.304, test acc 0.586, time 20.8 sec
epoch 2, loss 0.4707, train acc 0.634, test acc 0.676, time 20.2 sec
epoch 3, loss 0.2549, train acc 0.719, test acc 0.719, time 20.7 sec
epoch 4, loss 0.1698, train acc 0.743, test acc 0.741, time 21.4 sec
epoch 5, loss 0.1257, train acc 0.758, test acc 0.755, time 22.0 sec

in WSL2 Ubuntu

training on cuda
epoch 1, loss 1.8913, train acc 0.305, test acc 0.574, time 10.7 sec
epoch 2, loss 0.4851, train acc 0.619, test acc 0.665, time 10.3 sec
epoch 3, loss 0.2610, train acc 0.710, test acc 0.729, time 9.9 sec
epoch 4, loss 0.1728, train acc 0.737, test acc 0.743, time 10.3 sec
epoch 5, loss 0.1285, train acc 0.751, test acc 0.749, time 10.2 sec

training on cpu
epoch 1, loss 1.8504, train acc 0.322, test acc 0.583, time 6.5 sec
epoch 2, loss 0.4723, train acc 0.635, test acc 0.676, time 6.3 sec
epoch 3, loss 0.2590, train acc 0.713, test acc 0.723, time 5.6 sec
epoch 4, loss 0.1717, train acc 0.739, test acc 0.740, time 5.6 sec
epoch 5, loss 0.1268, train acc 0.754, test acc 0.750, time 5.6 sec

in WSL Ubuntu Docker

training on cuda
epoch 1, loss 1.8288, train acc 0.325, test acc 0.588, time 11.0 sec
epoch 2, loss 0.4789, train acc 0.622, test acc 0.674, time 10.7 sec
epoch 3, loss 0.2598, train acc 0.713, test acc 0.727, time 10.6 sec
epoch 4, loss 0.1707, train acc 0.739, test acc 0.747, time 10.9 sec
epoch 5, loss 0.1251, train acc 0.757, test acc 0.760, time 10.7 sec

training on cpu
epoch 1, loss 1.8938, train acc 0.302, test acc 0.561, time 5.8 sec
epoch 2, loss 0.4748, train acc 0.641, test acc 0.682, time 5.8 sec
epoch 3, loss 0.2510, train acc 0.718, test acc 0.728, time 5.7 sec
epoch 4, loss 0.1673, train acc 0.742, test acc 0.731, time 5.8 sec
epoch 5, loss 0.1236, train acc 0.758, test acc 0.757, time 6.1 sec

Two python scripts for the test:

lenet5.py

import os
import time
import torch
from torch import nn, optim

import sys
sys.path.append("..")
import d2lzh_pytorch as d2l

os.environ["CUDA_VISIBLE_DEVICES"] = "0"
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

print(torch.__version__)
print(device)

class LeNet(nn.Module):
    def __init__(self):
        super(LeNet, self).__init__()
        self.conv = nn.Sequential(
            nn.Conv2d(1, 6, 5), # in_channels, out_channels, kernel_size
            nn.Sigmoid(),
            nn.MaxPool2d(2, 2), # kernel_size, stride
            nn.Conv2d(6, 16, 5),
            nn.Sigmoid(),
            nn.MaxPool2d(2, 2)
        )
        self.fc = nn.Sequential(
            nn.Linear(16*4*4, 120),
            nn.Sigmoid(),
            nn.Linear(120, 84),
            nn.Sigmoid(),
            nn.Linear(84, 10)
        )

    def forward(self, img):
        feature = self.conv(img)
        output = self.fc(feature.view(img.shape[0], -1))
        return output

net = LeNet()
print(net)

batch_size = 256
train_iter, test_iter = d2l.load_data_fashion_mnist(batch_size=batch_size)

def evaluate_accuracy(data_iter, net, device=None):
    if device is None and isinstance(net, torch.nn.Module):
        device = list(net.parameters())[0].device
    acc_sum, n = 0.0, 0
    with torch.no_grad():
        for X, y in data_iter:
            if isinstance(net, torch.nn.Module):
                net.eval()
                acc_sum += (net(X.to(device)).argmax(dim=1) == y.to(device)).float().sum().cpu().item()
                net.train()
            else:
                if('is_training' in net.__code__.co_varnames):

                    acc_sum += (net(X, is_training=False).argmax(dim=1) == y).float().sum().item()
                else:
                    acc_sum += (net(X).argmax(dim=1) == y).float().sum().item()
            n += y.shape[0]
    return acc_sum / n

def train_ch5(net, train_iter, test_iter, batch_size, optimizer, device, num_epochs):
    net = net.to(device)
    print("training on ", device)
    loss = torch.nn.CrossEntropyLoss()
    batch_count = 0
    for epoch in range(num_epochs):
        train_l_sum, train_acc_sum, n, start = 0.0, 0.0, 0, time.time()
        for X, y in train_iter:
            X = X.to(device)
            y = y.to(device)
            y_hat = net(X)
            l = loss(y_hat, y)
            optimizer.zero_grad()
            l.backward()
            optimizer.step()
            train_l_sum += l.cpu().item()
            train_acc_sum += (y_hat.argmax(dim=1) == y).sum().cpu().item()
            n += y.shape[0]
            batch_count += 1
        test_acc = evaluate_accuracy(test_iter, net)
        print('epoch %d, loss %.4f, train acc %.3f, test acc %.3f, time %.1f sec'
              % (epoch + 1, train_l_sum / batch_count, train_acc_sum / n, test_acc, time.time() - start))

if __name__ == "__main__":
    lr, num_epochs = 0.001, 5
    optimizer = torch.optim.Adam(net.parameters(), lr=lr)
    train_ch5(net, train_iter, test_iter, batch_size, optimizer, device, num_epochs)

lenet5-cpu.py

import os
import time
import torch
from torch import nn, optim

import sys
sys.path.append("..")
import d2lzh_pytorch as d2l

os.environ["CUDA_VISIBLE_DEVICaES"] = "0"
device = torch.device('cpu')

print(torch.__version__)
print(device)

class LeNet(nn.Module):
    def __init__(self):
        super(LeNet, self).__init__()
        self.conv = nn.Sequential(
            nn.Conv2d(1, 6, 5), # in_channels, out_channels, kernel_size
            nn.Sigmoid(),
            nn.MaxPool2d(2, 2), # kernel_size, stride
            nn.Conv2d(6, 16, 5),
            nn.Sigmoid(),
            nn.MaxPool2d(2, 2)
        )
        self.fc = nn.Sequential(
            nn.Linear(16*4*4, 120),
            nn.Sigmoid(),
            nn.Linear(120, 84),
            nn.Sigmoid(),
            nn.Linear(84, 10)
        )

    def forward(self, img):
        feature = self.conv(img)
        output = self.fc(feature.view(img.shape[0], -1))
        return output

net = LeNet()
print(net)

batch_size = 256
train_iter, test_iter = d2l.load_data_fashion_mnist(batch_size=batch_size)

def evaluate_accuracy(data_iter, net, device=None):
    if device is None and isinstance(net, torch.nn.Module):
        device = list(net.parameters())[0].device
    acc_sum, n = 0.0, 0
    with torch.no_grad():
        for X, y in data_iter:
            if isinstance(net, torch.nn.Module):
                net.eval()
                acc_sum += (net(X.to(device)).argmax(dim=1) == y.to(device)).float().sum().cpu().item()
                net.train()
            else:
                if('is_training' in net.__code__.co_varnames):
                    acc_sum += (net(X, is_training=False).argmax(dim=1) == y).float().sum().item()
                else:
                    acc_sum += (net(X).argmax(dim=1) == y).float().sum().item()
            n += y.shape[0]
    return acc_sum / n

def train_ch5(net, train_iter, test_iter, batch_size, optimizer, device, num_epochs):
    net = net.to(device)
    print("training on ", device)
    loss = torch.nn.CrossEntropyLoss()
    batch_count = 0
    for epoch in range(num_epochs):
        train_l_sum, train_acc_sum, n, start = 0.0, 0.0, 0, time.time()
        for X, y in train_iter:
            X = X.to(device)
            y = y.to(device)
            y_hat = net(X)
            l = loss(y_hat, y)
            optimizer.zero_grad()
            l.backward()
            optimizer.step()
            train_l_sum += l.cpu().item()
            train_acc_sum += (y_hat.argmax(dim=1) == y).sum().cpu().item()
            n += y.shape[0]
            batch_count += 1
        test_acc = evaluate_accuracy(test_iter, net)
        print('epoch %d, loss %.4f, train acc %.3f, test acc %.3f, time %.1f sec'
              % (epoch + 1, train_l_sum / batch_count, train_acc_sum / n, test_acc, time.time() - start))

if __name__ == "__main__":
    lr, num_epochs = 0.001, 5
    optimizer = torch.optim.Adam(net.parameters(), lr=lr)
    train_ch5(net, train_iter, test_iter, batch_size, optimizer, device, num_epochs)

The only difference in the two scripts is in line 11. Device is disgnated to CPU in the lenet5-cpu.py.


title: math operator difference *= or +=

body:
I found a weird thing when I used the operator e.g. *= or +=

The code:

aa = Variable(torch.FloatTensor([[1,2],[3,4]]))
bb = aa
bb = bb*2
print(bb)
print(aa)

cc = Variable(torch.FloatTensor([[1,2],[3,4]]))
dd = cc
dd *= 2
print(cc)
print(dd)

The results showed as below:

Variable containing:
 2  4
 6  8
[torch.FloatTensor of size 2x2]

Variable containing:
 1  2
 3  4
[torch.FloatTensor of size 2x2]

Variable containing:
 2  4
 6  8
[torch.FloatTensor of size 2x2]

Variable containing:
 2  4
 6  8
[torch.FloatTensor of size 2x2]

As you see, when I used bb=bb*2, aa was not be affected. However, if using dd *= 2, cc seems to point towards (share) the same address as cc, it is changed.

Their respective previous line is the same, e.g. bb = aa and dd = cc. It seems that the *= operator changed the original deep copy to shallow copy and the change was made after the copy line itself.

I am wondering if this is a bug. If it is, it is important since it affects the basic mathematics operations. Generally, I thought that just use the built-in operation functions e.g. torch.add() is a good solution.

OS: Mac OS X
PyTorch version: 3.0
How you installed PyTorch (conda, pip, source): conda
Python version: 3.6
CUDA/cuDNN version: None
GPU models and configuration: None

*---------------------------------------

I understand dd *= 2 is multiplication in-place, but how the value of dd transfer into cc? but if I used dd = dd * 2, the new values did not transfer to cc? There is no difference in the previous line: dd = cc and bb =aa.
BTW, in python (not pytorch Variable or Tensor), dd *= 2 and dd = dd * 2 both will not affect cc value.


title: Sampling from a joint distribution in Pyro

body:
I understand how to sample from multidimensional categorical, or multivariate normal (with dependence within each column). For example, for a multivariate categorical, this can be done as below:

import pyro as p
import pyro.distributions as d
import torch as t
p.sample("obs1", d.Categorical(logits=logit_pobs1).independent(1), obs=t.t(obs1))

My question is, how can we do the same, if there are multiple distributions? For example, the following is not what I want as obs1, obs2 and obs3 are independent to each other.

p.sample("obs1", d.Categorical(logits=logit_pobs1).independent(1), obs=t.t(obs1))
p.sample("obs2", d.Normal(loc=mu_obs2, scale=t.ones(mu_obs2.shape)).independent(1), obs=t.t(obs2))
p.sample("obs3", d.Bernoulli(logits=logit_pobs3).independent(1),obs3)

I would like to do something like

p.sample("obs", d.joint(d.Bernoulli(...), d.Normal(...), d.Bernoulli(...)).independent(1),obs)


title: How to *securely* setup Jupyter-Notebook runnnig CUDA enabled PyTorch, inside docker, on a dedicated server in the internet?

body:
Gladly we are beyond basic setup things or questions like How to handle python module installation when running jupyter notebook in docker?, Best practices for writing Dockerfiles

This server already is running in our local intranet,
we are already using this as collaborative deep-learning dev-server for our team.

Our Team gets more distributed and now we are going to move this server into
an own DMZ to e.g. allow team acces via normal internet (and: not just via VPN).

The question therefore mainly is, how to securely do this, configure, user, permissions ...

Currently we installed cuda drivers etc. on the (Ubuntu) host,
use nvidia-docker to get cuda into the docker containers.
To make all cuda drivers of all containers compatible with the cuda version of host,
we are using manual dockerfile based aproach, where all containers derive from base image

nvcr.io/nvidia/cuda  9.0-cudnn7.1-devel-ubuntu16.04

(From there a intermediate dockerfile installs further things) but finally the final dockerfile
esp. (pip-) installs jupyter-notebook together with the needed (e.g. PyTorch 0.4.1) and all other needed deep-learning tools.

The docker container is started by adding a shared volume to the docker container,
i.e. a folder on the host belonging to docker:docker with group sticky bit.
Therein have the actual machine-learing code for the Jupyter-notebooks.

nvidia-docker run --runtime=nvidia \
  --interactive --tty \
  --volume /srv:/srv \
  --name ${CONT_NAME} \
  --hostname ${CONT_HOSTNAME} \
  ${IMAGE_FULLNAME}

Another script then starts inside the container the notebook with a dedicated AUTH token,
that is regularly shared among the users of this jupyter notebook.

jupyter notebook --allow-root --ip 0.0.0.0 --no-browser --NotebookApp.token='<TEMP_AUTH_TOKEN>'

The DMZ firewall allows web-access to this single port of the Jupyter-Notebook only.

The notebook (together with the GPU related python code) needs access to the cuda toolchain and the GPUs.
This is passed in by nvidia-docker wrapper,
and we currently run all stuff inside the docker container as root.

Now here is my need for help: How would we secure this, "all within reason"?
(Alas, my job is not to be a full-time admin, but be the deep-learning guy)
I have an esp. "little bad feeling" offering a Jupyter-notebook running as root (even if root of container) to the web.
I fear hacking Jupyter-Notebook, and already as root inside container execute stuff would be one thing. But from there escape this docker container and own the whole host... how would I hinder such scenarios?

So on the one side: Jupyter needs to run and access CUDA-GPU of the host, but also serve the notebook to the web...

I would very apreciate more explanatory answers, hinting also some in-depth tipps/resources
for this kind of adventures, so I actually become more professional...


title: How do I efficiently load data from disk during training of deep learning models in pytorch?

body:
I'm trying to train a deep learning model without loading the entire dataset into memory. My main question is, what's the best way of doing this?
It seems like HDF5 is a common method that people accomplish this, and is what I tried first. However when using pytorch's dataloader class, this ran extremely slowly. I created my own iterator which ran faster, however the data is not randomized every batch. I'm trying to understand why the pytorch dataloader is running slowly and if there is something I can do about it.
Below is my code
First I defined a dataset class that takes in a filepath to an HDF5 dataset. My understanding of this code is that it reads from disk whenever getitem is called.
class My_H5Dataset(torch.utils.data.Dataset):

def __init__(self, file_path):
    super(My_H5Dataset, self).__init__()
    h5_file = h5py.File(file_path , 'r')
    self.features = h5_file['features']
    self.labels = h5_file['labels']
    self.index = h5_file['index']
    self.labels_values = h5_file['labels_values']
    self.index_values = h5_file['index_values']

def __getitem__(self, index):

    return (torch.from_numpy(self.features[index,:]).float(),
            torch.from_numpy(self.labels[index,:]).float(),
           torch.from_numpy(self.index[index,:]).float(),
           torch.from_numpy(np.array(self.labels_values[index])),
           torch.from_numpy(np.array(self.index_values[index])))

def __len__(self):
    return self.features.shape[0]

Then I simply pass this into a pytorch dataloader as follows
train_dataset = My_H5Dataset(hdf5_data_folder_train)
train_ms = MySampler(train_dataset)
trainloader = torch.utils.data.DataLoader(train_dataset, batch_size=batch_size,
sampler=train_ms,num_workers=2)

My other method was to manually define an iterator. And this does run much faster. However I have not noticed a speed difference between the data stored in an HDD versus an SSD which makes me worried that there is a bottleneck somewhere that I am missing.
def hdf5_loader_generator(dataset, batch_size, as_tensor=True, n_samples = 10000):
    """Given an h5 path to a file that holds the arrays, returns a generator
    that can get certain data at a time."""
    stop = n_samples
    curr_index = start = 0
    while 1:
        stop_index = min([curr_index + batch_size, stop])
        ft, sp, index, sp_values, index_values = dataset[curr_index:stop_index]
        curr_index += batch_size
        if curr_index >= stop:
            curr_index = start
            continue

        yield ft, sp, index, sp_values, index_values

def hdf5_data_iterator(dataset, batch_size, as_tensor=True):
    return iter(hdf5_loader_generator(dataset, batch_size, as_tensor))

I timed pytorch's iterator as follows:
start = time.time()
for i, data in enumerate(trainloader, 0):
    ft, sp, index, sp_values, index_values = data
    ft, sp, index = ft.to(device), sp.to(device), index.to(device)
    if i > 500:
        break

end = time.time()

print(end-start)

I timed my iterator as follows:
train_dataset = My_H5Dataset(hdf5_data_folder_train)
samples = hdf5_data_iterator(train_dataset, batch_sizes)
start = time.time()
for i, data in enumerate(samples):
    ft, sp, index, sp_values, index_values = data
    ft, sp, index = ft.to(device), sp.to(device), index.to(device)
    if i > 500:
        break

end = time.time()

print(end-start)

Is there a better approach than what I am currently trying?
Thank you for the help!


title: GANs on color images

body:
Most (PyTorch) open source GANs work on  MNIST dataset, i.e. gray level image.

Can I use a GAN on each channel of a color image, then combine the result?


I'm following the pytorch sagemaker docs here and I'm stuck on this line

torch.jit.optimized_execution(True, {'target_device': 'eia:device ordinal'})

When I run it, I get the error optimized_execution() takes 1 positional argument but 2 were given. I'm using pytorch 1.3.1, but I tried with 1.4.0 and was running into similar problems. Can I use optimized execution without this second argument? How can I specify the accelerator?


I've been trying to deploy my model to the AI platform for Prediction through the console on my vm instance, but I've gotten the error "(gcloud.beta.ai-platform.versions.create) Create Version failed. Bad model detected with error:  "Failed to load model: Unexpected error when loading the model: problem in predictor - ModuleNotFoundError: No module named 'torchvision' (Error code: 0)"

I need to include both torch and torchvision. I followed the steps in this question Cannot deploy trained model to Google Cloud Ai-Platform with custom prediction routine: Model requires more memory than allowed, but I couldn't fetch the files pointed to by user gogasca. I tried downloading this .whl file from Pytorch website and uploading it to my cloud storage but got the same error that there is no module torchvision, even though this version is supposed to include both torch and torchvision. Also tried using Cloud AI compatible packages here, but they don't include torchvision.

I tried pointing to two separate .whl files for torch and torchvision in the --package-uris arguments, those point to files in my cloud storage, but then I got the error that the memory capacity was exceeded. This is strange, because collectively their size is around 130Mb. An example of my command that resulted in absence of torchvision looked like this:

gcloud beta ai-platform versions create version_1 \
  --model online_pred_1 \
  --runtime-version 1.15 \
  --python-version 3.7 \
  --origin gs://BUCKET/model-dir \
  --package-uris gs://BUCKET/staging-dir/my_package-0.1.tar.gz,gs://BUCKET/torchvision-dir/torch-1.4.0+cpu-cp37-cp37m-linux_x86_64.whl \
  --prediction-class predictor.MyPredictor

I've tried pointing to different combinations of .whl files that I obtained from different sources, but got either the no module error or not enough memory. I don't understand how the modules interact in this case and why the compiler thinks there is no such module. How can I resolve this? Or alternatively, how can I compile a package myself that include both torch and torchvision. Can you please give detailed answers because I'm not very familiar with package management and bash scripting.

Here's the code I used, torch_model.py:

from torch import nn

class EthnicityClassifier44(nn.Module):
    def __init__(self, num_classes=2):
        super().__init__()
        self.conv1 = nn.Conv2d(3, 32, kernel_size=7, stride=1, padding=3)
        self.maxpool1 = nn.MaxPool2d(kernel_size=2, stride=2)
        self.conv22 = nn.Conv2d(32, 32, kernel_size=3, stride=1, padding=1)
        self.maxpool2 = nn.MaxPool2d(kernel_size=2, stride=2)
        self.conv3 = nn.Conv2d(32, 64, kernel_size=3, stride=1, padding=1)
        self.maxpool3 = nn.MaxPool2d(kernel_size=2, stride=2)
        self.conv4 = nn.Conv2d(64, 128, kernel_size=3, stride=1, padding=1)
        self.maxpool4 = nn.MaxPool2d(kernel_size=2, stride=2)
        self.relu = nn.ReLU(inplace=False)
        self.fc1 = nn.Linear(8*8*128, 128)
        self.fc2 = nn.Linear(128, 128)
        self.fc4 = nn.Linear(128, num_classes)

    def forward(self, x):
        x = self.relu(self.conv1(x))
        x = self.maxpool1(x)
        x = self.relu(self.conv22(x))
        x = self.maxpool2(x)
        x = self.maxpool3(self.relu(self.conv3(x)))
        x = self.maxpool4(self.relu(self.conv4(x)))
        x = self.relu(self.fc1(x.view(x.shape[0], -1)))
        x = self.relu(self.fc2(x))
        x = self.fc4(x)
        return x

This is predictor_py:

from facenet_pytorch import MTCNN, InceptionResnetV1, extract_face
import torch
import torchvision
from torchvision import transforms
from torch.nn import functional as F
from PIL import Image
from sklearn.externals import joblib
import numpy as np
import os
import torch_model

class MyPredictor(object):

    import torch
    import torchvision

    def __init__(self, model, preprocessor, device):
        """Stores artifacts for prediction. Only initialized via `from_path`.
        """
        self._resnet = model
        self._mtcnn_mult = preprocessor
        self._device = device
        self.get_std_tensor = transforms.Compose([
            np.float32,
            np.uint8,
            transforms.ToTensor(),
        ])
        self.tensor2pil = transforms.ToPILImage(mode='RGB')
        self.trans_resnet = transforms.Compose([
            transforms.Resize((100, 100)),
            np.float32,
            transforms.ToTensor()
        ])

    def predict(self, instances, **kwargs):

        pil_transform = transforms.Resize((512, 512))

        imarr = np.asarray(instances)
        pil_im = Image.fromarray(imarr)
        image = pil_im.convert('RGB')
        pil_im_512 = pil_transform(image)

        boxes, _ = self._mtcnn_mult(pil_im_512)
        box = boxes[0]

        face_tensor = extract_face(pil_im_512, box, margin=40)
        std_tensor = self.get_std_tensor(face_tensor.permute(1, 2, 0))
        cropped_pil_im = self.tensor2pil(std_tensor)

        face_tensor = self.trans_resnet(cropped_pil_im)
        face_tensor4d = face_tensor.unsqueeze(0)
        face_tensor4d = face_tensor4d.to(self._device)

        prediction = self._resnet(face_tensor4d)
        preds = F.softmax(prediction, dim=1).detach().numpy().reshape(-1)
        print('probability of (class1, class2) = ({:.4f}, {:.4f})'.format(preds[0], preds[1]))

        return preds.tolist()

    @classmethod
    def from_path(cls, model_dir):
        import torch
        import torchvision
        import torch_model

        model_path = os.path.join(model_dir, 'class44_M40RefinedExtra_bin_no_norm_7860.joblib')
        classifier = joblib.load(model_path)

        mtcnn_path = os.path.join(model_dir, 'mtcnn_mult.joblib')
        mtcnn_mult = joblib.load(mtcnn_path)

        device_path = os.path.join(model_dir, 'device_cpu.joblib')
        device = joblib.load(device_path)

        return cls(classifier, mtcnn_mult, device)

And setup.py:

from setuptools import setup

REQUIRED_PACKAGES = ['opencv-python-headless', 'facenet-pytorch']

setup(
 name="my_package",
 version="0.1",
 include_package_data=True,
 scripts=["predictor.py", "torch_model.py"],
 install_requires=REQUIRED_PACKAGES
)


title: Torch code not running on GPU

body:
I am running an optimization problem in torch. My torch installation is GPU compatible but for some odd reason it does not use the GPU at all when running. Everything seems to be done by the CPU and my local RAM.

import numpy as np
import scipy.sparse.csgraph as csg
import torch
from torch.autograd import Variable
import torch.autograd as autograd
import matplotlib.pyplot as plt

%matplotlib inline

def cmdscale(D):
    # Number of points
    n = len(D)

    # Centering matrix
    H = np.eye(n) - np.ones((n, n))/n

    # YY^T
    B = -H.dot(D**2).dot(H)/2

    # Diagonalize
    evals, evecs = np.linalg.eigh(B)

    # Sort by eigenvalue in descending order
    idx   = np.argsort(evals)[::-1]
    evals = evals[idx]
    evecs = evecs[:,idx]

    # Compute the coordinates using positive-eigenvalued components only
    w, = np.where(evals > 0)
    L  = np.diag(np.sqrt(evals[w]))
    V  = evecs[:,w]
    Y  = V.dot(L)

    return Y, evals

Y = np.array([[0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0.],
              [0., 0., 0., 0., 0., 1., 1., 0., 1., 0., 0., 0., 0., 0., 0.],
              [0., 0., 0., 0., 1., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0.],
              [0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 1., 0., 0., 1., 0.],
              [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 1., 0.],
              [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
              [0., 1., 0., 1., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 1.],
              [0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],
              [1., 1., 1., 0., 0., 0., 0., 0., 0., 0., 0., 1., 1., 0., 1.],
              [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0.],
              [0., 0., 0., 1., 1., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0.],
              [0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 1., 1.],
              [0., 0., 0., 0., 0., 0., 0., 0., 1., 1., 0., 0., 0., 0., 0.],
              [0., 0., 0., 1., 1., 0., 0., 0., 0., 0., 1., 1., 0., 0., 0.],
              [0., 0., 0., 0., 0., 0., 1., 0., 1., 0., 0., 1., 0., 0., 0.]])

temp = Y[~np.all(Y == 0, axis=1)]
temp = temp[:,~np.all(Y == 0, axis=1)]
Y = np.asarray(temp, dtype='uint8')

n = np.shape(Y)[0]
k = 2

D = csg.shortest_path(Y, directed=True)
Z = cmdscale(D)[0][:,0:k]
Z = Z - Z.mean(axis=0, keepdims=True)

def distMatrix(m):
    n = m.size(0)
    d = m.size(1)
    x = m.unsqueeze(1).expand(n, n, d)
    y = m.unsqueeze(0).expand(n, n, d)
    return torch.sqrt(torch.pow(x - y, 2).sum(2) + 1e-4)

def loss(tY):
    d = -distMatrix(tZ)+B
    sigmoidD = torch.sigmoid(d)
    reduce = tY*torch.log(sigmoidD)+(1-tY)*torch.log(1-sigmoidD)
    #remove diagonal
    reduce[torch.eye(n).byte().cuda()] = 0
    return -reduce.sum()

tZ = autograd.Variable(torch.cuda.FloatTensor(Z), requires_grad=True)
B = autograd.Variable(torch.cuda.FloatTensor([0]), requires_grad=True)

tY = autograd.Variable(torch.cuda.FloatTensor(Y), requires_grad=False)

losses = []
biases = []
#rocAuc = []
learning_rate = 1e-3
epochs = 10000

percentDone = 0
percent = 5
for i in range(epochs):
    if i % (epochs*percent*0.01) == 0:
        percentDone += percent
        print(str(percentDone) + "%")

    l = loss(tY)
    l.backward(retain_graph=True)
    losses.append(float(l))
    biases.append(B.data)
    tZ.data = tZ.data - learning_rate * tZ.grad.data
    B.data = B.data - learning_rate * B.grad.data

    tZ.grad.data.zero_()
    B.grad.data.zero_()

plt.subplot(122)
plt.plot(losses)
plt.title('Loss')
plt.xlabel('Iteration')
plt.ylabel('loss')

plt.show()

Theres an awful lot of code, but it is a working example, how do I make this code run on my GPU? is it even possible? Any hints or nudges in the right directions would be greatly appreciated.


title: How to solve strange cuda error in PyTorch?

body:
I moved to PyTorch from Keras. I'm very new to the whole moving to  CUDA thing. I've spent hours, surfing the web and haven't been able to find anything? The fix is probably something a line or two. I'd appreciate it if someone knows how to solve this issue?

Here' my code, First I define my u-net model as a class of nn.Module like the following code:

import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.utils.data import Dataset, DataLoader
from torchvision import transforms, utils

class unet(nn.Module):
def __init__(self):
    super(unet, self).__init__()
    self.conv1 = nn.Conv3d(1, 32, 3, padding=1)
    self.conv1_1 = nn.Conv3d(32, 32, 3, padding=1)
    self.conv2 = nn.Conv3d(32, 64, 3, padding=1)
    self.conv2_2 = nn.Conv3d(64, 64, 3, padding=1)
    self.conv3 = nn.Conv3d(64, 128, 3, padding=1)
    self.conv3_3 = nn.Conv3d(128, 128, 3, padding=1)
    self.convT1 = nn.ConvTranspose3d(128, 64, 3, stride=(2,2,2), padding=1, output_padding=1)
    self.conv4 = nn.Conv3d(128, 64, 3, padding=1)
    self.conv4_4 = nn.Conv3d(64, 64, 3, padding=1)
    self.convT2 = nn.ConvTranspose3d(64, 32, 3,stride=(2,2,2), padding=1, output_padding=1)
    self.conv5 = nn.Conv3d(64, 32, 3, padding=1)
    self.conv5_5 = nn.Conv3d(32, 32, 3, padding=1)
    self.conv6 = nn.Conv3d(32, 1 ,3, padding=1)

def forward(self, inputs):
    conv1 = F.relu(self.conv1(inputs))
    conv1 = F.relu(self.conv1_1(conv1))
    pool1 = F.max_pool3d(conv1, 2)

    conv2 = F.relu(self.conv2(pool1))
    conv2 = F.relu(self.conv2_2(conv2))
    pool2 = F.max_pool3d(conv2, 2)
    conv3 = F.relu(self.conv3(pool2))
    conv3 = F.relu(self.conv3_3(conv3))
    conv3 = self.convT1(conv3)

    up1 = torch.cat((conv3, conv2), dim=1)
    conv4 = F.relu(self.conv4(up1))
    conv4 = F.relu(self.conv4_4(conv4))

    conv4 = self.convT2(conv4)
    up2 = torch.cat((conv4, conv1), dim=1)
    conv5 = F.relu(self.conv5(up2))
    conv5 = F.relu(self.conv5_5(conv5))

    conv6 = F.relu(self.conv6(conv5))

    return conv6

Then I run my unet like the following code. note that when defining the module I set it to the cuda. I also set the input data and its labels to the cuda.

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
model = unet().to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
loss_fn = nn.MSELoss()
datasets = torch.utils.data.TensorDataset(data_recon, data_truth)
train_loader = DataLoader(datasets, batch_size=2, shuffle=True)

def training_loop(n_epochs, optimizer, model, loss_fn, train_loader):
    for epoch in range(1, n_epochs + 1):
        loss_train = 0
        for imgs, labels in train_loader:
            imgs.to(device)
            labels.to(device)
            outputs = model(imgs)
            loss = loss_fn(outputs, labels)

            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

            loss_train += loss.item()

        print('{} Epoch {}, Training loss    {}'.format(datetime.datetime.now(), epoch, float(loss_train)))

training_loop(50, optimizer, model, loss_fn, train_loader)

But I get this error:

  RuntimeError                              Traceback (most recent call
  last)  in
  ----> 1 training_loop(50, optimizer, model, loss_fn, train_loader)

   in training_loop(n_epochs, optimizer,
  model, loss_fn, train_loader)
        5             imgs.to(device)
        6             labels.to(device)
  ----> 7             outputs = model(imgs)
        8             loss = loss_fn(outputs, labels)
        9

  /opt/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py
  in call(self, *input, **kwargs)
      491             result = self._slow_forward(*input, **kwargs)
      492         else:
  --> 493             result = self.forward(*input, **kwargs)
      494         for hook in self._forward_hooks.values():
      495             hook_result = hook(self, input, result)

   in forward(self, inputs)
       18
       19     def forward(self, inputs):
  ---> 20         conv1 = F.relu(self.conv1(inputs))
       21         conv1 = F.relu(self.conv1_1(conv1))
       22         pool1 = F.max_pool3d(conv1, 2)

  /opt/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py
  in call(self, *input, **kwargs)
      491             result = self._slow_forward(*input, **kwargs)
      492         else:
  --> 493             result = self.forward(*input, **kwargs)
      494         for hook in self._forward_hooks.values():
      495             hook_result = hook(self, input, result)

  /opt/anaconda3/lib/python3.7/site-packages/torch/nn/modules/conv.py in
  forward(self, input)
      474                             self.dilation, self.groups)
      475         return F.conv3d(input, self.weight, self.bias, self.stride,
  --> 476                         self.padding, self.dilation, self.groups)
      477
      478

  RuntimeError: Expected object of backend CPU but got backend CUDA for
  argument #2 'weight'

I've spent hours, surfing the web and haven't been able to find anything? The fix is probably something a line or two. I'd appreciate it if someone knows how to solve this issue?


title: Resnet18 first layer output dimensions

body: I am looking at the model implementation in PyTorch. The 1st layer is a convolutional layer with filter size = 7, stride = 2, pad = 3. The standard input size to the network is 224x224x3. Based on these numbers, the output dimensions are (224 + 3*2 - 7)/2 + 1, which is not an integer. Does the original implementation contain non-integer dimensions? I see that the network has adaptive pooling before the FC layer, so the variable input dimensions aren't a problem (I tested this by varying the input size). Am I doing something wrong, or why would the authors choose a non-integer dimension while designing the ResNet?


title: Get value from c10::Dict in Pytorch C++

body:
I'm using a TorchScript Model on Pytorch C++ Frontend.
The model in Python returns an output dict as  Dict[str, List[torch.Tensor]].
When I use it in C++, it returns a c10::Dict<c10::IValue, c10::IValue>. What is the equivalent of this Python code:
value_a = output['key_a']
value_b = output['key_b']

in C++ to get value from c10::Dict?
I have tried this but it does not work.
torch::IValue key_a("key_a");
torch::IValue key_b("key_b");
c10::IValue value_a = output[key_a];
c10::IValue value_b = output[key_b];
std::cout << value_a << std::endl;
std::cout << value_b << std::endl;

And error:
error: type 'c10::Dict<c10::IValue, c10::IValue>' does not provide a subscript operator


title: Transformer PyTorch Error- ValueError: too many values to unpack (expected 2)

body:
I am having issues getting my model to run. I am not sure which model to use in the translate_sentence function, I have tried model.transformer, model.encoder_de, etc. It is based off of the Transformer class and the forward() function I believe but I am getting a type error. These are the directions:

As in the forward(self, src, tgt) function of the
TransformerModel class, you need to create the appropriate
mask and encode the source sentence (just once).

You also need to create the appropriate mask and encode the
output sentence for sequential predictions. Unlike the source,
for every iteration, you need to re-encode the previous output and
pass both the source sentence and previous output into the
Transformer.

from torch.nn import Transformer
class TransformerModel(nn.Module):

    def __init__(self, ntoken_in, ntoken_out, ninp, nhead, npf_dim, nlayers, src_pad_idx, trg_pad_idx, dropout=0.5):
        super(TransformerModel, self).__init__()

        # --------------- param -----------------
        # ntoken_in: the idx of the input word after tokenization
        # ntoken_out: the idx of the input word w.r.t. the tokenization
        # ninp: the number of expected features in the encoder/decoder inputs
        # nhead: the number of multiAttention heads
        # npf_dim: the dimension of the feedforward layer
        # src_pad_idx: the token for padding in source language
        # trg_pad_idx: the token for padding in target language
        # ----------------------------------------

        self.model_type = 'Transformer'
        self.pos_encoder = PositionalEncoding(ninp, dropout)
        self.transformer = Transformer(d_model=ninp, nhead=nhead, num_encoder_layers=nlayers, num_decoder_layers=nlayers,
                                       dim_feedforward=npf_dim, dropout=dropout, activation='relu')

        self.encoder_en = nn.Embedding(ntoken_in, ninp)  # tok_embedding for input
        self.encoder_de = nn.Embedding(ntoken_out, ninp) # tok_embedding for output
        self.ninp = ninp
        self.decoder = nn.Linear(ninp, ntoken_out)

        self.src_pad_idx = src_pad_idx
        self.tgt_pad_idx = trg_pad_idx

        self.init_weights()

    def _generate_src_key_mask(self, src):
        # for key_padding_mask in transformer
        # the positions with the value of True will be ignored while the position
        # with the value of False will be unchanged. We mask all padding words.
        # The output dim is b*s
        src_mask = (src == self.src_pad_idx)
        return src_mask.T

    def _generate_tgt_mask(self, tgt, sz):
        # Beside key_padding_mask in transformer, the output or teacher input
        # should be masked sequentially to prevent the model get any information
        # from the future words it is going to predict
        tgt_key_mask = tgt == self.tgt_pad_idx

        # We provide FloatTensor attn_mask. It will be added to the attention weight.
        mask = (torch.triu(torch.ones(sz, sz)) == 1).transpose(0, 1)
        attn_mask = mask.float().masked_fill(mask == 0, float('-inf')).masked_fill(mask == 1, float(0.0)).to(tgt.device)
        return attn_mask, tgt_key_mask.T

    def init_weights(self):
        initrange = 0.1
        self.encoder_en.weight.data.uniform_(-initrange, initrange)
        self.encoder_de.weight.data.uniform_(-initrange, initrange)
        self.decoder.bias.data.zero_()
        self.decoder.weight.data.uniform_(-initrange, initrange)

    def forward(self, src, tgt):
        # src
        src_key_mask = self._generate_src_key_mask(src)
        src = self.encoder_en(src) * math.sqrt(self.ninp)  # use a learned encoder put stoi index to a feature space s*b --> s*b*e
        src = self.pos_encoder(src)  # add the pos feature toward feature space

        # tgt
        tgt_mask, tgt_key_mask = self._generate_tgt_mask(tgt, tgt.size(0))
        tgt = self.encoder_de(tgt) * math.sqrt(self.ninp)
        tgt = self.pos_encoder(tgt)

        output = self.transformer(src, tgt, tgt_mask=tgt_mask,
                                  src_key_padding_mask = src_key_mask,
                                  tgt_key_padding_mask = tgt_key_mask)
        output = self.decoder(output)
        return output

class PositionalEncoding(nn.Module):
    # The positional encoding as described in the paper
    # https://arxiv.org/pdf/1706.03762.pdf
    def __init__(self, d_model, dropout=0.1, max_len=5000):
        super(PositionalEncoding, self).__init__()
        self.dropout = nn.Dropout(p=dropout)

        pe = torch.zeros(max_len, d_model)
        position = torch.arange(0, max_len, dtype=torch.float).unsqueeze(1)
        div_term = torch.exp(torch.arange(0, d_model, 2).float() * (-math.log(10000.0) / d_model))
        pe[:, 0::2] = torch.sin(position * div_term)
        pe[:, 1::2] = torch.cos(position * div_term)
        pe = pe.unsqueeze(0).transpose(0, 1)
        self.register_buffer('pe', pe)

    def forward(self, x):
        x = x + self.pe[:x.size(0), :]
        return self.dropout(x)

# Here we intialize our model
INPUT_DIM = len(SRC.vocab)
OUTPUT_DIM = len(TRG.vocab)
print(INPUT_DIM, OUTPUT_DIM)

HID_DIM = 256
N_LAYERS = 3
N_HEADS = 8
N_PF_DIM = 512
DROPOUT = 0.1

SRC_PAD_IDX = SRC.vocab.stoi[SRC.pad_token]
TRG_PAD_IDX = TRG.vocab.stoi[TRG.pad_token]

model =TransformerModel(ntoken_in = INPUT_DIM, ntoken_out=OUTPUT_DIM, ninp=HID_DIM,
                        nhead=N_HEADS, npf_dim=N_PF_DIM, nlayers=N_LAYERS,
                        src_pad_idx=SRC_PAD_IDX, trg_pad_idx=TRG_PAD_IDX, dropout=DROPOUT).to(device)

def count_parameters(model: nn.Module):
    return sum(p.numel() for p in model.parameters() if p.requires_grad)

print(f'The model has {count_parameters(model):,} trainable parameters')

def initialize_weights(m):
    if hasattr(m, 'weight') and m.weight.dim() > 1:
        nn.init.xavier_uniform_(m.weight.data)

model.apply(initialize_weights)

----------
def translate_sentence(sentence, src_field, trg_field, model, device, max_len = 50):

    model.eval()

    if isinstance(sentence, str):
        nlp = spacy.load('de')
        tokens = [token.text.lower() for token in nlp(sentence)]
    else:
        tokens = [token.lower() for token in sentence]

    #tokens = [src_field.init_token] + tokens + [src_field.eos_token]
    src_indexes = [src_field.vocab.stoi[token] for token in tokens]
    src_tensor = torch.LongTensor(src_indexes).unsqueeze(1).to(device)

    with torch.no_grad():
        #model.?
        hidden, cell = model.encoder_en(src_tensor)

    # create a list to hold the output sentence, initialized with an <sos> token

    trg_indexes = [trg_field.vocab.stoi[trg_field.init_token]]

    for i in range(max_len):

        trg_tensor = torch.LongTensor(trg_indexes).unsqueeze(1).to(device)
        with torch.no_grad():
            #model.?
            output, hidden, cell = model.encoder_de(trg_tensor, hidden, cell)

        pred_token = output.argmax(1).item()

        trg_indexes.append(pred_token)

        if pred_token == trg_field.vocab.stoi[trg_field.eos_token]:
            break

    trg_tokens = [trg_field.vocab.itos[i] for i in trg_indexes]

    return trg_tokens[1:]

----------
#getting error here

example_idx = 18

src = vars(train_data.examples[example_idx])['src']
trg = vars(train_data.examples[example_idx])['trg']

print(f'src = {src}')
print(f'trg = {trg}')

translation = translate_sentence(src, TRG, SRC, model, device)

print(f'predicted trg = {translation}')


title: Looping over pytorch LSTM

body:
I am training a seq2seq model for machine translation in pytorch. I would like to gather the cell state at every time step, while still having the flexibility of multiple layers and bidirectionality, that you can find in the LSTM module of pytorch, for example.

To this end, I have the following encoder and forward method, where I loop over the LSTM module. The problem is, that the model does not train very well. Right after the loop terminates, you can see the normal way to use the LSTM module and with that, the model trains.

So, is the loop not a valid way to do this?

class encoder(nn.Module):
    def __init__(self, input_dim, emb_dim, hid_dim, n_layers, dropout):
        super().__init__()

        self.input_dim = input_dim
        self.emb_dim = emb_dim
        self.hid_dim = hid_dim
        self.n_layers = n_layers
        self.dropout = dropout

        self.embedding = nn.Embedding(input_dim, emb_dim)

        self.rnn = nn.LSTM(emb_dim, hid_dim, n_layers, dropout = dropout)

        self.dropout = nn.Dropout(dropout)

    def forward(self, src):

        #src = [src sent len, batch size]

        embedded = self.dropout(self.embedding(src))

        #embedded = [src sent len, batch size, emb dim]
        hidden_all = []

        for i in range(len(embedded[:,1,1])):
            outputs, hidden = self.rnn(embedded[i,:,:].unsqueeze(0))
            hidden_all.append(hidden)

        #outputs, hidden = self.rnn(embedded)

        #outputs = [src sent len, batch size, hid dim * n directions]
        #hidden = [n layers * n directions, batch size, hid dim]
        #cell = [n layers * n directions, batch size, hid dim]
        None
        #outputs are always from the top hidden layer

        return hidden


title: Callbacks in Fastai

body:
I am working on a deep learning project in Fastai and wish to use EarlyStoppingCallback with ReduceLROnPlateauCallback in it. Read callbacks.fastai but struggling to understand how to implement both and couldn't find any relevant example. Any help would be appreciated.

learn = cnn_learner(data, models.resnet50, metrics = [accuracy,quadratic_kappa])
learn.fit(50,2e-6)


title: Add values to a tensor element based on its position in pytorch

body:
In pytorch, I want to add values to elements in the tensor based on their position. For example consider,

Input = torch.tensor([1,2,3,4,5,6,7,8,9,0,1,2,3,4,5,6,7,8,9,0])

Between several offsets of the Input array, Offsets = [0,5,10,15,20], I want to add different values, ValuesToAdd = [10,100,1000,10000]

I expect the output to be

Output = torch.tensor([11,12,13,14,15,106,107,108,109,100,1001,1002,1003,1004,1005,10006,10007,10008,10009,10000])

Here, between indices Offsets[i] and Offsets[i+1] in Input array, ValuesToAdd[i] is added. For example, for indices 10,11,12,13 and 14 (Offsets[2] = 10 to Offsets[3]=15) in Input array, 1000 (ValuesToAdd[2]) is added.

How can I achieve this? Instead of looping over Offsets array, I am looking for a more efficient way.


title: What's the best way to load two pretrained model partially from state-dict in Pytorch?

body:
I am trying to load two separately trained models except for the last layer and want to train the last layer separately combining these two models. I defined e new nn.Module class and load these pretrained model inside that class and in the forward path tried to return the value before the last layer.
class New_net(nn.Module):
    def __init__(self):
        super(New_net, self).__init__()
        self.net1 = net1()
        self.net2 = net2()
        self.fc= nn.Linear(512, 2)
        self._initialize_weights()

    def _initialize_weights(self):
        checkpoint = torch.load('save_model/checkpoint_net1.t7')
        self.net1.load_state_dict(checkpoint['state_dict'])

        checkpoint = torch.load('save_model/checkpoint_net2.t7')
        self.net2.load_state_dict(checkpoint['state_dict'])

    def forward(self, x):
        x1 = self.net1(x)
        x2 = self.net2(x)
        x=torch.cat((x1,x2),dim=1)
        x=self.fc(x)
        return x

but it seems it is not loading the model accurately. What's the correct way to do that


title: Loading enormous custom dataset using IterableDataset

body:
I have a huge dataset with features (input_id,input_mask,segment_id,label_id) saved in batches of 64 in a pickle file. I read this file, create a TensorDataset and pass to dataloader for training. Since the features file is too big to create the full TensorDataset, I want to convert TensorDataset to IterableDataset so that one batch of samples can be retrieved from the features file at a time and passed on to the dataloader. But while training, I get the following error:
TypeError: iter() returned non-iterator of type 'TensorDataset'
Following is the custom dataset class I wrote:
class MyDataset(IterableDataset):

    def __init__(self,args):
        self.args=args

    def get_features(self,filename):
        with open(filename, "rb") as f:
            while True:
                try:
                    yield pickle.load(f)
                except EOFError:
                    break

    def process(self,args):
        if args.cached_features_file:
            cached_features_file = args.cached_features_file

        if os.path.exists(cached_features_file):
            features=self.get_features(cached_features_file)

        feat = next (features)
        li=list(feat)
        all_input_ids=torch.tensor([f.input_ids for f in li ], dtype=torch.long)
        all_input_mask= torch.tensor([f.input_mask for f in li ], dtype=torch.long)
        all_segment_ids= torch.tensor([f.segment_ids for f in li], dtype=torch.long)
        all_label_ids = torch.tensor([f.label_id for f in li ], dtype=torch.long)

        dataset = TensorDataset(all_input_ids, all_input_mask, all_segment_ids, all_label_ids)
        return dataset

    def __iter__(self):
        dataset=self.process(self.args)
        return dataset

And I use it like this:
train_dataset=MyDataset(args)
train_dataloader = DataLoader(train_dataset, batch_size=args.train_batch_size)

I understand that TensorDataset is map-style requiring index while IterableDataset is iterable-style which is the reason for error. Even if I return a list/tuple of feature tensors instead of TensorDataset, I get similar error. Can someone please tell me how to load a batched dataset in the correct way with IterableDataset?


I am trying to implement such CNN.

This is my implementation:
class Net(BaseFeaturesExtractor):
    def __init__(self, observation_space: gym.spaces.Box, features_dim: int = 256):
        super(Net, self).__init__(observation_space, features_dim)
        n_input_channels = observation_space.shape[0]
        print("Observation space shape:"+str(observation_space.shape))
        print("Number of channels:" + str(n_input_channels))
        self.cnn = nn.Sequential(
            nn.Conv2d(n_input_channels, 32, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.Conv2d(n_input_channels, 32, kernel_size=3, stride=2, padding=1),
            nn.ReLU(),
            nn.Conv2d(n_input_channels, 32, kernel_size=3, stride=2, padding=1),
            nn.ReLU(),
            nn.Flatten(),
            nn.Linear(in_features=128,out_features=64),
            nn.ReLU(),
            nn.Linear(in_features=64,out_features=7),
            nn.Sigmoid()
        )

    def forward(self, observations: th.Tensor) -> th.Tensor:
        print("Observation shape:"+str(observations[0].shape))
        return self.cnn(observations)

When I tried to run the code which uses this CNN, I am getting following log:
    Observation space shape:(3, 6, 7)
    Number of channels:3
    Observation shape:torch.Size([3, 6, 7])
    Traceback (most recent call last):   File "/Users/joe/Documents/JUPYTER/ConnectX/training3.py", line 250, in <module>
        learner.learn(total_timesteps=iterations, callback=eval_callback)
...
RuntimeError: Given groups=1, weight of size [32, 3, 3, 3], expected input[4, 32, 6, 7] to have 3 channels, but got 32 channels instead

What is the problem here? How can I solve it?


title: Display augmented image using Pytorch

body:
This is a snippet mostly provided by @ptrblck in Pytorch forum for data augmentation on some of images.

The task is segmentation, so I assume image and its correspond mask need to be augment.

I'm wondering how can I display some images and correspond mask after transformation in order to find out how they look like?

Here is the script:

import torch
from torch.utils.data.dataset import Dataset  # For custom data-sets
import torchvision.transforms as transforms
import torchvision.transforms.functional as tf
from PIL import Image
import numpy
import glob
import matplotlib.pyplot as plt
from split_dataset import test_loader
import os

class CustomDataset(Dataset):
    def __init__(self, image_paths, target_paths, transform_images, transform_masks):

    self.image_paths = image_paths
    self.target_paths = target_paths

    self.transform_images = transform_images
    self.transform_masks = transform_masks

    self.transformm = transforms.Compose([transforms.Lambda(lambda x: tf.rotate(x, 10)),
                                          transforms.Lambda(lambda x: tf.affine(x, angle=0,
                                      translate=(0, 0),
                                      scale=0.2,
                                      shear=0.2))
                                        ])

    self.transform = transforms.ToTensor()

    self.mapping = {
        0: 0,
        255: 1
    }

def mask_to_class(self, mask):
    for k in self.mapping:
        mask[mask==k] = self.mapping[k]
    return mask

def __getitem__(self, index):

    image = Image.open(self.image_paths[index])
    mask = Image.open(self.target_paths[index])

    if any([img in self.image_paths[index] for img in self.transform_images]):
        print('applying special transformation')
        image = self.transformm(image) #augmentation

    if any([msk in self.target_paths[index] for msk in self.transform_masks]):
        print('applying special transformation')
        image = self.transformm(mask) #augmentation

    t_image = image.convert('L')
    t_image = self.transform(t_image) # transform to tensor for image
    mask = self.transform(mask) # transform to tensor for mask

    mask = torch.from_numpy(numpy.array(mask, dtype=numpy.uint8))
    mask = self.mask_to_class(mask)
    mask = mask.long()

    return t_image, mask, self.image_paths[index], self.target_paths[index]

def __len__(self):  # return count of sample we have

    return len(self.image_paths)

image_paths = glob.glob("D:\\Neda\\Pytorch\\U-net\\my_data\\imagesResized\\*.png")
target_paths = glob.glob("D:\\Neda\\Pytorch\\U-net\\my_data\\labelsResized\\*.png")

transform_images = ['image_981.png', 'image_982.png','image_983.png', 'image_984.png', 'image_985.png',
                    'image_986.png','image_987.png','image_988.png','image_989.png','image_990.png',
                    'image_991.png']  # apply special transformation only on these images
print(transform_images)
#['image_991.png', 'image_991.png']

transform_masks = ['image_labeled_981.png', 'image_labeled_982.png','image_labeled_983.png', 'image_labeled_984.png',
                    'image_labeled_985.png', 'image_labeled_986.png','image_labeled_987.png','image_labeled_988.png',
                    'image_labeled_989.png','image_labeled_990.png',
                    'image_labeled_991.png']

dataset = CustomDataset(image_paths, target_paths, transform_images, transform_masks)

for transform_images in dataset:

    #print(transform_images)
    transform_images = Image.open(os.path.join(image_paths, transform_images))
    transform_images = numpy.array(transform_images)

    transform_masks = Image.open(os.path.join(target_paths, transform_masks))
    transform_masks = numpy.array(transform_masks)

    fig, (ax1, ax2) = plt.subplots(nrows=1, ncols=1, sharex=True, sharey=True, figsize = (6,6))

    img1 = ax1.imshow(transform_images, cmap='gray')
    ax1.axis('off')

    img2 = ax2.imshow(transform_masks)
    ax1.axis('off')
    plt.show()

currently it's causing an error

  path = os.fspath(path)
  TypeError: expected str, bytes or os.PathLike object, not tuple


title: Dealing with data which have different sizes in multi task learning (Pytorch)

body:
I want to create multi-task learning model with Pytorch.

The model I designed follows hard parameter sharing (http://ruder.io/multi-task/)

The problem is the size of each dataset is different. So they cannot be trained in the same for loop in the model like this:

for epoch in range(EPOCHS):
    for data in dataloader_train:
        output_a = model(data.a)
        output_b = model(data.b)
        output_c = model(data.c)

        loss_a = criterion(output_a, data.target_a)
        loss_b = criterion(output_b, data.target_b)
        loss_c = criterion(output_c, data.target_c)

        optimizer.zero_grad()
        loss.backward([loss_a, loss_b, loss_c])
        optimizer.step()

I would like to know how to deal with this situation. I thought I can use zip with itertools.cycle (Iterate over two list of different sizes in python) but it may affect the model significantly because it changes the frequency of certain data especially located in the early index:

data.a data.b
a      1
b      2
c      3
a      4
b      5
c      1
a      2
b      3

On the other hand, if I regard it as just the differences between epochs of the data sets, it seems that it has no problem. E.g. data.a has 5 epochs and data.b has 3 epochs when EPOCHS == 15.

Or if there is no problem to design model like this:

for epoch in range(EPOCHS):
    for data, target in data_a:
        output = model(data)
        loss = criterion(output, target)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

    for data, target in data_b:
        output = model(data)
        loss = criterion(output, target)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

    for data, target in data_c:
        output = model(data)
        loss = criterion(output, target)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

I think this will be the most simple solution. However, I don't know why but I got obsessed with the idea that I have to build the model like the former model.

All the codes above are pseudocode.

Thank you for the advice :)


title: How to build a conda environment with pytorch=0.3.1, since I keep getting conflicts

body:
I need to get an environment working with the following requirements, but it just won't do.
I have tried many things and always end up with something like below. Does anyone know how I can get an environment with the things that I need?

conda create -n myenv python=3.5 pytorch=0.3.1
Collecting package metadata (repodata.json): done
Solving environment: -
Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
failed

UnsatisfiableError: The following specifications were found to be incompatible with each other:

Package libffi conflicts for:
python=3.5 -> libffi[version='3.2.*|>=3.2.1,<4.0a0']
Package numpy conflicts for:
pytorch=0.3.1 -> numpy[version='>=1.11.3,<2.0a0']
Package openssl conflicts for:
python=3.5 -> openssl[version='1.0.*|1.0.*,>=1.0.2l,<1.0.3a|>=1.0.2m,<1.0.3a|>=1.0.2n,<1.0.3a|>=1.0.2o,<1.0.3a|>=1.0.2p,<1.0.3a']
Package libstdcxx-ng conflicts for:
pytorch=0.3.1 -> libstdcxx-ng[version='>=5.4.0']
python=3.5 -> libstdcxx-ng[version='>=7.2.0|>=7.3.0']
Package nccl conflicts for:
pytorch=0.3.1 -> nccl[version='<2']
Package libgcc-ng conflicts for:
pytorch=0.3.1 -> libgcc-ng[version='>=5.4.0']
python=3.5 -> libgcc-ng[version='>=7.2.0|>=7.3.0']
Package readline conflicts for:
python=3.5 -> readline[version='7.*|>=7.0,<8.0a0']
Package cudatoolkit conflicts for:
pytorch=0.3.1 -> cudatoolkit=8.0
Package cudnn conflicts for:
pytorch=0.3.1 -> cudnn[version='>=7.0.5,<=8.0a0']
Package cffi conflicts for:
pytorch=0.3.1 -> cffi
Package tk conflicts for:
python=3.5 -> tk[version='8.6.*|>=8.6.7,<8.7.0a0']
Package xz conflicts for:
python=3.5 -> xz[version='>=5.2.3,<6.0a0|>=5.2.4,<6.0a0']
Package zlib conflicts for:
python=3.5 -> zlib[version='>=1.2.11,<1.3.0a0']
Package python conflicts for:
pytorch=0.3.1 -> python[version='>=2.7,<2.8.0a0|>=3.5,<3.6.0a0|>=3.6,<3.7.0a0']
Package sqlite conflicts for:
python=3.5 -> sqlite[version='>=3.20.1,<4.0a0|>=3.22.0,<4.0a0|>=3.23.1,<4.0a0|>=3.24.0,<4.0a0']
Package pip conflicts for:
python=3.5 -> pip
Package mkl conflicts for:
pytorch=0.3.1 -> mkl[version='>=2018.0.2,<2019.0a0']
Package ncurses conflicts for:
python=3.5 -> ncurses[version='6.0.*|>=6.0,<7.0a0|>=6.1,<7.0a0']


title: How to save Pytorch tensor attributes to disk?

body:
I used setattr to save some information with a Pytorch Tensor, which can be retrieved as expected:

import torch
before_write = torch.Tensor()
setattr(before_write, "features", "my_features")
print(before_write.features)
> 'my_features'

Unexpectedly, when I write the tensor to disk, and then read it back in again, the features have disappeared:

torch.save(before_write, "~/my_tensor")
after_write = torch.load("~/my_tensor")
print(after_write.features)
> AttributeError: 'Tensor' object has no attribute 'features'

Why is the attribute lost when writing the tensor to disk? Is this expected behaviour? Would there be a workaround to save the features together with the tensor?

NB: using an empty Python class and writing it with pickle, does retain the attributes.


title: Loading a Pytorch model into C++ Using Dev Pytorch 1.0

body:
Pytorch 1.0 has a feature to convert a model into a torch script program (serialized in a way) to enable its execution in C++ without Python dependencies.

The details are in this tutorial.
https://pytorch.org/tutorials/advanced/cpp_export.html

This is how it is done:

import torch
import torchvision

# An instance of your model.
model = A UNET MODEL FROM FASTAI which has hooks as required by UNET

# An example input you would normally provide to your model's forward() method.
example = torch.rand(1, 3, 224, 224)

# Use torch.jit.trace to generate a torch.jit.ScriptModule via tracing.
traced_script_module = torch.jit.trace(model, example)

In my use case, I am using a UNET Model for semantic segmentation. However, I trace the model using this method, I get the following error.

Forward or backward hooks can't be compiled

UNET Model uses hooks to save intermediate features which is used at later layers in the network. Is there a way around it? or This is still a limitation of this new method that it cannot work with Models using such hooks.


title: Pytorch backprop is slower compared to Tensorflow?

body:
I’ve implemented a simple DDQN network in pytorch and tensorflow. The network is quite shallow.
While the forward pass is much faster in PyTorch compared to TF, the back-propagation step is much slower compared to TF. Both backprop steps were done on the CPU.
Any ideas on how to improve it.

The network part is:

def __init__(self, hidden_size_IP=100, hidden_size_rest=100, alpha=0.01, state_size=27, action_size=8,
             learning_rate=1e-6):
    super().__init__()

    # build hidden layers
    self.l1 = nn.Sequential(nn.Linear(in_features=500, out_features=400),
                            nn.LeakyReLU(negative_slope=alpha))
    self.l2 = nn.Sequential(nn.Linear(in_features=400, out_features=200),
                            nn.LeakyReLU(negative_slope=alpha))
    self.l3 = nn.Sequential(nn.Linear(in_features=200, out_features=200),
                            nn.LeakyReLU(negative_slope=alpha))
    # build output layer
    self.Qval = nn.Linear(in_features=200, out_features=24)

def forward(self, observation):
    if isinstance(observation, np.ndarray):
        observation = torch.from_numpy(observation).float()
    out1 = self.l1(observation)
    out2 = self.l2(out1)
    out3 = self.l3(out2)
    qval = self.Qval(out3)
    return qval

and the backprop code can be, for example:

self.optimizer = optim.Adam(self.q_net.parameters(), lr=1e-4)
self.optimizer.zero_grad()

state_batch=torch.rand([64,500])
act_batch=np.randi(0,24,[64,1]
act_batch_torch=torch.as_tensor(act_batch)
label_batch = torch.rand([64,500])
Q=self.q_net.forward(state_batch).gather(1, act_batch_torch) # q_net is an instance of the network above
loss = mse_loss(input=Q, target=label_batch.detach())
loss.backward()

self.optimizer.step()

Note that since inference is much faster using the CPU, I’m also doing backprop on the CPU. I have tried transferring the network to the GPU, and then do a backprop on the GPU, but it turned out to be slower.

Any ideas why pyTorch is slower? How can I improve the speed for this type of shallow network?


title: Difference in Computation DFT between PyTorch and Numpy

body:
I am trying to convert a numpy implementation of DFT using numpy.fft.fft2 to all PyTorch. When compairing torch.fft, the results are not the same. Why is this?

Numpy

>>> np.fft.fft2(test_cp, axes=(1,2)).shape
(100, 32, 32, 16)
>>> np.fft.fft2(test_cp, axes=(1,2))[0][0][0]
array([ 121.36078 +0.j,  862.5427  +0.j, -478.19095 +0.j,   -3.937337+0.j,
        124.33441 +0.j,   42.13216 +0.j,  401.703   +0.j,   43.894173+0.j,
       -155.7095  +0.j,  513.4581  +0.j, -119.220665+0.j, -166.82913 +0.j,
       -954.76184 +0.j, -401.38803 +0.j, -189.80426 +0.j, -231.5777  +0.j],
      dtype=complex64)
>>> abs(np.fft.fft2(test_cp, axes=(1,2))[0][0][0])  # to compare directly to real numbers in torch
array([121.36078 , 862.5427  , 478.19095 ,   3.937337, 124.33441 ,
        42.13216 , 401.703   ,  43.894173, 155.7095  , 513.4581  ,
       119.220665, 166.82913 , 954.76184 , 401.38803 , 189.80426 ,
       231.5777  ], dtype=float32)

Pytorch
Looking at the real numbers from the output of (*, 2)

>>> torch.fft(layer_acts, signal_ndim=2).shape
torch.Size([100, 32, 32, 16, 2])
>>> torch.fft(layer_acts, signal_ndim=2)[:, :,:,:,0][0][0][0]
tensor([   6.9696,  -41.5982, -144.1784,  -54.1054,   -2.1185,  -78.1295,
          72.2305,  107.3909,  172.7782,  107.3909,   72.2305,  -78.1295,
          -2.1185,  -54.1054, -144.1784,  -41.5982], grad_fn=<SelectBackward>)

What makes the results different in the way numpy computes compared to torch?

Thanks!


title: Can conditionality be added inside Pytorch nn.Sequential()

body:
Is there a way to add conditional statements inside the nn.Sequential(). Something similar to the code below.

import torch

class Building_Blocks(torch.nn.Module):
  def conv_block (self, in_features, out_features, kernal_size, upsample=False):
    block = torch.nn.Sequential(
        torch.nn.Conv2d(in_features, out_features, kernal_size),
        torch.nn.ReLU(inplace = True),
        torch.nn.Conv2d(out_features, out_features, kernal_size),
        torch.nn.ReLU(inplace = True),
        if(upsample):
          torch.nn.ConvTranspose2d(out_features, out_features, kernal_size)
        )
    return block

  def __init__(self):
    super(Building_Blocks, self).__init__()
    self.contracting_layer1 = self.conv_block(3, 64, 3, upsample=True)

  def forward(self, x):
    x=self.contracting_layer1(x)
    return x


title: Pytorch DataLoader doesn't work with remote interpreter

body:
I have the following error.

Expected: /home/ubuntu/.pycharm_helpers/pydev/pydevd_attach_to_process/attach_linux_amd64.so to exist.

And Here is the code:

import torch_geometric.transforms as T
category = 'Airplane'
path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'ShapeNet')
transform = T.Compose([
    T.RandomTranslate(0.01),
    T.RandomRotate(15, axis=0),
    T.RandomRotate(15, axis=1),
    T.RandomRotate(15, axis=2)
])
pre_transform = T.NormalizeScale()
train_dataset = ShapeNet(path, category, split='trainval', transform=transform,
                         pre_transform=pre_transform)
test_dataset = ShapeNet(path, category, split='test',
                        pre_transform=pre_transform)
train_loader = DataLoader(train_dataset, batch_size=12, shuffle=True, num_workers=6)
test_loader = DataLoader(test_dataset, batch_size=12, shuffle=False,
                         num_workers=6)

When I'm trying sample from dataset using the dataloader, the debugger crashes and returns this erros.

Have tried deleting remote helpers, but it didn't solve my problem.
My local machine in running on windows 10 and the remote one is running on Ubuntu 18.04


title: How do I use torchaudio with torch_xla on google colab TPU?

body:
I was using google colab (GPU enabled) to train my Automatic Speech Recognition Model based on pytorch and torchaudio. But when I tried to use google colab TPU I got the following error when training my model :
ImportError: /usr/local/lib/python3.6/dist-packages/_torch_sox.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN6caffe28TypeMeta21_typeMetaDataInstanceISt7complexIfEEEPKNS_6detail12TypeMetaDataEv site:stackoverflow.com

I'm almost sure it's a conflict between torchaudio and pytorch_xla versions
Here's the code that I use to install the librairies :
VERSION = "nightly"  #@param ["1.5" , "20200325", "nightly"]
!curl https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py -o pytorch-xla-env-setup.py
!pip install torchvision==0.6.1
!pip install torch==1.5.1
!pip install torchaudio==0.5.1
!python pytorch-xla-env-setup.py --version $VERSION

The solution proposed here https://stackoverflow.com/a/60929133/13847989 generated the following error :
ImportError: /usr/local/lib/python3.6/dist-packages/_torch_sox.cpython-36m-x86_64-linux-gnu.so: undefined symbol: THPVariableClass

What am I doing wrong ?


title: Higher Accuracy using SimpleTransformers vs Transformers Library with BERT

body:
I am working on a project for text classification using BERT
I am getting ca. 90% accuracy by using simple transformers.
But I only get like 60% using my own training for-loop (not published here) or using the trainer module from the transformers library.
Both are done with the default parameters of simple transformers.
I am really struggling to understand why there is such a difference in performance
Dataset is from Kaggle: https://www.kaggle.com/ankurzing/sentiment-analysis-for-financial-news
Imports:
from transformers import BertForSequenceClassification, AdamW, BertTokenizer, get_linear_schedule_with_warmup, Trainer, TrainingArguments
import torch
from torch.utils.data import DataLoader, RandomSampler, SequentialSampler, TensorDataset
import pandas as pd
from pathlib import Path
import sklearn
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, precision_recall_fscore_support
import numpy as np
from torch.nn import functional as F
from collections import defaultdict
import random
from simpletransformers.classification import ClassificationModel

Data Pre-Processing:
#loading phrase bank dataset
phrase_bank_dataset = "all-data.csv"
phrase_bank_dataset_file = Path(phrase_bank_dataset)
file_loaded = False
while not file_loaded:
  if phrase_bank_dataset_file.exists():
    phrase_bank_dataset = pd.read_csv(phrase_bank_dataset, encoding='latin-1')
    phrase_bank_dataset = phrase_bank_dataset.values.tolist()
    file_loaded = True
    print("Dataset Loaded")
  else:
    print("File not Found")
#correcting the format of phrase bank dataset
phrase_dataset = pd.DataFrame(columns=["news", "sentiment"])
for ele in phrase_bank_dataset:
  news = ele[1]
  #converting sentiment text into numbers
  sentiment = 0 if ele[0] == 'negative' else 1 if ele[0] == 'neutral' else 2
  row = [news, sentiment]
  phrase_dataset.loc[len(phrase_dataset)] = row
print(phrase_dataset)

Simple Transformers Code:
model = ClassificationModel('bert', 'bert-base-cased', num_labels=3,use_cuda=True)
train,eva = train_test_split(labeled_dataset,test_size = 0.2)

train_df = pd.DataFrame({
    'text': train['news'],
    'label': train['sentiment']
})

eval_df = pd.DataFrame({
    'text': eva['news'],
    'label': eva['sentiment']
})

model.train_model(train_df)

result, model_outputs, wrong_predictions = model.eval_model(eval_df)

lst = []
for arr in model_outputs:
    lst.append(np.argmax(arr))
true = eval_df['label'].tolist()
predicted = lst
sklearn.metrics.accuracy_score(true,predicted)

Transformers Trainer Code:
tokenizer = BertTokenizer.from_pretrained('bert-base-cased', stride = 0.8)
model = BertForSequenceClassification.from_pretrained('bert-base-cased', num_labels = 3)

if torch.cuda.is_available():
  print("\nUsing: ", torch.cuda.get_device_name(0))
  device = torch.device('cuda')
else:
  print("\nUsing: CPU")
  device = torch.device('cpu')
model = model.to(device)

#custom dataset class

class NewsSentimentDataset(torch.utils.data.Dataset):
  def __init__(self, encodings, labels):
        self.encodings = encodings
        self.labels = labels

  def __getitem__(self, idx):
      item = {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}
      item['labels'] = torch.tensor(self.labels[idx])
      return item

  def __len__(self):
      return len(self.labels)

#method for tokenizing dataset list

def tokenize_headlines(headlines, labels, tokenizer):

  encodings = tokenizer.batch_encode_plus(
      headlines,
      add_special_tokens = True,
      pad_to_max_length = True,
      return_attention_mask = True
  )

  dataset = NewsSentimentDataset(encodings, labels)
  return dataset

#splitting dataset into training and validation set
all_headlines = phrase_dataset['news'].tolist()
all_labels = phrase_dataset['sentiment'].tolist()

train_headlines, val_headlines, train_labels, val_labels = train_test_split(phrase_headlines, phrase_labels, test_size=.2)

val_dataset = tokenize_headlines(val_headlines, val_labels, tokenizer)
train_dataset = tokenize_headlines(train_headlines, val_labels, tokenizer)

#data loader
train_batch_size = 8
val_batch_size = 8

train_data_loader = DataLoader(train_dataset, batch_size = train_batch_size, sampler=RandomSampler(train_dataset))
val_data_loader = DataLoader(val_dataset, batch_size = val_batch_size, sampler=SequentialSampler(val_dataset))

#optimizer and scheduler
num_epochs = 1
num_steps = len(train_data_loader) * num_epochs
optimizer = AdamW(model.parameters(), lr=4e-5, eps=1e-8)
scheduler = get_linear_schedule_with_warmup(optimizer, num_warmup_steps=30, num_training_steps=num_steps)

#training and evaluation with trainer moduel from huggingfaces

def compute_metrics(pred):
    labels = pred.label_ids
    preds = pred.predictions.argmax(-1)
    precision, recall, f1, _ = precision_recall_fscore_support(labels, preds, average='micro')
    acc = accuracy_score(labels, preds)
    return {
        'accuracy': acc,
        'f1': f1,
        'precision': precision,
        'recall': recall
    }

training_args = TrainingArguments(
    output_dir='./results',          # output directory
    num_train_epochs=1,              # total number of training epochs
    per_device_train_batch_size=8,  # batch size per device during training
    per_device_eval_batch_size=8,   # batch size for evaluation
    warmup_steps=0,                # number of warmup steps for learning rate scheduler
    weight_decay=0,               # strength of weight decay
    logging_dir='./logs',            # directory for storing logs
    logging_steps=10,
)

trainer = Trainer(
    model=model,                         # the instantiated  Transformers model to be trained
    args=training_args,                  # training arguments, defined above
    train_dataset=train_dataset,         # training dataset
    eval_dataset=val_dataset  ,          # evaluation dataset
    compute_metrics=compute_metrics
)

trainer.train()
trainer.evaluate()


title: How forward pass and inference are different in pytorch?

body:
Looking at encoder part of Tacotron2 model:

https://github.com/mozilla/TTS/blob/master/layers/tacotron2.py#L63

https://github.com/mozilla/TTS/blob/master/layers/tacotron2.py#L77

I wonder why forward and inference are different?

def forward(self, x, input_lengths):
    x = self.convolutions(x)
    x = x.transpose(1, 2)
    input_lengths = input_lengths.cpu().numpy()
    x = nn.utils.rnn.pack_padded_sequence(
        x, input_lengths, batch_first=True)
    self.lstm.flatten_parameters()
    outputs, _ = self.lstm(x)
    outputs, _ = nn.utils.rnn.pad_packed_sequence(
        outputs,
        batch_first=True,
    )
    return outputs

def inference(self, x):
    x = self.convolutions(x)
    x = x.transpose(1, 2)
    self.lstm.flatten_parameters()
    outputs, _ = self.lstm(x)
    return outputs

Is it related to LSTM layer only?