title: Updating packages in conda

body:
I have a problem with updating packages in conda. The list of my installed packages is:
#
# Name                    Version                   Build  Channel
_anaconda_depends         2020.07                  py38_0
_ipyw_jlab_nb_ext_conf    0.1.0                    py38_0
alabaster                 0.7.12             pyhd3eb1b0_0
anaconda                  custom                   py38_1
anaconda-client           1.7.2                    py38_0
anaconda-navigator        1.10.0                   py38_0
anaconda-project          0.9.1              pyhd3eb1b0_1
argh                      0.26.2                   py38_0
argon2-cffi               20.1.0           py38h2bbff1b_1
ase                       3.21.0                   pypi_0    pypi
asn1crypto                1.4.0                      py_0
astroid                   2.5              py38haa95532_1
astropy                   4.2.1            py38h2bbff1b_1
async_generator           1.10               pyhd3eb1b0_0
atomicwrites              1.4.0                      py_0
attrs                     20.3.0             pyhd3eb1b0_0
autopep8                  1.5.6              pyhd3eb1b0_0
babel                     2.9.0              pyhd3eb1b0_0
backcall                  0.2.0              pyhd3eb1b0_0
backports                 1.0                pyhd3eb1b0_2
backports.functools_lru_cache 1.6.3              pyhd3eb1b0_0
backports.shutil_get_terminal_size 1.0.0              pyhd3eb1b0_3
backports.tempfile        1.0                pyhd3eb1b0_1
backports.weakref         1.0.post1                  py_1
bcrypt                    3.2.0            py38he774522_0
beautifulsoup4            4.9.3              pyha847dfd_0
bitarray                  1.9.1            py38h2bbff1b_1
bkcharts                  0.2                      py38_0
blas                      1.0                         mkl
bleach                    3.3.0              pyhd3eb1b0_0
blosc                     1.21.0               h19a0ad4_0
bokeh                     2.3.1            py38haa95532_0
boto                      2.49.0                   py38_0
bottleneck                1.3.2            py38h2a96729_1
brotli                    1.0.9                ha925a31_2
brotlipy                  0.7.0           py38h2bbff1b_1003
bzip2                     1.0.8                he774522_0
ca-certificates           2021.1.19            haa95532_1
certifi                   2020.12.5        py38haa95532_0
cffi                      1.14.5           py38hcd4344a_0
chardet                   4.0.0           py38haa95532_1003
charls                    2.2.0                h6c2663c_0
click                     7.1.2              pyhd3eb1b0_0
cloudpickle               1.6.0                      py_0
clyent                    1.2.2                    py38_1
colorama                  0.4.4              pyhd3eb1b0_0
comtypes                  1.1.9           py38haa95532_1002
conda                     4.10.0           py38haa95532_0
conda-build               3.20.5                   py38_1
conda-env                 2.6.0                         1
conda-package-handling    1.7.3            py38h8cc25b3_1
conda-verify              3.4.2                      py_1
console_shortcut          0.1.1                         4
contextlib2               0.6.0.post1                py_0
cpuonly                   1.0                           0    pytorch
cryptography              3.4.7            py38h71e12ea_0
curl                      7.71.1               h2a8f88b_1
cycler                    0.10.0                   py38_0
cython                    0.29.22          py38hd77b12b_0
cytoolz                   0.11.0           py38he774522_0
dask                      2021.4.0           pyhd3eb1b0_0
dask-core                 2021.4.0           pyhd3eb1b0_0
decorator                 5.0.6              pyhd3eb1b0_0
defusedxml                0.7.1              pyhd3eb1b0_0
diff-match-patch          20200713                   py_0
distributed               2021.4.0         py38haa95532_0
docutils                  0.17             py38haa95532_1
entrypoints               0.3                      py38_0
et_xmlfile                1.0.1                   py_1001
fastcache                 1.1.0            py38he774522_0
filelock                  3.0.12             pyhd3eb1b0_1
flake8                    3.9.0              pyhd3eb1b0_0
flask                     1.1.2              pyhd3eb1b0_0
freetype                  2.10.4               hd328e21_0
fsspec                    0.9.0              pyhd3eb1b0_0
future                    0.18.2                   py38_1
get_terminal_size         1.0.0                h38e98db_0
gevent                    21.1.2           py38h2bbff1b_1
giflib                    5.2.1                h62dcd97_0
glob2                     0.7                pyhd3eb1b0_0
gmpy2                     2.0.8            py38h7edee0f_3
googledrivedownloader     0.4                      pypi_0    pypi
greenlet                  1.0.0            py38hd77b12b_2
h5py                      2.10.0           py38h5e291fa_0
hdf5                      1.10.4               h7ebc959_0
heapdict                  1.0.1                      py_0
html5lib                  1.1                        py_0
icc_rt                    2019.0.0             h0cc432a_1
icu                       58.2                 ha925a31_3
idna                      2.10               pyhd3eb1b0_0
imagecodecs               2021.3.31        py38h5da4933_0
imageio                   2.9.0              pyhd3eb1b0_0
imagesize                 1.2.0              pyhd3eb1b0_0
importlib-metadata        3.10.0           py38haa95532_0
importlib_metadata        3.10.0               hd3eb1b0_0
iniconfig                 1.1.1              pyhd3eb1b0_0
intel-openmp              2020.2                      254
intervaltree              3.1.0                      py_0
ipykernel                 5.3.4            py38h5ca1d4c_0
ipynb                     0.5.1                    pypi_0    pypi
ipython                   7.22.0           py38hd4e2768_0
ipython_genutils          0.2.0              pyhd3eb1b0_1
ipywidgets                7.6.3              pyhd3eb1b0_1
isodate                   0.6.0                    pypi_0    pypi
isort                     5.8.0              pyhd3eb1b0_0
itsdangerous              1.1.0              pyhd3eb1b0_0
jdcal                     1.4.1                      py_0
jedi                      0.17.1                   py38_0
jinja2                    2.11.3             pyhd3eb1b0_0
joblib                    1.0.1              pyhd3eb1b0_0
jovian                    0.2.28                   pypi_0    pypi
jpeg                      9b                   hb83a4c4_2
json5                     0.9.5                      py_0
jsonschema                3.2.0                      py_2
jupyter                   1.0.0                    py38_7
jupyter_client            6.1.12             pyhd3eb1b0_0
jupyter_console           6.4.0              pyhd3eb1b0_0
jupyter_core              4.7.1            py38haa95532_0
jupyterlab                2.2.6                      py_0
jupyterlab_pygments       0.1.2                      py_0
jupyterlab_server         1.2.0                      py_0
jupyterlab_widgets        1.0.0              pyhd3eb1b0_1
keyring                   22.3.0           py38haa95532_0
kiwisolver                1.3.1            py38hd77b12b_0
krb5                      1.18.2               hc04afaa_0
lazy-object-proxy         1.6.0            py38h2bbff1b_0
lcms2                     2.12                 h83e58a3_0
lerc                      2.2.1                hd77b12b_0
libaec                    1.0.4                h33f27b4_1
libarchive                3.4.2                h5e25573_0
libcurl                   7.71.1               h2a8f88b_1
libdeflate                1.7                  h2bbff1b_5
libiconv                  1.15                 h1df5818_7
liblief                   0.10.1               ha925a31_0
libllvm9                  9.0.1                h21ff451_0
libpng                    1.6.37               h2a8f88b_0
libsodium                 1.0.18               h62dcd97_0
libspatialindex           1.9.3                h6c2663c_0
libssh2                   1.9.0                h7a1dbc1_1
libtiff                   4.2.0                hd0e1b90_0
libuv                     1.40.0               he774522_0
libxml2                   2.9.10               hb89e7f3_3
libxslt                   1.1.34               he774522_0
libzopfli                 1.0.3                ha925a31_0
llvmlite                  0.36.0           py38h34b8924_4
locket                    0.2.1            py38haa95532_1
lxml                      4.6.3            py38h9b66d53_0
lz4-c                     1.9.3                h2bbff1b_0
lzo                       2.10                 he774522_2
m2w64-gcc-libgfortran     5.3.0                         6
m2w64-gcc-libs            5.3.0                         7
m2w64-gcc-libs-core       5.3.0                         7
m2w64-gmp                 6.1.0                         2
m2w64-libwinpthread-git   5.0.0.4634.697f757               2
markupsafe                1.1.1            py38he774522_0
matplotlib                3.3.4            py38haa95532_0
matplotlib-base           3.3.4            py38h49ac443_0
mccabe                    0.6.1                    py38_1
menuinst                  1.4.16           py38he774522_1
mistune                   0.8.4           py38he774522_1000
mkl                       2020.2                      256
mkl-service               2.3.0            py38h196d8e1_0
mkl_fft                   1.3.0            py38h46781fe_0
mkl_random                1.1.1            py38h47e9c7a_0
mock                      4.0.3              pyhd3eb1b0_0
more-itertools            8.7.0              pyhd3eb1b0_0
mpc                       1.1.0                h7edee0f_1
mpfr                      4.0.2                h62dcd97_1
mpir                      3.0.0                hec2e145_1
mpmath                    1.2.1            py38haa95532_0
msgpack-python            1.0.2            py38h59b6b97_1
msys2-conda-epoch         20160418                      1
multipledispatch          0.6.0                    py38_0
navigator-updater         0.2.1                    py38_0
nbclient                  0.5.3              pyhd3eb1b0_0
nbconvert                 6.0.7                    py38_0
nbformat                  5.1.3              pyhd3eb1b0_0
nest-asyncio              1.5.1              pyhd3eb1b0_0
networkx                  2.5                        py_0
ninja                     1.10.2               h6d14046_1
nltk                      3.6.1              pyhd3eb1b0_0
nose                      1.3.7           pyhd3eb1b0_1006
notebook                  6.3.0            py38haa95532_0
numba                     0.53.1           py38hf11a4ad_0
numexpr                   2.7.3            py38hcbcaa1e_0
numpy                     1.19.2           py38hadc3359_0
numpy-base                1.19.2           py38ha3acd2a_0
numpydoc                  1.1.0              pyhd3eb1b0_1
olefile                   0.46                       py_0
opencv-python             4.5.1.48                 pypi_0    pypi
openjpeg                  2.3.0                h5ec785f_1
openpyxl                  3.0.7              pyhd3eb1b0_0
openssl                   1.1.1k               h2bbff1b_0
packaging                 20.9               pyhd3eb1b0_0
pandas                    1.2.3            py38hf11a4ad_0
pandoc                    2.12                 haa95532_0
pandocfilters             1.4.3            py38haa95532_1
paramiko                  2.7.2                      py_0
parso                     0.7.0                      py_0
partd                     1.2.0              pyhd3eb1b0_0
path                      15.1.2           py38haa95532_0
path.py                   12.5.0                        0
pathlib2                  2.3.5            py38haa95532_2
pathtools                 0.1.2                      py_1
patsy                     0.5.1                    py38_0
pep8                      1.7.1                    py38_0
pexpect                   4.8.0              pyhd3eb1b0_3
pickleshare               0.7.5           pyhd3eb1b0_1003
pillow                    8.2.0            py38h4fa10fc_0
pip                       20.2.4           py38haa95532_0
pkginfo                   1.7.0            py38haa95532_0
pluggy                    0.13.1           py38haa95532_0
ply                       3.11                     py38_0
powershell_shortcut       0.0.1                         3
prometheus_client         0.10.1             pyhd3eb1b0_0
prompt-toolkit            3.0.17             pyh06a4308_0
prompt_toolkit            3.0.17               hd3eb1b0_0
protobuf                  3.14.0                   pypi_0    pypi
psutil                    5.8.0            py38h2bbff1b_1
ptyprocess                0.7.0              pyhd3eb1b0_2
py                        1.10.0             pyhd3eb1b0_0
py-lief                   0.10.1           py38ha925a31_0
pycodestyle               2.6.0              pyhd3eb1b0_0
pycosat                   0.6.3            py38h2bbff1b_0
pycparser                 2.20                       py_2
pycurl                    7.43.0.6         py38h7a1dbc1_0
pydocstyle                6.0.0              pyhd3eb1b0_0
pyerfa                    1.7.2            py38h2bbff1b_0
pyflakes                  2.2.0              pyhd3eb1b0_0
pygments                  2.8.1              pyhd3eb1b0_0
pylint                    2.7.4            py38haa95532_1
pynacl                    1.4.0            py38h62dcd97_1
pyodbc                    4.0.30           py38ha925a31_0
pyopenssl                 20.0.1             pyhd3eb1b0_1
pyparsing                 2.4.7              pyhd3eb1b0_0
pyqt                      5.9.2            py38ha925a31_4
pyreadline                2.1                      py38_1
pyrsistent                0.17.3           py38he774522_0
pysocks                   1.7.1            py38haa95532_0
pytables                  3.6.1            py38ha5be198_0
pytest                    6.2.3            py38haa95532_2
python                    3.8.5                h5fd99cc_1
python-dateutil           2.8.1              pyhd3eb1b0_0
python-jsonrpc-server     0.4.0                      py_0
python-language-server    0.35.1                     py_0
python-libarchive-c       2.9                pyhd3eb1b0_1
python-louvain            0.15                     pypi_0    pypi
pytorch                   1.7.1               py3.8_cpu_0  [cpuonly]  pytorch
pytz                      2021.1             pyhd3eb1b0_0
pywavelets                1.1.1            py38he774522_2
pywin32                   227              py38he774522_1
pywin32-ctypes            0.2.0                 py38_1000
pywinpty                  0.5.7                    py38_0
pyyaml                    5.4.1            py38h2bbff1b_1
pyzmq                     20.0.0           py38hd77b12b_1
qdarkstyle                3.0.2              pyhd3eb1b0_0
qt                        5.9.7            vc14h73c81de_0
qtawesome                 1.0.2              pyhd3eb1b0_0
qtconsole                 5.0.3              pyhd3eb1b0_0
qtpy                      1.9.0                      py_0
rdflib                    5.0.0                    pypi_0    pypi
regex                     2021.4.4         py38h2bbff1b_0
requests                  2.25.1             pyhd3eb1b0_0
rope                      0.18.0                     py_0
rtree                     0.9.4            py38h21ff451_1
ruamel_yaml               0.15.100         py38h2bbff1b_0
scikit-image              0.18.1           py38hf11a4ad_0
scikit-learn              0.24.1           py38hf11a4ad_0
scipy                     1.6.2            py38h14eb087_0
seaborn                   0.11.1             pyhd3eb1b0_0
send2trash                1.5.0              pyhd3eb1b0_1
setuptools                52.0.0           py38haa95532_0
simplegeneric             0.8.1                    py38_2
singledispatch            3.6.1           pyhd3eb1b0_1001
sip                       4.19.13          py38ha925a31_0
six                       1.15.0           py38haa95532_0
snappy                    1.1.8                h33f27b4_0
snowballstemmer           2.1.0              pyhd3eb1b0_0
sortedcollections         2.1.0              pyhd3eb1b0_0
sortedcontainers          2.3.0              pyhd3eb1b0_0
soupsieve                 2.2.1              pyhd3eb1b0_0
sphinx                    3.2.1                      py_0
sphinxcontrib             1.0                      py38_1
sphinxcontrib-applehelp   1.0.2              pyhd3eb1b0_0
sphinxcontrib-devhelp     1.0.2              pyhd3eb1b0_0
sphinxcontrib-htmlhelp    1.0.3              pyhd3eb1b0_0
sphinxcontrib-jsmath      1.0.1              pyhd3eb1b0_0
sphinxcontrib-qthelp      1.0.3              pyhd3eb1b0_0
sphinxcontrib-serializinghtml 1.1.4              pyhd3eb1b0_0
sphinxcontrib-websupport  1.2.4                      py_0
spyder                    4.1.5                    py38_0
spyder-kernels            1.9.4                    py38_0
sqlalchemy                1.4.7            py38h2bbff1b_0
sqlite                    3.35.4               h2bbff1b_0
statsmodels               0.12.2           py38h2bbff1b_0
sympy                     1.7.1            py38haa95532_0
tbb                       2020.3               h74a9793_0
tblib                     1.7.0                      py_0
tensorboardx              2.1                      pypi_0    pypi
terminado                 0.9.4            py38haa95532_0
testpath                  0.4.4              pyhd3eb1b0_0
threadpoolctl             2.1.0              pyh5ca1d4c_0
tifffile                  2021.3.31          pyhd3eb1b0_1
tk                        8.6.10               he774522_0
toml                      0.10.2             pyhd3eb1b0_0
toolz                     0.11.1             pyhd3eb1b0_0
torch-cluster             1.5.8                    pypi_0    pypi
torch-geometric           1.6.3                    pypi_0    pypi
torch-scatter             2.0.5                    pypi_0    pypi
torch-sparse              0.6.8                    pypi_0    pypi
torch-spline-conv         1.2.0                    pypi_0    pypi
torchaudio                0.7.2                      py38    pytorch
torchvision               0.8.2                  py38_cpu  [cpuonly]  pytorch
tornado                   6.1              py38h2bbff1b_0
tqdm                      4.59.0             pyhd3eb1b0_1
traitlets                 5.0.5              pyhd3eb1b0_0
typing_extensions         3.7.4.3            pyha847dfd_0
ujson                     4.0.2            py38hd77b12b_0
unicodecsv                0.14.1                   py38_0
urllib3                   1.26.4             pyhd3eb1b0_0
uuid                      1.30                     pypi_0    pypi
vc                        14.2                 h21ff451_1
vs2015_runtime            14.27.29016          h5e58377_2
watchdog                  1.0.2            py38haa95532_1
wcwidth                   0.2.5                      py_0
webencodings              0.5.1                    py38_1
werkzeug                  1.0.1              pyhd3eb1b0_0
wheel                     0.36.2             pyhd3eb1b0_0
widgetsnbextension        3.5.1                    py38_0
win_inet_pton             1.1.0            py38haa95532_0
win_unicode_console       0.5                      py38_0
wincertstore              0.2                      py38_0
winpty                    0.4.3                         4
wrapt                     1.12.1           py38he774522_1
xlrd                      2.0.1              pyhd3eb1b0_0
xlsxwriter                1.3.8              pyhd3eb1b0_0
xlwings                   0.23.0           py38haa95532_0
xlwt                      1.3.0                    py38_0
xmltodict                 0.12.0                     py_0
xz                        5.2.5                h62dcd97_0
yaml                      0.2.5                he774522_0
yapf                      0.31.0             pyhd3eb1b0_0
zeromq                    4.3.3                ha925a31_3
zfp                       0.5.5                hd77b12b_6
zict                      2.0.0              pyhd3eb1b0_0
zipp                      3.4.1              pyhd3eb1b0_0
zlib                      1.2.11               h62dcd97_4
zope                      1.0                      py38_1
zope.event                4.5.0                    py38_0
zope.interface            5.3.0            py38h2bbff1b_0
zstd                      1.4.5                h04227a9_0

and I want to update
torch-cluster             1.5.8                    pypi_0    pypi
torch-geometric           1.6.3                    pypi_0    pypi
torch-scatter             2.0.5                    pypi_0    pypi
torch-sparse              0.6.8                    pypi_0    pypi
torch-spline-conv         1.2.0                    pypi_0    pypi

However, when I enter the command
conda update torch-cluster

I get
PackageNotInstalledError: Package is not installed in prefix.
  prefix: C:\Users\Rostyslav\anaconda3
  package name: torch-cluster

I may be wrong about it, but I think the reason of the error is that the packages that I want to update are located in different channels, namely pypi. But even if it true, I do not understand how to reach this channel.
Can anybody explain?
Thanks a lot!


title: PyTorch DataLoader uses identical random transformation across each epoch

body:
There is a bug in PyTorch/Numpy where when loading batches in parallel with a DataLoader (i.e. setting num_workers > 1), the same NumPy random seed is used for each worker, resulting in any random functions applied being identical across parallelized batches. This can be resolved by passing a seed generator to the worker_init_fn argument like so.
However the issue persists across multiple epochs.
Minimal example:
import numpy as np
from torch.utils.data import Dataset, DataLoader

class RandomDataset(Dataset):
    def __getitem__(self, index):
        return np.random.randint(0, 1000, 2)

    def __len__(self):
        return 4

dataset = RandomDataset()
dataloader = DataLoader(dataset, batch_size=1,
                        num_workers=2,
                        worker_init_fn = lambda x: np.random.seed(x))

for epoch in range(3):
    print(f'\nEpoch {epoch}')
    for batch in dataloader:
        print(batch)

As you can see, while parallelized batches within an epoch now produce different results, the results are identical across epochs:
Epoch 0
tensor([[684, 559]])
tensor([[ 37, 235]])
tensor([[629, 192]])
tensor([[908,  72]])

Epoch 1
tensor([[684, 559]])
tensor([[ 37, 235]])
tensor([[629, 192]])
tensor([[908,  72]])

Epoch 2
tensor([[684, 559]])
tensor([[ 37, 235]])
tensor([[629, 192]])
tensor([[908,  72]])

How can this be behaviour be fixed?

Using an empty argument e.g. worker_init_fn = lambda _: np.random.seed() appears to fix this - are there any issues with this workaround?


title: ouputs in semantic segmentation task

body:
Im working on the Kaggle semantic segmentation task,

In the testing part of my code,

model = model.eval()
predictions =[]
for data in testdataloader:
    data = t.autograd.Variable(data, volatile=True).cuda()
    output = model.forward(data)
    _,preds = t.max(output, 1, keepdim = True)

when i do the preds part,the array is only filled with ,i was hoping it to be an array of maximum locations im not sure what is going wrong.
The output part works well,I have attached a screenshot for visualization of output

Any sugestions on what is going wrong would be really helpful.

thanks


title: Multiple matrix multiplication loses weight updates

body:
When in forward method I only do one set of torch.add(torch.bmm(x, exp_w), self.b) then my model is back propagating correctly. When I add another layer - torch.add(torch.bmm(out, exp_w2), self.b2) - then the gradients are not updated and the model isn't learning. If I change the activation function from nn.Sigmoid to nn.ReLU then it works with two layers.

Been thinking about this a day now, and not figuring out why it's not working with nn.Sigmoid.

I've tried different learning rates, Loss functions and optimization functions, but no combination seems to work. When I add the weights together before and after training they are the same.

Code:

class MyModel(nn.Module):

    def __init__(self, input_dim, output_dim):
        torch.manual_seed(1)
        super(MyModel, self).__init__()
        self.input_dim = input_dim
        self.output_dim = output_dim
        hidden_1_dimentsions = 20
        self.w = torch.nn.Parameter(torch.empty(input_dim, hidden_1_dimentsions).uniform_(0, 1))
        self.b = torch.nn.Parameter(torch.empty(hidden_1_dimentsions).uniform_(0, 1))

        self.w2 = torch.nn.Parameter(torch.empty(hidden_1_dimentsions, output_dim).uniform_(0, 1))
        self.b2 = torch.nn.Parameter(torch.empty(output_dim).uniform_(0, 1))

    def activation(self):
        return torch.nn.Sigmoid()

    def forward(self, x):
        x = x.view((x.shape[0], 1, self.input_dim))

        exp_w = self.w.expand(x.shape[0], self.w.size(0), self.w.size(1))
        out = torch.add(torch.bmm(x, exp_w), self.b)
        exp_w2 = self.w2.expand(out.shape[0], self.w2.size(0), self.w2.size(1))
        out = torch.add(torch.bmm(out, exp_w2), self.b2)
        out = self.activation()(out)
        return out.view(x.shape[0])


title: How to get this variational auto encoder to use images with different resolution?

body:
I am trying to get an existing variational auto encoder to work with RGB images that have a resolution of 254 * 254

The code for the variational auto encoder works great for a dataset with resolutions of 100 x 100. However, I have no idea how to modify it to work with images of 254 X 254.

batch_size = 5
epochs = 10
size = 15
no_cuda = True
seed = 1
log_interval = 50
results_path = 'C:\\results\\'
train_root = 'C:\\train\\'
val_root = 'C:\\train\\'

cuda = not no_cuda and torch.cuda.is_available()

torch.manual_seed(seed)

device = torch.device("cuda" if cuda else "cpu")
kwargs = {'num_workers': 1, 'pin_memory': True} if cuda else {}

train_loader_food = torch.utils.data.DataLoader(
    datasets.ImageFolder(train_root, transform=transforms.ToTensor()),
    batch_size = batch_size, shuffle=True, **kwargs)

val_loader_food = torch.utils.data.DataLoader(
    datasets.ImageFolder(val_root, transform=transforms.ToTensor()),
    batch_size = batch_size, shuffle=True, **kwargs)

class VAE_CNN(nn.Module):
    def __init__(self):
        super(VAE_CNN, self).__init__()

        # Encoder
        self.conv1 = nn.Conv2d(3, 16, kernel_size=3, stride=1, padding=1, bias=False)
        self.bn1 = nn.BatchNorm2d(16)
        self.conv2 = nn.Conv2d(16, 32, kernel_size=3, stride=2, padding=1, bias=False)
        self.bn2 = nn.BatchNorm2d(32)
        self.conv3 = nn.Conv2d(32, 64, kernel_size=3, stride=1, padding=1, bias=False)
        self.bn3 = nn.BatchNorm2d(64)
        self.conv4 = nn.Conv2d(64, 16, kernel_size=3, stride=2, padding=1, bias=False)
        self.bn4 = nn.BatchNorm2d(16)

        # Latent vectors mu and sigma
        self.fc1 = nn.Linear(25 * 25 * 16, 2048)
        self.fc_bn1 = nn.BatchNorm1d(2048)
        self.fc21 = nn.Linear(2048, 2048)
        self.fc22 = nn.Linear(2048, 2048)

        # Sampling vector
        self.fc3 = nn.Linear(2048, 2048)
        self.fc_bn3 = nn.BatchNorm1d(2048)
        self.fc4 = nn.Linear(2048, 25 * 25 * 16)
        self.fc_bn4 = nn.BatchNorm1d(25 * 25 * 16)

        # Decoder
        self.conv5 = nn.ConvTranspose2d(16, 64, kernel_size=3, stride=2, padding=1, output_padding=1, bias=False)
        self.bn5 = nn.BatchNorm2d(64)
        self.conv6 = nn.ConvTranspose2d(64, 32, kernel_size=3, stride=1, padding=1, bias=False)
        self.bn6 = nn.BatchNorm2d(32)
        self.conv7 = nn.ConvTranspose2d(32, 16, kernel_size=3, stride=2, padding=1, output_padding=1, bias=False)
        self.bn7 = nn.BatchNorm2d(16)
        self.conv8 = nn.ConvTranspose2d(16, 3, kernel_size=3, stride=1, padding=1, bias=False)

        self.relu = nn.ReLU()

    def encode(self, x):
        conv1 = self.relu(self.bn1(self.conv1(x)))
        conv2 = self.relu(self.bn2(self.conv2(conv1)))
        conv3 = self.relu(self.bn3(self.conv3(conv2)))
        conv4 = self.relu(self.bn4(self.conv4(conv3))).view(-1, 25 * 25 * 16)

        fc1 = self.relu(self.fc_bn1(self.fc1(conv4)))

        r1 = self.fc21(fc1)
        r2 = self.fc22(fc1)

        return r1, r2

    def reparameterize(self, mu, logvar):
        if self.training:
            std = logvar.mul(0.5).exp_()
            eps = Variable(std.data.new(std.size()).normal_())
            return eps.mul(std).add_(mu)
        else:
            return mu

    def decode(self, z):
        fc3 = self.relu(self.fc_bn3(self.fc3(z)))
        fc4 = self.relu(self.fc_bn4(self.fc4(fc3))).view(-1, 16, 25, 25)

        conv5 = self.relu(self.bn5(self.conv5(fc4)))
        conv6 = self.relu(self.bn6(self.conv6(conv5)))
        conv7 = self.relu(self.bn7(self.conv7(conv6)))
        return self.conv8(conv7).view(-1, 3, 100, 100)

    def forward(self, x):
        mu, logvar = self.encode(x)
        z = self.reparameterize(mu, logvar)
        return self.decode(z), mu, logvar

class customLoss(nn.Module):
    def __init__(self):
        super(customLoss, self).__init__()
        self.mse_loss = nn.MSELoss(reduction="sum")

    def forward(self, x_recon, x, mu, logvar):
        loss_MSE = self.mse_loss(x_recon, x)
        loss_KLD = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp())

        return loss_MSE + loss_KLD

model = VAE_CNN().to(device)
optimizer = optim.Adam(model.parameters(), lr=1e-3)

loss_mse = customLoss()

val_losses = []
train_losses = []

def train(epoch):
    model.train()
    train_loss = 0
    for batch_idx, (data, _) in enumerate(train_loader_food):
        data = data.to(device)
        optimizer.zero_grad()
        recon_batch, mu, logvar = model(data)
        loss = loss_mse(recon_batch, data, mu, logvar)
        loss.backward()
        train_loss += loss.item()
        optimizer.step()
        if batch_idx % log_interval == 0:
            print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                epoch, batch_idx * len(data), len(train_loader_food.dataset),
                       100. * batch_idx / len(train_loader_food),
                       loss.item() / len(data)))

    print('====> Epoch: {} Average loss: {:.4f}'.format(
        epoch, train_loss / len(train_loader_food.dataset)))
    train_losses.append(train_loss / len(train_loader_food.dataset))

I replaced any refence to 25 * 25 * 16 with 128 * 128 * 4 etc.. assuming that would fix the issue but it just yields the following error:

shape '[-1, 64516]' is invalid for input of size 327680


title: Any guarantees that Torch won't mess up with an already allocated CUDA array?

body:
Assume we allocated some array on our GPU through other means than PyTorch, for example by creating a GPU array using numba.cuda.device_array. Will PyTorch, when allocating later GPU memory for some tensors, accidentally overwrite the memory space that is being used for our first CUDA array? In general, since PyTorch and Numba use the same CUDA runtime and thus I assume the same mechanism for memory management, are they automatically aware of memory regions used by other CUDA programs or does each one of them see the entire GPU memory as his own? If it's the latter, is there a way to make them aware of allocations by other CUDA programs?

EDIT: figured this would be an important assumption: assume that all allocations are done by the same process.


title: How to solve size mismatch error in Python (Pytorch)

body:
I am currently working on multivariate linear regression using PyTorch and I am getting the following error, I did search a lot about this error and the only thing I got to know is that there is a size mismatch between data and labels. But how to solve this error. Please help me or show me the right way to solve this problem.

size mismatch, m1: [824 x 1], m2: [8 x 8]

import torch
import torch.nn as nn
import numpy as np

Xtr = np.loadtxt("TrainData.csv")
Ytr = np.loadtxt("TrainLabels.csv")

X_train = torch.FloatTensor(Xtr)
Y_train = torch.FloatTensor(Ytr)

#### MODEL ARCHITECTURE ####

class Model(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = torch.nn.Linear(8,8)
        self.lin2 = torch.nn.Linear(8,1)

    def forward(self, x):
        x = self.lin2(x)
        y_pred = self.linear(x)
        return y_pred

model = Model()

loss_func = nn.MSELoss(size_average=False)
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
#print(len(list(model.parameters())))
def count_params(model):
    return sum(p.numel() for p in model.parameters() if p.requires_grad)

### TRAINING
for epoch in range(2):
    y_pred = model(X_train)

    loss = loss_func(y_pred, Y_train)

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    count = count_params(model)
    print(count)

test_exp = torch.FloatTensor([[6.0]])


title: persistent pip install in rapids.ai docker container

body: This is probably a really stupid question, but one has got to start somewhere. I am playing with NVDIA's rapids.ai gpu-enhanced docker container, but this (presumably by design) does not come with pytorch. Now, of course, I can do a pip install torch torch-ignite every time, but this is both annoying and resource-consuming (and pytorch is a large download). What is the approved method for persisting a pip install in a container?


title: How to deploy a pytorch model?

body:
I trained a model for Style Transfer and trained it on 1000's of images. I saved the Model for every 1000 of images and also saved the Transform Network final weights. Now I want it to be saved as a model so that I can use it in an app but none of the search are giving me a clear answer how to do that.

VG1 = vgg.VGG16("/kaggle/working/transformer_weight.pth")
example = torch.rand(1, 3, 800, 800)
traced_script_module = torch.jit.script(VG1, example)
traced_script_module.save('kaggle/working')

but it gives

RuntimeError:
Module 'Sequential' has no attribute '_modules' :
  File "/kaggle/working/vgg.py", line 43
        layers = {'3': 'relu1_2', '8': 'relu2_2', '15': 'relu3_3', '22': 'relu4_3'}
        features = {}
        for name, layer in self.features._modules.items():
                           ~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
            x = layer(x)
            if name in layers:

I am beginner and I have been trying for days. Pls tell if you want more information.
I want to save the model so that I can use it to an android studio to make app.

The notebook is in 'https://www.kaggle.com/starktony45/fast-neural-style'


title: How to pass an intermediate layer of one model to another model for skip connection in PyTorch

body:
I want to define an encoder decoder architecture as two separate models and later connect them using nn.Sequential() as shown in the code below. Now, let's say that I want to connect/concatenate the output of Encoder conv4 block to the deconv1 block of Decoder as a skip connection. Is there a way to achieve that without combining the two models (encoder and decoder) into one. I want to keep them separate to be able to use output of the same encoder as input of multiple decoders.

class Encoder(nn.Module):

    def __init__(self, conv_dim=64, n_res_blocks=2):
        super(Encoder, self).__init__()

        # Define the encoder
        self.conv1 = conv(3, conv_dim, 4)
        self.conv2 = conv(conv_dim, conv_dim*2, 4)
        self.conv3 = conv(conv_dim*2, conv_dim*4, 4)
        self.conv4 = conv(conv_dim*4, conv_dim*4, 4)

        # Define the resnet part of the encoder
        # Residual blocks
        res_layers = []
        for layer in range(n_res_blocks):
            res_layers.append(ResidualBlock(conv_dim*4))
        # use sequential to create these layers
        self.res_blocks = nn.Sequential(*res_layers)

        # leaky relu function
        self.leaky_relu = nn.LeakyReLU(negative_slope=0.2)

    def forward(self, x):
        # define feedforward behavior, applying activations as necessary
        conv1 = self.leaky_relu(self.conv1(x))
        conv2 = self.leaky_relu(self.conv2(conv1))
        conv3 = self.leaky_relu(self.conv3(conv2))
        conv4 = self.leaky_relu(self.conv4(conv3))

        out = self.res_blocks(conv4)

        return out

# Define the Decoder Architecture
class Decoder(nn.Module):

    def __init__(self, conv_dim=64, n_res_blocks=2):
        super(Decoder, self).__init__()

        # Define the resnet part of the decoder
        # Residual blocks
        res_layers = []
        for layer in range(n_res_blocks):
            res_layers.append(ResidualBlock(conv_dim*4))
        # use sequential to create these layers
        self.res_blocks = nn.Sequential(*res_layers)

        # Define the decoder
        self.deconv1 = deconv(conv_dim*4, conv_dim*4, 4)
        self.deconv2 = deconv(conv_dim*4, conv_dim*2, 4)
        self.deconv3 = deconv(conv_dim*2, conv_dim, 4)
        self.deconv4 = deconv(conv_dim, conv_dim, 4)

        # no batch norm on last layer
        self.out_layer = deconv(conv_dim, 3, 1, stride=1, padding=0, normalization=False)

        # leaky relu function
        self.leaky_relu = nn.LeakyReLU(negative_slope=0.2)

    def forward(self, x):
        # define feedforward behavior, applying activations as necessary
        res = self.res_blocks(x)

        deconv1 = self.leaky_relu(self.deconv1(res))
        deconv2 = self.leaky_relu(self.deconv2(deconv1))
        deconv3 = self.leaky_relu(self.deconv3(deconv2))
        deconv4 = self.leaky_relu(self.deconv4(deconv3))

        # tanh applied to last layer
        out = F.tanh(self.out_layer(deconv4))
        out = torch.clamp(out, min=-0.5, max=0.5)

        return out

def model():

    enc = Encoder(conv_dim=64, n_res_blocks=2)
    dec = Decoder(conv_dim=64, n_res_blocks=2)
    return nn.Sequential(enc, dec)


title: Implementing Coefficient of Variation Loss on Pytorch (Triplet Networks)

body:
I've been recently working on supervised contrastive learning. After several experiments using the triplet loss for image classification, I decided to implement a new function to add an extra penalty to this triplet loss. This function uses the coefficient of variation (stddev/mean) and my idea is based on this paper:
Learning 3D Keypoint Descriptors for Non-Rigid Shape Matching
Their GitHub repo is here, but it is not very well documented in my opinion.
I've been working on Pytorch for the last year and it's still difficult to me to fully understand how to create loss functions properly. This is the loss function I wrote:
class CoefficientVariationLoss(nn.Module):
    def __init__(self, class_to_idx, weighted=False):
        super(CoefficientVariationLoss, self).__init__()
        self.class_to_idx = class_to_idx
        self.weighted = weighted

    def forward(self, x, labels):
        total_coeff_variation = 0
        l2_norm_positive = F.pairwise_distance(x[0], x[1],2)

        for c in self.class_to_idx.values():
            count_label_pos = torch.sum(labels[0]==c).float()
            if count_label_pos <= 1:
                class_cv = 0
            else:
                class_pos = l2_norm_positive[labels[0]==c]

                if self.weighted:
                    weight = count_label_pos/labels[0].shape[0]
                    class_cv = (torch.std(class_pos)/torch.mean(class_pos))*weight
                else:
                    class_cv = (torch.std(class_pos)/torch.mean(class_pos))

                # if torch.isnan(class_cv):
                #     class_cv = 0

            total_coeff_variation += class_cv

        return total_coeff_variation

The two inputs of this function are:

x: x[0], x[1] and x[2] are the embeddings of a triplet (anchor, positive, negative)
labels: labels[0], labels[1] and labels[2] are the classes of each element in the triplet (anchor, positive, negative)

My custom dataloader is serving the data as a dictionary:
sample = {'images': [...], 'labels': [...]}

and the output of my model (triplet network) is a list of tensors in the same order: anchor, positive and negative.
The final loss is finally computed as
loss = loss_triplet + FACTOR * loss_coeffVariation

Have I created this loss correctly? I don't know if can use indexing in loss functions because of differentiability issues...
Thanks!


title: Is there any way to get torch.mode over multidimensional tensor

body:
is there any way torch.mode can be applied over multiple dimensions
for example
import numpy as np
import torch
x = np.random.randint(10, size=(3, 5))
y = torch.tensor(x)

lets say y has
[[6 3 7 3 0]
 [2 5 7 9 7]
 [6 1 4 6 3]]

torch.mode should return a size 3 tensor [3,7,6]
without using a loop


title: (image, mask) pair do not match one another in a semantic segmentation task

body:
I am writing a simple custom DataLoader (which I will add more features to later) for a segmentation dataset but the (image, mask) pair I return using __getitem()__ method are different; the returned mask belongs to a different image than the one which is returned. My directory structure is /home/bohare/data/images and /home/bohare/data/masks .
Following is the code I have:
import torch
from torch.utils.data.dataset import Dataset
from PIL import Image
import glob
import os
import matplotlib.pyplot as plt

class CustomDataset(Dataset):
    def __init__(self, folder_path):

        self.img_files = glob.glob(os.path.join(folder_path,'images','*.png'))
        self.mask_files = glob.glob(os.path.join(folder_path,'masks','*.png'))

    def __getitem__(self, index):

        image = Image.open(self.img_files[index])
        mask = Image.open(self.mask_files[index])

        return image, mask

    def __len__(self):
        return len(self.img_files)

data = CustomDataset(folder_path = '/home/bohare/data')
len(data)

This code correctly gives out the total size of the dataset.
But when I use:
img, msk = data.__getitem__(n) where n is the index of any (image, mask) pair and I plot the image and mask, they do not correspond to one another.
How can I modify/what can I add to the code to make sure the (image, mask) pair are returned correctly? Thanks for the help.


title: Pytorch apply operation to some rows but not all

body:
I'm trying to apply a softmax function on some rows of a tensor, but the problem is that some of my rows have all -inf values.  As such, softmax on these rows outputs NaN, which causes problems later in the model.
As such, I want to create a function that applies softmax to a row unless in is all -inf.  In that case, it outputs a zero vector.  Is there any easy way to do this?


title: How to writte a raster plot?

body:
I'm using pytorch on google colab.
I've got a tensor matrix below, this is the example, and actually the matrix size is about 50 neurons and 30,000~50,000 time.
a= torch.tensor([[0., 0., 0., 0., 0.],
                 [0., 0., 0., 0., 1.],
                 [0., 1., 0., 1., 0.]])

each of values are,
a= torch.tensor([[Neuron1(t=1), N2(t=1), N3(t=1), N4(t=1), N5(t=1)],
                 [N1(t=2), N2(t=2), N3(t=2), N4(t=2), N5(t=2)],
                 [N1(t=3), N2(t=3), N3(t=3), N4(t=3), N5(t=3)]])

and 1 means that neuron fire, 0 means not fire.
So Neuron5(t=2), Neuron2(t=3) and Neuron4(t=3) are firing.
Then, I want to make a raster plot or scatter plot like below using this matrix,
The dots show the firing neuron.

neuron number
1|
2|          *
3|
4|          *
5|__ *_____time
    1  2  3
What would be the best python code to do this?
I have no idea now.
Thank you for reading.


title: KeyError: 'true ' error when i try to convert labels to 0 and 1

body:
I am using Arabert (pre trained Bert for Arabic language) for binary classification labeled as true and false, i am trying to change the labels from "true" and "false" to 0 and one i used the code:
import pandas as pd

Data=pd.read_csv("/content/500-instances.csv")
DATA_COLUMN = 'sent'
LABEL_COLUMN = 'label'
Data.columns = [DATA_COLUMN, LABEL_COLUMN]
label_map = {
    'fake' : 0,
    'true' : 1
}

Data[DATA_COLUMN] = Data[DATA_COLUMN].apply(lambda x: preprocess(x, do_farasa_tokenization=False, use_farasapy = False))
Data[LABEL_COLUMN] = Data[LABEL_COLUMN].apply(lambda x: label_map[x])

i get an error: KeyError: 'true ' in the last line.
Do you have any solution?!
Thanks in advance


title: How to get cuda to load when running pytorch?

body:
I had an old conda virtual environment with pytorch and cudatoolkit 10.1. I was able to run the code and I would always get the following message at the beginning:
I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1

However, I created a new virtual environment with cudatoolkit 11.0, but now this message does not run at the beginning of the code (nor do I get an error message saying it failed to find the file), and the code crashes eventually when it tries to use the gpu. How do I make sure that it loads GPU support correctly?


title: Which loss function is suitable?

body:
I am using PyTorch and am still quite new to the library.
I have a relation between my input and output given by y = ax + b, where a and b are sampled from some distribution (say Uniform), that is, they are random. I would like to train a network to predict x upon seeing y and a. I am employing a network, named probability_network, with nn.Linear layers. There are N (say 10) classes to choose from for x.
class ProabilityNetwork(nn.Module):
    def __init__(self):
        super(ProabilityNetwork, self).__init__()
        self.fc1 = nn.Linear(8, 76)
        self.fc2 = nn.Linear(76, 150)
        self.fc3 = nn.Linear(150, 75)
        self.fc4 = nn.Linear(75, 14)
        self.fc5 = nn.Linear(14, 10)
        self.tanh = nn.Tanh()
        self.sigmoid = nn.Sigmoid()
        self.relu = nn.ReLU()
        self.softmax = nn.Softmax(dim=1)

    def forward(self, inputs):
        return self.softmax(self.fc5(self.fc4(self.relu(self.fc3(self.relu(self.fc2(self.relu(self.fc1(inputs)))))))))

probabilty_network = ProbabilityNetwork()

Upon seeing y, the loss function should help the network predict an x that minimizes ||y-ax||^2. All the quantities in y = ax + b are vectors (each of length 4, in this example). I have already tried the following loss function.
prob_values = probabilty_network(torch.cat([y, a], dim=0))  # shape: (batch_size, 10)
x_hat = mapping_tensor[torch.argmax(prob_values, dim=1)]  # Mapping from probability to one of 10 classes, mapping_tensor is an array of shape (10, 4)

mse_loss = nn.MSELoss()
loss = mse_loss(y, a*x_hat)

As an example, the mapping_tensor could contain binary representation of values from 0 (0000) to 9 (1001). The reason I need the binary representation of the class is that I need a vector x for the loss ||y-ax||^2. In this case, x is a 4 length vector whereas the output of the neural network is a 10  length vector.
The above setup doesn't work. Half the values in the predicted class (written out in binary) are always wrong, implying the network is confused while training.
Further, this is not an unachievable problem. Solution to the above mentioned loss function exists (with errors, of course, but the errors are far lesser than 50%) but is computationally intensive. I am trying to check if the network can somehow learn to predict at lower complexity. Any help is appreciated. Thanks.
Furthermore, from an optimization standpoint, the loss function is the best possible solution (that I know of). So, changing the loss function would only lead to poorer results.
Another way to look at the problem is as follows. Suppose the network sees y and a. The network then computes |y-ax| for each class x (out of 10 possible classes), and then chooses the class with least value of the computed value. And my question is, what loss function can I use to make the network train in this fashion?


title: My Pytorch model is giving very bad results

body:
I am new with Deep Learning with Pytorch. I am more experienced with Tensorflow, and thus I should say I am not new to Deep Learning itself.
Currently, I am working on a simple ANN classification. There are only 2 classes so quite naturally I am using a Softmax BCELoss combination.
The dataset is like this:
shape of X_train (891, 7)
Shape of Y_train (891,)
Shape of x_test (418, 7)

I transformed the X_train and others to torch tensors as train_data and so on. The next step is:
train_ds = TensorDataset(train_data, train_label)
# Define data loader
batch_size = 32
train_dl = DataLoader(train_ds, batch_size, shuffle=True)

I made the model class like:
class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()

        # an affine operation: y = Wx + b
        self.fc1 = nn.Linear(7, 32)
        self.bc1 = nn.BatchNorm1d(32)
        self.fc2 = nn.Linear(32, 64)
        self.bc2 = nn.BatchNorm1d(64)
        self.fc3 = nn.Linear(64, 128)
        self.bc3 = nn.BatchNorm1d(128)
        self.fc4 = nn.Linear(128, 32)
        self.bc4 = nn.BatchNorm1d(32)
        self.fc5 = nn.Linear(32, 10)
        self.bc5 = nn.BatchNorm1d(10)
        self.fc6 = nn.Linear(10, 1)
        self.bc6 = nn.BatchNorm1d(1)

        self.drop = nn.Dropout2d(p=0.5)

    def forward(self, x):
        torch.nn.init.xavier_uniform(self.fc1.weight)
        x = self.fc1(x)
        x = self.bc1(x)
        x = F.relu(x)

        x = self.drop(x)
        x = self.fc2(x)
        x = self.bc2(x)
        x = F.relu(x)

        #x = self.drop(x)
        x = self.fc3(x)
        x = self.bc3(x)
        x = F.relu(x)

        x = self.drop(x)
        x = self.fc4(x)
        x = self.bc4(x)
        x = F.relu(x)

        #x = self.drop(x)
        x = self.fc5(x)
        x = self.bc5(x)
        x = F.relu(x)

        x = self.drop(x)
        x = self.fc6(x)
        x = self.bc6(x)
        x = torch.sigmoid(x)
        return x

model = Net()

The loss function and the optimizer are defined:
loss = nn.BCELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.00001, betas=(0.9, 0.999), eps=1e-08, weight_decay=0, amsgrad=False)

At last, the task is to run the forward in epochs:
num_epochs = 1000
# Repeat for given number of epochs
for epoch in range(num_epochs):

    # Train with batches of data
    for xb,yb in train_dl:
        pred = model(xb)

        yb = torch.unsqueeze(yb, 1)

        #print(pred, yb)
        print('grad', model.fc1.weight.grad)

        l = loss(pred, yb)
        #print('loss',l)

        # 3. Compute gradients
        l.backward()

        # 4. Update parameters using gradients
        optimizer.step()

        # 5. Reset the gradients to zero
    optimizer.zero_grad()

    # Print the progress
    if (epoch+1) % 10 == 0:
        print('Epoch [{}/{}], Loss: {:.4f}'.format(epoch+1, num_epochs, l.item()))

I can see in the output that after each iteration with all the batches, the hard weights are non-zero, after this zero_grad is applied.
However, the model is pretty bad. I get an F1 score of around 50% only! And the model is bad when I call it to predict the train_dl  itself!!!
I am wondering what the reason is. The grad of weights not zero but not updating properly? The optimizer not optimizing the weights? Or what else?
Can someone please have a look?
I already tried different loss functions and optimizers. I tried with smaller datasets, bigger batches, different hyperparameters.
Thanks! :)


title: Dice loss working only when probs are squared at denominator

body:
I'm experiencing an interesting and frustrating issue with the Dice loss used in image segmentation with Unet.
I have to segment images in two classes: background and region of interest. The region of interest is typically 4% of the pixels of the whole image. Images are about 1600x1600 pixels.
I found the Dice loss working much better than Cross Entropy.
However, if I use the standard Dice loss formula my Unet does not provide a correct output, i.e. all the pixels are predicted as background.
With standard Dice loss I mean:

where x_{c,i} is the probability predicted by Unet for pixel i and for channel c, and y_{c,i} is the corresponding ground-truth label.
The modified version I use is:

Note the squared x at the denominator.
For some reason the latter one makes the net to produce a correct output, although the loss converges to ~0.5.
I do not understand why the latter works and the former doesn't. The latter works even if I use the power of three at the denominator.
Here below my implementation:
def make_one_hot(labels, classes):
    one_hot = torch.FloatTensor(labels.size()[0], classes, labels.size()[2], labels.size()[3]).zero_().to(labels.device)
    target = one_hot.scatter_(1, labels.data, 1)
    return target

class DiceLoss(nn.Module):

    def __init__(self,):
        super(DiceLoss, self).__init__()

    def forward(self, output, target):

        target = make_one_hot(target.unsqueeze(dim=1), classes=output.size()[1])
        output = F.softmax(output, dim=1)

        numerator = (output * target).sum(dim=(2, 3))
        denominator = output.pow(2).sum(dim=(2, 3)) + target.sum(dim=(2, 3))

        iou = numerator / denominator

        return 1 - iou.mean()


title: Failure ONNX InferenceSession ONNX model exported from PyTorch

body:
I am trying to export a custom PyTorch model to ONNX to perform inference but without success... The tricky thing here is that I'm trying to use the script-based exporter as shown in the example here in order to call a function from my model.
I can export the model without any complain but then when trying to start an InferenceSession I get the following error:
Fail: [ONNXRuntimeError] : 1 : FAIL : Load model from ner.onnx failed:Type Error: Type parameter (T) bound to different types (tensor(int64) and tensor(float) in node (Concat_1260).

I tried to identify the root cause of that problem and it seems to be generated by the use of torch.matmul() in the following function (quite nasty cause I'm trying to only use pytorch operators):
@torch.jit.script
def valid_sequence_output(sequence_output, valid_mask):
    X = torch.where(valid_mask.unsqueeze(-1) == 1, sequence_output, torch.zeros_like(sequence_output))
    bs, max_len, _ = X.shape

    tu = torch.unique(torch.nonzero(X)[:, :2], dim=0)
    batch_axis = tu[:, 0]
    rows_axis = tu[:, 1]

    a = torch.arange(bs).repeat(batch_axis.shape).reshape(batch_axis.shape[0], -1)
    a = torch.transpose(a, 0, 1)

    T = torch.cumsum(torch.where(batch_axis == a, torch.ones_like(a), torch.zeros_like(a)), dim=1) - 1
    cols_axis = T[batch_axis, torch.arange(batch_axis.shape[0])]

    A = torch.zeros((bs, max_len, max_len))
    A[(batch_axis, cols_axis, rows_axis)] = 1.0

    valid_output = torch.matmul(A, X)
    valid_attention_mask = torch.where(valid_output[:, :, 0] != 0, torch.ones_like(valid_mask),
                                       torch.zeros_like(valid_mask))
    return valid_output, valid_attention_mask

It seems like torch.matmul isn't supported (according to the docs) so I tried a bunch of workaround (e.g. A.matmul(X), torch.baddbmm) but I still get the same issue...
Any suggestions on how to fix this behavior would be awesome :D
Thanks for your help!


title: Gradient-based saliency of input words in a pytorch model from transformers library

body:
The following code is used to determine the impact of input words on the most probable output unit.
def _register_embedding_list_hook(model, embeddings_list):
    def forward_hook(module, inputs, output):
        embeddings_list.append(output.squeeze(0).clone().cpu().detach().numpy())
    embedding_layer = model.bert.embeddings.word_embeddings
    handle = embedding_layer.register_forward_hook(forward_hook)
    return handle

def _register_embedding_gradient_hooks(model, embeddings_gradients):
    def hook_layers(module, grad_in, grad_out):
        embeddings_gradients.append(grad_out[0])
    embedding_layer = model.bert.embeddings.word_embeddings
    hook = embedding_layer.register_backward_hook(hook_layers)
    return hook

def saliency_map(model, input_ids, segment_ids, input_mask):
    torch.enable_grad()
    model.eval()
    embeddings_list = []
    handle = _register_embedding_list_hook(model, embeddings_list)
    embeddings_gradients = []
    hook = _register_embedding_gradient_hooks(model, embeddings_gradients)

    model.zero_grad()
    A = model(input_ids, token_type_ids=segment_ids, attention_mask=input_mask)
    pred_label_ids = np.argmax(A.logits[0].detach().numpy())
    A.logits[0][pred_label_ids].backward()
    handle.remove()
    hook.remove()

    saliency_grad = embeddings_gradients[0].detach().cpu().numpy()
    saliency_grad = np.sum(saliency_grad[0] * embeddings_list[0], axis=1)
    norm = np.linalg.norm(saliency_grad, ord=1)
    saliency_grad = [e / norm for e in saliency_grad]

    return saliency_grad

which is used in the following way (for a sentiment analysis model):
from transformers import AutoTokenizer, AutoModelForSequenceClassification
tokenizer = AutoTokenizer.from_pretrained("textattack/bert-base-uncased-imdb")
model = AutoModelForSequenceClassification.from_pretrained("textattack/bert-base-uncased-imdb")

tokens = tokenizer('A really bad movie')

input_ids = torch.tensor([tokens['input_ids']], dtype=torch.long)
token_type_ids = torch.tensor([tokens['token_type_ids']], dtype=torch.long)
attention_ids = torch.tensor([tokens['attention_mask']], dtype=torch.long)

saliency_scores = saliency_map(model, input_ids,
                                token_type_ids,
                                attention_ids)

But it produces the following scores for the tokens which are nonsense since for example "bad" has a negative effect on the predicted class (which is negative). What's wrong with this code?

Here are some more examples:


title: Derive the gradient through torch.topk

body:
I want to derive the gradient through torch.topk function.
Suppose I the input is a vector
,
then it is transformed by a parameter matrix
 ,
and the top k values of the vector are selected
.
The result vector is further transformed by element-wise multiplication.

Finally the loss is computed by
.
I wonder, is the loss differentiable with respect to W? Formally, can we calculate the following gradient?


title: Spiral Loss Function for Music Encoding

body:
I am trying to develop an autoencoder for music generation; in pursuit of that end, I am attempting to develop a loss function which captures musical relationships.

My current idea is a 'Spiral' loss function, which is to say that if the system predicts the same note in a different octave, the loss should be smaller than if the note is just wrong. Additionally, notes that are close to the correct note, such as B and D to C should also have small losses. One can conceptually think of this as finding the distance between two points on a coil or spiral, such that the same notes in different octaves lie along a line tangent to the coil, but separated by some loop distance.

I am working in PyTorch, and my input representation is a 36 by 36 Tensor, where the rows represent the notes (MIDI range 48:84, the middle three octaves of a piano) and the columns represent time steps (1 column = 1/100th of a second). The values in the matrix are either 0 or 1, signifying that a note is on at a particular time.

Here is my current implementation of the loss:

def SpiralLoss():
    def spiral_loss(input, output):
        loss = Variable(torch.FloatTensor([0]))
        d = 5
        r = 10
        for i in xrange(input.size()[0]):
            for j in xrange(input.size()[3]):
                # take along the 1 axis because it's a column vector
                inval, inind = torch.max(input[i, :, :, j], 1)
                outval, outind = torch.max(output[i, :, :, j], 1)
                note_loss = (r*30*(inind%12 - outind%12)).float()
                octave_loss = (d*(inind/12 - outind/12)).float()
                loss += torch.sqrt(torch.pow(note_loss, 2) + torch.pow(octave_loss, 2))
        return loss
    return spiral_loss

The problem with this loss is that the max function is not differentiable. I cannot think of a way to make this loss differentiable, and was wondering if anyone might have any ideas or suggestions?

I'm not sure if this is the right place for a post like this, and so if it isn't, I would really appreciate any pointers towards a better location.


title: Indexing a batched set of images

body:
Suppose I have two index tensors and an image tensor, how can I sample the (x, y) points from the image?

img.shape # -> (batch x H x W x 3)
x.shape # -> (batch x H x W)
y.shape # -> batch x H x W)

(H x W being height x width)
Basically I want to perform something like a batch "shuffle" of the image pixel intensities.


title: PyTorch will not fit straight line to two data points

body:
I'm facing issues in fitting a simple y= 4x1 line with 2 data points using pytorch. While running the inference code, the model seems to output same value to any input which is strange. Pls find the code attached along with the data files used by me. Appreciate any help here.

import torch
import numpy as np
import pandas as pd

df = pd.read_csv('data.csv')
test_data = pd.read_csv('test_data.csv')

inputs = df[['x1']]
target = df['y']
inputs = torch.tensor(inputs.values).float()
target = torch.tensor(target.values).float()

test_data = torch.tensor(test_data.values).float()
#Defining Network Architecture
import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):

  def __init__(self):
    super(Net,self).__init__()

    hidden1 = 3
#     hidden2 = 5

    self.fc1 = nn.Linear(1,hidden1)
    self.fc3 = nn.Linear(hidden1,1)

  def forward(self,x):
    x = F.relu(self.fc1(x))
    x = self.fc3(x)
    return x

#instantiate the model

model = Net()
print(model)

criterion = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(),lr=0.01)

model.train()

#epochs
epochs = 100

for x in range(epochs):
  #initialize the training loss to 0
  train_loss = 0
  #clear out gradients
  optimizer.zero_grad()

  #calculate the output
  output = model(inputs)

  #calculate loss
  loss = criterion(output,target)

  #backpropagate
  loss.backward()

  #update parameters
  optimizer.step()

  if ((x%5)==0):
    print('Training Loss after epoch {:2d} is {:2.6f}'.format(x,loss))

#set the model in evaluation mode
model.eval()

#Test the model on unseen data

test_output = model(test_data)

print(test_output)

Below is the model output

#model output
tensor([[56.7579],
        [56.7579],
        [56.7579],
        [56.7579],
        [56.7579],
        [56.7579],
        [56.7579],
        [56.7579],
        [56.7579],
        [56.7579],
        [56.7579],
        [56.7579],
        [56.7579],
        [56.7579],
        [56.7579],
        [56.7579],
        [56.7579],
        [56.7579],
        [56.7579],
        [56.7579]], grad_fn=<AddmmBackward>)


title: Got Very Different Scores After Translating Simple Test Model from Keras to PyTorch

body:
I'm trying to transition from Keras to PYTorch.

After reading tutorials and similar questions, I came up with the following simple models to test. However, the two models below gives me very different scores: Keras (0.9), PyTorch (0.03).

Could someone give me guidance?

Basically my dataset has 120 features and multilabels with 3 classes that look like below.

[
    [1,1,1],
    [0,1,1],
    [1,0,0],
    ...
]

def score(true, pred):
    lrl = label_ranking_loss(true, pred)
    lrap = label_ranking_average_precision_score(true, pred)
    print('LRL:', round(lrl), 'LRAP:', round(lrap))

#Keras:
model= Sequential()
model.add(Dense(60, activation="relu", input_shape=(120,)))
model.add(Dense(30, activation="relu"))
model.add(Dense(3, activation="sigmoid"))
model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])
model.fit(x_train, y_train, batch_size=32, epochs=100)
pred = model.predict(x_test)
score(y_test, pred)

#PyTorch
model = torch.nn.Sequential(
    torch.nn.Linear(120, 60),
    torch.nn.ReLU(),
    torch.nn.Linear(60, 30),
    torch.nn.ReLU(),
    torch.nn.Linear(30, 3),
    torch.nn. Sigmoid())
loss_fn = torch.nn. BCELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)

epochs = 100
batch_size = 32
n_batch = int(x_train.shape[0]/batch_size)
for epoch in range(epochs):
    avg_cost = 0
    for i in range(n_batch):
        x_batch = x_train[i*batch_size:(i+1)*batch_size]
        y_batch = y_train[i*batch_size:(i+1)*batch_size]
        x, y = Variable(torch.from_numpy(x_batch).float()), Variable(torch.from_numpy(y_batch).float(), requires_grad=False)
        pred = model(x)
        loss = loss_fn(pred, y)
        loss.backward()
        optimizer.step()
        avg_cost += loss.item()/n_batch
    print(epoch, avg_cost)

x, y = Variable(torch.from_numpy(x_test).float()), Variable(torch.from_numpy(y_test).float(), requires_grad=False)
pred = model(x)
score(y_test, pred.data.numpy())


title: Pytorch - Batch Normalizaiton simple question

body:
I implemented a model with batch normalization:

class FFNet(torch.nn.Module):
    def __init__(self, D_in, H_1, H_2, D_out):

        super(FFNet, self).__init__()
        self.linear1 = torch.nn.Linear(D_in, H_1)
        self.linear2 = torch.nn.Linear(H_1, H_2)
        self.bn2 = torch.nn.BatchNorm1d(H_2)
        self.linear4 = torch.nn.Linear(H_2, D_out)

    def forward(self, x):

        h_relu_1=F.relu(self.linear1(x))
        h_relu_2=F.relu(self.bn2(self.linear2(h_relu_1)))
        y_pred=self.linear4(h_relu_2)
        return y_pred

Also, I wrote the training loop:

for epoch in range(epoches):

    running_loss = 0.0
    cnt = 0
    for i, data in enumerate(train_data, 0):
        local_X, local_y = data
        y_pred = model.forward(local_X)
        loss = criterion(y_pred, local_y)
        optimizer.zero_grad()
        #loss = criterion(y_pred, Y_local_output)
        loss.backward() # back props
        optimizer.step()
        running_loss = running_loss + loss.item()
        cnt+=1

    Validation_loss = 0.0
    cnt2 = 0
    # Validation
    for i, data in enumerate(validation_data, 0):
        Val_X, Val_Y = data

        y_pred = model.forward(Val_X)

        loss=criterion(y_pred, Val_Y)
        Validation_loss = Validation_loss + loss.item()
        cnt2+=1

I have two questions:
1. Is there no need to use model.train() in this code?
2. How to evaluate this model using eval? I have one data sample whose size is (1xD_in), and batch size is more than 1. When I use the below code, there is an error:

    test_single = torch.tensor([aa, ab, ac, ad, ae, af, ag])
    test_single = test_single.unsqueeze(0)

    model.eval()
    [bb,cc] = model.forward(test_single)

The error is 'not enough values to unpack (expected 2, got 1)'


title: What does the "@" operator do in Python?

body:
I am trying to understand some code in which I found this statement. I don't know what "@" sign is doing here.
This is a running code for landmark detection, you can check full code here: https://github.com/cleardusk/3DDFA/blob/master/utils/ddfa.py
vertex = p @ (u + w_shp @ alpha_shp + w_exp @ alpha_exp).reshape(3, -1, order='F') + offset


title: Understanding Pytorch memory allocation on GPU

body:
I have the following code snippet to do complex multiplication in Pytorch

Variation 3

def complex_mult(x, y):
  a, b = x[:, :, :, :, :, 0], x[:, :, :, :, :, 1]
  c, d = y[:, :, :, :, :, 0], y[:, :, :, :, :, 1]
  out = torch.stack([a*c - b*d, a*d + b*c], dim=-1)
  return out

As I was investigating why I am getting RuntimeError: CUDA out of memory,
I realized that the memory allocation depending on the number of instruction I explicit give.

I wrote that code in three different ways, the other two as as bellow.

Variation 2

def complex_mult(x, y):
  a, b = x[:, :, :, :, :, 0], x[:, :, :, :, :, 1]
  c, d = y[:, :, :, :, :, 0], y[:, :, :, :, :, 1]
  real = a*c - b*d
  imag = a*d + b*c
  pair = [real, imag]
  out = torch.stack(pair, dim=-1)
  return out

Variation 1

def complex_mult(x, y):
  a, b = x[:, :, :, :, :, 0], x[:, :, :, :, :, 1]
  c, d = y[:, :, :, :, :, 0], y[:, :, :, :, :, 1]

  r1 = a*c
  r2 = b*d
  real = r1 - r2

  i1 = a*d
  i2 = b*c
  imag = i1 + i2

  pair = [real, imag]
  out = torch.stack(pair, dim=-1)
  return out

From Variation 1 Gist , we have:

All the tensors use float32, which is 4 bytes
The size of _x is (128, 64, 32, 32, 2), thus 64 MB, 32 MB each component (real, imag)
Each component multiplication (a*c, b*d, a*d and b*c) will have 2048 MB

Which is 64 times the 32 MB of each component of x
Subtotal of 8 GB

Plus 2 GB, if consider each sum result as storage on its own variable

Subtotal of 12GB

Stacking real and imag requires more 4 GB

Total of 16 GB

RuntimeError: CUDA out of memory. Tried to allocate 4.00 GiB (GPU 0; 14.73 GiB total capacity; 12.12 GiB already allocated; 1.82 GiB free; 12.12 GiB reserved in total by PyTorch)

Intuitively, I then decided to stick with Variation 3 by thinking

  "Without those temporary variable (place) holders, Torch will know that they won't be used again, thus releasing their memory. Thus using only 4 GB by the end of my execution."

But, what I don't understand, and here is my question, is

  Why, on Variation 3, I still get CUDA out of memory after a few calls to that method? The gist show that in the third call it crashes for not having enough memory.

What is even more interesting is that, though one can think

  "That is because there is already 4 GB that is allocated with the previous result."

However, given that it was able to execute twice, I don't think that is the issue.


title: Creating and shaping data for 1D CNN

body:
I have a (244, 108) numpy array. It contains percentage change of close value of a trade for each minute in one day ie 108 values and like that for 244 days. Basically its a 1D vector. So in order to do 1D CNN how should I shape my data?

What i have done:

x.shape = (244, 108)
x = np.expand_dims(x, axis=2)
x.shape = (243, 108, 1)
y.shape = (243,)

Model:

class Net(torch.nn.Module):
    def __init__(self):
        super(Net, self).__init__()

        self.layer1 = torch.nn.Conv1d(in_channels=108, out_channels=1, kernel_size=1, stride=1)
        self.act1 = torch.nn.ReLU()
        self.act2 = torch.nn.MaxPool1d(kernel_size=1, stride=1)
        self.layer2 = torch.nn.Conv1d(in_channels=1, out_channels=1, kernel_size=1, stride=1)
        self.act3 = torch.nn.ReLU()
        self.act4 = torch.nn.MaxPool1d(kernel_size=1, stride=1)

        self.linear_layers = nn.Linear(1, 1)

    # Defining the forward pass
    def forward(self, x):
        x = self.layer1(x)
        x = self.act1(x)
        x = self.act2(x)
        x = self.layer2(x)
        x = self.act3(x)
        x = self.act4(x)
        x = self.linear_layers(x)
        return x


title: MSE with predictions not expected

body:
Here is a regression model where I attempt to predict the y values (outputs) from x values (inputs) . Each class is given a different mean and normalized with l2 normalization:
x_values = sklearn.preprocessing.normalize(x_values, norm="l2")

This may appear as a classification problem attempting to be solved using regression. I'm attempting to understand multiclass regression in PyTorch, as the PyTorch doc gives the following example which suggests multiclass regression is possible:
>>> loss = nn.MSELoss()
>>> input = torch.randn(3, 5, requires_grad=True)
>>> target = torch.randn(3, 5)
>>> output = loss(input, target)
>>> output.backward()

src: https://pytorch.org/docs/master/generated/torch.nn.MSELoss.html
Entire code:
% reset - f

from datetime import datetime
from sklearn import metrics
import torch
import torch.nn as nn
import torchvision
import torchvision.transforms as transforms
import numpy as np
import matplotlib.pyplot as plt
import torch.utils.data as data_utils
import torch.nn as nn
import torch.nn.functional as F
import random
from torch.autograd import Variable
import pandas as pd
import unittest
import time
from collections import Counter
import sklearn

x_values = []
y_values = []
input_size = 17
lr = .1

# Class1
mu, sigma = 0, 0.1  # mean and standard deviation
x_values.append(np.random.normal(mu, sigma, input_size))
x_values.append(np.random.normal(mu, sigma, input_size))
x_values.append(np.random.normal(mu, sigma, input_size))

# Class2
mu, sigma = 5, 0.5  # mean and standard deviation
x_values.append(np.random.normal(mu, sigma, input_size))
x_values.append(np.random.normal(mu, sigma, input_size))
x_values.append(np.random.normal(mu, sigma, input_size))

# Class3
mu, sigma = 10, 1.0  # mean and standard deviation
x_values.append(np.random.normal(mu, sigma, input_size))
x_values.append(np.random.normal(mu, sigma, input_size))
x_values.append(np.random.normal(mu, sigma, input_size))

# Class4
mu, sigma = 15, 1.5  # mean and standard deviation
x_values.append(np.random.normal(mu, sigma, input_size))
x_values.append(np.random.normal(mu, sigma, input_size))
x_values.append(np.random.normal(mu, sigma, input_size))

# Class5
mu, sigma = 20, 2.0  # mean and standard deviation
x_values.append(np.random.normal(mu, sigma, input_size))
x_values.append(np.random.normal(mu, sigma, input_size))
x_values.append(np.random.normal(mu, sigma, input_size))

x_values = sklearn.preprocessing.normalize(x_values, norm="l2")

y_values.append(0)
y_values.append(0)
y_values.append(0)

y_values.append(1)
y_values.append(1)
y_values.append(1)

y_values.append(2)
y_values.append(2)
y_values.append(2)

y_values.append(3)
y_values.append(3)
y_values.append(3)

y_values.append(4)
y_values.append(4)
y_values.append(4)

num_classes = len(y_values)

class NeuralNet(nn.Module):
    def __init__(self):
        super(NeuralNet, self).__init__()
        self.criterion = torch.nn.MSELoss()
        self.model = torch.nn.Sequential(
            torch.nn.Linear(input_size, 100),
            torch.nn.ReLU(),
            torch.nn.Linear(100, 50),
            torch.nn.ReLU(),
            torch.nn.Linear(50, num_classes)
            #                         torch.nn.ReLU()
        )
        self.optimizer = torch.optim.Adam(self.model.parameters(), lr)

    def update(self, state, action):
        y_pred = self.model(torch.Tensor(state))
        loss = self.criterion(y_pred, Variable(torch.Tensor(action)))
        self.optimizer.zero_grad()
        loss.backward()
        self.optimizer.step()
        return loss

    def predict(self, s):
        with torch.no_grad():
            return self.model(torch.Tensor(s))

def weights_init(m):
    if type(m) == nn.Linear:
        m.weight.data.normal_(0.0, 1)

model = NeuralNet()
model.apply(weights_init)
print('len(states)', len(x_values))

i = 0

for s in range(7000):

    if i == 15:
        i = 0
    x = x_values[i]
    loss_value = model.update(x, y_values)

    if s % 1000 == 0:
        print('loss_value', loss_value)

    i = i + 1

Predicting on the x_values:
[torch.argmax(model.predict(s)) for s in x_values]

returns:
[tensor(14),
 tensor(14),
 tensor(14),
 tensor(14),
 tensor(14),
 tensor(14),
 tensor(14),
 tensor(14),
 tensor(14),
 tensor(14),
 tensor(14),
 tensor(14),
 tensor(14),
 tensor(14),
 tensor(14)]

As I have defined classes with difference means and the final loss value is low (4.7370e-15) I expect the predicted values to be closer to:
[tensor(0)
 tensor(0),
 tensor(0),
 tensor(1),
 tensor(1),
 tensor(1),
 tensor(2),
 tensor(2),
 tensor(2),
 tensor(3),
 tensor(3),
 tensor(3),
 tensor(4),
 tensor(4),
 tensor(4)]

What are the predicted outputs not closed to my expectation?
Have I set up the model incorrectly?


title: Numpy from an array, for each element create a matrix N*M with all values set as the element without for loops

body:
I have a numpy array like np.array([1, 2, 3])
Without using for loops but just use numpy or pytorch methods, i want a matrix with dimension len(array) * N * M composed by matrices N*M that the first matrix is composed by all ones, the second only by two values and the third only by 3 values.
For istance
N = 4 M = 3
[[[1,1,1,1],[1,1,1,1],[1,1,1,1]],
  [[2,2,2,2],[2,2,2,2],[2,2,2,2]],
  [[3,3,3,3],[3,3,3,3],[3,3,3,3]]]

I tried different methods to achive this matrix like unsqueeze and repeat but i wasn't able to find a solution, any suggestion?


title: Seemingly inconsistent tensor sizes in pytorch

body:
I'm building a convolutional autoencoder, but want the encoding to be in a linear form so I can more easily feed it as input into an MLP. I have two convolutional layers on the encoder along with a linear inner layer to reduce dimension. This encoding is then fed into the corresponding decoder.
When I flatten the output of the second convolutional layer, based on my calculation (using the standard formula: Calculate the Output size in Convolution layer) should come out to a 1x100352 rank 1 tensor. However, when I set the input dimension of the linear layer to be 100352, the flattened rank 1 tensor has dimension 1x50176. Then comes the weird part.
I tried changing the input dimension of the linear layer to be 50176, assuming I had miscalculated. When I do this, the reshaped rank 1 tensor confusingly becomes 1x100352, and then the aforementioned weight matrix becomes 50176x256 as expected.
This response to modifying the linear layer's input dimension doesn't make sense to me. That hyperparameter controls the weight matrix correctly, but I guess I'm uncertain why it has any bearing on the linear layer's input since that's just a reshaped tensor output from a convolutional layer whose hyperparameters are unrelated to the hyperparameter in question.
I apologize if I'm just missing something obvious. I'm very new to pytorch, and I couldn't find any other posts which discussed this sort of issue.
Here's what I believe to be the minimal reproducible example:
import os
import torch
import torchvision
import torchvision.transforms as transforms
import matplotlib.pyplot as plt
import torch.autograd as autograd
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

from torchvision import datasets
from torch.utils.data import DataLoader
from torchvision.utils import save_image

class convAutoEncoder(nn.Module):
        def __init__(self,**kwargs):
                super().__init__()
                #Creating network structure
                #Encoder portion of autoencoder
                self.enc1 = nn.Conv2d(in_channels = kwargs["inputChannels"], out_channels = kwargs["channelsEncoderMid"], kernel_size = kwargs["kernelSize"])
                self.enc2 = nn.Conv2d(in_channels = kwargs["channelsEncoderMid"], out_channels = kwargs["channelsEncoderInner"], kernel_size = kwargs["kernelSize"])
                self.enc3 = nn.Linear(in_features = kwargs["intoLinear"], out_features = kwargs["linearEncoded"])
                #Decoder portion of autoencoder
                self.dec1 = nn.Linear(in_features = kwargs["linearEncoded"], out_features = kwargs["intoLinear"])
                self.dec2 = nn.ConvTranspose2d(in_channels = kwargs["channelsEncoderInner"], out_channels = kwargs["channelsDecoderMid"], kernel_size = kwargs["kernelSize"])
                self.dec3 = nn.ConvTranspose2d(in_channels = kwargs["channelsDecoderMid"], out_channels = kwargs["inputChannels"], kernel_size = kwargs["kernelSize"])

        def forward(self,x):
                #Encoding
                x = F.relu(self.enc1(x))
                x = F.relu(self.enc2(x))
                x = x.reshape(1,-1)
                x = x.squeeze()
                x = F.relu(self.enc3(x))
                #Decoding
                x = F.relu(self.dec1(x))
                x = x.reshape([32,4,28,28])
                x = F.relu(self.dec2(x))
                x = F.relu(self.dec3(x))

                return x

def encodeDecodeConv(numEpochs = 20, input_Channels = 3, batchSize = 32,
channels_Encoder_Inner = 4, channels_Encoder_Mid = 8, into_Linear = 100352,
linear_Encoded = 256,  channels_Decoder_Mid = 8, kernel_Size = 3,
learningRate = 1e-3):
        #Pick a device. If GPU available, use that. Otherwise, use CPU.
        device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
        #Define data transforms
        transform = transforms.Compose([transforms.ToTensor(),transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])
        #Define training dataset
        trainSet = datasets.CIFAR10(root = './data', train = True, download = True, transform = transform)
        #Define testing dataset
        testSet = datasets.CIFAR10(root = './data', train = False, download = True, transform = transform)
        #Define data loaders
        trainLoader = DataLoader(trainSet, batch_size = batchSize, shuffle = True)
        testLoader = DataLoader(testSet, batch_size = batchSize, shuffle = True)
        #Initialize neural network
        model = convAutoEncoder(inputChannels = input_Channels, channelsEncoderMid = channels_Encoder_Mid, channelsEncoderInner = channels_Encoder_Inner, intoLinear = into_Linear, linearEncoded = linear_Encoded, channelsDecoderMid = channels_Decoder_Mid, kernelSize = kernel_Size)
        #Optimization setup
        criterion = nn.MSELoss()
        optimizer = optim.Adam(model.parameters(),lr = learningRate)
        lossTracker = []
        for epoch in range(numEpochs):
                loss = 0
                for data,_ in trainLoader:
                        data = data.to(device)
                        optimizer.zero_grad()
                        outputs = model(data)
                        train_loss = criterion(outputs,data)
                        train_loss.backward()
                        optimizer.step()
                        loss += train_loss.item()
                loss = loss/len(trainLoader)
                print('Epoch {} of {}, Train loss: {:.3f}'.format(epoch+1,numEpochs,loss))

encodeDecodeConv()

Edit2: Somewhere in the CIFAR10 dataset, the data appears to change dimension. After playing around with print statements more, I discovered that setting the relevant hyperparameter to 100352 works great for many entries, but then seemingly one image pops up that has a different size. Not sure why that would occur, though.


title: I get error while downloading BERT models for summarization

body:
I'm a novice in writing neural networks. I have just started using BERT models, while running BERT for text summarization using the examples in bert extractive summarizer I get the following error with the pretrained model getting halted at 57%.
 OSError: Couldn't reach server at 'https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased-pytorch_model.bin' to download pretrained weights.
How do I resolve this error.
Thanks.


title: Pytorch - Should 'CenterCrop' be used to test set? Does this count as cheating？

body:
I'm learning image  classification with Pytorch. I found some papers code use 'CenterCrop' to both train set and test set，e.g. Resize to larger size，then apply CenterCrop to obtain a smaller size. The smaller size is a general size in this research direction.
In my experience, I found apply CenterCrop can get a significant improvement(e.g. 1% or 2%) on test,  compare to without CenterCrop on test set.
Because it is used in the top conference papers, confused me. So, Should CenterCrop be used to test set this count as cheating？ In addition, should I use any data augmentation to test set except 'Resize' and 'Normalization'?
Thank you for your answer.


title: "@" for tensor multiplication using pytorch

body:
This article https://towardsdatascience.com/understand-kaiming-initialization-and-implementation-detail-in-pytorch-f7aa967e9138 about intelligent weights initialization uses the syntax
x@w

to signify tensor (/matrix) multiplication.  I had not seen that before and instead had presumed we would need to "spell it out" as :
 torch.mm(x, w.t())

What is required to use the former (nicer) syntax? That article did not show a complete set of the imports they were using.


title: Output activation function for tensors for multilabel classification

body:
My expected label is a tensor like tensor([[0, 1, 0, 1], [1, 1, 1, 0]]).
The output from my model (note that I stopped training just to obtain the values so they do not accurately represent the activations of neurons) is also a tensor like tensor([[-10.6964, -13.8998, 0.8348, -45.7040], [-10.3260, -13.8385, -9.2342, -5.3424]])
Am I wrong in using BCEWithLogitsLoss directly on the outputs and labels? Would I need to convert the output tensor to a binary one similar to the expected label prior to using BCEWithLogitsLoss? I understand that BCEWithLogitsLoss is just BCELoss + Sigmoid activation. How can I obtain values of the type of the expected label tensor and what loss should I use in that case?


title: GRU Loss decreased upto 0.9 but not further, PyTorch

body:
the code that I am using for experimenting with GRU.
import torch
import torch.nn as nn
import torch.nn.functional as F

from collections import *

class N(nn.Module):
    def __init__(self):
        super().__init__()

        self.embed = nn.Embedding(5,2)

        self.layers = 4
        self.gru = nn.GRU(2, 512, self.layers, batch_first=True)
        self.bat = nn.BatchNorm1d(4)
        self.bat1 = nn.BatchNorm1d(4)
        self.bat2 = nn.BatchNorm1d(4)
        self.fc = nn.Linear(512,100)
        self.fc1 = nn.Linear(100,100)
        self.fc2 = nn.Linear(100,5)
        self.s = nn.Softmax(dim=-1)

    def forward(self,x):
        h0 = torch.zeros(self.layers, x.size(0), 512).requires_grad_()

        x = self.embed(x)

        x,hn = self.gru(x,h0)
        x  = self.bat(x)
        x = self.fc(x)
        x = nn.functional.relu(x)

        x = self.bat1(x)
        x = self.fc1(x)
        x = nn.functional.relu(x)

        x = self.bat2(x)
        x  = self.fc2(x)
        softmaxed = self.s(x)
        return  softmaxed

inp = torch.tensor([[4,3,2,1],[2,3,4,1],[4,1,2,3],[1,2,3,4]])
out = torch.tensor([[3,2,1,4],[3,2,4,1],[1,2,3,4],[2,3,4,1]])
k = 0
n = N()
opt = torch.optim.Adam(n.parameters(),lr=0.0001)

while k<10000:
    print(inp.shape)
    o = n(inp)

    o = o.view(-1, o.size(-1))
    out = out.view(-1)
    loss = nn.functional.cross_entropy(o.view(-1,o.size(-1)),out.view(-1)-1)

    acc = ((torch.argmax(o, dim=1) == (out -1)).sum().item() / out.size(0))
    if k==10000:
        print(torch.argmax(o, dim=1))
        print(out-1)
        exit()

    print(loss,acc)
    loss.backward()
    opt.step()
    opt.zero_grad()
    k+=1
print(o[0])

Shrinked Output:
torch.Size([4, 4])
tensor(0.9593, grad_fn=<NllLossBackward>) 0.9375
torch.Size([4, 4])
tensor(0.9593, grad_fn=<NllLossBackward>) 0.9375
tensor([4.8500e-01, 9.7813e-06, 5.1498e-01, 6.2428e-06, 7.5929e-06],
       grad_fn=<SelectBackward>)

The Loss is 0.9593 and accuracy reached up to 0.9375. For this simple input data, the GRU loss is this big. What is the reason? Is there anything wrong in this code? I used cross_entropy as loss function and Adam as the optimizer. Learning rate is 0.001. I tried multiple learning rates but all gave the same final result. I added batch normalization, it speed up the training, but the same loss and accuracy. Why loss does not decrease up to 0.2 or something.


title: Efficiently find the dot product of two lists of vectors stored as PyTorch tensors & preserve backprop

body:
Suppose I had tensors X and Y which are both (batch_size, d) dimensional. I would like to find the (batch_size x 1) tensor resulting from [X[0]@Y[0].T, X[1]@Y[1].T, ...]
There are two ways I can think of doing this, neither of which are particularly efficient.
Way 1
product = torch.eye(batch_size) * X@Y.T
product = torch.sum(product, dim=1)

This works, but for large matrices there are a LOT of wasted computations
Way 2
product = torch.cat(
    [ X[i]@Y[i].T for i in X.size(0) ],
    dim=0
)

This is good in that no cycles are wasted, but it won't leverage any of the built-in parallelism torch offers.
I'm aware that numpy has a method that will do this, but converting the tensors to np arrays will destroy the chain of backpropagation, and this is for a neural net, so that's not an option.
Am I missing an obvious built in torch method, or am I stuck with these two options?


title: What is the best way to create augmentation on image dataset while training instance segmentation?

body:
Background : I am using YOLACT instance segmentation model to train set of images. The dataset size is very small (~20 images). The model doesn't converge properly (of-course given the dataset size). I wanted to increase the dataset size by adding some augmented images. I know we have various image augmentation techniques and packages like imgaug , albumentation, opencv etc. but I need image & annotation file ( COCO JSON ) format to train the model.
So my question is :
Is there a package that helps me to automatically generate the annotations of augmented images ?
or
Is there a better way to address my issue ?
Thank you in advance for your help!


title: Multiprocessing, file not found

body:
I'm using AlphaPose from GitHub and I'd like to run the script script/demo_inference.py from another script I created in AlphaPose root called run.py. In run.py I imported demo_inference.py as ap using this script:
def import_module_by_path(path):
  name = os.path.splitext(os.path.basename(path))[0] spec =
  importlib.util.spec_from_file_location(name, path) mod =
  importlib.util.module_from_spec(spec) spec.loader.exec_module(mod) return mod

and
ap = import_module_by_path('./scripts/demo_inference.py')

Then, in demo_inference.py I substituted
if __name__ == "__main__":

with
def startAlphapose():

and in run.py I wrote
ap.StartAlphapose().

Now I got this error:
Load SE Resnet...
Loading YOLO model..
Process Process-3:
Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/home/vislab/guerri/alphagastnet/insieme/alphapose/utils/detector.py", line 251, in image_postprocess
    (orig_img, im_name, boxes, scores, ids, inps, cropped_boxes) = self.wait_and_get(self.det_queue)
  File "/home/vislab/guerri/alphagastnet/insieme/alphapose/utils/detector.py", line 121, in wait_and_get
    return queue.get()
  File "/usr/lib/python3.6/multiprocessing/queues.py", line 113, in get
    return _ForkingPickler.loads(res)
  File "/home/vislab/guerri/alphagastnet/lib/python3.6/site-packages/torch/multiprocessing/reductions.py", line 284, in rebuild_storage_fd
    fd = df.detach()
  File "/usr/lib/python3.6/multiprocessing/resource_sharer.py", line 57, in detach
    with _resource_sharer.get_connection(self._id) as conn:
  File "/usr/lib/python3.6/multiprocessing/resource_sharer.py", line 87, in get_connection
    c = Client(address, authkey=process.current_process().authkey)
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 487, in Client
    c = SocketClient(address)
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 614, in SocketClient
    s.connect(address)
FileNotFoundError: [Errno 2] No such file or directory

What does it mean?


title: Why is the timm visual transformer position embedding initializing to zeros?

body:
I'm looking at the timm implementation of visual transformers and for the positional embedding, he is initializing his position embedding with zeros as follows:
self.pos_embed = nn.Parameter(torch.zeros(1, num_patches + 1, embed_dim))

See here:
https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/vision_transformer.py#L309
I'm not sure how this actually embeds anything about the position when it is later added to the patch?
x = x + self.pos_embed

Any feedback is appreciated.


title: Exporting ONNX Model from Deep Learning Frameworks at Operator-level

body:
Hi, I have some questions regarding exporting ONNX models.
Let's say we have an LSTM cell from PyTorch.
Using torch.onnx.export produces ONNX model with LSTM layer.
However, I am interested in whether it can produce the ONNX model at the operator level, i.e, matmul, add.
Is there a way to do so?
If not, is there another way to make an operator level ONNX model?
Thanks,
Jake


title: How to train a Masked Language Model with a big text corpus(200GB) using PyTorch？

body:
Recently I am training a masked language model with a big text corpus(200GB) using transformers. The training data is too big to fit into computer equiped with 512GB memory and V100(32GB)*8. Is it possible to find a elegant way to train model with big data?
Now I split entire training data into 20 pieces and create 20 DataLoaders, each dataloader loads corresponding data into memory and uses DistributedDataParallel to train the model. The code is below.
def get_dataloader(rank, world_size, tokenizer, part_of_data, args):
    train_data_df = get_DataFrame(args, part_of_data)
    train_dataset = CustomDataset(train_data_df, tokenizer, args.max_len)

    data_collator = transformers.DataCollatorForLanguageModeling(
        tokenizer=tokenizer, mlm=True, mlm_probability=0.15
    )

    train_sampler = torch.utils.data.distributed.DistributedSampler(
        train_dataset,
        num_replicas=world_size,
        rank=rank,
    )

    train_loader = torch.utils.data.DataLoader(
        dataset=train_dataset,
        batch_size=args.batch_size,
        shuffle=False,
        num_workers=2,
        pin_memory=True,
        collate_fn=data_collator,
        sampler=train_sampler)

    return train_loader

By reading PyTorch Doc I find a dataloader called IterableDataset, maybe IterableDataset is more suitable for my task, but there is a NOTE in the Doc said neither sampler nor batch_sampler is compatible with iterable-style datasets, since such datasets have no notion of a key or an index.
So I wonder if it is possible to use IterableDataset, DistributedSampler and DistributedDataParallel to train model, or other elegant method to train model instead of splitting data. Thanks a lot.