title: Why are some weights int8 and some float32 (PyTorch 1.7 quantisation aware training)

body:
I trained a simple model (Conv2D + ReLU + Conv2D) with quantisation aware training in pytorch 1.7. After saving the state dict of the quantised model and opening it in Netron, I noticed that some weights are int8, but others are still float32.
For example, the Conv2D weights are int8, but the biases are float32. Why is this the case? And how does pytorch convert the biases to int8 when I load the model?
Edit: Below is the printed state_dict of the quantised model.
OrderedDict([('quant.scale', tensor([0.0058])), ('quant.zero_point', tensor([0])), ('Conv1.weight', tensor([[[[-0.1155, -0.1615, -0.0945],
          [-0.1274,  0.1510,  0.1589],
          [ 0.0775,  0.1497,  0.1274]],

         [[ 0.0775,  0.0709,  0.1339],
          [-0.1615,  0.0538,  0.0932],
          [-0.0249, -0.1011,  0.0341]],

         [[-0.1260,  0.1090, -0.0420],
          [ 0.0762, -0.1667, -0.1379],
          [-0.0105, -0.1549, -0.0026]]],

        [[[-0.0675, -0.1732,  0.0528],
          [-0.1130, -0.1482,  0.1864],
          [ 0.1746,  0.1379,  0.0528]],

         [[-0.1453,  0.1629, -0.0382],
          [-0.1159,  0.1291, -0.0088],
          [ 0.0293,  0.0719,  0.1379]],

         [[-0.0059,  0.0249, -0.0998],
          [ 0.1115, -0.1864,  0.1247],
          [-0.1423, -0.0558, -0.1643]]],

        [[[ 0.1288, -0.1288, -0.0186],
          [ 0.1248, -0.0637,  0.1434],
          [-0.1633, -0.1089,  0.1673]],

         [[-0.1700,  0.0120,  0.1474],
          [ 0.0637, -0.0359,  0.0518],
          [ 0.0664,  0.0292,  0.0637]],

         [[ 0.0996,  0.0359,  0.1129],
          [-0.1023,  0.0398,  0.1129],
          [ 0.0730, -0.1155,  0.0186]]],

        ...,

        [[[-0.1497, -0.1468,  0.0785],
          [-0.0044, -0.1061, -0.0988],
          [-0.0334,  0.0073, -0.0610]],

         [[ 0.1584, -0.0451,  0.1846],
          [ 0.0596,  0.0523, -0.1076],
          [-0.1337,  0.1453, -0.0727]],

         [[-0.0988, -0.1541,  0.1642],
          [-0.1512,  0.0669,  0.0770],
          [ 0.0959, -0.0480, -0.0363]]],

        [[[-0.0015, -0.0239, -0.0060],
          [ 0.1134, -0.0880, -0.0313],
          [-0.0507, -0.0627,  0.0462]],

         [[ 0.0224,  0.1536,  0.1164],
          [ 0.0462,  0.0164, -0.1686],
          [-0.1059, -0.1193, -0.1238]],

         [[-0.1119, -0.0403,  0.1417],
          [ 0.1895, -0.0462, -0.0746],
          [-0.0030, -0.0149, -0.1895]]],

        [[[-0.1158, -0.0868, -0.0116],
          [ 0.0767, -0.0999,  0.0449],
          [ 0.0174, -0.0999,  0.0043]],

         [[-0.0970, -0.0868,  0.0318],
          [ 0.1838, -0.0883,  0.0883],
          [ 0.0897,  0.0666, -0.0897]],

         [[-0.1129, -0.1838,  0.0174],
          [-0.1332, -0.0275, -0.1650],
          [-0.1172, -0.0825,  0.0449]]]], size=(128, 3, 3, 3),
       dtype=torch.qint8, quantization_scheme=torch.per_channel_affine,
       scale=tensor([0.0013, 0.0015, 0.0013, 0.0014, 0.0015, 0.0014, 0.0015, 0.0014, 0.0015,
        0.0015, 0.0014, 0.0015, 0.0015, 0.0015, 0.0014, 0.0015, 0.0015, 0.0015,
        0.0015, 0.0015, 0.0014, 0.0015, 0.0015, 0.0014, 0.0015, 0.0015, 0.0012,
        0.0015, 0.0015, 0.0014, 0.0015, 0.0015, 0.0014, 0.0014, 0.0015, 0.0015,
        0.0015, 0.0015, 0.0015, 0.0014, 0.0015, 0.0015, 0.0014, 0.0015, 0.0015,
        0.0014, 0.0015, 0.0014, 0.0015, 0.0013, 0.0015, 0.0014, 0.0014, 0.0015,
        0.0015, 0.0015, 0.0014, 0.0015, 0.0015, 0.0015, 0.0015, 0.0015, 0.0015,
        0.0015, 0.0015, 0.0015, 0.0015, 0.0014, 0.0015, 0.0014, 0.0014, 0.0015,
        0.0014, 0.0015, 0.0015, 0.0015, 0.0015, 0.0015, 0.0015, 0.0015, 0.0015,
        0.0015, 0.0015, 0.0015, 0.0014, 0.0015, 0.0014, 0.0015, 0.0013, 0.0014,
        0.0015, 0.0014, 0.0015, 0.0015, 0.0015, 0.0015, 0.0015, 0.0015, 0.0015,
        0.0015, 0.0015, 0.0015, 0.0015, 0.0015, 0.0014, 0.0015, 0.0014, 0.0015,
        0.0014, 0.0015, 0.0015, 0.0014, 0.0014, 0.0015, 0.0014, 0.0015, 0.0015,
        0.0014, 0.0015, 0.0014, 0.0015, 0.0015, 0.0014, 0.0015, 0.0014, 0.0015,
        0.0015, 0.0014], dtype=torch.float64),
       zero_point=tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0]),
       axis=0)), ('Conv1.bias', tensor([-0.0076, -0.1628,  0.0198,  0.0671,  0.0580, -0.0119,  0.1381,  0.1515,
         0.0006, -0.1816, -0.0398, -0.1390, -0.1904, -0.1584, -0.1862, -0.0725,
         0.1018,  0.0192,  0.0459,  0.0122, -0.0790,  0.0689,  0.1705, -0.0765,
        -0.0684,  0.1773, -0.1782, -0.0136, -0.0989, -0.1515, -0.1222, -0.1603,
         0.0768, -0.0196,  0.1086,  0.1526,  0.1713, -0.1683,  0.0665, -0.1556,
         0.0356, -0.1606,  0.1206,  0.0589,  0.1702, -0.1242,  0.1160,  0.1241,
        -0.0293,  0.0719, -0.1161, -0.0638,  0.1791,  0.0327,  0.0470, -0.1005,
        -0.0801,  0.0491, -0.0330,  0.1024, -0.1336, -0.1146,  0.1859, -0.0381,
         0.0134, -0.0438, -0.0574, -0.1637,  0.1240, -0.0087, -0.1361, -0.0820,
        -0.1605, -0.0396,  0.0912,  0.1553, -0.0816,  0.0428,  0.0644,  0.0374,
        -0.1021,  0.0983,  0.0982, -0.1161,  0.1612, -0.0651,  0.0618,  0.1647,
        -0.0890,  0.1622, -0.0400,  0.0981,  0.0385, -0.1839,  0.0367, -0.0605,
        -0.1760, -0.0574,  0.1601,  0.1603,  0.1676,  0.0800,  0.1583, -0.1492,
         0.0910, -0.0997, -0.0313, -0.0552,  0.1784, -0.1193, -0.0585, -0.1764,
         0.0099, -0.0066,  0.0491,  0.0959, -0.0493,  0.1617,  0.1466,  0.1692,
         0.1415,  0.0424, -0.1162,  0.1862,  0.0290, -0.0040,  0.0939,  0.1713],
       requires_grad=True)), ('Conv1.scale', tensor(0.0187)), ('Conv1.zero_point', tensor(75)), ('Conv2.weight', tensor([[[[-0.0155, -0.0042,  0.0136],
          [-0.0194,  0.0005,  0.0187],
          [-0.0058,  0.0044,  0.0148]],

         [[-0.0226,  0.0053, -0.0247],
          [ 0.0079,  0.0226, -0.0129],
          [ 0.0014,  0.0044,  0.0164]],

         [[ 0.0185,  0.0247,  0.0046],
          [ 0.0122,  0.0259, -0.0148],
          [ 0.0226,  0.0173,  0.0083]],

         ...,

         [[-0.0215,  0.0051, -0.0136],
          [-0.0159,  0.0000, -0.0203],
          [-0.0219, -0.0141, -0.0282]],

         [[-0.0065,  0.0085, -0.0139],
          [-0.0028, -0.0039,  0.0171],
          [ 0.0254,  0.0152,  0.0282]],

         [[-0.0293,  0.0049,  0.0002],
          [ 0.0245, -0.0046, -0.0042],
          [ 0.0240,  0.0236, -0.0256]]],

        [[[-0.0247,  0.0194,  0.0127],
          [ 0.0134,  0.0240,  0.0288],
          [ 0.0136, -0.0198, -0.0120]],

         [[ 0.0005, -0.0088,  0.0044],
          [-0.0014, -0.0101,  0.0062],
          [-0.0053,  0.0293,  0.0085]],

         [[-0.0201, -0.0101,  0.0083],
          [-0.0201, -0.0263, -0.0184],
          [-0.0113, -0.0062,  0.0062]],

         ...,

         [[ 0.0260,  0.0288, -0.0002],
          [ 0.0127,  0.0272,  0.0270],
          [-0.0168, -0.0157, -0.0092]],

         [[-0.0065,  0.0067, -0.0164],
          [ 0.0124, -0.0244,  0.0035],
          [ 0.0293, -0.0028, -0.0152]],

         [[-0.0154,  0.0014, -0.0196],
          [ 0.0078, -0.0044,  0.0115],
          [-0.0048, -0.0071, -0.0088]]],

        [[[ 0.0069, -0.0099,  0.0259],
          [ 0.0268,  0.0102,  0.0192],
          [ 0.0173, -0.0206,  0.0203]],

         [[-0.0016, -0.0183, -0.0217],
          [ 0.0226,  0.0129, -0.0092],
          [-0.0018,  0.0166,  0.0164]],

         [[-0.0229,  0.0201, -0.0187],
          [-0.0074,  0.0183, -0.0273],
          [-0.0139, -0.0023, -0.0226]],

         ...,

         [[ 0.0224, -0.0099, -0.0196],
          [ 0.0157,  0.0018, -0.0194],
          [ 0.0016,  0.0201,  0.0044]],

         [[-0.0217, -0.0252, -0.0141],
          [-0.0116,  0.0270, -0.0085],
          [ 0.0231,  0.0291,  0.0153]],

         [[ 0.0148, -0.0215, -0.0116],
          [ 0.0155, -0.0049, -0.0166],
          [-0.0254,  0.0032, -0.0125]]]], size=(3, 128, 3, 3),
       dtype=torch.qint8, quantization_scheme=torch.per_channel_affine,
       scale=tensor([0.0002, 0.0002, 0.0002], dtype=torch.float64),
       zero_point=tensor([0, 0, 0]), axis=0)), ('Conv2.bias', tensor([-0.0076,  0.0025, -0.0067], requires_grad=True)), ('Conv2.scale', tensor(0.0026)), ('Conv2.zero_point', tensor(27))])

Apparently the conv weights are also float, for some reason Netron converts those to int8.


title: Time series forecasting. How use future values

body:
I have a time series dataset containing hourly data from a few year, like below. Let's assume that I want to make prediction for the next 3 hours (2021-01-01 19:00, 2021-01-01 20:00, 2021-01-01 21:00). The output from model (e.g. LSTM) is the following sequence: [Feature_1(t+1), Feature_1(t+2), Feature_1(t+3)]. One of the features of my training data is weather data.

Date
Feature_1
Weather_feature

2019-01-01 15:00
value(t0)
value(t0)

2019-01-01 16:00
value(t0)
value(t0)

....
...
...

2021-01-01 18:00
value(t0)
value(t0)

Can I use data, that I already know in my model? I would like to use the weather forecast for the next hours to improve my predctions.
For example at 2021-01-01 18:00 i know value Feature_1 and  Weather_feature in time t(0) but also i know already Weather_feature in (t+1), (t+2), (t+3). How can i use it in the model?
At the moment, this is what one training sample looks like: [[Feature_1(t-n), Weather_feature(t-n)]...[Feature_1(t-1),Weather_feature(t-1), [Feature_1(t0), Weather_feature(t0)]]


title: How to use fine-tuned BERT model for sentence encoding?

body:
I fine-tuned the BERT base model on my own dataset following the script here:
https://github.com/cedrickchee/pytorch-pretrained-BERT/tree/master/examples/lm_finetuning
I saved the model as a .pt file and I want to use it now for a sentence similarity task. Unfortunately, it is not clear to me, how to load the fine-tuned model. I tried the following:
model = BertModel.from_pretrained('trained_model.pt')
model.eval()

This doesn't work. It says:
ReadError: not a gzip file

So apparently, loading a .pt file with the from_pretrained method is not possible. Can anyone help me out here? Thank's a lot!! :)
Edit: I saved the model in a s3 bucket as follows:
# Convert model to buffer
buffer = io.BytesIO()
torch.save(model, buffer)
# Save in s3 bucket
output_model_file = output_folder + "trained_model.pt"
s3_.put_object(Bucket="power-plant-embeddings", Key=output_model_file, Body=buffer.getvalue())


title: RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. GPU not detected by pytorch

body:
Having trouble with CUDA + Pytorch this is the error. I reinstalled CUDA and cudnn multiple times.
Conda env is detecting GPU but its giving errors with pytorch and certain cuda libraries. I tried with Cuda 10.1 and 10.0, and cudnn version 8 and 7.6.5, Added cuda to path and everything.
However anaconda is showing cuda tool kit 9.0 is installed, whilst I clearly installed 10.0, so I am not entirely sure what's the deal with that.
    => loading model from models/pytorch/pose_coco/pose_hrnet_w32_256x192.pth
Traceback (most recent call last):
  File "hydroman2.py", line 580, in <module>
    pose_model.load_state_dict(torch.load(cfg.TEST.MODEL_FILE), strict=False)
  File "C:\Users\Fardin\anaconda3\envs\myenv\lib\site-packages\torch\serialization.py", line 593, in load
    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
  File "C:\Users\Fardin\anaconda3\envs\myenv\lib\site-packages\torch\serialization.py", line 773, in _legacy_load
    result = unpickler.load()
  File "C:\Users\Fardin\anaconda3\envs\myenv\lib\site-packages\torch\serialization.py", line 729, in persistent_load
    deserialized_objects[root_key] = restore_location(obj, location)
  File "C:\Users\Fardin\anaconda3\envs\myenv\lib\site-packages\torch\serialization.py", line 178, in default_restore_location
    result = fn(storage, location)
  File "C:\Users\Fardin\anaconda3\envs\myenv\lib\site-packages\torch\serialization.py", line 154, in _cuda_deserialize
    device = validate_cuda_device(location)
  File "C:\Users\Fardin\anaconda3\envs\myenv\lib\site-packages\torch\serialization.py", line 138, in validate_cuda_device
    raise RuntimeError('Attempting to deserialize object on a CUDA '
RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.

System info
   System info:
--------------------------------------------------------------------------------
__Time Stamp__
Report started (local time)                   : 2021-03-19 19:59:06.957967
UTC start time                                : 2021-03-19 15:59:06.957967
Running time (s)                              : 4.003899

__Hardware Information__
Machine                                       : AMD64
CPU Name                                      : znver1
CPU Count                                     : 12
Number of accessible CPUs                     : 12
List of accessible CPUs cores                 : 0 1 2 3 4 5 6 7 8 9 10 11
CFS Restrictions (CPUs worth of runtime)      : None

CPU Features                                  : 64bit adx aes avx avx2 bmi bmi2
                                                clflushopt clzero cmov cx16 cx8
                                                f16c fma fsgsbase fxsr lzcnt mmx
                                                movbe mwaitx pclmul popcnt prfchw
                                                rdrnd rdseed sahf sha sse sse2
                                                sse3 sse4.1 sse4.2 sse4a ssse3
                                                xsave xsavec xsaveopt xsaves

Memory Total (MB)                             : 16334
Memory Available (MB)                         : 8787

__OS Information__
Platform Name                                 : Windows-10-10.0.19041-SP0
Platform Release                              : 10
OS Name                                       : Windows
OS Version                                    : 10.0.19041
OS Specific Version                           : 10 10.0.19041 SP0 Multiprocessor Free
Libc Version                                  : ?

__Python Information__
Python Compiler                               : MSC v.1916 64 bit (AMD64)
Python Implementation                         : CPython
Python Version                                : 3.8.5
Python Locale                                 : en_US.cp1252

__LLVM Information__
LLVM Version                                  : 10.0.1

__CUDA Information__
CUDA Device Initialized                       : True
CUDA Driver Version                           : 11020
CUDA Detect Output:
Found 1 CUDA devices
id 0     b'GeForce GTX 1070'                              [SUPPORTED]
                      compute capability: 6.1
                           pci device id: 0
                              pci bus id: 6
Summary:
        1/1 devices are supported

CUDA Librairies Test Output:
Finding cublas from <unknown>
        named  cublas.dll
        trying to open library...       ERROR: failed to open cublas:
Could not find module 'cublas.dll' (or one of its dependencies). Try using the full path with constructor syntax.
Finding cusparse from <unknown>
        named  cusparse.dll
        trying to open library...       ERROR: failed to open cusparse:
Could not find module 'cusparse.dll' (or one of its dependencies). Try using the full path with constructor syntax.
Finding cufft from <unknown>
        named  cufft.dll
        trying to open library...       ERROR: failed to open cufft:
Could not find module 'cufft.dll' (or one of its dependencies). Try using the full path with constructor syntax.
Finding curand from <unknown>
        named  curand.dll
        trying to open library...       ERROR: failed to open curand:
Could not find module 'curand.dll' (or one of its dependencies). Try using the full path with constructor syntax.
Finding nvvm from <unknown>
        named  nvvm.dll
        trying to open library...       ERROR: failed to open nvvm:
Could not find module 'nvvm.dll' (or one of its dependencies). Try using the full path with constructor syntax.
Finding cudart from <unknown>
        named  cudart.dll
        trying to open library...       ERROR: failed to open cudart:
Could not find module 'cudart.dll' (or one of its dependencies). Try using the full path with constructor syntax.
Finding libdevice from <unknown>
        searching for compute_20...     ERROR: can't open libdevice for compute_20
        searching for compute_30...     ERROR: can't open libdevice for compute_30
        searching for compute_35...     ERROR: can't open libdevice for compute_35
        searching for compute_50...     ERROR: can't open libdevice for compute_50

__ROC information__
ROC Available                                 : False
ROC Toolchains                                : None
HSA Agents Count                              : 0
HSA Agents:
None
HSA Discrete GPUs Count                       : 0
HSA Discrete GPUs                             : None

__SVML Information__
SVML State, config.USING_SVML                 : True
SVML Library Loaded                           : True
llvmlite Using SVML Patched LLVM              : True
SVML Operational                              : True

__Threading Layer Information__
TBB Threading Layer Available                 : False
+--> Disabled due to Unknown import problem.
OpenMP Threading Layer Available              : True
+-->Vendor: MS
Workqueue Threading Layer Available           : True
+-->Workqueue imported successfully.

__Numba Environment Variable Information__
None found.

__Conda Information__
Conda Build                                   : 3.20.5
Conda Env                                     : 4.9.2
Conda Platform                                : win-64
Conda Python Version                          : 3.8.5.final.0
Conda Root Writable                           : True

__Installed Packages__
_pytorch_select           1.1.0                       cpu    anaconda
_tflow_select             2.3.0                       mkl    anaconda
absl-py                   0.12.0                   pypi_0    pypi
alabaster                 0.7.12                   pypi_0    pypi
appdirs                   1.4.3            py36h28b3542_0    anaconda
argparse                  1.4.0                    pypi_0    pypi
asn1crypto                1.3.0                    py36_0    anaconda
astor                     0.8.1              pyh9f0ad1d_0    conda-forge
astunparse                1.6.3                    pypi_0    pypi
atomicwrites              1.4.0                      py_0    anaconda
attrs                     19.3.0                     py_0    anaconda
babel                     2.9.0                    pypi_0    pypi
backcall                  0.2.0                      py_0    anaconda
backports                 1.0                        py_2    anaconda
backports.weakref         1.0.post1       py36h9f0ad1d_1001    conda-forge
blas                      1.0                         mkl    anaconda
bleach                    1.5.0                    py36_0    conda-forge
blinker                   1.4                        py_1    conda-forge
brotlipy                  0.7.0           py36he774522_1000    anaconda
bzip2                     1.0.8                he774522_0    anaconda
ca-certificates           2020.10.14                    0    anaconda
cachetools                4.1.1                      py_0    anaconda
certifi                   2020.6.20                py36_0    anaconda
cffi                      1.14.0           py36h7a1dbc1_0    anaconda
chardet                   3.0.4                 py36_1003    anaconda
click                     7.1.2              pyh9f0ad1d_0    conda-forge
cloudpickle               1.4.1                      py_0    anaconda
colorama                  0.4.3                      py_0    anaconda
contextlib2               0.6.0.post1                py_0    anaconda
cpuonly                   1.0                           0    pytorch
cryptography              2.9.2            py36h7a1dbc1_0    anaconda
cudatoolkit               9.0                           1    anaconda
cudnn                     7.6.5                 cuda9.0_0    anaconda
curl                      7.71.0               h2a8f88b_0    anaconda
cycler                    0.10.0           py36h009560c_0    anaconda
cython                    0.29.22                  pypi_0    pypi
cytoolz                   0.10.1           py36he774522_0    anaconda
dask-core                 2.19.0                     py_0    anaconda
decorator                 4.4.2                      py_0    anaconda
defusedxml                0.6.0                      py_0    anaconda
dlib                      19.20            py36h5653133_1    conda-forge
docker-py                 4.2.1            py36h9f0ad1d_0    conda-forge
docker-pycreds            0.4.0                      py_0    anaconda
docutils                  0.16                     pypi_0    pypi
easydict                  1.7                      pypi_0    pypi
entrypoints               0.3                      py36_0    anaconda
ffmpeg                    2.7.0                         0    menpo
flake8                    3.8.3                      py_0    anaconda
flake8-polyfill           1.0.2                    py36_0    anaconda
flake8-quotes             3.0.0              pyh9f0ad1d_0    conda-forge
flatbuffers               1.12                     pypi_0    pypi
freetype                  2.10.2               hd328e21_0    anaconda
gast                      0.2.2                    pypi_0    pypi
geos                      3.8.1                h33f27b4_0    anaconda
gettext                   0.19.8.1          hb01d8f6_1002    conda-forge
git                       2.23.0               h6bb4b03_0    anaconda
glib                      2.58.3          py36h04c7ab9_1004    conda-forge
google-auth               1.28.0                   pypi_0    pypi
google-auth-oauthlib      0.4.3                    pypi_0    pypi
google-pasta              0.2.0              pyh8c360ce_0    conda-forge
grpcio                    1.32.0                   pypi_0    pypi
h5py                      2.10.0           py36h5e291fa_0    anaconda
hdf5                      1.10.4               h7ebc959_0    anaconda
html5lib                  0.9999999                py36_0    conda-forge
icc_rt                    2019.0.0             h0cc432a_1    anaconda
icu                       58.2                 ha925a31_3    anaconda
idna                      2.10                       py_0    anaconda
imageio                   2.8.0                      py_0    anaconda
imageio-ffmpeg            0.4.2                      py_0    conda-forge
imagesize                 1.2.0                    pypi_0    pypi
imgaug                    0.4.0                    pypi_0    pypi
importlib-metadata        1.7.0                    py36_0    anaconda
importlib_metadata        1.7.0                         0    anaconda
intel-openmp              2019.4                      245    anaconda
ipykernel                 5.3.0            py36h5ca1d4c_0    anaconda
ipyparallel               6.3.0                    pypi_0    pypi
ipython                   7.16.1           py36h5ca1d4c_0    anaconda
ipython_genutils          0.2.0                    py36_0    anaconda
ipywidgets                7.5.1                      py_0    anaconda
jedi                      0.17.1                   py36_0    anaconda
jinja2                    2.11.2                     py_0    anaconda
joblib                    0.15.1                     py_0    anaconda
jpeg                      9d                   he774522_0    conda-forge
json-tricks               3.15.5                   pypi_0    pypi
jsonschema                3.2.0                    py36_0    anaconda
jupyter                   1.0.0                    py36_7    anaconda
jupyter_client            6.1.3                      py_0    anaconda
jupyter_console           6.1.0                      py_0    anaconda
jupyter_core              4.6.3                    py36_0    anaconda
keras-applications        1.0.8                      py_1    anaconda
keras-preprocessing       1.1.2                    pypi_0    pypi
kiwisolver                1.2.0            py36h74a9793_0    anaconda
krb5                      1.18.2               hc04afaa_0    anaconda
leptonica                 1.78.0               h919f142_2    conda-forge
libarchive                3.3.3                h0643e63_5    anaconda
libcurl                   7.71.0               h2a8f88b_0    anaconda
libffi                    3.2.1             h6538335_1007    conda-forge
libgpuarray               0.7.6             hfa6e2cd_1003    conda-forge
libiconv                  1.15             vc14h29686d3_5  [vc14]  anaconda
libmklml                  2019.0.5                      0    anaconda
libpng                    1.6.37               h2a8f88b_0    anaconda
libprotobuf               3.12.3               h7bd577a_0    anaconda
libsodium                 1.0.18               h62dcd97_0    anaconda
libssh2                   1.9.0                h7a1dbc1_1    anaconda
libtiff                   4.1.0                h56a325e_0    anaconda
libwebp                   1.0.2                hfa6e2cd_5    conda-forge
libxml2                   2.9.10               h464c3ec_1    anaconda
libxslt                   1.1.34               he774522_0    anaconda
lxml                      4.5.0            py36h1350720_0    anaconda
lz4-c                     1.8.1.2              h2fa13f4_0    anaconda
lzo                       2.10                 he774522_2    anaconda
m2w64-gcc-libgfortran     5.3.0                         6    conda-forge
m2w64-gcc-libs            5.3.0                         7    conda-forge
m2w64-gcc-libs-core       5.3.0                         7    conda-forge
m2w64-gmp                 6.1.0                         2    conda-forge
m2w64-libwinpthread-git   5.0.0.4634.697f757               2    conda-forge
mako                      1.1.0                      py_0    anaconda
markdown                  3.3.4                    pypi_0    pypi
markupsafe                1.1.1            py36he774522_0    anaconda
matplotlib                3.1.3                    py36_0    anaconda
matplotlib-base           3.1.3            py36h64f37c6_0    anaconda
mccabe                    0.6.1                    py36_1    anaconda
mistune                   0.8.4            py36he774522_0    anaconda
mkl                       2018.0.3                      1    anaconda
mkl_fft                   1.0.6            py36hdbbee80_0    anaconda
mkl_random                1.0.1            py36h77b88f5_1    anaconda
mock                      4.0.3                    pypi_0    pypi
more-itertools            8.4.0                      py_0    anaconda
moviepy                   1.0.1                      py_0    conda-forge
msys2-conda-epoch         20160418                      1    conda-forge
nbconvert                 5.6.1                    py36_0    anaconda
nbformat                  5.0.7                      py_0    anaconda
networkx                  2.4                        py_0    anaconda
ninja                     1.9.0            py36h74a9793_0    anaconda
nose                      1.3.7                    pypi_0    pypi
notebook                  6.0.3                    py36_0    anaconda
numpy                     1.19.5                   pypi_0    pypi
oauthlib                  3.1.0                      py_0    anaconda
olefile                   0.46                     py36_0    anaconda
opencv-python             3.4.1.15                 pypi_0    pypi
openjpeg                  2.3.1                h57dd2e7_3    conda-forge
openssl                   1.1.1h               he774522_0    anaconda
opt-einsum                3.3.0                    pypi_0    pypi
packaging                 20.4                       py_0    anaconda
pandas                    1.0.3            py36h47e9c7a_0    anaconda
pandoc                    2.9.2.1                       0    anaconda
pandocfilters             1.4.2                    py36_1    anaconda
parso                     0.7.0                      py_0    anaconda
pcre                      8.44                 ha925a31_0    anaconda
pep8-naming               0.8.2                    py36_0    anaconda
pickleshare               0.7.5                    py36_0    anaconda
pillow                    7.1.2            py36hcc1f983_0    anaconda
pip                       20.2.4                   py36_0    anaconda
pluggy                    0.13.1                   py36_0    anaconda
poppler                   0.87.0               hdbe765f_0    conda-forge
poppler-data              0.4.9                         1    conda-forge
proglog                   0.1.9                      py_0    conda-forge
prometheus_client         0.8.0                      py_0    anaconda
prompt-toolkit            3.0.5                      py_0    anaconda
prompt_toolkit            3.0.5                         0    anaconda
protobuf                  3.12.3           py36h33f27b4_0    anaconda
psutil                    5.8.0                    pypi_0    pypi
py                        1.9.0                      py_0    anaconda
pyasn1                    0.4.8                      py_0    anaconda
pyasn1-modules            0.2.8                    pypi_0    pypi
pycocotools               2.0                      pypi_0    pypi
pycodestyle               2.6.0                      py_0    anaconda
pycparser                 2.20                       py_0    anaconda
pyflakes                  2.2.0                      py_0    anaconda
pygments                  2.6.1                      py_0    anaconda
pygpu                     0.7.6           py36h7725771_1001    conda-forge
pyjwt                     1.7.1                      py_0    conda-forge
pyopenssl                 19.1.0                   py36_0    anaconda
pyparsing                 2.4.7                      py_0    anaconda
pyqt                      5.9.2            py36h6538335_2    anaconda
pyreadline                2.1                   py36_1001    conda-forge
pyrsistent                0.16.0           py36he774522_0    anaconda
pysocks                   1.7.1                    py36_0    anaconda
pytesseract               0.3.3              pyh8c360ce_0    conda-forge
pytest                    5.4.3                    py36_0    anaconda
python                    3.6.10               h9f7ef89_1    anaconda
python-dateutil           2.8.1                      py_0    anaconda
python_abi                3.6                     1_cp36m    conda-forge
pytorch                   1.5.1               py3.6_cpu_0  [cpuonly]  pytorch
pytz                      2020.1                     py_0    anaconda
pywavelets                1.1.1            py36he774522_0    anaconda
pywin32                   223              py36hfa6e2cd_1    anaconda
pywinpty                  0.5.7                    py36_0    anaconda
pyyaml                    5.3.1            py36he774522_0    anaconda
pyzmq                     19.0.1           py36ha925a31_1    anaconda
qt                        5.9.7            vc14h73c81de_0  [vc14]  anaconda
qtconsole                 4.7.5                      py_0    anaconda
qtpy                      1.9.0                      py_0    anaconda
requests                  2.24.0                     py_0    anaconda
requests-oauthlib         1.3.0              pyh9f0ad1d_0    conda-forge
rsa                       4.6                pyh9f0ad1d_0    conda-forge
scikit-image              0.16.2           py36h47e9c7a_0    anaconda
scikit-learn              0.20.1           py36hb854c30_0    anaconda
scipy                     1.4.1                    pypi_0    pypi
send2trash                1.5.0                    py36_0    anaconda
setuptools                50.3.0           py36h9490d1a_1    anaconda
shapely                   1.6.4                    pypi_0    pypi
simplejson                3.17.0           py36he774522_0    anaconda
sip                       4.19.8           py36h6538335_0    anaconda
six                       1.15.0                     py_0    anaconda
sklearn                   0.0                      pypi_0    pypi
slidingwindow             0.0.14                   pypi_0    pypi
snowballstemmer           2.1.0                    pypi_0    pypi
sphinx                    3.5.2                    pypi_0    pypi
sphinxcontrib-applehelp   1.0.2                    pypi_0    pypi
sphinxcontrib-devhelp     1.0.2                    pypi_0    pypi
sphinxcontrib-htmlhelp    1.0.3                    pypi_0    pypi
sphinxcontrib-jsmath      1.0.1                    pypi_0    pypi
sphinxcontrib-qthelp      1.0.3                    pypi_0    pypi
sphinxcontrib-serializinghtml 1.1.4                    pypi_0    pypi
sqlite                    3.32.3               h2a8f88b_0    anaconda
swig                      3.0.12               h047fa9f_3    anaconda
tbb                       2020.0               h74a9793_0    anaconda
tbb4py                    2020.0           py36h74a9793_0    anaconda
tensorboard               1.13.1                   pypi_0    pypi
tensorboard-plugin-wit    1.8.0                    pypi_0    pypi
tensorboardx              1.6                        py_0    conda-forge
tensorflow                2.4.1                    pypi_0    pypi
tensorflow-estimator      1.13.0                   pypi_0    pypi
tensorflow-gpu            1.13.1                   pypi_0    pypi
tensorflow-gpu-estimator  2.1.0                    pypi_0    pypi
termcolor                 1.1.0                    pypi_0    pypi
terminado                 0.8.3                    py36_0    anaconda
testpath                  0.4.4                      py_0    anaconda
theano                    1.0.4           py36h003fed8_1002    conda-forge
threadpoolctl             2.1.0              pyh5ca1d4c_0    anaconda
tk                        8.6.10               he774522_0    anaconda
toolz                     0.10.0                     py_0    anaconda
torchfile                 0.1.0                      py_0    conda-forge
torchvision               0.6.1                  py36_cpu  [cpuonly]  pytorch
tornado                   6.0.4            py36he774522_1    anaconda
tqdm                      4.47.0                     py_0    anaconda
traitlets                 4.3.3                    py36_0    anaconda
typing-extensions         3.7.4.3                  pypi_0    pypi
urllib3                   1.25.11                    py_0    anaconda
vc                        14.1                 h0510ff6_4    anaconda
visdom                    0.1.8.9                       0    conda-forge
vs2015_runtime            14.16.27012          hf0eaf9b_3    anaconda
vs2017_win-64             19.16.27038          h2e3bad8_2    conda-forge
vswhere                   2.7.1                h21ff451_0    anaconda
wcwidth                   0.2.5                      py_0    anaconda
webencodings              0.5.1                    py36_1    anaconda
websocket-client          0.57.0                   py36_1    anaconda
werkzeug                  1.0.1              pyh9f0ad1d_0    conda-forge
wget                      1.16.3                        0    menpo
wheel                     0.35.1                     py_0    anaconda
widgetsnbextension        3.5.1                    py36_0    anaconda
win_inet_pton             1.1.0                    py36_0    anaconda
wincertstore              0.2              py36h7fe50ca_0    anaconda
winpty                    0.4.3                         4    anaconda
wrapt                     1.12.1           py36h68a101e_1    conda-forge
xz                        5.2.5                h62dcd97_0    anaconda
yacs                      0.1.8                    pypi_0    pypi
yaml                      0.1.7                hc54c509_2    anaconda
zeromq                    4.3.2                ha925a31_2    anaconda
zipp                      3.3.1                      py_0    anaconda
zlib                      1.2.11               h62dcd97_4    anaconda
zstd                      1.3.7                h508b16e_0    anaconda

No errors reported.


title: How to feed a image to Generator in GAN Pytorch

body:
So, I'm training a DCGAN model in pytorch on celeba dataset (people). And here is the architecture of the generator:
Generator(
  (main): Sequential(
    (0): ConvTranspose2d(100, 512, kernel_size=(4, 4), stride=(1, 1), bias=False)
    (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (2): ReLU(inplace=True)
    (3): ConvTranspose2d(512, 256, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1), bias=False)
    (4): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (5): ReLU(inplace=True)
    (6): ConvTranspose2d(256, 128, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1), bias=False)
    (7): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (8): ReLU(inplace=True)
    (9): ConvTranspose2d(128, 64, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1), bias=False)
    (10): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (11): ReLU(inplace=True)
    (12): ConvTranspose2d(64, 3, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1), bias=False)
    (13): Tanh()
  )
)

So after training, I want to check what generator outputs if I feed an occluded image like this:
 (size: 64X64)
But as u might have guessed that the image has 3 channels and my generator accepts a latent vector of 100 channels at the starting, so what is the correct way to feed this image to the generator and check the output. (I'm expecting that the generator tries to generate only the occluded part of the image). If you want a reference code then try this demo file of pytorch. I have modified this file according to my own needs, so for referring, this will do the trick.


So I went through some of the threads and posts and I was able to get rid of some issues. But, I am still facing this error:
RuntimeError: expected stride to be a single integer value or a list of 1 values to match the convolution dimensions, but got stride=[1, 1]

In my case, I have assigned my own weights and I am using a 3rd party library to just see how the network is working (torchinfo ). I am giving the input as (1,1,28,28) . My model is as follows:
net(
  (conv1): Conv2d(1, 1, kernel_size=(2, 2), stride=(1, 1))
  (pool): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  (conv2): Conv2d(1, 1, kernel_size=(2, 2), stride=(1, 1))
)

Any suggestions on getting rid of the error?


title: GradReverseLayer :NotImplementedError: You must implement the backward function for custom autograd.Function

body:
I'm recently in a domain adaptation program which involves using gradreverselayer.
This is what I'm using.

class GradReverse(Function):
    def forward(ctx, x, alpha):
        ctx.alpha = alpha

        return x

    def backward(ctx, grad_output):
        output = grad_output.neg() * ctx.alpha

        return output, None

It seems there's nothing wrong with it.
However, when I run my code, it raised this error.

Traceback (most recent call last):
  File "main.py", line 125, in <module>
    class_criterion, domain_criterion,  optimizer, trainloader1, valloader, trainloader2, testloader2)
  File "/content/drive/MyDrive/TRANSFER/Train.py", line 325, in train_original
    src_domain_loss.backward(retain_graph=True)
  File "/usr/local/lib/python3.7/dist-packages/torch/tensor.py", line 245, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/usr/local/lib/python3.7/dist-packages/torch/autograd/__init__.py", line 147, in backward
    allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag
  File "/usr/local/lib/python3.7/dist-packages/torch/autograd/function.py", line 89, in apply
    return self._forward_cls.backward(self, *args)  # type: ignore
  File "/usr/local/lib/python3.7/dist-packages/torch/autograd/function.py", line 201, in backward
    raise NotImplementedError("You must implement the backward function for custom"
NotImplementedError: You must implement the backward function for custom autograd.Function.

I can't figure it out. So I add

src_domain_loss=torch.tensor(src_domain_loss, requires_grad=True)

And there's no warning.
Howerver, when I check my net's grad, it's like this:

-->name: domain_classifier.c_fc1.weight -->grad_requirs: True -->grad_value: None
-->name: domain_classifier.c_fc1.bias -->grad_requirs: True -->grad_value: None
-->name: domain_classifier.c_fc2.weight -->grad_requirs: True -->grad_value: None
-->name: domain_classifier.c_fc2.bias -->grad_requirs: True -->grad_value: None

And once I remove the gradreverselayer, it works well. No warning, and the grad is fine.
It really bothers me. I wish someone could help me.
Thank you in advance.


title: How to compute iSTFT in Swift?

body:
I've been trying to compute iSTFT in swift using Accelerate and DSP Functions for the past ten days without success. In Python, I usually use torch library to complete this process, and it works well:
import numpy as np
import torch
from librosa.output import write_wav
import wav_file

win_length = 4096
hop_length = 1024
win = torch.hann_window(win_length)
saveFile = torch.istft(wav_file.arr, win_length, hop_length=hop_length, window=win)
write_wav('out.wav', np.asfortranarray(saveFile.squeeze().numpy()), 44100)

In swift however, I used a combination of vDSP_vsmul, vDSP_ztoc, as well as vDSP_fft_zip with FFT_INVERSE. This is my code in Swift:
View Controller:
override func viewDidLoad() {
    super.viewDidLoad()

    let path = Bundle.main.path(forResource: "wav_file", ofType: "json")!
    let data = try! Data(contentsOf: URL(fileURLWithPath: path))
    let stft_masked:[[[[Double]]]] = try! JSONSerialization.jsonObject(with: data) as! [[[[Double]]]]

    let a1 = istft(data: stft_masked[0])
    let a2 = istft(data: stft_masked[1])

    saveWav(buf: [a1.convertToFloat, a2.convertToFloat])

}

Functions
func istft(data:[[[Double]]]) -> [Double] {
    let win_length = 4096
    let resultReal = data.map { $0.map { $0[0] }}
    let resultImag = data.map { $0.map { $0[1] }}

    let resultReal_transposed = resultReal.transposed
    let resultImag_Transposed = resultImag.transposed
    var FlatReal = resultReal_transposed.reduce([], +).convertToFloat
    var FlatImag = resultImag_Transposed.reduce([], +).convertToFloat

    var result : [Float] = [Float](repeating: 0.0, count: FlatReal.count)
    var resultComplex : UnsafeMutablePointer<DSPComplex>? = nil
    result.withUnsafeMutableBytes {
        resultComplex = $0.baseAddress?.bindMemory(to: DSPComplex.self, capacity: FlatReal.count)
    }

    var splitComplexBuffer = DSPSplitComplex(realp: &FlatReal, imagp: &FlatImag)
    let log2Sizes = vDSP_Length(log2f(Float(12)))
    let setupFFTz = vDSP_create_fftsetup(log2Sizes, FFTRadix(FFT_RADIX2))!
    vDSP_fft_zip(setupFFTz, &splitComplexBuffer, 1, vDSP_Length(12), FFTDirection(FFT_INVERSE));
    vDSP_ztoc(&splitComplexBuffer, 1, resultComplex!, 1, vDSP_Length(12));
    var scale : Float = 1.0;
    var copyOfResult = result;
    vDSP_vsmul(&result, 1, &scale, &copyOfResult, 1, vDSP_Length(12));
    result = copyOfResult

    let resultsData = result.convertToDouble

    let flatMatrix = Array(resultsData.chunked(into: win_length).transposed)
    let flatMatrixReduced = flatMatrix.reduce([], +)

    let nn = vDSP_Length(win_length)
    var win = [Float](repeating: 0, count: Int(nn))
    vDSP_hann_window(&win, nn, Int32(vDSP_HANN_DENORM))
    let flatwindow = win.convertToDouble

    let newMatrixCols = flatwindow.count
    let newMatrixRows = flatMatrix[0].count

    var resultCounter = [Double](repeating: 0.0, count: newMatrixCols*newMatrixRows)
    for index in 0..<resultCounter.count {
        let doub:Double = flatwindow[index/flatMatrix[0].count]
        let res:Double = flatMatrixReduced[index]
        if ((res == 0) || (doub == 0)) {
            resultCounter[index] = 0
        } else {
            resultCounter[index] = res/doub
        }
    }

    let flatMatrixL = Array(resultCounter.chunked(into: win_length).transposed)

    var inverseArray = [Double]()
    let odds = (0...flatMatrixL[0].count).filter { $0 % 2 == 0}
    for number in odds {
        for index in 0..<flatMatrixL.count {
            inverseArray.append(flatMatrixL[index][number])
        }
    }

    var getarr = inverseArray
    for _ in 0..<win_length/2 {
        getarr.removeFirst()
        getarr.removeLast()
    }

    return getarr
}

func mul(_ a:Double,_ b:Double) -> Double { return a*b }

func saveWav(buf: [[Float]]) {
    let url = Bundle.main.url(forResource: "audio_example", withExtension: "wav")!
    let file = try! AVAudioFile(forReading: url)

    let format = AVAudioFormat(commonFormat: .pcmFormatFloat32, sampleRate: 44100, channels: 2, interleaved: false)!
    let pcmBuf = AVAudioPCMBuffer(pcmFormat: format, frameCapacity: AVAudioFrameCount(buf[0].count))
    memcpy(pcmBuf?.floatChannelData?[0], buf[0], 4 * buf[0].count)
    memcpy(pcmBuf?.floatChannelData?[1], buf[1], 4 * buf[1].count)
    pcmBuf?.frameLength = UInt32(buf[0].count)

    let fileManager = FileManager.default
    do {
        let documentDirectory = try fileManager.url(for: .documentDirectory, in: .userDomainMask, appropriateFor:nil, create:false)
        var fileURL = documentDirectory.appendingPathComponent("final")
        try FileManager.default.createDirectory(atPath: fileURL.path, withIntermediateDirectories: true, attributes: nil)
        fileURL = fileURL.appendingPathComponent("out.wav")
        print(fileURL.path)
        let audioFile = try AVAudioFile(forWriting: fileURL, settings: file.fileFormat.settings)
        try audioFile.write(from: pcmBuf!)
    } catch {
        print(error)
    }
}

Extensions
public extension Array {
    func chunked(into size: Int) -> [[Element]] {
        return stride(from: 0, to: count, by: size).map {
            Array(self[$0 ..< Swift.min($0 + size, count)])
        }
    }
}

extension Array where Element == [Double] {

    public var transposed: [[Double]] {
        let matrix = self
        let newMatrixCols = matrix.count
        let newMatrixRows = matrix.first?.count ?? 1

        var results = [Double](repeating: 0.0, count: newMatrixCols*newMatrixRows)

        vDSP_mtransD(matrix.flatMap { $0 }, 1, &results, 1, vDSP_Length(newMatrixRows), vDSP_Length(newMatrixCols))

        return results.chunked(into: newMatrixCols)
    }
}

extension Collection where Iterator.Element == Double {
    var convertToFloat: [Float] {
        return compactMap{ Float($0) }
    }
}

extension Collection where Iterator.Element == Float {
    var convertToDouble: [Double] {
        return compactMap{ Double($0) }
    }
}

The results for Python and Swift are different. Python saves a correct audio file while Swift saves empty audio file with the same length.
My question is, what should I change to make the swift function save the wav audio correctly. Any help is appreciated.

Note: If you would like to run the issue on your machine, I prepared a GitHub repository for this issue to make testing faster and easier:
https://github.com/abokhalel2/istft


title: PyTorch Installation Error - Could not find a version that satisfies the requirement

body:
Hello there! I’ve been trying to install PyTorch, but so far all I got is error messages. I tried the command line
pip install torch==1.8.1+cpu torchvision==0.9.1+cpu torchaudio===0.8.1 -f https://download.pytorch.org/whl/torch_stable.html

that was given on the tutorial, and I get the error:
ERROR: Could not find a version that satisfies the requirement torch==1.8.1+cpu
ERROR: No matching distribution found for torch==1.8.1+cpu

Later I tried to simplify the command to just “pip install torch”, which causes:
ERROR: Command errored out with exit status 1:
ModuleNotFoundError: No module named ‘tools.nnwrap’
ERROR: Failed building wheel for torch
...
Running setup.py clean for torch
ERROR: Command errored out with exit status 1:
ERROR: Failed cleaning build dir for torch

I’d be glad if you could help me, thanks!


title: expected np.ndarray (got DataFrame)

body:
I wanted to find out the output_size of a convolution operation, but I am strugglin with converting my dataframe into a tensor.
output_size = torch.nn.Conv2d(3, 5, 5,stride=1, padding=3,
                dilation=1, groups=1, bias=True, padding_mode='zeros')

fashion = torch.from_numpy(load_fashion)
input_ = torch.Tensor((fashion.values), dtype=torch.float)

output = output_size(input_


I am still grappling with PyTorch, having played with Keras for a while (which feels a lot more intuitive).
Anyway - I have the nn.linear model code below, which works fine for just one input feature, where:
inputDim = 1

I am now trying to expand the same code to include 2 features, and so I have included another column in my feature dataframe and also set:
inputDim = 2

However, when I run the code, I get the dreaded error:
RuntimeError: mat1 dim 1 must match mat2 dim 0

This error references line 63, which is:
    outputs = model(inputs)

I have gone through several other posts here relating to this dimensionality error, but I still can't see what is wrong with my code. Any help would be appreciated.
The full code looks like this:
import numpy as np
import pandas as pd
import torch
from torch.autograd import Variable
import matplotlib.pyplot as plt

device = 'cuda' if torch.cuda.is_available() else 'cpu'

df = pd.read_csv('Adjusted Close - BAC-UBS-WFC.csv')
x = df[['BAC', 'UBS']]
y = df['WFC']

# number_of_features = x.shape[1]
# print(number_of_features)

x_train = np.array(x, dtype=np.float32)
x_train = x_train.reshape(-1, 1)

y_train = np.array(y, dtype=np.float32)
y_train = y_train.reshape(-1, 1)

class linearRegression(torch.nn.Module):
    def __init__(self, inputSize, outputSize):
        super(linearRegression, self).__init__()
        self.linear = torch.nn.Linear(inputSize, outputSize)

    def forward(self, x):
        out = self.linear(x)
        return out

inputDim = 2
outputDim = 1
learningRate = 0.01
epochs = 500

# Model instantiation
torch.manual_seed(42)
model = linearRegression(inputDim, outputDim)
if torch.cuda.is_available(): model.cuda()

criterion = torch.nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=learningRate)

# Model training
loss_series = []
for epoch in range(epochs):
    # Converting inputs and labels to Variable
    inputs = Variable(torch.from_numpy(x_train).cuda())
    labels = Variable(torch.from_numpy(y_train).cuda())

    # Clear gradient buffers because we don't want any gradient from previous epoch to carry forward, dont want to cummulate gradients
    optimizer.zero_grad()

    # get output from the model, given the inputs
    outputs = model(inputs)

    # get loss for the predicted output
    loss = criterion(outputs, labels)
    loss_series.append(loss.item())
    print(loss)
    # get gradients w.r.t to parameters
    loss.backward()

    # update parameters
    optimizer.step()

    print('epoch {}, loss {}'.format(epoch, loss.item()))

# Calculate predictions on training data
with torch.no_grad():  # we don't need gradients in the testing phase
    predicted = model(Variable(torch.from_numpy(x_train).cuda())).cpu().data.numpy()


title: Visualizing self-attention with pytorch

body:
I have a simple model that uses a self-attention mechanism.
The model processes natural language queries.
At the end of my attention calculation, I have my self-attention matrix describing the weights of each word in the input with respect to any other word in the input.
I am looking for a way to create a pretty color map (confusion matrix?) of the attention matrix in order to visualize the results of the attention.
How can I create and plot this map with python?


title: How to change PyTorch sigmoid function to be steeper

body:
My model works when I use torch.sigmoid. I tried to make the sigmoid steeper by creating a new sigmoid function:
def sigmoid(x):
    return 1 / (1 + torch.exp(-1e5*x))

But for some reason the gradient doesn't flow through it (I get NaN). Is there a problem in my function, or is there a way to simply change the PyTorch implementation to be steeper (as my function)?
Code example:
def sigmoid(x):
  return 1 / (1 + torch.exp(-1e5*x))

a = torch.tensor(0.0, requires_grad=True)
b = torch.tensor(0.58, requires_grad=True)

c = sigmoid(a-b)
c.backward()
a.grad
>>> tensor(nan)


title: PyTorch backward() on a tensor element affected by nan in other elements

body:
Consider the following two examples:
x = torch.tensor(1., requires_grad=True)
y = torch.tensor(0., requires_grad=True)
z = torch.full((2, ), float("nan"))
z0 = x * y / y
z1 = x + y
print(z0, z1) # tensor(nan, grad_fn=<DivBackward0>) tensor(1., grad_fn=<AddBackward0>)
z1.backward()
print(x.grad) # tensor(1.)

x = torch.tensor(1., requires_grad=True)
y = torch.tensor(0., requires_grad=True)
z = torch.full((2, ), float("nan"))
z[0] = x * y / y
z[1] = x + y
print(z) # tensor([nan, 1.], grad_fn=<CopySlices>)
z[1].backward()
print(x.grad) # tensor(nan)

In example 1, z0 does not affect z1, and the backward() of z1 executes as expected and x.grad is not nan. However, in example 2, the backward() of z[1] seems to be affected by z[0], and x.grad is nan.
How do I prevent this (example 1 is desired behaviour)? Specifically I need to retain the nan in z[0] so adding epsilon to division does not help.


title: What's the most elegant method to transform models between pytorch and caffe?

body:
Just like title says, models pytorch to caffe, caffe to pytorch.

Not only inference, but also trainable models after transformation.

Any good points?


title: Store weight updates for momemntum

body:
I am trying to implement momentum in my implementation of SGD with momentum.
From my understanding this update look like this:

parameters -= (lr * (p.grad*0.1 + p_delta_prev*0.9))

My question is how I should store my previous deltas from every update

Here is what I have in my update function:

#we now want to do the update with momentum
#momentum takes derivative, multiplies it by 0.1, then takes the previous update,
#multiplies it by 0.9 and we add the two together
#alpha = 0.1, beta = 0.9;  p-=grad*0.1 + p*0.9
def update(x,y,lr):
    wd = 1e-5
    y_hat = model(x)
    # weight decay
    w2 = 0.
    for p in model.parameters(): w2 += (p**2).sum()
    # add to regular loss
    loss = loss_func(y_hat, y) + w2*wd
    loss.backward()
    with torch.no_grad():
        for p in model.parameters():
            #p.grad is the slope of the line of that parameter
            #current_p-previous_p to get difference
            p_update = (lr * (p.grad*0.1 + p*0.9))
            p.sub_(p_update)
            p.grad.zero_()
    return loss.item()

Here the p*0.9 should be replace by the p_delta_prev. But how should I store these deltas for every parameter? If I save them to a tensor wouldn't I would be effectively copying the weight deltas to memory making my model two times the size. What would be a good way to accomplish this? I do not want to use an inbuilt function that does that activation for me. I did look into the pytorch sgd.py and it looks like the store the states.

I have updated the code:

#we now want to do the update with momentum
#momentum takes derivative, multiplys it by 0.1, then takes the previous update,
#multiplies it by 0.9 and we add the two together
#alpha = 0.1, beta = 0.9;  p-=grad*0.1 + p*0.9
p_delta = {}
def update(x,y,lr):
    wd = 1e-5
    y_hat = model(x)
    # weight decay
    w2 = 0.
    for p in model.parameters(): w2 += (p**2).sum()
    # add to regular loss
    loss = loss_func(y_hat, y) + w2*wd
    loss.backward()
    with torch.no_grad():
        i = 0
        for p in model.parameters():
            #p.grad is the slope of the line of that parameter
            if i not in p_delta:#check if key exists
                p_delta[i] = torch.zeros_like(p)
            p_update = (lr *p.grad) + (p_delta[i]*0.9)
            p_delta[i] = p_update.clone()
            p.sub_(p_update)
            p.grad.zero_()
            print((p_delta[i]))
            i+=1
    return loss.item()

I think the code in the Excel spreadsheet is incorrect. Jeremy seems to show: lr* ((p.grad*0.1) + (p_delta[i]*0.9)) but many tutorials seem to show: (lr *p.grad) + (p_delta[i]*0.9) If we implement Jeremy’s code the loss actually is slower than vanilla GD. The part of the video is here: https://youtu.be/CJKnDu2dxOE?t=6581


title: dll in pytorch failed

body:
I'm using Windows 10 and pytorch 1 I've used pytorch successfully, however, after I loaded matplotlib the following error occurred:
ImportError: DLL load failed: The specified module could not be found. for torch._C.
This error  persist even though I have uninstall first matplotlib, then pytorch and re-installed pytorch. I have even created a new environment and installed pytorch into it. None of the posts I have seen helped me resolve the problem. It would appear that something has happened to the global environment.


title: Why is an OOM happening on my model init()?

body:
A single line in my model, tr.nn.Linear(hw_flat * num_filters*8, num_fc), is causing an OOM error on initialization of the model. Commenting it out removes the memory issue.

import torch as tr
from layers import Conv2dSame, Flatten

class Discriminator(tr.nn.Module):
    def __init__(self, cfg):
        super(Discriminator, self).__init__()
        num_filters = 64
        hw_flat = int(cfg.hr_resolution[0] / 2**4)**2
        num_fc = 1024

        self.model = tr.nn.Sequential(
            # Channels in, channels out, filter size, stride, padding
            Conv2dSame(cfg.num_channels, num_filters, 3),
            tr.nn.LeakyReLU(),
            Conv2dSame(num_filters, num_filters, 3, 2),
            tr.nn.BatchNorm2d(num_filters),
            tr.nn.LeakyReLU(),
            Conv2dSame(num_filters, num_filters*2, 3),
            tr.nn.BatchNorm2d(num_filters*2),
            tr.nn.LeakyReLU(),
            Conv2dSame(num_filters*2, num_filters*2, 3, 2),
            tr.nn.BatchNorm2d(num_filters*2),
            tr.nn.LeakyReLU(),
            Conv2dSame(num_filters*2, num_filters*4, 3),
            tr.nn.BatchNorm2d(num_filters*4),
            tr.nn.LeakyReLU(),
            Conv2dSame(num_filters*4, num_filters*4, 3, 2),
            tr.nn.BatchNorm2d(num_filters*4),
            tr.nn.LeakyReLU(),
            Conv2dSame(num_filters*4, num_filters*8, 3),
            tr.nn.BatchNorm2d(num_filters*8),
            tr.nn.LeakyReLU(),
            Conv2dSame(num_filters*8, num_filters*8, 3, 2),
            tr.nn.BatchNorm2d(num_filters*8),
            tr.nn.LeakyReLU(),
            Flatten(),
            tr.nn.Linear(hw_flat * num_filters*8, num_fc),
            tr.nn.LeakyReLU(),
            tr.nn.Linear(num_fc, 1),
            tr.nn.Sigmoid()
        )
        self.model.apply(self.init_weights)

    def forward(self, x_in):
        x_out = self.model(x_in)
        return x_out

    def init_weights(self, layer):
        if type(layer) in [tr.nn.Conv2d, tr.nn.Linear]:
            tr.nn.init.xavier_uniform_(layer.weight)

This is strange, as hw_flat = 96*96 = 9216, and num_filters*8 = 512, so hw_flat * num_filters = 4718592, which is the number of parameters in that layer. I have confirmed this calculation as changing the layer to tr.nn.Linear(4718592, num_fc) results in the same output.

To me this makes no sense as dtype=float32, so the expected size of this would be 32*4718592 = 150,994,944 bytes. This is equivalent to about 150mb.

Error message is:

Traceback (most recent call last):
  File "main.py", line 116, in <module>
    main()
  File "main.py", line 112, in main
    srgan = SRGAN(cfg)
  File "main.py", line 25, in __init__
    self.discriminator = Discriminator(cfg).to(device)
  File "/home/jpatts/Documents/ECE/ECE471-SRGAN/models.py", line 87, in __init__
    tr.nn.Linear(hw_flat * num_filters*8, num_fc),
  File "/home/jpatts/.local/lib/python3.6/site-packages/torch/nn/modules/linear.py", line 51, in __init__
    self.weight = Parameter(torch.Tensor(out_features, in_features))
RuntimeError: $ Torch: not enough memory: you tried to allocate 18GB. Buy new RAM! at /pytorch/aten/src/TH/THGeneral.cpp:201

I am only running batch sizes of 1 as well (not that that affects this error), with overall input shape to the network being (1, 3, 1536, 1536), and shape after flatten layer being (1, 4718592).

Why is this happening?


title: Python __getattr__ called and enter infinite recursion even the attribute is not missing

body:
I define a NE_Conv2d class and override its __getattr__. When I try to get an existing attribute from one instance of this class, __getattr__ is called and enter infinite recursion.

I know one need  to deal with __getattr__ carefully to avoid infinite recursion, but apparently the conv attribute already exists after __init__ is called. So when I try to get conv attribute, __getattr__ should not be called.

from torch import nn
class NE_Conv2d(nn.Module):
    '''Nonexpansive conv2d'''

    def __init__(self, *k, **kw):
        super(NE_Conv2d, self).__init__()
        self.conv = nn.Conv2d(*k, **kw)
        print('foo')

    def __getattr__(self, attr):
        return getattr(self.conv, attr)

a = NE_Conv2d(3, 32, 5)
print(a)
print(a.conv)

Above code should print info about a and a.conv, but enter infinite recursion when trying to get a.conv.


title: Why Tensor must be Integer in Pytorch?

body:
I want to get my accuracy. So I divide the correct number(tensor) by the total number(tensor). Then I get a integrate instead of a float. I wonder why. The code is as follows. Also, if I transfer tensor to numpy, why I cannot get an exact number? I get description like "built-in method numpy of Tensor object at 0x00000296AEDC1168" I still do not know the exact number.

I tried tensor.numpy

print(correct)
tensor(9668)

print(total)
10000

print(correct/total)
tensor(0)

print(100 * correct/total)
tensor(96)

print((100 * correct/total).numpy)
<built-in method numpy of Tensor object at 0x00000296AEDC1168>


title: System memory usage increase after using GPU?

body:
When I use pytorch/tensorflow to build my neural network, I find that system memory usage increase when I use GPU.

The example code is pretty simple using pytorch, a small Conv1d neural network.

When I set use_gpu=False, the system memory usage is just 131M, . But when I set use_gpu=True, system memory usage increases to more than 2G.

As far as I know, when using gpu, many parameters will be pinned into GPU'S memory, that means the system memory usage will decrease.

Could anyone help to explain the situation ?


title: Right place to do class imbalance regularisation (data level or batch level)

body:
I have two binary imbalanced dataset where the labels are ether 0 or 1 and prediction output is between 0 and 1. The positive case has 10000 samples, while the negative case has 90000 samples. I'm using a batch of 100 when training.

when calculating the BinaryCrossEntropyLoss (in pytorch) its possible to supply the per batch element regularisation weight.

My question is:
To calculate the general class weight dose it make more sense to calculate it 1 time at the start (so 1/(10000/(100000) for the positive case) and scale the loss of each sample with this value

or:

Calculate the weight at the batch level, by firstly finding the batch class imbalance (e.g in the batch it might be 25 positives and 75 negatives, hence 1/(25/(25+75) for the positive case)

I'm asking this because the loss is averaged across the batch


title: How to transform network's output images to colored pictures with target is PIL's P mode?

body:
The original value in target images is 20 and 16 at training process with P mode of PIL,so I transform 20 to 1 and 16 to 2 in order to train the segmentation task.
But when I want to get the output images,the pictures aren't colored although I used the code

        pred=pred.reshape([512,512]).astype('uint8')
        (x, y) = pred.shape
        for xx in range(x):
            for yy in range(y):
                if pred[xx, yy] == 2:
                    pred[xx, yy] = 16
                elif pred[xx, yy] == 1:
                    pred[xx, yy] = 20
        pp = Image.fromarray(pred).convert('P')
        pp.save(r'E:\python_workspace\0711\run\pascal\{}.png'.format(i))

But the output image is

I have see the value with PIL.open and transform it to numpy to see the values,the part of things is convert to 16 and 20,the mode is P too.
How can I deal with this problem?


title: How I make pytorch read the numpy format?

body:
I'm trying to create a code with PyTorch and Keras that uses the BERT algorithm to detect fake news, but I got an error tells me:

can't convert np.ndarray of type numpy.bool_.
The only supported types are: double, float, float16, int64, int32, and uint8.

Please access the code on my Google Codelab. The error can be seen in the last cell.

The only requirement for running it is downloading a CSV file for the training process.


title: Loss for binary sparsity

body:
I have binary images (as the one below) at the output of my net. I need the '1's to be further from each other (not connected), so that they would form a sparse binary image (without white blobs). Something like salt-and-pepper noise. I am looking for a way to define a loss (in pytorch) that would punish based on the density of the '1's.

Thanks.

I


title: Pycharm/Pytorch: finding usages of nn.Module.forward

body:
Then you click no, and it only finds explicit mentions of .forward in the code, rather than model(x) calls. If you click yes, it finds explicit mentions of forward in the pytorch source. Is this fixable? Is there another IDE that supports this?

Pycharm Version:

PyCharm 2019.3.3 (Professional Edition)
Runtime version: 11.0.5+10-b520.38 x86_64
VM: OpenJDK 64-Bit Server VM by JetBrains s.r.o
macOS 10.15.1
GC: ParNew, ConcurrentMarkSweep
Memory: 725M
Cores: 12
Registry:
Non-Bundled Plugins: Key promoter, mobi.hsz.idea.latex, name.kropp.intellij.makefile, net.seesharpsoft.intellij.plugins.csv


title: How to debug if weight keep increasing. Pytorch program

body:
I m having some doubt when practicing Pytorch program.

I have function like y = m1x1 + m2x2 + c (just 2 weights to learn here). The expected values of weight should be 16,-14 and bias should be 36. But in every epoch the learned wight goes very big. Can any one help me to debug and understand this 20 lines of code, what going wrong here.

import torch

x = torch.randint(size = (1,2), high = 10)
w = torch.Tensor([16,-14])
b = 36
#Compute Ground Truth
y = w * x + b

#Find weights by program
epoch = 20
learning_rate = 30

#initialize random
w1 = torch.rand(size= (1,2), requires_grad= True)
b1 = torch.ones(size = [1], requires_grad= True)

for i in range(epoch):
    y1 = w1 * x + b1

    #loss function RMSQ
    loss = torch.sum((y1-y)**2)

    #Find gradient
    loss.backward()

    with torch.no_grad():
        #update parameters
        w1 -= (learning_rate * w1.grad)
        b1 -= (learning_rate * b1.grad)

        w1.grad.zero_()
        b1.grad.zero_()

    print("B ", b1)
    print("W ", w1)

Thanks,
  Ganesh


title: How can i fix or freeze the subset of convolution filter in Pytorch?

body:
for batch_idx, (inputs, targets) in enumerate(testloader):
    optimizer.zero_grad()

    inputs = inputs.float().cuda()
    inputs, targets = inputs.to(device), targets.to(device)

    outputs = im_net(inputs)

    ce_loss = criterion(outputs, targets)

    loss = criterion(outputs, targets)

    loss.backward()
    im_net.features[25].weight.grad[temp_5.values] = 0

    print(im_net.features[25].weight.grad[temp_5.values])

    optimizer.step()

Above is my code for trying to fix the subset (temp_5.values) of 25th convolution layer's filter. I try to set the subset filter's gradient as 0 so not to update.
When I try to print the gradient, they are all zero, but the weight has been changed.

So two questions:

I wonder why the zero gradient makes the weight change;
Is there other solution to fix the subset convolution filter?


title: Wrap two tensors in pytorch to get size of new tensor as 2

body:
I have two tensors say x and y:

x has shape: [21314, 3, 128, 128]
y has shape: [21314]

Can I get new tensor of shape : [ [21314, 3, 128, 128], [21314] ], basically of shape 2


title: Implement x=T if abs(x)>T as an activation function in pytorch

body:
I would like to implement the following activation function in pytorch:

x = T if abs(x)>T else x

I could do something close with torch.clamp(min=-T, max=T) but it's not exactly the behavior I want (this would behave the same as above for x>-T but would return -T for x<-T). Is there any torch function that could help me achieve this?


title: How should I sketch the architecture of a neural network?

body:
I'm reading One Shot Learning with Siamese Networks in PyTorch by Harshvardhan Gupta.

I'm trying to figure out the size and dimensions of the network because I want to sketch out the network architecture. How would one draw an architectural diagram of this network?

His neural network class can be found below:

class SiameseNetwork(nn.Module):
    def __init__(self):
        super(SiameseNetwork, self).__init__()
        self.cnn1 = nn.Sequential(
            nn.ReflectionPad2d(1),
            nn.Conv2d(1, 4, kernel_size=3),
            nn.ReLU(inplace=True),
            nn.BatchNorm2d(4),

            nn.ReflectionPad2d(1),
            nn.Conv2d(4, 8, kernel_size=3),
            nn.ReLU(inplace=True),
            nn.BatchNorm2d(8),

            nn.ReflectionPad2d(1),
            nn.Conv2d(8, 8, kernel_size=3),
            nn.ReLU(inplace=True),
            nn.BatchNorm2d(8),

        )

        self.fc1 = nn.Sequential(
            nn.Linear(8*100*100, 500),
            nn.ReLU(inplace=True),

            nn.Linear(500, 500),
            nn.ReLU(inplace=True),

            nn.Linear(500, 5))

    def forward_once(self, x):
        output = self.cnn1(x)
        output = output.view(output.size()[0], -1)
        output = self.fc1(output)
        return output

    def forward(self, input1, input2):
        output1 = self.forward_once(input1)
        output2 = self.forward_once(input2)
        return output1, output2

To my understanding so far, his network has a depth of 7 layers; 3 convolution layers, 3 fully-connected layers, and the output layer. I'm not sure what qualifies as a node in each layer.


title: Non-intersection of two n-dimentional pytorch tensors

body:
Thanks everyone in advance for your help! What I'm trying to do in PyTorch is to compute non-intersection ( lets call it torch.nonintersection) for tensors of many dimenstions (without for loops because I want it to be effectively performed on GPU). So here is the example how it should work:
a = torch.tensor([[ 0.,  0.], [ 0.,  1.], [ 0.,  2.], [ 1.,  0.], [ 1.,  1.], [ 1.,  2.], [ 1.,  3.],
                  [ 2.,  0.], [ 2.,  1.], [ 2.,  2.]])
b = torch.tensor([[ 2.,  0.], [ 2.,  1.], [ 2.,  2.], [ 1.,  0.], [ 1.,  1.], [ 1.,  2.], [ 1.,  3.]])

torch.spec_unique(a,b) = torch.tensor([ 0.,  0.], [ 0.,  1.], [ 0.,  2.])

I have analogs with for loops, but they take too much time right now. Any Ideas how it can be done? Much appreciated!


title: Workaround for index_add / masked_scatter pytorch operators

body:
I’m currently trying to export a model to ONNX and run it using an external library that parses ONNX. Unfortunately my model uses index_add and masked_scatter_ and the external library can’t parse the ONNX operations these two are transformed into (I believe index_add doesn’t even have an ONNX equivalent for opset 11 which is what I’m using).
Is there any way I can rewrite these operations into “simpler” ONNX compatible operations, without a major dip in performance (i. e. without having to write explicit for loops in python)? I have been trying for a while but I have not been able to find any equivalent way to express these two.


title: Pytorch image segmentation transfer learning

body: I am new in Pytorch. My question is: How do I apply transfer learning to a custom dataset? I am doing image segmentation on brain tumors. I can find examples which use U-net structure but I could not find examples using weights of the pre-trained models for a U-net image segmentation?


title: Why I can't use pytoch on spyder?

body:
I am using Windows 10 OS, Python Version - 3.8
Installed Method -  conda install pytorch torchvision cudatoolkit=10.2 -c pytorch
It was successfully installed, when trying to run torch.cuda.is_available() throws up an error as -
AttributeError: partially initialized module 'torch' has no attribute 'cuda' (most likely due to a circular import)

I faced this error other importing such as -torch.nn as nn-
I checked the package list (!pip list ) it is present
Why I can't use pytoch on spyder ? How to tackle this problem ?


title: Gradient Checkpointing returning values

body:
I have a checkpoint callback function (i.e, custom_dec) that returns a Tensor, and a dictionary. But it seems like this function does not return dictionaries (or other data types), but only tensors. What is the workaround to this, as the module that I want to checkpoint is returning a tensor, plus a data type as dictionary:
def custom_dec(self, module):
        def custom_forward(*inputs):
            output = module(inputs[0], inputs[1],
                            encoder_attn_mask=inputs[2],
                            decoder_padding_mask=inputs[3],
                            layer_state=inputs[4],
                            causal_mask=inputs[5],
                            output_attentions=inputs[6],
                            )
            # output[2] is a python dictionary
            return output[0], output[2]

The following is the checkpoint call:
x, layer_past = \
                checkpoint.checkpoint(
                    self.custom_dec(decoder_layer),
                    x,
                    encoder_hidden_states,
                    encoder_padding_mask,
                    decoder_padding_mask,
                    layer_state,
                    decoder_causal_mask,
                    output_attentions,
                )

The error:

TypeError: CheckpointFunctionBackward.forward: expected Variable (got
dictionary) for return value 1


title: How to update AlexNet to use FCN? Confused on upsampling

body:
The main issue is I don't understand how the upsampling works.  The AlexNet architecture has the following classifier
Dropout(),
Linear(in_features=9216, out_features=4096),
ReLU(),
Dropout(),
Linear(in_features=4096, out_features=4096),
ReLU(),
Linear(in_features=4096, out_features=1000)

The above is used on cifar so the num classes is 1000.  If we change the classifier to use FCN (following FCN for Semantic Segmentation) and use VOC we'll have 20 classes (why does the figure show 21 channels?). Images are resized to 256x256.  The architecture from the paper is shown below

Using pytorch I have the following
class AlexNetFCN(nn.Module):

    def __init__(self):
        super().__init__()
        self.features = AlexNet().features
        self.classifier = nn.Sequential(
                nn.Dropout(),
                nn.Conv2d(256, 4096, kernel_size=6),
                nn.ReLU(),
                nn.Dropout(),
                nn.Conv2d(4096, 4096, kernel_size=1),
                nn.ReLU(),
                nn.Conv2d(4096, 20, kernel_size=1),
                )

    def forward(self, x):
        x = self.features(x)
        x = self.classifier(x)
        return x

I know after the feature portion my images are 7x7.  How do i upsample back up to 256x256?  Maybe nn.Bilinear but this isn't clear to me from the paper.


title: pytorch RNN parameter, loss function

body:
I'm a beginner at Pytorch. I want to apply optimization problem in geophysics to RNN.
I didn't use nn.RNN or nn.LSTM because the relation of output and trainable parameter is defined by physical formula.
This is the code with nn.Module. In the forward function, u3(output) is obtained previous wavefield u1, u2 and the trainable parameter vp_inv is involved.
class WaveCell(nn.Module):
def __init__(self, nnz, dz, dt):
    super(WaveCell, self).__init__()

    self.nnz = nnz
    self.dz = dz
    self.dt = dt

    nz=self.nnz
    npz=self.nnz+4

    vp_ini = np.zeros((npz), dtype=np.float32)
    # vp_ini[0:30]=2.
    min_vp = 2.
    max_vp = 5.

    vp_ini[0:30] = 2.
    for iz in range(30, 170):
        vp_ini[iz] = min_vp + (max_vp - min_vp) * float(iz) / (nz + 1)
    vp_ini[170:npz] = 5.

    vp_ini = gaussian_filter1d(vp_ini, 5, mode='nearest')

    self.vp_inv = Parameter(torch.tensor(data=vp_ini, dtype=torch.float32))
    # self.vp_inv = Variable(vp_ini,requires_grad=True)

def forward(self, u1, u2, src):
    nnz = self.nnz
    dz = self.dz
    dt = self.dt

    nz = nnz
    vp_in=self.vp_inv.clamp(2.,5.)
    vp_in[nz+1:]=5.

    npz = nnz + 4

    srcf = torch.zeros((npz))
    srcf[1] = 1.

    u3 = torch.zeros((npz))
    u3[nnz] = -vp_in[nz] * dt * (u2[nnz] - u2[nnz - 1]) / dz + u2[nnz]

    u3[1:nnz] = 2. * u2[1:nnz] - u1[1:nnz] + (dt) ** 2 * \
                (vp_in[1:nnz]**2*(u2[2:nnz + 1] - 2. * u2[1:nnz] + u2[0:nnz - 1]) / dz ** 2 + srcf[1:nnz] * src)

    return u2, u3, u3[1]

and this code is the process of optimization. Every time step I calculate loss value and backward after time loop. My question is
Q1: Does it work properly as RNN?
Q2: I want to update vp_inv directly by defining Parameter. But, I'm not sure it is updated well.
rnn = WaveCell(nnz, dz, dt)
epochs=1000
lr=100.
loss_func = nn.MSELoss()

optimizer = torch.optim.Adam(rnn.parameters(), lr=lr)

p1= torch.zeros(npz)
p2= torch.zeros(npz)
p3= torch.zeros(npz)

srcf = torch.zeros((npz))

p0= torch.zeros(1)

for i in range(epochs):
    rnn.zero_grad()
    total_loss = 0

    p1 = torch.zeros(npz)
    p2 = torch.zeros(npz)
    p3 = torch.zeros(npz)

    outs=[]

    for it in range(1,nt):
        target = true_rec[it]

        p3 = torch.zeros((npz))
        srcf = torch.zeros((npz))
        p0 = torch.zeros(1)

        p1, p2, p0 = rnn(p1, p2, source[it])
        loss = loss_func(p0, target)

        total_loss += loss
        outs.append(p0)

    total_loss.backward(retain_graph=True)

    optimizer.step()


title: Why does my autoencoder for de-noising in PyTorch learns to zero out everything?

body:
I need to build a de-noising autoencoder using pytorch for the task of cleaning out signals.
For example, I can take the cosine function and sample it in intervals (where i have two parameters - B and K. B is the number of intervals I take in each example and K is how many sampling points (equally distanced) are in each interval) so for instance, I can take B = 5 intervals and measure K = 8 points in each interval. Hence the distance between each point is 2pi / 8 and i have a total of 40 points. The number of functions I try to generalize is L and I treat this like different channels. Then I add a random starting position for each example (to make it slightly different) then random noise and send it to the autoencoder to train.
The thing is, no matter what architecture or learning rate, gradually it learns to output nothing but zeros. The autoencoder is super simple so I dont reckon there's a problem with it but rather a problem of how I generate my data.
I attach both of the codes anyways:
class ConvAutoencoder(nn.Module):
def __init__(self, enc_channels, dec_channels):
    super(ConvAutoencoder, self).__init__()
    ## encoder layers ##

    encoder_layers = []
    decoder_layers = []

    in_channels = enc_channels[0]

    for i in range(1, len(enc_channels)):
        out_channels = enc_channels[i]

        encoder_layers += [nn.ConvTranspose2d(in_channels, out_channels, kernel_size=1, bias=True),
                           nn.ReLU()]
        in_channels = out_channels

    in_channels = dec_channels[0]

    for i in range(1, len(dec_channels)):
        out_channels = dec_channels[i]

        decoder_layers += [nn.ConvTranspose2d(in_channels, out_channels, kernel_size=1, bias=True),
                           nn.ReLU()]
        in_channels = out_channels

    self.encoder = nn.Sequential(*encoder_layers)
    self.decoder = nn.Sequential(*decoder_layers)

def forward(self, x):

    if len(x.shape) == 3:
        x = x.unsqueeze(dim=-1)

    res = self.decoder(self.encoder(x)).squeeze(-1)

    return res

And the data generation is as follows:
def generate_data(batch_size: int, intervals: int, sample_length: int, channels_functions, noise_scale=1)->torch.tensor:
    channels = len(channels_functions)
    mul_term = 2 * np.pi / sample_length # each sample is 2pi and equally distance

    # each example is K * B long
    positions = np.arange(0, sample_length * intervals)
    x = positions * mul_term

    # creating random start points (from negative to positive)
    random_starting_pos = (np.random.rand(batch_size) - 0.5) * 10000
    start_pos_mat = np.tile(random_starting_pos , (sample_length * intervals, 1))
    start_pos_mat = np.tile(start_pos_mat , (channels, 1)).T
    start_pos_mat = np.reshape(start_pos_mat , (batch_size, channels, sample_length * intervals))

    X = np.tile(x, (channels, 1))
    X = np.repeat(X[np.newaxis, :, :], batch_size, axis=0)

    X += start_pos_mat #adding the random starting position

    # apply each function to a different channel
    for i, function in enumerate(channels_functions):
        X[:, i, :] = function(X[:, i, :])

    clean = X
    noise = np.random.normal(scale=noise_scale, size=clean.shape)
    noisy = clean + noise

    # normalizing each sample
    row_sums = np.linalg.norm(clean, axis=2)
    clean = clean / row_sums[:, :, np.newaxis]
    row_sums = np.linalg.norm(noisy, axis=2)
    noisy = noisy / row_sums[:, :, np.newaxis]

    clean = torch.from_numpy(clean)
    noisy = torch.from_numpy(noisy)

    return clean, noisy

Edit - Added the entire training loop:
if __name__ == '__main__':

func_list = [lambda x: np.cos(x),
             lambda x: np.cos((x**4) / 10),
             lambda x: np.sin(x**3 * np.cos(x**2)),
             lambda x: 0.25*np.cos(x**2) - 10*np.sin(0.25*x)]

L = len(func_list)
K = 3
B = 4

enc_channels = [L, 64, 128, 256]

num_epochs = 100

model = models.ConvAutoencoder(enc_channels, enc_channels[::-1])

criterion = torch.nn.BCELoss()

optimizer = torch.optim.Adam(model.parameters(), lr=0.005, weight_decay=1e-5)

for epoch in range(num_epochs):
    clean, noisy = util.generate_data(128, K, B, func_list)

    # ===================forward=====================
    output = model(noisy.float())
    loss = criterion(output.float(), clean.float())
    # ===================backward====================
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    # ===================log========================
    print('epoch [{}/{}], loss:{:.4f}'.format(epoch + 1, num_epochs, loss.data))

    if epoch % 10 == 0:
        show_clean, show_noisy = util.generate_data(1, K, B, func_list)
        print("clean\n{}".format(show_clean))
        print("noisy\n{}".format(show_noisy))
        print("denoised\n{}".format(model(show_noisy.float())))

sure enough after like 10 epochs the model outputs:
clean vector
tensor([[[ 0.3611, -0.1905, -0.3611,  0.1905,  0.3611, -0.1905, -0.3611,
           0.1905,  0.3611, -0.1905, -0.3611,  0.1905],
         [ 0.3387, -0.0575, -0.2506, -0.3531, -0.3035,  0.3451,  0.3537,
          -0.2416,  0.2652, -0.3126, -0.3203, -0.1707],
         [-0.0369,  0.4412, -0.1323,  0.1802, -0.2943,  0.3590,  0.4549,
           0.0827, -0.0164,  0.4350, -0.1413, -0.3395],
         [ 0.3997,  0.3516,  0.2451,  0.1136, -0.0458, -0.1944, -0.3225,
          -0.3925, -0.3971, -0.3382, -0.2457, -0.1153]]], dtype=torch.float64)
noisy vector
tensor([[[-0.1071, -0.0671,  0.0993, -0.2029,  0.1587, -0.4407, -0.0867,
          -0.2598,  0.2426, -0.6939, -0.3011, -0.0870],
         [ 0.0889, -0.3415, -0.1434, -0.2393, -0.4708,  0.0144,  0.2620,
          -0.1186,  0.6424,  0.0886, -0.2192, -0.1562],
         [ 0.1989,  0.2794,  0.0848, -0.2729, -0.2168,  0.1475,  0.5294,
           0.4788,  0.1754,  0.2333, -0.0549, -0.3665],
         [ 0.3611,  0.3535,  0.1957,  0.1980, -0.1115, -0.1912, -0.2713,
          -0.4087, -0.3669, -0.3675, -0.2991, -0.1356]]], dtype=torch.float64)
denoised vector
tensor([[[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
         [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
         [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
         [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]]], grad_fn=<SqueezeBackward1>)

Thanks


title: AzureML SDK not working with PyTorch 1.5?

body:
Has anyone got PyTorch 1.5 to work with the AzureML SDK (versions 1.11 and 1.12)? torch.cuda.is_available() returns False even on GPU-enabled machines. Exactly the same setup works fine (is_available() is True) with PyTorch 1.3, 1.4 and 1.6. Any pointers welcome. These are the (possibly) relevant parts of my Conda environment file, with the values of pytorch and azureml-sdk varied as required.
channels:
  - defaults
  - pytorch
dependencies:
  - python=3.7.3
  - pytorch=1.5.0
  - pip:
      - azureml-sdk==1.12.0

Thanks


title: How to get absolute depth map from estimated depth map

body:
I have a list of images with a list of random N points (for each image) which also contains their corresponding depth values.
I'm using MidasV2 model in order to estimate the relative depth map of the image. (https://pytorch.org/hub/intelisl_midas_v2/)
After I got the result, I'm trying to understand how I can combine the output of the model (relative depth map) and the list w/ the points and their corresponding depth values in order to output an absolute depth map for that image?
AFAIK: ratio between medians of ground-truth (which are the points and their correponding depth values) and current prediction is
employed to acquire absolute depth - if so, how can I quantify the error? should it just be just a sum of all the errors (difference between the points in relative depth map values and the n points of the image?) or should I define it somehow else?
And on another note, are there any other methods to get the absolute depth map w/o using the output of the relative depth map from MidasV2?


title: Target dimension mismatch

body:
I am trying to learn deep learning for a school project. I used ESNet as the network architecture and as of the moment, I am using a simple dataset (cats vs dogs) just to test how the architecture performs. I first saved my dataset in a separate file and later loaded it to my model. Since my dataset is a NumPy array, I converted it to tensors and then resized it. When trying to train, I got the following error:
RuntimeError: only batches of spatial targets supported (3D tensors) but got targets of dimension: 2
Based on the suggestions I found online, the error points out to my target not having the right shape. My input has the shape [batch_size, 3, 100, 100].
Based on this shape the output would have the shape [batch_size, 2, 104, 104] and thus the target should have the shape [batch_size, 104, 104] and contain values in [0, 1]. Yet, whenever i try to train in and try to print what shape my target's shape I got [batch_size, 2].
and this is my training code:
BATCH_SIZE = 10
EPOCHS = 1

for epoch in range(EPOCHS):
    for i in tqdm(range(0, len(train_x), BATCH_SIZE)):
        batch_x = train_x[i:i+BATCH_SIZE].view(-1, 1, 100, 100)
        batch_y = train_y[i:i+BATCH_SIZE]

        print(batch_y.shape)

        model.zero_grad()

        outputs = model(batch_x)
        print(outputs.shape)
        loss = loss_function(outputs, batch_y)
        loss.backward()
        optimizer.step()    # Does the update

    print(f"Epoch: {epoch}. Loss: {loss}")

I've tried every suggestion I found online but found no success. Can anyone help me out? Thank you so much.


title: Mini batches with DataLoader and a 3D input. (Pytorch)

body:
I have been struggling to manage and create batches for a 3D tensor. I have used it before as a way to create batches for 1D tensor. However, in my current research, I need to create batches out of a tensor with shape (1024,1024,2).
I created custom data to use as my input for the DataLoader method in pytorch. I created the following for the 1D array:
class CustomDataset(Dataset):
def __init__(self, x_tensor, y_tensor):
    self.xdomain = x_tensor
    self.ydomain = y_tensor

def __getitem__(self, index):
    return (self.xdomain[index], self.ydomain[index])

def __len__(self):
    return len(self.xdomain)

It works pretty well, however, I realized that this doesn’t work for tensors x_tensor and y_tensor of shape (1024,1024,2) and (1024,1024,1) respectively. I understand that I have to change the __ getitem __ and __ len __ function in a way so it can divide the tensors into batches.
I tried many things, but one I know it could work is that I could flatten these tensors into shapes (1024 x1024,2) and (1024x1024,1). However, I would have to not only change my NN definition but must of my code.
So I want to keep it as is and try to understand how to create these functions if possible. What I understand of these functions are:
__len__ so that len(dataset) returns the size of the dataset.
__getitem__ to support the indexing such that dataset[i] can be used to get ith sample.
With this knowledge, I created this class, that finds the indexes of the first 2 dimensions(to find the ith sample). However, this created the input of the NN to be (1024x1024,2) and output (1024x1024,1). And I want it to be (1024,1024,2) and (1024,1024,1).
If someone with a better understanding of Data Loader and mini-batches could explain what am I missing, that could be amazing. An first of all is this possible?
Thanks for reading this, sorry if this question is too basic. I hope is clear.


title: how to Implement replication_pad2d layer in to coremltools converters as torch_op

body:
I've been trying to convert my pytorch model into coreML format, However one of the layers is currently not supported replication_pad2d. Therefore I was trying to implement it using the register operator decorator @register_torch_op  to reimplement the layer for coremltools.converters, However I'm struggling to understand the input types to be able to implement the function currently. I got this, which is an implementation roughly translated from pytorch but it's not working
from coremltools.converters.mil import Builder as mb
from coremltools.converters.mil import register_torch_op
from coremltools.converters.mil.frontend.torch.ops import _get_inputs

@register_torch_op
def replication_pad2d(context, node):
  inputs = _get_inputs(context, node)
  x = inputs[0]
  a = len(x)
  L_list, R_list = [], []
  U_list, D_list = [], []
  for i in range(a):#i:0, 1
    l = x[:, :, :, (a-i):(a-i+1)]
    L_list.append(l)
    r = x[:, :, :, (i-a-1):(i-a)]
    R_list.append(r)
  L_list.append(x)
  x = mb.concat(L_list+R_list[::-1], axis=3, name=node.name)
  for i in range(a):
    u = x[:, :, (a-i):(a-i+1), :]
    U_list.append(u)
    d = x[:, :, (i-a-1):(i-a), :]
    D_list.append(d)
  U_list.append(x)
  x = mb.concat(U_list+D_list[::-1], axis=3, name=node.name)
  context.add(x)

but getting the following error
<ipython-input-12-cf14ed84cb93> in replication_pad2d(context, node)
     59   inputs = _get_inputs(context, node)
     60   x = inputs[0]
---> 61   a = len(x)
     62   L_list, R_list = [], []
     63   U_list, D_list = [], []

TypeError: object of type 'Var' has no len()

would be great if someone could help me understand this better especially input type node and context


title: How to create a stacked-RNN model in Pytorch?

body:
I have a template of LSTM in Pytorch:
# RNN structure parameters
input_dim = 1
hidden_dim = 32
num_layers = 2
output_dim = 1
num_epochs = 50

class LSTM(nn.Module):
    def __init__(self, input_dim, hidden_dim, num_layers, output_dim):
        super(LSTM, self).__init__()
        self.hidden_dim = hidden_dim
        self.num_layers = num_layers

        self.lstm = nn.LSTM(input_dim, hidden_dim, num_layers, batch_first=True)
        self.fc = nn.Linear(hidden_dim, output_dim)

    def forward(self, x):
        h0 = torch.zeros(self.num_layers, x.size(0), self.hidden_dim).requires_grad_()
        c0 = torch.zeros(self.num_layers, x.size(0), self.hidden_dim).requires_grad_()
        out, (hn, cn) = self.lstm(x, (h0.detach(), c0.detach()))
        out = self.fc(out[:, -1, :])
        return out

Now I want to stack a GRU layer on it. So I modified the template to
class LG(nn.Module):
    def __init__(self, input_dim, hidden_dim, num_layers, output_dim):
        super(LG, self).__init__()
        self.hidden_dim = hidden_dim
        self.num_layers = num_layers

        self.lstm = nn.LSTM(input_dim, hidden_dim, num_layers, batch_first=True)
        self.gru = nn.GRU(hidden_dim, hidden_dim, 1, batch_first=True) # Modification 1: Add a GRU layer
        self.fc = nn.Linear(hidden_dim, output_dim)

    def forward(self, x):
        h0 = torch.zeros(self.num_layers, x.size(0), self.hidden_dim).requires_grad_()
        c0 = torch.zeros(self.num_layers, x.size(0), self.hidden_dim).requires_grad_()
        out, (hn, cn) = self.lstm(x, (h0.detach(), c0.detach()))
        out2, _ = self.gru(out) # Modification 2: Add gru forward part
        out = self.fc(out2[:, -1, :])
        return out

The modified version worked, but I have a few questions:

Is this a standard way to add RNN layers in Pytorch?
What does hn, cn mean to the forward function? I feel they seem some records of weight updating of the RNN, and the .required_grad_() seems an option to keep records of gradient descendent. Thus the hn, cn can be captured.
I added the GRU forward part but never involve any h, c here. Is it fine I don't need those updating records to create the model?


title: problem in installing CV3 package in google colab

body:
!pip install cv3

I was trying to install cv3 package on google collab but it is throwing the error below:
ERROR: Could not find a version that satisfies the requirement cv3 (from versions: none)
ERROR: No matching distribution found for cv3


title: The difference of loading model parameters between load_state_dict and nn.Parameter in pytorch

body:
When I wanna assign part of pre-trained model parameters to another module defined in a new model of PyTorch, I got two different outputs using two different methods.
The Network is defined as follows:
class Net:
    def __init__(self):
        super(Net, self).__init__()

        self.resnet = torch.hub.load('pytorch/vision', 'resnet18', pretrained=True)
        self.resnet = nn.Sequential(*list(self.resnet.children())[:-1])
        self.freeze_model(self.resnet)
        self.classifier = nn.Sequential(
            nn.Dropout(),
            nn.Linear(512, 256),
            nn.ReLU(),
            nn.Linear(256, 3),
        )

    def forward(self, x):
        out = self.resnet(x)
        out = out.flatten(start_dim=1)
        out = self.classifier(out)
        return out

What I want is to assign pre-trained parameters to classifier in the net module. Two different ways were used for this task.
# First way
net.load_state_dict(torch.load('model_CNN_pretrained.ptl'))

# Second way
params = torch.load('model_CNN_pretrained.ptl')
net.classifier[1].weight = nn.Parameter(params['classifier.1.weight'], requires_grad =False)
net.classifier[1].bias = nn.Parameter(params['classifier.1.bias'], requires_grad =False)
net.classifier[3].weight = nn.Parameter(params['classifier.3.weight'], requires_grad =False)
net.classifier[3].bias = nn.Parameter(params['classifier.3.bias'], requires_grad =False)

The parameters were assigned correctly but got two different outputs from the same input data. The first method works correctly, but the second doesn't work well. Could some guys point what the difference of these two methods?


title: create tensor in specific indexes without loops

body:
I have a tensor t1 (with shape (2*n, 2*n), and I need to create tensor t2 (with shape (2*n)) with the values of t1 at [i,(i+n) mod 2n] for each row i.
For example, given:
t1 = torch.tensor([[1, 2, 3, 4],
                   [5, 6, 7, 8],
                   [9 ,10,11,12],
                   [13,14,15,16]])

Here n=2.
t2 has to be [3,8,9,14].
I have this code:
t2 = torch.tensor([t1[i,(i+n)%(2*n)] for i in range(2*n)])

but Im searching a way without loops.


title: Crossentropyloss Pytorch: Targetsize does not match Torchsize

body:
I want to use the Crossentropyloss of pytorch but somehow my code only works with batchsize 2, so i am asuming there is something wrong with the shapes of target and output.
I get following error:
Value Error: Expected target size (50, 2), got torch.Size([50, 3])

My targetsize is (N=50,batchsize=3) and the output of my model is (N=50, batchsize=3, number of classes =2). Before the output layer my shape is (N=50,batchsize=3,dimensions=64).
How do i need to change the shapes so that the Crossentropyloss works?