title: Why does the quantization after pruning result in worse performance?

body:
I quantized resent18 model of mine, using graph quantization which lead to a forward pass of about 100 ms (on my cpu) and a size reduction of 40MBs (from initial 85 to 45 MBs).
I then went on and pruned this model to 14.5M parameters from the initial 25M and its size shrunk from 85 to 58 MB. I then quantized the resulting model hoping for further improvements but instead faced diminishing results.
That is, I don't see what I expected for inference speed.  Instead of being faster, the newer model simply is worse than the initial model (quantized from initial model).
Is this expected behavior? Here are the  two models for comparison:
https://gofile.io/d/zCzHAF
And this is their runtime benchmark results:
Quantized (no pruning done beforehand):
1>[ RUN      ] EmbedderModelForwardFixture.ModelEmbedderBench (10 runs, 10 iterations per run)
1>[     DONE ] EmbedderModelForwardFixture.ModelEmbedderBench (5672.357520 ms)
1>[   RUNS   ]        Average time: 567235.752 us (~31674.053 us)
1>                    Fastest time: 530900.462 us (-36335.290 us / -6.406 %)
1>                    Slowest time: 640024.562 us (+72788.810 us / +12.832 %)
1>                     Median time: 561095.762 us (1st quartile: 548392.562 us | 3rd quartile: 577176.062 us)
1>
1>             Average performance: 1.76294 runs/s
1>                Best performance: 1.88359 runs/s (+0.12066 runs/s / +6.84409 %)
1>               Worst performance: 1.56244 runs/s (-0.20050 runs/s / -11.37282 %)
1>              Median performance: 1.78223 runs/s (1st quartile: 1.82351 | 3rd quartile: 1.73257)
1>
1>[ITERATIONS]        Average time: 56723.575 us (~3167.405 us)
1>                    Fastest time: 53090.046 us (-3633.529 us / -6.406 %)
1>                    Slowest time: 64002.456 us (+7278.881 us / +12.832 %)
1>                     Median time: 56109.576 us (1st quartile: 54839.256 us | 3rd quartile: 57717.606 us)
1>
1>             Average performance: 17.62935 iterations/s
1>                Best performance: 18.83592 iterations/s (+1.20657 iterations/s / +6.84409 %)
1>               Worst performance: 15.62440 iterations/s (-2.00495 iterations/s / -11.37282 %)
1>              Median performance: 17.82227 iterations/s (1st quartile: 18.23511 | 3rd quartile: 17.32574)

Pruned-Quantized model:
1>[ RUN      ] EmbedderModelForwardFixture.ModelEmbedderBench (10 runs, 10 iterations per run)
1>[     DONE ] EmbedderModelForwardFixture.ModelEmbedderBench (5778.116020 ms)
1>[   RUNS   ]        Average time: 577811.602 us (~26296.168 us)
1>                    Fastest time: 537246.562 us (-40565.040 us / -7.020 %)
1>                    Slowest time: 617859.662 us (+40048.060 us / +6.931 %)
1>                     Median time: 585275.362 us (1st quartile: 554311.262 us | 3rd quartile: 594753.362 us)
1>
1>             Average performance: 1.73067 runs/s
1>                Best performance: 1.86134 runs/s (+0.13067 runs/s / +7.55054 %)
1>               Worst performance: 1.61849 runs/s (-0.11218 runs/s / -6.48174 %)
1>              Median performance: 1.70860 runs/s (1st quartile: 1.80404 | 3rd quartile: 1.68137)
1>
1>[ITERATIONS]        Average time: 57781.160 us (~2629.617 us)
1>                    Fastest time: 53724.656 us (-4056.504 us / -7.020 %)
1>                    Slowest time: 61785.966 us (+4004.806 us / +6.931 %)
1>                     Median time: 58527.536 us (1st quartile: 55431.126 us | 3rd quartile: 59475.336 us)
1>
1>             Average performance: 17.30668 iterations/s
1>                Best performance: 18.61343 iterations/s (+1.30675 iterations/s / +7.55054 %)
1>               Worst performance: 16.18491 iterations/s (-1.12177 iterations/s / -6.48174 %)
1>              Median performance: 17.08597 iterations/s (1st quartile: 18.04041 | 3rd quartile: 16.81369)

To put it simply after 10 iterations, each take:
r18_default : 805.72 ms (mean)
quantized_model : 560 ms (mean)
r18_pruned : 7,466.78  ms
pruned_then_quantized: 578 ms (mean)

Not only the second model is not faster, it's worse, it has become slower! You can also see that the pruned model is extremely slow. 10x slower than the default model.
Note:
In case it matters, training (pruning, and finetuning the model) is done using pytorch 1.5.1 and the final graph quantization is done under windows using pytorch 1.6.
Note 2:
This is being tested and evaluated using libtorch (1.6) on Windows 10 machine, but there shouldn't be any discrepancy between Python and C++.


title: Why loaded Pytorch model's loss highly increased?

body:
I'm trying to train Arcface with reference to.
As far as I know, Arcface requires more than 200 training epochs on CASIA-webface with a large batch size.
Within 100 epochs of training, I stopped the training for a while because I was needed to use GPU for other tasks. And the checkpoints of the model(Resnet) and margin are saved. Before it was stopped, its loss recorded a value between 0.3~1.0, and training accuracy was mount to 80~95%.
When I resume the Arcface training by loading the checkpoint files using load_sate, it seems normal when the first batch is processed. But suddenly the loss increased sharply and the accuracy became very low.
Its loss suddenly became increased. How did this happen? I had no other way so anyway continued the training, but I don't think the loss is decreasing well even though it is a trained model over 100 epochs...
When I searched for similar issues, they told the problem was that the optimizer was not saved (Because the reference github page didn't save the optimizer, so did I. Is it true?
My losses after loading


title: Error installing pretrained models for pytorch

body:
I'm working on a Windows 10 machine (yes, I know, don't laugh!), and with python 3.7, and I'm trying to install the pretrained models here:
https://github.com/meliketoy/fine-tuning.pytorch
The commands that the website suggests are:
$ git clone https://github.com/Cadene/pretrained-models.pytorch.git
$ pretrained-models.pytorch
$ python setup.py install

Although the website says this is for Python 3.5, and I have 3.7, I think the 3.7 version should be back-compatible, right?
I successfully ran the git clone, and the pretrained-models.pytorch was actually a cd command (which threw me for a loop for a second!). Then I ran into trouble with python setup.py install
The error I'm getting is:
[Errno 2] No such file or directory: 'build\\bdist.win-amd64\\egg\\pretrainedmodels\\models\\resnext_features\\__pycache__\\resnext101_32x4d_features.cpython-37.pyc.1702181039952'

How can I fix this error?
EDIT (in response to a comment): Someone asked for the full traceback. Here it is!
(base) G:\>python setup.py install
running install
running bdist_egg
running egg_info
creating pretrainedmodels.egg-info
writing pretrainedmodels.egg-info\PKG-INFO
writing dependency_links to pretrainedmodels.egg-info\dependency_links.txt
writing requirements to pretrainedmodels.egg-info\requires.txt
writing top-level names to pretrainedmodels.egg-info\top_level.txt
writing manifest file 'pretrainedmodels.egg-info\SOURCES.txt'
reading manifest file 'pretrainedmodels.egg-info\SOURCES.txt'
writing manifest file 'pretrainedmodels.egg-info\SOURCES.txt'
installing library code to build\bdist.win-amd64\egg
running install_lib
running build_py
creating build
creating build\lib
creating build\lib\pretrainedmodels
copying pretrainedmodels\utils.py -> build\lib\pretrainedmodels
copying pretrainedmodels\version.py -> build\lib\pretrainedmodels
copying pretrainedmodels\__init__.py -> build\lib\pretrainedmodels
creating build\lib\pretrainedmodels\datasets
copying pretrainedmodels\datasets\utils.py -> build\lib\pretrainedmodels\datasets
copying pretrainedmodels\datasets\voc.py -> build\lib\pretrainedmodels\datasets
copying pretrainedmodels\datasets\__init__.py -> build\lib\pretrainedmodels\datasets
creating build\lib\pretrainedmodels\models
copying pretrainedmodels\models\bninception.py -> build\lib\pretrainedmodels\models
copying pretrainedmodels\models\cafferesnet.py -> build\lib\pretrainedmodels\models
copying pretrainedmodels\models\dpn.py -> build\lib\pretrainedmodels\models
copying pretrainedmodels\models\fbresnet.py -> build\lib\pretrainedmodels\models
copying pretrainedmodels\models\inceptionresnetv2.py -> build\lib\pretrainedmodels\models
copying pretrainedmodels\models\inceptionv4.py -> build\lib\pretrainedmodels\models
copying pretrainedmodels\models\nasnet.py -> build\lib\pretrainedmodels\models
copying pretrainedmodels\models\nasnet_mobile.py -> build\lib\pretrainedmodels\models
copying pretrainedmodels\models\pnasnet.py -> build\lib\pretrainedmodels\models
copying pretrainedmodels\models\polynet.py -> build\lib\pretrainedmodels\models
copying pretrainedmodels\models\resnext.py -> build\lib\pretrainedmodels\models
copying pretrainedmodels\models\senet.py -> build\lib\pretrainedmodels\models
copying pretrainedmodels\models\torchvision_models.py -> build\lib\pretrainedmodels\models
copying pretrainedmodels\models\utils.py -> build\lib\pretrainedmodels\models
copying pretrainedmodels\models\vggm.py -> build\lib\pretrainedmodels\models
copying pretrainedmodels\models\wideresnet.py -> build\lib\pretrainedmodels\models
copying pretrainedmodels\models\xception.py -> build\lib\pretrainedmodels\models
copying pretrainedmodels\models\__init__.py -> build\lib\pretrainedmodels\models
creating build\lib\pretrainedmodels\models\resnext_features
copying pretrainedmodels\models\resnext_features\resnext101_32x4d_features.py -> build\lib\pretrainedmodels\models\resnext_features
copying pretrainedmodels\models\resnext_features\resnext101_64x4d_features.py -> build\lib\pretrainedmodels\models\resnext_features
copying pretrainedmodels\models\resnext_features\__init__.py -> build\lib\pretrainedmodels\models\resnext_features
creating build\bdist.win-amd64
creating build\bdist.win-amd64\egg
creating build\bdist.win-amd64\egg\pretrainedmodels
creating build\bdist.win-amd64\egg\pretrainedmodels\datasets
copying build\lib\pretrainedmodels\datasets\utils.py -> build\bdist.win-amd64\egg\pretrainedmodels\datasets
copying build\lib\pretrainedmodels\datasets\voc.py -> build\bdist.win-amd64\egg\pretrainedmodels\datasets
copying build\lib\pretrainedmodels\datasets\__init__.py -> build\bdist.win-amd64\egg\pretrainedmodels\datasets
creating build\bdist.win-amd64\egg\pretrainedmodels\models
copying build\lib\pretrainedmodels\models\bninception.py -> build\bdist.win-amd64\egg\pretrainedmodels\models
copying build\lib\pretrainedmodels\models\cafferesnet.py -> build\bdist.win-amd64\egg\pretrainedmodels\models
copying build\lib\pretrainedmodels\models\dpn.py -> build\bdist.win-amd64\egg\pretrainedmodels\models
copying build\lib\pretrainedmodels\models\fbresnet.py -> build\bdist.win-amd64\egg\pretrainedmodels\models
copying build\lib\pretrainedmodels\models\inceptionresnetv2.py -> build\bdist.win-amd64\egg\pretrainedmodels\models
copying build\lib\pretrainedmodels\models\inceptionv4.py -> build\bdist.win-amd64\egg\pretrainedmodels\models
copying build\lib\pretrainedmodels\models\nasnet.py -> build\bdist.win-amd64\egg\pretrainedmodels\models
copying build\lib\pretrainedmodels\models\nasnet_mobile.py -> build\bdist.win-amd64\egg\pretrainedmodels\models
copying build\lib\pretrainedmodels\models\pnasnet.py -> build\bdist.win-amd64\egg\pretrainedmodels\models
copying build\lib\pretrainedmodels\models\polynet.py -> build\bdist.win-amd64\egg\pretrainedmodels\models
copying build\lib\pretrainedmodels\models\resnext.py -> build\bdist.win-amd64\egg\pretrainedmodels\models
creating build\bdist.win-amd64\egg\pretrainedmodels\models\resnext_features
copying build\lib\pretrainedmodels\models\resnext_features\resnext101_32x4d_features.py -> build\bdist.win-amd64\egg\pretrainedmodels\models\resnext_features
copying build\lib\pretrainedmodels\models\resnext_features\resnext101_64x4d_features.py -> build\bdist.win-amd64\egg\pretrainedmodels\models\resnext_features
copying build\lib\pretrainedmodels\models\resnext_features\__init__.py -> build\bdist.win-amd64\egg\pretrainedmodels\models\resnext_features
copying build\lib\pretrainedmodels\models\senet.py -> build\bdist.win-amd64\egg\pretrainedmodels\models
copying build\lib\pretrainedmodels\models\torchvision_models.py -> build\bdist.win-amd64\egg\pretrainedmodels\models
copying build\lib\pretrainedmodels\models\utils.py -> build\bdist.win-amd64\egg\pretrainedmodels\models
copying build\lib\pretrainedmodels\models\vggm.py -> build\bdist.win-amd64\egg\pretrainedmodels\models
copying build\lib\pretrainedmodels\models\wideresnet.py -> build\bdist.win-amd64\egg\pretrainedmodels\models
copying build\lib\pretrainedmodels\models\xception.py -> build\bdist.win-amd64\egg\pretrainedmodels\models
copying build\lib\pretrainedmodels\models\__init__.py -> build\bdist.win-amd64\egg\pretrainedmodels\models
copying build\lib\pretrainedmodels\utils.py -> build\bdist.win-amd64\egg\pretrainedmodels
copying build\lib\pretrainedmodels\version.py -> build\bdist.win-amd64\egg\pretrainedmodels
copying build\lib\pretrainedmodels\__init__.py -> build\bdist.win-amd64\egg\pretrainedmodels
byte-compiling build\bdist.win-amd64\egg\pretrainedmodels\datasets\utils.py to utils.cpython-37.pyc
byte-compiling build\bdist.win-amd64\egg\pretrainedmodels\datasets\voc.py to voc.cpython-37.pyc
byte-compiling build\bdist.win-amd64\egg\pretrainedmodels\datasets\__init__.py to __init__.cpython-37.pyc
byte-compiling build\bdist.win-amd64\egg\pretrainedmodels\models\bninception.py to bninception.cpython-37.pyc
byte-compiling build\bdist.win-amd64\egg\pretrainedmodels\models\cafferesnet.py to cafferesnet.cpython-37.pyc
byte-compiling build\bdist.win-amd64\egg\pretrainedmodels\models\dpn.py to dpn.cpython-37.pyc
byte-compiling build\bdist.win-amd64\egg\pretrainedmodels\models\fbresnet.py to fbresnet.cpython-37.pyc
byte-compiling build\bdist.win-amd64\egg\pretrainedmodels\models\inceptionresnetv2.py to inceptionresnetv2.cpython-37.pyc
byte-compiling build\bdist.win-amd64\egg\pretrainedmodels\models\inceptionv4.py to inceptionv4.cpython-37.pyc
byte-compiling build\bdist.win-amd64\egg\pretrainedmodels\models\nasnet.py to nasnet.cpython-37.pyc
byte-compiling build\bdist.win-amd64\egg\pretrainedmodels\models\nasnet_mobile.py to nasnet_mobile.cpython-37.pyc
byte-compiling build\bdist.win-amd64\egg\pretrainedmodels\models\pnasnet.py to pnasnet.cpython-37.pyc
byte-compiling build\bdist.win-amd64\egg\pretrainedmodels\models\polynet.py to polynet.cpython-37.pyc
byte-compiling build\bdist.win-amd64\egg\pretrainedmodels\models\resnext.py to resnext.cpython-37.pyc
byte-compiling build\bdist.win-amd64\egg\pretrainedmodels\models\resnext_features\resnext101_32x4d_features.py to resnext101_32x4d_features.cpython-37.pyc
error: [Errno 2] No such file or directory: 'build\\bdist.win-amd64\\egg\\pretrainedmodels\\models\\resnext_features\\__pycache__\\resnext101_32x4d_features.cpython-37.pyc.1702181039952'


title: transpose on PyTorch : IndexError: Dimension out of range (expected to be in range of [-2, 1], but got 2)

body:
I want to transpose the data I have using transpose, but I am encountering such an error. My data and related process uploaded to github.
https://github.com/nurkbts/error/blob/main/error.ipynb


title: How to access Spark DataFrame data in GPU from ML Libraries such as PyTorch or Tensorflow

body:
Currently I am studying the usage of Apache Spark 3.0 with Rapids GPU Acceleration. In the official spark-rapids docs I came across this page which states:

There are cases where you may want to get access to the raw data on the GPU, preferably without copying it. One use case for this is exporting the data to an ML framework after doing feature extraction.

To me this sounds as if one could make data that is already available on the GPU from some upstream Spark ETL process directly available to a framework such as Tensorflow or PyTorch. If this is the case how can I access the data from within any of these frameworks? If I am misunderstanding something here, what is the quote exactly referring to?


title: Can't load a dataset from torchvision

body:
I'm trying to load the LSUN dataset following PyTorch's code. I used their other datasets but this one seems to give me errors.
import torch
import torchvision.transforms as transforms

#convert the data to torch tensors
transform = transforms.Compose([transforms.ToTensor()])

from torchvision.datasets import LSUN
data = LSUN(root = './', transform=transform)

>>>Error: .//bedroom_train_lmdb: No such file or directory

Am I doing something wrong here? The code works just fine with MNIST/CIFAR/etc. (with a slight modification data = MNIST(root = './', train=False, download=True, transform=transform)
Update
Cloned the repo and downloaded the dataset:
!git clone https://github.com/fyu/lsun.git
cd lsun
# Download testing set
!python3 download.py -c test

Tried running the code as before with
data = LSUN(root = '',classes='test_lmdb.zip', transform=transform)

But getting this error now:
ValueError: Unknown value 'test_lmdb.zip' for argument classes. Valid values are {'train', 'val', 'test'}.

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
/usr/local/lib/python3.7/dist-packages/torchvision/datasets/utils.py in verify_str_arg(value, arg, valid_values, custom_msg)
    348             msg = msg.format(value=value, arg=arg,
    349                              valid_values=iterable_to_str(valid_values))
--> 350         raise ValueError(msg)
    351
    352     return value

ValueError: Unknown value '' for LSUN class. Valid values are {'bedroom', 'bridge', 'church_outdoor', 'classroom', 'conference_room', 'dining_room', 'kitchen', 'living_room', 'restaurant', 'tower'}.

When I change it to
data = LSUN(root = '',classes='test', transform=transform)

I get this error:
Error: /test_lmdb: No such file or directory


title: filter class/subfolder with pytorch ImageFolder

body:
Here's my folder structure
image-folders/
   ├── class_0/
   |   ├── 001.jpg
   |   ├── 002.jpg
   └── class_1/
   |   ├── 001.jpg
   |   └── 002.jpg
   └── class_2/
       ├── 001.jpg
       └── 002.jpg

By using ImageFolder from torchvision, I can create dataset with this syntax :
dataset = ImageFolder("image-folders",...)
But this will read the entire subfolder and create 3 target classes. I don't want to include the class_2 folder, I want my dataset to only contains class_0 and class_1 only, is there any way to achieve this besides delete/move the class_2 folder?


title: How to index a tensor in pytorch?

body:
I'm having some trouble with pytorch.
When calculating mu and sig this message appears:
ValueError: result of slicing is an empty tensor

Here's my code:

x            = mu + sig*torch.randn(N,1)
S            = -x**2 + 100
sorted_v , I = torch.sort(S,0)
mu           = alpha*torch.mean(x[I[(1-rho)*N:N]]) + (1-alpha)*mu
sig          = alpha*torch.std(x[I[(1-rho)*N:N]])  + (1-alpha)*sig

In MATLAB I would do something like this:

mu    = alpha*mean(x(I((1-rho)*N:N))) + (1-alpha)*mu;
sig   = alpha*std(x(I((1-rho)*N:N)))  + (1-alpha)*sig;

What is wrong and why?
Thak you so much.


title: Matrix-vector multiplication for only one dimension in a tensor

body:
Is it possible to multiply only one (last) dimension in a tensor alone with other vectors?

For example, assume a tensor T=[100, 20, 400] and a matrix M =[400, 400].
Is it possible to make the operation h_{transpose}*M*h, where h is the last dimension in the tensor T? In other words, is it possible to make use of (possibly pytorch) built-in functions to get the resulting tensor of size [100, 20, 1]?


title: When not to use Tensors

body:
I have been using PyTorch for a while now and am making a general RL framework. I am running into the question of whether to use np.arrays or tensors.

When would you not want to use tensors when available? What would make you choose numpy over pytorch? Obviously tensors are important for ML models, but what if you want to just do basic image processing or list manipulation?

I am tempted to use Tensors whenever possible but do not know of any pitfalls. (graph confusion? memory leaks??)

For example, I have a basic unfinished code snippet of collecting actions for an env, not sure whether to stick with numpy or not.

@dataclass
class Action(object):
    """
    Handles actions, action space, and value verification.
    """
    taken_action: np.array
    raw_action: np.array
    n_possible_values: int
    action_space: gym.Space

    def __post_init__(self):
        if type(self.taken_action) is not np.array: taken_action = np.array([self.taken_action])

@pytest.mark.parametrize("env", sorted([env.id for env in gym.envs.registry.all()]))
def test_action_data_structure(env):
    try:
        init_env = gym.make(env)
    except error.DependencyNotInstalled as e:
        print(e)
        return

    taken_action = init_env.action_space.sample()
    raw_action = np.random.rand(init_env.action_space.n)
    state, reward, done, info = init_env.step(taken_action)

    action_dataclass = Action(taken_action=taken_action, raw_action=raw_action,
                              n_possible_values=init_env.action_space.n, action_space=init_env.action_space)


title: Why does runtime not scale with flops - Pointwise Multiplication vs 2D Convolutions

body:
Background

Based on the convolution theorem of Fourier Transforms convolutions in the spatial domain are equivalent to pointwise multiplications in the Fourier domain (and the other way around). I implemented a torch.nn.Conv2d that "operates" in the Fourier domain by performing pointwise multiplication instead of convolution (with a kernel that is transformed to the size of the intput) in PyTorch (as described e.g. here: https://arxiv.org/pdf/1312.5851.pdf)

Expectation & Results

I found that it did not perform well, similar to: Keras/Tensorflow - fourier pointwise multiplication implementation of conv2d running 4x slower than spatial convolution

After a number of benchmarks it appears that the pointwise multiplication is the major bottleneck of the operation. During the benchmarks I excluded the FFT process to isolate the operations of the layer (and used saved kernels in the appropriate size).

This is confusing because when considering the number of FLOPs required for a 2D convolution (stride=1) and element-wise multiplication:

Conv2d FLOPs: Kernel_H * Kernel_W * C_in * C_out * H * W
Pointwise FLOPs: C_in * C_out * H * W

E.g. given H = 32, W = 60, C_in = 64, C_out = 256:

Conv2d FLOPs (k = 16): 16 * 16 * 32 * 60 * 64 * 256 = 8053 MFLOPs
Pointwise (FLOPs): 64 * 256 * 32 * 60 = 31.46 MFLOPs

Considering that large difference in FLOPs I would expect the 2D convolution to take much longer to run (have read GPUs are optimized well for dot products)

I created a simple script to benchmark pointwise multiplication of torch.Tensor against torch.nn.Conv2d, since it appeared that the performance was equivalent or even slower for elementwise multiplication compared to 2D convolutions.

Below is an overview of 2 such benchmark result on a CPU and GPU (i9900k with torch.set_num_threads(1))

Results - CPU (i9900k)

(# Kernel Size = 16)

Benchmark Overview (device = cpu):
    Number of test iterations: 100
    Number of warm-up iterations: 5
    Pointwise: [1, 256, 32, 60] * [64, 256, 32, 60]
    Conv2d(in_ch=256, out_ch=64, kernel_size=16): Conv2d([1, 256, 32, 60)
    FLOP Estimation:
        Conv2d:      8053.06368 MFlops
        Pointwise:   31.45728 MFlops

Benchmark Results (device = cpu)
    Pointwise:   16.139 +/- 0.786 ms
    Conv2d:      12.947 +/- 0.784 ms

-------------------------
(# Kernel Size = 5)

Benchmark Overview (device = cpu):
    Number of test iterations: 100
    Number of warm-up iterations: 5
    Pointwise: [1, 256, 32, 60] * [64, 256, 32, 60]
    Conv2d(in_ch=256, out_ch=64, kernel_size=5): Conv2d([1, 256, 32, 60)
    FLOP Estimation:
        Conv2d:      786.432 MFlops
        Pointwise:   31.45728 MFlops

Benchmark Results (device = cpu)
    Pointwise:   36.085 +/- 3.668 ms
    Conv2d:      9.344 +/- 0.952 ms

Results - GPU (RTX Titan)

(# Kernel Size = 16)

Benchmark Overview (device = cuda:1):
    Number of test iterations: 1000
    Number of warm-up iterations: 5
    Pointwise: [1, 256, 32, 60] * [64, 256, 32, 60]
    Conv2d(in_ch=256, out_ch=64, kernel_size=16): Conv2d([1, 256, 32, 60)
    FLOP Estimation:
        Conv2d:      8053.06368 MFlops
        Pointwise:   31.45728 MFlops

Benchmark Results (device = cuda:1)
    Pointwise:   0.698 +/- 0.031 ms
    Conv2d:      2.916 +/- 0.161 ms

------------------------------------

(# Kernel size = 3)

Benchmark Overview (device = cuda:1):
    Number of test iterations: 100
    Number of warm-up iterations: 5
    Pointwise: [1, 256, 32, 60] * [64, 256, 32, 60]
    Conv2d(in_ch=256, out_ch=64, kernel_size=3): Conv2d([1, 256, 32, 60)
    FLOP Estimation:
        Conv2d:      283.11552 MFlops
        Pointwise:   31.45728 MFlops
        FreqConv:    62.91456 MFlops

Benchmark Results (device = cuda:1)
    Pointwise:   0.681 +/- 0.011 ms
    Conv2d:      0.126 +/- 0.034 ms

The results do not appear to change significantly if I change the H or W or the channels. But for smaller kernels pointwise appears significantly slower.

Question

Can anyone suggest why the pointwise multiplication appears to be so slow when the FLOPs are at least 2 orders of magnitude larger, or where there may be an error in my thinking or code?

Benchmark Implementation

import torch
import numpy as np
from torch import nn
from time import time

torch.set_num_threads(1)

in_ch = 256
out_ch = 64
height = 32
width = 60
kernel_size = 16

warmup = 5
iters = 100

flops_pointwise = (out_ch * in_ch * height * width)
m_flops_conv = (flops_pointwise * kernel_size ** 2) / 1e6
m_flops_pw = (flops_pointwise) / 1e6

# Device to run benchmark on, e.g. 'cpu' or 'cuda:X'
device = 'cpu'

print(f'Benchmark Overview (device = {device}):')
print(f'\tNumber of test iterations: {iters}')
print(f'\tNumber of warm-up iterations: {warmup}')
print(f'\tPointwise: [1, {in_ch}, {height}, {width}] * [{out_ch}, {in_ch}, {height}, {width}]')
print(f'\tConv2d(in_ch={in_ch}, out_ch={out_ch}, kernel_size={kernel_size}): Conv2d([1, {in_ch}, {height}, {width})')

print('\tFLOP Estimation:')
print(f'\t\tConv2d:\t\t {m_flops_conv} MFlops')
print(f'\t\tPointwise:\t {m_flops_pw} MFlops')
print(f'\t\tFreqConv:\t {m_flops_freq_conv} MFlops')

print()

def benchmark(input_gen, operation, warmup=5, iters=1000):
    duration = []
    for i in range(iters + warmup):

        input = input_gen()

        start = time() # start timer
        with torch.no_grad():
            operation(input)

        # Sync if using cuda
        if device[:4] == 'cuda':
            torch.cuda.synchronize(device)
        end = time() # end timer

        if i < warmup:
            continue

        duration.append((end - start) * 1e3) # ms

    return np.array(duration)

def pointwise(input):
    x, y = input
    x * y

# Helper methods to generate new data
# for every iteration inside of the benchmark method

def _gen_pw_input(in_ch, out_ch, height, width):
    x = torch.rand(1, in_ch, height, width).to(device)
    k = torch.randn(out_ch, in_ch, height, width).to(device)
    return x, k

gen_pw_input = lambda : _gen_pw_input(in_ch, out_ch, height, width)

def _gen_conv_input(in_ch, out_ch, height, width):
    x = torch.rand(1, in_ch, height, width).to(device)
    return x

gen_conv_input = lambda : _gen_conv_input(in_ch, out_ch, height, width)

conv2d = nn.Conv2d(in_ch, out_ch, kernel_size=kernel_size).to(device)

pw_res = benchmark(gen_pw_input, pointwise, warmup=warmup, iters=iters)
conv_res = benchmark(gen_conv_input, conv2d, warmup=warmup, iters=iters)

print(f'Benchmark Results (device = {device})')
print('\tPointwise:\t {:.3f} +/- {:.3f} ms'.format(pw_res.mean(), pw_res.std()))
print('\tConv2d:\t\t {:.3f} +/- {:.3f} ms'.format(conv_res.mean(), conv_res.std()))

Eigen

I also implemented a basic benchmark test in Eigen (C++) to compare the elementwise multiplication and it was similar (slightly slower) to results observed in PyTorch; the backend BLAS used by PyTorch appears optimized.


title: How do i save images for each class using Pytorch? (no grid)

body:
I am using acgan to do image augmentation. At present, sample images are generated in grid format. But I want to save images for each class separately. (e.g. 1.png; 2.png ...) How should I modify this code? Or is there an answer that I would like to refer to?

class Generator(nn.Module):
def __init__(self):
    super(Generator, self).__init__()

    self.label_emb = nn.Embedding(opt.n_classes, opt.latent_dim)

    self.init_size = opt.img_size // 4  # Initial size before upsampling
    self.l1 = nn.Sequential(nn.Linear(opt.latent_dim, 128 * self.init_size ** 2))

    self.conv_blocks = nn.Sequential(
        nn.BatchNorm2d(128),
        nn.Upsample(scale_factor=2),
        nn.Conv2d(128, 128, 3, stride=1, padding=1),
        nn.BatchNorm2d(128, 0.8),
        nn.LeakyReLU(0.2, inplace=True),
        nn.Upsample(scale_factor=2),
        nn.Conv2d(128, 64, 3, stride=1, padding=1),
        nn.BatchNorm2d(64, 0.8),
        nn.LeakyReLU(0.2, inplace=True),
        nn.Conv2d(64, opt.channels, 3, stride=1, padding=1),
        nn.Tanh(),
    )

..

generator = Generator()

..

def sample_image(n_row, batches_done):
   """Saves a grid of generated digits ranging from 0 to n_classes"""
   # Sample noise
   z = Variable(FloatTensor(np.random.normal(0, 1, (n_row ** 2, opt.latent_dim))))
   # Get labels ranging from 0 to n_classes for n rows
   labels = np.array([num for _ in range(n_row) for num in range(n_row)])
   labels = Variable(LongTensor(labels))
   gen_imgs = generator(z, labels)
   save_image(gen_imgs.data, "images/%d.png" % batches_done, nrow=n_row, normalize=True)


title: What is a pytorch snapshot and a Dlib model? (Running Hopenet)

body:
I am trying to make Hopenet run on my computer, using the code on github

I have installed all required libraries. This is my forked code, with test_hopenet.py updated to python 3.

I installed all required libraries, and using pip install "pillow<7" because of some old requirements.

python code/test_on_video_dlib.py --snapshot PATH_OF_SNAPSHOT --face_model PATH_OF_DLIB_MODEL --video PATH_OF_VIDEO --output_string STRING_TO_APPEND_TO_OUTPUT --n_frames N_OF_FRAMES_TO_PROCESS --fps FPS_OF_SOURCE_VIDEO

It looks like I am missing some basic understanding which is supposed to be obvious:

What is a SNAPSHOT? Where do I get one?
What is a Dlib Model? Where do I get the right one?

Again - I just want to make this code run, but I can't understand the instructions.


title: CNN not learning correctly

body:
I've a small dataset of 500 plant images and I have to predict a number for a single image in range [1, 10]. There is a order relation between the numbers (10 > 9 > ... > 1). This problem is similar to age estimation based on a single photo.

I tried regression using Resnet18, Resnet34 and VGG16. None of them gave a very good result.

The interesting point is that when I plotted the heatmap for a few images it showed that the model was picking the wrong spots to predict the answer. It's like, if I was suppose to predict age based on facial photo, the cnn gave more value to the background than to the actual face.

I tried other approachs as well, like classification and learning to rank, but it happens the same thing when I do heatmap. In these approachs, the best accuracy I get is 30% using classification and 35% using learning to rank.

The regression and classification approachs I used Fastai implementation with pretrained. The learning to rank approach I used this : https://github.com/Raschka-research-group/coral-cnn. I changed a little bit to be able to use a pretrained model as well.

Another important point is that the dataset is unbalanced. 80% of the dataset corresponds to classes 6 to 10.

Does anyone has any tips to improve it or another approach I could try?

EDIT:
My data augmentation looks like this:

transforms.Compose([
                  transforms.Resize(256), transforms.CenterCrop(224),
                  transforms.RandomHorizontalFlip(p=0.5),
                  transforms.ColorJitter(brightness=0.15),
                  transforms.ToTensor(),
                  transforms.Normalize([0.485, 0.456, 0.406], [0.299, 0.224, 0.225])
])


title: Torch Divide Two Tensors With a Fallback Value if Unsafe - Fast

body:
I am dividing two tensors that include zeros with one another.

test = torch.zeros(10000)
div = torch.div(test, test) # 99497.19it/s

However, the result div includes nans, and using this output tensor messes up gradient decent, so I want to fill div with a fallback value of 0.

div[div != div] = 0 # 28546.27it/s

This operation for fixing the tensor is very costly, and changes the entire sequence from running 100,000 times a second to around 5500.

Is there a faster way to either divide with a fallback value, or fill out all nans with 0?


title: Pytorch | I don't know why it is throwing an error? (Beginner)

body:
import torch.nn as nn
import torch.nn.functional as F

## TODO: Define the NN architecture
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        # linear layer (784 -> 1 hidden node)
        self.fc1 = nn.Linear(28 * 28, 512)
        self.fc2 = nn.Linear(512 * 512)
        self.fc3 = nn.Linear(512 * 10)

    def forward(self, x):
        # flatten image input
        x = x.view(-1, 28 * 28)
        # add hidden layer, with relu activation function
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.relu(self.fc3(x))
        return x

# initialize the NN
model = Net()
print(model)

When I run this, it throws this error. Why?

TypeError: __ init __() missing 1 required positional argument: 'out_features'


title: Custom dataset os.path.join () returns type error

body:
I am writing custom dataset but it returns type error when I am merging root directory path with pandas iloc of image name in csv file:

 img_path = os.path.join(self.root_dir, self.annotations.iloc[index,0])

error:  TypeError: join() argument must be str or bytes, not 'int64'

I have tried converting the annotation.iloc to string type but it's still giving me the same error.

csvfile with filenames and labels:

custom dataset class:

class patientdataset(Dataset):

    def __init__(self, csv_file, root_dir, transform=None):
            self.annotations = pd.read_csv(csv_file)
            self.root_dir = root_dir
            self.transform = transform

    def __len__(self):
        return len(self.annotations)

    def __getitem__(self, index):
        img_path = os.path.join(self.root_dir, self.annotations.iloc[index,0])
        image= np.array(np.load(img_path))
        y_label = torch.tensor(self.annotations.iloc[index, 1]).long()

        if self.transform:
            imagearrays = self.transform(image)
            image = imagearrays[None, :, :, :]
            imaget = np.transpose(image, (0, 2, 1, 3))
            image = imaget

        return (image, y_label)


title: How to save arrays in a .npz structure compatible with FBK Fairseq for Direct Speech Translation?

body:
I generated a npz folder with numpy with the code
np.savez(outpath + "/data.npz", **keywords)
where keywords is a dictionary structured as:

"0" : array
"1" : array

Each array is a 2D array containing MFCC features extracted with speechpy.
For instance the array for the key 0 has shape (518, 13).

This is the folder and filename structure of the resulting npz folder:

data.npz
    0.npy
    1.npy

Since I have to feed the npz file to a sequence modeling toolkit for speech translation FBK-Fairseq-ST build on pytorch, the function to read the npz file is as it follows:

def reader_npz(path):
   with open(path, 'rb') as f:
      shape = np.load(f)
      for i in range(int(shape[0])):
         yield torch.from_numpy(np.load(f))

I had to modify the line for i in range(int(shape["0"])) to for i in range(int(shape[0])) as suggested in the comments by @V. Ayrat, to avoid a key error.

The problem is that this results in the TypeError: only size-1 arrays can be converted to Python scalars since I am giving a 2D array to int.

In fact, if my .npz folder contains 100 npy files, shape=np.load(f) will result in a set of a 2D arrays from shape["0"] to shape["99"].

How should I save the .npy files in the .npz file in order to make the .npz folder readable by the function reader_npz(path) in the FAIR's fairseq script above?

Thanks in advance!


title: Pytorch reinforcement learning explanation

body:
I am attempting reinforcement learning to create a chess engine. I found some code online in Python with Pytorch. I am more familiar with tensorflow so can somebody help explain what the code means and maybe help "translate" it into tensorflow code?
Also, is the currently defined model "robust" enough to be decent? If not, how can I improve it? I have tried adding more nn.Linear lines to it but it seems to weaken the model.
EDIT: Here is the original link to the code: https://colab.research.google.com/drive/1Xk9MibJ9Fli5tIlDvo88hcZrI76rqZN5#scrollTo=ZxHEghUq9JWM
EDIT2: Also, I want to figure out how to implement a system to use its memory of previous results into the current code. Can somebody point me in the right direction?
Note that the code below was ADAPTED from the link.
Adapted Code:
import chess
import chess.pgn
import chess.engine
import torch
import numpy as np
import os
import torch.nn as nn
from torch.nn import functional as F
#os.remove("Games.txt")
def board_to_tensor(board):
    # Python chess uses flattened representation of the board
    x = torch.zeros(64, dtype=torch.float)
    for pos in range(64):
        piece = board.piece_type_at(pos)
        if piece:
            color = int(bool(board.occupied_co[chess.BLACK] & chess.BB_SQUARES[pos]))
            col = int(pos % 8)
            row = int(pos / 8)
            x[row * 8 + col] = -piece if color else piece
    x = x.reshape(8, 8)
    return x

def move_to_index_tensor(move):
    index_tensor = torch.LongTensor([0])
    square_to_pick_figure = move.from_square
    # Can decode exact position this way:
    #square_to_pick_figure_row = int(square_to_pick_figure / 8)
    #square_to_pick_figure_col = int(squre_to_pick_figure % 8)
    square_to_put_figure = move.to_square
    # Can decode exact position this way:
    #square_to_put_figure_row = int(square_to_put_figure / 8)
    #square_to_put_figure_col = int(square_to_put_figure % 8)
    index = square_to_pick_figure * 64 + square_to_put_figure
    index_tensor = torch.LongTensor([index])
    return index_tensor

def filter_legal_moves(legal_moves):
    filtered_legal_moves = []
    for legal_move in legal_moves:
        # Here we check if it is a promotion and
        # only leave promotion if it is a promotion to a queen
        if legal_move.promotion is not None:
            if legal_move.promotion == 5:
                filtered_legal_moves.append(legal_move)
            continue
        filtered_legal_moves.append(legal_move)
    return filtered_legal_moves

def legal_moves_to_index_tensors(legal_moves):
    legal_moves_index_tensors = [move_to_index_tensor(legal_move) for legal_move in legal_moves]
    return legal_moves_index_tensors

# The input to the network is a tensor of size 8*8 (it is flattened)
# The output of the network is 64*64 (it is flattened too)
# The size of the hidden layer should be 512

class Network(nn.Module):
    def __init__(self, number_of_actions=64*64):
        super(Network, self).__init__()

        # Fill up the values below in nn.Linear()
        self.layer1 = nn.Linear(64, 512)
        self.layer5 = nn.Linear(512, number_of_actions)

        # Initialization of weights in the layers
        nn.init.xavier_uniform_(self.layer1.weight)
        nn.init.xavier_uniform_(self.layer5.weight)

    def forward(self, x):
        x =  F.relu( self.layer1(x) )
        # Logits will be fed into softmax layer to get probabilities
        # for each move later.
        logits =  self.layer5(x)

        return logits

net = Network(number_of_actions=64*64)

def discount_rewards(collected_moves, gamma=0.99):
    running_reward = 0.0
    for index, collected_move in enumerate(reversed(collected_moves)):
        reward = collected_move[1]
        running_reward = running_reward * gamma + reward
        collected_move[1] = running_reward

def normalize_rewards(collected_moves):
    normalized_rewards = np.asarray(list(map(lambda x: x[1], collected_moves)), dtype=np.float)
    normalized_rewards -= np.mean(normalized_rewards)
    normalized_rewards /= np.std(normalized_rewards)

    for index, collected_move in enumerate(collected_moves):
        collected_move[1] = normalized_rewards[index]

from torch.distributions import Categorical
import random

def get_games_data(policy_net, episodes=100,d=1):
    all_moves = []
    lost_count = 0
    draw_count = 0
    win_count = 0
    game_lengths_sum = 0.0

    for episode in range(episodes):

        engine = chess.engine.SimpleEngine.popen_uci("stockfish-5-linux/Linux/stockfish_14053109_x64")
        engine.configure({"Clear Hash": True})

        board = chess.Board(fen='rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1')
        collected_moves = []
        board_sign = 1
        move_counter = 0.0
        all = ""
        while not board.is_game_over():
            if board_sign == 1:
                # Converting the board to a tensor representation
                board_tensor = board_to_tensor(board).reshape(-1)
                board_tensor_batched = board_tensor.unsqueeze(0)
                # Getting the logits output
                logits = policy_net(board_tensor_batched)
                # Now we need to select only legal moves
                # in python-chess format
                current_legal_moves = filter_legal_moves(list(board.legal_moves))
                legal_moves_index_tensors = legal_moves_to_index_tensors(current_legal_moves)
                legal_moves_logits = logits[:, legal_moves_index_tensors]
                # Here we sample the action using valid logits
                categorical_sampler = Categorical(logits=(legal_moves_logits))
                sampled_action = categorical_sampler.sample()
                sampled_action_move_object = current_legal_moves[sampled_action]
                log_prob = categorical_sampler.log_prob(sampled_action)
                board.push(sampled_action_move_object)
                #print(str(sampled_action_move_object),end = " ")
                all += str(sampled_action_move_object)
                # Board tensor, legal_moves_indexes, sampled_action_label, reward (0 if not known yet)
                collected_moves.append([log_prob, 0.0])
            else:
                result = engine.play(board, chess.engine.Limit(depth=d, nodes=3))
                board.push(result.move)
                all+=" "
                all+= str(result.move)
                all+=" "
                #print(str(result.move),end=" ")

            board_sign = board_sign * -1
            move_counter = move_counter + 1

        #print(f"\n--------------{board.result()}---------------\n")
        with open("Games.txt","a") as file:
          file.write(all)
          file.write(str(board.result()))
          file.write("\n\n\n")
        if board.is_checkmate():
          if board_sign == 1:
            reward = -1.0
            lost_count = lost_count + 1
          else:
            reward = 1.0
            win_count = win_count + 1
        if not board.is_checkmate():
          reward = 0.1
          draw_count = draw_count + 1
        game_lengths_sum = game_lengths_sum + move_counter
        collected_moves[-1][1] = reward
        discount_rewards(collected_moves)
        all_moves.extend(collected_moves)
        engine.quit()

    average_game_length = game_lengths_sum / episodes
    stats = { "lost": lost_count,
              "draw": draw_count,
              "win": win_count,
            }
    normalize_rewards(all_moves)
    return all_moves, stats, win_count

import torch.optim as optim

net = Network(number_of_actions=64*64)
optimizer = optim.Adam(net.parameters(), lr=0.01)

checkpoint = torch.load("/content/drive/My Drive/Checkpoint2Layer.pt")
net.load_state_dict(checkpoint['model_state_dict'])
optimizer.load_state_dict(checkpoint['optimizer_state_dict'])

from livelossplot import PlotLosses
liveloss = PlotLosses()
# Stop it when you are happy with the displayed results
# Each iteration takes a while, be patient
dep=1
while dep<11:
  try:
    collected_moves, stats, wins = get_games_data(policy_net=net, episodes=100,d=dep)
    if int(wins)>70:
      dep+=1
    logs = [collected_move[0] for collected_move in collected_moves]
    rewards = [collected_move[1] for collected_move in collected_moves]
    logs_tensor = torch.cat(logs)
    rewards_tensor = torch.FloatTensor(rewards)
    optimizer.zero_grad()
    policy_loss = -logs_tensor * rewards_tensor
    policy_loss = policy_loss.sum()
    policy_loss.backward()
    optimizer.step()
    liveloss.update(stats)
    liveloss.draw()
  except KeyboardInterrupt:
    break
torch.save({
            'model_state_dict': net.state_dict(),
            'optimizer_state_dict': optimizer.state_dict(),
            }, "Checkpoint.pt")

#torch.save(net.state_dict(),"ChessNet.pt")


title: calculate Entropy for each class of the test set to measure uncertainty on pytorch

body:
I am trying to calculate Entropy of each class of the dataset for an image classification task to measure uncertainty on pytorch,using the MC Dropout method and the solution proposed in this link
Measuring uncertainty using MC Dropout on pytorch
First,I have calculated the mean of each class per batch across different forward passes (class_mean_batch) and then for all the testloader (classes_mean) and then did some transformations to get (total_mean) to use it for calculating Entropy as shown in the code below
def mcdropout_test(batch_size,n_classes,model,T):

    #set non-dropout layers to eval mode
    model.eval()

    #set dropout layers to train mode
    enable_dropout(model)

    softmax = nn.Softmax(dim=1)
    classes_mean = []

    for images,labels in testloader:
        images = images.to(device)
        labels = labels.to(device)
        classes_mean_batch = []

        with torch.no_grad():
          output_list = []

          #getting outputs for T forward passes
          for i in range(T):
            output = model(images)
            output = softmax(output)
            output_list.append(torch.unsqueeze(output, 0))

        concat_output = torch.cat(output_list,0)

        # getting mean of each class per batch across multiple MCD forward passes
        for i in range (n_classes):
          mean = torch.mean(concat_output[:, : , i])
          classes_mean_batch.append(mean)

        # getting mean of each class for the testloader
        classes_mean.append(torch.stack(classes_mean_batch))

    total_mean = []
    concat_classes_mean = torch.stack(classes_mean)

    for i in range (n_classes):
      concat_classes = concat_classes_mean[: , i]
      total_mean.append(concat_classes)

    total_mean = torch.stack(total_mean)
    total_mean = np.asarray(total_mean.cpu())

    epsilon = sys.float_info.min
    # Calculating entropy across multiple MCD forward passes
    entropy = (- np.sum(total_mean*np.log(total_mean + epsilon), axis=-1)).tolist()
    for i in range(n_classes):
      print(f'The uncertainty of class {i+1} is {entropy[i]:.4f}')

Can anyone please correct or confirm the implementation i have used to calculate Entropy of each class.


title: Why do we need the custom dataset class and use of _getitem_ method in NLP, BERT fine tuning etc

body:
I am a newbie in NLP and has been studying the usage of BERT for NLP tasks. In many notebooks, I See that a custom dataset class is defined and getitem method is defined (along with len).
Tweetdataset class in this notebook - https://www.kaggle.com/abhishek/roberta-inference-5-folds
and text_Dataset class in this notebook - https://engineering.wootric.com/when-bert-meets-pytorch
Can some one please explain the reason, need for defining the custom dataset class and the getitem (and len) method. thank you


title: Use Libtorch on Mac - can't figure out how to include torch

body:
at this point I'm actually frustrated, because I can't figure out how to use Libtorch. I hope this is not a question too dumb to be answered. So I tried following this tutorial (https://pytorch.org/cppdocs/installing.html), but I just can not figure out how to include libtorch into projects in IDEs (e.g. CLion). Even modifying the CMake file doesn't work out for me, as I try to #include<torch/torch.h> the compiler just won't find it.
Can anybody give me a step by step guide or maybe knows some on the internet on how to use libtorch for dummies? Thanks so much.
(Using MacOS btw)


title: PyTorch: Finding variable needed for gradient computation that has been modified by inplace operation - Multitask Learning

body:
I recently did a massive refactor to my PyTorch LSTM code, in order to support multitask learning. I created an MTLWrapper, which holds a BaseModel (which can be one of several variations on a regular LSTM network), which remained the same as it was before the refactor, minus a linear hidden2tag layer (takes hidden sequence and converts to tag space), which now sits in the wrapper. The reason for this is that for multitask learning, all the parameters are shared, except for the final linear layer, which I have one of for each task. These are stored in a nn.ModuleList, not just a regular python list.
What happens now is that my forward pass returns a list of tag scores tensors (one for each task), rather than a single tensor of the tag scores for a single task. I compute the losses for each of these tasks and then try to backpropagate with the average of these losses (technically also averaged over all the sentences of a batch, but this was true before the refactor too). I call model.zero_grad() before running the forward pass on each sentence in a batch.
I don't know exactly where it happened, but after this refactor, I started getting this error (on the second batch):

RuntimeError: Trying to backward through the graph a second time, but
the buffers have already been freed. Specify retain_graph=True when
calling backward the first time.

Following the advice, I added the retain_graph=True flag, but now I get the following error instead (also on the second backward step):

RuntimeError: one of the variables needed for gradient computation has
been modified by an inplace operation: [torch.FloatTensor [100, 400]],
which is output 0 of TBackward, is at version 2; expected version 1
instead. Hint: the backtrace further above shows the operation that
failed to compute its gradient. The variable in question was changed
in there or anywhere later. Good luck!

The hint in the backtrace is not actually helpful, because I have no idea where a tensor of the shape [100, 400] even came from - I don't have any parameters of size 400.
I have a sneaky suspicion that the problem is actually that I shouldn't need the retain_graph=True, but I have no way to confirm that vs. finding the mystery variable that is being changed according to the second error. Either way, I'm at a complete loss how to solve this issue. Any help is appreciated!
Code snippets:
import torch
import torch.nn as nn
import torch.nn.functional as F

class MTLWrapper(nn.Module):

    def __init__(self, embedding_dim, hidden_dim, dropout,..., directions=1, device='cpu', model_type):
        super(MTLWrapper, self).__init__()
        self.base_model = model_type(embedding_dim, hidden_dim, dropout, ..., directions, device)
        self.linear_taggers = []
        for tagset_size in tagset_sizes:
            self.linear_taggers.append(nn.Linear(hidden_dim*directions, tagset_size))
        self.linear_taggers = nn.ModuleList(self.linear_taggers)

    def init_hidden(self, hidden_dim):
        return self.base_model.init_hidden(hidden_dim)

    def forward(self, sentence):
        lstm_out = self.base_model.forward(sentence)
        tag_scores = []
        for linear_tagger in self.linear_taggers:
            tag_space = linear_tagger(lstm_out.view(len(sentence), -1))
            tag_scores.append(F.log_softmax(tag_space))
        tag_scores = torch.stack(tag_scores)
        return tag_scores

Inside the train function:
for i in range(math.ceil(len(train_sents)/batch_size)):
    batch = r[i*batch_size:(i+1)*batch_size]
    losses = []
    for j in batch:

        sentence = train_sents[j]
        tags = train_tags[j]

        # Step 1. Remember that Pytorch accumulates gradients.
        # We need to clear them out before each instance
        model.zero_grad()

        # Also, we need to clear out the hidden state of the LSTM,
        # detaching it from its history on the last instance.
        model.hidden = model.init_hidden(hidden_dim)

        sentence_in = sentence
        targets = tags

        # Step 3. Run our forward pass.
        tag_scores = model(sentence_in)

        loss = [loss_function(tag_scores[i], targets[i]) for i in range(len(tag_scores))]
        loss = torch.stack(loss)
        avg_loss = sum(loss)/len(loss)
        losses.append(avg_loss)
losses = torch.stack(losses)
total_loss = sum(losses)/len(losses)  # average over all sentences in batch
total_loss.backward(retain_graph=True)
running_loss += total_loss.item()
optimizer.step()
count += 1

And code for one possible BaseModel (the others are practically identical):
class LSTMTagger(nn.Module):

def __init__(self, embedding_dim, hidden_dim, dropout, vocab_size, alphabet_size,
             directions=1, device='cpu'):

    super(LSTMTagger, self).__init__()
    self.device = device

    self.hidden_dim = hidden_dim
    self.directions = directions
    self.dropout = nn.Dropout(dropout)

    self.word_embeddings = nn.Embedding(vocab_size, embedding_dim)

    # The LSTM takes word embeddings as inputs, and outputs hidden states
    # with dimensionality hidden_dim.
    self.lstm = nn.LSTM(embedding_dim, hidden_dim, dropout=dropout, bidirectional=directions == 2)

    # The linear layer that maps from hidden state space to tag space
    self.hidden = self.init_hidden(hidden_dim)

def init_hidden(self, dim):
    # Before we've done anything, we don't have any hidden state.
    # Refer to the PyTorch documentation to see exactly
    # why they have this dimensionality.
    # The axes semantics are (num_layers, minibatch_size, hidden_dim)
    return (torch.zeros(self.directions, 1, dim).to(device=self.device),
            torch.zeros(self.directions, 1, dim).to(device=self.device))

def forward(self, sentence):
    word_idxs = []
    for word in sentence:
        word_idxs.append(word[0])

    embeds = self.word_embeddings(torch.LongTensor(word_idxs).to(device=self.device))

    lstm_out, self.hidden = self.lstm(
        embeds.view(len(sentence), 1, -1), self.hidden)
    lstm_out = self.dropout(lstm_out)
    return lstm_out


title: Reusing oversized pinned memory buffers with PyTorch

body:
I'm trying to improve the performance of my PyTorch code by using a preallocated page locked tensor to copy all my results from the GPU into. The results from the GPU will vary in size in the first dimension so I created a pinned buffer large enough to handle the max sized result.
What is the most performant way to copy the undersized results in the pinned memory and then to numpy arrays of their original size?
import torch

channels = 8
maxsize = 10

# create a page locked buffer on the cpu of masize in first dimension
buf = torch.zeros([maxsize, channels], pin_memory=True)

# create smaller tensors on the gpu
data = [
    torch.empty((i, channels), device="cuda").fill_(i)
    for i in range(1, 6)
]

model = lambda x: x
results = []

for res in data:

    # run dummy model
    res = model(res)

    # best way to async copy the smaller res tensor from the gpu?
    buf[:res.shape[0], :] = res

    # check buffer looks correct
    print(buf[:, 0], buf.data_ptr())

    # do something with the correct sized results
    results.append(buf[:res.shape[0], :].clone().numpy())

# check results look correct
for res in results:
    print(res[:, 0])

tensor([1., 0., 0., 0., 0., 0., 0., 0., 0., 0.]) 140124862021632
tensor([2., 2., 0., 0., 0., 0., 0., 0., 0., 0.]) 140124862021632
tensor([3., 3., 3., 0., 0., 0., 0., 0., 0., 0.]) 140124862021632
tensor([4., 4., 4., 4., 0., 0., 0., 0., 0., 0.]) 140124862021632
tensor([5., 5., 5., 5., 5., 0., 0., 0., 0., 0.]) 140124862021632
[1.]
[2. 2.]
[3. 3. 3.]
[4. 4. 4. 4.]
[5. 5. 5. 5. 5.]


title: EasyOCR used under Python / Torch Multiprocessing is defaulting to CPU

body:
I am using EasyOCR for text extraction from images. It uses PyTorch. There are multiple images in different folders and the sequence in which these folders are read isn't consequential.
When run in sequence, EasyOCR is by default using GPU and is faster compared to when run on CPU. But when Python / Torch Multiprocessing is invoked, so that the multiple folders are read in parallel, EasyOCR is defaulting to CPU.
torch.cuda.is_available returns False.
How can I solve this?


title: Is is possible to finetune SimCLR on an instance segmentation task using PyTorch Lightning Bolts?

body:
I have a dataset of high-resolution drone images, which I have split into tiles of size 512x512. The tiles will be annotated by delineating a specific type of vegetation and stored in COCO format. I would like to use self-supervised learning for instance segmentation in the tiles. The Pytorch Lightning bolts module provides already pre-trained self-supervised models, such as SimCLR. According to the bolts module documentation, it should be possible to finetune SimCLR on any computer vision task, such as instance segmentation, using bolts.
The only examples I found, however, show how SimCLR is finetuned on classification tasks. How would one finetune SimCLR (or another self-supervised model) on an instance segmentation task?


title: What is the difference between Neural Network Frameworks and RL Algorithm Libraries?

body:
I know this is a silly question, but I cannot find a good way to put it.
I've worked with TensorFlow and TFAgents, and am now moving to Ray RLlib. Looking at all the RL frameworks/libraries, I got confused about the difference between the two below:

frameworks such as Keras, TensorFlow, PyTorch
RL implementation libraries such as TFAgents, RLlib, OpenAi Baseline, Tensorforce, KerasRL, etc

For example, there are Keras codes in TensorFlow and Ray RLlib supports both TensorFlow and PyTorch. How are they all related?
My understanding so far is that Keras allows to make neural networks and TensorFlow is more of a math library for RL (I don't have enough understanding about PyTorch). And libraries like TFAgents and RLlib use frameworks like Keras and TensorFlow to implement existing RL algorithms so that programmers can utilize them with ease.
Can someone please explain how they are interconnected/different? Thank you very much.


title: Multi-step time series forecasting

body:
I have standard many-to-one different RNN models which I used for 1 target time series prediction using other time series as features. I use 14 as the input sequence length and 1 value which corresponds to the target time series at the time moment which corresponds to the end of the input sequence. I want to generalize this approach to have some forecast horizon and predict some predefined steps ahead. Then, I would construct my training batches as follows(with forecast horizon = 2 and in_seq_len = 4):
 . . ./. ./. . . . .
       |
/0 0 0 0/0 0 0 0 0 0

The next sample is shifted by one position:
  . . . ./. ./. . . .
          |
  0/0 0 0 0/0 0 0 0 0

As expected, I train the network to predict the sequence instead of just one value. My question is: How should I get the final predictions of the network if I got the output of my network of shape [batch_size, forecast_horizon]. The output tensors will be overlapping because the samples are shifted by one position as Pytorch RNN expects it to be. Should I take only the last samples as the network predictions? i.e. [batch_size, -1]


title: How to Create Scikitlearn or Pytorch like Pipeline Design Pattern in Python?

body:
I was studying the sklearn Pipeline object and also Pytorch nn module. In both cases it is possible to attach other object in to the constructor subclass of Pytorch or add a tuple in the list of Pipeline object. So when the pipeline is executed it runs the whole transformation sequentially both for Pytorch and slkearn. What I understood on studying sklearn Pipeline doc that each pipeline object should implement an interface which should have fit, transform and fit_transform methods. Then later adding that in the Pipeline the objects can run as regular pipeline object. My Question which kind of Design Pattern is this and How I can create my own module like that. The sklearn repository is so big I am just lost where to find what? . For example I can create a New custom Transformer class such as:
class MultiColumnEncoder:
    def __init__(self, columns=None):
        """Multicolumn Encoder class encode the provided columns in to integer"""
        self.columns = columns

    def fit(self, df: pd.DataFrame, y=None):
        """Just a Dummy Return"""
        # In proper class definition that should be a fit
        ....
        ....
        return self

    def transform(self, df: pd.DataFrame):
        """
        Transforms columns of X specified in self.columns using
        LabelEncoder(). If no columns specified, transforms all
        columns in X.
        """
       ....
       ....
       return transformed_df

    def fit_transform(self, df: pd.DataFrame, y=None):
        .....
        .....
        return self.fit(df).transform(df)

Now as it satisfies the Pipiline interface in can use in my pipeline along with other transformer:
preprocess_pipeline = Pipeline([
    ('encoding', MultiColumnEncoder(columns=['column_1', 'column_2']),
    ('stdscaler', StandardScaler())
])

For pytorch I can add each sequential operation like convulation, dorp out, batch norm in the the __init__ method and the forward method will run them sequentially. So which kind of design pattern is needed to design such architecture? I have searched several resources but have not got the answer.
My final goal is to create my own Pipeline with several functionality like transform, fit, predict, plot and stack them together to make a pipeline. Such as:
pipeline = MyOwnPipeline([
    ('name1', Object1(enable=True),
    ('name2', Object2(enable=False),
    ('name3', Object3(enable=True)])
    ('model', Model()])
    ('persist', Plot(enable=True)])

Based on enable parameter it should run that Transformer on the data, other wise not.


title: Pytorch got CUDA error: device-side assert triggered when training conv1d classifier

body:
I'm implementing a CNN multi-label classifier with PyTorch, while it always shows this error:
"CUDA error: device-side assert triggered". In the error message it points out the loss function, but when I change a loss function, it still occurs and points to the other parts (really like randomly picked). When I changed to CPU, it said "index out of range in self", however when I investigated my dataloader, it is nothing weird.
I have 15 classes, 59462 unique tokens, and the len of each document is 30,000(token).
My model and loss function is like this:

class model(nn.Module):
    def __init__(self, num_classes=15):
        super(model, self).__init__()

        self.embedding = nn.Sequential(nn.Embedding(59462,400),nn.Dropout(0.15))
        self.features = nn.Sequential(
            nn.Conv1d(400, 500, kernel_size=3, stride=1, padding=False), nn.ReLU(),
            nn.Dropout(0.05), nn.MaxPool1d(kernel_size=2), nn.Dropout(0.15))

        self.linear = nn.Linear(500*14999, 15)

    def forward(self, x):
        x = self.embedding(x)
        x = x.permute(0,2,1)
        x = self.features(x)
        x = x.view(x.size(0), 500*14999)
        x = self.linear(x)

        return x

model = model()
model = model.to(device)

def loss_fn(outputs, targets):
    return torch.nn.BCEWithLogitsLoss(pos_weight=pos_weight.to(device))(outputs, targets).to(device)

#pos_weight is for my unbalanced data

This is the error messsage when using CPU:
/usr/local/lib/python3.6/dist-packages/IPython/core/interactiveshell.py in run_cell_magic(self, magic_name, line, cell)
   2115             magic_arg_s = self.var_expand(line, stack_depth)
   2116             with self.builtin_trap:
-> 2117                 result = fn(magic_arg_s, cell)
   2118             return result
   2119

<decorator-gen-60> in time(self, line, cell, local_ns)

/usr/local/lib/python3.6/dist-packages/IPython/core/magic.py in <lambda>(f, *a, **k)
    186     # but it's overkill for just that one bit of state.
    187     def magic_deco(arg):
--> 188         call = lambda f, *a, **k: f(*a, **k)
    189
    190         if callable(arg):

/usr/local/lib/python3.6/dist-packages/IPython/core/magics/execution.py in time(self, line, cell, local_ns)
   1191         else:
   1192             st = clock2()
-> 1193             exec(code, glob, local_ns)
   1194             end = clock2()
   1195             out = None

<timed exec> in <module>()

<ipython-input-67-09829392983d> in train_epoch(model, data_loader, loss_fn, optimizer, device, scheduler, n_examples)
     27         targets = d["targets"].to(device)
     28         outputs = model(
---> 29           tokens
     30         )
     31

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

<ipython-input-59-2c2ec554cb05> in forward(self, x)
     16
     17     def forward(self, x):
---> 18         x = self.embedding(x)
     19         x = x.permute(0,2,1)
     20         x = self.features(x)

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/container.py in forward(self, input)
    115     def forward(self, input):
    116         for module in self:
--> 117             input = module(input)
    118         return input
    119

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/sparse.py in forward(self, input)
    124         return F.embedding(
    125             input, self.weight, self.padding_idx, self.max_norm,
--> 126             self.norm_type, self.scale_grad_by_freq, self.sparse)
    127
    128     def extra_repr(self) -> str:

/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
   1850         # remove once script supports set_grad_enabled
   1851         _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-> 1852     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
   1853
   1854

IndexError: index out of range in self

Does anyone know what this problem is, and how I can solve it?
Thanks in advance.


title: Pytorch-size mismatch when loading the pretrained model

body:
Could someone help me with this issue as I am unable to get this right.
I want to load my pretrained models for reinforcement learning. That's how I save and load the model:
def save(self, folder_to_save='./'):
    if folder_to_save[-1] != '/':
        folder_to_save = folder_to_save + '/'
    torch.save(self.enc.state_dict(), folder_to_save + 'enc.model')
    torch.save(self.dec.state_dict(), folder_to_save + 'dec.model')
    torch.save(self.lp.state_dict(), folder_to_save + 'lp.model')

    pickle.dump(self.lp.order, open(folder_to_save + 'order.pkl', 'wb'))

def load(self, folder_to_load='./'):
    if folder_to_load[-1] != '/':
        folder_to_load = folder_to_load + '/'

    order = pickle.load(open(folder_to_load + 'order.pkl', 'rb'))
    self.lp = LP(distr_descr=self.latent_descr + self.feature_descr,
                 tt_int=self.tt_int, tt_type=self.tt_type,
                 order=order)

    self.enc.load_state_dict(torch.load(folder_to_load + 'enc.model'))
    self.dec.load_state_dict(torch.load(folder_to_load + 'dec.model'))
    self.lp.load_state_dict(torch.load(folder_to_load + 'lp.model'))

and in .ipynb files I use:
model.save('./saved_gentrl/')
enc = gentrl.RNNEncoder(latent_size=50)
dec = gentrl.DilConvDecoder(latent_input_size=50)
model = gentrl.GENTRL(enc, dec, 50 * [('c', 20)], [('c', 20)], beta=0.001)
model.cuda();
model.load('saved_gentrl/')
model.cuda();

But the error occurs:
RuntimeError: Error(s) in loading state_dict for RNNEncoder:size mismatch for embs.weight: copying a param with shape torch.Size([28, 256]) from checkpoint, the shape in current model is torch.Size([31, 256]).

The traceback is:
/Colab Notebooks/GENTRL-master/gentrl/gentrl.py in load(self, folder_to_load)
114                      order=order)
115
116         self.enc.load_state_dict(torch.load(folder_to_load + 'enc.model'))
117         self.dec.load_state_dict(torch.load(folder_to_load + 'dec.model'))
118         self.lp.load_state_dict(torch.load(folder_to_load + 'lp.model'))

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in load_state_dict(self, state_dict, strict)

1050         load = None  # break load->load reference cycle
1052         return _IncompatibleKeys(missing_keys, unexpected_keys)
1053
1054     def _named_members(self, get_members_fn, prefix='', recurse=True):

I guess this may because I added 3 more features in token list, which lead vocab_size change from 28 to 31, but this parameter only use as：
self.embs = nn.Embedding(get_vocab_size(), hidden_size)
self.vocab_size = get_vocab_size()

I can not figure out why, please help me.
Thanks!


title: Casting Pytorch's tensor elements the type "float" instead of "double"

body:
I had a matrix saved as a numpy type, call it "X_before" (for example, its shape is 100*30).
Since I want to feed it to an AutoEncoder using Pytorch library, I converted it to torch.tensor like this:
X_tensor = torch.from_numpy(X_before, dtype=torch)

Then, I got the following error:

expected scalar type Float but found Double

Next, I tried to make elements as "float" and then convert them torch.tensor:
X_before = X_before.astype(float)
X_tensor = torch.from_numpy(X_before)

Again, the same error happens.
How should I solve this issue? How can I convert the type of elements in a torch.tensor object to another type?
Thanks in advance


title: Pytorch Conv1d on simple 1d signal

body:
1d-convolution is pretty simple when it is done by hand. However, I want to implement what is done here using nn.Conv1d and it is not simple for me to do it. In this example h=[1,2,-1], x=[4,1,2,5] and the output is going to be y=[4,9,0,8,8,-5]. To do it using Pytorch we need to define h=nn.Conv1d(in, out, k) and x=torch.tensor(*) and y=h(x) should be the result.
Note: please do not use nn.Conv2d to implement it.


title: CNN Classifier only guesses one thing - PyTorch

body:
I'm trying to make a model predict the race of a 75x75 image's ethnicity, but when ever I train the model, the accuracy always stays completely still at 53.2%. I didn't realize why until I actually ran it on some of photos. It turned out, that no matter what the photo was, it would always predict 'other'. I'm not entirely sure why though.
I copied the code over from the official PyTorch Quickstart tutorial, and in that dataset or the standard MNIST data, it worked fine. I changed the dataset to the UTKFace, and then it started only predicting one label, all the time.
Here's my code:
import torch
from torch import nn
from torch.utils.data import DataLoader
from torchvision.datasets import ImageFolder
from torchvision.transforms import ToTensor
import torch.nn.functional as F

training_data = ImageFolder(
    root = "data_training/",
    transform = ToTensor(),
)

testing_data = ImageFolder(
    root = "data_testing/",
    transform = ToTensor()
)

training_dataloader = DataLoader(training_data, batch_size=64, shuffle=True)
test_dataloader = DataLoader(testing_data, batch_size=64, shuffle=True)

class NeuralNetwork(nn.Module):
    def __init__(self):
        super(NeuralNetwork, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)

        self.fc1 = nn.Linear(1296, 1024)
        self.fc2 = nn.Linear(1024, 1024)
        self.fc3 = nn.Linear(1024, 512)
        self.fc4 = nn.Linear(512, 84)
        self.fc5 = nn.Linear(84, 5)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))

        x = x.view(x.size(0), -1)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.relu(self.fc3(x))
        x = F.relu(self.fc4(x))

        x = self.fc5(x)
        return x

model = NeuralNetwork().to("cpu")
loss_fn = nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=1e-3)

def train(dataloader, model, loss_fn, optimizer):
    size = len(dataloader.dataset)
    for batch, (X, y) in enumerate(dataloader):
        X, y = X.to("cpu"), y.to("cpu")
        # Compute prediction error
        pred = model(X)
        loss = loss_fn(pred, y)

        # Backpropagation
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if batch % 100 == 0:
            loss, current = loss.item(), batch * len(X)
            print(f"loss: {loss:>7f}  [{current:>5d}/{size:>5d}]")

def tests(dataloader, model):
    size = len(dataloader.dataset)
    model.eval()
    test_loss, correct = 0, 0
    with torch.no_grad():
        for X, y in dataloader:
            X, y = X.to("cpu"), y.to("cpu")
            pred = model(X)
            test_loss += loss_fn(pred, y).item()
            correct += (pred.argmax(1) == y).type(torch.float).sum().item()
    test_loss /= size
    correct /= size
    print(f"Test Error: \n Accuracy: {(100*correct):>0.1f}%, Avg loss: {test_loss:>8f} \n")

epochs = 10
for t in range(epochs):
    print(f"Epoch {t+1}\n-------------------------------")
    train(training_dataloader, model, loss_fn, optimizer)
    tests(test_dataloader, model)

torch.save(model.state_dict(), "model.pth")

The training logs:
Epoch 1
-------------------------------
loss: 1.628994  [    0/23705]
loss: 1.620698  [ 6400/23705]
loss: 1.615423  [12800/23705]
loss: 1.596390  [19200/23705]
Test Error:
 Accuracy: 53.2%, Avg loss: 0.024725

Epoch 2
-------------------------------
loss: 1.593613  [    0/23705]
loss: 1.581375  [ 6400/23705]
loss: 1.583656  [12800/23705]
loss: 1.591942  [19200/23705]
Test Error:
 Accuracy: 53.2%, Avg loss: 0.024165

Epoch 3
-------------------------------
loss: 1.541260  [    0/23705]
loss: 1.592345  [ 6400/23705]
loss: 1.540908  [12800/23705]
loss: 1.540741  [19200/23705]
Test Error:
 Accuracy: 53.2%, Avg loss: 0.023705

Epoch 4
-------------------------------
loss: 1.566888  [    0/23705]
loss: 1.524875  [ 6400/23705]
loss: 1.540764  [12800/23705]
loss: 1.510044  [19200/23705]
Test Error:
 Accuracy: 53.2%, Avg loss: 0.023323

Epoch 5
-------------------------------
loss: 1.530084  [    0/23705]
loss: 1.498773  [ 6400/23705]
loss: 1.537755  [12800/23705]
loss: 1.508989  [19200/23705]
Test Error:
 Accuracy: 53.2%, Avg loss: 0.022993
....

No matter how many epochs I set it to, or how many layers I add in to try to get it to overfit, it always just seems to guess the same thing over and over again, with no signs of improvement.
I separated the UTKFace dataset into folders based on the ethnicity category of the name. There are 23705 images in the training data and 10134 in the testing.
I'm not sure why this is happening. Is my dataset not large enough? Are there not enough layers?


Can't trace the model using torch.jit.trace. This is a resnet 101 based segmentation model.
I am using python 3.7, torch 1.8, rtx 3070 8gb.
My code:
Net=FCN.Net(CatDic.CatNum)
Net.load_state_dict(torch.load('./model.torch', map_location=torch.device('cuda')), strict=False)
Net.eval()

c = torch.jit.trace(Net, torch.randn(1, 640, 640, 3).cuda())

My neural network structure:
class Net(nn.Module):
    def __init__(self, CatDict):
            super(Net, self).__init__()
            self.Encoder = models.resnet101(pretrained=True)
            self.PSPScales = [1, 1 / 2, 1 / 4, 1 / 8]

            self.PSPLayers = nn.ModuleList()
            for Ps in self.PSPScales:
                self.PSPLayers.append(nn.Sequential(
                    nn.Conv2d(2048, 1024, stride=1, kernel_size=3, padding=1, bias=True)))
            self.PSPSqueeze = nn.Sequential(
                nn.Conv2d(4096, 512, stride=1, kernel_size=1, padding=0, bias=False),
                nn.BatchNorm2d(512),
                nn.ReLU(),
                nn.Conv2d(512, 512, stride=1, kernel_size=3, padding=0, bias=False),
                nn.BatchNorm2d(512),
                nn.ReLU()
            )

            self.SkipConnections = nn.ModuleList()
            self.SkipConnections.append(nn.Sequential(
                nn.Conv2d(1024, 512, stride=1, kernel_size=1, padding=0, bias=False),
                nn.BatchNorm2d(512),
                nn.ReLU()))
            self.SkipConnections.append(nn.Sequential(
                nn.Conv2d(512, 256, stride=1, kernel_size=1, padding=0, bias=False),
                nn.BatchNorm2d(256),
                nn.ReLU()))
            self.SkipConnections.append(nn.Sequential(
                nn.Conv2d(256, 256, stride=1, kernel_size=1, padding=0, bias=False),
                nn.BatchNorm2d(256),
                nn.ReLU()))
            # ------------------Skip squeeze applied to the (concat of upsample+skip conncection layers)-----------------------------------------------------------------------------
            self.SqueezeUpsample = nn.ModuleList()
            self.SqueezeUpsample.append(nn.Sequential(
                nn.Conv2d(1024, 512, stride=1, kernel_size=1, padding=0, bias=False),
                nn.BatchNorm2d(512),
                nn.ReLU()))
            self.SqueezeUpsample.append(nn.Sequential(
                nn.Conv2d(256 + 512, 256, stride=1, kernel_size=1, padding=0, bias=False),
                nn.BatchNorm2d(256),
                nn.ReLU()))
            self.SqueezeUpsample.append(nn.Sequential(
                nn.Conv2d(256 + 256, 256, stride=1, kernel_size=1, padding=0, bias=False),
                nn.BatchNorm2d(256),
                nn.ReLU()))

            self.OutLayersList =nn.ModuleList()
            self.OutLayersDict={}
            for f,nm in enumerate(CatDict):
                    self.OutLayersDict[nm]= nn.Conv2d(256, 2, stride=1, kernel_size=3, padding=1, bias=False)
                    self.OutLayersList.append(self.OutLayersDict[nm])

    def forward(self,Images, UseGPU = True, TrainMode=False, FreezeBatchNormStatistics=False):
                RGBMean = [123.68,116.779,103.939]
                RGBStd = [65,65,65]
                if TrainMode:
                        tp=torch.FloatTensor
                else:
                    self.half()
                    tp=torch.HalfTensor
                    self.eval()
                #InpImages = torch.autograd.Variable(torch.from_numpy(Images), requires_grad=False).transpose(2,3).transpose(1, 2).type(torch.FloatTensor)

                InpImages = torch.autograd.Variable(Images, requires_grad=False).transpose(2,3).transpose(1, 2).type(tp)
                if FreezeBatchNormStatistics==True: self.eval()
                if UseGPU:
                    InpImages=InpImages.cuda()
                    self.cuda()
                else:
                    self=self.cpu()
                    self.float()
                    InpImages=InpImages.type(torch.float).cpu()
                for i in range(len(RGBMean)): InpImages[:, i, :, :]=(InpImages[:, i, :, :]-RGBMean[i])/RGBStd[i] # normalize image values
                x=InpImages
                SkipConFeatures=[] # Store features map of layers used for skip connection
                x = self.Encoder.conv1(x)
                x = self.Encoder.bn1(x)
                x = self.Encoder.relu(x)
                x = self.Encoder.maxpool(x)
                x = self.Encoder.layer1(x)
                SkipConFeatures.append(x)
                x = self.Encoder.layer2(x)
                SkipConFeatures.append(x)
                x = self.Encoder.layer3(x)
                SkipConFeatures.append(x)
                x = self.Encoder.layer4(x)
                PSPSize=(x.shape[2],x.shape[3]) # Size of the original features map

                PSPFeatures=[] # Results of various of scaled procceessing
                for i,PSPLayer in enumerate(self.PSPLayers): # run PSP layers scale features map to various of sizes apply convolution and concat the results
                      NewSize=(np.array(PSPSize)*self.PSPScales[i]).astype(np.int)
                      if NewSize[0] < 1: NewSize[0] = 1
                      if NewSize[1] < 1: NewSize[1] = 1

                      y = nn.functional.interpolate(x, tuple(NewSize), mode='bilinear')
                      y = PSPLayer(y)
                      y = nn.functional.interpolate(y, PSPSize, mode='bilinear')
                      PSPFeatures.append(y)
                x=torch.cat(PSPFeatures,dim=1)
                x=self.PSPSqueeze(x)
                for i in range(len(self.SkipConnections)):
                  sp=(SkipConFeatures[-1-i].shape[2],SkipConFeatures[-1-i].shape[3])
                  x=nn.functional.interpolate(x,size=sp,mode='bilinear') #Resize
                  x = torch.cat((self.SkipConnections[i](SkipConFeatures[-1-i]),x), dim=1)
                  x = self.SqueezeUpsample[i](x)

                self.OutLbDict = {}

                ret_arr = np.eye(640, 640)
                for nm in self.OutLayersDict:
                  l=self.OutLayersDict[nm](x)
                  l = nn.functional.interpolate(l,size=InpImages.shape[2:4],mode='bilinear') # Resize to original image size
                  tt, Labels = l.max(1)  # Find label per pixel
                  self.OutLbDict[nm] = Labels
                  array = np.asarray(self.OutLbDict[nm].cpu())
                  resx = np.reshape(array, ((array.shape)[2], (array.shape)[1]))
                  ret_arr = list(ret_arr + resx*10)

                return ret_arr

I get an error:
RuntimeError: Tracer cannot infer type of [array([..])]
:Could not infer type of list element: Only tensors and (possibly nested) tuples of tensors, lists, or dictsare supported as inputs or outputs of traced functions, but instead got value of type ndarray.

If I remove all numpy arrays from the code, then I get a different error:
C:\anaconda3\lib\site-packages\torch\jit_trace.py in _check_trace(check_inputs, func, traced_func, check_tolerance, strict, force_outplace, is_trace_module, _module_class)
517         diag_info = graph_diagnostic_info()
518         if any(info is not None for info in diag_info):
--> 519             raise TracingCheckError(*diag_info)
520
521
TracingCheckError: Tracing failed sanity checks!
ERROR: Graphs differed across invocations!
    Graph diff:
          graph(%self.1 : __torch__.FCN_NetModel.Net,
                %Images : Tensor):
            %2 : __torch__.torch.nn.modules.container.ModuleList = prim::GetAttr[name="SqueezeUpsample"](%self.1)
            %3 : __torch__.torch.nn.modules.container.Sequential = prim::GetAttr[name="2"](%2)
            %4 : __torch__.torch.nn.modules.container.ModuleList = prim::GetAttr[name="SkipConnections"](%self.1)
            %5 : __torch__.torch.nn.modules.container.Sequential = prim::GetAttr[name="2"](%4)

........

The interesting thing is that if I run c = torch.jit.trace (Net, torch.randn (1, 640, 640, 3) .cuda ()) again, the last error does not occur and the tracing is successful. But this traced model doesn't work. I would be grateful for your help.


title: In Pytorch, is there a difference between (x<0) and x.lt(0)?

body:
Suppose x is a tensor in Pytorch. One can either write:
x_lowerthanzero = x.lt(0)

or:
x_lowerthanzero = (x<0)

with seemingly the exact same results. Many other operations have Pytorch built-in equivalents: x.gt(0) for (x>0), x.neg() for -x, x.mul() etc.
Is there a good reason to use one form over the other?


title: Odd behaviour of "out" argument in torch.empty()

body:
I have a sample tensor like this:

In [137]: x = x.new_ones((5, 3), dtype=torch.double)
In [138]: x
Out[138]:
tensor([[1., 1., 1.],
        [1., 1., 1.],
        [1., 1., 1.],
        [1., 1., 1.],
        [1., 1., 1.]], dtype=torch.float64)

Now, I want to free the memory of this tensor by overwriting the contents using torch.empty() which takes an out argument.

In [139]: torch.empty((5, 3), out=x)
Out[139]:
tensor([[1., 1., 1.],
        [1., 1., 1.],
        [1., 1., 1.],
        [1., 1., 1.],
        [1., 1., 1.]], dtype=torch.float64)

However, the values in the original tensor x still remains the same. If this is the case, then what is the purpose of this keyword argument out in torch.empty? What am I missing here?


title: Pytorch : significance of grad_h_relu.clone() in Backprop

body:
I am learning Pytorch, while I'm looking to the tutorial on their site I can't understand the significance of grad_h = grad_h_relu.clone().

h = x.mm(w1)
h_relu = h.clamp(min=0)
y_pred = h_relu.mm(w2)

# Compute and print loss
loss = (y_pred - y).pow(2).sum().item()
print(t, loss)

# Backprop to compute gradients of w1 and w2 with respect to loss
grad_y_pred = 2.0 * (y_pred - y)
grad_w2 = h_relu.t().mm(grad_y_pred)
grad_h_relu = grad_y_pred.mm(w2.t())

grad_h = grad_h_relu.clone() # what is the signifigance of this line?

grad_h[h < 0] = 0
grad_w1 = x.t().mm(grad_h)

# Update weights using gradient descent
w1 -= learning_rate * grad_w1
w2 -= learning_rate * grad_w2


title: how to multiply all rows by column of single matrix in pytorch in a vectorized way

body:
I need to multiply all rows of a matrix by column, i think with an example it will be clearer:

matrix is:

1,2,3
4,5,6
7,8,9

An i need an operation that returns:

28,80,162

But i can't find anything in the documentation and blogs and other SO question  only are related to matrix multiplication and dot product, which is not what i need in this case,how can it be achieved in a vectorized fashion(instead of for loop based) ?

For example this is easy to achieve for the case of sum like:

the_matrix.sum(dim=0)

But there's not something like:

the_matrix.mul(dim=0)


title: How to parallelize model prediction from a pytorch model?

body: I have a PyTorch instance segmentation model that predicts the masks of images one by one. Is there any way to parallelize this task so that it can be completed faster. I have tried to use multiprocessing pool.apply_async() to call the method that performs prediction passing the required arguments, but it throws me segmentation fault. Any help is much appreciated. Thanks.


title: LPPool with trainable norm type easily result in nan gradient

body:
I want to have nn.LPPool2d(norm_type=xxx) decide the proper norm_type itself by making norm_type trainable.

After a lot of experiments, I found that the gradient of norm_type can easily become nan even in the simplest case with single precision (not half). Here is the reproducing code:

import torch
import torch.nn
import torch.nn.functional as F

x = torch.randn(64, 3, 32, 32, device='cuda')
y = torch.randint(0, 10, [64], device='cuda')
k = torch.tensor(1., device='cuda', requires_grad=True)
w = torch.randn(3, 10, device='cuda', requires_grad=True)

nn.CrossEntropyLoss()(F.lp_pool2d(x, k, 32).view(len(x), -1) @ w, y).backward()
print(k.grad)

This is simply a global LP pool with a linear layer after that. The result is

tensor(nan, device='cuda:0')

I understand that overflow/underflow is more likely to happen because norm_type is mathematically on the exponent of x. But I'm wondering is there any way to avoid these so that I can train things normally.

I tried sacling the loss value before back propagating it, but the gradient of norm_type is still nan no matter how much I increase or decrease it (e.g. multiply 65536).

PS: LP Norm basically means sumpool(x ^ norm_type) ^ (1/norm_type) where sumpool is avgpool but do sum instead


title: Batch training DAE with LSTM Encoder, Decoder properly in PyTorch

body:
def denoise_train(x: DataLoader):
    loss = 0
    x_padded = list(map(lambda s: pad_string(s), x))
    x_idx_tensor = strings_to_index_tensor(x_padded)
    noisy_x = list(map(lambda s: noise_name(s), x))
    noisy_x_padded = list(map(lambda s: pad_string(s), noisy_x))
    noisy_x_idx_tensor = strings_to_index_tensor(noisy_x_padded)
    noisy_x_rnn_tensor = to_rnn_tensor(noisy_x_idx_tensor)
    batch_sz = len(x)
    encoder_hidden = encoder.init_hidden(batch_size=batch_sz)

    for i in range(noisy_x_rnn_tensor.shape[0]):
       _, encoder_hidden = encoder(noisy_x_rnn_tensor[i].unsqueeze(0), encoder_hidden)

    decoder_input = strings_to_tensor([SOS] * batch_sz)
    decoder_hidden = encoder_hidden
    names = [''] * batch_sz

    for i in range(x_idx_tensor.shape[0]):
        decoder_probs, decoder_hidden = decoder(decoder_input, decoder_hidden)
        nonzero_indexes = x_idx_tensor[i]
        best_indexes = torch.squeeze(torch.argmax(decoder_probs, dim=2), dim=0)
        decoder_probs = torch.squeeze(decoder_probs, dim=0)
        best_chars = list(map(lambda idx: index_to_char(int(idx)), best_indexes))
        loss += criterion(decoder_probs, nonzero_indexes.type(torch.LongTensor))

        for i, char in enumerate(best_chars):
            names[i] += char

        decoder_input = strings_to_tensor(best_chars)

    loss.backward()
    return names, noisy_x, loss.item()

I have this code for a denoising autoencoder on first names. It takes in a first name then pre-pads it with PAD char which I denote. I set it up like this so I could do batch training, but after training it for a couple days, I ran a name through it and it just printed out pads, which makes sense because it backprops on every iteration of a char so it’s being rewarded for just generating pads. How do I set it up not to learn this? I need the padding for batch training, because the names aren’t all consistent length so the padding allows them to be a consistent length. Currently right now after training if I put a name through it just generates pads.


title: Reduce the dimension of a tensor using max-pooling layer

body:
My question is very simple:

How can I reduce the dimension of a list or a tensor using max-pooling layer to 512 elements in the list:

I'm trying the following code:

    input_ids = tokenizer.encode(question, text)
    print(input_ids) # input_ids is a list of 700 elements
    m = nn.AdaptiveMaxPool1d(512)
    input_ids = m(torch.tensor([[input_ids]])) # convert the list to tensor and apply max-pooling layer

But I get the following error:

RuntimeError: "adaptive_max_pool2d_cpu" not implemented for 'Long'

So, please help to figure out where is the error


title: Training models interactively in Pytorch

body: I need to train two models in parallel. Each model has a different activation function with trainable parameters. I want to train model one and model two in the way that the parameters of the activation function from model one (e.g., alpha1) is separated from the parameters in model two (e.g., alpha2) by a gap of 2; i.e., |alpha_1 - alpha_2| > 2. I wonder how I could include it into the loss function for training.


title: how to get the index of the subarray in pytorch?

body:
a and b are torch tensor No repeating elements
a shape is [n,2] like:

[[1,2]
[2,3]
[4,6]
...]

b is[m,2] like:

[[1,2]
[4,6]
....
]

how to get the index of b in a, example:

a = [[1,2]
[2,4]
[6,7]
]
b = [[1,2]
[6,7]]

the index should be (0,3), we can use gpu，


title: Neural network layer without all connections

body:
The weights in a dense layer of a neural network is a (n,d) matrix, and I want to force some of these weights to always be zero.  I have another (n,d) matrix which is the mask of which entries can be non-zero.  The idea is that the layer should not be truly dense, but have some connections missing (i.e. equal to 0).
How can achieve this while training with PyTorch (or Tensorflow)?  I don't want these weights to become non-zero while training.
One method, if it doesn't support it directly, would be to zero-out the desired entries after each iteration of training.


title: How to pass new pre-trained embeddings while sharing the same vocabulary across torchtext.Field?

body:
I'm new to pytorch and torchtext and I was trying to concatenate two embedding layers in my CNN from two different pre-trained embeddings, before applying my convolutions.
Here's the basic workflow:
# Load pre-trained embeddings (dim=100)
from torchtext.vocab import Vectors
vectors_amazon = Vectors(name='gensim_embeddings_amazon.txt', cache='{}/{}'.format(PROJECT_FOLDER, OUTPUT_FOLDER))
vectors_imdb = Vectors(name='gensim_embeddings_imdb.txt', cache='{}/{}'.format(PROJECT_FOLDER, OUTPUT_FOLDER))

Then I create my text_field and label_field as follow:
# Custom_tokenizer is my tokenizer, MAX_SIZE=20000
text_field = Field(sequential=True, use_vocab=True, tokenize=custom_tokenizer)
label_field = LabelField()

text_field.build_vocab(train_data,
                       max_size=MAX_SIZE,
                       vectors=vectors_amazon)
label_field.build_vocab(train_data)

and after creating my train/valid/test iterator with BucketIterator, I create my CNN model for sentiment analysis.
So far so good, the problem is that I'd like to create another embedding considering also the vectors_imdb and I'm stuck here, since for the embedding layer I'd do the following:
pretrained_embeddings = text_field.vocab.vectors
model.embedding.weight.data.copy_(pretrained_embeddings)

but I have no idea how I can pass to a model.second_embedding.weight.data the values in vectors_imdb while keeping the correct alignment between embeddings and sharing the same vocab (coming from my training data)
I tried something like
second_text_field = text_field
second_text_field.vocab.set_vectors(text_field.vocab.stoi, vectors_imdb, 100)

but of course it doesn't work since changing vectors in second_text_field also modify text_field ones.
How can I modify vectors while keeping the correct mapping word:vector representation?
I'm afraid the only way is to loop through vectors_imdb keeping only words that are in my vocab, sorting them so that the two embeddings match and pass the result to the second_embedding layer, right?


title: Appropriate loss function in pytorch when output is an array of float numbers

body:
I am writing an encoder/decoder model very similar to https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html
The only difference is, here, the words are represented by some indices. I want to show them based on another metric, which are represented by flaot numbers.
The loss function nn.criterion = nn.NLLLoss(), seems to be working for times we are only workin with classes.
If my output array is not an array of integers, but an array of float numbers, what kind of loss function I can use? Considering all other parts are similar to the tutorial?
Thanks in advance.


title: How to dynamically update batch norm momentum in TF2?

body:
I found a PyTorch implementation that decays the batch norm momentum parameter from 0.1 in the first epoch to 0.001 in the final epoch. Any suggestions on how to do this with the batch norm momentum parameter in TF2? (i.e., start at 0.9 and end at 0.999) For example, this is what is done in the PyTorch code:
# in training script
momentum = initial_momentum * np.exp(-epoch/args.epochs * np.log(initial_momentum/final_momentum))
model_pos_train.set_bn_momentum(momentum)

# model class function
def set_bn_momentum(self, momentum):
    self.expand_bn.momentum = momentum
    for bn in self.layers_bn:
        bn.momentum = momentum

SOLUTION:
The selected answer below provides a viable solution when using the tf.keras.Model.fit() API. However, I was using a custom training loop. Here is what I did instead:
After each epoch:
mi = 1 - initial_momentum  # i.e., inital_momentum = 0.9, mi = 0.1
mf = 1 - final_momentum  # i.e., final_momentum = 0.999, mf = 0.001
momentum = 1 - mi * np.exp(-epoch / epochs * np.log(mi / mf))
model = set_bn_momentum(model, momentum)

set_bn_momentum function (credit to this article):
def set_bn_momentum(model, momentum):
    for layer in model.layers:
        if hasattr(layer, 'momentum'):
            print(layer.name, layer.momentum)
            setattr(layer, 'momentum', momentum)

    # When we change the layers attributes, the change only happens in the model config file
    model_json = model.to_json()

    # Save the weights before reloading the model.
    tmp_weights_path = os.path.join(tempfile.gettempdir(), 'tmp_weights.h5')
    model.save_weights(tmp_weights_path)

    # load the model from the config
    model = tf.keras.models.model_from_json(model_json)

    # Reload the model weights
    model.load_weights(tmp_weights_path, by_name=True)
    return model

This method did not add significant overhead to the training routine.