title: How to properly display Pytorch's math notation inside docstring?

body:
When looking at the docstring of Pytorch functions, the math notations are not properly displayed, e.g.:
https://pytorch.org/docs/stable/_modules/torch/nn/modules/loss.html

.. math::
    \text{loss}(x, class) = weight[class] \left(-x[class] + \log\left(\sum_j \exp(x[j])\right)\right)

Only if I use my IDE to display the documentation, it renders the latex notation properly.

Is there any switch for displaying math on the website?


title: Feed multi resolution images to Neural Network Pytorch

body:
I am new in this area so pardon if my question seems stupid.
I have created a multiresolution image pyramid using

  skimage.transform.pyramid_gaussian

The images are 2D. Now I want to feed these images to a neural network. The structure of the neural network is not fixed. But I can't do that since the images are not of the same size. Can anyone guide me to any resource regarding if this can be done?

Thank you


title: How to reduce the dimensions of a tensor with neural networks

body:
I have a 3D tensor of size [100,70,42] (batch, seq_len, features) and I would like to get a tensor of size [100,1,1] by using a neural network based on linear transformations (nn.Linear in Pytorch).

I have implemented the following code

class Network(nn.Module):
   def __init__(self):
      super(Network, self).__init__()
      self.fc1 = nn.Linear(42, 120)
      self.fc2 = nn.Linear(120,1)

   def forward(self, input):
      model = nn.Sequential(self.fc1,
                            nn.ReLU(),
                            self.fc2)
      output = model(input)
      return output

However, upon training this only gives me an output of the shape [100,70,1], which is not the desired one.

Thanks!


title: Pytorch or Numpy Batch Matrix Operation

body:
I am trying to us torch.bmm to do the following matrix operation,

If matrix is a M * N tensor, batch is a N * B tensor, how can i achieve,
In each batch, matrix @ batch_i, which gives M, and put the batch size together, the output tensor looks like M * B

There two questions here,

1.To use torch.bmm, it seems need both matrix need be batch, but my first input is not

The batch size need be the first dimension, while my batch size in the end

I guess it is the same question for Numpy users


title: Using past and attention_mask at the same time for gpt2

body:
I am processing a batch of sentences with different lengths, so I am planning to take advantage of the padding + attention_mask functionality in gpt2 for that.

At the same time, for each sentence I need to add a suffix phrase and run N different inferences. For instance, given the sentence "I like to drink coke", I may need to run two different inferences: "I like to drink coke. Coke is good" and "I like to drink coke. Drink is good". Thus, I am trying to improve the inference time for this by using the "past" functionality: https://huggingface.co/transformers/quickstart.html#using-the-past so I just process the original sentence (e.g. "I like to drink coke") once, and then I somehow expand the result to be able to be used with two other sentences: "Coke is good" and "Drink is good".

Below you will find a simple code that is trying to represent how I was trying to do this. For simplicity I'm just adding a single suffix phrase per sentence (...but I still hope my original idea is possible though):

from transformers.tokenization_gpt2 import GPT2Tokenizer
from transformers.modeling_gpt2 import GPT2LMHeadModel

tokenizer = GPT2Tokenizer.from_pretrained('gpt2', pad_token='<|endoftext|>')
model = GPT2LMHeadModel.from_pretrained('gpt2')

# Complete phrases are: "I like to drink soda without sugar" and "Go watch TV alone, I am not going"
docs = ["I like to drink soda", "Go watch TV"]
docs_tensors = tokenizer.batch_encode_plus(
    [d for d in docs], pad_to_max_length=True, return_tensors='pt')

docs_next = ["without sugar", "alone, I am not going"]
docs_next_tensors = tokenizer.batch_encode_plus(
    [d for d in docs_next], pad_to_max_length=True, return_tensors='pt')

# predicting the first part of each phrase
_, past = model(docs_tensors['input_ids'], attention_mask=docs_tensors['attention_mask'])

# predicting the rest of the phrase
logits, _ = model(docs_next_tensors['input_ids'], attention_mask=docs_next_tensors['attention_mask'], past=past)
logits = logits[:, -1]
_, top_indices_results = logits.topk(30)

The error I am getting is the following:

Traceback (most recent call last):
  File "/Applications/PyCharm CE.app/Contents/plugins/python-ce/helpers/pydev/pydevd.py", line 1434, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/Applications/PyCharm CE.app/Contents/plugins/python-ce/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/Users/damiox/Workspace/xxLtd/yy/stress-test-withpast2.py", line 26, in <module>
    logits, _ = model(docs_next_tensors['input_ids'], attention_mask=docs_next_tensors['attention_mask'], past=past)
  File "/Users/damiox/.local/share/virtualenvs/yy-uMxmjV2h/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/Users/damiox/.local/share/virtualenvs/yy-uMxmjV2h/lib/python3.7/site-packages/transformers/modeling_gpt2.py", line 593, in forward
    inputs_embeds=inputs_embeds,
  File "/Users/damiox/.local/share/virtualenvs/yy-uMxmjV2h/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/Users/damiox/.local/share/virtualenvs/yy-uMxmjV2h/lib/python3.7/site-packages/transformers/modeling_gpt2.py", line 476, in forward
    hidden_states, layer_past=layer_past, attention_mask=attention_mask, head_mask=head_mask[i]
  File "/Users/damiox/.local/share/virtualenvs/yy-uMxmjV2h/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/Users/damiox/.local/share/virtualenvs/yy-uMxmjV2h/lib/python3.7/site-packages/transformers/modeling_gpt2.py", line 226, in forward
    self.ln_1(x), layer_past=layer_past, attention_mask=attention_mask, head_mask=head_mask
  File "/Users/damiox/.local/share/virtualenvs/yy-uMxmjV2h/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/Users/damiox/.local/share/virtualenvs/yy-uMxmjV2h/lib/python3.7/site-packages/transformers/modeling_gpt2.py", line 189, in forward
    attn_outputs = self._attn(query, key, value, attention_mask, head_mask)
  File "/Users/damiox/.local/share/virtualenvs/yy-uMxmjV2h/lib/python3.7/site-packages/transformers/modeling_gpt2.py", line 150, in _attn
    w = w + attention_mask
RuntimeError: The size of tensor a (11) must match the size of tensor b (6) at non-singleton dimension 3

Process finished with exit code 1

Initially I thought this was related to https://github.com/huggingface/transformers/issues/3031 - so I re-built latest master to try the fix, but I still experience the issue.


title: How to execute a script in a virtual environment from a bash script on server?

body:
I am trying to run a python script in a virtual environment in a server using oarsub :

So firstly I run this command in a server name "a" :

oarsub -l /host=1/gpu=1,walltime=2:00:00 './training_corpus1.sh'

training_corpus1.sh looks like this at the beguinning :

#!/bin/bash

cd /home/ge/ke/anaconda3/envs

source activate env

cd ~/eXP/bert

CUDA_VISIBLE_DEVICES=0,1 python training.py \
    --exp_name bert \ ...

At the beguinning,  I am suppose to open my virtual environment and then run the script but I am always getting this error in the OAR.18651.stderr file :

./training_corpus1.sh: line 5: activate: No such file or directory
Traceback (most recent call last):
  File "training.py", line 18, in <module>
    from xlm.slurm import init_signal_handler, init_distributed_mode
  File "/home/ge/ke/eXP/bert/xlm/slurm.py", line 11, in <module>
    import torch
ImportError: No module named torch

Torch is located in my virtual environment , it seems that It did not open.Whenusing "conda " a thte place of "source" I get:

./training_corpus1.sh: line 5: conda: command not found


title: FASTAI:Buffered data was truncated after reaching the output size limit

body:
I was training the CNN ResNet model on google colab, with the learn.fit for 100 epochs saving the best model using the EarlyStoppingCallback and ReduceLROnPlateauCallback. But colab did a weird thing at 46th epoch where it got "Buffered data was truncated after reaching the output size limit", I know the meaning of it. The cell continues to run but doesn’t display anything. So I have no idea what the final loss and accuracy were. Is there a way to retrieve this from the model? I am working with FASTAI Library

import torchvision
from fastai.callbacks import *
learn5= cnn_learner(data, models.resnet18, metrics =[accuracy])
from fastai.callbacks import *
EarlySC = EarlyStoppingCallback(learn=learn5, monitor='accuracy', min_delta=0.01, patience=20)
reduceLR = ReduceLROnPlateauCallback(learn=learn5, monitor = 'accuracy', patience = 20, factor = 0.2, min_delta = 0)
learn5.fit(100,0.001,callbacks=[reduceLR,EarlySC])
learn5.save('resnet18')


title: Pytorch for Raspberry Pi 3 B+

body:
I have successfully cross compiled Pytorch 1.7 for Raspberry Pi3 B+ and the following wheel was generated as the result:
torch-1.7.0a0-cp37-cp37m-linux_armv7l.whl

However, when I try to install the wheel on the pi, I get this error:
ERROR: torch-1.7.0a0-cp37-cp37m-linux_armv7l.whl is not a supported wheel on this platform.

When I do lscpu, I see that the arm7l was correctly used as the target hw:
    Architecture:          armv7l
    Byte Order:            Little Endian
    CPU(s):                4
    On-line CPU(s) list:   0-3
    Thread(s) per core:    1
    Core(s) per socket:    4
    Socket(s):             1
    Model:                 4
    Model name:            ARMv7 Processor rev 4 (v7l)
    CPU max MHz:           1400.0000
    CPU min MHz:           600.0000
    BogoMIPS:              38.40
    Flags:                 half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32

And here is the output of pep425tags:
[('cp38', 'cp38', 'linux_armv7l'), ('cp38', 'abi3', 'linux_armv7l'),
('cp38', 'none', 'linux_armv7l'), ('cp37', 'abi3', 'linux_armv7l'),
('cp36', 'abi3', 'linux_armv7l'), ('cp35', 'abi3', 'linux_armv7l'),
('cp34', 'abi3', 'linux_armv7l'), ('cp33', 'abi3', 'linux_armv7l'),
('cp32', 'abi3', 'linux_armv7l'), ('py3', 'none', 'linux_armv7l'),
('cp38', 'none', 'any'), ('cp3', 'none', 'any'), ('py38', 'none',
'any'), ('py3', 'none', 'any'), ('py37', 'none', 'any'), ('py36',
'none', 'any'), ('py35', 'none', 'any'), ('py34', 'none', 'any'),
('py33', 'none', 'any'), ('py32', 'none', 'any'), ('py31', 'none',
'any'), ('py30', 'none', 'any')]

So I am wondering, what am I missing?


title: Pytorch's nn.TransformerEncoder "src_key_padding_mask" not functioning as expected

body:
Im working with Pytorch's nn.TransformerEncoder module. I got input samples with (as normal) the shape (batch-size, seq-len, emb-dim). All samples in one batch have been zero-padded to the size of the biggest sample in this batch. Therefore I want the attention of the all zero values to be ignored.
The documentation says, to add an argument src_key_padding_mask to the forward function of the nn.TransformerEncoder module. This mask should be a tensor with shape (batch-size, seq-len) and have for each index either True for the pad-zeros or False for anything else.
I achieved that by doing:
. . .

def forward(self, x):
    # x.size -> i.e.: (200, 28, 200)

    mask = (x == 0).cuda().reshape(x.shape[0], x.shape[1])
    # mask.size -> i.e.: (200, 20)

    x = self.embed(x.type(torch.LongTensor).to(device=device))
    x = self.pe(x)

    x = self.transformer_encoder(x, src_key_padding_mask=mask)

    . . .

Everything works good when I dont set the src_key_padding_mask. But the error I get when I do is the following:
File "/home/me/.conda/envs/py37/lib/python3.7/site-packages/torch/nn/functional.py", line 4282, in multi_head_attention_forward
    assert key_padding_mask.size(0) == bsz
AssertionError

Seems seems like it is comparing the first dimension of the mask, which is the batch-size, with bsz which probably stands for batch-size. But why is it failing then? Help very much appreciated!


title: Pytorch print(x) gives Illegal instruction (core dumped)

body:
I am using anaconda python2.7. I was able to install pytorch by using conda install pytorch from the installation guide at http://pytorch.org/

When I enter python and test the example

from __future__ import print_function
import torch
from torch.autograd import Variable
x = Variable(torch.ones(2, 2), requires_grad=True) ## works fine
print (x) ## broke here
x ## also broke, python can't give the values stored in x

I get the error Illegal instruction (core dumped) after the using the print(x).

I am able to do all the pytorch operation on x except for printing its value. At some point, I would have to look at the value in x. Does anybody have the same problem or know how to fix this?

Thanks.


title: CUDA error: out of memory when using subprocess to launch multiple training process in Pytorch

body:
I am training some neural networks, and I have a script to search for hyperparameters.

"""Peform hyperparemeters search"""

import argparse
import os
from subprocess import check_call
import sys
import numpy as np

import utils

PYTHON = sys.executable
parser = argparse.ArgumentParser()
parser.add_argument('--parent_dir', default='experiments/learning_rate',
                    help='Directory containing params.json')
parser.add_argument('--data_dir', default='data', help="Directory containing the dataset")

def launch_training_job(parent_dir, data_dir, job_name, params):
    """Launch training of the model with a set of hyperparameters in parent_dir/job_name

    Args:
        model_dir: (string) directory containing config, weights and log
        data_dir: (string) directory containing the dataset
        params: (dict) containing hyperparameters
    """
    # Create a new folder in parent_dir with unique_name "job_name"
    model_dir = os.path.join(parent_dir, job_name)
    if not os.path.exists(model_dir):
        os.makedirs(model_dir)

    # Write parameters in json file
    json_path = os.path.join(model_dir, 'params.json')
    params.save(json_path)

    # Launch training with this config
    cmd = "{python} -W ignore train.py --model_dir={model_dir} --data_dir {data_dir}".format(python=PYTHON,
                                                                                model_dir=model_dir,
                                                                                data_dir=data_dir)
    print(cmd)
    check_call(cmd, shell=True)

if __name__ == "__main__":
    # Load the "reference" parameters from parent_dir json file
    args = parser.parse_args()
    json_path = os.path.join(args.parent_dir, 'params.json')
    assert os.path.isfile(json_path), "No json configuration file found at {}".format(json_path)
    params = utils.Params(json_path)

    # Perform hypersearch over one parameter
    learning_rates = [1e-5]

    for learning_rate in learning_rates:
        # Modify the relevant parameter in params
        params.learning_rate = learning_rate

        # Launch job (name has to be unique)
        job_name = "learning_rate_{}".format(learning_rate)
        launch_training_job(args.parent_dir, args.data_dir, job_name, params)

If I try calling train.py by itself, it does not run out of memory - it only uses around 2500MB of the 12000MB available on the GPU.

But if I call this script to search hyperparameters, it will run out of memory EVEN if I call it with a single subprocess, specifically just testing ONE learning rate.

This is the error message: RuntimeError: CUDA error: out of memory.

Note that this does not happen for another similar model that I have that's smaller.

Why does this happen? And how can I solve it?


title: Setting dimensions of layers in a convolutional neural network

body:
Say I have 3x100x100 images in batches of 4 as input and I'm trying to make my first convolutional neural networks with pytorch. I'm really not sure if I'm getting convolutional neural networks right because when I train my input through the following arrangement I run into the error:

Expected input batch_size (1) to match target batch_size (4).

The following is my forward nnet:

Then If I were to pass it through:

nn.Conv2d(3, 6, 5)

I would get 6 layers of maps each with dimensions (100-5+1).

Then If I were to pass it through:

nn.MaxPool2d(2, 2)

I would get 6 layers of maps each with dimensions (96/2)

Then if I were to pass it through:

nn.Conv2d(6, 16, 5)

I would get 16 layers of maps each with dimensions (48-5+1)

Then If I were to pass it through:

self.fc1 = nn.Linear(44*44*16, 120)

I would get 120 neurons

Then If I were to pass it through:

self.fc2 = nn.Linear(120, 84)

I would get 84 neurons

Then If I were to pass it through:

self.fc3 = nn.Linear(84, 3)

I would get 3 outputs which would be perfect because I have 3 classes of labels. But as I said before, this leads to an error which is really surprising because this makes a lot of sense to me.

Full neural network code:

import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(44*44*16, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 3)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 *44*44)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

net = Net()
net.to(device)


title: State persistence in shared LSTM layers in Keras

body:
I am trying to use a shared LSTM layer with state in a Keras model, but it seems that the internal state is modified by each parallel use. This raises two questions:

When training a model with a shared LSTM layer and using stateful=True, are the parallel uses updating the same state also during training?
If my observation is valid, is there a way to use weight-sharing LSTMs such that the state is stored independently for each of the parallel uses?

The code below exemplifies the problem with three sequences sharing the LSTM. The prediction of a full input is compared with the result from splitting the prediction input into two halves and feeding them into the network consecutively.

What can be observed, is that the a1 is the same as the first half of aFull, meaning that the the uses of the LSTM really are in parallel with independent states during the first prediction. I.e., z1 is not affected by the parallel call creating z2 and z3. But a2 is different from the second half of aFull, so there is some interaction between the states of the parallel uses.

What I was hoping is that the concatenation of the two pieces a1 and a2 would be the same as the result from calling the prediction with a longer input sequence, but this doesn't seem to be the case. A further concern is that when this kind of interaction takes place in the prediction, is it also happening during the training.

import keras
import keras.backend as K
import numpy as np

nOut = 3
xShape = (3, 50, 4)
inShape = (xShape[0], None, xShape[2])
batchInShape = (1, ) + inShape
x = np.random.randn(*xShape)

# construct network
xIn = keras.layers.Input(shape=inShape, batch_shape=batchInShape)

# shared LSTM layer
sharedLSTM = keras.layers.LSTM(units=nOut, stateful=True, return_sequences=True, return_state=False)

# split the input on the first axis
x1 = keras.layers.Lambda(lambda x: x[:,0,:,:])(xIn)
x2 = keras.layers.Lambda(lambda x: x[:,1,:,:])(xIn)
x3 = keras.layers.Lambda(lambda x: x[:,2,:,:])(xIn)

# pass each input through the LSTM
z1 = sharedLSTM(x1)
z2 = sharedLSTM(x2)
z3 = sharedLSTM(x3)

# add a singleton dimension
y1 = keras.layers.Lambda(lambda x: K.expand_dims(x, axis=1))(z1)
y2 = keras.layers.Lambda(lambda x: K.expand_dims(x, axis=1))(z2)
y3 = keras.layers.Lambda(lambda x: K.expand_dims(x, axis=1))(z3)

# combine the outputs
y = keras.layers.Concatenate(axis=1)([y1, y2, y3])

model = keras.models.Model(inputs=xIn, outputs=y)
model.compile(loss='mse', optimizer='adam')
model.summary()

# no need to train, since we're interested only what is happening mechanically

# reset to a known state and predict for full input
model.reset_states()
aFull = model.predict(x[np.newaxis,:,:,:])

# reset to a known state and predict for the same input, but in two pieces
model.reset_states()
a1 = model.predict(x[np.newaxis,:,:xShape[1]//2,:])
a2 = model.predict(x[np.newaxis,:,xShape[1]//2:,:])
# combine the pieces
aSplit = np.concatenate((a1, a2), axis=2)

print('full diff: {}, first half diff: {}, second half diff: {}'.format(str(np.sum(np.abs(aFull - aSplit))), str(np.sum(np.abs(aFull[:,:,:xShape[1]//2,:] - aSplit[:,:,:xShape[1]//2,:]))), str(np.sum(np.abs(aFull[:,:,xShape[1]//2:,:] - aSplit[:,:,xShape[1]//2:,:])))))

Update: The behaviour described above was observed with Keras using Tensorflow 1.14 and 1.15 as the backend. Running the same code with tf2.0 (with the adjusted imports) changes the result so that a1 is no longer the same as the first half of aFull. This can be still accomplished by setting stateful=False in the layer instantiation.

This would suggest to me that the way I'm trying to use the recursive layer with shared parameters, but own states for parallel uses, is not really possible like this.

Update 2: It seems that the same functionality has been missed by also other earlier: closed, unanswered question at Keras' github.

For a comparison, here is a scribbling in pytorch (the first time I've tried to use it) implementing a simple network with N parallel LSTMs sharing the weights, but having independent states. In this case the states are stored explicitly in a list and provided to the LSTM cell manually.

import torch
import numpy as np

class sharedLSTM(torch.nn.Module):

    def __init__(self, batchSz, nBands, nDims, outDim):
        super(sharedLSTM, self).__init__()
        self.internalLSTM = torch.nn.LSTM(input_size=nDims, hidden_size=outDim, num_layers=1, bias=True, batch_first=True)
        allStates = list()
        for bandIdx in range(nBands):
            h_0 = torch.zeros(1, batchSz, outDim)
            c_0 = torch.zeros(1, batchSz, outDim)
            allStates.append((h_0, c_0))

        self.allStates = allStates
        self.nBands = nBands

    def forward(self, x):
        allOut = list()
        for dimIdx in range(self.nBands):
            thisSlice = x[:,dimIdx,:,:] # (batchSz, nSteps, nFeats)
            thisState = self.allStates[dimIdx]

            thisY, thisState = self.internalLSTM(thisSlice, thisState)
            self.allStates[dimIdx] = thisState
            allOut.append(thisY[:,None,:,:]) # => (batchSz, 1, nSteps, nFeats)

        y = torch.cat(allOut, dim=1) # => (batchSz, nDims, nSteps, nFeats)

        return y

    def resetStates(self):
        for bandIdx in range(nBands):
            self.allStates[bandIdx][0][:] = 0.0
            self.allStates[bandIdx][1][:] = 0.0

batchSz = 5
nBands = 3
nFeats = 4
nOutDims = 2
net = sharedLSTM(batchSz, nBands, nFeats, nOutDims)
net = net.float()
print(net)

N = 20
x = torch.from_numpy(np.random.rand(batchSz, nBands, N, nFeats)).float()
x1 = x[:, :, :N//2, :]
x2 = x[:, :, N//2:, :]

aa = net.forward(x)
net.resetStates()
a1 = net.forward(x1)
a2 = net.forward(x2)

print('(with reset) first half abs diff: {}'.format(str(torch.sum(torch.abs(a1 - aa[:,:,:N//2,:])).detach().numpy())))
print('(with reset) second half abs diff: {}'.format(str(torch.sum(torch.abs(a2 - aa[:,:,N//2:,:])).detach().numpy())))

Result: the output is the same regardless if we do the prediction in one go or in pieces.

I've tried to replicate this in Keras using sub-classing, but without success:

import keras
import numpy as np

class sharedLSTM(keras.Model):
    def __init__(self, batchSz, nBands, nDims, outDim):
        super(sharedLSTM, self).__init__()
        self.internalLSTM = keras.layers.LSTM(units=outDim, stateful=True, return_sequences=True, return_state=True)
        self.internalLSTM.build((batchSz, None, nDims))
        self.internalLSTM.reset_states()
        allStates = list()
        allSlicers = list()
        for bandIdx in range(nBands):
            allStates.append(None)
            allSlicers.append(keras.layers.Lambda(lambda x, b: x[:, :, b, :], arguments = {'b' : bandIdx}))

        self.allStates = allStates
        self.allSlicers = allSlicers
        self.Concat = keras.layers.Lambda(lambda x: keras.backend.concatenate(x, axis=2))

        self.nBands = nBands

    def call(self, x):
        allOut = list()
        for bandIdx in range(self.nBands):
            thisSlice = self.allSlicers[bandIdx]( x )
            thisState = self.allStates[bandIdx]

            thisY, *thisState = self.internalLSTM(thisSlice, initial_state=thisState)
            self.allStates[bandIdx] = thisState.copy()
            allOut.append(thisY[:,:,None,:])

        y = self.Concat( allOut )
        return y

batchSz = 1
nBands = 3
nFeats = 4
nOutDims = 2
N = 20

model = sharedLSTM(batchSz, nBands, nFeats, nOutDims)
model.compile(optimizer='SGD', loss='mae')

x = np.random.rand(batchSz, N, nBands, nFeats)
x1 = x[:, :N//2, :, :]
x2 = x[:, N//2:, :, :]

aa = model.predict(x)

model.reset_states()
a1 = model.predict(x1)
a2 = model.predict(x2)

print('(with reset) first half abs diff: {}'.format(str(np.sum(np.abs(a1 - aa[:,:N//2,:,:])))))
print('(with reset) second half abs diff: {}'.format(str(np.sum(np.abs(a2 - aa[:,N//2:,:,:])))))

If you now ask "why don't you then use torch and shut up?", the answer is that the surrounding experimental framework has been built assuming Keras and changing it would be a non-negligible amount of work.


title: libtorch (PyTorch C++) weird class syntax

body:
In the official PyTorch C++ examples on GitHub Here
you can witness a strange definition of a class:

class CustomDataset : public torch::data::datasets::Dataset<CustomDataset> {...}

My understanding is that this defines a class CustomDataset which "inherits from" or "extends" torch::data::datasets::Dataset<CustomDataset>. This is weird to me since the class we're creating is inheriting from another class which is parameterized by the class we're creating...How does this even work? What does it mean? This seems to me like an Integer class inheriting from vector<Integer>, which seems absurd.


title: How can I keep a PyTorch submodule in eval mode?

body:
I have a pretrained model that I'm using in conjunction with a model being trained. I want the pretrained model to always be in eval mode, but the other model will be moving back and forth between eval and train mode. I'd still like the pretrained model to be a submodule of the other one, though (e.g. so that all parameters stay on the same device). Is there a way to do this? Here's a minimal example:

from torch import nn

class FixedModule(nn.Module):
    pass

class TrainableModule(nn.Module):
    def __init__(self, fixed_module):
        super().__init__()
        self.fixed_module = fixed_module

fixed = FixedModule().eval()
assert not fixed.training

trainable = TrainableModule(fixed)
assert trainable.training and not trainable.fixed_module.training

trainable.train()
assert trainable.fixed_module.training  # I'd like this to give an error

I know I can work around this by, e.g., always doing

trainable.train()
trainable.fixed_module.eval()

but that's error-prone and doesn't work well with existing code.


title: GPU out of memory even when using a lazy dataset and custom dataloader

body:
EDIT - the following solutions worked for me:

Instead of epoch_loss += loss.item(), use loss = loss.item() and epoch_loss += loss. This overwrites the loss variable freeing up a ton of  memory.

Check for memory leak (https://www.programcreek.com/python/example/1870/gc.get_objects). I used this to count the number of tensors in memory at each batch and noticed it was going up over time. Overwriting variables solved it.

Overwrite variables.  For example, x = embedded(x) is better than embedded_x = embedded_layer(x) because the second option keeps two variables in memory (x and embedded_x) while the first only has one (x).

Check if the problem is only in certain batches (too long sequences in some batches,for example).
 try:
     loss = train_step(model,source, src_len, targets,clip)
 except RuntimeError as e:
     if 'out of memory' in str(e):
         print(f'WARNING: ran out of memory, skipping batch number {batch_num:,}')

I ended up not using this but a common suggestion was: gc.collect() right before loss.backward() to make sure large variables are cleaned up before the most memory intensive step.

For a translation tasks, I have a large dataset, so I created a lazy dataset. I also used a custom collate_fn in the dataloader to pad the sequences in the batches. However, I'm getting CUDA out of memory almost immediately even after decreasing the batch size down to almost nothing. =
Is there an error in my implementation?
class LazyDataset(Dataset):
def __init__(self, path, filename,text_vocab,target_vocab):
    self.path = path
    self.text_vocab = text_vocab
    self.text_init = self.text_vocab.init_token
    self.text_eos = self.text_vocab.eos_token
    self.target_vocab = target_vocab
    self.target_init = self.target_vocab.init_token
    self.target_eos = self.target_vocab.eos_token

    self._filepath = os.path.join(path,filename)
    self._total_data = int(subprocess.check_output("wc -l "+self._filepath, shell=True).split()[0])

def __len__(self):
    'Denotes the total number of samples'
    return self._total_data

def tokens_to_idx(self,text,target):
    #TODO: add arguments to make bos & eos optional
    # add sos and eos tokens
    text = [self.text_init] + text + [self.text_eos]
    target = [self.target_init] + target + [self.target_eos]
    #tokens to indices
    text = [self.text_vocab.vocab.stoi[t] for t in text]
    target = [self.target_vocab.vocab.stoi[t] for t in target]
    return text,target

def __getitem__(self, index):
    'Generates one sample of data'
    line = linecache.getline(self._filepath, index + 1)
    text,target = line.split('\t')

    # string to list, tokenizing on white space
    # TODO: add more tokenization options
    text, target = text.split(), target.split()
    text,target = self.tokens_to_idx(text, target)
    text_lens = len(text)
    return text,target,text_lens

The custom collate is:
def sort_batch(batch):
batch = sorted(batch, key=lambda x: len(x[0]), reverse=True)
source, targets,src_len = zip(*[(s,t,l) for (s,t,l) in batch])
source = pad_sequence([torch.LongTensor(s) for s in source],batch_first=False, padding_value=0)
targets = pad_sequence([torch.LongTensor(t) for t in targets],batch_first=False, padding_value=0)
src_len = torch.LongTensor(src_len) #[bsz]
return source, targets,src_len


title: Weights not updating on my neural net (Pytorch)

body:
I'm completely new to neural nets, so I tried to roughly follow some tutorials to create a neural net that can just distinguish if a given binary picture contains a white circle or if it is all black. So, I generated 1000 arrays of size 10000 representing a 100x100 picture with half of them containing a white circle somewhere. The generation of my dataset looks like this:
for i in range(1000):
   image = [0] * (IMAGE_SIZE * IMAGE_SIZE)

   if random() < 0.5:
      dataset.append([image, [[0]]])

   else:
      #inserts circle in image
      #...

      dataset.append([image, [[1]]])

np.random.shuffle(dataset)
np.save("testdataset.npy", dataset)

The double list around the classifications is because the net seemed to give that format as an output, so I matched that.
Now since I don't really have any precise idea of how pytorch works, I don't really now which parts of the code are relevant for solving my problem and which aren't. Therefore, I gave the code for the net and the training down below and really hope that someone can explain to me where I went wrong. I'm sorry if it's too much code. The code runs without errors, but if I print the parameters before and after training they didn't change in any way and the net will always just return a 0 for every image/array.
IMAGE_SIZE = 100
EPOCHS = 3
BATCH_SIZE = 50
VAL_PCT = 0.1

class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(IMAGE_SIZE * IMAGE_SIZE, 64)
        self.fc2 = nn.Linear(64, 64)
        self.fc3 = nn.Linear(64, 64)
        self.fc4 = nn.Linear(64, 1)

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.relu(self.fc3(x))
        x = self.fc4(x)
        return F.log_softmax(x, dim = 1)

net = Net()
optimizer = optim.Adam(net.parameters(), lr = 0.01)
loss_function = nn.MSELoss()
dataset = np.load("testdataset.npy", allow_pickle = True)

X = torch.Tensor([i[0] for i in dataset]).view(-1, 10000)
y = torch.Tensor([i[1] for i in dataset])

val_size = int(len(X) * VAL_PCT)

train_X = X[:-val_size]
train_y = y[:-val_size]

test_X = X[-val_size:]
test_y = y[-val_size:]

for epoch in range(EPOCHS):
    for i in range(0, len(train_X), BATCH_SIZE):
        batch_X = train_X[i:i + BATCH_SIZE].view(-1, 1, 10000)
        batch_y = train_y[i:i + BATCH_SIZE]

        net.zero_grad()

        outputs = net(batch_X)
        loss = loss_function(outputs, batch_y)
        loss.backward()
        optimizer.step()


title: Pytorch installation results in python setup.py egg_info

body:
I am trying to run:

pip3 install --no-cache-dir torch

but get:

Command "python setup.py egg_info" failed with error code 1 in C:\Users\Bla\AppData\Local\Temp\pip-install-hcy4wtm0\torch\

Google search gives conflicting results and I tried a few of them. Btw, I have a shiny new Anaconda installation.

PS:

So we established that I should use, which I also tried previously:

conda install pytorch -c pytorch
pip3 install torchvision

The error I am getting now is:

CondaHTTPError: HTTP 404 NOT FOUND for url <https://pypi.python.org/simple/noarch/repodata.json>
Elapsed: 00:00.119647

The remote server could not find the noarch directory for the
requested channel with url: https://pypi.python.org/simple

As of conda 4.3, a valid channel must contain a `noarch/repodata.json` and
associated `noarch/repodata.json.bz2` file, even if `noarch/repodata.json` is
empty. please request that the channel administrator create
`noarch/repodata.json` and associated `noarch/repodata.json.bz2` files.
$ mkdir noarch
$ echo '{}' > noarch/repodata.json
$ bzip2 -k noarch/repodata.json

You will need to adjust your conda configuration to proceed.
Use `conda config --show channels` to view your configuration's current state.
Further configuration help can be found at <https://conda.io/docs/config.html>.

A reportable application error has occurred. Conda has prepared the above report.
If submitted, this report will be used by core maintainers to improve
future releases of conda.
Would you like conda to send this report to the core maintainers?


title: cv2.warpAffine fails on higher dimension input

body:
I am trying to rewrite a script such that it'll be able to run on higher dimension tensor (on batches of data).

This is a function that works, though it does not support batches of center/scale:

def get_affine_transform(center,
                         scale,
                         rot,
                         output_size,
                         shift=np.array([0, 0], dtype=np.float32),
                         inv=0):
    if not isinstance(scale, np.ndarray) and not isinstance(scale, list):
        print(scale)
        scale = np.array([scale, scale])

    scale_tmp = scale * 200.0
    src_w = scale_tmp[0]
    dst_w = output_size[0]
    dst_h = output_size[1]

    rot_rad = np.pi * rot / 180
    src_dir = get_dir([0, src_w * -0.5], rot_rad)
    dst_dir = np.array([0, dst_w * -0.5], np.float32)

    src = np.zeros((3, 2), dtype=np.float32)
    dst = np.zeros((3, 2), dtype=np.float32)
    src[0, :] = center + scale_tmp * shift
    src[1, :] = center + src_dir + scale_tmp * shift
    dst[0, :] = [dst_w * 0.5, dst_h * 0.5]
    dst[1, :] = np.array([dst_w * 0.5, dst_h * 0.5]) + dst_dir

    src[2:, :] = get_3rd_point(src[0, :], src[1, :])
    dst[2:, :] = get_3rd_point(dst[0, :], dst[1, :])

    if inv:
        trans = cv2.getAffineTransform(np.float32(dst), np.float32(src))
    else:
        trans = cv2.getAffineTransform(np.float32(src), np.float32(dst))

    return trans

My new version of this function that gives similar output as the function above (if for example i have a batch of two bboxes, this function returns the same output as if i ran the above function separately on each bbox):

def get_affine_transforms(
        num_boxes,
        center,
        scale,
        rot,
        output_size,
        shift=np.array([0, 0], dtype=np.float32),
        inv=0
):
    if not isinstance(scale, np.ndarray) and not isinstance(scale, list):
        print(scale)
        scale = np.array([scale, scale])

    scale_tmp = scale * 200.0
    src_w = scale_tmp[:, 0]
    dst_w = output_size[0]
    dst_h = output_size[1]

    rot_rad = np.pi * rot / 180
    src_dir, dst_dir = [], []
    for i in range(num_boxes):
        src_dir.append(get_dir([0, src_w[i] * -0.5], rot_rad))
        dst_dir.append(np.array([0, dst_w * -0.5], np.float32))

    src = np.zeros((num_boxes, 3, 2), dtype=np.float32)
    dst = np.zeros((num_boxes, 3, 2), dtype=np.float32)
    src[:, 0, :] = center + scale_tmp * shift
    src[:, 1, :] = center + src_dir + scale_tmp * shift
    dst[:, 0, :] = [dst_w * 0.5, dst_h * 0.5]
    dst[:, 1, :] = np.array([dst_w * 0.5, dst_h * 0.5]) + dst_dir

    src[:, 2:, :] = get_3rd_points(src[:, 0, :], src[:, 1, :])
    dst[:, 2:, :] = get_3rd_points(dst[:, 0, :], dst[:, 1, :])

    trans = []
    for dst_, src_ in zip(src, dst):
        if inv:
            trans.append(cv2.getAffineTransform(np.float32(dst_), np.float32(src_)))
        else:
            trans.append(cv2.getAffineTransform(np.float32(dst_), np.float32(src_)))

    return np.array(trans)

Though, later in the program, the following code snippet fails because of cv2.warpAffine (data_numpy is an image):

    trans = get_affine_transforms(len(boxes), c, s, r, config.MODEL.IMAGE_SIZE)
    input = cv2.warpAffine(data_numpy,
                           trans,
                           (int(config.MODEL.IMAGE_SIZE[0]),
                            int(config.MODEL.IMAGE_SIZE[1])),
                           flags=cv2.INTER_LINEAR)

    transform = transforms.Compose([
        transforms.ToTensor(),
        transforms.Normalize(mean=[0.485, 0.456, 0.406],
                             std=[0.229, 0.224, 0.225]),
    ])

    input = transform(input).unsqueeze(0)

The failure:

OpenCV(3.4.1) Error: Assertion failed ((M0.type() == 5 || M0.type() == 6) && M0.rows == 2 && M0.cols == 3) in warpAffine, file /io/opencv/modules/imgproc/src/imgwarp.cpp, line 2700

How can i call cv2.warpAffine so i can create higher dimension inputs (that eventualy will be fed into a neural network).


title: PyTorch matrix factorization embedding error

body:
I'm trying to use a single hidden layer NN to perform matrix factorization. In general, I'm trying to solve for a tensor, V, with dimensions [9724x300] where there are 9724 items in inventory, and 300 is the arbitrary number of latent features.

The data that I have is a [9724x9724] matrix, X, where columns and rows represent the number of mutual likes. (eg X[0,1] represents the sum of users who like both item 0 and item 1. Diagonal entries are not of importance.

My goal is to use MSE loss, such that the dot product of V[i,:] on V[j,:] transposed is very, very close to X[i,j].

Below is code that I've adapted from the below link.

https://blog.fastforwardlabs.com/2018/04/10/pytorch-for-recommenders-101.html

import torch
from torch.autograd import Variable

class MatrixFactorization(torch.nn.Module):
    def __init__(self, n_items=len(movie_ids), n_factors=300):
        super().__init__()

        self.vectors = nn.Embedding(n_items, n_factors,sparse=True)

    def forward(self, i,j):
        return (self.vectors([i])*torch.transpose(self.vectors([j]))).sum(1)

    def predict(self, i, j):
        return self.forward(i, j)

model = MatrixFactorization(n_items=len(movie_ids),n_factors=300)
loss_fn = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)

for i in range(len(movie_ids)):
    for j in range(len(movie_ids)):
    # get user, item and rating data
        rating = Variable(torch.FloatTensor([Xij[i, j]]))
        # predict
#         i = Variable(torch.LongTensor([int(i)]))
#         j = Variable(torch.LongTensor([int(j)]))
        prediction = model(i, j)
        loss = loss_fn(prediction, rating)

        # backpropagate
        loss.backward()

        # update weights
        optimizer.step()

The error returned is:

TypeError: embedding(): argument 'indices' (position 2) must be Tensor, not list

I'm very new to embeddings. I had tried replacing embeddings as a simple float tensor, however the MatrixFactorization class, which I defined, did not recognize the tensor as a model parameters to be optimized over.

Any thoughts on where I'm going wrong?


title: Understanding cdist() function

body:
What does this new_cdist() function actually do? More specifically:

Why is there a sqrt() operation when the AdderNet paper does not use it in its backward propagation equation?
How is needs_input_grad[] used?

def new_cdist(p, eta):
    class cdist(torch.autograd.Function):
        @staticmethod
        def forward(ctx, W, X):
            ctx.save_for_backward(W, X)
            out = -torch.cdist(W, X, p)
            return out

        @staticmethod
        def backward(ctx, grad_output):
            W, X = ctx.saved_tensors
            grad_W = grad_X = None
            if ctx.needs_input_grad[0]:
                _temp1 = torch.unsqueeze(X, 2).expand(X.shape[0], X.shape[1], W.shape[0]).permute(1, 0, 2)
                _temp2 = torch.unsqueeze(W.transpose(0, 1), 1)
                _temp = torch.cdist(_temp1, _temp2, p).squeeze().transpose(0, 1)
                grad_W = torch.matmul(grad_output, _temp)
                # print('before norm: ', torch.norm(grad_W))
                grad_W = eta * np.sqrt(grad_W.numel()) / torch.norm(grad_W) * grad_W
                print('after norm: ', torch.norm(grad_W))
            if ctx.needs_input_grad[1]:
                _temp1 = torch.unsqueeze(W, 2).expand(W.shape[0], W.shape[1], X.shape[0]).permute(1, 0, 2)
                _temp2 = torch.unsqueeze(X.transpose(0, 1), 1)
                _temp = torch.cdist(_temp1, _temp2, p).squeeze().transpose(0, 1)
                _temp = torch.nn.functional.hardtanh(_temp, min_val=-1., max_val=1.)
                grad_X = torch.matmul(grad_output.transpose(0, 1), _temp)
            return grad_W, grad_X
    return cdist().apply

I mean that it seems to be related to a new type of back-propagation equation and adaptive learning rate.


title: Why some people chain the parameters of two different networks and train them with same optimizer?

body:
I was looking at CycleGAN's official pytorch implementation and there, author chained the parameters of both networks and used a single optimizer for both network. How does this work? Is it better than using two different optimizers for two different networks ?

all_params = chain(module_a.parameters(), module_b.parameters())
optimizer = torch.optim.Adam(all_params)


title: how to change the labels in a datafolder of pytorch?

body:
I first load an unlabeled dataset as following:
unlabeled_set = DatasetFolder("food-11/training/unlabeled", loader=lambda x: Image.open(x), extensions="jpg", transform=train_tfm)
and now since I'm trying to conduct semi-supervised learning: I'm trying to define the following function. The input "dataset" is the unlabeled_set I just loaded.
As I want to change the label of the dataset to be the one I predicted, not the original labels(all of the original labels were 1's), how can I do?
I have tried using dataset.targets to change the labels, but it doesn't work at all.
the following is my function:
import torch
def get_pseudo_labels(dataset, model, threshold=0.07):
    # This functions generates pseudo-labels of a dataset using given model.
    # It returns an instance of DatasetFolder containing images whose prediction confidences exceed a given threshold.
    # You are NOT allowed to use any models trained on external data for pseudo-labeling.
    device = "cuda" if torch.cuda.is_available() else "cpu"
    x = []
    y = []

    # print(dataset.targets[0])

    # Construct a data loader.
    data_loader = DataLoader(dataset, batch_size=batch_size, shuffle=False)

    # Make sure the model is in eval mode.
    model.eval()
    # Define softmax function.
    softmax = nn.Softmax()
    counter = 0
    # Iterate over the dataset by batches.
    for batch in tqdm(data_loader):
        img, _ = batch

        # Forward the data
        # Using torch.no_grad() accelerates the forward process.
        with torch.no_grad():
            logits = model(img.to(device))

        # Obtain the probability distributions by applying softmax on logits.
        probs = softmax(logits)
        count = 0
        # ---------- TODO ----------
        # Filter the data and construct a new dataset.
        dataset.targets = torch.tensor(dataset.targets)
        for p in probs:
          if torch.max(p) >= threshold:
            if not(counter in x):
              x.append(counter)
            dataset.targets[counter] = torch.argmax(p)

          counter += 1

    # Turn off the eval mode.
    model.train()
    # dat = DataLoader(ImgDataset(x,y), batch_size=batch_size, shuffle=False)
    print(dataset.targets[10])
    new = torch.utils.data.Subset(dataset, x)

    return new```


title: How to sample through a small dataset for more iterations than data size?

body:
I have one small and one large dataset and they signify two separate classes. The network I am training is style transfer, so I need one image of each class in order to keep training. The training stops though, as soon as the smaller dataset runs out. How can I keep sampling from the small dataset randomly beyond its size?

I tried RandomSampler() but that didn't work. Here's my code for the small dataset :

sampler = RandomSampler(self)
dataloader = DataLoader(self, batch_size=26, shuffle=False, sampler=sampler)
while True:
    for data in dataloader:
        yield data

I also tried iterator.cycle but that didn't help either.

loader = iter(cycle(self.dataset.gen(attribute_id, True)))
A, y_A = next(loader)
B, y_B = next(self.dataset.gen(attribute_id, False))


title: load multi-modal data with pytorch

body:
I'm trying to load multi-modal data (e.g. text and image) in pytorch for image classification. I do not know how to load them simultaneously, like the following code.

 def __init__(self, img_path, txt_path, transform=None, loader=default_loader):

def __len__(self):
    return len(self.img_name)

def __getitem__(self, item):

Can anyone help me?


title: pytorch 1D Dropout leads to unstable learning

body:
I'm implementing an Inception-like CNN in pytorch.  After the blocks of convolution layers, I have three fully-connected linear layers followed by a sigmoid activation to give me my final regression output.  I'm testing the effects of dropout layers in this network, but it's giving me some unexpected results.

Here is the code:

class MyInception(nn.Module):
    def __init__(self, in_channels, verbose=False):
        super(MyInception, self).__init__()
        self.v = verbose
        ic=in_channels; oc=16
        self.inceptionBlock1 = InceptionBlock(in_channels=ic, out_channels=oc, maxpool=False, verbose=verbose)
        self.inceptionBlock2 = InceptionBlock(in_channels=oc * 6, out_channels=oc, maxpool=False, verbose=verbose)
        self.pool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1)

        self.regressor = nn.Sequential(
            nn.Linear(oc * 6 * 35 * 35, 1024, bias=True),
            nn.ReLU(inplace=True),
            nn.Dropout(p=0.2, inplace=False),  # <--- Dropout 1
            nn.Linear(1024, 128, bias=True),
            nn.ReLU(inplace=True),
            nn.Dropout(p=0.2, inplace=False),  # <--- Dropout 2
            nn.Linear(128, 1, bias=True),
            nn.Sigmoid()
        )

    def forward(self, x):
        x = self.inceptionBlock1(x)
        x = self.inceptionBlock2(x)
        x = self.pool(x)
        x = torch.flatten(x, 1)
        x = self.regressor(x)
        return x

def train(epochs=10, dot_every=25):
    running = pd.DataFrame(columns=['Epoch','Round','TrainLoss','TestLoss','LearningRate'])
    for epoch in range(epochs):
        train_losses = []
        model.train()
        counter = 0

        for images, targets in train_loader:
            images = images.to(device)
            targets = targets.to(device)

            optimizer.zero_grad()
            outputs = model(images)
            loss = loss_fn(torch.flatten(outputs), targets)
            train_losses.append( loss.item() )
            loss.backward()
            optimizer.step()

            counter += 1
            if counter % dot_every == 0:
                print(".",  end='.', flush=True)
                test_loss = test()
            else:
                test_loss = -1.
            lr = np.squeeze(scheduler.get_lr())
            running = running.append(pd.Series([epoch, counter, loss.item(), test_loss, lr], index=running.columns), ignore_index=True)

        test_loss = test()
        train_loss = np.mean(np.asarray(train_losses))
        running = running.append(pd.Series([epoch, counter, train_loss, test_loss, lr], index=running.columns), ignore_index=True)
        print("")
        print(f"Epoch {epoch+1}, Train Loss: {np.round(train_loss,4)}, Test Loss: {np.round(test_loss, 4)}, Learning Rate: {np.format_float_scientific(lr, precision=4)}")
    return running

def test():
    model.eval()
    test_losses = []
    for i, (images,targets) in enumerate(test_loader):
        images = images.to(device)
        targets = targets.to(device)
        outputs = model(images)
        loss = loss_fn(torch.flatten(outputs), targets)
        test_losses.append( loss.item() )

    mean_loss = np.mean(np.asarray(test_losses))
    return mean_loss

# instantiate the model
model = MyInception(in_channels=4, verbose=False).to(device)
# define the optimizer and loss function
optimizer = Adam(model.parameters(), lr=0.001, weight_decay=0.0001)
loss_fn = nn.MSELoss()

# run it
results = train(epochs=10, dot_every=20)

Here is a plot of the MSE losses for the training data.  (red = no dropout, green = second dropout only, blue = first dropout only, purple = both dropouts)
Runs with dropout have big increases in losses at the epoch boundaries (dashed vertical lines), with the double dropout even having a big jump in loss at the start of epoch 10.

The important thing is the test loss.  That is much more stable and not too different between either condition after the 5th epoch, so maybe I shouldn't care. But I would like to understand what is going on.


title: Is it possible to see the read data of a pytorchtext.data.Tabulardataset?

body:
train, test = data.TabularDataset.splits(path="./data/", train="train.csv",test="test.csv",format="csv",fields=[("Tweet",TEXT), ("Affect Dimension",LABEL)])

I have this code and want to evaluate, if the loaded data is correct or if it's using wrong columns for the actual text fields etc.

If my file has the columns "Tweet" for the Texts and "Affect Dimension" for the Class name, is it correct to put them like this is the fields section?

Edit: TabularDataset includes an Example object, in which the data can be read. When reading csv files, only a "," is accepted as a delimiter. Everything else will result in corrupted data.


title: What is the range for nll_loss?

body:
I thought the range was only within the positive domain, but I am getting both negative numbers and positive numbers for F.nll_loss?

Why is that? I am confused. Softmax ranges from 0 to 1 and -log(0 to 1) is from infinity to 0. So, why am I getting negative numbers?


title: (Distributed) Why all gpus are giving the same output?

body:
I am a new comer to PyTorch and I'm confused when I am running the official example of torch.distributed
at PyTorch ImageNet main.py L304.

I have made some small modification on the evaluation part of the source code like below:

model.eval()
    with torch.no_grad():
        end = time.time()
        for i, (images, target, image_ids) in enumerate(val_loader):
            if args.gpu is not None:
                images = images.cuda(args.gpu, non_blocking=True)

            target = target.cuda(args.gpu, non_blocking=True)
            image_ids = image_ids.data.cpu().numpy()
            output = model(images)
            loss = criterion(output, target)

            # Get acc1, acc5 and update
            acc1, acc5 = accuracy(output, target, topk=(1, 5))
            losses.update(loss.item(), images.size(0))
            top1.update(acc1[0], images.size(0))
            top1.update(acc1[0], images.size(0))
            top5.update(acc5[0], images.size(0))

            # print at i-th batch of images only
            dist.barrier()
            if i==0:
                if args.gpu==0:
                    print("gpu 0",acc1,output.shape)
                if args.gpu==1:
                    print("gpu 1",acc1,output.shape)
                if args.gpu==2:
                    print("gpu 2",acc1,output.shape)
                if args.gpu==3:
                    print("gpu 3",acc1,output.shape)

And above code gives the following output:

Use GPU: 0 for training
Use GPU: 1 for training
Use GPU: 3 for training
Use GPU: 2 for training
=> loading checkpoint model_best.pth.tar'
...
gpu 3 tensor([75.], device='cuda:3') torch.Size([32, 200])
gpu 2 tensor([75.], device='cuda:2') torch.Size([32, 200])
gpu 1 tensor([75.], device='cuda:1') torch.Size([32, 200])
gpu 0 tensor([75.], device='cuda:0') torch.Size([32, 200])

As I am using 4 GPU with a batch size of 128, I think 128 images have been divided and fed into 4 GPU respectively. So all the four GPU have output.shape[0]=32(where 200 is num_classes).

But what has really confused me is that, all the 4 GPU are showing the same acc1. In my understanding, as 4 GPUs are taking different input portion (32 images respectively), they should also give different output and accuracy corresponding to their input respectively. However, in my print test, these GPU are showing the same output and accuracy. And I don't know why, shouldn't they be different ?

Looking for help. Thank you in advance !


title: Android(kotlin), how getting assets file path? (at pytorch mobile)

body:
I'm trying PyTorch Mobile tutorial by kotlin. I want to load module, "model.pt" in assets file. But no idea to load module in assets file.

Java (written in PyTorch Mobile Tutorial "hello world")

Module module = Module.load(assetFilePath(this, "model.pt"));

kotlin

val module = Module.load("?????")


title: How can I change my Pytorch module (with in-place operations) to be differentiable?

body:
My layer is like this (I am making an LSTM layer with dropout applied in every time-step, the input is passed through 10 times, and the average of the outputs is returned)

import torch
from torch import nn

class StochasticLSTM(nn.Module):
    def __init__(self, input_size: int, hidden_size: int, dropout_rate: float):
        """
        Args:
        - dropout_rate: should be between 0 and 1
        """
        super(StochasticLSTM, self).__init__()

        self.iter = 10
        self.input_size = input_size
        self.hidden_size = hidden_size

        if not 0 <= dropout_rate <= 1:
            raise Exception("Dropout rate should be between 0 and 1")
        self.dropout = dropout_rate
        self.bernoulli_x = torch.distributions.Bernoulli(
            torch.full((self.input_size,), 1 - self.dropout)
        )
        self.bernoulli_h = torch.distributions.Bernoulli(
            torch.full((hidden_size,), 1 - self.dropout)
        )

        self.Wi = nn.Linear(self.input_size, self.hidden_size)
        self.Ui = nn.Linear(self.hidden_size, self.hidden_size)

        self.Wf = nn.Linear(self.input_size, self.hidden_size)
        self.Uf = nn.Linear(self.hidden_size, self.hidden_size)

        self.Wo = nn.Linear(self.input_size, self.hidden_size)
        self.Uo = nn.Linear(self.hidden_size, self.hidden_size)

        self.Wg = nn.Linear(self.input_size, self.hidden_size)
        self.Ug = nn.Linear(self.hidden_size, self.hidden_size)

    def forward(self, input, hx=None):
        """
        input shape (sequence, batch, input dimension)
        output shape (sequence, batch, output dimension)
        return output, (hidden_state, cell_state)
        """

        T, B, _ = input.shape

        if hx is None:
            hx = torch.zeros((self.iter, T + 1, B, self.hidden_size), dtype=input.dtype)
        else:
            hx = hx.unsqueeze(0).repeat(self.iter, T + 1, B, self.hidden_size)

        c = torch.zeros((self.iter, T + 1, B, self.hidden_size), dtype=input.dtype)
        o = torch.zeros((self.iter, T, B, self.hidden_size), dtype=input.dtype)

        for it in range(self.iter):
            # Dropout
            zx = self.bernoulli_x.sample()
            zh = self.bernoulli_h.sample()

            for t in range(T):
                x = input[t] * zx
                h = hx[it, t] * zh

                i = torch.sigmoid(self.Ui(h) + self.Wi(x))
                f = torch.sigmoid(self.Uf(h) + self.Wf(x))

                o[it, t] = torch.sigmoid(self.Uo(h) + self.Wo(x))
                g = torch.tanh(self.Ug(h) + self.Wg(x))

                c[it, t + 1] = f * c[it, t] + i * g
                hx[it, t + 1] = o[it, t] * torch.tanh(c[it, t + 1])

        o = torch.mean(o, axis=0)
        c = torch.mean(c[:, 1:], axis=0)
        hx = torch.mean(hx[:, 1:], axis=0)

        return o, (hx, c)

When I optimizer the network, I have the error one of the variables needed for gradient computation has been modified by an inplace operation. We can spot several inplace operations such as o[it, t] = torch.sigmoid(self.Uo(h) + self.Wo(x)).

How can I avoid this inplace operation when I want to find the average?

Thanks


title: why is the accuracy of my pretrained resnet-152 model so low?

body:
I am fairly new to deep learning and neural networks. I recently built a facial emotions recognition classifier using the FER-2013 dataset. I am using the pretrained resnet-152 model for classification, but the accuracy of my model is very low, both training and validation accuracies. I am getting an accuracy of around 36%, which is not good. I suppose that using transfer learning, the accuracies should be high, why is it that im getting such a low accuracy. should I change the hyperparameters? here is my code.

    model= models.resnet152(pretrained=True)

    for param in model.parameters():
      param.requires_grad= False

    print(model)
    from collections import OrderedDict

    classifier= nn.Sequential(OrderedDict([
                                           ('fc1',nn.Linear(2048, 512)),
                                           ('relu', nn.ReLU()),
                                           ('dropout1', nn. Dropout(p=0.5)),
                                           ('fc2', nn.Linear(512, 7)),
                                           ('output', nn.LogSoftmax(dim=1))
    ]))
    model.fc= classifier
    print(classifier)
    def train_model(model, criterion, optimizer, scheduler, num_epochs=10):
      since= time.time()

      best_model_wts= copy.deepcopy(model.state_dict())
      best_acc= 0.0

      for epoch in range(1, num_epochs + 1):
        print('Epoch {}/{}'.format(epoch, num_epochs))
        print('-' * 10)

        for phase in ['train', 'validation']:
          if phase == 'train':
            scheduler.step()
            model.train()
          else:
            model.eval()

          running_loss= 0.0
          running_corrects=0

          for inputs, labels in dataloaders[phase]:
            inputs, labels= inputs.to(device), labels.to(device)

            optimizer.zero_grad()

            with torch.set_grad_enabled(phase== 'train'):
              outputs= model(inputs)
              loss= criterion(outputs, labels)
              _, preds= torch.max(outputs, 1)

              if phase == 'train':
                loss.backward()
                optimizer.step()

            running_loss += loss.item() * inputs.size(0)
            running_corrects += torch.sum(preds== labels.data)

          epoch_loss= running_loss / dataset_sizes[phase]
          epoch_acc= running_corrects.double() / dataset_sizes[phase]

          print('{} Loss: {:.4f} Acc: {:.4f}'.format(phase, epoch_loss, epoch_acc))

          if phase == 'validation' and epoch_acc > best_acc:
            best_acc= epoch_acc
            best_model_wts= copy.deepcopy(model.state_dict())

      time_elapsed= time.time() - since
      print('Training complete in {:.0f}m {:.0f}s'.format(
          time_elapsed // 60, time_elapsed % 60))
      print('Best valid accuracy: {:4f}'.format(best_acc))

      model.load_state_dict(best_model_wts)
      return model

    use_gpu= torch.cuda.is_available()
    num_epochs= 10
    if use_gpu:
      print('Using GPU: '+ str(use_gpu))
      model= model.cuda()

    criterion= nn.NLLLoss()

    optimizer= optim.SGD(model.fc.parameters(), lr = .0006, momentum=0.9)
    exp_lr_scheduler = lr_scheduler.StepLR(optimizer, step_size=5, gamma=0.1)

    model_ft = train_model(model, criterion, optimizer, exp_lr_scheduler, num_epochs=10)

Can someone please guide me. I am a beginner at it, and I could really make use of some help in it.


title: How to include pytorch in pyinstaller app?

body:
I am trying to use pyinstaller to "freeze" a python app I've written that uses pytorch. I see that pyinstaller has a hook for torch: https://github.com/pyinstaller/pyinstaller/blob/develop/PyInstaller/hooks/hook-torch.py

But it doesn't appear to work. I still get the following error:

  File "torch/_utils_internal.py", line 53, in get_source_lines_and_file
  OSError: Can't get source for . TorchScript requires source access in order to carry out
  compilation, make sure original .py files are available. Original
  error: could not get source code [28326] Failed to execute script
  timeslide

I also tried this work-around with no success:

https://github.com/pyinstaller/pyinstaller/issues/2666#issuecomment-370789381

I just need a way to get pytorch to install in a pyinstaller app. It doesn't need to be clean or beautiful. Can anyone suggest a solution?

(macOS and python 3.7)


title: What are some ways to speed up data loading on large sparse arrays (~1 million x 1 million, density ~0.0001) in Pytorch?

body:
I am working on a binary classification problem. I have ~1.5 million data points, and the dimensionality of the feature space is 1 million. This dataset is stored as a sparse array, with a density of ~0.0001. For this post, I'll limit the scope to assume that the model is a shallow feedforward neural network, and also assume that the dimensionality has already been optimized (so cannot be reduced below 1 million). Naiive approaches to create mini-batches out of this data to feed into the network would take a lot of time (As an example, a basic approach of creating a TensorDataset (map style) from a torch.sparse.FloatTensor representation of the input array, and wrapping a DataLoader around it, means ~20s to get a mini-batch of 32 to the network, as opposed to say ~0.1s to perform the actual training). I am looking for ways to speed this up.
What I've tried

I first figured that reading from such a large sparse array in every iteration of the DataLoader was computationally intensive, so I broke down this sparse array into smaller sparse arrays
For the DataLoader to read from these multiple sparse arrays in an iterative fashion, I replaced the map style dataset that I had inside the DataLoader with an IterableDataset, and streamed these smaller sparse arrays into this IterableDataset like so:

from itertools import chain
from scipy import sparse

class SparseIterDataset(torch.utils.data.IterableDataset):

    def __init__(self, fpaths):
        super(SparseIter).__init__()
        self.fpaths = fpaths

    def read_from_file(self, fpath):
        data = sparse.load_npz(fpath).toarray()
        for d in data:
            yield torch.Tensor(d)

    def get_stream(self, fpaths):
        return chain.from_iterable(map(self.read_from_file, fpaths))

    def __iter__(self):
        return self.get_stream(self.fpaths)

With this approach, I was able to bring down the time from the naiive base case of ~20s to ~0.2s per minibatch of 32. However, given that my dataset has ~1.5 million samples, this still implies a lot of time spent on even making one pass through the dataset. (As a comparison, even though it's slightly apples to oranges, running a logistic regression on scikit-learn on the original sparse array takes about ~6s per iteration through the whole dataset. With pytorch, with the approach I just outlined, it would take ~3000s just to load all the minibatches in an epoch)
One thing which I am aware of but yet to try is using multiprocess data loading by setting the num_workers argument in the DataLoader. I believe this has its own catches in the case of iterable style datasets though. Plus even a 10x speedup would still mean ~300s per epoch in loading mini batches. I feel I'm being inordinately slow! Are there any other approaches/improvements/best practices that you could suggest?


title: Error in loading state_dict for customed model

body:
I had problems when loading the weights of model. Here's some parts of the model

class InceptionV4(nn.Module):

   def __init__(self, num_classes=1001):
       super(InceptionV4, self).__init__()
       # Special attributs
       self.input_space = None
       self.input_size = (299, 299, 3)
       self.mean = None
       self.std = None
       # Modules
       self.features = nn.Sequential(
           BasicConv2d(3, 32, kernel_size=3, stride=2),
           BasicConv2d(32, 32, kernel_size=3, stride=1),
           BasicConv2d(32, 64, kernel_size=3, stride=1, padding=1),
           Mixed_3a(),
           Mixed_4a(),
           Mixed_5a(),
           Inception_A(),
           Inception_A(),
           Inception_A(),
           ...
       )
       self.avg_pool = nn.AvgPool2d(8, count_include_pad=False)
       self.last_linear = nn.Linear(1536, num_classes)

I have tried to save the weights, something like torch.save(model.state_dict(), weight_name) and then reload again model.load_state_dict(torch.load(weight_name))
but got these errors:

Missing key(s) in state_dict: "features.0.conv.weight", "features.0.bn.weight", "features.0.bn.bias", "features.0.bn.running_mean", "features.0.bn.running_var", "features.1.conv.weight", "features.1.bn.weight", "features.1.bn.bias", "features.1.bn.running_mean", "features.1.bn.running_var", "features.2.conv.weight", "features.2.bn.weight

and also:

Unexpected key(s) in state_dict: "conv.0.conv1.0.weight", "conv.0.conv1.0.bias", "conv.0.conv1.2.weight", "conv.0.conv1.2.bias", "conv.0.conv1.2.running_mean", "conv.0.conv1.2.running_var", "conv.0.conv1.2.num_batches_tracked", "conv.0.conv2.0.weight", "conv.0.conv2.0.bias", "conv.0.conv2.2.weight", "conv.0.conv2.2.bias", "conv.0.conv2.2.running_mean", "conv.0.conv2.2.running_var", "conv.0.conv2.2.num_batches_tracked", "conv.1.conv1.0.weight", "conv.1.conv1.0.bias", "conv.1.conv1.2.weight", "conv.1.conv1.2.bias", "conv.1.conv1.2.running_mean", "conv.1.conv1.2.running_var", "conv.1.conv1.2.num_batches_tracked

Any hints on this? Thanks in advance.


title: I am getting an error while installing PATE from SYFT

body:
I am doing a Differential Privacy Course and have to work with syft. When running the below command, I encountered a error

I have already installed syft package required for the analysis. And I am working in Anaconda on Windows 10

from syft.frameworks.torch.differential_privacy import pate

WARNING:tf_encrypted:Falling back to insecure randomness since the required custom op could not be found for the installed version of TensorFlow (1.13.1). Fix this by compiling custom ops.


title: Optimizer parameters missing in Pytorch

body:
I have 2 nets sharing one optimizer using different learning rate. Simple code shown as below:

optim = torch.optim.Adam([
{'params': A.parameters(), 'lr': args.A},
{'params': B.parameters(), 'lr': args.B}])

Is this right? I ask this because when I check parameters in optimizer (using code below), I found only 2 parameters.

for p in optim.param_groups:
outputs = ''
for k, v in p.items():
    if k is 'params':
        outputs += (k + ': ' + str(v[0].shape).ljust(30) + ' ')
    else:
        outputs += (k + ': ' + str(v).ljust(10) + ' ')
print(outputs)

Only 2 parameters are printed:

params: torch.Size([16, 1, 80])        lr: 1e-05      betas: (0.9, 0.999) eps: 1e-08      weight_decay: 0          amsgrad: False

params: torch.Size([30, 10])           lr: 1e-05      betas: (0.9, 0.999) eps: 1e-08      weight_decay: 0          amsgrad: False

Actually, 2 nets have more than 100 parameters. I thought all parameters will be printed. Why is this happening? Thank you!


title: "No such file" when loading csv data stored in G drive to torchtext format using torchtext.data.TabularDataset,

body:
I have stored a csv file in G drive and try to load it to torchtext data.TabularDataset. The error message is "FileNotFoundError: [Errno 2] No such file or directory: 'https://.....'"

Is it impossible to load csv file from g drive directly to torchtext TabularDataset?

Here is the code. I have also made a public colab notebook with data publicly available.

import torch
from torchtext import data, datasets

!pip install -U -q PyDrive
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
from google.colab import auth
from oauth2client.client import GoogleCredentials

auth.authenticate_user()
gauth = GoogleAuth()
gauth.credentials = GoogleCredentials.get_application_default()
drive = GoogleDrive(gauth)

TEXT = data.Field(tokenize = 'spacy', batch_first = True, lower=False)
LABEL = data.LabelField(sequential=False, dtype = torch.float)

train = data.TabularDataset(path = 'https://drive.google.com/open?id=1eWMjusU3H34m0uml5SdJvYX6gQuB8zta',
                            format = 'csv',
                            fields = [('Insult', LABEL), (None, None), ('Comment', TEXT)],
                            skip_header=False)


Some older code of mine was working fine for the last 2 months until it suddenly started giving this error today. I can't figure out what changed because I didn't touch this code. But I don't understand this new error:

INFO:pytorch_transformers.tokenization_utils:loading file https://s3.amazonaws.com/models.huggingface.co/bert/xlm-clm-ende-1024-vocab.json from cache at /root/.cache/torch/pytorch_transformers/6e42a59f5e60f1efc6116fd1a2c05a72ecf713a3022b9c274b727ed6469e6ac1.2c29a4b393decdd458e6a9744fa1d6b533212e4003a4012731d2bc2261dc35f3
INFO:pytorch_transformers.tokenization_utils:loading file https://s3.amazonaws.com/models.huggingface.co/bert/xlm-mlm-ende-1024-merges.txt from cache at /root/.cache/torch/pytorch_transformers/85d878ffb1bc2c3395b785d10ce7fc91452780316140d7a26201d7a912483e44.42fa32826c068642fdcf24adbf3ef8158b3b81e210a3d03f3102cf5a899f92a0
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-31-c365f437b895> in <module>()
      9 tokenizer = tokenizer_class.from_pretrained(args['model_name'])
     10
---> 11 model = model_class.from_pretrained(args['model_name'])
     12 model.to(device);
     13

3 frames
/usr/local/lib/python3.6/dist-packages/pytorch_transformers/modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    534
    535         # Instantiate model.
--> 536         model = cls(config, *model_args, **model_kwargs)
    537
    538         if state_dict is None and not from_tf:

/usr/local/lib/python3.6/dist-packages/pytorch_transformers/modeling_xlm.py in __init__(self, config)
    842         self.num_labels = config.num_labels
    843
--> 844         self.transformer = XLMModel(config)
    845         self.sequence_summary = SequenceSummary(config)
    846

/usr/local/lib/python3.6/dist-packages/pytorch_transformers/modeling_xlm.py in __init__(self, config)
    543         if config.n_langs > 1 and config.use_lang_emb:
    544             self.lang_embeddings = nn.Embedding(self.n_langs, self.dim)
--> 545         self.embeddings = nn.Embedding(self.n_words, self.dim, padding_idx=self.pad_index)
    546         self.layer_norm_emb = nn.LayerNorm(self.dim, eps=config.layer_norm_eps)
    547

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/sparse.py in __init__(self, num_embeddings, embedding_dim, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse, _weight)
     86         if padding_idx is not None:
     87             if padding_idx > 0:
---> 88                 assert padding_idx < self.num_embeddings, 'Padding_idx must be within num_embeddings'
     89             elif padding_idx < 0:
     90                 assert padding_idx >= -self.num_embeddings, 'Padding_idx must be within num_embeddings'

AssertionError: Padding_idx must be within num_embeddings

Can someone shed any light on what might be happening?

Thanks a lot!


I have two columns in the file: [Text Label]
I'm trying to create a vocab using torchtext.Data.TabularDataset

dataFields =  [("Text", TEXT), ("Label", LABEL)]

    trainTDB = TabularDataset(
        path="data/trainData.tsv",
        format='tsv',
        skip_header=True,
        fields=dataFields)

Upon trying the above code I'm getting following error: AttributeError: 'Example' object has no attribute 'Label'

Any fix or clue?


I am trying to run a python file that makes use of libtorch  however I am getting the following error

terminate called after throwing an instance of 'c10::Error'
  what():  CUDA error: invalid device ordinal (exchangeDevice at /pytorch/c10/cuda/impl/CUDAGuardImpl.h:31)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x33 (0x7f6af0e6c813 in /cs/home/mm470/Documents/Workspace/Dissertation/libtorch/lib/libc10.so)
frame #1: <unknown function> + 0xdf2d (0x7f6aedf75f2d in /cs/home/mm470/Documents/Workspace/Dissertation/libtorch/lib/libc10_cuda.so)
frame #2: <unknown function> + 0x41beafa (0x7f6af524fafa in /cs/home/mm470/Documents/Workspace/Dissertation/libtorch/lib/libtorch.so)
frame #3: <unknown function> + 0x3c78098 (0x7f6af4d09098 in /cs/home/mm470/Documents/Workspace/Dissertation/libtorch/lib/libtorch.so)
frame #4: <unknown function> + 0x1bd01f1 (0x7f6af2c611f1 in /cs/home/mm470/Documents/Workspace/Dissertation/libtorch/lib/libtorch.so)
frame #5: at::native::to(at::Tensor const&, c10::TensorOptions const&, bool, bool) + 0x272 (0x7f6af2c61b92 in /cs/home/mm470/Documents/Workspace/Dissertation/libtorch/lib/libtorch.so)
frame #6: <unknown function> + 0x1ef9810 (0x7f6af2f8a810 in /cs/home/mm470/Documents/Workspace/Dissertation/libtorch/lib/libtorch.so)
frame #7: <unknown function> + 0x3a89ce3 (0x7f6af4b1ace3 in /cs/home/mm470/Documents/Workspace/Dissertation/libtorch/lib/libtorch.so)
frame #8: void torch::nn::Module::to_impl<c10::Device&, bool&>(c10::Device&, bool&) + 0x14f (0x7f6af5187cef in /cs/home/mm470/Documents/Workspace/Dissertation/libtorch/lib/libtorch.so)
frame #9: void torch::nn::Module::to_impl<c10::Device&, bool&>(c10::Device&, bool&) + 0x83 (0x7f6af5187c23 in /cs/home/mm470/Documents/Workspace/Dissertation/libtorch/lib/libtorch.so)
frame #10: torch::nn::Module::to(c10::Device, bool) + 0x1b (0x7f6af518426b in /cs/home/mm470/Documents/Workspace/Dissertation/libtorch/lib/libtorch.so)
frame #11: Init + 0x16f (0x7f6b32954cef in /cs/home/mm470/Documents/Workspace/Dissertation/combopt-zero/vertex-cover/lib/build/lib_vertex_cover.so)
frame #12: ffi_call_unix64 + 0x4c (0x7f6b3915be2c in /lib64/libffi.so.6)
frame #13: ffi_call + 0x1f5 (0x7f6b3915b755 in /lib64/libffi.so.6)
frame #14: _ctypes_callproc + 0x2a0 (0x7f6b3936e680 in /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so)
frame #15: <unknown function> + 0x97c6 (0x7f6b393677c6 in /usr/lib64/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so)
<omitting python frames>
frame #35: __libc_start_main + 0xf5 (0x7f6b4ad80555 in /lib64/libc.so.6)
frame #36: python() [0x400c40]


title: Different usage of ,  and  tokens

body:
I found that there are many different usages of <PAD>, <EOS>, and <GO> tokens.

Personally, I separate those three tokens and assign different embeddings to them, assigning an all-zero embedding vector to <PAD> token specifically (with RNN-based seq2seq model).

The majority of codes show that <PAD>, <EOS> and <GO> are all represented as <PAD> token.

I want to know if there is the optimum usage of those tokens (in terms of RNN-based models or transformer-based models).


title: How to create a 1D sparse tensors from given list of indices and values?

body:
I have a list of indices and values. I want to create a sparse tensor of size 30000 from this indices and values as follows.
indices = torch.LongTensor([1,3,4,6])
values = torch.FloatTensor([1,1,1,1])

So, I want to build a 30k dimensional sparse tensor in which the indices [1,3,4,6] are ones and the rest are zeros. How can I do that?
I want to store the sequences of such sparce tensors efficiently.


title: Training a CNN model with DataLoader on a GPU in PyTorch

body:
I am trying to classify chest x-rays into two categories: 'normal', and 'pneumonia'. My training and test sets are DataLoader objects with num_workers=0, pin_memory=True. Cuda is available on my device (GTX 1060 6GB). After creating the CNN I call
model = CNN().cuda(). When I try to train the model I get RuntimeError: Expected object of backend CPU but got backend CUDA for argument #2 'weight'. What do I have to change in order to train the model on my GPU?
The code:
root = 'chest_xray/'

train_data = datasets.ImageFolder(os.path.join(root, 'train'), transform=train_transform)
test_data = datasets.ImageFolder(os.path.join(root, 'test'), transform=test_transform)

train_loader = DataLoader(train_data,batch_size=10,shuffle=True,num_workers=0,pin_memory=True)
test_loader = DataLoader(test_data,batch_size=10,shuffle=False,num_workers=0,pin_memory=True)

class_names = train_data.classes

class CNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(3, 12, 5, 1)
        self.conv2 = nn.Conv2d(12, 24, 5, 1)
        self.conv3 = nn.Conv2d(24, 30, 5, 1)
        self.conv4 = nn.Conv2d(30, 36, 5, 1)
        self.fc1 = nn.Linear(58*58*36, 256)
        self.fc2 = nn.Linear(256, 144)
        self.fc3 = nn.Linear(144, 84)
        self.fc4 = nn.Linear(84, 16)
        self.fc5 = nn.Linear(16, 2)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        x = F.max_pool2d(x, 2, 2)
        x = F.relu(self.conv2(x))
        x = F.max_pool2d(x, 2, 2)
        x = F.relu(self.conv3(x))
        x = F.max_pool2d(x, 2, 2)
        x = F.relu(self.conv4(x))
        x = F.max_pool2d(x, 2, 2)

        x = x.view(-1, 58*58*36)

        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.relu(self.fc3(x))
        x = F.relu(self.fc4(x))
        x = self.fc5(x)

        return F.log_softmax(x, dim=1)

model = CNN().cuda()

criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)

epochs = 2

train_losses = []
train_correct = []
for i in range(epochs):
    trn_correct = 0
    tst_correct = 0

    for b, (X_train, y_train) in enumerate(train_loader):
        y_pred = model(X_train)
        loss = criterion(y_pred, y_train)

        predicted = torch.max(y_pred.data, 1)[1]
        batch_correct = (predicted == y_train).sum()
        trn_correct += batch_correct

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if b % 200 == 0:
            print(f'epoch: {i+1} batch: {b} progress: {10*b/len(train_data)} loss: {loss.item()} accuracy: {10*trn_correct/b}%')

    train_losses.append(loss)
    train_correct.append(trn_correct)