import torch

import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

class sentimentAnalyser(nn.Module) :
    def __init__(self, vocab_dim, embed_dim, num_filters, FFN_units, dropout_rate, training, batch_size) :
    super(sentimentAnalyser, self).__init__()
    self.sentence_len = 73
    self.embed_dim = embed_dim
    self.batch_size = batch_size
    self.embedding = nn.Embedding(vocab_dim, embed_dim)
    self.bigram = nn.Conv2d(in_channels=1,
                           out_channels=num_filters,
                           kernel_size=(2, embed_dim))
    self.trigram = nn.Conv2d(in_channels=1,
                            out_channels=num_filters,
                            kernel_size=(3, embed_dim))
    self.fourgram = nn.Conv2d(in_channels=1,
                             out_channels=num_filters,
                             kernel_size=(4, embed_dim))

    self.maxpool = lambda x : torch.max(x, dim=-1).values
    self.dense1 = nn.Linear(in_features=num_filters*3,
                           out_features=FFN_units)
    self.dropout = nn.Dropout(p=dropout_rate)
    self.dense2 = nn.Linear(in_features=FFN_units,
                           out_features=2)

    def forward(self, x, training) :

        x = self.embedding(x).view((self.batch_size, 1, self.sentence_len, self.embed_dim))
        x1 = torch.squeeze(self.bigram(x))
        x2 = torch.squeeze(self.trigram(x))
        x3 = torch.squeeze(self.fourgram(x))

        x1_maxpool = self.maxpool(x1)
        x2_maxpool = self.maxpool(x2)
        x3_maxpool = self.maxpool(x3)

        concat_x = torch.cat((x1_maxpool, x2_maxpool, x3_maxpool), dim=-1)
        output = self.dense1(concat_x)
        output = F.relu(output)
        if training : output = self.dropout(output)

        output = F.softmax(self.dense2(output)).type(torch.float32)
        return output

x = torch.from_numpy(train_data).type(torch.long)
y = torch.from_numpy(train_labels).type(torch.long)
bs = 64 # batch size is 64
EPOCH = 5
model = sentimentAnalyser(vocab_dim=VOCAB_SIZE,
                     embed_dim=EMB_DIM,
                     num_filters=NUM_FILTERS,
                     FFN_units=FFN_UNITS,
                     dropout_rate=0.2,
                     training=True,
                     batch_size=bs)

loss = nn.BCELoss()
optimizer = optim.SGD(model.parameters(), lr=0.01)

for epoch in range(EPOCH) :
    i = 0
    while i < x.shape[0] :
    inp = x[i : min(i + bs, x.shape[0])]
    out = y[i : min(i + bs, x.shape[0])]
    i = i + bs
    pred_out = model.forward(inp, training=True)
    pred_out = torch.max(pred_out, dim=1).indices
    l = loss(pred_out, out)
    l.backward()
    optimizer.step()

    optimizer.zero_grad()
    print(f'Epoch : {epoch}, Loss is {l}')

When running this code, I get an error :
    _thnn_binary_cross_entropy_forward not supported on CPUType for Long

It seems that BCELoss is not defined for tensors of type torch.long, but on the other hand, nn.Embedding
layer is only defined for torch.long tensors. I have tried to pass the input tensor as long to nn.embedding and then casting it to torch.float32 and other types, but still, it doesn't work.
Please Help!!


title: Using the T5 model with huggingface's mask-fill pipeline

body:
Does anyone know if it is possible to use the T5 model with hugging face's mask-fill pipeline?  The below is how you can do it using the default model but i can't seem to figure out how to do is using the T5 model specifically?

from transformers import pipeline
nlp_fill = pipeline('fill-mask')
nlp_fill('Hugging Face is a French company based in ' + nlp_fill.tokenizer.mask_token)

Trying this for example raises the error "TypeError: must be str, not NoneType" because nlp_fill.tokenizer.mask_token is None.

nlp_fill = pipeline('fill-mask',model="t5-base", tokenizer="t5-base")
nlp_fill('Hugging Face is a French company based in ' + nlp_fill.tokenizer.mask_token)


title: Does my for loop run parallely, if all the tensors involved in the loop are on the GPU?

body:
I have a list of tensors and all of them are present on the GPU. I obtained this list by splitting one tensor on the GPU using torch.split. I want to get list of sums of the list of tensors I have. So, in simple terms, I want to get a list in which, the first element is sum of first tensor in the list, and so on. If I run a for loop for this, does it get parallelised? If not, is there a way to make it run parallely? I want to parallelize it since the list is pretty long, and the sum operation can be done parallely, and independently on every tensor present on the list. If this operation can be performed on the GPU, the performance gain would be immense.
UPDATE : Consider I have a list of tensors as follows :
ls
[tensor([[0.8469, 0.3712, 0.2956],
         [0.6548, 0.5284, 0.8682],
         [0.5748, 0.2390, 0.1402],
         [0.0010, 0.1794, 0.6048],
         [0.4636, 0.4101, 0.6543]], device='cuda:0'),
 tensor([[0.2138, 0.3613, 0.8712],
         [0.4689, 0.0503, 0.7342],
         [0.1368, 0.0688, 0.9223]], device='cuda:0'),
 tensor([[0.3131, 0.6142, 0.1555],
         [0.4099, 0.5000, 0.7578],
         [0.7353, 0.2425, 0.4407],
         [0.5943, 0.0377, 0.4820],
         [0.5898, 0.9585, 0.6993]], device='cuda:0'),
 tensor([[0.8629, 0.3172, 0.4248],
         [0.9957, 0.6998, 0.0931],
         [0.0258, 0.9898, 0.5250]], device='cuda:0'),
 tensor([[0.0298, 0.4033, 0.9465],
         [0.2763, 0.9412, 0.4873]], device='cuda:0')]

As you can see, I have a list of 5 tensors of different shapes. Each tensor has a shape of 3 in their first dimension. The shape is different because of 0th dimension. So, in this example, the shapes of the tensor in the list are [[5,3], [3, 3], [5, 3], [3, 3], [2,3]]. I want to get a list of tensors from this list as follows :
sums = [torch.sum(li, axis=0) for li in ls]
sums
[tensor([2.5412, 1.7280, 2.5632], device='cuda:0'),
 tensor([0.8195, 0.4804, 2.5277], device='cuda:0'),
 tensor([2.6424, 2.3528, 2.5352], device='cuda:0'),
 tensor([1.8844, 2.0068, 1.0429], device='cuda:0'),
 tensor([0.3062, 1.3445, 1.4338], device='cuda:0')]

So, as you can see, the first tensor in the list is sum of first tensor in the list ls along the dimension 0. The second tensor is sum of second tensor in list ls along the dimension 0 and so on.
To do this task, I'm currently using a for loop. which iteratively calculates the sums and appends it to sums list. However, this is very inefficient as my list of tensors is really big, of the order of 100K, and doing this in each iteration is super inefficient. I wanted to find out if there is any way to do this more efficiently.
The list ls of tensors is obtained by splitting a big tensor like this :
splitter = [5, 3, 5, 3, 2]

A = torch.rand(18, 3).cuda()

ls = torch.split(A, splitter)
ls
(tensor([[0.1969, 0.6113, 0.3563],
         [0.9180, 0.7759, 0.5953],
         [0.0279, 0.4014, 0.2268],
         [0.9026, 0.3821, 0.1498],
         [0.3630, 0.9144, 0.3277]], device='cuda:0'),
 tensor([[2.1312e-02, 5.2311e-01, 8.9177e-02],
         [4.7427e-01, 2.4503e-04, 1.2559e-01],
         [5.1641e-01, 9.1357e-01, 9.5637e-01]], device='cuda:0'),
 tensor([[0.3730, 0.4251, 0.9437],
         [0.5634, 0.3086, 0.5891],
         [0.5602, 0.0872, 0.2128],
         [0.7717, 0.1920, 0.3977],
         [0.5787, 0.3488, 0.7499]], device='cuda:0'),
 tensor([[0.9338, 0.4330, 0.8843],
         [0.5646, 0.0574, 0.8790],
         [0.4692, 0.5831, 0.9160]], device='cuda:0'),
 tensor([[0.9786, 0.5209, 0.9364],
         [0.4370, 0.4917, 0.3672]], device='cuda:0'))

So, if avoiding the for loop is not possible, do anyone have any ideas on summing the main tensor A, according to a splitter provided? So, for example, in the code above, the splitter is [5, 3, 5, 3, 2]. So, I want to obtain a tensor res from tensor A such that the first row of res is sum of first 5 rows of A(because splitter[0] = 5) along dim=0. The second row of res is sum of next 3 rows (row 5 to row 7) of A. And so on. Can I do this without using a for loop? Or can I parallelise this for loop since the opearation it is doing are independent of each other and are mutually exclusive and exhaustive.
I hope the added details are enough. If I need to add any further details to the question, please let me know. Thanks in advance :)


title: Segmentation fault (core dumped) with RTX 3080

body:
Hi~ I just get RTX 3080 and meet lots of version problems. One of them is the problem that pop up “Segmentation fault (core dumped)” and multiprocessing issue.
There is my device list below:

GPU: RTX 3080
CPU: Intel i7-10700
CUDA: 11.1
RAM: 32GB
Python: 3.7.5
Pytorch: 1.7.0.dev20200927+cu110

And the following is the issue:
https://i.stack.imgur.com/AUGWH.png
Did anyone encounter the same problem? Thanks~~


title: AttributeError: 'dict' object has no attribute 'get_epoch_log_metrics' Pytorch Lightning

body:
I have created a lightning module which is working fine for single validation dataset but throws following error while using multiple validation dataset:
self._log_on_evaluation_epoch_end_metrics(epoch_logs)
  File "/home/ubuntu/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/pytorch_lightning/trainer/connectors/logger_connector.py", line 186, in _log_on_evaluation_epoch_end_metrics
    logger_metrics = reduced_epoch_metrics.get_epoch_log_metrics()
AttributeError: 'dict' object has no attribute 'get_epoch_log_metrics'

Here's my validation_step method. To support single as well as multiple validation dataset I have kept dataset_idx = 0
    def validation_step(self, batch, batch_idx, dataset_idx=0):
        qa_pairs, labels = batch
        # labels = labels.type(torch.LongTensor)
        labels = labels.long()#(torch.LongTensor)
        # run the model for the inputs
        outputs = self(qa_pairs)
        # compute the loss
        if self.loss.lower() == "cross_entropy":
            loss = F.cross_entropy(outputs, labels)

        self.valid_acc(outputs, labels)
        self.log('val_loss', loss, on_step=True)
        self.log('val_acc', self.valid_acc, on_step=True, on_epoch=True, prog_bar=True)

system:
pytorch: 1.6
ubuntu 18.04


title: Increase speed Huggingface tokenizer ouput

body:
I need to get the last layer of embeddings from a BERT model using HuggingFace. The following code works, however is extremely slow, how can I increase the speed?
This is a toy example, my real data is made of thousands of examples with long texts.
import transformers
import pandas as pd
from transformers import BertModel, BertTokenizer

tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
model = BertModel.from_pretrained("bert-base-uncased")

def getPrediction(text):
  encoded_input = tokenizer(text, return_tensors='pt')
  outputs = model(**encoded_input)
  embedding = outputs[0][:, -1]
  embedding_1ist = embedding.cpu().detach().tolist()[0]
  return embedding_1ist

df = pd.DataFrame({'text':['First text', 'Second text']})
results =  pd.DataFrame(df.apply(lambda x: getPrediction(x.text), axis=1))


title: Pytorch tensor dimension multiplication

body:
I'm trying to implement the grad-camm algorithm:
https://arxiv.org/pdf/1610.02391.pdf
My arguments are:
activations: Tensor with shape torch.Size([1, 512, 14, 14])
alpha values : Tensor with shape torch.Size([512])
I want to multiply each activation (in dimension index 1 (sized 512)) in each corresponding alpha value: for example if the i'th index out of the 512 in the activation is 4 and the i'th alpha value is 5, then my new i'th activation would be 20.
The shape of the output should be torch.Size([1, 512, 14, 14])


title: PyTorch - How to get the gradient of the first layer relative to the input?

body:
With my understanding, by using backward hooks the gradient input at index 0 gives me the gradient relative to the input. This works with all layers, except the first one. In this one, the value at index 0 is None.

Can someone explain how to get the gradient realtive to the input image in PyTorch?

The PyTorch Docs for backward hooks  and an example: https://pytorch.org/docs/stable/nn.html#torch.nn.Module.register_backward_hook
https://pytorch.org/tutorials/beginner/former_torchies/nnft_tutorial.html#forward-and-backward-function-hooks


title: How to read images from pandas column in a pytorch class

body:
I need to define a pytorch dataset class that reads the absolute path from a column to pull the image. When I try this with multiple packages, I get errors.  Below are the errors for each package I'm aware of:

Pathlib.Path: TypeError: expected str, bytes or os.PathLike object, not method

glob.glob: TypeError: expected str, bytes or os.PathLike object, not method

os.path: TypeError: 'module' object is not callable

Below is my pytorch custom dataset.

class image_Dataset(Dataset):
'''
image data class

'''
def __init__(self, data, transform = None):
    '''
    Args:
    ------------------------------------------------------------
        data = dataframe
        image = column in dataframe with absolute path to the image
        label = column in dataframe that is the target classification variable
        policy = ID variable

    '''
    self.image_frame = data
    self.transform = transform

def __len__(self):
    return len(self.image_frame)

def __getitem__(self, idx):
    if torch.is_tensor(idx):
        idx = idx.tolist()

    label = self.image_frame.iloc[idx, 16]
    pic = Path(self.image_frame.iloc[idx,19])
    img = Image.open(pic)
    policy = self.image_frame.iloc[idx, 1]
    sample = {'image': img, 'policy': policy, 'label':label}

    if self.transform:
        image = self.transform(image)

    return image, label, policy

path column in my dataset values looks like the following:

D:/Models/Photos/Train/train_img.jpg
Full trace back below:

IndexError                                Traceback (most recent call last)
<ipython-input-261-38143c041795> in <module>
      7
      8 for i in range(len(roof_data_test)):
----> 9     sample = roof_data_test[i]
     10
     11     print(i, sample[0].shape, sample[1], sample[2])

<ipython-input-260-1bd25b938fea> in __getitem__(self, idx)
     25
     26         label = self.image_frame.iloc[idx, 16]
---> 27         pic = Path(self.image_frame.iloc[idx,19])
     28         img = Image.open(pic)
     29         policy = self.image_frame.iloc[idx, 1]

C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\indexing.py in __getitem__(self, key)
   1492             except (KeyError, IndexError, AttributeError):
   1493                 pass
-> 1494             return self._getitem_tuple(key)
   1495         else:
   1496             # we by definition only have the 0th axis

C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\indexing.py in _getitem_tuple(self, tup)
   2141     def _getitem_tuple(self, tup):
   2142
-> 2143         self._has_valid_tuple(tup)
   2144         try:
   2145             return self._getitem_lowerdim(tup)

C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\indexing.py in _has_valid_tuple(self, key)
    221                 raise IndexingError('Too many indexers')
    222             try:
--> 223                 self._validate_key(k, i)
    224             except ValueError:
    225                 raise ValueError("Location based indexing can only have "

C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\indexing.py in _validate_key(self, key, axis)
   2068             return
   2069         elif is_integer(key):
-> 2070             self._validate_integer(key, axis)
   2071         elif isinstance(key, tuple):
   2072             # a tuple should already have been caught by this point

C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\indexing.py in _validate_integer(self, key, axis)
   2137         len_axis = len(self.obj._get_axis(axis))
   2138         if key >= len_axis or key < -len_axis:
-> 2139             raise IndexError("single positional indexer is out-of-bounds")
   2140
   2141     def _getitem_tuple(self, tup):

IndexError: single positional indexer is out-of-bounds


title: Why does 'dimension' mean several different things in the machine-learning world?

body:
I've noticed that AI community refers to various tensors as 512-d, meaning 512 dimensional tensor, where the term 'dimension' seems to mean 512 different float values in the representation for a single datapoint. e.g. in 512-d word-embeddings means 512 length vector of floats used to represent 1 english-word e.g. https://medium.com/@jonathan_hui/nlp-word-embedding-glove-5e7f523999f6

But it isn't 512 different dimensions, it's only 1 dimensional vector? Why is the term dimension used in such a different manner than usual?

When we use the term conv1d or conv2d which are convolutions over 1-dimension and 2-dimensions, a dimension is used in the typical way it's used in math/sciences but in the word-embedding context, a 1-d vector is said to be a 512-d vector, or am I missing something?

Why is this overloaded use of the term dimension? What context determines what dimension means in machine-learning as the term seems overloaded?


title: How to right shift each row of a matrix?

body:
I have a matrix whose shape is (TxK, and K << T). I want to extend it into shape TxT, and right shift the i-th row with i steps.
For an example:
inputs: T= 5, and K = 3
1 2 3
1 2 3
1 2 3
1 2 3
1 2 3

expected outputs:
1 2 3 0 0
0 1 2 3 0
0 0 1 2 3
0 0 0 1 2
0 0 0 0 1

My solutions:
right_pad = T - K + 1
output = F.pad(input, (0, right_pad), 'constant', value=0)
output = output.view(-1)[:-T].view(T, T)

My solution will cause the error -- gradient computation has been modified by an in-place operation. Is there an efficient and feasible way to achieve my purpose?


title: Pytorch: use pretrained vectors to initialize nn.Embedding, but this embedding layer is not updated during the training

body:
I initialized nn.Embedding with some pretrain parameters (they are 128 dim vectors), the following code demonstrates how I do this:
self.myvectors = gensim.models.KeyedVectors.load_word2vec_format(cfg.vec_dir)
self.vec_weights = torch.FloatTensor(self.myvectors.vectors)
self.embeds = torch.nn.Embedding.from_pretrained(self.vec_weights)

cfg.vec_dir is a json file where vec_dir indicates the path of the pretrained 128 dim vectors I used to initialize this layer.
After the model is trained, I print out this embedding layer, and I found that the parameters are exactly the same as I initialized them, so clearly the parameters are not updated during the training. Why is this happening? What should I do in order to update these vectors?


title: Scatternd not supported

body:
I have a tensor with shape [1,100,34] which needs to be added with 2 tensors [1,100,17] shape at alternative indexes. Current Implementation is like this in pytorch.
kps[..., ::2] += xs.view(batch, K, 1).expand(batch, K, num_joints)
kps[..., 1::2] += ys.view(batch, K, 1).expand(batch, K, num_joints)

This gets successfully converted to onnx model using opset>=11. This is achieved using the scatternd operation. However, this operation is not yet supported in tensorrt. It fails with error
ERROR: INVALID_ARGUMENT: getPluginCreator could not find plugin ScatterND version 1
Parse Failed, Layers =  4447
In node -1 (importFallbackPluginImporter): UNSUPPORTED_NODE: Assertion failed: creator && "Plugin not found, are the plugin name, version, and namespace correct?"

Is there anyway to avoid converting this code to use scatternd operation in onnx. Or any other alternative workaround?


title: torchtext data build_vocab / data_field

body:
I want to ask you some about torchtext.

I have a task about abstractive text summarization, and I build a seq2seq model with pytorch.

I just wonder about data_field constructed by build_vocab function in torchtext.

In machine translation, i accept that two data_fields(input, output) are needed.

But, in summarization, input data and output data are same language.

Here, should I make two data_field(full_sentence, abstract_sentence) in here?

Or is it okay to use only one data_field?

I'm afraid that my wrong choice make model's performance down.

Please, give me a hint.


title: PyTorch: Dynamically freeze layers in my network?

body:
I have a network and I want to only update the first few layers' weights. The thing is, I want to change the amount of layers I update dynamically through the training process.

Is there a way to do this without having to change each weight to trainable = False? (or requires_grad = False)


title: Implementing LeNet in Pytorch

body:
Sorry if this question is incredibly basic. I feel like there is a wealth of resources online, but most of them are half-complete or skip over the details that I want to know.

I am trying to implement LeNet with Pytorch for practice.

https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html

How come in this examples and many examples online, they define the convolutional layers and the fc layers in init, but the subsampling and activation functions in forward?
What is the purpose of using torch.nn.functional for some functions, and torch.nn for others? For example, you have convolution with torch.nn (https://pytorch.org/docs/stable/nn.html#conv1d) and convolution with torch.nn.functional (https://pytorch.org/docs/stable/nn.functional.html#conv1d). Why choose one or the other?
Let's say I want to try different image sizes, like 28x28 (MNIST). The tutorial recommends I resize MNIST. Is there a way to instead change the values of LeNet? What happens if I don't change them?
What is the purpose of num_flat_features? If you wanted to flatten the features, couldn't you just do x = x.view(-1, 16*5*5)?


title: Use of scheduler with self-ajusting optimizers in PyTorch

body: In PyTorch, the weight adjustment policy is determined by the optimizer, and the learning rate is adjusted with a scheduler. When the optimizer is SGD, there is only one learning rate and this is straightforward. When using Adagrad, Adam, or any similar optimizer which inherently adjusts the learning rate on a per-parameter basis, is there something in particular to look out for? Can I ignore the scheduler completely since the algorithm adjusts its own learning rates? Should I parameterize it very differently than if I'm using SGD?


title: Slurm sbatch for a PyTorch script draining node; gres/gpu: count changed for node node002 from 0 to 1

body:
We have a user whose script always drains a node.

Note this error: "gres/gpu: count changed for node node002 from 0 to 1" Could it be misleading? What could cause the node to drain? Here are the contents of the user's SBATCH file. Could the piping having an effect here? Another thing that I just noticed while typing this up is perhaps an attempt to use a mix of library versions. So he has a module load cuda10.0 but then a module load pytorch-py36-cuda10.1-gcc/1.3.1 module load ml-pythondeps-py36-cuda10.1-gcc/3.0.0 python3.6

#!/bin/sh
#SBATCH -N 1
#SBATCH -n 1
#SBATCH --mail-type=ALL
#SBATCH --gres=gpu:1
#SBATCH --job-name=$1sequentialBlur_squeezenet_training_imagewoof_crossval
module purge
module load gcc5 cuda10.0
module load openmpi/cuda/64
module load pytorch-py36-cuda10.1-gcc/1.3.1
module load ml-pythondeps-py36-cuda10.1-gcc/3.0.0
python3.6 SequentialBlur_untrained.py squeezenet 100 imagewoof $1 | tee squeeze_100_imwoof_seq_longtrain_cv_$1.txt
/u/run_seq_blur2.py

Here are the script contents:

# Banks 1978 paper:
# 1 month:  2.4 cyc/deg
# 2 month:  2.8 cyc/deg
# 3 month:  4 cyc/deg
# 224 pixels:
# 20 deg -> 11 pix in deg;  4.6 pix blur;  4 pix blur;  2.8 pix blur
# 4 deg -> 56 pix in deg; 23 pix blur (1 mo); 20 pix blur (2 mo); 14 pix blur (3 mo)

import torch
import torchvision
import torchvision.transforms as transforms
from torchvision import models
import torchvision.datasets
import matplotlib.pyplot as plt
import numpy as np
import matplotlib.pyplot as plt
import numpy as np
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import os
import sys
import scipy
from torch.utils.data.sampler import SubsetRandomSampler
import h5py

args = sys.argv
modelType = args[1] # 'alexnet', 'squeezenet', 'vgg16'
numEpochs = args[2] # int
image_set = str(args[3]) # 'imagewoof', 'imagenette'
block_call = args[4] # int {0:4}

# Example call:
# python3 alexnet 100 imagenette 1

def get_train_valid_loader(data_dir,block,augment=0,random_seed=69420,valid_size=0.2,shuffle=False,
                                                show_sample=False,num_workers=4, pin_memory=False, batch_size=128):
        # valid_size gotta be in [0,1]
        # block must be an int between 0:(1/valid_size) (0:4 for valid_size==0.2)
        transform = transforms.Compose([
                transforms.Resize(256),
                transforms.CenterCrop(224),
                transforms.ToTensor(),
                transforms.Normalize(
                mean=[0.485, 0.456, 0.406],
                std=[0.229, 0.224, 0.225]
        )])
        train_dataset = torchvision.datasets.ImageFolder(root=data_dir,transform=transform)
        valid_dataset = torchvision.datasets.ImageFolder(root=data_dir,transform=transform)
        num_train = len(train_dataset)
        indices = list(range(num_train))
        split = int(np.floor(valid_size * num_train))
        split1 = int(np.floor(block*split))
        split2 = int(np.floor((block+1)*split))
        # if shuffle:
        np.random.seed(100)
        np.random.shuffle(indices)
        valid_idx = indices[split1:split2]
        train_idx = np.append(indices[:split1],indices[split2:])
        train_idx = train_idx.astype('int32')
        if block != 0:
                for b in range(block):
                        indices = [indices[(i + split) % len(indices)] for i, x in enumerate(indices)]
        # train_idx, valid_idx = indices[split:], indices[:split]
        train_sampler = SubsetRandomSampler(train_idx)
        # train_sampler = torch.utils.data.Subset(dataset, indices)
        valid_sampler = SubsetRandomSampler(valid_idx)
        train_loader = torch.utils.data.DataLoader(
                train_dataset, sampler=train_sampler, batch_size=batch_size,
                num_workers=num_workers, pin_memory=pin_memory,
        )
        valid_loader = torch.utils.data.DataLoader(
                valid_dataset, sampler=valid_sampler, batch_size=batch_size,
                num_workers=num_workers, pin_memory=pin_memory,
        )
        return (train_loader, valid_loader)

transform = transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize(
        mean=[0.485, 0.456, 0.406],
        std=[0.229, 0.224, 0.225]
 )])

blurTypes = ['gaussian']

data_dir = "/path/to/dir/" + image_set + "-320_blur/"

classes = []
for directory, subdirectories, files in os.walk(data_dir):
        for file in files:
                if directory.split("\\")[-1] not in classes:
                        classes.append(directory.split("\\")[-1])

criterion = nn.CrossEntropyLoss()
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
def train():

        for epoch in range(int(numEpochs)):
                prev_loss = 100000.0
                running_loss = 0.0
                for i, data in enumerate(trainloader, 0):
                        # get the inputs; data is a list of [inputs, labels]
                        inputs, labels = data
                        inputs = inputs.to(device)
                        labels = labels.to(device)

                        # zero the parameter gradients
                        optimizer.zero_grad()

                        # forward + backward + optimize
                        outputs = net(inputs)
                        loss = criterion(outputs, labels)
                        loss.backward()
                        optimizer.step()

                        running_loss += loss.item()

                if epoch % 10 == 9:
                        print('[%d, %5d] loss: %.3f' %
                                (epoch + 1, i + 1, running_loss / 100))

allAccs = []
for blurType in blurTypes: # multiple types of blur
        print(blurType)
        print('-' * 10)
        # for block in range(5):
        block = int(block_call)
        print("\nFOLD " + str(block+1) + ":")
        for i in range(5):
                if i == 0:
                        blurLevels = [23, 11, 5, 3, 1]
                elif i == 1:
                        blurLevels = [11, 5, 3, 1]
                elif i == 2:
                        blurLevels = [5, 3, 1]
                elif i == 3:
                        blurLevels = [3, 1]
                elif i == 4:
                        blurLevels = [1]

                if modelType == 'vgg16':
                        net = torchvision.models.vgg16(pretrained=False)
                        num_ftrs = net.classifier[6].in_features
                        net.classifier[6] = nn.Linear(num_ftrs, len(classes))
                elif modelType == 'alexnet':
                        net = torchvision.models.alexnet(pretrained=False)
                        num_ftrs = net.classifier[6].in_features
                        net.classifier[6] = nn.Linear(num_ftrs, len(classes))
                else:
                        net = torchvision.models.squeezenet1_1(pretrained=False)
                        net.classifier[1] = nn.Conv2d(512, len(classes), kernel_size=(1, 1), stride=(1, 1))
                        net.num_classes = len(classes)
                optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)
                net = net.to(device)
                for i in range(len(blurLevels)): #5 levels of blur: 1, 3, 5, 11, 23
                        mult = blurLevels[i]

                        trainloader, validloader = get_train_valid_loader(data_dir=data_dir + blurType + '/' + image_set +
                                '-320_' + str(mult) + '/train',
                                block=block,shuffle=False,num_workers=0,batch_size=128)
                        print('Start training on blur window of ' + str(mult))
                        train()
                        print('Finished Training on ' + blurType + ' with blur window of ' + str(mult))

                accs = []
                permBlurLevels = [23, 11, 5, 3, 1]
                for j in range(len(permBlurLevels)):
                        tempMult = permBlurLevels[j]
                        correct = 0
                        total = 0
                        # newTestSet = torchvision.datasets.ImageFolder(root=data_dir + blurType + '/' + image_set + '-320_' +
                        #       str(tempMult) + '/val',
                        #       transform=transform)
                        # newTestLoader = torch.utils.data.DataLoader(newTestSet, batch_size=128,
                        #       shuffle=True, num_workers=0)
                        t2, validloader2 = get_train_valid_loader(data_dir=data_dir + blurType + '/' + image_set +
                                '-320_' + str(mult) + '/train',
                                block=block,shuffle=False,num_workers=0,batch_size=128)

                        with torch.no_grad():
                                for data in validloader2:
                                        images, labels = data
                                        images = images.to(device)
                                        labels = labels.to(device)
                                        outputs = net(images)
                                        _, predicted = torch.max(outputs.data, 1)
                                        total += labels.size(0)
                                        correct += (predicted == labels).sum().item()
                                        acc = 100 * correct / total
                        print('Accuracy: %f %%' % (acc))
                        accs.append(acc)
                allAccs.append(accs)

And here are the errors we see each time he runs this:

[2020-03-13T08:54:02.269] gres/gpu: count changed for node node002 from 0
to 1
[2020-03-13T08:54:02.269] error: Setting node node002 state to DRAIN
[2020-03-13T08:54:02.269] drain_nodes: node node002 state set to DRAIN
[2020-03-13T08:54:02.269] error: _slurm_rpc_node_registration node=node002:
Invalid argument

I could only find one reference to a 2015 SlurMD bug that wasn't a bug that even mentions this gres/gpu: count changed for node...


title: Custom Layer with pytorch autograd makes gradient error when training

body:
I'm a beginner in Pytorch and want to make a custom layer, PACT Function, "https://arxiv.org/pdf/1805.06085.pdf". I implemented this layer and replaced all ReLU functions in ResNet. But When I started to train my model, this error occurred.

RuntimeError: Function _pact_functionBackward returned an invalid gradient at index 1 - got [] but expected shape compatible with [1]

PACT Class:

class PACT(nn.Module):
  def __init__(self, num_bits=4):
    super(PACT, self).__init__()

    self.num_bits = num_bits
    self.alpha = nn.Parameter(torch.Tensor([5.]), requires_grad=True)

    self.pact_function = PACT_function()
    self.pact_quantize = PACT_quantize(self.alpha, self.num_bits)

  def forward(self, input):
    y = self.pact_function(input, self.alpha)
    yq = self.pact_quantize(y)

  return yq

And how to forward and backward the activation with "pact_function":

def PACT_function():
  class _pact_function(torch.autograd.function):

    # See https://github.com/obilaniu/GradOverride/blob/master/functional.py
    # See https://arxiv.org/pdf/1805.06085.pdf

    @staticmethod
    def forward(ctx, x, alpha):
      ctx.save_for_backward(x, alpha)
      return x.clamp(min=0.).min(alpha)

    @staticmethod
    def backward(ctx, grad_output):
      x, alpha = ctx.saved_variables

      lt0      = x < 0
      gta      = x > alpha
      gi       = 1.0-lt0.float()-gta.float()

      grad_x    = grad_output*gi
      grad_alpha = torch.sum(grad_output*x.ge(alpha).float())

      return grad_x, grad_alpha   # why grad_x, None acts?? but this makes the values of alpha same in all layers!

  return _pact_function.apply

This is very odd because if I remove grad_alpha, returns like grad_x, None, then the code acts. However, when I print the values of alpha in my model (ResNet), the values are all same even if those PACT layers are different layers.

Epoch: 10 | LR: 0.01000  [============================ 196/196
===========================>]  Step: 61ms | Tot: 22s604ms | Loss: 0.555 | Acc: 80.896% (40448/50000)
PACT() Parameter containing: tensor([4.5349], device='cuda:0',   requires_grad=True)
PACT() Parameter containing: tensor([4.5349],       device='cuda:0',
requires_grad=True)
PACT() Parameter containing:          tensor([4.5349],
device='cuda:0', requires_grad=True)
PACT()          Parameter    containing: tensor([4.5349], device='cuda:0',             requires_grad=True)
PACT() Parameter containing: tensor([4.5349], device='cuda:0',    requires_grad=True)

So, I'm wondered why returning (grad_x, None) acts, not (grad_x, grad_alpha), and why alpha parameters are same in all layers when using (grad_x, None) with nn.Parameter.
It's really grateful if you have some answers.


title: Weights in BCEWithLogitsLoss

body: I am facing an issue in a multi label, multi class classification task. I have a dataset of size 33000, each samples containing 104 classes. I split the dataset in 16500 samples with labels such as [1, 0, 1, 0, 0, …], [0, 1, 1, 0, 1, …], [1, 0, 0, 0] (each label has at least one element 1 in it) and 16500 labels such as [0, 0, 0, …], [0, 0, 0, …] (all elements in all labels are 0). When calculating the pos_count for each class, the number pos_count_0 for class 0 is how many of 1 appear in the first position of each label in my dataset. For class 1, pos_count_1 the number of 1 in the second position and so on. And after that, the pos_weight of class 0 is (33000-pos_count_0)/pos_count_0, pos_weight of class 1 is (33000-pos_count_1)/pos_count_1 ? I am a little bit confused how neg_count and pos_count for a class are calculated.


title: Compute matrix multiplication only for given coordinates

body:
In PyTorch, I would like to compute

E * A.mm(B)

where E can be a very sparse matrix consisting of 0's and 1's. In other words, I want to compute A.mm(B) and then leave only the certain coordinates. Is there a way to compute such a sparse matrix efficiently? I have full control over matrix representations.

Also, in most cases, E consists only of 1's, so I would like this case also to be handled efficiently.


title: When training a small self attention model, raise a GPU Out of Memory Error. I don't know why

body:
When I train a small model, whose target is to do self-attention on several inputs, it raises a GPU Out of Memory Error, as follows.

RuntimeError: CUDA out of memory. Tried to allocate 2.00 MiB (GPU 0; 11.93 GiB total capacity; 11.45 GiB already allocated; 1.06 MiB free; 15.75 MiB cached)

Here is my code:
import torch

class SelfAttention():
    ## this same as self attention code in Transformer

def self_attention_model(input):
    self_attention = SelfAttention()
    distance = nn.MSELoss()
    optimizer = torch.optim.Adam(instances_slf_att.parameters(), lr=0.01, weight_decay=0.00005)
    num_epochs_slf_att = 10
    for epoch in num_epochs_slf_att:
        output = self_attention(input)
        loss = distance_slf_att(output, input)
        optimizer.zero_grad()
        loss.backward()
        optimizer_slf_att.step()
    torch.cuda.empty_cache()
    return output

# It is a dictionary containing users and related items.
# Items are embeddings and the shape is dynamic.
# Different users have different items. The numbers of items of different users are different.
user_items = {}

user_output = [] # a list to put outputs
for user in users:
    this_user_items = user_items[user] # this_user_items.shape = torch.Size([1,n,100]). Different user has different n. The range of n is (1,10).
    output = self_attention_model(this_user_items) # self attention code is the same as code in Tranformer.
    user_output[user] = output

Here are 1450 users. Each time I train this model after user 900 or 700, it raises this GPU Out of Memory Error. I don't know why. For each user, I have empty the cuda cache.


title: Convert from Keras to Pytorch - conv2d

body:
I am trying to convert the following Keras code into PyTorch.
    tf.keras.Sequential([
          Conv2D(128, 1, activation=tf.nn.relu),
          Conv2D(self.channel_n, 1, activation=None),
    ])

When creating the model summary with self.channels=16 i get the following summary.
Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
conv2d (Conv2D)              (1, 3, 3, 128)            6272
_________________________________________________________________
conv2d_1 (Conv2D)            (1, 3, 3, 16)             2064
=================================================================
Total params: 8,336
Trainable params: 8,336
Non-trainable params: 0

How would one convert?
I have attempted it as such:
import torch
from torch import nn

class CellCA(nn.Module):
    def __init__(self, channels, dim=128):
        super().__init__()
        self.net = nn.Sequential(
            nn.Conv2d(in_channels=channels,out_channels=dim, kernel_size=1),
            nn.ReLU(),
            nn.Conv2d(in_channels=dim, out_channels=channels, kernel_size=1),
        )
    def forward(self, x):
        return self.net(x)

However, I get 4240 params


title: What is the correct way to use a PyTorch Module inside a PyTorch Function?

body:
We have a custom torch.autograd.Function z(x, t) which computes an output y in a way not amenable to direct automatic differentiation, and have computed the Jacobian of the operation with respect to its inputs x and t, so we can implement the backward method.
However, the operation involves making several internal calls to a neural network, which we have implemented for now as a stack of torch.nn.Linear objects, wrapped in net, a torch.nn.Module. Mathematically, these are parameterized by t.
Is there any way that we can have net itself be an input to the forward method of z? Then, we would return from our backward the list of products of the upstream gradient Dy and parameter Jacobia dydt_i, one for each of the parameters ti that are children of net (in addition to Dy*dydx, although x is data and does not need gradient accumulation).
Or do we really instead need to take t (actually a list of individual t_i), and reconstruct internally in z.forward the actions of all the Linear layers in net?


title: Pytorch : Pass from grid to image coordinate convention for use of grid_sample

body:
I need to interpolate some deformation grid in PyTorch and decided to use the function grid_sample (see doc). I need to reshape grids as images back and forth with theses conventions :

N Batch size
D grid depth (for 3D images)
H grid height
W grid width
d grid dimension (=2 or 3)

images format is (N,2,H,W) in 2D (resp. (N,3,D,H,W) in 3D)
when grid format is (N,H,W,2) (resp. (N,D,H,W,3)in 3D)
I can't use reshape or view because they do not arrange the data as I wish. I need to have (for example)
grid_in_grid_convention[0,:,:,0] == grid_in_image_convention[0,0,:,:]

I came up with theses functions to make the reshaping which works fine but I am sure that there is a more compact/fast way to do so. What are your taught ?
def grid2im(grid):
    """Reshape a grid tensor into an image tensor
        2D  [T,H,W,2] -> [T,2,H,W]
        3D  [T,D,H,W,2] -> [T,D,H,W,3]
    """
    if grid.shape[0] == 1 and grid.shape[-1] == 2: # 2D case, batch =1
        return torch.stack((grid[0,:,:,0],grid[0,:,:,1]),dim = 0).unsqueeze(0)

    elif grid.shape[0] == 1 and grid.shape[-1] == 3: # 3D case, batch =1
        return torch.stack((grid[0,:,:,:,0],grid[0,:,:,:,1],grid[0,:,:,:,2]),
                            dim = 0).unsqueeze(0)

    elif grid.shape[-1] == 2:
        N,H,W,d = grid.shape
        temp = torch.zeros((N,H,W,d))
        for n in range(N):
            temp[n,:,:,:] = torch.stack((grid[n,:,:,0],grid[n,:,:,1]),dim = 0).unsqueeze(0)
        return temp

    elif grid.shape[-1] == 3:
        N,D,H,W,d =grid.shape
        temp = torch.zeros((N,D,H,W,d))
        for n in range(N):
            temp[n,:,:,:,:] = torch.stack((grid[n,:,:,:,0],
                                           grid[n,:,:,:,1],
                                           grid[n,:,:,:,2]),
                            dim = 0).unsqueeze(0)
    else:
        raise ValueError("input argument expected is [N,H,W,2] or [N,D,H,W,3]",
                         "got "+str(grid.shape)+" instead.")

def im2grid(image):
    """Reshape an image tensor into a grid tensor
        2D case [T,2,H,W]   ->  [T,H,W,2]
        3D case [T,3,D,H,W] ->  [T,D,H,W,3]
    """
    # No batch
    if image.shape[0:2] == (1,2):
        return torch.stack((image[0,0,:,:],image[0,1,:,:]),dim= 2).unsqueeze(0)
    elif image.shape[0:2] == (1,3):
        return torch.stack((image[0,0,:,:],image[0,1,:,:],image[0,2,:,:]),
                           dim = 2).unsqueeze(0)
    # Batch size > 1
    elif image.shape[0] > 0 and image.shape[1] == 2 :
        N,d,H,W = image.shape
        temp = torch.zeros((N,H,W,d))
        for n in range(N):
            temp[n,:,:,:] = torch.stack((image[n,0,:,:],image[n,1,:,:]),dim= 2).unsqueeze(0)
        return temp
    elif image.shape[0] > 0 and image.shape[1] == 3 :
        N,d,D,H,W = image.shape
        temp = torch.zeros((N,D,H,W,d))
        for n in range(N):
            temp[n,:,:,:] = torch.stack((image[n,0,:,:],
                                         image[n,1,:,:],
                                         image[n,2,:,:]),
                           dim = 2).unsqueeze(0)
        return temp
    else:
        raise ValueError("input argument expected is [1,2,H,W] or [1,3,D,H,W]",
                         "got "+str(image.shape)+" instead.")


title: PyTorch's grid_sample conversion to CoreML (via coremltools)

body:
torch.nn.functional.grid_sample (source here, click on docs for documentation) is currently unsupported operation by CoreML (and their conversion utilities library: coremltools).
What I'm looking for is a way to export layer shown below from PyTorch's torchscript (docs here) to CoreML (either using custom op created via Swift or via efficient PyTorch rewrite of grid_sample).
For details and tips to get you started see Tips section
Minimal verifiable example
import coremltools as ct
import torch

class GridSample(torch.nn.Module):
    def forward(self, inputs, grid):
        # Rest could be the default behaviour, e.g. bilinear
        return torch.nn.functional.grid_sample(inputs, grid, align_corners=True)

# Image could also have more in_channels, different dimension etc.,
# for example (2, 32, 64, 64)
image = torch.randn(2, 3, 32, 32)  # (batch, in_channels, width, height)
grid = torch.randint(low=-1, high=2, size=(2, 64, 64, 2)).float()

layer = GridSample()
# You could use `torch.jit.script` if preferable
scripted = torch.jit.trace(layer, (image, grid))

# Sanity check
print(scripted(image, grid).shape)

# Error during conversion
coreml_layer = ct.converters.convert(
    scripted,
    source="pytorch",
    inputs=[
        ct.TensorType(name="image", shape=image.shape),
        ct.TensorType(name="grid", shape=grid.shape),
    ],
)

which raises the following error:
Traceback (most recent call last):
  File "/home/REDACTED/Downloads/sample.py", line 23, in <module>
    coreml_layer = ct.converters.convert(
  File "/home/REDACTED/.conda/envs/REDACTED/lib/python3.9/site-packages/coremltools/converters/_converters_entry.py", line 175, in convert
    mlmodel = mil_convert(
  File "/home/REDACTED/.conda/envs/REDACTED/lib/python3.9/site-packages/coremltools/converters/mil/converter.py", line 128, in mil_convert
    proto = mil_convert_to_proto(, convert_from, convert_to,
  File "/home/REDACTED/.conda/envs/REDACTED/lib/python3.9/site-packages/coremltools/converters/mil/converter.py", line 171, in mil_convert_to_proto
    prog = frontend_converter(, **kwargs)
  File "/home/REDACTED/.conda/envs/REDACTED/lib/python3.9/site-packages/coremltools/converters/mil/converter.py", line 85, in __call__
    return load(*args, **kwargs)
  File "/home/REDACTED/.conda/envs/REDACTED/lib/python3.9/site-packages/coremltools/converters/mil/frontend/torch/load.py", line 81, in load
    raise e
  File "/home/REDACTED/.conda/envs/REDACTED/lib/python3.9/site-packages/coremltools/converters/mil/frontend/torch/load.py", line 73, in load
    prog = converter.convert()
  File "/home/REDACTED/.conda/envs/REDACTED/lib/python3.9/site-packages/coremltools/converters/mil/frontend/torch/converter.py", line 227, in convert
    convert_nodes(self.context, self.graph)
  File "/home/REDACTED/.conda/envs/REDACTED/lib/python3.9/site-packages/coremltools/converters/mil/frontend/torch/ops.py", line 54, in convert_nodes
    raise RuntimeError(
RuntimeError: PyTorch convert function for op 'grid_sampler' not implemented.

Dependencies
Python (conda):

coremltools==4.1
torch==1.8.0

You could also use nightly/master builds (at least for the day of writing: 2021-03-20)
Tips
Those were split into two possible solutions I currently see:
PyTorch only
Rewrite torch.nn.functional.grid_sample from scratch.

This would require sticking only to PyTorch operations on tensors as loops (e.g. triple nested) would hang the converter and be too inefficient
You cannot use __getitem__ on list or related types - seems to work with torch.Tensor but had problems with that so you should have it in mind if you get RuntimeError: PyTorch convert function for op '__getitem__' not implemented

Pros:

No need for two languages & sticking to single technology

Cons:

Limited with loops and would require sticking to vectorized operations (most/all of the time)

Swift & CoreML
Register custom layer which is responsible for running grid_sample. CPU only implementation would be fine (although using Apple's Metal for GPU speedups would be great).
As I'm not into Swift, I've gathered a few resources which might help you:

https://coremltools.readme.io/docs/custom-operators - starting point, Python only, quite easy, only registering layer for conversion
https://developer.apple.com/documentation/coreml/mlcustomlayer - API of the layer which one would have to code in Swift
https://developer.apple.com/documentation/coreml/core_ml_api/creating_a_custom_layer - more about aforementioned (but not much)
https://machinethink.net/blog/coreml-custom-layers/ - blog post with example and dispatching layer to devices (GPU, CPU). Needs Swift (CPU version), Metal (GPU implementation). Eventual Metal implementation might be based off PyTorch's CUDA impl, CPU and Swift might also be related.
3 years old, so be aware of that, swish activation layer, seems to be a good starting point (and other posts from the same author cast some light on the CoreML itself).
https://github.com/hollance/CoreML-Custom-Layers - repo for the above

Pros:

Possibility to use loops and finer control over the algorithm
Might be easier as we're not limited to operations which CoreML can currently read

Cons:

Two languages
Sparse documentation


title: CUDA out of memory with matrix multiply

body:
I'm trying to multiply 3 matrices, but am running out of CUDA memory.

# A: 3000 x 100 (~2MB)
# B: 100 x 100  (~0.05MB)
# C: 100 x 3MM  (~2GB)

A = np.random.randn(3000, 100)
B = np.random.randn(100, 100)
C = np.random.randn(100, 3e6)

A_gpu = torch.from_numpy(A).cuda()
B_gpu = torch.from_numpy(B).cuda()
C_gpu = torch.from_numpy(C).cuda()

R_gpu = (A_gpu @ B_gpu @ C_gpu)

Cuda is requesting about 90GB of memory for this operation.  I don't understand why.


title: Pytorch - Custom DataLoader runs forever

body:
class TripletImageLoader(torch.utils.data.Dataset):
    def __init__(self):
        self.data = [0]*10000000

    def __getitem__(self, index):
        pid = os.getpid() % WORKER_SIZE
        # My code here only uses pid, doesnt use index

        return torch.tensor(batch.data), torch.tensor(batch.label)

    def __len__(self):
        return len(self.data)

I need my data loader to run forever. Right now it always terminates after hitting 10000000 or whatever the max integer size is. How do i make this run forever, I dont care about 'index' I am not using it. I am simply using the worker capabilities of this class


title: Finetune PyTorch model after training fc layers

body:
I am trying to do Transfer Learning using PyTorch. I wan to train fc layers first and then to finetune the whole network. Unfortunately after training fc layers and then passing my network to finetune, I am losing the accuracy that was acquired in the first training. Is this an expected behaviour or am I doing something wrong here?

Here is the code:

model = torchvision.models.resnet50(pretrained=True)
for param in model.parameters():
    param.requires_grad = False

num_ftrs = model.fc.in_features
model.fc = nn.Linear(num_ftrs, 4)
model = model.to(device)

criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)

model = trainer.fit_model(dataloader, model, criterion, optimizer, num_epochs=10)
# fit model is basic PyTorch training function found here: https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html#convnet-as-fixed-feature-extractor The only difference is that scheduler is an optional param.

for param in model.parameters():
    param.requires_grad = True

torch.cuda.empty_cache()
exp_lr_scheduler = lr_scheduler.StepLR(optimizer, step_size=7, gamma=0.1)

# Here I am finetuning the model
model_ft = trainer.fit_model(
    dataloader, model, criterion, optimizer, scheduler=exp_lr_scheduler, num_epochs=10
)

Am I missing something here or should I just train the model once?


title: How to build Pytorch Mobile sample HelloWorld app?

body:
https://pytorch.org/mobile/android/

I am trying to build a Pytorch demo HelloWorld app for Android. My machine is MacOS Mojave where I installed Python3 and Torchvision via conda. I am new to Pytorch, Pytorch mobile and gradlew etc.. In the past I used CMake and Make for C/C++ builds. I tried the following steps from the pytorch website but installDebug is not found. gradlew tasks definitely doesn't have any installDebug tasks.

Is the documentation old or am I missing a step or two below? I do have Android SDK and NDK installed as far as I can tell.

$ git clone https://github.com/pytorch/android-demo-app.git
$ cd android-demo-app
$ cd HelloWorldApp
$ python trace_model.py
$ ./gradlew installDebug
FAILURE: Build failed with an exception.

* What went wrong:
Task 'installDebug' not found in root project 'HelloWorldApp'.

* Try:
Run gradlew tasks to get a list of available tasks. Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 0s
$ ./gradlew tasks


title: Does PyTorch loss() and backpropagation understand lambda layers?

body:
I've been working with a resnet56 model from the code provided here: https://github.com/akamaster/pytorch_resnet_cifar10/blob/master/resnet.py.

I noticed that the implementation is different from many of the other available ResNet examples online, and I was wondering if PyTorch's backpropagation algorithm using loss() can account for the lambda layer and shortcut in the code provided.

If that is the case, can anyone provide insight into how PyTorch is able to interpret the lambda layer for backpropagation (i.e. how does PyTorch know how to differentiate with respect to the layer's operations)?

P.S. I also had to modify the code to fit my own use-case, and it seems like my own implementation with option == 'A' does not produce great results. This may simply be because option == 'B,' which uses convolutional layers instead of padding, is better for my data.

        self.shortcut = nn.Sequential()
        if stride != 1 or in_planes != planes:
            if option == 'A':
                top = (int) ((self.expansion*planes - in_planes) / 2)
                bot = (self.expansion*planes - in_planes) - top
                self.shortcut = LambdaLayer(lambda x:
                                            F.pad(x[:, :, ::stride, ::stride], (0, 0, 0, 0, top, bot), "constant", 0))


title: Training loss increase after loading model, optimizer and scheduler using PyTorch

body:
I'm working in a research computer vision project and I have a particular problem that doesn't allow me to resume training after a crash or interrupt, this is my code to load the checkpoint:

from torch.optim import lr_scheduler
N_EPOCHS = 120
if load_weights:
    optimizer = torch.optim.Adam(model.parameters(),lr=checkpoint['last_lr'], weight_decay=weight_decay)
    optimizer.load_state_dict(checkpoint['optimizer'])
    scheduler = lr_scheduler.StepLR(optimizer, step_size=step_of_scheduler, gamma=0.9, last_epoch=loaded_epochs)
    scheduler.load_state_dict(checkpoint['scheduler'])
else:
    optimizer = torch.optim.Adam(model.parameters(),lr=initial_lr, weight_decay=weight_decay)
    scheduler = lr_scheduler.StepLR(optimizer, step_size=step_of_scheduler, gamma=0.9)

I even tried adding optimizer.state_dict()['param_groups'][0]['params'] = checkpoint['optimizer']['param_groups'][0]['params'] but the result is even worse, my piece of code to save the checkpoint after validation is:

# Checkpoint
       checkpoint = {'model':model.state_dict(),
                     'epoch':loaded_epochs + epoch + 1,
                     'last_validation_acc': val_acc_db_avg[-1],
                     'hyperparameters': hyperparameters,
                     'last_lr': optimizer.param_groups[0]['lr'],
                     'best_general_val': max(val_acc_db_avg),
                     'last_train_loss': train_loss_db[-1],
                     'optimizer': optimizer.state_dict(),
                     'scheduler': scheduler.state_dict(),
                     'img_resize': 512 if lr_scaled else 256,
                     'best_train_acc': max(train_acc_db),
                     'lr_scaled': lr_scaled
                     }

       # Backup in drive
       torch.save(checkpoint, filename_of_checkpoint)

I did make sure to call model.train() before training starts and also I called scheduler.step(), this being said, my increase in training loss is from 0.81 to 0.89 or 0.90 if I manually assign the params, I don't know what might be going wrong here since I also load the model state dict properly.

Edit: this is how I load my model state dict:

if load_weights:
    model.load_state_dict(checkpoint_load['model'])

# Transfering model to GPU if available
model = model.to(DEVICE)


title: conversion issue: pytorch -> ONNX -> TensorFlow -> tflite

body:
I have created a simple neural net (for image classification on CIFAR dataset, given on pytorch official docs 60 mins. blitz section). Now, after doing the following conversion:
pytorch -> ONNX -> TensorFlow -> tflite
I’m using the following code to convert .pb -> .tflite:
converter = tf.compat.v1.lite.TFLiteConverter.from_frozen_graph(
    graph_def_file="../models/my_cifar_net.pb", input_arrays=["my_input"], output_arrays=["my_output"])

converter.optimizations = [tf.lite.Optimize.DEFAULT]
tf_lite_model = converter.convert()

with tf.io.gfile.GFile('../models/my_cifar_net.tflite', 'wb') as f:
    f.write(tf_lite_model)

All other converted models (like, .pth, .onnx, .pb) are giving correct output on an example data except .tflite, as I’m getting the following error:
snapshot of the error
I’m using following versions of diff. libraries:

numpy==1.18.2
pandas==1.0.3
torch==1.5.1
torchvision==0.6.1
onnx==1.7.0
coremltools==4.0b1
onnxmltools==1.7.0
onnx-tf==1.6.0
tensorflow==2.2.0
tensorflow-addons==0.10.0


title: Torch not compiled with CUDA enabled

body:
How I install PyTorch:
conda install pytorch torchvision cudatoolkit=10.1 -c pytorch

How I install Tensor Flow:
pip install tensorflow

CUDA Toolkit is installed and checked with:
nvcc --version

cuDNN 7.6.4 SDK imported to CUDA folder as described, example here:

NVIDIDA Card and Driver Information

=======================================
THE CODE:
In [1]:
from __future__ import print_function
from __future__ import division
import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader, TensorDataset, Dataset
from torch import Tensor
import numpy as np
import pandas
import os
from PIL import Image
import glob
from os import listdir
from os.path import isfile, join
import torchvision
from torchvision import datasets, models, transforms
import matplotlib.pyplot as plt
import time
import math
import copy
import torch.utils.data
import cv2
import tensorflow as tf
print("PyTorch Version: ",torch.__version__)
print("Torchvision Version: ",torchvision.__version__)
print("Tensor Flow Version: ", tf.__version__)
tf.test.gpu_device_name()

Out [1]:
PyTorch Version:  1.6.0
Torchvision Version:  0.7.0
Tensor Flow Version:  2.3.0

'/device:GPU:0'

In [2]:
torch.cuda.current_device()

Out [2]:
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-2-3380d2c12118> in <module>
----> 1 torch.cuda.current_device()

~\Anaconda3\lib\site-packages\torch\cuda\__init__.py in current_device()
382 def current_device() -> int:
383     r"""Returns the index of a currently selected device."""
--> 384     _lazy_init()
385     return torch._C._cuda_getDevice()
386

~\Anaconda3\lib\site-packages\torch\cuda\__init__.py in _lazy_init()
184             raise RuntimeError(
185                 "Cannot re-initialize CUDA in forked subprocess. " + msg)
--> 186         _check_driver()
187         if _cudart is None:
188             raise AssertionError(

~\Anaconda3\lib\site-packages\torch\cuda\__init__.py in _check_driver()
59 def _check_driver():
60     if not hasattr(torch._C, '_cuda_isDriverSufficient'):
---> 61         raise AssertionError("Torch not compiled with CUDA enabled")
62     if not torch._C._cuda_isDriverSufficient():
63         if torch._C._cuda_getDriverVersion() == 0:

AssertionError: Torch not compiled with CUDA enabled

Is anybody able to help me understand what is happening here?
I've uninstalled, and reinstalled PyTorch and Tensor Flow several times without success.


title: Unable to install pillow 6.2.1 in conda

body:
After upgrading pytorch / torch-vision the following error occurs:
python -c "import torch ; import torchvision as tv; print(torch.__version__, tv.__version__)
> "
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/pointr/anaconda3/lib/python3.7/site-packages/torchvision/__init__.py", line 4, in <module>
    from torchvision import datasets
  File "/home/pointr/anaconda3/lib/python3.7/site-packages/torchvision/datasets/__init__.py", line 9, in <module>
    from .fakedata import FakeData
  File "/home/pointr/anaconda3/lib/python3.7/site-packages/torchvision/datasets/fakedata.py", line 3, in <module>
    from .. import transforms
  File "/home/pointr/anaconda3/lib/python3.7/site-packages/torchvision/transforms/__init__.py", line 1, in <module>
    from .transforms import *
  File "/home/pointr/anaconda3/lib/python3.7/site-packages/torchvision/transforms/transforms.py", line 17, in <module>
    from . import functional as F
  File "/home/pointr/anaconda3/lib/python3.7/site-packages/torchvision/transforms/functional.py", line 5, in <module>
    from PIL import Image, ImageOps, ImageEnhance, PILLOW_VERSION
ImportError: cannot import name 'PILLOW_VERSION' from 'PIL' (/home/pointr/anaconda3/lib/python3.7/site-packages/PIL/__init__.py)

This has been noted as due to an incompatibility between conda and pytorch 7.0.0  https://github.com/pytorch/vision/issues/1712 . So  I need to downgrade to pillow 6.2.1:
The command posted to do this is:  conda install pillow=6.2.1 -y
However that is failing:
 (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: |

That is hanging.  So now what ?   The workaround needs itself a workaround ..  Is conda broken for torchvision now?   The primary reason I am using conda in the first place is torch / torchvision ..
Update Conda tried to resolve conflicts. After 20 minutes of this it was 13% done. Ridiculous. This is a 2020 core i7 mini-tower. No sane program takes more than low double digits seconds to resolve dependencies.  I finally killed it.  I am going to try the suggestion to do directly from pip:
pip install Pillow==6.2.1

OK - that is hanging ..  I am going to uninstall pillow and reinstall it with that version.
Another update  @erip has recommended conda install -c conda-forge pillow=6.2.1 - so here we go:
conda install -c conda-forge pillow=6.2.1
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: \

I will let the dust settle on that one but looking quite unlikely.


title: wandb - RuntimeError: CUDA out of memory

body:
I'm trying to run the model given in the hyperparameter optimization example from the simple transformers documentation, but while searching for hyperparameters after a certain number of iterations, a CUDA out of memory error occurs. Also during the search for hyperparameters GPU Memory Allocated is constantly increased.
Here is a graph of memory allocation:

Here's my code. Code I'm launching on Google Collab. How I can fix this error?
import logging

import pandas as pd
import sklearn

import wandb
from simpletransformers.classification import (
    ClassificationArgs,
    ClassificationModel,
)

sweep_config = {
    "method": "bayes",  # grid, random
    "metric": {"name": "train_loss", "goal": "minimize"},
    "parameters": {
        "num_train_epochs": {"values": [2, 3, 5]},
        "learning_rate": {"min": 5e-5, "max": 4e-4},
    },
}

sweep_id = wandb.sweep(sweep_config, project="Simple Sweep")

logging.basicConfig(level=logging.INFO)
transformers_logger = logging.getLogger("transformers")
transformers_logger.setLevel(logging.WARNING)

# Preparing train data
train_data = [
    ["Aragorn was the heir of Isildur", "true"],
    ["Frodo was the heir of Isildur", "false"],
]
train_df = pd.DataFrame(train_data)
train_df.columns = ["text", "labels"]

# Preparing eval data
eval_data = [
    ["Theoden was the king of Rohan", "true"],
    ["Merry was the king of Rohan", "false"],
]
eval_df = pd.DataFrame(eval_data)
eval_df.columns = ["text", "labels"]

model_args = ClassificationArgs()
model_args.reprocess_input_data = True
model_args.overwrite_output_dir = True
model_args.evaluate_during_training = True
model_args.manual_seed = 4
model_args.use_multiprocessing = True
model_args.train_batch_size = 16
model_args.eval_batch_size = 8
model_args.labels_list = ["true", "false"]
model_args.wandb_project = "Simple Sweep"

def train():
    # Initialize a new wandb run
    wandb.init()

    # Create a TransformerModel
    model = ClassificationModel(
        "roberta",
        "roberta-base",
        use_cuda=True,
        args=model_args,
        sweep_config=wandb.config,
    )

    # Train the model
    model.train_model(train_df, eval_df=eval_df)

    # Evaluate the model
    model.eval_model(eval_df)

    # Sync wandb
    wandb.join()

wandb.agent(sweep_id, train)


title: Cannot setup package in conda environment with Pytorch installed

body:
All
After setting up the PyTorch 1.7.1 with CUDA 11.2 on a conda virtual environment, I run python setup.py install it always returns me the following error message.
Traceback (most recent call last):
  File "setup.py", line 2, in <module>
    from torch.utils.cpp_extension import BuildExtension, CUDAExtension
  File "anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/__init__.py", line 189, in <module>
    _load_global_deps()
  File "anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/__init__.py", line 142, in _load_global_deps
    ctypes.CDLL(lib_path, mode=ctypes.RTLD_GLOBAL)
  File "anaconda3/envs/pytorch/lib/python3.7/ctypes/__init__.py", line 364, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/lib/../../../../libcublas.so.11: symbol free_gemm_select version libcublasLt.so.11 not defined in file libcublasLt.so.11 with link time reference

Could anyone can help with this?
Thanks!


I am trying to show results of GAN network on some specified epochs. The function for printing the current result was used previously with TF. I need to change in to pytorch.
def show_result(G_net, z_, num_epoch, show=False, save=False, path='result.png'):

  #test_images = sess.run(G_z, {z: z_, drop_out: 0.0})
  test_images = G_net(z_)

  size_figure_grid = 5
  fig, ax = plt.subplots(size_figure_grid, size_figure_grid, figsize=(5, 5))

  for i, j in itertools.product(range(size_figure_grid), range(size_figure_grid)):
     ax[i, j].get_xaxis().set_visible(False)
     ax[i, j].get_yaxis().set_visible(False)

  for k in range(5*5):
     i = k // 5
     j = k % 5
     ax[i, j].cla()
     ax[i, j].imshow(np.reshape(test_images[k], (28, 28)), cmap='gray')

  label = 'Epoch {0}'.format(num_epoch)
  fig.text(0.5, 0.04, label, ha='center')

  plt.savefig(name)
  file = drive.CreateFile({'title': label, "parents": [{"kind": "https://drive.google.com/drive/u/0/folders/", "id": folder_id}]})
  file.SetContentFile(name)
  file.Upload()

  if num_epoch == 10 or num_epoch == 20 or num_epoch == 50 or num_epoch == 100:
     plt.show()

  plt.close()

The results I need to obtain looks like that:
result img
I am getting this error, however I am not sure what I did incorrectly

Can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first


title: Trying to reproduce results from StackGAN paper on Text-to-Photo synthesis

body:
I am trying to run the StackGAN code on aws ec2 but the demo does not work. I have downloaded the text-embeddings and preprocessed the data.

However I keep getting this error

IOError: [Errno 2] No such file or directory: 'Data/birds/example_captions.t7'

I have tried to manually install pytorch but the build fails on ec2
I have also went into the torch environment pytorch_p27 provided but the same error occurs

I have downloaded some pretrained models in the hopes that those will allow me to perform the StackGAN birds demo included in the repo.

But, I cannot get past this error.


When I'm running a pytorch code,I got this error information below:

class BatchRNN(nn.Module):
def __init__(self, input_size, hidden_size, rnn_type=nn.LSTM,
        bidirectional=False, batch_norm=True, dropout = 0.1):
    super(BatchRNN, self).__init__()
    self.input_size = input_size
    self.hidden_size = hidden_size
    self.bidirectional = bidirectional
    self.batch_norm = SequenceWise(nn.BatchNorm1d(input_size)).cuda() if batch_norm else None
    self.rnn = rnn_type(input_size=input_size, hidden_size=hidden_size,
                        bidirectional=bidirectional, dropout = dropout, bias=False,batch_first=True).cuda()

def forward(self, x):
    if self.batch_norm is not None:
        x = self.batch_norm(x).cuda()
    x, _ = self.rnn(x)
    self.rnn.cuda().flatten_parameters()
    return x

class CTC_RNN(nn.Module):
def __init__(self, rnn_input_size, rnn_hidden_size, rnn_layers=1,
        rnn_type=nn.LSTM, bidirectional=True,
        batch_norm=True, num_class=1232, drop_out = 0.1):
    super(CTC_RNN, self).__init__()
    self.rnn_input_size = rnn_input_size
    self.rnn_hidden_size = rnn_hidden_size
    self.rnn_layers = rnn_layers
    self.rnn_type = rnn_type
    self.num_class = num_class
    self.num_directions = 2 if bidirectional else 1
    #cnn
    self.conv1_cnn=nn.Conv2d(1,256,(2,rnn_input_size)).cuda()
    self.conv2_cnn=nn.Conv2d(1,256,(2,256)).cuda()
    self.fc_cnn=nn.Linear(256,rnn_hidden_size).cuda()
    self.softmax_cnn=torch.nn.Softmax().cuda()
    rnns = []
    rnn = BatchRNN(input_size=rnn_input_size, hidden_size=rnn_hidden_size,
                    rnn_type=rnn_type, bidirectional=bidirectional,
                    batch_norm=False).cuda()
    rnns.append(('0', rnn))
    for i in range(rnn_layers-1):
        rnn = BatchRNN(input_size=self.num_directions*rnn_hidden_size,
                hidden_size=rnn_hidden_size, rnn_type=rnn_type,
                bidirectional=bidirectional, dropout = drop_out, batch_norm = batch_norm).cuda()
        rnns.append(('%d' % (i+1), rnn))

    self.rnns = nn.Sequential(OrderedDict(rnns)).cuda()

    if batch_norm :
        fc = nn.Sequential(nn.BatchNorm1d(self.num_directions*rnn_hidden_size).cuda(),
                    nn.Linear(self.num_directions*rnn_hidden_size, rnn_hidden_size, bias=False).cuda()).cuda()
    else:
        fc = nn.Linear(self.num_directions*rnn_hidden_size, rnn_hidden_size, bias=False).cuda()
    self.fc = SequenceWise(fc).cuda()
    self.inference_log_softmax = InferenceBatchLogSoftmax().cuda()
    self.softmax=torch.nn.Softmax().cuda()
    #self.inference_softmax = InferenceBatchSoftmax()
    #tddf fusion lstm
    self.tddf_lstm=nn.LSTMCell(rnn_hidden_size,rnn_hidden_size).cuda()
    self.fc_s=nn.Linear(rnn_hidden_size,2,bias=True).cuda()
    self.fc_c=nn.Linear(rnn_hidden_size,2,bias=True) .cuda()
    self.hx=Variable(torch.zeros(100,rnn_hidden_size),requires_grad=True).cuda()
    self.cx=Variable(torch.zeros(100,rnn_hidden_size),requires_grad=True).cuda()
    self.fc_tddf=nn.Linear(rnn_hidden_size,num_class).cuda()
def forward(self, x,y):  #x: packed padded sequence [x.data: the origin data] [x.batch_sizes: the batch_size of each frames] [x_len: type:list not torch.IntTensor]
    #ipdb.set_trace()
    x = self.rnns(x)
    x = self.fc(x)
    x = self.inference_log_softmax(x)#(max_step,batch_size,dim)
    x=x.transpose(0,1)
    #x = self.inference_softmax(x)
    y=self.conv1_cnn(y)
    #banben 2_relu
    y=F.relu(y)
    y=self.conv2_cnn(torch.transpose(y,1,3))
    y=F.relu(y)
    y=self.fc_cnn(torch.transpose(y,1,3))#(batch_size,1,max_step,dim)
    #y=torch.transpose(y,1,3)
    y=y.view(100,-1,self.rnn_hidden_size)
    y=torch.transpose(y,0,1)

    output=Variable(torch.zeros(x.cpu().data.numpy().shape[0],100,self.rnn_hidden_size)).cuda()
    for i in range(x.cpu().data.numpy().shape[0]):
        #ipdb.set_trace()
        if i==0:
            st=F.softmax(self.fc_s(self.hx))
            ct=F.sigmoid(self.fc_c(self.hx))
            at=st*ct
            tddf_input_i_x=x[i]*at[:,0].contiguous().view(100,1).expand(100,self.rnn_hidden_size)
            tddf_input_i_y=y[i]*at[:,1].contiguous().view(100,1).expand(100,self.rnn_hidden_size)
            tddf_input_i=tddf_input_i_x+tddf_input_i_y
            hx,cx=self.tddf_lstm(tddf_input_i,(self.hx,self.cx))
            output[i]=hx
        else:
            st=F.softmax(self.fc_s(hx))
            ct=F.sigmoid(self.fc_c(hx))
            at=st*ct
            tddf_input_i_x=x[i]*at[:,0].contiguous().view(100,1).expand(100,self.rnn_hidden_size)
            tddf_input_i_y=y[i]*at[:,1].contiguous().view(100,1).expand(100,self.rnn_hidden_size)
            tddf_input_i=tddf_input_i_x+tddf_input_i_y
            #tddf_input_i=x[i]*at[:,0].contiguous().view(100,1).expand(100,self.rnn_hidden_size)+y[i]*at[:1].contiguous().view(100,1).expand(100,self.rnn_hidden_size)
            hx,cx=self.tddf_lstm(tddf_input_i,(hx,cx))
            output[i]=hx
    return self.fc_tddf(output)

And the error shows like:

Traceback (most recent call last):
File "/home/xinhaoran/PycharmProjects/TDDF/PH_ctc_cnn_tddf.py", line 358, in <module>
main()
File "/home/xinhaoran/PycharmProjects/TDDF/PH_ctc_cnn_tddf.py", line 353, in main
train()
File "/home/xinhaoran/PycharmProjects/TDDF/PH_ctc_cnn_tddf.py", line 256, in train
probs = model(feats,feats_cnn).cuda(async=True)
File "/home/xinhaoran/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 357, in __call__
result = self.forward(*input, **kwargs)
File "/home/xinhaoran/PycharmProjects/TDDF/model_tddf.py", line 167, in forward
x = self.rnns(x)
File "/home/xinhaoran/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 357, in __call__
result = self.forward(*input, **kwargs)
File "/home/xinhaoran/anaconda3/lib/python3.6/site-packages/torch/nn/modules/container.py", line 67, in forward
input = module(input)
File "/home/xinhaoran/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 357, in __call__
result = self.forward(*input, **kwargs)
File "/home/xinhaoran/PycharmProjects/TDDF/model_tddf.py", line 115, in forward
x, _ = self.rnn(x).cuda()
File "/home/xinhaoran/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 357, in __call__
result = self.forward(*input, **kwargs)
File "/home/xinhaoran/anaconda3/lib/python3.6/site-packages/torch/nn/modules/rnn.py", line 204, in forward
output, hidden = func(input, self.all_weights, hx)
File "/home/xinhaoran/anaconda3/lib/python3.6/site-packages/torch/nn/_functions/rnn.py", line 385, in forward
return func(input, *fargs, **fkwargs)
File "/home/xinhaoran/anaconda3/lib/python3.6/site-packages/torch/nn/_functions/rnn.py", line 245, in forward
nexth, output = func(input, hidden, weight)
File "/home/xinhaoran/anaconda3/lib/python3.6/site-packages/torch/nn/_functions/rnn.py", line 85, in forward
hy, output = inner(input, hidden[l], weight[l])
File "/home/xinhaoran/anaconda3/lib/python3.6/site-packages/torch/nn/_functions/rnn.py", line 158, in forward
hidden = inner(step_input, hidden, *weight)
File "/home/xinhaoran/anaconda3/lib/python3.6/site-packages/torch/nn/_functions/rnn.py", line 32, in LSTMCell
gates = F.linear(input, w_ih, b_ih) + F.linear(hx, w_hh, b_hh)
File "/home/xinhaoran/anaconda3/lib/python3.6/site-packages/torch/nn/functional.py", line 837, in linear
output = input.matmul(weight.t())
File "/home/xinhaoran/anaconda3/lib/python3.6/site-packages/torch/autograd/variable.py", line 386, in matmul
return torch.matmul(self, other)
File "/home/xinhaoran/anaconda3/lib/python3.6/site-packages/torch/functional.py", line 174, in matmul
return torch.mm(tensor1, tensor2)
RuntimeError: Expected object of type Variable[torch.FloatTensor] but found type Variable[torch.cuda.FloatTensor] for argument #1 'mat2'

I'd appreciate it if someone could help me out!!!~~
For the record, this seems to work on pytorch0.2.0(I'm using 0.3.0 now)


title: Deep Learning methods for Text Generation (PyTorch)

body:
Greetings to everyone,

I want to design a system that is able to generate stories or poetry based on a large dataset of text, without being needed to feed a text description/start/summary as input at inference time.

So far I did this using RNN's, but as you know they have a lot of flaws. My question is, what are the best methods to achieve this task at the time?
I searched for possibilities using Attention mechanisms, but it turns out that they are fitted for translation tasks.

I know about GPT-2, Bert, Transformer, etc., but all of them need a text description as input, before the generation and this is not what I'm seeking. I want a system able to generate stories from scratch after training.

Thanks a lot!


title: Re-using a classification CNN model for autoencoding - pytorch

body:
I am very new to pytorch so I need a bit of handholding. I am trying to re-use an old CNN classification model -- reusing the already trained convolutional layers as the encoder in an autoencoder and then training the decoder layers. The below code is what I have.

class Autoencoder(nn.Module):
  def __init__(self, model, specs):

    super(Autoencoder, self).__init__()

    self.encoder = nn.Sequential(
        *list(model.conv_layer.children())
        )

    self.decoder = nn.Sequential(
        nn.ConvTranspose2d(in_channels=C7, out_channels=C6, kernel_size=pooling, padding=0),
        nn.ReLU(inplace=True),
        nn.ConvTranspose2d(in_channels=C6, out_channels=C5, kernel_size=pooling, padding=0),
        nn.ReLU(inplace=True),
        nn.ConvTranspose2d(in_channels=C5, out_channels=C4, kernel_size=pooling, padding=0),
        nn.ReLU(inplace=True),
        nn.ConvTranspose2d(in_channels=C4, out_channels=C3, kernel_size=pooling, padding=0),
        nn.ReLU(inplace=True),
        nn.ConvTranspose2d(in_channels=C3, out_channels=C2, kernel_size=pooling, padding=0),
        nn.ReLU(inplace=True),
        nn.ConvTranspose2d(in_channels=C2, out_channels=C1, kernel_size=pooling, padding=0),
        nn.ReLU(inplace=True),
        nn.ConvTranspose2d(in_channels=C1, out_channels=C0, kernel_size=pooling, padding=0),
        nn.ReLU(inplace=True),
        nn.ConvTranspose2d(in_channels=C0, out_channels=3, kernel_size=pooling, padding=0),
        nn.ReLU(inplace=True),
        )
    for param in self.encoder.parameters():
      param.requires_grad = False

    for p in self.decoder.parameters():
      if p.dim() > 1:
        nn.init.kaiming_normal_(p)
        pass

    def forward(self, x):
      x = self.encoder(x)
      x = self.decoder(x)
      return x

However, I am getting a "NotImplementedError". What am I doing wrong? When I initiate an instance of that class, I would be passing the pretrained CNN classification model and self.encoder should take care of taking the layers I am interested from the model (those in conv_layer). When I:

model = pretrainedCNNmodel
autoencoder = Autoencoder(model, specs)
print(autoencoder)

the print looks okay, it has all layers and everything I am hoping for, but when I try to train on it I get the "NotImplementedError:".

Edit

Here is the entire error:

---------------------------------------------------------------------------
NotImplementedError                       Traceback (most recent call last)
<ipython-input-20-9adc467b2472> in <module>()
      2 optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate, weight_decay=L2_lambda)
      3
----> 4 train(x, train_loader, test_loader, optimizer, criterion)

2 frames
<ipython-input-5-b25edb14cf5f> in train(model, train_loader, test_loader, optimizer, criterion)
     15       data, target = data.cuda(), target.cuda()
     16       optimizer.zero_grad()
---> 17       output = model(data)
     18       loss = criterion(output, target)
     19       loss.backward()

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--> 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in forward(self, *input)
     94             registered hooks while the latter silently ignores them.
     95         """
---> 96         raise NotImplementedError
     97
     98     def register_buffer(self, name, tensor):

NotImplementedError:


title: Differential Privacy decreases the model performance significantly

body:
Background Information

I trained a classifier to predict three labels: COVID/Pneumonia/Healthy based on chest X-Ray images. It's a PyTorch implementation of COVID-Net. I use a training set to train on, validation set to save the best performing model, and then a test set to measure the "real" performance of the model. However, I noticed that my model "learned" to classify normal/pneumonia really good, but it just ignored the underpopulated COVID set. Therefore I choose to undersample (reduce the number of training instances of the other classes (normal and pneumonia) in order to get equal populations). This worked well, but my sample set has been reduced to ~1500 samples (low!). The results are somewhat worse than COVID-Net, I achieve an accuracy of ~80% and lower sensitivity on underpopulated classes (COVID) then they report. I suppose that they report better performance because they do not use a validation-set and use the test-set each epoch. I figured that they might indirectly overfit on the test-set because of that. I have chosen to explain this so that the reader gets a context.

Question

I tried adding privacy to the training procedure by using Differential Privacy. Specifically, I used Facebook's PyTorch-DP module. Training works just as well if I choose to add almost no-privacy (this can be achieved by choosing a really low noise multiplier value (sigma), i.e. 1e-7) and a really high delta. So it's not that the module itself is not working/faulty, but, if I use a lower sigma (so I add more noise) then I get more privacy (epsilon decreases) but the model fails to fit the data at all.
The question is: how do I manage to add privacy to a somewhat meaningful degree while making sure that my model somewhat fits the data still?

Performance differences

Confusion Matrix of Model without Differential Privacy added. It's not "good" but it's at least somewhat meaningful and the model reaches an accuracy of ~80%.

Confusion Matrix of Model with Differential Privacy (epsilon: 2.3) after 100 epochs. It looks as if the model does not know what to do, at all.

Possible explanations

I read a paper that stated that adding Differential Privacy can cause bad performance in the sense that the accuracy decreases for underpopulated classes. But, I used undersampling and I think this should've solved that, but the accuracy stays bad (for all classes!).

Maybe because my sample set is so small, differential privacy is much harder to achieve, and therefore the performance is bad? However, even if add a really tiny bit of privacy, with an epsilon value >20000, the model still struggles in learning how to classify. So I'm not sure.