title: PyTorch forward pass using weights trained by Theano

body:
I've trained a small size CNN binary classifier in Theano. To have a simpler code, I wanted to port the trained weights to PyTorch or numpy forward pass for predictions. The predictions by original Theano program are satisfying but the PyTorch forward pass predicted all the examples to one class.

Here is how I save trained weights in Theano using h5py:

layer0_w = layer0.W.get_value(borrow=True)
layer0_b = layer0.b.get_value(borrow=True)
layer1_w = layer1.W.get_value(borrow=True)
layer1_b = layer1.b.get_value(borrow=True)
layer2_w = layer2.W.get_value(borrow=True)
layer2_b = layer2.b.get_value(borrow=True)
sm_w = layer_softmax.W.get_value(borrow=True)
sm_b = layer_softmax.b.get_value(borrow=True)

h5_l0w = h5py.File('./model/layer0_w.h5', 'w')
h5_l0w.create_dataset('layer0_w', data=layer0_w)
h5_l0b = h5py.File('./model/layer0_b.h5', 'w')
h5_l0b.create_dataset('layer0_b', data=layer0_b)
h5_l1w = h5py.File('./model/layer1_w.h5', 'w')
h5_l1w.create_dataset('layer1_w', data=layer1_w)
h5_l1b = h5py.File('./model/layer1_b.h5', 'w')
h5_l1b.create_dataset('layer1_b', data=layer1_b)
h5_l2w = h5py.File('./model/layer2_w.h5', 'w')
h5_l2w.create_dataset('layer2_w', data=layer2_w)
h5_l2b = h5py.File('./model/layer2_b.h5', 'w')
h5_l2b.create_dataset('layer2_b', data=layer2_b)
h5_smw = h5py.File('./model/softmax_w.h5', 'w')
h5_smw.create_dataset('softmax_w', data=sm_w)
h5_smb = h5py.File('./model/softmax_b.h5', 'w')
h5_smb.create_dataset('softmax_b', data=sm_b)

Then load the weights to build a forward pass using Pytorch and Numpy:

import torch
import numpy as np
import torch.nn.functional as F
def model(data):

    conv0_out = F.conv2d(input=np2var(data),
                         weight=np2var(layer0_w),
                         bias=np2var(layer0_b)
                        )
    layer0_out = relu(var2np(conv0_out))

    conv1_out = F.conv2d(input=np2var(layer0_out),
                         weight=np2var(layer1_w),
                         bias=np2var(layer1_b)
                        )
    layer1_out = np.max(relu(var2np(conv1_out)), axis=2)

    dense_out=relu(np.matmul(layer1_out, layer2_w) + layer2_b)

    softmax_out = softmax(np.matmul(dense_out, softmax_w) + softmax_b)

    return softmax_out

def relu(x):
    return x * (x > 0)
def np2var(x):
    return torch.autograd.Variable(torch.from_numpy(x))
def var2np(x):
    return x.data.numpy()
def softmax(x):
    e_x = np.exp(x - np.max(x))
    return e_x / e_x.sum()

The input and kernel shapes for conv2d functions are the same for Theano and PyTorch, and network structures in two frameworks are the same. I couldn't detect any errors step by step. What could go wrong here?


title: Pytorch CUDA OutOfMemory Error while training

body:
I'm trying to train a PyTorch FLAIR model in AWS Sagemaker.
While doing so getting the following error:
RuntimeError: CUDA out of memory. Tried to allocate 84.00 MiB (GPU 0; 11.17 GiB total capacity; 9.29 GiB already allocated; 7.31 MiB free; 10.80 GiB reserved in total by PyTorch)

For training I used sagemaker.pytorch.estimator.PyTorch class.
I tried with different variants of instance types from ml.m5, g4dn to p3(even with a 96GB memory one).
In the ml.m5 getting the error with CPUmemoryIssue, in g4dn with GPUMemoryIssue and in the P3 getting GPUMemoryIssue mostly because Pytorch is using only one of the GPU of 12GB out of 8*12GB.
Not getting anywhere to complete this training, even in local tried with a CPU machine and got the following error:
RuntimeError: [enforce fail at ..\c10\core\CPUAllocator.cpp:72] data. DefaultCPUAllocator: not enough memory: you tried to allocate 67108864 bytes. Buy new RAM!

The model training script:
    corpus = ClassificationCorpus(data_folder, test_file='../data/exports/val.csv', train_file='../data/exports/train.csv')

    print("finished loading corpus")

    word_embeddings = [WordEmbeddings('glove'), FlairEmbeddings('news-forward-fast'), FlairEmbeddings('news-backward-fast')]

    document_embeddings = DocumentLSTMEmbeddings(word_embeddings, hidden_size=512, reproject_words=True, reproject_words_dimension=256)

    classifier = TextClassifier(document_embeddings, label_dictionary=corpus.make_label_dictionary(), multi_label=False)

    trainer = ModelTrainer(classifier, corpus, optimizer=Adam)

    trainer.train('../model_files', max_epochs=12,learning_rate=0.0001, train_with_dev=False, embeddings_storage_mode="none")

P.S.: I was able to train the same architecture with a smaller dataset in my local GPU machine with a 4GB GTX 1650 DDR5 memory and it was really quick.


title: How to get the image torchvision.utils.save_image saves, without reading it back from disk?

body:
from torchvision.utils import save_image

...
save_image(im, f'im_name.png')

In my case (standard mnist), using code from here, im is a Tensor:96, and save_image works.
I want that image in memory to show it in other plots, and I don't want to read it back after saving it, which seems kind of stupid.
Is there a way to separate the functionality of generating the image and of saving it?

Edit
clarification:
I want an equivalent to
save_image(im, f'im_name.png')
reread = plt.imread(f'im_name.png')

without saving the image and reading it back.
I just want the image, and I want to save it later.
the save_image function does some work, like stacking multiple images into one, converting the tensor to images of correct sizes and so on. I want only that part without the saving to disk.


title: Deploy PyTorch model on React Native

body:
Does it possible to deploy pytorch model on React Native? Any way that work similar like deploying a tensorflow model using tensorflow.js? I'm trying to build a object detector with YOLOv5 which is implemented in PyTorch.
Update:
In the mean time, Im using AWS as my backend cloud services. If i cant deploy on react native , the most efficient way is to store the model in s3 and made prediction by importing the model using lambda function right?


title: How to ignore specific columns for calculating softmax attention?

body:
I am implementing a model which is based on MemoryNetworks. I have triplets data of (context, query, answer). And I want to calculate attention. The attention indicates which sentences in a context should be focused.

To formulate mini-batch, I use zero-paddings to create context data. So the following is attention data. And 0 values show a result of embeddings of zero-padded context.
In such a data, I want to apply softmax to indices 0, 1, 2, 3, last. So the model should ignore zero padding columns.

So how do I realize this? I want to know such a technique when we use zero-padding and attention mechanisms. Although I prefer pytorch, any tools or just description answers will be also appreciated.

Before softmax.  torch.bmm(contex, q)
 109.8601
  77.6376
  68.3927
 199.1673
   0.0000
   0.0000
   0.0000
   0.0000
   0.0000
   0.0000
   0.0000
   0.0000
   0.0000
   0.0000
 348.0155
[torch.cuda.FloatTensor of size 15 (GPU 0)]

After softmax. F.softmax( torch.bmm(contex, q) )
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1
[torch.cuda.FloatTensor of size 15 (GPU 0)]


title: PyTorch - to NumPy yields unsized object?

body:
Converting a PyTorch tensor to NumPy I get

print(nn_result.shape)
# (2433, 2)
np_result = torch.argmax(nn_result).numpy()
type(np_result)
# <type 'numpy.ndarray'>
print(len(np_result))
TypeError: len() of unsized object

Why? I thought per documentation the numpy() function would return a proper ndarray, yet it seems to be incomplete somehow?


title: Simplest LSTM with attention (Encoder-Decoder architecture) using Pytorch

body:
please, help me understand how to write LSTM (RNN) with attention using Encoder-Decoder architecture. I've watched a lot of videos on YouTube, read some articles on towardsdatascience.com and so on but those examples are very complex for me. I need the most simple example of RNN that can do what I said before.
Anyway, I did some attempts to write it myself (and it works!) but the loss is big all the time and it is almost not changing. I can explain you how I think it should work (and I'm going to describe every step I did in my code) but please can you correct me (I mean say, where my mistakes were) and make from my code (if it possible) the workable LSTM  with attention with Encoder-Decoder architecture. I would be so happy to get this important knowledge for me. Let's go!
Firstly, here is the full code:
import torch.nn as nn
import torch.optim as optim
import torch
import numpy as np

class Encoder(nn.Module):
  def __init__(self):
    super(Encoder, self).__init__()
    self.lstm = nn.LSTM(1, 1, 1, batch_first=True)

  def forward(self, input):
    output, hidden_state = self.lstm(input)
    #print('DATA:', self.lstm.weight_hh_l0[0])
    return output

class Decoder(nn.Module):
  def __init__(self):
    super(Decoder, self).__init__()
    self.lstm = nn.LSTM(10, 1, 1, batch_first=True)
    self.linear = nn.ModuleList([nn.Linear(10, 10) for i in range(10)])
    self.softmax = nn.Softmax(dim=0)

  def forward(self, input):
    input = input.view(1, 10)
    initial_input = input.clone()
    for i in range(10):
      if i == 0:
        input = self.linear[i](input)
        #print(input)
      else:
        input = torch.cat((input, self.linear[i](initial_input)), 0)
        #print(input)
    input = input.view(-1, 10, 10)
    output, hidden_state = self.lstm(input)
    output = output.view(1, 10)
    return output

class RNN(nn.Module):
  def __init__(self):
    super(RNN, self).__init__()

    self.encoder = Encoder()
    self.decoder = Decoder()
    print(self.encoder)
    print(self.decoder)

    self.loss = nn.CrossEntropyLoss()
    self.encoder_optimizer = optim.Adagrad(self.encoder.parameters(), lr=0.006)
    self.decoder_optimizer = optim.Adagrad(self.decoder.parameters(), lr=0.006)

  def train(self, input, target, test):

    # Encoder
    output_encoder = self.encoder(input)

    # Decoder
    output_decoder = self.decoder(output_encoder)

    if test:
      print(np.argmax(torch.nn.functional.softmax(output_decoder).detach().numpy().flatten()))

    self.encoder_optimizer.zero_grad()
    self.decoder_optimizer.zero_grad()

    loss = self.loss(output_decoder, target)
    print(loss)

    loss.backward()

    self.encoder_optimizer.step()
    self.decoder_optimizer.step()

    return 0

def main():
  rnn = RNN()

  input = [[10, 1, 1, 1, 1, 1, 1, 1, 1, 1], [10, 1, 1, 1, 1, 1, 1, 1, 1, 2], [10, 1, 1, 1, 1, 1, 1, 1, 1, 3], [10, 1, 1, 1, 1, 1, 1, 1, 1, 4], [10, 1, 1, 1, 1, 1, 1, 1, 1, 5], [10, 1, 1, 1, 1, 1, 1, 1, 1, 6], [10, 1, 1, 1, 1, 1, 1, 1, 1, 7], [10, 1, 1, 1, 1, 1, 1, 1, 1, 8], [10, 1, 1, 1, 1, 1, 1, 1, 1, 9], [10, 1, 1, 1, 1, 1, 1, 1, 1, 10]]
  labels = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

  labels = torch.tensor(labels).type(dtype=torch.long)

  losses = []
  for epoch in range(100):
    input_copy = input[epoch % 10]
    labels_copy = [labels[epoch % 10]]
    labels_copy = torch.tensor(labels_copy).type(dtype=torch.long)
    input_copy = torch.tensor(input_copy).float().requires_grad_(True).view(1, 10, 1)
    rnn.train(input_copy, labels_copy, False)

  rnn.train(input_copy, labels_copy, True)

main()

Secondly, here is the picture I've made to help you understand what I did:
[click here to see image][1]
Also, I should say that this LSTM has no any real meaning, I just want to know how to write such type of RNN.
Let's dive into the code closer (READ COMMENTS, PLEASE), here is Encoder:
class Encoder(nn.Module):
  def __init__(self):
    super(Encoder, self).__init__()
    self.lstm = nn.LSTM(1, 1, 1, batch_first=True) #input is like [[1], [2], ..., [10]], also, I'm planning to get WHOLE output from this nn.LSTM (not the last state), I mean the variable 'output' down below

  def forward(self, input):
    output, hidden_state = self.lstm(input) # just implement nn.LSTM
    #print('DATA:', self.lstm.weight_hh_l0[0])
    return output

Here is Decoder (the most complex part for me):
class Decoder(nn.Module):
  def __init__(self):
    super(Decoder, self).__init__()
    self.lstm = nn.LSTM(10, 1, 1, batch_first=True) #initialize RNN for the decoder
    self.linear = nn.ModuleList([nn.Linear(10, 10) for i in range(10)]) #generating linear layer in order to kind of apply sub neural network that will somehow predict weights for important 'words' in our 'sentence' and I decided that those 10 linear layers (they were generated with different weights) should be applied to output from Encoder and each this linear layer should produce the 'input word' for the Decoder's LSTM (you can see what I said in the picture)
    self.softmax = nn.Softmax(dim=0)

  def forward(self, input): #doing what I said before
    input = input.view(1, 10)
    initial_input = input.clone()
    for i in range(10):
      if i == 0:
        input = self.linear[i](input)
        #print(input)
      else:
        input = torch.cat((input, self.linear[i](initial_input)), 0) #merge all outputs from linear layers in one tensor, so, we've made the input 'sentence' for our Decoder's LSTM
        #print(input)
    input = input.view(-1, 10, 10)
    output, hidden_state = self.lstm(input)
    output = output.view(1, 10)
    return output

In the class called 'RNN' we just initialize our Encoder and Decoder and we do the standart things for every simple RNN (anyway, I'm not sure whether I'm implementing all of them in a right order because now I have kind of two RNNs in one program (if it would be only one RNN, I'm sure that I'm doing everything right)).
Well, here are the rersults (this is the loss):
tensor(2.4289, grad_fn=<NllLossBackward>)
tensor(2.3605, grad_fn=<NllLossBackward>)
tensor(2.4271, grad_fn=<NllLossBackward>)
tensor(2.4054, grad_fn=<NllLossBackward>)
tensor(2.4332, grad_fn=<NllLossBackward>)
tensor(2.2394, grad_fn=<NllLossBackward>)
tensor(2.2026, grad_fn=<NllLossBackward>)
tensor(2.0889, grad_fn=<NllLossBackward>)
tensor(2.3166, grad_fn=<NllLossBackward>)
tensor(2.3591, grad_fn=<NllLossBackward>)
tensor(2.3845, grad_fn=<NllLossBackward>)
tensor(2.3353, grad_fn=<NllLossBackward>)
tensor(2.4073, grad_fn=<NllLossBackward>)
tensor(2.3933, grad_fn=<NllLossBackward>)
tensor(2.4244, grad_fn=<NllLossBackward>)
tensor(2.2370, grad_fn=<NllLossBackward>)
tensor(2.2015, grad_fn=<NllLossBackward>)
tensor(2.0889, grad_fn=<NllLossBackward>)
tensor(2.3139, grad_fn=<NllLossBackward>)
tensor(2.3567, grad_fn=<NllLossBackward>)
tensor(2.3785, grad_fn=<NllLossBackward>)
tensor(2.3313, grad_fn=<NllLossBackward>)
tensor(2.4027, grad_fn=<NllLossBackward>)
tensor(2.3904, grad_fn=<NllLossBackward>)
tensor(2.4217, grad_fn=<NllLossBackward>)
tensor(2.2371, grad_fn=<NllLossBackward>)
tensor(2.2024, grad_fn=<NllLossBackward>)
tensor(2.0907, grad_fn=<NllLossBackward>)
tensor(2.3134, grad_fn=<NllLossBackward>)
tensor(2.3556, grad_fn=<NllLossBackward>)
tensor(2.3751, grad_fn=<NllLossBackward>)
tensor(2.3290, grad_fn=<NllLossBackward>)
tensor(2.3998, grad_fn=<NllLossBackward>)
tensor(2.3885, grad_fn=<NllLossBackward>)
tensor(2.4197, grad_fn=<NllLossBackward>)
tensor(2.2374, grad_fn=<NllLossBackward>)
tensor(2.2034, grad_fn=<NllLossBackward>)
tensor(2.0925, grad_fn=<NllLossBackward>)
tensor(2.3135, grad_fn=<NllLossBackward>)
tensor(2.3549, grad_fn=<NllLossBackward>)
tensor(2.3727, grad_fn=<NllLossBackward>)
tensor(2.3274, grad_fn=<NllLossBackward>)
tensor(2.3975, grad_fn=<NllLossBackward>)
tensor(2.3870, grad_fn=<NllLossBackward>)
tensor(2.4181, grad_fn=<NllLossBackward>)
tensor(2.2376, grad_fn=<NllLossBackward>)
tensor(2.2043, grad_fn=<NllLossBackward>)
tensor(2.0942, grad_fn=<NllLossBackward>)
tensor(2.3136, grad_fn=<NllLossBackward>)
tensor(2.3545, grad_fn=<NllLossBackward>)
tensor(2.3709, grad_fn=<NllLossBackward>)
tensor(2.3261, grad_fn=<NllLossBackward>)
tensor(2.3956, grad_fn=<NllLossBackward>)
tensor(2.3857, grad_fn=<NllLossBackward>)
tensor(2.4168, grad_fn=<NllLossBackward>)
tensor(2.2378, grad_fn=<NllLossBackward>)
tensor(2.2052, grad_fn=<NllLossBackward>)
tensor(2.0957, grad_fn=<NllLossBackward>)
tensor(2.3139, grad_fn=<NllLossBackward>)
tensor(2.3541, grad_fn=<NllLossBackward>)
tensor(2.3693, grad_fn=<NllLossBackward>)
tensor(2.3250, grad_fn=<NllLossBackward>)
tensor(2.3940, grad_fn=<NllLossBackward>)
tensor(2.3846, grad_fn=<NllLossBackward>)
tensor(2.4155, grad_fn=<NllLossBackward>)
tensor(2.2380, grad_fn=<NllLossBackward>)
tensor(2.2060, grad_fn=<NllLossBackward>)
tensor(2.0972, grad_fn=<NllLossBackward>)
tensor(2.3141, grad_fn=<NllLossBackward>)
tensor(2.3539, grad_fn=<NllLossBackward>)
tensor(2.3679, grad_fn=<NllLossBackward>)
tensor(2.3240, grad_fn=<NllLossBackward>)
tensor(2.3925, grad_fn=<NllLossBackward>)
tensor(2.3836, grad_fn=<NllLossBackward>)
tensor(2.4144, grad_fn=<NllLossBackward>)
tensor(2.2382, grad_fn=<NllLossBackward>)
tensor(2.2068, grad_fn=<NllLossBackward>)
tensor(2.0985, grad_fn=<NllLossBackward>)
tensor(2.3144, grad_fn=<NllLossBackward>)
tensor(2.3537, grad_fn=<NllLossBackward>)
tensor(2.3667, grad_fn=<NllLossBackward>)
tensor(2.3232, grad_fn=<NllLossBackward>)
tensor(2.3912, grad_fn=<NllLossBackward>)
tensor(2.3828, grad_fn=<NllLossBackward>)
tensor(2.4134, grad_fn=<NllLossBackward>)
tensor(2.2384, grad_fn=<NllLossBackward>)
tensor(2.2075, grad_fn=<NllLossBackward>)
tensor(2.0998, grad_fn=<NllLossBackward>)
tensor(2.3147, grad_fn=<NllLossBackward>)
tensor(2.3535, grad_fn=<NllLossBackward>)
tensor(2.3656, grad_fn=<NllLossBackward>)
tensor(2.3224, grad_fn=<NllLossBackward>)
tensor(2.3900, grad_fn=<NllLossBackward>)
tensor(2.3819, grad_fn=<NllLossBackward>)
tensor(2.4125, grad_fn=<NllLossBackward>)
tensor(2.2386, grad_fn=<NllLossBackward>)
tensor(2.2082, grad_fn=<NllLossBackward>)
tensor(2.1009, grad_fn=<NllLossBackward>)
tensor(2.3150, grad_fn=<NllLossBackward>)
tensor(2.3534, grad_fn=<NllLossBackward>)

Well, can you help me? This RNN should predict which sequence (like [10, 1, 1, 1, 1, 1, 1, 1, 1, 1]) belongs to what class.
Please, if it's possible, change my code in order to I could look at workable example and improve my understanding. I also know that LSTM with attention is needed to work with very big "sequence_length' but I just want to understand the concept of suce architecture. Or can you write me new simplest example? Well, any help will be appreciated.
[1]: https://i.stack.imgur.com/D2V0S.jpg


title: Increasing the size of images displayed in Pytorch

body:
I want to display few images and their respective labels using Pytorch dataloader.

However the image displayed is very tiny grid.

How do I increase the width of each image so it's bigger.

Here's the code I used:

mean_nums = [0.485, 0.456, 0.406]
std_nums = [0.229, 0.224, 0.225]

def imshow(inp, title=None):
    """Imshow for Tensor."""
    inp = inp.numpy().transpose((1, 2, 0))
    mean = np.array(mean_nums)
    std = np.array(std_nums)
    inp = std * inp + mean
    inp = np.clip(inp, 0, 1)
    plt.imshow(inp)
    if title is not None:
        plt.title(title)
    plt.pause(0.001)  # pause a bit so that plots are updated

# Get a batch of training data
inputs, classes = next(iter(dataloaders['trainLoader']))

# Make a grid from batch
out = torchvision.utils.make_grid(inputs,nrow=2)

imshow(out, title=[image_datasets['train'].classes[x] for x in classes])


title: What is the best way to handle the background pixel classes (ignore_label), when training deep learning models for semantic segmentation?

body:
I am trying to train a UNET model on the cityscapes dataset which has 20 'useful' semantic classes and a bunch of background classes that can be ignored (ex. sky, ego vehicle, mountains, street lights). To train the model to ignore these background pixels I am using the following popular solution on the internet :

I assign a common ignore_label (ex: ignore_label=255) for all
the pixels belonging to the ignore classes
Train the model using the cross_entropy loss for each pixel prediction
Provide the ignore_label parameter in the cross_entropy loss, therefore the
loss computed ignores the pixels with the unnecessary classes.

But this approach has a problem. Once trained, the model ends up classifying these background pixels as belonging to one of the 20 classes instead. This is expected as in the loss we do not penalize the model for whatever classification it makes for the background pixels.

The second obvious solution is therefore to use a extra class for all the background pixels. Therefore it is the 21st class in cityscapes. However, here I am worried that I will 'waste' my model's capacity by teaching it to classify this additional unnecessary class.

What is the most accurate way of handling the background pixel classes ?


title: Padding a tensor until reaching required size

body:
I'm working with certian tensors with shape of (X,42) while X can be in a range between 50 to 70.
I want to pad each tensor that I get until it reaches a size of 70. so all tensors will be (70,42).
is there anyway to do this when I the begining size is a variable X? thanks for the help!


title: What to define as entrypoint when initializing a pytorch estimator with a custom docker image for training on AWS Sagemaker?

body:
So I created a docker image for training. In the dockerfile I have an entrypoint defined such that when docker run is executed, it will start running my python code.
To use this on aws sagemaker in my understanding I need to create a pytorch estimator in a jupyter notebook in sagemaker. I tried something like this:

import sagemaker
from sagemaker.pytorch import PyTorch

sagemaker_session = sagemaker.Session()

role = sagemaker.get_execution_role()

estimator = PyTorch(entry_point='train.py',
                    role=role,
                    framework_version='1.3.1',
                    image_name='xxx.ecr.eu-west-1.amazonaws.com/xxx:latest',
                    train_instance_count=1,
                    train_instance_type='ml.p3.xlarge',
                    hyperparameters={})

estimator.fit({})

In the documentation I found that as image name I can specify the link the my docker image on aws ecr. When I try to execute this it keeps complaining

[Errno 2] No such file or directory: 'train.py'

It complains immidiatly, so surely I am doing something completely wrong. I would expect that first my docker image should run, and than it could find out that the entry point does not exist.

But besides this, why do I need to specify an entry point, as in, should it not be clear that the entry to my training is simply docker run?

For maybe better understanding. The entrypoint python file in my docker image looks like this:

if __name__=='__main__':
    parser = argparse.ArgumentParser()

    # Hyperparameters sent by the client are passed as command-line arguments to the script.
    parser.add_argument('--epochs', type=int, default=5)
    parser.add_argument('--batch_size', type=int, default=16)
    parser.add_argument('--learning_rate', type=float, default=0.0001)

    # Data and output directories
    parser.add_argument('--output_data_dir', type=str, default=os.environ['OUTPUT_DATA_DIR'])
    parser.add_argument('--train_data_path', type=str, default=os.environ['CHANNEL_TRAIN'])
    parser.add_argument('--valid_data_path', type=str, default=os.environ['CHANNEL_VALID'])

    # Start training
    ...

Later I would like to specify the hyperparameters and data channels. But for now I simply do not understand what to put as entry point. In the documentation it says that the entrypoint is required and it should be a local/global path to the entrypoint...


title: PyTorch: How to define a new neural network that utilizes transfer learning

body:
I am migrating from Keras/TF frameworks and I have litte troubles understanding the transfer learning process in PyTorch.

I want to use pytorch-lightning framework and I want to switch between different neural networks in one script.

Per this example we can switch between different neural networks in their implementation:

class BERT(pl.LightningModule):
def __init__(self, model_name, task):
    self.task = task

    if model_name == 'transformer':
        self.net = Transformer()
    elif model_name == 'my_cool_version':
        self.net = MyCoolVersion()

The question is: how to create a new neural network that extends the nn.Module and utilizes transfer learning process?

My own implementation looks like this: I am using vgg16 network and replaced the classifier layer with only one fc with two output neurons.

class VGGNetwork(nn.Module):
    def __init__(self):
        super(VGGNetwork, self).__init__()
        # vgg16 is the default model here, we can use bn etc...
        self.model = vgg16(pretrained=True)

        # removing the last three layers of classifier only 2 ...
        self.model.classifier = nn.Sequential(nn.Linear(512 * 7 * 7, 2))

def forward(self, x):
    return self.model.forward(x)

Is this the correct way how to do that?


title: Torchtext on Colab: Cannot import name 'dtype_to_attr'

body:
having trouble importing torchtext.data or .datasets with the following error on Google Colab. Tried to 1/ upgrade torch to 1.7, 2/ upgrade torchtext to 0.4 or 0.5 or latest as per some posts. But none seem to help.
Wondering if you have any insights here? Trying to import to embed / tokenize word sequences.

ImportError                               Traceback (most recent call last)
 in ()
1 # For data loading.
----> 2 from torchtext import data, datasets
2 frames
/usr/local/lib/python3.6/dist-packages/torchtext/data/field.py in ()
8 from .dataset import Dataset
9 from .pipeline import Pipeline
---> 10 from .utils import get_tokenizer, dtype_to_attr, is_tokenizer_serializable
11 from ..vocab import Vocab, SubwordVocab
12
ImportError: cannot import name 'dtype_to_attr'


title: Pytorch lightning saving model during the epoch

body:
I would like to save a checkpoint every time a validation loop ends. I set up the val_check_interval to be 0.2 so I have 5 validation loops during each epoch but the checkpoint callback saves the model only at the end of the epoch. I couldn't find an easy (or hard) way to save the model after each validation loop. It seems a bit strange cause I can't see a reason to make the validation loop other then saving a checkpoint.
What is the simplest way of doing that?
Thank you,
Gilad


title: GRU Language Model not Training Properly

body:
I’ve tried reimplementing a simple GRU language model using just a GRU and a linear layer (the full code is also at https://www.kaggle.com/alvations/gru-language-model-not-training-properly):

class Generator(nn.Module):
    def __init__(self, vocab_size, embedding_size, hidden_size, num_layers):
        super(Generator, self).__init__()

        # Initialize the embedding layer with the
        # - size of input (i.e. no. of words in input vocab)
        # - no. of hidden nodes in the embedding layer
        self.embedding = nn.Embedding(vocab_size, embedding_size, padding_idx=0)

        # Initialize the GRU with the
        # - size of the input (i.e. embedding layer)
        # - size of the hidden layer
        self.gru = nn.GRU(embedding_size, hidden_size, num_layers)

        # Initialize the "classifier" layer to map the RNN outputs
        # to the vocabulary. Remember we need to -1 because the
        # vectorized sentence we left out one token for both x and y:
        # - size of hidden_size of the GRU output.
        # - size of vocabulary
        self.classifier = nn.Linear(hidden_size, vocab_size)

    def forward(self, inputs, use_softmax=False, hidden=None):
        # Look up for the embeddings for the input word indices.
        embedded = self.embedding(inputs)
        # Put the embedded inputs into the GRU.
        output, hidden = self.gru(embedded, hidden)

        # Matrix manipulation magic.
        batch_size, sequence_len, hidden_size = output.shape
        # Technically, linear layer takes a 2-D matrix as input, so more manipulation...
        output = output.contiguous().view(batch_size * sequence_len, hidden_size)
        # Put it through the classifier
        # And reshape it to [batch_size x sequence_len x vocab_size]
        output = self.classifier(output).view(batch_size, sequence_len, -1)

        return (F.softmax(output,dim=2), hidden) if use_softmax else (output, hidden)

    def generate(self, max_len, temperature=1.0):
        pass

And the training routine:

device = 'cuda' if torch.cuda.is_available() else 'cpu'

# Set the hidden_size of the GRU
embed_size = 100
hidden_size = 100
num_layers = 1

# Setup the data.
batch_size=50
kilgariff_data = KilgariffDataset(tokenized_text)
dataloader = DataLoader(dataset=kilgariff_data, batch_size=batch_size, shuffle=True)

criterion = nn.CrossEntropyLoss(ignore_index=kilgariff_data.vocab.token2id['<pad>'], size_average=True)
model = Generator(len(kilgariff_data.vocab), embed_size, hidden_size, num_layers).to(device)

learning_rate = 0.003
optimizer = optim.Adam(model.parameters(), lr=learning_rate)

#model = nn.DataParallel(model)

losses = []

def train(num_epochs, dataloader, model, criterion, optimizer):
    plt.ion()
    for _e in range(num_epochs):
        for batch in tqdm(dataloader):
            x = batch['x'].to(device)
            x_len = batch['x_len'].to(device)
            y = batch['y'].to(device)
            # Zero gradient.
            optimizer.zero_grad()
            # Feed forward.
            output, hidden = model(x, use_softmax=True)
            # Compute loss:
            # Shape of the `output` is [batch_size x sequence_len x vocab_size]
            # Shape of `y` is [batch_size x sequence_len]
            # CrossEntropyLoss expects `output` to be [batch_size x vocab_size x sequence_len]

            _, prediction = torch.max(output, dim=2)
            loss = criterion(output.permute(0, 2, 1), y)
            loss.backward()
            optimizer.step()
            losses.append(loss.float().data)

            clear_output(wait=True)
            plt.plot(losses)
            plt.pause(0.05)

train(50, dataloader, model, criterion, optimizer)

#learning_rate = 0.05
#optimizer = optim.SGD(model.parameters(), lr=learning_rate)
#train(4, dataloader, model, criterion, optimizer)

But when the model is predicting, we see that it’s only predicting “the” and comma “,”.

Anyone spot something wrong with my code? Or hyperparameters?

The full code:

# coding: utf-8

# In[1]:

# IPython candies...
from IPython.display import Image
from IPython.core.display import HTML

from IPython.display import clear_output

# In[2]:

import numpy as np
from tqdm import tqdm

import pandas as pd

from gensim.corpora import Dictionary

import torch
from torch import nn, optim, tensor, autograd
from torch.nn import functional as F
from torch.utils.data import Dataset, DataLoader

from torch.nn.utils.rnn import pack_padded_sequence, pad_packed_sequence

device = 'cuda' if torch.cuda.is_available() else 'cpu'

# In[3]:

import matplotlib.pyplot as plt
import seaborn as sns

sns.set_style("darkgrid")
sns.set(rc={'figure.figsize':(12, 8)})

torch.manual_seed(42)

# In[4]:

try: # Use the default NLTK tokenizer.
    from nltk import word_tokenize, sent_tokenize
    # Testing whether it works.
    # Sometimes it doesn't work on some machines because of setup issues.
    word_tokenize(sent_tokenize("This is a foobar sentence. Yes it is.")[0])
except: # Use a naive sentence tokenizer and toktok.
    import re
    from nltk.tokenize import ToktokTokenizer
    # See https://stackoverflow.com/a/25736515/610569
    sent_tokenize = lambda x: re.split(r'(?<=[^A-Z].[.?]) +(?=[A-Z])', x)
    # Use the toktok tokenizer that requires no dependencies.
    toktok = ToktokTokenizer()
    word_tokenize = word_tokenize = toktok.tokenize

# In[5]:

import os
import requests
import io #codecs

# Text version of https://kilgarriff.co.uk/Publications/2005-K-lineer.pdf
if os.path.isfile('language-never-random.txt'):
    with io.open('language-never-random.txt', encoding='utf8') as fin:
        text = fin.read()
else:
    url = "https://gist.githubusercontent.com/alvations/53b01e4076573fea47c6057120bb017a/raw/b01ff96a5f76848450e648f35da6497ca9454e4a/language-never-random.txt"
    text = requests.get(url).content.decode('utf8')
    with io.open('language-never-random.txt', 'w', encoding='utf8') as fout:
        fout.write(text)

# In[6]:

# Tokenize the text.
tokenized_text = [list(map(str.lower, word_tokenize(sent)))
                  for sent in sent_tokenize(text)]

# In[7]:

class KilgariffDataset(nn.Module):
    def __init__(self, texts):
        self.texts = texts

        # Initialize the vocab
        special_tokens = {'<pad>': 0, '<unk>':1, '<s>':2, '</s>':3}
        self.vocab = Dictionary(texts)
        self.vocab.patch_with_special_tokens(special_tokens)

        # Keep track of the vocab size.
        self.vocab_size = len(self.vocab)

        # Keep track of how many data points.
        self._len = len(texts)

        # Find the longest text in the data.
        self.max_len = max(len(txt) for txt in texts)

    def __getitem__(self, index):
        vectorized_sent = self.vectorize(self.texts[index])
        x_len = len(vectorized_sent)
        # To pad the sentence:
        # Pad left = 0; Pad right = max_len - len of sent.
        pad_dim = (0, self.max_len - len(vectorized_sent))
        vectorized_sent = F.pad(vectorized_sent, pad_dim, 'constant')
        return {'x':vectorized_sent[:-1],
                'y':vectorized_sent[1:],
                'x_len':x_len}

    def __len__(self):
        return self._len

    def vectorize(self, tokens, start_idx=2, end_idx=3):
        """
        :param tokens: Tokens that should be vectorized.
        :type tokens: list(str)
        """
        # See https://radimrehurek.com/gensim/corpora/dictionary.html#gensim.corpora.dictionary.Dictionary.doc2idx
        # Lets just cast list of indices into torch tensors directly =)

        vectorized_sent = [start_idx] + self.vocab.doc2idx(tokens) + [end_idx]
        return torch.tensor(vectorized_sent)

    def unvectorize(self, indices):
        """
        :param indices: Converts the indices back to tokens.
        :type tokens: list(int)
        """
        return [self.vocab[i] for i in indices]

# In[8]:

kilgariff_data = KilgariffDataset(tokenized_text)
len(kilgariff_data.vocab)

# In[9]:

batch_size = 10
dataloader = DataLoader(dataset=kilgariff_data, batch_size=batch_size, shuffle=True)

for data_dict in dataloader:
    # Sort indices of data in batch by lengths.
    sorted_indices = np.array(data_dict['x_len']).argsort()[::-1].tolist()
    data_batch = {name:_tensor[sorted_indices]
                  for name, _tensor in data_dict.items()}
    print(data_batch)
    break

# In[97]:

class Generator(nn.Module):
    def __init__(self, vocab_size, embedding_size, hidden_size, num_layers):
        super(Generator, self).__init__()

        # Initialize the embedding layer with the
        # - size of input (i.e. no. of words in input vocab)
        # - no. of hidden nodes in the embedding layer
        self.embedding = nn.Embedding(vocab_size, embedding_size, padding_idx=0)

        # Initialize the GRU with the
        # - size of the input (i.e. embedding layer)
        # - size of the hidden layer
        self.gru = nn.GRU(embedding_size, hidden_size, num_layers)

        # Initialize the "classifier" layer to map the RNN outputs
        # to the vocabulary. Remember we need to -1 because the
        # vectorized sentence we left out one token for both x and y:
        # - size of hidden_size of the GRU output.
        # - size of vocabulary
        self.classifier = nn.Linear(hidden_size, vocab_size)

    def forward(self, inputs, use_softmax=False, hidden=None):
        # Look up for the embeddings for the input word indices.
        embedded = self.embedding(inputs)
        # Put the embedded inputs into the GRU.
        output, hidden = self.gru(embedded, hidden)

        # Matrix manipulation magic.
        batch_size, sequence_len, hidden_size = output.shape
        # Technically, linear layer takes a 2-D matrix as input, so more manipulation...
        output = output.contiguous().view(batch_size * sequence_len, hidden_size)
        # Put it through the classifier
        # And reshape it to [batch_size x sequence_len x vocab_size]
        output = self.classifier(output).view(batch_size, sequence_len, -1)

        return (F.softmax(output,dim=2), hidden) if use_softmax else (output, hidden)

    def generate(self, max_len, temperature=1.0):
        pass

# In[98]:

# Set the hidden_size of the GRU
embed_size = 12
hidden_size = 10
num_layers = 4

_encoder = Generator(len(kilgariff_data.vocab), embed_size, hidden_size, num_layers)

# In[99]:

# Take a batch.
_batch = next(iter(dataloader))
_inputs, _lengths = _batch['x'], _batch['x_len']
_targets = _batch['y']
max(_lengths)

# In[100]:

_output, _hidden = _encoder(_inputs)
print('Output sizes:\t', _output.shape)
print('Input sizes:\t', batch_size, kilgariff_data.max_len -1, len(kilgariff_data.vocab))
print('Target sizes:\t', _targets.shape)

# In[101]:

_, predicted_indices = torch.max(_output, dim=2)
print(predicted_indices.shape)
predicted_indices

# In[103]:

device = 'cuda' if torch.cuda.is_available() else 'cpu'

# Set the hidden_size of the GRU
embed_size = 100
hidden_size = 100
num_layers = 1

# Setup the data.
batch_size=50
kilgariff_data = KilgariffDataset(tokenized_text)
dataloader = DataLoader(dataset=kilgariff_data, batch_size=batch_size, shuffle=True)

criterion = nn.CrossEntropyLoss(ignore_index=kilgariff_data.vocab.token2id['<pad>'], size_average=True)
model = Generator(len(kilgariff_data.vocab), embed_size, hidden_size, num_layers).to(device)

learning_rate = 0.003
optimizer = optim.Adam(model.parameters(), lr=learning_rate)

#model = nn.DataParallel(model)

losses = []

def train(num_epochs, dataloader, model, criterion, optimizer):
    plt.ion()
    for _e in range(num_epochs):
        for batch in tqdm(dataloader):
            x = batch['x'].to(device)
            x_len = batch['x_len'].to(device)
            y = batch['y'].to(device)
            # Zero gradient.
            optimizer.zero_grad()
            # Feed forward.
            output, hidden = model(x, use_softmax=True)
            # Compute loss:
            # Shape of the `output` is [batch_size x sequence_len x vocab_size]
            # Shape of `y` is [batch_size x sequence_len]
            # CrossEntropyLoss expects `output` to be [batch_size x vocab_size x sequence_len]

            _, prediction = torch.max(output, dim=2)
            loss = criterion(output.permute(0, 2, 1), y)
            loss.backward()
            optimizer.step()
            losses.append(loss.float().data)

            clear_output(wait=True)
            plt.plot(losses)
            plt.pause(0.05)

train(50, dataloader, model, criterion, optimizer)

#learning_rate = 0.05
#optimizer = optim.SGD(model.parameters(), lr=learning_rate)
#train(4, dataloader, model, criterion, optimizer)

# In[ ]:

list(kilgariff_data.vocab.items())

# In[105]:

start_token = '<s>'
hidden_state = None
max_len = 20
temperature=0.8

i = 0

while start_token not in ['</s>', '<pad>'] and i < max_len:
    i += 1
    start_state = torch.tensor(kilgariff_data.vocab.token2id[start_token]).unsqueeze(0).unsqueeze(0).to(device)
    model.embedding(start_state)
    output, hidden_state = model.gru(model.embedding(start_state), hidden_state)

    batch_size, sequence_len, hidden_size = output.shape
    output = output.contiguous().view(batch_size * sequence_len, hidden_size)

    output = model.classifier(output).view(batch_size, sequence_len, -1)
    _, prediction = torch.max(F.softmax(output, dim=2), dim=2)

    start_token = kilgariff_data.vocab[int(prediction.squeeze(0).squeeze(0))]

    print(start_token, end=' ')


title: Dataloader on top of protobuf file using pytorch's torch.utils.data.IterableDataset

body:
I am building a RNN network using pytorch.
The data is stored in various protobuf file.
Each record in protobuf represents one training example with multiple timestamp.

As this is very large dataset, reading the whole data in memory or random read by extending torch.utils.data.Dataset class isn't feasible.

As per the docs using the torch.utils.data.IterableDataset is recommended.

DataLoader on top of IterableDataset would be able to achieve parallelism

However I am not able to find an implementation of this on custom data,  docs only talk about a simple range iterator.

import math

import stream
from src import record_pb2
import torch

class MyIterableDataset(torch.utils.data.IterableDataset):
    def __init__(self, pb_file):
        self.pb_file = pb_file
        self.start = 0
        self.end = 0
        # One time read of the data to get the total count of records in the dataset
        with stream.open(self.pb_file, 'rb') as data_stream:
            for _ in data_stream:
                self.end += 1

    def __iter__(self):
        worker_info = torch.utils.data.get_worker_info()
        if worker_info is None:  # Single-process data loading, return the full iterator
            iter_start = self.start
            iter_end = self.end
        else:
            # in a worker process, split the workload
            per_worker = int(math.ceil((self.end - self.start))/float(worker_info.num_workers))
            worker_id = worker_info.id
            iter_start = self.start + worker_id * per_worker
            iter_end = min(iter_start + per_worker, self.end)

        data_stream = stream.open(self.pb_file, 'rb')

        # Block to skip the streaming data till the iter start for the current worker process
        i = 0
        for _ in data_stream:
            i += 1
            if i >= iter_start:
                break

        return iter(self.pb_stream)

I am expecting a mechanism by which a parallel data feeder could be designed on top of a large streaming data (protobuf)


title: avoid gradient breaking with `torch.where` for combining tensors

body:
Here is a a very brief example of the problem I am having and what I am after

nan = float("nan")
t = th.Tensor([[nan, nan], [0, 0], [.1, .1],])

l = nn.Linear(2, 2)
tf = l(t)

tf = th.where(t != t, th.zeros_like(t), tf)
(10 - tf.sum()).backward()
print(l.weight.grad)

why is the gradient nan (especially when the following is just fine)

nan = float("nan")
t = th.Tensor([[nan, nan], [0, 0], [.1, .1],])
t_o = th.where(t != t, th.zeros_like(t), t)

l = nn.Linear(2, 2)
tf = l(t_o)

(10 - tf.sum()).backward()
print(l.weight.grad)

giving the correct gradient:

tensor([[-0.1000, -0.1000],
        [-0.1000, -0.1000]])

The Full Question

Suppose I have t tensors of shape (n, f_i) where f_i can be different for each t. Furthermore, each of these t tensors may have some nan rows (assume each tensor represents a type of object and I want a shared embedding of all those objects). What I would like to do is "assimilate" these tensors into a shared space.

Here is a minimal module attempting to do this:

import torch as th
import torch.nn as n

class Assimilation(nn.Module):
    def __init__(self, assimilation_dim: int, *type_in_dim):
        super().__init__()
        self._assimilation_dim = assimilation_dim
        self._type_in_dim = type_in_dim
        self._mixins = nn.ModuleList(
            [
                nn.Linear(in_dim, self._assimilation_dim, bias=False)
                for in_dim in self._type_in_dim
            ]
        )

    def forward(self, *x: th.Tensor):
        assims, = []
        for to_assim, assim_layer in zip(x, self._mixins):
            assimilated = assim_layer(to_assim)
            assims.append(assimilated)

        # the problem area (I believe)
        current = assims[0]
        for i in range(1, len(assims)):
            current = th.where(current != current, assims[i], current)
        return current

Note, the implementation has some more requirements like allowing tensors of more arbitrary shapes (*, n, f_i) for more leading dimensions. For this reason, I wanted to avoid tracking indices as things get more complex when batching this kind of data (or adding time, etc) - especially since torch has bad fancy indexing.

However, when trying to use it you can observe that the gradient breaks from the th.where

ass = Assimilation(2, 2, 3) # :)
nan = float("nan")

t1 = th.Tensor([[nan, nan], [0, 0], [.1, .1],])
t2 = th.Tensor([[100, 100, 100], [nan, nan, nan], [nan, nan, nan],])

res = ass(t1, t2)
print(t1.shape, t2.shape)
print(res.shape)
print(res)
(10 - res.sum()).backward()
print(ass._mixins[0].weight.grad) # all nan
print(ass._mixins[1].weight.grad) # all nan

Is there a reason th.where should be breaking the gradient here? How can I avoid this?


title: How does one save torch.nn.Sequential models in pytorch properly?

body:
I am very well aware of loading the dictionary and then having a instance of be loaded with the old dictionary of parameters (e.g. this great question & answer). Unfortunately, when I have a torch.nn.Sequential I of course do not have a class definition for it.
So I wanted to double check, what is the proper way to do it. I believe torch.save is sufficient (so far my code has not collapsed), though these things can be more subtle than one might expect (e.g. I get a warning when I use pickle but torch.save uses it internally so it's confusing). Also, numpy has it's own save functions (e.g. see this answer) which tend to be more efficient, so there might be a subtle trade off I might be overlooking.

My test code:

# creating data and running through a nn and saving it

import torch
import torch.nn as nn

from pathlib import Path
from collections import OrderedDict

import numpy as np

import pickle

path = Path('~/data/tmp/').expanduser()
path.mkdir(parents=True, exist_ok=True)

num_samples = 3
Din, Dout = 1, 1
lb, ub = -1, 1

x = torch.torch.distributions.Uniform(low=lb, high=ub).sample((num_samples, Din))

f = nn.Sequential(OrderedDict([
    ('f1', nn.Linear(Din,Dout)),
    ('out', nn.SELU())
]))
y = f(x)

# save data torch to numpy
x_np, y_np = x.detach().cpu().numpy(), y.detach().cpu().numpy()
np.savez(path / 'db', x=x_np, y=y_np)

print(x_np)
# save model
with open('db_saving_seq', 'wb') as file:
    pickle.dump({'f': f}, file)

# load model
with open('db_saving_seq', 'rb') as file:
    db = pickle.load(file)
    f2 = db['f']

# test that it outputs the right thing
y2 = f2(x)

y_eq_y2 = y == y2
print(y_eq_y2)

db2 = {'f': f, 'x': x, 'y': y}
torch.save(db2, path / 'db_f_x_y')

print('Done')

db3 = torch.load(path / 'db_f_x_y')
f3 = db3['f']
x3 = db3['x']
y3 = db3['y']
yy3 = f3(x3)

y_eq_y3 = y == y3
print(y_eq_y3)

y_eq_yy3 = y == yy3
print(y_eq_yy3)

Related:

related question from forum: https://discuss.pytorch.org/t/how-to-save-nn-sequential-as-a-model/89117/14


title: Pytorch + BERT+ batch_encode_plus() Code running fine in Colab but producing problems with Kaggle in mismatch input shapes

body:
I tried to use a Google Colab initialised Notebook for Kaggle and found a strange behaviour as it gave me something like:
 16   # text2tensor
---> 17   train_seq,train_mask,train_y = textToTensor(train_text,train_labels,pad_len)
     18   val_seq,val_mask,val_y       = textToTensor(val_text,val_labels,pad_len)
     19

<ipython-input-9-ee85c4607a30> in textToTensor(text, labels, max_len)
      4   tokens = tokenizer.batch_encode_plus(text.tolist(), max_length=max_len, padding='max_length', truncation=True)
      5
----> 6   text_seq = torch.tensor(tokens['input_ids'])
      7   text_mask = torch.tensor(tokens['attention_mask'])
      8

ValueError: expected sequence of length 38 at dim 1 (got 13)

The error came from the code below:
def textToTensor(text,labels=None,max_len=38):#max_len is 38

  tokens = tokenizer.batch_encode_plus(text.tolist(), max_length=max_len, padding='max_length', truncation=True)

  text_seq = torch.tensor(tokens['input_ids']) # ERROR CAME FROM HERE
  text_mask = torch.tensor(tokens['attention_mask'])

  text_y = None
  if isinstance(labels,np.ndarray):
    text_y = torch.tensor(labels.tolist())

  return text_seq, text_mask, text_y

train_seq,train_mask,train_y = textToTensor(train_text,train_labels,pad_len)
train_data = TensorDataset(train_seq, train_mask, train_y)
train_sampler = RandomSampler(train_data)
train_dataloader = DataLoader(train_data, sampler=train_sampler, batch_size=batch_size)

I again ran this code on COLAB and it ran smoothly. Can it be because of the Version and something like that? Can someone please this?
Kaggle Configs:
transformers: '2.11.0'
torch: '1.5.1'
python: 3.7.6

Colab Configs:
torch: 1.7.0+cu101
transformers: 3.5.1
python: 3.6.9

EDIT:
My train_text is a numpy array of texts and train_labels is 1-d numerical array with 4 classes ranging 0-3.
Also: I initialized my tokenizer as:
from transformers import BertTokenizerFast
tokenizer = BertTokenizerFast.from_pretrained('bert-base-uncased')


title: Load .txt file with pickle on non GPU machine

body:
I am trying to load a .txt file on a non GPU machine. The .txt file was dumped on a GPU machine:

with open("Data/all_rewards.txt", "wb") as f:
    pickle.dump(all_rewards, f)

The file contains just text data, no network models or anything else.
On a GPU machine, I have no trouble loading the data:

with open("all_rewards.txt", 'rb') as f:
    losses = pickle.load(f)

On a non GPU machine however, I get the error:

AssertionError('Torch not compiled with CUDA enabled',)

Since Torch uses pickle too to load models, I have tried to load the data with :

losses = torch.load("all_rewards.txt", map_location='cpu')

but that didn´t work.
Could you help me plaese?

EDIT:
Somehow this problem solved itself magically. I have reinstalled pickle, which did not solve the problem initially but now it works...


title: Store images and labels to hdf5 files

body: I have a folder of images and a csv file with a label corresponding to each image. I am trying to train a pytorch model, the problem is data loading seems to be a bottleneck,so i want to store the images to a HDf5 file and read them.


title: Pytorch: How to predict a 1D vector from a 2D vector/image?

body:
I'm attempting use Pytorch to predict a 1D vector (a frame of clean speech data) using regression from a 2D vector (a sequence of frames of noisy speech data) - which has been done before. The sequence of frames provide temporal context for the frame to more accurately predict the clean frame. The vectors can be thought of as analogous to a 2D grayscale image and 1D grayscale image.

When the batch size is 64, window length is 5 and frame length is 257 the input tensor has a shape of [64, 1, 5, 257] and the target tensor has a shape of [64, 1, 1, 257].

There are some examples of this being done in TensorFlow, but none that I can find using Pytorch. This is my best attempt so far to replicate this paper (https://www.isca-speech.org/archive/Interspeech_2017/pdfs/1465.PDF).

def __init__(self, window_length, frame_length, batch_size):
        super(Net, self).__init__()
        self.layer1 = nn.Sequential(
            nn.Conv2d(1, 12, kernel_size=(1,13), stride=1, padding=(0,6)),
            # nn.BatchNorm2d(12),
            nn.ReLU())
        self.layer2 = nn.Sequential(
            nn.Conv2d(12, 16, kernel_size=(1,11), stride=1, padding=(0,5)),
            # nn.BatchNorm2d(16),
            nn.ReLU())
        self.layer3 = nn.Sequential(
            nn.Conv2d(16, 20, kernel_size=(1,9), stride=1, padding=(0,4)),
            # nn.BatchNorm2d(20),
            nn.ReLU())
        self.layer4 = nn.Sequential(
            nn.Conv2d(20, 24, kernel_size=(1,7), stride=1, padding=(0,3)),
            # nn.BatchNorm2d(24),
            nn.ReLU())
        self.layer5 = nn.Sequential(
            nn.Conv2d(24, 32, kernel_size=(1,7), stride=1, padding=(0,3)),
            # nn.BatchNorm2d(32),
            nn.ReLU())
        self.layer6 = nn.Sequential(
            nn.Conv2d(32, 24, kernel_size=(1,7), stride=1, padding=(0,3)),
            # nn.BatchNorm2d(24),
            nn.ReLU())
        self.layer7 = nn.Sequential(
            nn.Conv2d(24, 20, kernel_size=(1,9), stride=1, padding=(0,4)),
            # nn.BatchNorm2d(20),
            nn.ReLU())
        self.layer8 = nn.Sequential(
            nn.Conv2d(20, 16, kernel_size=(1,11), stride=1, padding=(0,5)),
            # nn.BatchNorm2d(16),
            nn.ReLU())
        self.layer9 = nn.Sequential(
            nn.Conv2d(16, 12, kernel_size=(1,13), stride=1, padding=(0,6)),
            # nn.BatchNorm2d(12),
            nn.ReLU())
        self.conv_out = nn.Sequential(
            nn.Conv2d(12, 1, kernel_size=(1,1), stride=1, padding=(0,0)),
            )
        self.fc1 = nn.Linear(batch_size * window_length * frame_length, frame_length)

    def forward(self, x):
        out = self.layer1(x)
        out = self.layer2(out)
        out = self.layer3(out)
        out = self.layer4(out)
        out = self.layer5(out)
        out = self.layer6(out)
        out = self.layer7(out)
        out = self.layer8(out)
        out = self.layer9(out)
        out = self.conv_out(out)
        out = self.fc1(out)
        return out

Calling .forward() on this net results in the following error message:

RuntimeError: size mismatch, m1: [320 x 257], m2: [82240 x 257]

How can I reduce the output layer to 1x257 per sample to match the target (a single frame of length 257)?


title: Running into CUDA out of memory

body:
I am trying to evaluate MAE for images using a CNN model with pre-trained weights and after 240th iteration I run into this error (in the image)

mae = 0
for i in range(len(img_paths)):
    img = 255.0 * F.to_tensor(Image.open(img_paths[i]).convert('RGB'))

    img[0,:,:]=img[0,:,:]-92.8207477031
    img[1,:,:]=img[1,:,:]-95.2757037428
    img[2,:,:]=img[2,:,:]-104.877445883
    img = img.cuda()
    #img = transform(Image.open(img_paths[i]).convert('RGB')).cuda()
    gt_file = h5py.File(img_paths[i].replace('.jpg','.h5').replace('images','ground'),'r')
    groundtruth = np.asarray(gt_file['density'])
    output = model(img.unsqueeze(0))
    del img
    mae += float(abs(output.detach().cpu().sum().numpy()-np.sum(groundtruth)))
    print (i,mae)
print (mae/len(img_paths))

However , the funny thing is , the exact same code threw no errors/warnings about memory or anything at all just a few weeks ago. I havent changed anything. But yesterday when I tried to run it , it threw this memory run out error. The batch size is 1. I tried the solutions from other similar threads but nothing seems to be working.
I am not very familiar with using pytorch . So can anyone suggest what to do?
One other thing I observed is , when the code runs into this error and stop , running the command - nvidia-smi- shows the following warning :
WARNING: infoROM is corrupted at gpu 0000:01:00.0

This warning doesnt show up when I reboot my system. Appears only after this code is run. Does this mean my Graphics Card (GTX-1650) is damaged?


title: PyTorch Factorial Function

body:
There does not seem to be a PyTorch function for computing a factorial. Is there a method to do this in PyTorch? I am looking to manually compute a Poisson distribution in Torch (I am aware this exists: https://pytorch.org/docs/stable/generated/torch.poisson.html) and the formula requires a factorial in the denominator.
Poisson Distribution: https://en.wikipedia.org/wiki/Poisson_distribution


title: How PyTorch implements Convolution Backward?

body:
I read about the Pytorch's source code, and I find it's weird that it doesn't implement the convolution_backward function, The only convolution_backward_overrideable function is directly raises an error and supposed not to fall here.
So I referred to CuDNN / MKLDNN implementation, they both implements functions like cudnn_convolution_backward.
I got the following question:

What are the native implementation of CUDA/ CPU? I can find something like thnn_conv2d_backward_out, but I could not find where are this is called.

Why PyTorch didn't put the convolution_backward function in Convolution.cpp? It offers an _convolution_double_backward() function. But this is the double backward, it's the gradient of gradient. Why don't they offer a single backward function?

If I want to call the native convolution/ convolution_backward function for my pure cpu/cuda tensor, how should I write code? Or where could I refer to? I couldn't find example for this.

Thanks !


title: How to check if a model is in train or eval mode in Pytorch?

body: How to check from within a model if it is currently in train or eval mode?


title: PyTorch Linear Regression Issue

body:
I am trying to implement a simple linear model in PyTorch that can be given x data and y data, and then trained to recognize the equation y = mx + b. However, whenever I try to test my model after training, it thinks that the equation is y= mx + 2b. I'll show my code, and hopefully someone will be able to spot an issue. Thank you in advance for any help.

import torch

D_in = 500
D_out = 500
batch=200
model=torch.nn.Sequential(
     torch.nn.Linear(D_in,D_out),
)

Next I create some data and set a rule. Let's do 3x+4.

x_data=torch.rand(batch,D_in)
y_data=torch.randn(batch,D_out)

for i in range(batch):
    for j in range(D_in):
         y_data[i][j]=3*x_data[i][j]+5 # model thinks y=mx+c -> y=mx+2c?

loss_fn=torch.nn.MSELoss(size_average=False)
optimizer=torch.optim.Adam(model.parameters(),lr=0.001)

Now to training...

for epoch in range(500):
    y_pred=model(x_data)
    loss=loss_fn(y_pred,y_data)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

Then I test my model with a Tensor/matrix of just 1's.

test_data=torch.ones(batch,D_in)
y_pred=model(test_data)

Now, I'd expect to get 3*1 + 4 = 7, but instead, my model thinks it is 11.

[[ 10.7286,  11.0499,  10.9448,  ...,  11.0812,  10.9387,
      10.7516],
    [ 10.7286,  11.0499,  10.9448,  ...,  11.0812,  10.9387,
      10.7516],
    [ 10.7286,  11.0499,  10.9448,  ...,  11.0812,  10.9387,
      10.7516],
    ...,
    [ 10.7286,  11.0499,  10.9448,  ...,  11.0812,  10.9387,
      10.7516],
    [ 10.7286,  11.0499,  10.9448,  ...,  11.0812,  10.9387,
      10.7516],
    [ 10.7286,  11.0499,  10.9448,  ...,  11.0812,  10.9387,
      10.7516]])

Similarly, if I change the rule to y=3x+8, my model guesses 19. So, I am not  sure what is going on. Why is the constant being added twice? By the way, if I just set the rule to y=3x, my model correctly infers 3, and for y=mx in general my model correctly infers m. For some reason, the constant term is throwing it off. Any help to solve this problem is much appreciated. Thanks!


title: How Weight update in Dynamic Computation Graph of pytorch works?

body:
How does the Weight Update works in Pytorch code of Dynamic Computation Graph when Weights are shard (=reused multiple times)

https://pytorch.org/tutorials/beginner/examples_nn/dynamic_net.html#sphx-glr-beginner-examples-nn-dynamic-net-py

import random
import torch

class DynamicNet(torch.nn.Module):
    def __init__(self, D_in, H, D_out):
    """
    In the constructor we construct three nn.Linear instances that we will use
    in the forward pass.
    """
    super(DynamicNet, self).__init__()
    self.input_linear = torch.nn.Linear(D_in, H)
    self.middle_linear = torch.nn.Linear(H, H)
    self.output_linear = torch.nn.Linear(H, D_out)

def forward(self, x):
    """
    For the forward pass of the model, we randomly choose either 0, 1, 2, or 3
    and reuse the middle_linear Module that many times to compute hidden layer
    representations.

    Since each forward pass builds a dynamic computation graph, we can use normal
    Python control-flow operators like loops or conditional statements when
    defining the forward pass of the model.

    Here we also see that it is perfectly safe to reuse the same Module many
    times when defining a computational graph. This is a big improvement from Lua
    Torch, where each Module could be used only once.
    """
    h_relu = self.input_linear(x).clamp(min=0)
    for _ in range(random.randint(0, 3)):
        h_relu = self.middle_linear(h_relu).clamp(min=0)
    y_pred = self.output_linear(h_relu)
    return y_pred

I want to know what happens to middle_linear weight at each backward which is used multiple times at a step


title: pytorch.load() dict object has not attribute 'features'

body:
import torch
    import torch.nn as nn
    from torch.autograd import Variable
    import torchvision.models as models

    class AlexSal(nn.Module):
        def __init__(self):
            super(AlexSal, self).__init__()
            self.features = nn.Sequential(*list(torch.load('alexnet_places365.pth.tar').features.children())[:-2])
            self.relu = nn.ReLU()
            self.sigmoid = nn.Sigmoid()
            self.conv6 = nn.Conv2d(256, 1, kernel_size=(1, 1), stride=(1, 1))

        def forward(self, x):
            x = self.relu(self.features(x))
            x = self.sigmoid(self.conv6(x))
            x = x.squeeze(1)
            return x
    model = AlexSal().cuda()

Traceback (most recent call last):
  File "main.py", line 23, in <module>
    model = AlexSal().cuda()
  File "main.py", line 13, in __init__
    self.features = nn.Sequential(*list(torch.load('alexnet_places365.pth.tar').features.children())[:-2])
AttributeError: 'dict' object has no attribute 'features'

I got this piece of code from internet , i downloaded alexnet_places365.pth.tar ,and when i am running this , it is showing the above error


title: Is there a way to update the weights of a layer/variable directly after a training step?

body:
There are ways to get and update the weight directly outside of training, but what I am looking to do is after the each training step and after the gradients have updated a particular variable or layer, I would like to save those weights to file and then replace the layer or variable weights with brand new ones. And then continue with the next step (forward pass using the new values in the variable or layer, then backward pass with the calculated loss/gradients) of training.

I have thought about just calling each training step individually, but I am wondering if that is somehow very time/memory inefficient.


title: Is there an implementation of forward mapping/warping in a neural network framework?

body:
Introduction

I want to train a neural network and require a differentiable implementation of a forward mapping to warp a source image into a destination image, see e.g. https://www.cs.princeton.edu/courses/archive/fall00/cs426/lectures/warp/warp.pdf.

What I already found is backward mapping via pytorch (grid_sample), in which every pixel in destination image has only a single connection to one particular pixel in source image (+ neighbours).

Code

import torch
import torch.nn.functional as F
import numpy as np
import matplotlib.pyplot as plt

# create artificial image
im = np.zeros((9,9))
im[4,4] = 1

# warping images
u = np.zeros((9,9))
v = np.zeros((9,9))

u[4,3] += 1

# shift to pytorch
h,w = np.shape(im)

im_t = torch.from_numpy(im).float().view((1,1,h,w)).cpu()
u_t = torch.from_numpy(u).float().view((1,h,w,1)).cpu()
v_t = torch.from_numpy(v).float().view((1,h,w,1)).cpu()

n, _, h, w = im_t.size()
batchSize, _, _, nmbSamples = im_t.size()

# warp source image
U = torch.arange(start=0, end=w).expand(h, w).float().view((n, h, w, 1)).cpu()
V = torch.arange(start=0, end=h).expand(w, h).t().float().view((n, h, w, 1)).cpu()

U_u = (U + u_t) / ((w-1) / 2) - 1
V_v = (V + v_t) / ((h-1) / 2) - 1

grid = torch.cat((U_u, V_v), dim=3).float().cpu()

warped_t = F.grid_sample(im_t, grid)

# plot routine
plt.figure()
plt.subplot(121)
plt.imshow(im_t.data.cpu().numpy()[0,0,:,:])
plt.grid()

plt.subplot(122)
plt.imshow(warped_t.data.cpu().numpy()[0,0,:,:])
plt.grid()

plt.show(block=False)
plt.draw()

Problem

What I need, is to render the destination image from different, non neighbouring pixels in source image.

Help appreciated.
Regards


title: Why do we need " state_dict = state_dict.copy() "

body:
In my task I want to load the weights from the already pretrained model to my local model. But I don’t understand why we need this line " state_dict = state_dict.copy() " if the two have the same name " state_dict"

# copy state_dict so _load_from_state_dict can modify it
metadata = getattr(state_dict, '_metadata', None)
state_dict = state_dict.copy()
if metadata is not None:
    state_dict._metadata = metadata

def load(module, prefix=''):

    local_metadata = {} if metadata is None else metadata.get(prefix[:-1], {})
    module._load_from_state_dict(
        state_dict, prefix, local_metadata, True, missing_keys, unexpected_keys, error_msgs)
    for name, child in module._modules.items():
        if child is not None:
            load(child, prefix + name + '.')
start_prefix = ''
# print("hasattr(model, 'bert')",hasattr(model, 'bert')  ) :false
if not hasattr(model, 'bert') and any(s.startswith('bert.') for s in state_dict.keys()):
    start_prefix = 'bert.'
load(model, prefix=start_prefix)

Note: the above code is from huggingface, (line: 669)
https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_utils.py


title: Pytorch geometric Data object edge_attr for undirected graphs

body:
How would one construct a edge_attr list for undirected graphs in pytorch geometric Data object. Say we had an undirected graph such as this one. The graph COO matrix required by the pytorch geometric object is:

[['a', 'b', 'a', 'c', 'b', 'c']  ['b', 'a', 'c', 'a', 'c', 'b']]

How would one construct the edge_attr list then (which is an array of one-hot encoded vectors for the features of each edge). Since the graph is undirected, would one simply append two of the same one-hot encoded feature vectors at a time. For example, say these are the edge features :

(a, b) = [0,0,0,1]  (a, c) = [1,0,0,0]  (b, c) = [0,1,0,0]

Would the edge_attr list look like:

[[0,0,0,1], [0,0,0,1], [1,0,0,0], [1,0,0,0], [0,1,0,0], [0,1,0,0]

Note how each one-hot encoded feature vector is repeated twice, and the feature vector index in edge_attr corresponds its respective edge in the graph COO matrix. Because the graph is undirected, we just use the same feature matrix. Is the right way to do it, or is there some other way?


title: Pytorch: image transformer not converting to grayscale

body:
My input file contains images or faces showing emotions, and there are 8 labels, 1 for neutral and the others for the different emotions.
I want to train my CNN to classify the emotions of the images.
I need to convert my images to grayscale. This is an image before transformation;

However the image I get afterwards is

My tranformer code is as follows;

   _transforms = transforms.Compose([
        transforms.ToPILImage(),
        transforms.Grayscale(num_output_channels=1),
        transforms.Resize((ImageSize.width, ImageSize.height)),
        transforms.ToTensor()
    ])
    train_dataset = EmotionDatasetTrain(X_tr, y_tr, transform=_transforms)
    test_dataset = EmotionDatasetTest(X_t, y_t, transform=_transforms)
    val_dataset = EmotionDatasetVal(X_v, y_v, transform=_transforms)

My EmotionDatasetTrain class (and super class) is;

import torch import torchvision as tv from torchvision import
transforms from torch.utils.data import Dataset from settings import
ImageSize
class EmotionDataSet():
def init(self, transform):
self.transform = transform
self.classes = ["neutral", "anger", "contempt", "disgust", "fear", "happy", "sadness", "surprise"]
def get_x_y(self, idx, _X, _Y):
    if torch.is_tensor(idx):
        idx = idx.tolist()

    y = _Y[idx].item()

    if self.transform:
        x = self.transform(_X[idx].reshape(ImageSize.width, ImageSize.height))
        #x = self.transform(_X[idx])
    return x, y

class EmotionDatasetTrain(EmotionDataSet, Dataset):
def __init__(self, X_Train, Y_Train, transform=None):
    super().__init__(transform)
    self.X_Train = X_Train
    self.Y_Train = Y_Train

def __len__(self):
    return len(self.X_Train)

def __getitem__(self, idx):
    x, y = super().get_x_y(idx, self.X_Train, self.Y_Train)
    return x, y

So why is my image being converted to a very low resolution, purple and yellow image? I am expecting a high resolution black and white image. My CNN is not able to learn from images transformed like this.


title: Pytorch - RuntimeError: invalid multinomial distribution (encountering probability entry < 0)

body:
I am using Stable Baselines 3 to train an agent to play Connect 4 game. I am trying to take the case into account when an agent starts a game as a second player.
self.env = self.ks_env.train([opponent, None])

When I am trying to run the code, I am getting the following error:
invalid multinomial distribution (encountering probability entry < 0)
/opt/conda/lib/python3.7/site-packages/torch/distributions/categorical.py in sample(self, sample_shape)
samples_2d = torch.multinomial(probs_2d, sample_shape.numel(), True).T

However, there is no problem when an agent is first player:
self.env = self.ks_env.train([None, opponent])

I think problem is related to the Pytorch library. My question is how can I fix this issue?


title: Pytorch nn.CrossEntropyLoss giving, ValueError: Expected target size (x, y), got torch.Size([x, z]) for 3d tensor

body:
I am following the example here, where the documentation says:

Input: (N, C) where C = number of classes
Target: (N) where each value is 0 ≤ targets[i] ≤ C−1

And this is the case with the example given for a 2d tensor
loss = nn.CrossEntropyLoss()
input = torch.randn(3, 5, requires_grad=True)
target = torch.empty(3, dtype=torch.long).random_(5)
output = loss(input, target)
output.backward()

But for a 2d tensor, I am getting an error
import torch.nn as nn
import torch
loss = nn.CrossEntropyLoss(ignore_index=0)

inputs = torch.rand(32, 128, 3)
targets = torch.ones(32, 128)

loss(inputs, targets.long())

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-26-61e7f03039a6> in <module>
      7 targets = torch.ones(32, 128)
      8
----> 9 loss(inputs, targets.long())

/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

/opt/conda/lib/python3.8/site-packages/torch/nn/modules/loss.py in forward(self, input, target)
    959
    960     def forward(self, input: Tensor, target: Tensor) -> Tensor:
--> 961         return F.cross_entropy(input, target, weight=self.weight,
    962                                ignore_index=self.ignore_index, reduction=self.reduction)
    963

/opt/conda/lib/python3.8/site-packages/torch/nn/functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction)
   2466     if size_average is not None or reduce is not None:
   2467         reduction = _Reduction.legacy_get_string(size_average, reduce)
-> 2468     return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
   2469
   2470

/opt/conda/lib/python3.8/site-packages/torch/nn/functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)
   2271         out_size = (n,) + input.size()[2:]
   2272         if target.size()[1:] != input.size()[2:]:
-> 2273             raise ValueError('Expected target size {}, got {}'.format(
   2274                 out_size, target.size()))
   2275         input = input.contiguous()

ValueError: Expected target size (32, 3), got torch.Size([32, 128])

As far as I can tell, I am doing everything right regarding setting up the dimensions. The error message seems to think that I am giving a 2d vector, but I gave it a 3d vector, the 128 size dimension is missing.
Is there something that I didn't set up correctly for this loss function?


title: Calculating Distance in Parameter Space for PyTorch Network

body:
I'm new to PyTorch. I want to keep track of the distance in parameter-space my model travels through its optimization. This is the code I'm using.

class ParameterDiffer(object):
    def __init__(self, network):
        network_params = []
        for p in network.parameters():
            network_params.append(p.data.numpy())
        self.network_params = network_params

    def get_difference(self, network):
        total_diff = 0.0
        for i, p in enumerate(network.parameters()):
            p_np = p.data.numpy()
            diff = self.network_params[i] - p_np
            # print(diff)
            scalar_diff = np.sum(diff ** 2)
            total_diff += scalar_diff
        return total_diff

Will this work? I keep track of total_diff through time, and am logging it, but it seems to be zero ALWAYS. Even though the model's performance is improving, which confuses me greatly.


title: How to Multi-Head learning

body:
I have about 5 models that work pretty well trained individually but I want to fuse them together in order to have one big model.
I'm looking into it because one big model is more easy to update (in production) than many small model
this is an image of what I want to achieve.

my question are, is it ok to do it like this ?
having one dataset per head model, how am I supposed to train the whole model ?


title: calculating loss and perplexity when evaluating GPT2 model even when not defined

body:
When I'm trying to evaluate GPT2 model for text generation task, I printed loss and perplexity as given below in the code, but it is not defined in the code by me.

with torch.no_grad():

    for _ in range(length):
        outputs = model(generated)
        next_token_logits = outputs[0][:, -1, :]
        next_token = torch.multinomial(F.softmax(next_token_logits, dim=-1), num_samples=1)

        generated = torch.cat((generated, next_token), dim=1)

print(f"loss {loss}")
print(f"perplexity {perplexity}")

...

part from the output:

loss 3.644557237625122
perplexity 38.26582717895508

This prints out very realistic values for loss and perplexity and I'm not sure from where this is coming from, because if I change it to loss1 and perplexity1 it would give me this error

     38
---> 39 print(f"loss {loss1}")
     40 print(f"perplexity {perplexity}")
     41
NameError: name 'loss1' is not defined

Although here, for every generation, the loss and the perplexity will be the same.

So, I'm so confused as to how this is calculated automatically and would like some clarification since I'm very new to Pytorch and ML. Any help is appreciated.


title: Where do the input flatten value come from in fully connected 1 (fc1) layer (MNIST Example)

body:
here's some of the convolutional neural network sample code from Pytorch's examples directory on their github:
https://github.com/pytorch/examples/blob/master/mnist/main.py
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 32, 3, 1)
        self.conv2 = nn.Conv2d(32, 64, 3, 1)
        self.dropout1 = nn.Dropout2d(0.25)
        self.dropout2 = nn.Dropout2d(0.5)
        self.fc1 = nn.Linear(9216, 128)
        self.fc2 = nn.Linear(128, 10)

If I understand this, we need to flatten the output from the last convolutional layer before we can pass it through a linear layer (fc1). So, looking at this code, we see the input to the first fully connected layer is: 9216.
Where has this number (9216) come from?


title: PyTorch differentiable mask

body:
How would I go about blacking out a portion of an image or feature map such that AutoGrad can backprop through the operation?
Specifically I want to black out everything except for n layers of border pixels. So if we consider a single channel of the feature map which looks like:
[
  [1, 1, 1, 1],
  [1, 1, 1, 1],
  [1, 1, 1, 1],
  [1, 1, 1, 1],
]

I set a constant n=1 so my operation does the following to the input:
[
  [1, 1, 1, 1],
  [1, 0, 0, 1],
  [1, 0, 0, 1],
  [1, 1, 1, 1],
]

In my case I'd be doing it to a multi channel feature map and all channels would be treated the same way.
If possible, I want to do it in a functional manner.


title: How can one get state of the art results on Cifar10 with the All Convolution Net?

body:
I was trying to replicate the results of Striving for Simplicity: The All Convolutional Net  but I seem unable to. I've copied all the hyperparams from the implementation provided by StefOe and I'm not sure what to do. I get ~0.2 error instead of ~0.1. Has anyone been able to replicate this result on Pytorch?

The training looks fine, so I'm not sure what is going wrong:

Other related links:

https://discuss.pytorch.org/t/pytorch-net-from-striving-for-simplicity-the-all-convolutional-net/19297/2

Code:

https://github.com/StefOe/all-conv-pytorch/blob/master/allconv.py

https://github.com/StefOe/all-conv-pytorch/blob/master/cifar10.ipynb


title: In PyTorch, how to convert the cuda() related codes into CPU version?

body:
I have some existing PyTorch codes with cuda() as below, while net is a MainModel.KitModel object:

net = torch.load(model_path)
net.cuda()

and

im = cv2.imread(image_path)
im = Variable(torch.from_numpy(im).unsqueeze(0).float().cuda())

I want to test the code in a machine without CPU, so I want to convert the cuda-code into CPU version. I tried to look at some relevant posts regarding the CPU/GPU switch of PyTorch, but they are related to the usage of device and thus doesn't apply to my case.


title: Test predictions to class in fast.ai?

body:
After using get_preds I get a list of tensors. Something like:

[
    [0.9, 0.1],
    [0.85, 0.15],
    [0.92, 0.08],
    ...
]

And my classes are [0,1]. How should I translate these tensors to the corresponding (the most likely) class? Please see my current approach below

probs = learn.get_preds(ds_type=DatasetType.Test)[0]

def probs2class(item):
    return max(range(len(item)), key=item.__getitem__)

print(map(probs2class, probs))

I searched the documentation like crazy but maybe for wrong terms? What is the general way to go from probs to class predictions?


title: Reinforcement Learning with Pytorch. [Error: KeyError ]

body:
I am new to reinforcement learning as well as pytorch. I am learning it from Udemy. However, the code I have is the same as it is shown but I am having an error. I guess it is a pytorch error but can't debug it. If anyone helps it is really appreciated.

import gym
import time
import torch
import matplotlib.pyplot as plt
from gym.envs.registration import register
register(id='FrozenLakeNotSlippery-v0',entry_point='gym.envs.toy_text:FrozenLakeEnv',kwargs={'map_name': '4x4', 'is_slippery':False})
env = gym.make('FrozenLakeNotSlippery-v0')
number_of_states = env.observation_space.n
number_of_actions = env.action_space.n
Q = torch.zeros([number_of_states,number_of_actions])
num_episodes = 1000
steps_total = []
gamma = 1
for i in range(num_episodes):
    state = env.reset()
    step = 0
    while True:
        step += 1
        #action = env.action_space.sample()
        random_values = Q[state]+torch.rand(1,number_of_actions)/1000
        action = torch.max(random_values,1)[1][0]
        new_state, reward, done, info = env.step(action)
        Q[state, action] = reward + gamma * torch.max(Q[new_state])
        state = new_state
        #time.sleep(0.4)
        #env.render()
        if done:
            steps_total.append(step)
            print ("Episode Finished after %i steps" %step)
            break

print ("Average Num Steps: %2f" %(sum(steps_total)/num_episodes))
plt.plot(steps_total)
plt.show()

The error I am having is the below one

KeyError                                  Traceback (most recent call last)
<ipython-input-11-a6aa419c3767> in <module>
  8         random_values = Q[state]+torch.rand(1,number_of_actions)/1000
  9         action = torch.max(random_values,1)[1][0]
---> 10         new_state, reward, done, info = env.step(action)
 11         Q[state, action] = reward + gamma * torch.max(Q[new_state])
 12         state = new_state

c:\users\souradip\appdata\local\programs\python\python36\lib\site-packages\gym\envs\toy_text\discrete.py in step(self, a)
 53
 54     def step(self, a):
---> 55         transitions = self.P[self.s][a]
 56         i = categorical_sample([t[0] for t in transitions], self.np_random)
 57         p, s, r, d= transitions[i]

KeyError: tensor(3)


title: How to upgrade to pytorch-nightly in google colab?

body:
I got into problem ,
how can i solve this?
I want to run pytorch-nightly on colab, I have all codes in pytorch-nightly version because of new packages in it,
I tried to search about this and tried this code but it is not working even after restarting runtime
from os.path import exists
cuda_output = !ldconfig -p|grep cudart.so|sed -e 's/.*\.\([0-9]*\)\.\([0-9]*\)$/cu\1\2/'
accelerator = cuda_output[0] if exists('/dev/nvidia0') else 'cpu'
!pip install torchvision_nightly
!pip install torch_nightly -f https://download.pytorch.org/whl/nightly/{accelerator}/torch_nightly.html

Help!!


title: 'ValueError: optimizing a parameter that doesn't require gradients' do not understand the error

body:
as a part of an online python course, I am developing a an AI program that can identify flowers. After my first submission, I was told that I forgot to freeze the parameters after loading the model so the aren't modified during backprop. To solve this, in the block where I load my model I did this:

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")model = models.vgg16(pretrained = True)
model
for param in model.parameters():
    param.requires_grad = False

Then, when running the block where I define a new feedforward (code block below, I get the error   'ValueError: optimizing a parameter that doesn't require gradients'

features = list(model.classifier.children())[:-1]

num_filters = model.classifier[len(features)].in_features
features.extend([
    nn.Linear(25088, 2048),
    nn.ReLU(True),
    nn.Linear(2048, 512),
    nn.ReLU(True),
    nn.Linear(512, 102),
    nn.LogSoftmax(dim=1)
])

model.classifier = nn.Sequential(*features)
criterion = nn.NLLLoss()
model = model.to('cuda')
optimizer = optim.Adam(model.classifier.parameters(), lr = 0.001)
model

I'd be great if anyone could help me with this.


title: Questions about Batch Normalization in Pytorch

body:
Recently when I use the BN in the PyTorch, I have several questions.

Based on the BN2d documentation in PyTorch, when inferencing(evaluation), it will automatically use the mean and variance (running estimate when training) for BN layer. However, my first question is that when we save out the model after training, does it contain the running mean and variance? I originally thought the model will only save the learnable parameters. But the running mean and variance are not really learnable?

By default, when we use eval() in PyTorch, the BN layer will use some running mean and variance from a specific variable (maybe the results from training), right? But not to calculate the mean and variance of mini-batch? (Because I see some answers mention that the bad performance of BN when inferencing is cased by the 1-size batch. But it should use the running mean and variance of training, why the batch size of testing will influence?
The third question is kind of related to the second question. Will the BN performs the same when inferencing. For example, when I iterate the test set twice. Will the results behave differently? (The maybe more direct question is that will the mean and variance change when inferencing?)
I also want to know if it is possible to use mini-batch mean & variance or running mean & variance for inferencing. If I only set BN layer to train, its learnable parameter will also update. But maybe I just want them to get new mean and variance. Is there a possible way to do that?
The last question is related to the fourth question. So is it fair to use mean & variance of all test data, or batch of test data to calculate the mean & variance? I mean fair here is that is it improper (tricky?) to use some attributes of the test set?

Look forward to your answers. I'm a kind of new bird, willing to learn and discuss with you!!
Thanks in advance!!!


title: Titan XP vs Quadro P400 GPU in Pytorch

body:
I gave the the two GPUs on my machine a try and I expected the Titan-XP to be faster than the Quadro-P400. However, both gave almost the same execution time.

I need to know if PyTorch will dynamically choose one GPU over another, or, I myself will have to specify which one PyTorch will use, during run-time.

Here is the code snippet used in the test:

import torch
import time

def do_something(gpu_device):
    torch.cuda.set_device(gpu_device)  # torch.cuda.set_device(device_num)
    print("current GPU device ", torch.cuda.current_device())
    strt = time.time()
    a = torch.randn(100000000).cuda()
    xx = time.time() - strt
    print("execution time, to create 1E8 random numbers, is ", xx)
    # print(a)
    # print(a + 2)

no_of_GPUs= torch.cuda.device_count()
print("how many GPUs are there:", no_of_GPUs)
for i  in range(0, no_of_GPUs):
    print(i, "th GPU is", torch.cuda.get_device_name(i))
    do_something(i)

Sample output:

how many GPUs are there: 2
0 th GPU is TITAN Xp COLLECTORS EDITION
current GPU device  0
execution time, to create 1E8 random numbers, is  5.527713775634766

1 th GPU is Quadro P400
current GPU device  1
execution time, to create 1E8 random numbers, is  5.511776685714722


title: extracting frame features for videos using I3D pytorch implementation

body: I have a large datasets of videos related to human activities. I have converted the dataset to RGB frames. I want to generate features for these frames from the I3D pytorch architecture. I don't have the flow frames as of now, is it possible to extract features without the flow. Also if anyone can please help me with the process to extract features with I3D. Thank you very much.


title: How to use torch to speed up some common computations?

body:
I am trying make some common computations, like matrix multiplication, but without gradient computation. An example of my computation is like

import numpy as np
from scipy.special import logsumexp

var = 1e-8
a = np.random.randint(0,10,(128,20))
result = np.logsumexp(a, axis=1) / 2. + np.log(np.pi * var)

I want to use torch (gpu) to speed up the computation. Here is the code

import numpy as np
import torch

var = 1e-8
a = np.random.randint(0,10,(128,20))
a = torch.numpy_from(a).cuda()

result = torch.logsumexp(a, dim=1)/ 2. + np.log(np.pi*var)

but i have some questions:

Could the above code speed up the computation? I don't know if it works.
Do I need to convert all values into torch.tensor, like from var to torch.tensor(var).cuda() and from np.log(np.pi*var) to a torch.tensor?
Do I need to convert all tensors into gpu by myself, especially for some intermediate variable?
If the above code doesn't work, how can I speed up the computation with gpu?


title: pytorch sort/median on GPU as slow as numpy on CPU. Am I missing something?

body:
I need to compute the median array out of a list of arrays. Here is the timing results of a short exercise. I have an NVIDIA Quadro P2000 GPU, and have verified with FFTs and convolutions that cuda/torch runs 100's of times faster than numpy. However, when it comes to sorting, the GPU is fairly "stuck" and isn't giving me much meaningful acceleration.

import numpy as np
import torch as th
c=np.random.rand(3,480,700).astype(np.float32) # cpu version
g=th.tensor(c).to(device='cuda:0')             # gpu version
%timeit np.median(c,axis=0)        # 14.3 ms ± 61.6 µs per loop
%timeit np.sort(c,axis=0)[1]       # 9.59 ms ± 26.5 µs per loop
%timeit th.median(g,dim=0).values  # 11.8 ms ± 9.13 µs per loop
%timeit th.sort(g,dim=0).values[1] # 9.28 ms ± 6.01 µs per loop

I have verified with nvidia-smi that the GPU is running during the torch calls, fwiw.

There are plenty of reports (like this) that show torch's sorting algorithm beating the pants off of numpy, but I'm not getting it to happen. Anyone know whether there is a simple trick to make the torch sort/median on the GPU work faster?


title: Logging with Distributed Data Parallel with PyTorch

body:
I am trying to setup a training workflow with PyTorch DistributedDataParallel (DDP). Generally when I train I pass a logger through to track outputs and record useful information. However, I am having trouble using the logger I have with the DDP method. Right now my code is as follows:

import torch
import torch.multiprocessing as mp

class BaseModel:
    def __init__(self, *args, **kwargs):
        ...
        "does things"

    def fit(self, *args, **kwargs):
        ...
        'set up stuff'
        mp.spawn(self.distributed_training, nprocs=self.num_gpus, args=(self.params, training_input, self.logger))

    def distributed_training(params, training_input, logger):
        ...
        for e in epochs:
            'trains for an epoch'
            logger.info(print_line)

I know I am supposed to use the QueueHandler and QueueListener tools from logging with the import, but I have been scouring the internet and still do not have a clear understanding as to how. Any help would be greatly appreciated.


title: how initialize a torch of matrices

body:
Hello I m trying to create a tensor that will have inside N matrices of n by n size. I tried to initialize it with
Q=torch.zeros(N, (n,n))

but i get the following error
zeros(): argument 'size' must be tuple of ints, but found element of type tuple at pos 2

Also I want to fill it later with random matrices with integer values and I will turn it to semidefinte so I thought of the following
for i in range(0,N):
   Q[i]=torch.randint(0,10,(n,n))
Q = Q*Q.t()

Is it correct? Is there any other faster way with a build in command?


title: element-wise operation in pytorch

body:
I have two Tensors A and B, A.shape is (b,c,100,100), B.shape is (b,c,80,80),
how can I get tensor C with shape (b,c,21,21) subject to
C[:, :, i, j] = torch.mean(A[:, :, i:i+80, j:j+80] - B)?
I wonder whether there's an efficient way to solve this?
Thanks very much.


title: RNN - RuntimeError: input must have 3 dimensions, got 2

body:
I’m getting the following error:

RuntimeError: input must have 3 dimensions, got 2

I have a single feature column that I am trying to feed into a GRU neural net.
Below are my data loader and neural net. I have also included the output of my data loader when I retrieve a batch of data.
What am I doing wrong???
def batch_data(feature1, sequence_length, batch_size):
“”"
Batch the neural network data using DataLoader
:param feature1: the single feature column
:param sequence_length: The sequence length of each batch
:param batch_size: The size of each batch; the number of sequences in a batch
:return: DataLoader with batched data
“”"
    # total number of batches we can make
    n_batches = len(feature1)//batch_size

    # Keep only enough characters to make full batches
    feature1= feature1[:n_batches * batch_size]

    y_len = len(feature1) - sequence_length

    x, y = [], []
    for idx in range(0, y_len):
        idx_end = sequence_length + idx
        x_batch = feature1[idx:idx_end]
        x.append(x_batch)
        # only making predictions after the last item in the batch
        batch_y = feature1[idx_end]
        y.append(batch_y)

    # create tensor datasets
    data = TensorDataset(torch.from_numpy(np.asarray(x)), torch.from_numpy(np.asarray(y)))

    data_loader = DataLoader(data, shuffle=False, batch_size=batch_size)

    # return a dataloader
    return data_loader

# test dataloader on subset of actual data

test_text = data_subset_b
t_loader = batch_data(test_text, sequence_length=5, batch_size=10)

data_iter = iter(t_loader)
sample_x, sample_y = data_iter.next()

print(sample_x.shape)
print(sample_x)
print()
print(sample_y.shape)
print(sample_y)

When I pass in data, the following batch is generated…
torch.Size([10, 5])
tensor([[ 0.0045, 0.0040, -0.0008, 0.0005, -0.0012],
[ 0.0040, -0.0008, 0.0005, -0.0012, 0.0000],
[-0.0008, 0.0005, -0.0012, 0.0000, -0.0015],
[ 0.0005, -0.0012, 0.0000, -0.0015, 0.0008],
[-0.0012, 0.0000, -0.0015, 0.0008, 0.0000],
[ 0.0000, -0.0015, 0.0008, 0.0000, 0.0000],
[-0.0015, 0.0008, 0.0000, 0.0000, -0.0008],
[ 0.0008, 0.0000, 0.0000, -0.0008, -0.0039],
[ 0.0000, 0.0000, -0.0008, -0.0039, -0.0026],
[ 0.0000, -0.0008, -0.0039, -0.0026, -0.0082]], dtype=torch.float64)

torch.Size([10])
tensor([ 0.0000, -0.0015, 0.0008, 0.0000, 0.0000, -0.0008, -0.0039, -0.0026,
-0.0082, 0.0078], dtype=torch.float64)

Here is my neural net architecture:
class GRU(nn.Module):
def __init__(self, input_dim, output_dim, n_hidden, n_layers, dropout=0.25, lr=0.001):
    super().__init__()

    # set class variables
    self.output_dim = output_dim
    self.n_layers = n_layers
    self.n_hidden = n_hidden

    # define model layers

    # define gru layer
    self.gru = nn.GRU(input_dim, n_hidden, n_layers, dropout=dropout, batch_first=True)

    # define a dropout layer
    self.dropout = nn.Dropout(dropout)

    # define the final, fully-connected output layer
    self.fc = nn.Linear(n_hidden, output_dim)

def forward(self, nn_input, hidden):
    ''' Forward pass through the network.
        These inputs are x, and the hidden/cell state `hidden`. '''

    # batch_size equals the input's first dimension
    batch_size = nn_input.size(0)

    # Get the outputs and the new hidden state from the lstm
    r_output, hidden = self.gru(nn_input, hidden)

    # pass through a dropout layer
    out = self.dropout(r_output)

    # Stack up LSTM outputs using view
    # you may need to use contiguous to reshape the output
    out = out.contiguous().view(-1, self.n_hidden)

    # put nn_input through the fully-connected layer
    out = self.fc(out)

    # reshape into (batch_size, seq_length, output_size)
    out = out.view(batch_size, -1, self.output_dim)

    # get last batch
    out = out[:, -1]

    # return the final output and the hidden state
    return out, hidden

def init_hidden(self, batch_size):
    ''' Initializes hidden state '''
    # Create two new tensors with sizes n_layers x batch_size x n_hidden,
    # initialized to zero, for hidden state and cell state of LSTM
    weight = next(self.parameters()).data

    if (train_on_gpu):
        hidden = (weight.new(self.n_layers, batch_size, self.n_hidden).zero_().cuda(),
                  weight.new(self.n_layers, batch_size, self.n_hidden).zero_().cuda())
    else:
        hidden = (weight.new(self.n_layers, batch_size, self.n_hidden).zero_(),
                  weight.new(self.n_layers, batch_size, self.n_hidden).zero_())

    return hidden


title: Batches of points with the same label on Pytorch

body:
I want to train a neural network using gradient descent on batches that contain N training points each. I would like these batches to only contain points with the same label, instead of being randomly sampled from the training set.

For example, if I'm training using MNIST, I would like to have batches that look like the following:

batch_1 = {0,0,0,0,0,0,0,0}

batch_2 = {3,3,3,3,3,3,3,3}

batch_3 = {7,7,7,7,7,7,7,7}

.....

and so on.

How can I do it using pytorch?


title: PyTorch non-deterministic dropout

body:
I'm trying to make output of BLSTM deterministic, after investigation its appeared that my dropout layer creates not deterministic dropout masks, so I was researching about how to fix random seed in pytorch. I found this page and other suggestions though I put everything in code it did not help. Here is my code:

import sys
import random
import datetime as dt

import numpy as np
import torch

torch.manual_seed(42)
torch.cuda.manual_seed(42)
np.random.seed(42)
random.seed(42)
torch.backends.cudnn.deterministic = True

ex = torch.ones(10)
torch.nn.functional.dropout(ex, p=0.5, training=True)
# Out[29]: tensor([0., 0., 2., 0., 0., 0., 0., 0., 2., 2.])

torch.nn.functional.dropout(ex, p=0.5, training=True)
# Out[30]: tensor([0., 2., 0., 2., 2., 0., 0., 2., 2., 2.])

Please help me get deterministic output from dropout for the same input