title: pytorch 4d numpy array applying transfroms inside custom dataset

body:
Inside my custom dataset, I want to apply transforms.Compose() to a NumPy array.

My images are in a NumPy array format with shape (num_samples, width, height, channels).

How can I apply the follwoing transforms to the full numpy array?

img_transform = transforms.Compose([
        transforms.Scale((224,224)),
        transforms.ToTensor(),
        transforms.Normalize([0.46, 0.48, 0.51], [0.32, 0.32, 0.32])
    ])

My attempts are ending in multiple errors as the transforms accept a PIL image not a 4-d NumPy array.

from torchvision import transforms
import numpy as np
import torch

img_transform = transforms.Compose([
        transforms.Scale((224,224)),
        transforms.ToTensor(),
        transforms.Normalize([0.46, 0.48, 0.51], [0.32, 0.32, 0.32])
    ])

a = np.random.randint(0,256, (299,299,3))
print(a.shape)

img_transform(a)


title: Change expectation to multi dimensional rather than 1D in nn.crossentropyloss

body:
My goal is to do multi class image classification in Pytorch using the EMNIST dataset.
As a loss function, I would like to use Multi-Class Cross-Entropy Loss.
Currently, I define my loss function as follows:
criterion = nn.CrossEntropyLoss()

I train my model as follows:
iter = 0
for epoch in range(num_epochs):
    for i, (images, labels) in enumerate(train_loader):

        # Add a single channel dimension
        # From: [batch_size, height, width]
        # To: [batch_size, 1, height, width]
        images = images.unsqueeze(1)

        # Forward pass to get output/logits
        outputs = model(images)

        # Clear gradients w.r.t. parameters
        optimizer.zero_grad()

        # Forward pass to get output/logits
        outputs = model(images)

        # Calculate Loss: softmax --> cross entropy loss
        loss = criterion(outputs, labels)

        # Getting gradients w.r.t. parameters
        loss.backward()

        # Updating parameters
        optimizer.step()

        iter += 1

        if iter % 500 == 0:
            # Calculate Accuracy
            correct = 0
            total = 0
            # Iterate through test dataset
            for images, labels in test_loader:

                images = images.unsqueeze(1)

                # Forward pass only to get logits/output
                outputs = model(images)

                # Get predictions from the maximum value
                _, predicted = torch.max(outputs.data, 1)

                # Total number of labels
                total += labels.size(0)

                correct += (predicted == labels).sum()

            accuracy = 100 * correct / total

            # Print Loss
            print('Iteration: {}. Loss: {}. Accuracy: {}'.format(iter, loss.data[0], accuracy))

However, the error that I get is:
RuntimeError                              Traceback (most recent call last)
<ipython-input-15-c26c43bbc32e> in <module>()
     21
     22         # Calculate Loss: softmax --> cross entropy loss
---> 23         loss = criterion(outputs, labels)
     24
     25         # Getting gradients w.r.t. parameters

3 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)
   2113                          .format(input.size(0), target.size(0)))
   2114     if dim == 2:
-> 2115         ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
   2116     elif dim == 4:
   2117         ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)

RuntimeError: 1D target tensor expected, multi-target not supported

My CNN outputs 26 variables and my target variables are also 26D.
How can I change my code so that nn.crossentropyloss() expects a 26D input rather than 1D?


title: TypeError: forward() missing 1 required positional argument: 'target' when using AdaptiveLogSoftmaxWithLoss

body:
I am trying to build a next word prediction model with pytorch in google colab. As my vocabulary size is over 1.5 million, I am using AdaptiveLogSoftmaxWithLoss module of pytorch to reduce RAM consumption.
The simple BiLSTM model definition is as follows:
class BLSTM(nn.Module):
    def __init__(self, emb_size, hidden_size, num_layers, vocab_size, cutoffs):
        super(BLSTM, self).__init__()
        self.hidden_size = hidden_size
        self.num_layers = num_layers
        self.blstm = nn.LSTM(emb_size, hidden_size, num_layers, batch_first=True, bidirectional=True)
        self.fc = nn.AdaptiveLogSoftmaxWithLoss(hidden_size*2, vocab_size, cutoffs)
        # self.fc = nn.Linear(hidden_size*2, vocab_size)

    def forward(self, x):
        # Set initial states
        h0 = torch.zeros(self.num_layers*2, x.size(0), self.hidden_size)
        c0 = torch.zeros(self.num_layers*2, x.size(0), self.hidden_size)

        # Forward propagate LSTM
        embed = nn.Embedding(vocab_size, emb_size)
        out, _ = self.blstm(embed(x), (h0, c0))  # out: tensor of shape (batch_size, seq_length, hidden_size*2)

        # Decode the hidden state of the last time step
        out = self.fc(out[:, -1, :])

        return out

The model and loss function are called as follows:
model = BLSTM(emb_size, hidden_size, num_layers, vocab_size, cutoffs)

# Loss and optimizer
criterion = nn.NLLLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

Inside the epoch the loss is calculated as follows:
        inputs = x[0].to(device)
        targets = x[1].to(device)

        # Forward pass
        outputs = model(inputs)
        outputs = outputs.to(device)
        loss = criterion(outputs, targets)
        print(loss.item())

        #Backward and optimize
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

This is the complete error trace I am getting after running the epoch loop:
TypeError                                 Traceback (most recent call last)
<ipython-input-33-51e15380f8c7> in <module>()
      8
      9         # Forward pass
---> 10         outputs = model(inputs)
     11         outputs = outputs.to(device)
     12         loss = criterion(outputs, targets)

2 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--> 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

TypeError: forward() missing 1 required positional argument: 'target'

I tried the same code with a simple nn.Linear() layer and the code runs fine. But when I replace the Linear layer with AdaptiveLogSoftmaxWithLoss, I get the above mentioned error.


title: Error in transformation of EMNIST data through Pytorch

body:
I was trying to train my model for prediction of EMNIST by using Pytorch.
Edit:- Here's the link of colab notebook for the problem.
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(28, 64, (5, 5), padding=2)
        self.conv1_bn = nn.BatchNorm2d(64)
        self.conv2 = nn.Conv2d(64, 128, 2, padding=2)
        self.fc1 = nn.Linear(2048, 1024)
        self.dropout = nn.Dropout(0.3)
        self.fc2 = nn.Linear(1024, 512)
        self.bn = nn.BatchNorm1d(1)
        self.fc3 = nn.Linear(512, 128)
        self.fc4 = nn.Linear(128, 47)
    def forward(self, x):
        x = F.relu(self.conv1(x))
        x = F.max_pool2d(x, 2, 2)
        x = self.conv1_bn(x)
        x = F.relu(self.conv2(x))
        x = F.max_pool2d(x, 2, 2)
        x = x.view(-1, 2048)
        x = F.relu(self.fc1(x))
        x = self.dropout(x)
        x = self.fc2(x)
        x = x.view(-1, 1, 512)
        x = self.bn(x)
        x = x.view(-1, 512)
        x = self.fc3(x)
        x = self.fc4(x)
        return F.log_softmax(x, dim=1)
        return x

I am getting this  type of error as shown below, whenever I am training my model.
<ipython-input-11-07c68cf1cac2> in forward(self, x)
     24     def forward(self, x):
     25         x = F.relu(self.conv1(x))
---> 26         x = F.max_pool2d(x, 2, 2)
     27         x = self.conv1_bn(x)
RuntimeError: Given input size: (64x28x1). Calculated output size: (64x14x0). Output size is too small

I tried to searched for the solutions and found that I should transform the data before. So i tried transforming it by the most common suggestion:-
transform_valid = transforms.Compose(
    [
     transforms.ToTensor(),
    ])

But then again I am getting the error mentioned below. Maybe the problem lies here in the transformation part.
/opt/conda/lib/python3.7/site-packages/torchvision/datasets/mnist.py:469: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at  /opt/conda/conda-bld/pytorch_1595629403081/work/torch/csrc/utils/tensor_numpy.cpp:141.)
  return torch.from_numpy(parsed.astype(m[2], copy=False)).view(*s)

I wanted to make that particular numpy array writable by using "ndarray.setflags(write=None, align=None, uic=None)" but I'm not able to figure out from where and what type of array should I make writable, as I'm directly loading the dataset using ->
"datasets.EMNIST(root, split="balanced", train=False, download=True, transform=transform_valid)"


title: Tensor sizes CrossEntropyLoss

body:
In the snippet:

criterion = nn.CrossEntropyLoss()
raw_loss = criterion(output.view(-1, ntokens), targets)

output size is torch.Size([5, 5, 8967]), targets size is torch.Size([25]), and ntokens is 8967

After modifying the code, my

output size is torch.Size([5, 8967]) and targets size is torch.Size([25])

which rises dimensionality issues when computing the loss.

Is it sensible to increase the size of my Linear activation that produces the output by 5, so that I can resize the output later to be of the size torch.Size([5, 5, 8967])?

The problem with increasing the size of the tensor is that ntokens can become quite large and I can easily run out of memory because of that. Is there an alternative approach?


title: How to generate burst of images from a single image by adding misalignment?

body: I'm learning about image denoising and Pytorch.I want to get burst of images generated from a single image. For example, I have an image, then random crop a patch of specific size from it. Then I want to add an 1 or 2 pixels shift on it to get a new image with tiny difference. What could I do? Is it better to use some techniques in PIL or others?


title: Deep learning with Pytorch: understanding the neural network example

body:
I'm reading the Pytorch documentation and I have a couple of questions about the neural network that is introduced. The documentation defines the following network:

import torch
import torch.nn as nn
import torch.nn.functional as F

    class Net(nn.Module):

        def __init__(self):
            super(Net, self).__init__()
            # 1 input image channel, 6 output channels, 3x3 square convolution
            # kernel
            self.conv1 = nn.Conv2d(1, 6, 3)
            self.conv2 = nn.Conv2d(6, 16, 3)
            # an affine operation: y = Wx + b
            self.fc1 = nn.Linear(16 * 6 * 6, 120)  # 6*6 from image dimension
            self.fc2 = nn.Linear(120, 84)
            self.fc3 = nn.Linear(84, 10)

        def forward(self, x):
            # Max pooling over a (2, 2) window
            x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
            # If the size is a square you can only specify a single number
            x = F.max_pool2d(F.relu(self.conv2(x)), 2)
            x = x.view(-1, self.num_flat_features(x))
            x = F.relu(self.fc1(x))
            x = F.relu(self.fc2(x))
            x = self.fc3(x)
            return x

        def num_flat_features(self, x):
            size = x.size()[1:]  # all dimensions except the batch dimension
            num_features = 1
            for s in size:
                num_features *= s
            return num_features

Later on, the following statement is made:

Let try a random 32x32 input. Note: expected input size of this net (LeNet) is 32x32. To use this net on MNIST dataset, please resize the images from the dataset to 32x32.

Question 1: Why do the images need to be 32x32 (where I assume that this means 32 pixels by 32)?

The first convolution applies six kernel to an image, with every kernel being 3x3. This means that if the input channel is 32x32, the six output channels all have dimensions 30x30 (the 3x3 kernel grid makes you lose 2 pixels in width and height). The second convolution applies more kernels so that there now are sixteen output channels of dimensions 28x28 (again the 3x3 kernel grid makes you lose 2 pixels in width and height). Now I would expect 16x28x28 nodes in the next layer, since every one of the sixteen output channels has 28x28 pixels. Somehow, this is incorrect, and the next layer contains 16x6x6 nodes. Why is this true?

Question 2: The second convolution layer goes from six input channels to sixteen output channels. How is this done?

In the first convolution layer we go from one input channel to six input channels, which makes sense to me. You can just apply six kernels to the single input channel to arrive at six output channels. Going from six input channels to sixteen output channels does not make as much sense to me. How are the different kernels applied? Do you apply two kernels to the first five input channels to arrive at ten output channels, and apply six kernels to the last input channel, so that the total comes to sixteen output channels? Or does the neural network learn itself to use x kernels and apply them to the input channels that it finds most suitable?


title: Numpy random.choice probablities don't sum up to 1

body:
I wanna randomly select sample points based on the probability distribution specified by prob for a given row. However, I get the error in np.random.choice that the probabilities don't add up to 1. This is very weird because I first normalize using the L1-norm along the rows and then I define a uniform distribution if the values are smaller than the threshold 1e-6.

import numpy as np
import torch.nn.functional as F

prob = F.normalize(outputs, p=1, dim=1).clone().data.cpu().numpy() # outputs is a torch.Tensor of shape (14, 6890)
all_zero = np.where(prob.max(1) < 1e-6)[0] # find indices of rows where all values are smaller
prob[all_zero] = np.full(prob.shape[1], 1 / prob.shape[1]) # fill those rows uniformly
# ... somewhere later inside a method
for j in range(14):
    sample = np.random.choice(6890, 4, replace=False, p=prob[j])

Do you understand, why that is?


title: What does required_grad do in PyTorch? (Not requires_grad)

body:
I have been trying to carry out transfer learning on a multiclass classification task using resnet as my backbone.

In many tutorials, it was stated that it would be wise to try and train only the last layer (usually a fully connected layer) again, while freezing the other layers. The freezing would be done as so:

for param in model.parameters():
    param.requires_grad = False

However, I just realized that all of my layers were actually not freezed, and while checking on my code I realized I had made a typo:

for param in model.parameters():
    param.required_grad = False

In a way that I wrote required_grad instead of requires_grad.

I can't seem to find information on required_grad - what it is, nor what it does. The only thing I found out was that it did not change the requires_grad flag, and that there is a separate required_grad flag which is set to False instead.

Can anyone explain what required_grad does? Have I been 'not freezing' my other layers all this time?


title: Gradient accumulation in an RNN

body:
I ran into some memory issues (GPU) when running a large RNN network, but I want to keep my batch size reasonable so I wanted to try out gradient accumulation. In a network where you predict the output in one go, that seems self-evident but in an RNN you do multiple forward passes for each input step. Because of that, I fear that my implementation does not work as intended. I started from user albanD's excellent examples here , but I think they should be modified when using an RNN. The reason I think that is because you accumulate much more gradients because you do multiple forwards per sequence.
My current implementation looks like this, at the same time allowing for AMP in PyTorch 1.6 which seems important - everything needs to be called in the right place. Note that this is just an abstract version, which might seem like a lot of code but it is mostly comments.
def train(epochs):
    """Main training loop. Loops for `epoch` number of epochs. Calls `process`."""
    for epoch in range(1, epochs + 1):
        train_loss = process("train")
        valid_loss = process("valid")
        # ... check whether we improved over earlier epochs
        if lr_scheduler:
            lr_scheduler.step(valid_loss)

def process(do):
    """Do a single epoch run through the dataloader of the training or validation set.
       Also takes care of optimizing the model after every `gradient_accumulation_steps` steps.
       Calls `step` for each batch where it gets the loss from."""
    if do == "train":
        model.train()
        torch.set_grad_enabled(True)
    else:
        model.eval()
        torch.set_grad_enabled(False)

    loss = 0.
    for batch_idx, batch in enumerate(dataloaders[do]):
        step_loss, avg_step_loss = step(batch)
        loss += avg_step_loss

        if do == "train":
            if amp:
                scaler.scale(step_loss).backward()

                if (batch_idx + 1) % gradient_accumulation_steps == 0:
                    # Unscales the gradients of optimizer's assigned params in-place
                    scaler.unscale_(optimizer)
                    # clip in-place
                    clip_grad_norm_(model.parameters(), 2.0)
                    scaler.step(optimizer)
                    scaler.update()
                    model.zero_grad()
            else:
                step_loss.backward()
                if (batch_idx + 1) % gradient_accumulation_steps == 0:
                    clip_grad_norm_(model.parameters(), 2.0)
                    optimizer.step()
                    model.zero_grad()

        # return average loss
        return loss / len(dataloaders[do])

    def step():
        """Processes one step (one batch) by forwarding multiple times to get a final prediction for a given sequence."""
        # do stuff... init hidden state and first input etc.
        loss = torch.tensor([0.]).to(device)

        for i in range(target_len):
            with torch.cuda.amp.autocast(enabled=amp):
                # overwrite previous decoder_hidden
                output, decoder_hidden = model(decoder_input, decoder_hidden)

                # compute loss between predicted classes (bs x classes) and correct classes for _this word_
                item_loss = criterion(output, target_tensor[i])

                # We calculate the gradients for the average step so that when
                # we do take an optimizer.step, it takes into account the mean step_loss
                # across batches. So basically (A+B+C)/3 = A/3 + B/3 + C/3
                loss += (item_loss / gradient_accumulation_steps)

            topv, topi = output.topk(1)
            decoder_input = topi.detach()

        return loss, loss.item() / target_len

The above does not seem to work as I had hoped, i.e. it still runs into out-of-memory issues very quickly. I think the reason is that step already accumulates so much information, but I am not sure.


title: Training custom model

body:
I am trying to train my dataset on yolov5 I normalized data as discussed in the docs on github but I always end up with this error.
              from  n    params  module                                  arguments
  0             -1  1      8800  models.common.Focus                     [3, 80, 3]
  1             -1  1    115520  models.common.Conv                      [80, 160, 3, 2]
  2             -1  1    315680  models.common.BottleneckCSP             [160, 160, 4]
  3             -1  1    461440  models.common.Conv                      [160, 320, 3, 2]
  4             -1  1   3311680  models.common.BottleneckCSP             [320, 320, 12]
  5             -1  1   1844480  models.common.Conv                      [320, 640, 3, 2]
  6             -1  1  13228160  models.common.BottleneckCSP             [640, 640, 12]
  7             -1  1   7375360  models.common.Conv                      [640, 1280, 3, 2]
  8             -1  1   4099840  models.common.SPP                       [1280, 1280, [5, 9, 13]]
  9             -1  1  20087040  models.common.BottleneckCSP             [1280, 1280, 4, False]
 10             -1  1    820480  models.common.Conv                      [1280, 640, 1, 1]
 11             -1  1         0  torch.nn.modules.upsampling.Upsample    [None, 2, 'nearest']
 12        [-1, 6]  1         0  models.common.Concat                    [1]
 13             -1  1   5435520  models.common.BottleneckCSP             [1280, 640, 4, False]
 14             -1  1    205440  models.common.Conv                      [640, 320, 1, 1]
 15             -1  1         0  torch.nn.modules.upsampling.Upsample    [None, 2, 'nearest']
 16        [-1, 4]  1         0  models.common.Concat                    [1]
 17             -1  1   1360960  models.common.BottleneckCSP             [640, 320, 4, False]
 18             -1  1    922240  models.common.Conv                      [320, 320, 3, 2]
 19       [-1, 14]  1         0  models.common.Concat                    [1]
 20             -1  1   5025920  models.common.BottleneckCSP             [640, 640, 4, False]
 21             -1  1   3687680  models.common.Conv                      [640, 640, 3, 2]
 22       [-1, 10]  1         0  models.common.Concat                    [1]
 23             -1  1  20087040  models.common.BottleneckCSP             [1280, 1280, 4, False]
 24   [17, 20, 23]  1         0  models.yolo.Detect                      [3, [[10, 13, 16, 30, 33, 23], [30, 61, 62, 45, 59, 119], [116, 90, 156, 198, 373, 326]]]
Traceback (most recent call last):
  File "train.py", line 404, in <module>
    train(hyp)
  File "train.py", line 80, in train
    model = Model(opt.cfg).to(device)
  File "/content/yolov5/models/yolo.py", line 62, in __init__
    m.stride = torch.tensor([128 / x.shape[-2] for x in self.forward(torch.zeros(1, ch, 128, 128))])  # forward
  File "/content/yolov5/models/yolo.py", line 90, in forward
    return self.forward_once(x, profile)  # single-scale inference, train
  File "/content/yolov5/models/yolo.py", line 107, in forward_once
    x = m(x)  # run
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/content/yolov5/models/yolo.py", line 26, in forward
    x[i] = x[i].view(bs, self.na, self.no, ny, nx).permute(0, 1, 3, 4, 2).contiguous()
RuntimeError: shape '[1, 3, 8, 16, 16]' is invalid for input of size 81920

These are the flags used
!python train.py --img 1024 --batch 4 --epochs 30 \
  --data ./data/mask.yaml --cfg ./models/yolov5x.yaml --weights yolov5x.pt \
   --cache --name maskmodel

this is the file structure


title: How to convert tensorflow.js model weights to pytorch tensors, and back?

body:
I am using ml5.js, a wrapper around tensorflowjs. I want to train a neural network in the browser, download the weights, process them as tensors in pyTorch, and load them back into the browser's tensorflowjs model. How do I convert between these formats tfjs <-> pytorch?
The browser model has a save() function which generates three files. A metadata file specific to ml5.js (json), a topology file describing model architecture (json), and a binary weights file (bin).
// Browser
model.save()

// HTTP/Download
model_meta.json   (needed by ml5.js)
model.json        (needed by tfjs)
model.weights.bin (needed by tfjs)

# python backend
import json

with open('model.weights.bin', 'rb') as weights_file:
    with open('model.json', 'rb') as model_file:
        weights = weights_file.read()
        model = json.loads(model_file.read())
        ####
        pytorch_tensor = convert2tensor(weights, model) # whats in this function?
        ####
        # Do some processing in pytorch

        ####
        new_weights_bin = convert2bin(pytorch_tensor, model) # and in this?
        ####

Here is sample javascript code to generate and load the 3 files in the browser. To load, select all 3 files at once in the dialog box. If they are correct, a popup will show a sample prediction.


title: Best Way to Overcome Early Convergence for Machine Learning Model

body:
I have a machine learning model built that tries to predict weather data, and in this case I am doing a prediction on whether or not it will rain tomorrow (a binary prediction of Yes/No).

In the dataset there is about 50 input variables, and I have 65,000 entries in the dataset.

I am currently running a RNN with a single hidden layer, with 35 nodes in the hidden layer. I am using PyTorch's NLLLoss as my loss function, and Adaboost for the optimization function. I've tried many different learning rates, and 0.01 seems to be working fairly well.

After running for 150 epochs, I notice that I start to converge around .80 accuracy for my test data. However, I would wish for this to be even higher. However, it seems like the model is stuck oscillating around some sort of saddle or local minimum. (A graph of this is below)

What are the most effective ways to get out of this "valley" that the model seems to be stuck in?


title: Dealing with large slowdown when moving PyTorch code to GPU

body:
I have a Graph Neural Network model I have written using Pytorch. On my CPU I am not getting fantastic performance, so I tried to port it over to a V100 GPU I have access to. In this process, I have received a huge performance decrease (around 10times slower).

I have two ideas of where might be the issue, but I would like some input to try get the optimum performance from my model. The first problem might be coming from my custom graph convolutional layer:

class GraphConvLayer(torch.nn.Module):
    """
    Based, basically, on https://arxiv.org/abs/1609.02907
    Have some modifications:
    https://towardsdatascience.com/how-to-do-deep-learning-on-graphs-with-graph-convolutional-networks-7d2250723780
    This helped:
    https://pytorch.org/docs/master/notes/extending.html

    """
    def __init__(self, input_features, output_features, device, bias=True):
        super(GraphConvLayer, self).__init__()

        self.input_features = input_features
        self.output_features = output_features
        self.device = device

        self.weight = nn.Parameter(torch.FloatTensor(self.input_features, self.output_features))

        if bias:
            self.bias = nn.Parameter(torch.FloatTensor(self.output_features))
        else:
            self.register_parameter('bias', None)

        # Not a very smart way to initialize weights
        self.weight.data.uniform_(-0.1, 0.1)
        if bias is not None:
            self.bias.data.uniform_(-0.1, 0.1)

    def forward(self,input, adj):
        # Here, we put in the forward pass:
        # Our forward pass needs to be:
        # D^-1 * (A + 1) * X * weights
        input, adj = input.float(), adj.float()

        Identity = torch.eye( len(adj[0]), device = self.device)
        A_hat = adj + Identity

        D = torch.sum(A_hat, dim=0)
        len_D = len(D)
        zero = torch.zeros(len_D,len_D, device = self.device)
        mask = torch.diag(torch.ones_like(D, device = self.device))
        D = mask*torch.diag(D) + (1. - mask)*zero

        D_inv = torch.inverse(D)
        out = torch.mm(input, self.weight)
        out = torch.spmm(A_hat,out)
        out = torch.spmm(D_inv, out)

        if self.bias is not None:
            return out + self.bias
        else:
            return out

        return out

    def extra_repr(self):
        # (Optional)Set the extra information about this module. You can test
        # it by printing an object of this class.
        return 'node_features={}, length of weights={}, bias={}'.format(
            self.node_features, self.input_features, self.bias is not None
        )

Specifically, in the forward pass I am doing a selection of transformations that are described in the towardsdatascience link in the class. Is there something here that is causing this large slow-down? It seems to me that the tensors are all being initialised on the GPU.

Secondly, as all my graphs are different sizes I am being forced into using a batch size of 1. In my training loop I have this:

        for batch in tqdm(train_loader):
            opt.zero_grad()
            adjacency, features, _, nodes = batch
            adjacency = adjacency.to(device)
            features = features.to(device)
            nodes = nodes.to(device)

            output = model(features[0], adjacency[0])

            loss = F.nll_loss(output, nodes[0])
            loss.backward()
            opt.step()

This means (as I interpret it) that every single piece of data is being moved to the GPU individually, every loop. This seems like an obvious cause of inefficiency. Is there a way to move all the data into GPU memory at once, outside the training loop, allowing me to remove the adjacency = adjacency.to(device) lines?

Any help would be really appreciated.


title: PyTorch - WSD using LSTM

body:
I'm trying to replicate Google's research paper on WSD with neural models using PyTorch.

I'm having some issues traying to overfit the model before training on large datasets.

Using this training set:

  The film was also intended to be the first in a trilogy.

this model definition:

class WordGuesser(nn.Module):
    def __init__(self, hidden_dim, context_dim, embedding_dim, vocabulary_dim, batch_dim, window_dim):
        super(WordGuesser, self).__init__()
        self.hidden_dim = hidden_dim
        self.batch_dim = batch_dim
        self.window_dim = window_dim
        self.word_embeddings = nn.Embedding(vocabulary_dim, embedding_dim)
        self.lstm = nn.LSTM(embedding_dim, hidden_dim)
        #self.extract_context = nn.Linear((2 * window_dim + 1) * hidden_dim, context_dim)
        self.extract_context = nn.Linear(hidden_dim, context_dim)
        self.predict = nn.Linear(context_dim, vocabulary_dim)
        self.hidden = self.init_hidden()

    def init_hidden(self):
        return (autograd.Variable(torch.zeros(1, self.batch_dim, self.hidden_dim).cuda()),
                autograd.Variable(torch.zeros(1, self.batch_dim, self.hidden_dim).cuda()))

    def forward(self, sentence, hidden):
        embeddings = self.word_embeddings(sentence)
        out, self.hidden = self.lstm(embeddings.permute(1, 0, 2), hidden)
        lstm_out = out[-1]
        context = self.extract_context(lstm_out)
        prediction = self.predict(context)
        return prediction, context

and this training routine:

num_epoch = 100
hidden_units = 512
embedding_dim = 256
context_dim = 256

def mytrain():
    lines = open('training/overfit.txt').readlines()
    sentences = data.split_to_sentences(lines) #uses spaCy to detect sentences from each line
    word2idx=dict() #dictionary is built from the training set
    idx2word =dict()
    i = 0
    for s in sentences:
        for t in s.split(' '):
            if t in word2idx:
                continue
            word2idx[t] = i
            idx2word[i] = t
            i += 1
    word2idx['$'] = i #the token to guess the missing word in a sentence
    idx2word[i] = '$'
    X = list()
    Y = list()
    for sentence in sentences:
        sentence = sentence.split(' ')
        for i in range(len(sentence)):
            newsentence = list(sentence)
            newsentence[i] = '$'
            if not sentence[i] in word2idx:
                continue
            indices = [word2idx[w] for w in newsentence]
            label = word2idx[sentence[i]]
            X.append(indices)
            Y.append(label)
    model = WordGuesser(hidden_units, context_dim, embedding_dim, len(word2idx), len(X), len(X[0]))
    model.train()
    model.cuda()
    input = torch.LongTensor(X).cuda()
    output = torch.LongTensor(Y).cuda()
    criterion = nn.CrossEntropyLoss()
    optimizer = optim.Adam(model.parameters(), lr=0.01)
    model.hidden = model.init_hidden()
    for epoch in range(num_epoch):
        model.hidden = model.init_hidden()
        model.zero_grad()
        input_tensor = autograd.Variable(input)
        target_tensor = autograd.Variable(output)

        predictions, context = model(input_tensor, model.hidden)
        for i, prediction in enumerate(predictions):
            sorted_val = sorted(enumerate(np.array(prediction.data)), key=lambda x : x[1], reverse=True)
            print([(idx2word[x[0]], x[1]) for x in sorted_val[:5]], idx2word[Y[i]])
        loss = criterion(predictions, target_tensor)
        loss.backward()
        optimizer.step()

        print(epoch, loss.data[0])

    torch.save(model, "train2.pt")

during the training it seems that the model is able to overfit just after the 21st epoch, as you can see from the following scores (top 5 words from the predictions and the last word in a line is the label for that sentence):

  [('The', 11.362326), ('film', 11.356865), ('also', 7.5573149), ('to',
  5.3518314), ('intended', 4.3520432)] The

  [('film', 11.073805), ('The', 10.451499), ('also', 7.5498624), ('was',
  4.9684553), ('be', 4.0730805)] film

  [('was', 11.232123), ('also', 9.9741745), ('the', 6.0156212), ('be',
  4.9949703), ('The', 4.5516477)] was

  [('also', 9.6998224), ('was', 9.6202812), ('The', 6.345758), ('film',
  4.9122157), ('be', 2.6727715)] also

  [('intended', 18.344809), ('to', 16.410078), ('film', 10.147289),
  ('The', 9.8423424), ('$', 9.6181822)] intended

  [('to', 12.442947), ('intended', 10.900065), ('film', 8.2598763),
  ('The', 8.0493736), ('$', 4.4901967)] to

  [('be', 12.189278), ('also', 7.7172523), ('was', 7.5415096), ('the',
  5.2521734), ('The', 4.1723843)] be

  [('the', 15.59604), ('be', 9.3750105), ('first', 8.9820032), ('was',
  8.6859236), ('also', 5.0665498)] the

  [('first', 10.191225), ('the', 5.1829329), ('in', 3.6020348), ('be',
  3.4108081), ('a', 1.5569853)] first

  [('in', 14.731103), ('first', 9.3131113), ('a', 5.982264), ('trilogy',
  4.2928643), ('be', 0.49548936)] in

  [('a', 14.357709), ('in', 8.3088198), ('trilogy', 6.3918238),
  ('first', 6.2178354), ('intended', 0.95656234)] a

  [('trilogy', 14.351434), ('a', 4.5073452), ('in', 4.2348137), ('$',
  3.7552347), ('intended', 3.5101018)] trilogy

  [('.', 18.152126), ('$', 12.028764), ('to', 9.6003456), ('intended',
  8.1202478), ('The', 4.9225812)] .

When running another Python script which loads the model and queries it for the following words (using the same code to print out the scores during the training):

  The film was also intended to $ the first in a trilogy.   be

  The film $ also intended to be the first in a trilogy.    was

  $ film was also intended to be the first in a trilogy.    The

I'm getting these scores:

  [('film', 24.066889), ('$', 20.107487), ('was', 16.855488), ('a',
  12.969441), ('in', 8.1248817)] be

  [('film', 24.089062), ('$', 20.116539), ('was', 16.891994), ('a',
  12.982826), ('in', 8.1167336)] was

  [('film', 23.993624), ('$', 20.108011), ('was', 16.891005), ('a',
  12.960193), ('in', 8.1577587)] The

I've also tried setting to False the model.train() mode, using model.eval() as well as calling topk on the LSTM scores, but the results aren't satifying,


title: Pytorch (torchscript) error : attribute 'downsample' of type 'NoneType' is not usable in a script method (did you forget to add it __constants__?)

body:
I am converting pytorch code for AlignedReID++ to C++ code using torchscript. During the process I encountered the following error:

Traceback (most recent call last):
  File "selfDemo.py", line 28, in <module>
    traced_model = torch.jit.script(model)
  File "/nfs/engine/rajagopalar/anaconda3/envs/torchExp/lib/python3.6/site-packages/torch/jit/__init__.py", line 1162, in script
    return _convert_to_script_module(obj)
  File "/nfs/engine/rajagopalar/anaconda3/envs/torchExp/lib/python3.6/site-packages/torch/jit/__init__.py", line 1812, in _convert_to_script_module
    return WeakScriptModuleProxy(mod, stubs)
  File "/nfs/engine/rajagopalar/anaconda3/envs/torchExp/lib/python3.6/site-packages/torch/jit/__init__.py", line 1386, in init_then_register
    original_init(self, *args, **kwargs)
  File "/nfs/engine/rajagopalar/anaconda3/envs/torchExp/lib/python3.6/site-packages/torch/jit/__init__.py", line 1736, in __init__
    _create_methods_from_stubs(self, stubs)
  File "/nfs/engine/rajagopalar/anaconda3/envs/torchExp/lib/python3.6/site-packages/torch/jit/__init__.py", line 1347, in _create_methods_from_stubs
    self._c._create_methods(self, defs, rcbs, defaults)
  File "/nfs/engine/rajagopalar/anaconda3/envs/torchExp/lib/python3.6/site-packages/torch/jit/__init__.py", line 982, in _make_strong_submodule
    new_strong_submodule = _convert_to_script_module(module)
  File "/nfs/engine/rajagopalar/anaconda3/envs/torchExp/lib/python3.6/site-packages/torch/jit/__init__.py", line 1791, in _convert_to_script_module
    return _create_constant_iterable_module(mod)
  File "/nfs/engine/rajagopalar/anaconda3/envs/torchExp/lib/python3.6/site-packages/torch/jit/__init__.py", line 963, in _create_constant_iterable_module
    modules[key] = _create_constant_iterable_module(submodule)
  File "/nfs/engine/rajagopalar/anaconda3/envs/torchExp/lib/python3.6/site-packages/torch/jit/__init__.py", line 965, in _create_constant_iterable_module
    modules[key] = _convert_to_script_module(submodule)
  File "/nfs/engine/rajagopalar/anaconda3/envs/torchExp/lib/python3.6/site-packages/torch/jit/__init__.py", line 1812, in _convert_to_script_module
    return WeakScriptModuleProxy(mod, stubs)
  File "/nfs/engine/rajagopalar/anaconda3/envs/torchExp/lib/python3.6/site-packages/torch/jit/__init__.py", line 1386, in init_then_register
    original_init(self, *args, **kwargs)
  File "/nfs/engine/rajagopalar/anaconda3/envs/torchExp/lib/python3.6/site-packages/torch/jit/__init__.py", line 1736, in __init__
    _create_methods_from_stubs(self, stubs)
  File "/nfs/engine/rajagopalar/anaconda3/envs/torchExp/lib/python3.6/site-packages/torch/jit/__init__.py", line 1347, in _create_methods_from_stubs
    self._c._create_methods(self, defs, rcbs, defaults)
RuntimeError:
attribute 'downsample' of type 'NoneType' is not usable in a script method (did you forget to add it __constants__?):
at /nfs/engine/rajagopalar/anaconda3/envs/torchExp/lib/python3.6/site-packages/torchvision/models/resnet.py:109:12
        out = self.relu(out)

        out = self.conv2(out)
        out = self.bn2(out)
        out = self.relu(out)

        out = self.conv3(out)
        out = self.bn3(out)

        if self.downsample is not None:
           ~~~~~~~~~~~~~~~ <--- HERE
            identity = self.downsample(x)
        out += identity
        out = self.relu(out)

        return out
'__module__.__torch__.torchvision.models.resnet.___torch_mangle_32.Bottleneck.forward' is being compiled since it was called from '__module__.__torch__.models.ResNet.ResNet50.forward'
at /nfs/engine/rajagopalar/anaconda3/envs/torchExp/lib/python3.6/site-packages/torchvision/models/resnet.py:114:16

        out = self.conv3(out)
        out = self.bn3(out)

        if self.downsample is not None:
            identity = self.downsample(x)
        out += identity
        out = self.relu(out)

        return out
               ~~~ <--- HERE

Further, the error comes from the torchvision Resnet model and not my code which confused me as to where the error was

I am running the code with pytorch==1.2. I have tried using Optional[T] tags but it did not work as expected still

I want to know if there's anything wrong on my end or must I create more files for scripting to happen.

Any help or suggestions would be welcome

The first model

traced_model = torch.jit.script(model)

And the second model

myexactor = FeatureExtractor(traced_model, exact_list)

...
scripted_trace = torch.jit.script(myexactor, img1)

The code is taken from github : https://github.com/michuanhaohao/AlignedReID

and I made changes in the file :
https://github.com/michuanhaohao/AlignedReID/blob/master/Alignedreid_demo.py


title: Correct dimensions to convert a wide data frame to Pytorch tensor object?

body:
I have this time-series  data frame that has 56 columns and 36508 samples; 55 are predictors and the last one is the output. I'm trying to fit a LSTM neural network and while I'll be able to fit the model, I'm having some hard time converting the features to Pytorch tensor objects. Currently I have already normalised the data between 0 and 1 and also split the data into train and test sets.

import torch
import torch.nn as nn

print(x_train.shape)
(27380, 55)

print(y_train.shape)
(27380,)

print(x_test.shape)
(9128, 55)

print(y_test.shape)
(9128,)

I've had no problems converting the target to a tensor object, since the series is only 1D like so:

y_train = torch.FloatTensor(y_train).view(-1)
print(y_train[:5])
tensor([0.7637, 0.6220, 0.6566, 0.6922, 0.6774])

But when it comes to converting the features, then I'm unable to figure out the dimensions that need to be specified. I've tried this:

x_train = torch.FloatTensor(x_train).view(-1, 55)

ValueError: could not determine the shape of object type 'DataFrame'

How do I properly convert the features dataset to a tensor object? Sadly the documentation seems vague.


title: How can I access a submatrix specified by row and column index vectors from a matrix (2D tensor) in pytorch?

body:
Please consider a matrix
A= torch.tensor([[1,2,3,4], [1,2,3,4], [1,2,3,4], [1,2,3,4]]
and vectors of indexes
ind_1= [0,1] and ind_2= [2,3].

Is it possible to access the top-right sub-matrix of the matrix A ([[3,4], [3,4]]) without using any for loops?  Naively using A[ind_1, ind_2] merely returns the elements 3 and 4 (the diagonal of the required sub-matrix).


title: Making transformers BertForSequenceClassification initial layers non-trainable for pytorch training

body:
I'm trying to do a transfer learning with BertForSequenceClassification https://huggingface.co/transformers/model_doc/bert.html#bertforsequenceclassification

This is my simple NN model for classification.

from transformers import BertTokenizer, BertForSequenceClassification
class NN(nn.Module):
    def __init__(self):
        super(NN, self).__init__()
        self.bert = BertForSequenceClassification.from_pretrained('bert-base-uncased', num_labels = 17)
    def forward(self, x):
        return self.bert(x)

a = NN()

Once I print my model I get this:

NN(
  (bert): BertForSequenceClassification(
    (bert): BertModel(
      (embeddings): BertEmbeddings(
        (word_embeddings): Embedding(30522, 768, padding_idx=0)
        (position_embeddings): Embedding(512, 768)
        (token_type_embeddings): Embedding(2, 768)
        (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
        (dropout): Dropout(p=0.1, inplace=False)
      )
      (encoder): BertEncoder(
        (layer): ModuleList(
          (0): BertLayer(
            (attention): BertAttention(
              (self): BertSelfAttention(
                (query): Linear(in_features=768, out_features=768, bias=True)
                (key): Linear(in_features=768, out_features=768, bias=True)
                (value): Linear(in_features=768, out_features=768, bias=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
              (output): BertSelfOutput(
                (dense): Linear(in_features=768, out_features=768, bias=True)
                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
            )
            (intermediate): BertIntermediate(
              (dense): Linear(in_features=768, out_features=3072, bias=True)
            )
            (output): BertOutput(
              (dense): Linear(in_features=3072, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (1): BertLayer(
            (attention): BertAttention(
              (self): BertSelfAttention(
                (query): Linear(in_features=768, out_features=768, bias=True)
                (key): Linear(in_features=768, out_features=768, bias=True)
                (value): Linear(in_features=768, out_features=768, bias=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
              (output): BertSelfOutput(
                (dense): Linear(in_features=768, out_features=768, bias=True)
                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
            )
            (intermediate): BertIntermediate(
              (dense): Linear(in_features=768, out_features=3072, bias=True)
            )
            (output): BertOutput(
              (dense): Linear(in_features=3072, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (2): BertLayer(
            (attention): BertAttention(
              (self): BertSelfAttention(
                (query): Linear(in_features=768, out_features=768, bias=True)
                (key): Linear(in_features=768, out_features=768, bias=True)
                (value): Linear(in_features=768, out_features=768, bias=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
              (output): BertSelfOutput(
                (dense): Linear(in_features=768, out_features=768, bias=True)
                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
            )
            (intermediate): BertIntermediate(
              (dense): Linear(in_features=768, out_features=3072, bias=True)
            )
            (output): BertOutput(
              (dense): Linear(in_features=3072, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (3): BertLayer(
            (attention): BertAttention(
              (self): BertSelfAttention(
                (query): Linear(in_features=768, out_features=768, bias=True)
                (key): Linear(in_features=768, out_features=768, bias=True)
                (value): Linear(in_features=768, out_features=768, bias=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
              (output): BertSelfOutput(
                (dense): Linear(in_features=768, out_features=768, bias=True)
                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
            )
            (intermediate): BertIntermediate(
              (dense): Linear(in_features=768, out_features=3072, bias=True)
            )
            (output): BertOutput(
              (dense): Linear(in_features=3072, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (4): BertLayer(
            (attention): BertAttention(
              (self): BertSelfAttention(
                (query): Linear(in_features=768, out_features=768, bias=True)
                (key): Linear(in_features=768, out_features=768, bias=True)
                (value): Linear(in_features=768, out_features=768, bias=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
              (output): BertSelfOutput(
                (dense): Linear(in_features=768, out_features=768, bias=True)
                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
            )
            (intermediate): BertIntermediate(
              (dense): Linear(in_features=768, out_features=3072, bias=True)
            )
            (output): BertOutput(
              (dense): Linear(in_features=3072, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (5): BertLayer(
            (attention): BertAttention(
              (self): BertSelfAttention(
                (query): Linear(in_features=768, out_features=768, bias=True)
                (key): Linear(in_features=768, out_features=768, bias=True)
                (value): Linear(in_features=768, out_features=768, bias=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
              (output): BertSelfOutput(
                (dense): Linear(in_features=768, out_features=768, bias=True)
                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
            )
            (intermediate): BertIntermediate(
              (dense): Linear(in_features=768, out_features=3072, bias=True)
            )
            (output): BertOutput(
              (dense): Linear(in_features=3072, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (6): BertLayer(
            (attention): BertAttention(
              (self): BertSelfAttention(
                (query): Linear(in_features=768, out_features=768, bias=True)
                (key): Linear(in_features=768, out_features=768, bias=True)
                (value): Linear(in_features=768, out_features=768, bias=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
              (output): BertSelfOutput(
                (dense): Linear(in_features=768, out_features=768, bias=True)
                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
            )
            (intermediate): BertIntermediate(
              (dense): Linear(in_features=768, out_features=3072, bias=True)
            )
            (output): BertOutput(
              (dense): Linear(in_features=3072, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (7): BertLayer(
            (attention): BertAttention(
              (self): BertSelfAttention(
                (query): Linear(in_features=768, out_features=768, bias=True)
                (key): Linear(in_features=768, out_features=768, bias=True)
                (value): Linear(in_features=768, out_features=768, bias=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
              (output): BertSelfOutput(
                (dense): Linear(in_features=768, out_features=768, bias=True)
                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
            )
            (intermediate): BertIntermediate(
              (dense): Linear(in_features=768, out_features=3072, bias=True)
            )
            (output): BertOutput(
              (dense): Linear(in_features=3072, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (8): BertLayer(
            (attention): BertAttention(
              (self): BertSelfAttention(
                (query): Linear(in_features=768, out_features=768, bias=True)
                (key): Linear(in_features=768, out_features=768, bias=True)
                (value): Linear(in_features=768, out_features=768, bias=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
              (output): BertSelfOutput(
                (dense): Linear(in_features=768, out_features=768, bias=True)
                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
            )
            (intermediate): BertIntermediate(
              (dense): Linear(in_features=768, out_features=3072, bias=True)
            )
            (output): BertOutput(
              (dense): Linear(in_features=3072, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (9): BertLayer(
            (attention): BertAttention(
              (self): BertSelfAttention(
                (query): Linear(in_features=768, out_features=768, bias=True)
                (key): Linear(in_features=768, out_features=768, bias=True)
                (value): Linear(in_features=768, out_features=768, bias=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
              (output): BertSelfOutput(
                (dense): Linear(in_features=768, out_features=768, bias=True)
                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
            )
            (intermediate): BertIntermediate(
              (dense): Linear(in_features=768, out_features=3072, bias=True)
            )
            (output): BertOutput(
              (dense): Linear(in_features=3072, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (10): BertLayer(
            (attention): BertAttention(
              (self): BertSelfAttention(
                (query): Linear(in_features=768, out_features=768, bias=True)
                (key): Linear(in_features=768, out_features=768, bias=True)
                (value): Linear(in_features=768, out_features=768, bias=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
              (output): BertSelfOutput(
                (dense): Linear(in_features=768, out_features=768, bias=True)
                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
            )
            (intermediate): BertIntermediate(
              (dense): Linear(in_features=768, out_features=3072, bias=True)
            )
            (output): BertOutput(
              (dense): Linear(in_features=3072, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (11): BertLayer(
            (attention): BertAttention(
              (self): BertSelfAttention(
                (query): Linear(in_features=768, out_features=768, bias=True)
                (key): Linear(in_features=768, out_features=768, bias=True)
                (value): Linear(in_features=768, out_features=768, bias=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
              (output): BertSelfOutput(
                (dense): Linear(in_features=768, out_features=768, bias=True)
                (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
                (dropout): Dropout(p=0.1, inplace=False)
              )
            )
            (intermediate): BertIntermediate(
              (dense): Linear(in_features=768, out_features=3072, bias=True)
            )
            (output): BertOutput(
              (dense): Linear(in_features=3072, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
        )
      )
      (pooler): BertPooler(
        (dense): Linear(in_features=768, out_features=768, bias=True)
        (activation): Tanh()
      )
    )
    (dropout): Dropout(p=0.1, inplace=False)
    (classifier): Linear(in_features=768, out_features=17, bias=True)
  )
)

I want to make only the last Linear layer trainable but I can't access the layers of my model. I have tried iterating model as a list and features but both gives error that there is no such attribute.

a.features

How can make all my layers frozen except the last linear layer?


title: Autoencoder to encode features/categories of data

body:
My question is regarding the use of autoencoders (in PyTorch). I have a tabular dataset with a categorical feature that has 10 different categories. Names of these categories are quite different - some names consist of one word, some of two or three words. But all in all I have 10 unique category names. What I'm trying to do is to create an autoencoder which will encode names of these categories - for example, if I have a category named 'Medium size class', I want to see if it is possible to train autoencoder to encode this name as something like 'mdmsc' or something like that. The use of it would be to found out which data points are hard to encode or not typical or something like that. I tried to adapt autoencoder architectures from various tutorials online however nothing seems to work for me or I simply do not know how to use them as they are all about images. Maybe someone has any idea how this type of autoencoder might be accomplished if it is at all possible?

Edit: here's the model I have so far (I just tried to adapt some architectures I found online):

class Autoencoder(nn.Module):

def __init__(self, input_shape, encoding_dim):
    super(Autoencoder, self).__init__()

    self.encode = nn.Sequential(
        nn.Linear(input_shape, 128),
        nn.ReLU(True),
        nn.Linear(128, 64),
        nn.ReLU(True),
        nn.Linear(64, encoding_dim),
    )

    self.decode = nn.Sequential(
        nn.Linear(encoding_dim, 64),
        nn.ReLU(True),
        nn.Linear(64, 128),
        nn.ReLU(True),
        nn.Linear(128, input_shape)
    )

def forward(self, x):
    x = self.encode(x)
    x = self.decode(x)
    return x

model = Autoencoder(input_shape=10, encoding_dim=5)

And also I use LabelEncoder() and then OneHotEncoder()to give these features/categories I mentioned numerical form. However, after training, output is the same as was input (no changes on the category name) but when I try to use only encoder part I'm unable to apply LabelEncoder() and then OneHotEncoder() because of dimension issues. I feel like maybe I can do something differently at the beginning, then I try to give those features numerical form, however I'm not sure what should I do.


title: Difference between DCGAN & WGAN

body: In my understanding, DCGAN use convolution layer in both Generator and Discriminator, and WGAN adjust the loss function, optimizer, clipping and last sigmoid function. The part they control is not overlapping. So are there any conflict if i implement both changes of DCGAN & WGAN in one model?


title: How can I get the IDs of specific items in a Pytorch dataloader-based dataset with a query?

body: I have a large dataset (approx. 500GB and 180k data points plus labels) in a Pytorch dataloader. Until now, I used torch.utils.data.random_split to split the dataset randomly into training and validation. However, this lead to serious overfitting. Now, I want to rather use a deterministic split, i.e. based on the paths stored in the dataloader, I could figure out a non-random split. However, I have no idea how to do so... The question is: How can I get the IDs of about 10% of the data points based on some query that has a look at the information about the files stored in the data loader (e.g. the paths)?


title: Mismatch in batch size

body:
**My complete code is here: I am following github code for accomplishing my task. I am getting dimensions mismatch error. The code is giving me dimension mismatch error. I am receiving the following error: ValueError: Expected input batch_size (1) to match target batch_size (64).I am confused, i don't know what should i change in this code. Please help me in resolving this issue. **

def windowz(data, size):
   start = 0
   while start < len(data):
       yield start, start + size
      start += (size // 2)

 def segment_pa2(x_train,y_train,window_size):
segments = np.zeros(((len(x_train)//(window_size//2))-1,window_size,52))
labels = np.zeros(((len(y_train)//(window_size//2))-1))
i_segment = 0
i_label = 0
for (start,end) in windowz(x_train,window_size):
    if(len(x_train[start:end]) == window_size):
        m = stats.mode(y_train[start:end])
        segments[i_segment] = x_train[start:end]
        labels[i_label] = m[0]
        i_label+=1
        i_segment+=1

  return segments, labels
  print ('starting...')
 start_time = time.time()

dataset = sys.argv[1]
 path = '/Users/tehreem/Desktop/PAMAP2/PAMAP2_Dataset/pamap2.h5'

f = h5py.File(path, 'r')
print(f)

   x_train = f.get('train').get('inputs')[()]
   y_train = f.get('train').get('targets')[()]

  x_test = f.get('test').get('inputs')[()]
  y_test = f.get('test').get('targets')[()]

 print ("x_train shape =",x_train.shape)
 print ("y_train shape =",y_train.shape)
 print ("x_test shape =" ,x_test.shape)
 print ("y_test shape =",y_test.shape)

x_train = x_train[::3,:]
y_train = y_train[::3]
x_test = x_test[::3,:]
y_test = y_test[::3]
print ("x_train shape(downsampled) = ", x_train.shape)
print ("y_train shape(downsampled) =",y_train.shape)
print ("x_test shape(downsampled) =" ,x_test.shape)
print ("y_test shape(downsampled) =",y_test.shape)

 print (np.unique(y_train))
 print (np.unique(y_test))
unq = np.unique(y_test)

input_width = 52
print("segmenting signal...")
train_x, train_y = segment_pa2(x_train,y_train,input_width)
test_x, test_y = segment_pa2(x_test,y_test,input_width)
print ("signal segmented.")

train = pd.get_dummies(train_y)
test = pd.get_dummies(test_y)
train, test = train.align(test, join='inner', axis=1)

train_y = np.asarray(train)
test_y = np.asarray(test)

input_height = 1
input_width = input_width
num_labels = 11
num_channels = 52

   batch_size = 64
   stride_size = 2
   kernel_size_1 = 7
  kernel_size_2 = 3
  kernel_size_3 = 1
  depth_1 = 128
 depth_2 = 128
depth_3 = 128
num_hidden = 512

 dropout_1 = 0.1 #0.1
 dropout_2 = 0.25 #0.25
 dropout_3 = 0.5 #0.5

learning_rate = 0.0005
training_epochs = 50
total_batches = train_x.shape[0] // batch_size

  train_x = train_x.reshape(len(train_x),1,input_width,num_channels)
  test_x = test_x.reshape(len(test_x),1,input_width,num_channels)
 print ("test_x_reshaped = " , test_x.shape)
 print ("train_x shape =",train_x.shape)
 print ("train_y shape =",train_y.shape)
 print ("test_x shape =",test_x.shape)
 print ("test_y shape =",test_y.shape)

 train_x = train_x.reshape(-1,input_width,num_channels)
 test_x = test_x.reshape(-1,input_width,num_channels)

 def init_weights(m):
if type(m) == nn.LSTM:
    for name, param in m.named_parameters():
        if 'weight_ih' in name:
            torch.nn.init.orthogonal_(param.data)
        elif 'weight_hh' in name:
            torch.nn.init.orthogonal_(param.data)
        elif 'bias' in name:
            param.data.fill_(0)
  elif type(m) == nn.Conv1d or type(m) == nn.Linear:
    torch.nn.init.orthogonal_(m.weight)
    m.bias.data.fill_(0)
 model.apply(init_weights)

 import torch
 import torch.nn as nn
import torch.nn.functional as F

  class  CharCNN(nn.Module):
     def __init__(self):
     super(CharCNN, self).__init__()

    self.conv1 = nn.Sequential(
        nn.Conv1d(num_channels, depth_1, kernel_size=kernel_size_1, stride=stride_size),
        nn.ReLU(),
        nn.MaxPool1d(kernel_size=kernel_size_1, stride=stride_size),
        nn.Dropout(0.1),
    )

    self.conv2 = nn.Sequential(
        nn.Conv1d(depth_1, depth_2, kernel_size=kernel_size_2, stride=stride_size),
        nn.ReLU(),
        nn.MaxPool1d(kernel_size=kernel_size_2, stride=stride_size),
        nn.Dropout(0.25)
    )

    self.fc1 = nn.Sequential(
        nn.Linear(128*64, num_hidden),
        nn.ReLU(),
        nn.Dropout(0.5)
    )

    self.fc2 = nn.Sequential(
        nn.Linear(num_hidden, 11),
        nn.ReLU(),
        nn.Dropout(0.5)
    )

def forward(self, x):
    out = self.conv1(x)
    out = self.conv2(out)

    # collapse
    out = out.reshape(-1,128*64)
    #out = out.view(out.size(0), -1)
    # linear layer
    out = self.fc1(out)
    # output layer
    out = self.fc2(out)
    #out = self.log_softmax(x,dim=1)

    return out

 model =  CharCNN()
 print(model)

def iterate_minibatches(inputs, targets, batch_size, shuffle=True):
assert len(inputs) == len(targets)
if shuffle:
    indices = np.arange(len(inputs))
    np.random.shuffle(indices)
for start_idx in range(0, len(inputs) - batch_size + 1, batch_size):
    if shuffle:
        excerpt = indices[start_idx:start_idx + batch_size]
    else:
        excerpt = slice(start_idx, start_idx + batch_size)
    yield inputs[excerpt], targets[excerpt]

 criterion = nn.CrossEntropyLoss()
opt = torch.optim.Adam(model.parameters(),lr=learning_rate)

for e in range(training_epochs):
   if(train_on_gpu):
     net.cuda()
   train_losses = []

for batch in iterate_minibatches(train_x, train_y, batch_size):
    x, y = batch
    inputs, targets = torch.from_numpy(x), torch.from_numpy(y)
    if(train_on_gpu):
        inputs, targets = inputs.cuda(), targets.cuda()
        #inputs= inputs.view(batch_size,  128*64)
        #targets = targets.view(batch_size)
    opt.zero_grad()
    output = model(inputs)

    loss = criterion(output, targets.long())
    train_losses.append(loss.item())
    loss.backward()
    opt.step()
val_losses = []
accuracy=0
f1score=0
print("Epoch: {}/{}...".format(e+1, training_epochs),
              "Train Loss: {:.4f}...".format(np.mean(train_losses)))

I received the following error:

<ipython-input-468-7a508893b28d> in <module>
     21         output = model(inputs)
     22
---> 23         loss = criterion(output, targets.long())
     24         train_losses.append(loss.item())
     25         loss.backward()

~\AppData\Local\Continuum\anaconda3\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--> 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

~\AppData\Local\Continuum\anaconda3\lib\site-packages\torch\nn\modules\loss.py in forward(self, input, target)
    914     def forward(self, input, target):
    915         return F.cross_entropy(input, target, weight=self.weight,
--> 916                                ignore_index=self.ignore_index, reduction=self.reduction)
    917
    918

~\AppData\Local\Continuum\anaconda3\lib\site-packages\torch\nn\functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction)
   2019     if size_average is not None or reduce is not None:
   2020         reduction = _Reduction.legacy_get_string(size_average, reduce)
-> 2021     return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
   2022
   2023

~\AppData\Local\Continuum\anaconda3\lib\site-packages\torch\nn\functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)
   1834     if input.size(0) != target.size(0):
   1835         raise ValueError('Expected input batch_size ({}) to match target batch_size ({}).'
-> 1836                          .format(input.size(0), target.size(0)))
   1837     if dim == 2:
   1838         ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)

ValueError: Expected input batch_size (1) to match target batch_size (64).```


title: What is the right calculation of epoch loss in training?

body:
I am reading Pytorch official tutorial for fine tuning and I am faced with one problem and that is calculation of loss in each epoch.
Before this , I calculate loss for batch of data, accumulate these batch losses and find mean of these values as loss of epoch. But in that example, the calculation is as follow:
for inputs, labels in dataloaders[phase]:
            inputs = inputs.to(device)
            labels = labels.to(device)

            # zero the parameter gradients
            optimizer.zero_grad()

            # forward
            # track history if only in train
            with torch.set_grad_enabled(phase == 'train'):
                outputs = model(inputs)
                _, preds = torch.max(outputs, 1)
                loss = criterion(outputs, labels)

                # backward + optimize only if in training phase
                if phase == 'train':
                    loss.backward()
                    optimizer.step()

            # statistics
            running_loss += loss.item() * inputs.size(0)
            running_corrects += torch.sum(preds == labels.data)

my question is in this line running_loss += loss.item() * inputs.size(0). It is multiply loss value of batch in bach size. What is the true way to calculate loss of epoch?
and what is the unit of loss? What is the range of loss value?


title: How to convert a pytorch geometric graph to Networkx Multigraph?

body:
I have a weighted graph stored in Data object, and I want to convert this graph to Networkx MultiGraph. Here is what I have tried:
data = Data(x=x, edge_index=edge_index)
print(edge_index).

.torch.Size([3, 237])

conversion
first trial
from torch_geometric.utils import to_networkx, from_networkx
G=to_networkx(data, to_undirected=True)

ValueError: too many values to unpack (expected 2)

second trial
G = nx.MultiGraph()
edges = data.edge_index.numpy()
edge_list = []
for i in range(data.num_edges):
    edge_list.append(tuple(edges[:, i]))
G.add_edges_from(edge_list)
data.G = G
adj=nx.adjacency_matrix(G)
print('here the adj is', adj.shape)

Result=(18, 18) expected (23,23)

What else can I do? Thank you very much for your Input


title: How to get probabilities when using Pytorch's densenet?

body:
I want to do a binary classification and I used the DenseNet from Pytorch.

Here is my predict code:

densenet = torch.load(model_path)
densenet.eval()
output = densenet(input)
print(output)

And here is the output:

Variable containing:
54.4869 -54.3721
[torch.cuda.FloatTensor of size 1x2 (GPU 0)]

I want to get the probabilities of each class. What should I do?

I have noticed that torch.nn.Softmax() could be used when there are many categories, as discussed here.


title: Masking: Mask everything after a specified token (eos)

body:
My tgt tensor is in shape of [12, 32, 1] which is sequence_length, batch_size, token_idx.

What is the best way to create a mask which has ones for entries with <eos> and before in sequence, and zeros afterwards?

Currently I'm calculating my mask like this, which simply puts zeros where <blank> is, ones otherwise.

mask = torch.zeros_like(tgt).masked_scatter_((tgt != tgt_padding), torch.ones_like(tgt))

But the problem is, that my tgt can contain <blank> as well (before <eos>), in which cases I don't want to mask it out.

My temporary solution:

mask = torch.ones_like(tgt)
for eos_token in (tgt == tgt_eos).nonzero():
    mask[eos_token[0]+1:,eos_token[1]] = 0


title: Why pytorch(CUDA) couldn't allocate 290MB when there are 5GB of memory free

body:
Solution: I solved this problem by uninstalling the Nvidia GPU driver then reinstalling it. Running the driver installer without uninstalling it doesn’t work.

——————————————————————-

The Original Problem:

I got this error when I'm training a large CNN model.

    Traceback (most recent call last):
  File "C:\Users\dkdk\miniconda3\envs\transfer\lib\site-packages\torch\nn\modules\module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\Users\dkdk\miniconda3\envs\transfer\lib\site-packages\torch\nn\modules\batchnorm.py", line 107, in forward
    exponential_average_factor, self.eps)
  File "C:\Users\dkdk\miniconda3\envs\transfer\lib\site-packages\torch\nn\functional.py", line 1670, in batch_norm
    training, momentum, eps, torch.backends.cudnn.enabled
RuntimeError: CUDA out of memory. Tried to allocate 290.00 MiB (GPU 0; 8.00 GiB total capacity; 673.67 MiB already allocated; 5.27 GiB free; 686.00 MiB reserved in total by PyTorch)

You can see from the error message there are obviously 5.27GB of memory free in my GPU, but pytorch is only reserving 686MB. It used to have more than 5GB memory reserved. I tried another project running with Keras, Keras runs the project fine and allocated 5GB of memory.

I'm using GeForce RTX2080, Pytorch 1.4, CUDA Toolkit v10.1 on Windows

I thought this might be a driver issue. I reinstalled the Nvidia driver, which doesn't fix anything. I also tried downgrade to toolkit v9.2 and doesn't work as well. So I wonder what would cause this to happen and how should I fix this?

Output from running nvidia-smi when process is not running:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 442.19       Driver Version: 442.19       CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce RTX 2080   WDDM  | 00000000:02:00.0  On |                  N/A |
| 32%   29C    P8     1W / 225W |    911MiB /  8192MiB |      1%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0       908    C+G   C:\Windows\System32\MicrosoftEdgeCP.exe    N/A      |
|    0      6524    C+G   ...5n1h2txyewy\StartMenuExperienceHost.exe N/A      |
|    0     13760    C+G   ...DIA GeForce Experience\NVIDIA Share.exe N/A      |
|    0     14892    C+G   C:\Windows\System32\MicrosoftEdgeCP.exe    N/A      |
|    0     16452    C+G   C:\Windows\System32\MicrosoftEdgeCP.exe    N/A      |
|    0     24656    C+G   C:\Program Files\NZXT CAM\NZXT CAM.exe     N/A      |
|    0     26168    C+G   ...t_cw5n1h2txyewy\ShellExperienceHost.exe N/A      |
|    0     31108    C+G   ...AppData\Local\slack\app-4.4.0\slack.exe N/A      |
|    0     36420    C+G   C:\Windows\System32\MicrosoftEdgeCP.exe    N/A      |
|    0     37340    C+G   ...oftEdge_8wekyb3d8bbwe\MicrosoftEdge.exe N/A      |
|    0     37428    C+G   ...dows.Cortana_cw5n1h2txyewy\SearchUI.exe N/A      |
|    0     37440    C+G   ...osoft.LockApp_cw5n1h2txyewy\LockApp.exe N/A      |
|    0     37588    C+G   C:\Windows\System32\MicrosoftEdgeCP.exe    N/A      |
|    0     38396    C+G   Insufficient Permissions                   N/A      |
|    0     40804    C+G   ...hell.Experiences.TextInput.InputApp.exe N/A      |
|    0     41584    C+G   ...6)\Google\Chrome\Application\chrome.exe N/A      |
|    0     44740    C+G   C:\Windows\explorer.exe                    N/A      |
+-----------------------------------------------------------------------------+

outputs from running nvidia-smi when the process is running:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 442.19       Driver Version: 442.19       CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce RTX 2080   WDDM  | 00000000:02:00.0  On |                  N/A |
| 32%   31C    P8     1W / 225W |   1291MiB /  8192MiB |      1%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0       908    C+G   C:\Windows\System32\MicrosoftEdgeCP.exe    N/A      |
|    0      6524    C+G   ...5n1h2txyewy\StartMenuExperienceHost.exe N/A      |
|    0     13284      C   ...miniconda3\envs\transfer\python.exe     N/A      |
|    0     13760    C+G   ...DIA GeForce Experience\NVIDIA Share.exe N/A      |
|    0     14892    C+G   C:\Windows\System32\MicrosoftEdgeCP.exe    N/A      |
|    0     16452    C+G   C:\Windows\System32\MicrosoftEdgeCP.exe    N/A      |
|    0     24656    C+G   C:\Program Files\NZXT CAM\NZXT CAM.exe     N/A      |
|    0     26168    C+G   ...t_cw5n1h2txyewy\ShellExperienceHost.exe N/A      |
|    0     31108    C+G   ...AppData\Local\slack\app-4.4.0\slack.exe N/A      |
|    0     36420    C+G   C:\Windows\System32\MicrosoftEdgeCP.exe    N/A      |
|    0     37340    C+G   ...oftEdge_8wekyb3d8bbwe\MicrosoftEdge.exe N/A      |
|    0     37428    C+G   ...dows.Cortana_cw5n1h2txyewy\SearchUI.exe N/A      |
|    0     37440    C+G   ...osoft.LockApp_cw5n1h2txyewy\LockApp.exe N/A      |
|    0     37588    C+G   C:\Windows\System32\MicrosoftEdgeCP.exe    N/A      |
|    0     38396    C+G   Insufficient Permissions                   N/A      |
|    0     40804    C+G   ...hell.Experiences.TextInput.InputApp.exe N/A      |
|    0     41584    C+G   ...6)\Google\Chrome\Application\chrome.exe N/A      |
|    0     44740    C+G   C:\Windows\explorer.exe                    N/A      |
+-----------------------------------------------------------------------------+

Besides the error above, sometimes the error becomes "CUDNN_STATUS_INTERNAL_ERROR":

Traceback (most recent call last):
  File "E:/research/docs/transfer/train.py", line 32, in <module>
    lrn.train()
  File "E:\research\docs\transfer\main.py", line 592, in train
    outputs = self.model(train_inputs.float())
  File "C:\Users\dkdk\miniconda3\envs\transfer\lib\site-packages\torch\nn\modules\module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "E:\research\docs\transfer\builder.py", line 679, in forward
    x = self.bn_list[i](F.relu(self.conv_list[i](x)))
  File "C:\Users\dkdk\miniconda3\envs\histotransfer\lib\site-packages\torch\nn\modules\module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\Users\dkdk\miniconda3\envs\histotransfer\lib\site-packages\torch\nn\modules\conv.py", line 345, in forward
    return self.conv2d_forward(input, self.weight)
  File "C:\Users\dkdk\miniconda3\envs\histotransfer\lib\site-packages\torch\nn\modules\conv.py", line 342, in conv2d_forward
    self.padding, self.dilation, self.groups)
RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR


title: Pytorch: How to optimize multiple variables with respect to multiple losses?

body:
I want different losses to have their gradients computed with respect to different variables, and those variables to then all step together.

Here's a simple example demonstrating what I want:

import torch as T
x = T.randn(3, requires_grad = True)
y = T.randn(4, requires_grad = True)
z = T.randn(5, requires_grad = True)

x_opt = T.optim.Adadelta([x])
y_opt = T.optim.Adadelta([y])
z_opt = T.optim.Adadelta([z])

for i in range(n_iter):
  x_opt.zero_grad()
  y_opt.zero_grad()
  z_opt.zero_grad()

  shared_computation = foobar(x, y, z)

  x_loss = f(x, y, z, shared_computation)
  y_loss = g(x, y, z, shared_computation)
  z_loss = h(x, y, z, shared_computation)

  x_loss.backward_with_respect_to(x)
  y_loss.backward_with_respect_to(y)
  z_loss.backward_with_respect_to(z)

  x_opt.step()
  y_opt.step()
  z_opt.step()

My question is how do we do that backward_with_respect_to part in PyTorch? I only want x's gradient w.r.t. x_loss, etc.. And then I want all the optimizers to step together (based on the current values of x, y, and z).


title: Loss not reducing in Linear Regression with Pytorch

body:
I'm working on a Linear Regression problem with Pytorch. The dataset I'm using is the Housing Prices from Kaggle. While training the model I see the loss is not reducing. It shows an erratic pattern. This is the Loss I'm getting after 100 epochs:
Epoch [10/100], Loss: 222273830912.0000
Epoch [20/100], Loss: 348813688832.0000
Epoch [30/100], Loss: 85658296320.0000
Epoch [40/100], Loss: 290305572864.0000
Epoch [50/100], Loss: 59399933952.0000
Epoch [60/100], Loss: 80360054784.0000
Epoch [70/100], Loss: 90352918528.0000
Epoch [80/100], Loss: 534457679872.0000
Epoch [90/100], Loss: 256064503808.0000
Epoch [100/100], Loss: 102400483328.0000

This is the code:
import torch
import numpy as np
from torch.utils.data import TensorDataset
import torch.nn as nn
from torch.utils.data import DataLoader
import torch.nn.functional as F

inputs = normalized_X
targets = np.array(train_y)

# Tensors
inputs = torch.from_numpy(inputs)
targets = torch.from_numpy(targets)
targets = targets.view(-1, 1)
train_ds = TensorDataset(inputs, targets.squeeze())
batch_size = 5
train_dl = DataLoader(train_ds, batch_size, shuffle=True)

model = nn.Linear(10, 1)
# Define Loss func
loss_fn = F.mse_loss
# Optimizer
opt = torch.optim.SGD(model.parameters(), lr = 1e-1)

num_epochs = 100
model.train()
for epoch in range(num_epochs):
    # Train with batches of data
    for xb, yb in train_dl:

        # 1. Generate predictions
        pred = model(xb.float())

        # 2. Calculate loss
        yb = yb.view(yb.size(0), -1)
        loss = loss_fn(pred, yb.float())

        # 3. Compute gradients
        loss.backward()

        # 4. Update parameters using gradients
        opt.step()

        # 5. Reset the gradients to zero
        opt.zero_grad()

    if (epoch+1) % 10 == 0:
        print('Epoch [{}/{}], Loss: {:.4f}'.format(epoch +
                                                   1, num_epochs,
                                                   loss.item()))


title: Explain CUDA out of memory in Pytorch

body:
Can anybody help me to explain the meaning of this common problem in Pytorch?
Model: EfficientDet-D4
GPU: RTX 2080Ti
Batch size: 2
CUDA out of memory. Tried to allocate 14.00 MiB (GPU 0; 11.00 GiB total capacity; 8.32 GiB already allocated; 2.59 MiB free; 8.37 GiB reserved in total by PyTorch)

Anyway, I think the model and GPU are not important here and I know the solution should be reduced batch size, try to turn off the gradient while validating, etc. But I just want to know what is the meaning of 8.32 GiB while I have 11 GiB but can not allocate 14.00 MiB more?
Addition: I try to watch nvidia-smi while training with batch size = 1, it took 9.5 GiB in my GPU.


title: Replacing a max pooling layer with an average pooling layer on a VGG model

body:
I'm following this article and I try to implement this function:
def replace_max_pooling(model):
 '''
 The function replaces max pooling layers with average pooling layers with
 the following properties: kernel_size=2, stride=2, padding=0.
 '''
 for layer in model.layers:
    if layer is max pooling:
        replace

But I get an error on the iteration that says:

ModuleAttributeError: 'VGG' object has no attribute 'layers'...

How can I do this correctly?


title: Using nn.ModuleList over Python list dramatically slows down training

body:
I'm training a very simple model that takes the number of hidden layers as a parameter. I originally stored these hidden layers in a vanilla python list [], however when converting this list to a nn.ModuleList, training slows down dramatically by at least one order of magnitude!

AdderNet

class AdderNet(nn.Module):
    def __init__(self, num_hidden, hidden_width):
        super(AdderNet, self).__init__()
        self.relu = nn.ReLU()

        self.hiddenLayers = []
        self.inputLayer = nn.Linear(2, hidden_width)
        self.outputLayer = nn.Linear(hidden_width, 1)

        for i in range(num_hidden):
            self.hiddenLayers.append(nn.Linear(hidden_width, hidden_width))

        self.hiddenLayers = nn.ModuleList(self.hiddenLayers)  # <--- causes DRAMATIC slowdown!

    def forward(self, x):
        out = self.inputLayer(x)
        out = self.relu(out)

        for layer in self.hiddenLayers:
            out = layer(out)
            out = self.relu(out)

        return self.outputLayer(out)

Training

for epoch in range(num_epochs):
    for i in range(0,len(data)):
        out = model.forward(data[i].x)
        loss = lossFunction(out, data[i].y)

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()


title: How to reproduce RNN results on several runs?

body:
I call same model on same input twice in a row and I don't get the same result, this model have nn.GRU layers so I suspect that it have some internal state that should be release before second run?

How to reset RNN hidden state to make it the same as if model was initially loaded?

UPDATE:

Some context:

I'm trying to run model from here:

https://github.com/erogol/WaveRNN/blob/master/models/wavernn.py#L93

I'm calling generate:

https://github.com/erogol/WaveRNN/blob/master/models/wavernn.py#L148

Here it's actually have some code using random generator in pytorch:

https://github.com/erogol/WaveRNN/blob/master/models/wavernn.py#L200

https://github.com/erogol/WaveRNN/blob/master/utils/distribution.py#L110

https://github.com/erogol/WaveRNN/blob/master/utils/distribution.py#L129

I have placed (I'm running code on CPU):

torch.manual_seed(0)
torch.cuda.manual_seed_all(0)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
np.random.seed(0)

in

https://github.com/erogol/WaveRNN/blob/master/utils/distribution.py

after all imports.

I have checked GRU weights between runs and they are the same:

https://github.com/erogol/WaveRNN/blob/master/models/wavernn.py#L153

Also I have checked logits and sample between runs and logits are the same but sample are not, so @Andrew Naguib seems were right about random seeding, but I'm not sure where the code that fixes random seed should be placed?

https://github.com/erogol/WaveRNN/blob/master/models/wavernn.py#L200

UPDATE 2:

I have placed seed init inside generate and now results are consistent:

https://github.com/erogol/WaveRNN/blob/master/models/wavernn.py#L148


title: Loading a torchscript model in C++: Cannot initialize CUDA without ATen_cuda library

body:
I have built a CNN with Pytorch with GPU support. It works fine in python and I would like to import it in my C++ application using VS2017. I am following these instructions here and here.I can build and run the application, but I get this error when loading the model:

error loading the model
Cannot initialize CUDA without ATen_cuda library. PyTorch splits its backend into two shared
libraries: a CPU library and a CUDA library; this error has occurred because you are trying to use
some CUDA functionality, but the CUDA library has not been loaded by the dynamic linker for some
reason.  The CUDA library MUST be loaded, EVEN IF you don't directly use any symbols from the CUDA
library! One common culprit is a lack of -Wl,--no-as-needed in your link arguments; many dynamic
linkers will delete dynamic library dependencies if you don't depend on any of their symbols.  You
can check if this has occurred by using ldd on your binary to see if there is a dependency on
*_cuda.so library. (initCUDA at
C:\w\1\s\windows\pytorch\aten\src\ATen/detail/CUDAHooksInterface.h:63)
(no backtrace available)

I cannot find any ATen_cuda file in the C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1.Has anyone a Windows C++ VS2017 Pytorch with GPU working sample?


title: The location of source code for torch.mean()

body:
I'm trying to find the source code for torch.mean and am unable to find it in the pytorch github. It is under math operations but I can't find it at all.

I've looked everywhere and inspected most pages under pytorch/torch and am still unable to find it.

I even did ?? in a jupyter notebook but it just returned a useless docstring


title: XLNetForSequenceClassification Pretrained model unable to load

body:
I tried loading the XLNet pretrained but this occurred. I've tried this before and it worked, however, now it doesn't. Any suggestion on how to fix this problem?

model = XLNetForSequenceClassification.from_pretrained("xlnet-large-cased", num_labels = 2)
model.to(device)

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-55-d6f698a3714b> in <module>()
----> 1 model = XLNetForSequenceClassification.from_pretrained("xlnet-large-cased", num_labels = 2)
      2 model.to(device)

3 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/sparse.py in __init__(self, num_embeddings, embedding_dim, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse, _weight)
     95         self.scale_grad_by_freq = scale_grad_by_freq
     96         if _weight is None:
---> 97             self.weight = Parameter(torch.Tensor(num_embeddings, embedding_dim))
     98             self.reset_parameters()
     99         else:

RuntimeError: Trying to create tensor with negative dimension -1: [-1, 1024]


title: Multi-dimensional tensor dot product in pytorch

body:
I have two tensors of shapes (8, 1, 128) as follows.

q_s.shape
Out[161]: torch.Size([8, 1, 128])

p_s.shape
Out[162]: torch.Size([8, 1, 128])

Above two tensors represent a batch of eight 128 dimensional vectors. I want the dot product of batch q_s with batch p_s. How can I do this? I tried to use torch.tensordot function as follows. It works as expected as well. But it also does the extra work, which I don't want it to do. See the following example.

dt = torch.tensordot(q_s, p_s, dims=([1,2], [1,2]))

dt
Out[176]:
tensor([[0.9051, 0.9156, 0.7834, 0.8726, 0.8581, 0.7858, 0.7881, 0.8063],
        [1.0235, 1.5533, 1.2155, 1.2048, 1.3963, 1.1310, 1.1724, 1.0639],
        [0.8762, 1.3490, 1.2923, 1.0926, 1.4703, 0.9566, 0.9658, 0.8558],
        [0.8136, 1.0611, 0.9131, 1.1636, 1.0969, 0.9443, 0.9587, 0.8521],
        [0.6104, 0.9369, 0.9576, 0.8773, 1.3042, 0.7900, 0.8378, 0.6136],
        [0.8623, 0.9678, 0.8163, 0.9727, 1.1161, 1.6464, 0.9765, 0.7441],
        [0.6911, 0.8392, 0.6931, 0.7325, 0.8239, 0.7757, 1.0456, 0.6657],
        [0.8493, 0.8174, 0.8041, 0.9013, 0.8003, 0.7451, 0.7408, 1.1771]],
       grad_fn=<AsStridedBackward>)

dt.shape
Out[177]: torch.Size([8, 8])

As we can see, this produces the tensor of size (8,8) with the dot products I want lying on the diagonal. Is there any different way to obtain a smaller required tensor of shape (8,1), which just contains the elements lying on the diagonal in above result. To be more clear, the elements lying on the diagonal are the correct required dot products we want as a dot product of two batches. Element at index [0][0] is dot product of q_s[0] and p_s[0]. Element at index [1][1] is dot product of q_s[1] and p_s[1] and so on.

Is there a better way to obtain the desired dot product in pytorch?


title: Build vocabulary only from training data or entire data?

body:
Should I build the vocabulary only from train data or all data, wouldn't that effect test data in both ways? I mean :

If we only build the vocab from train data, The model wouldn't recognize a lot of the words in the validation and testing data, if the word is not available in the vocabulary.

Would considering a pre-trained word embedding help in this situation (i.e. the model learns the new word not from training data but from the pre-trained word embedding)?

If yes, Would a randomly Initialized word embedding have the same effect?

On the contrary, I've seen many examples where the coders build their vocab from the entire data,  testing and validation data are shared with training data. Wouldn't this be an obvious data leakage problem?


title: `Exception: process 0 terminated with exit code 1` error when using `torch.multiprocessing.spawn` to parallelize over multiple GPUs

body:
I have the following code below using torch.multiprocessing.spawn to parallelize over multiple GPUs:
import numpy as np
import torch
from torch.multiprocessing import Pool, set_start_method, spawn

X = np.array([[1, 3, 2, 3], [2, 3, 5, 6], [1, 2, 3, 4]])
X = torch.DoubleTensor(X)
# X = torch.DoubleTensor(X).cuda()

def X_power_func(j):
    X_power = X.cuda()**j
    return X_power

if __name__ == '__main__':
    results = spawn(X_power_func, range(4), nprocs=1)

results

But I am getting this error below when I run the code:
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
<ipython-input-9-97eb990d7396> in <module>()
     12
     13 if __name__ == '__main__':
---> 14     results = spawn(X_power_func, range(4), nprocs=1)
     15
     16 results

2 frames
/usr/local/lib/python3.6/dist-packages/torch/multiprocessing/spawn.py in join(self, timeout)
    111                 raise Exception(
    112                     "process %d terminated with exit code %d" %
--> 113                     (error_index, exitcode)
    114                 )
    115

Exception: process 0 terminated with exit code 1

What I have done wrong in my code?


title: what is the function of default_loader in torch?

body:
import os
    import pandas as pd
    import numpy as np
    from torchvision.datasets.folder import default_loader
    from torchvision.datasets.utils import download_url
    from torch.utils.data import Dataset

 class Sample_Class(Dataset):
      def __init__(self,root,train=True,transform=None,loader=default_loader):
        self.root = os.path.expanduser(root)
        self.transform = transform
        self.loader = default_loader

In the above code snippet , what is the significance of loader=default_loader, what exactly does that do?


title: Cannot Obtain Similar DL Prediction Result in Pytorch C++ API Compared to Python

body:
I have trained a deep learning model using unet architecture in order to segment the nuclei in python and pytorch. I would like to load this pretrained model and make prediction in C++. For this reason, I obtained trace file(with pt extension). Then, I have run this code:

#include <iostream>

#include <torch/script.h> // One-stop header.

#include <iostream>
#include <memory>

#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>

using namespace cv;

int main(int argc, const char* argv[]) {

    Mat image;
    image = imread("C:/Users/Sercan/PycharmProjects/samplepyTorch/test_2.png", CV_LOAD_IMAGE_COLOR);

    std::shared_ptr<torch::jit::script::Module> module = torch::jit::load("C:/Users/Sercan/PycharmProjects/samplepyTorch/epistroma_unet_best_model_trace.pt");
    module->to(torch::kCUDA);

    std::vector<int64_t> sizes = { 1, 3, image.rows, image.cols };
    torch::TensorOptions options(torch::ScalarType::Byte);
    torch::Tensor tensor_image = torch::from_blob(image.data, torch::IntList(sizes), options);
    tensor_image = tensor_image.toType(torch::kFloat);
    auto result = module->forward({ tensor_image.to(at::kCUDA) }).toTensor();

    result = result.squeeze().cpu();
    result = at::sigmoid(result);

    cv::Mat img_out(image.rows, image.cols, CV_32F, result.data<float>());

    cv::imwrite("img_out.png", img_out);

}

Image outputs ( First image: test image, Second image: Python prediction result, Third image: C++ prediction result):

As you see, C++ prediction output is not similar to python prediction output. Could you offer a solution to fix this problem?


title: how to specify python==3.6.8 for PyTorch Estimator (conda_packages not sufficient)

body:
I need to run my python script under Azure Machine Learning, using python=3.6.8 (not the default 3.6.2).  I am using the AML "PyTorch()" Estimator, setting the "conda_packages" arg to ["python==3.6.8"].

I am relying on this doc page for the PyTorch Estimator:

  https://docs.microsoft.com/en-us/python/api/azureml-train-core/azureml.train.dnn.pytorch?view=azure-ml-py

When my script runs, I print out "sys.version" and see that it is still set to python 3.6.2:

python: 3.6.2 | packaged by conda-forge | (default, Jul 23 2017, 22:59:30)
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)]

I expected to see python 3.6.8, since I specified that in the PyTorch Estimator's conda_packages arg.

I also tried moving the "python==3.6.8" from conda_packages to pip_packages, but received an error saying pip could not locate that package.

FYI, I have another package specified in pip_packages, and that does get installed correctly during this process.  It seems like the value of the "conda_packages" arg is not being used (I can find no mention of a conda or python install error in the AML logs for my job).