title: Find a GPU with enough memory

body:
I want to programmatically find out the available GPUs and their current memory usage and use one of the GPUs based on their memory availability. I want to do this in PyTorch.

I have seen the following solution in this post:

import torch.cuda as cutorch

for i in range(cutorch.device_count()):
    if cutorch.getMemoryUsage(i) > MEM:
        opts.gpuID = i
        break

but it is not working in PyTorch 0.3.1 (there is no function called, getMemoryUsage). I am interested in a PyTorch based (using the library functions) solution. Any help would be appreciated.


title: Pytorch select values from the last tensor dimension with indices from another tenor with a smaller dimension

body:
I have a tensor a with three dimensions. The first dimension corresponds to minibatch size, the second to the sequence length, and the third to the feature dimension. E.g.,

>>> a = torch.arange(1, 13, dtype=torch.float).view(2,2,3)  # Consider the values of a to be random
>>> a
tensor([[[ 1.,  2.,  3.],
         [ 4.,  5.,  6.]],

        [[ 7.,  8.,  9.],
         [10., 11., 12.]]])

I have a second, two-dimensional tensor. Its first dimension corresponds to the minibatch size and its second dimension to the sequence length. It contains values in the range of the indices of the third dimension of a. as third dimension has size 3, so b can contain values 0, 1 or 2. E.g.,

>>> b = torch.LongTensor([[0, 2],[1,0]])
>>> b
tensor([[0, 2],
        [1, 0]])

I want to obtain a tensor c that has the shape of b and contains all the values of a that are referenced by b.
In the upper scenario I would like to have:

c = torch.empty(2,2)
c[0,0] = a[0, 0, b[0,0]]
c[1,0] = a[1, 0, b[1,0]]
c[0,1] = a[0, 1, b[0,1]]
c[1,1] = a[1, 1, b[1,1]]

>>> c
tensor([[ 1.,  5.],
        [ 8., 10.]])

How can I create the tensor c fast? Further, I also want c to be differentiable (be able to use .backprob()). I am not too familiar with pytorch, so I am not sure, if a differentiable version of this exists.

As an alternative, instead of c having the same shape as b I could also use a c with the same shape of a, having only zeros, but at the places referenced by b ones. Then I could multiply a and c to obtain a differentiable tensor.

Like follows:

c = torch.zeros(2,2,3, dtype=torch.float)
c[0,0,b[0,0]] = 1
c[1,0,b[1,0]] = 1
c[0,1,b[0,1]] = 1
c[1,1,b[1,1]] = 1

>>> a*c
tensor([[[ 1.,  0.,  0.],
         [ 0.,  5.,  0.]],

        [[ 0.,  8.,  0.],
         [10.,  0.,  0.]]])


title: pytorch : unable to understand model.forward function

body:
I am learning deep learning and am trying to understand the pytorch code given below. I'm struggling to understand how the probability calculation works. Can somehow break it down in lay-man terms. Thanks a ton.

  ps = model.forward(images[0,:])

# Hyperparameters for our network
input_size = 784
hidden_sizes = [128, 64]
output_size = 10

# Build a feed-forward network
model = nn.Sequential(nn.Linear(input_size, hidden_sizes[0]),
                      nn.ReLU(),
                      nn.Linear(hidden_sizes[0], hidden_sizes[1]),
                      nn.ReLU(),
                      nn.Linear(hidden_sizes[1], output_size),
                      nn.Softmax(dim=1))
print(model)

# Forward pass through the network and display output
images, labels = next(iter(trainloader))
images.resize_(images.shape[0], 1, 784)
print(images.shape)
ps = model.forward(images[0,:])


title: [RUNTIME ERROR]: element 0 of tensors does not require grad and does not have a grad_fn

body:
I’m working on a classification problem. I’m trying to classify URLs as malicious or benign. I’m implementing logistic regression for this problem, and here is my code:

import torch
import numpy as np
import pandas as pd
import torch.nn as nn
import torch.utils.data
from torch.autograd import Variable

class LogisticRegressionModel(nn.Module):

    def __init__(self, in_dim, num_classes):
        super().__init__()
        self.linear = nn.Linear(in_dim, num_classes)

    def forward(self, x):
        out = self.linear(x)
        return out

class Train(LogisticRegressionModel):

    def __init__(self, in_dim, num_classes, lr, batch_size):
        super().__init__(in_dim, num_classes)
        self.batch_size = batch_size
        self.learning_rate = lr
        self.input_layer_dim = in_dim
        self.output_layer_dim = num_classes
        self.criterion = nn.CrossEntropyLoss()
        self.model = LogisticRegressionModel(self.input_layer_dim, self.output_layer_dim)
        self.device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
        self.model = self.model.to(self.device)
        self.optimizer = torch.optim.Adam(self.model.parameters(), lr = self.learning_rate)

    def epochs(self, iterations, train_dataset, batch_size):
        epochs = int(iterations/(len(train_dataset)/batch_size))
        return epochs

    def train_model(self, training_data, n_iters):
        batch = self.batch_size
        epochs = self.epochs(n_iters, training_data, batch)
        training_data = torch.utils.data.DataLoader(dataset = training_data, batch_size = batch, shuffle = False)

        for epoch in range(epochs):
            for i, data in enumerate(training_data):

                X_train = data[:, :-1]
                Y_train = data[:, -1]

                if torch.cuda.is_available():
                    x = Variable(X_train).cuda()
                    y = Variable(Y_train).cuda()

                else:
                    x = X_train.float()
                    y = Y_train.type(torch.LongTensor)

                self.optimizer.zero_grad()
                out = self.model(x).data
                loss = self.criterion(out, y)
                loss.backward()
                self.optimizer.step()

batch_size = 1000
train_class = Train((training_set.shape[1]-1), number_of_target_labels+1, 0.001, batch_size)
rain_class.train_model(training_set, batch_size)

However, I get the following error when I run this code block:

RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

I don't know why that is the case. If you'd like to have a look at my Notebook and the dataset, here is the link to it: https://github.com/islamaymansais/Malicious-URL-Classifier

This is a classification project to classify URLs as benign or a malicious category (phishing, defacement, spam, malware). Let me know if you have any questions that I can clarify.


title: How to get a specific sample from pytorch DataLoader?

body:
In Pytorch, is there any way of loading a specific single sample using the torch.utils.data.DataLoader class? I'd like to do some testing with it.
The tutorial uses
trainloader = torch.utils.data.DataLoader(...)
images, labels = next(iter(trainloader))

to fetch a random batch of samples. Is there are way, using DataLoader, to get a specific sample?
Cheers


title: Pytorch: Modifying VGG16 Architecture

body:
I'm currently trying to modify the VGG16 network architecture so that it's able to accept 400x400 px images.

Based on literature that I've read, the way to do it would be to covert the fully connected (FC) layers into convolutional (CONV) layers. This would essentially " allow the network to efficiently “slide” across a larger input image and make multiple evaluations of different parts of the image, incorporating all available contextual information." Afterwards, an Average Pooling layer is used to "average the multiple feature vectors into a single feature vector that summarizes the input image".

I've done this using this function, and have come up with the following network architecture:

----------------------------------------------------------------
        Layer (type)               Output Shape         Param #
================================================================
            Conv2d-1         [-1, 64, 400, 400]           1,792
              ReLU-2         [-1, 64, 400, 400]               0
            Conv2d-3         [-1, 64, 400, 400]          36,928
              ReLU-4         [-1, 64, 400, 400]               0
         MaxPool2d-5         [-1, 64, 200, 200]               0
            Conv2d-6        [-1, 128, 200, 200]          73,856
              ReLU-7        [-1, 128, 200, 200]               0
            Conv2d-8        [-1, 128, 200, 200]         147,584
              ReLU-9        [-1, 128, 200, 200]               0
        MaxPool2d-10        [-1, 128, 100, 100]               0
           Conv2d-11        [-1, 256, 100, 100]         295,168
             ReLU-12        [-1, 256, 100, 100]               0
           Conv2d-13        [-1, 256, 100, 100]         590,080
             ReLU-14        [-1, 256, 100, 100]               0
           Conv2d-15        [-1, 256, 100, 100]         590,080
             ReLU-16        [-1, 256, 100, 100]               0
        MaxPool2d-17          [-1, 256, 50, 50]               0
           Conv2d-18          [-1, 512, 50, 50]       1,180,160
             ReLU-19          [-1, 512, 50, 50]               0
           Conv2d-20          [-1, 512, 50, 50]       2,359,808
             ReLU-21          [-1, 512, 50, 50]               0
           Conv2d-22          [-1, 512, 50, 50]       2,359,808
             ReLU-23          [-1, 512, 50, 50]               0
        MaxPool2d-24          [-1, 512, 25, 25]               0
           Conv2d-25          [-1, 512, 25, 25]       2,359,808
             ReLU-26          [-1, 512, 25, 25]               0
           Conv2d-27          [-1, 512, 25, 25]       2,359,808
             ReLU-28          [-1, 512, 25, 25]               0
           Conv2d-29          [-1, 512, 25, 25]       2,359,808
             ReLU-30          [-1, 512, 25, 25]               0
        MaxPool2d-31          [-1, 512, 12, 12]               0
           Conv2d-32           [-1, 4096, 1, 1]     301,993,984
             ReLU-33           [-1, 4096, 1, 1]               0
          Dropout-34           [-1, 4096, 1, 1]               0
           Conv2d-35           [-1, 4096, 1, 1]      16,781,312
             ReLU-36           [-1, 4096, 1, 1]               0
          Dropout-37           [-1, 4096, 1, 1]               0
           Conv2d-38              [-1, 3, 1, 1]          12,291
AdaptiveAvgPool2d-39              [-1, 3, 1, 1]               0
          Softmax-40              [-1, 3, 1, 1]               0
================================================================
Total params: 333,502,275
Trainable params: 318,787,587
Non-trainable params: 14,714,688
----------------------------------------------------------------
Input size (MB): 1.83
Forward/backward pass size (MB): 696.55
Params size (MB): 1272.21
Estimated Total Size (MB): 1970.59
----------------------------------------------------------------

My question is simple: Is the use of the average pooling layer at the end necessary? It seems like by the last convolutional layer, we get a 1x1 image with 3 channels. Doing an average pooling on that would seem to not have any effect.

If there is anything amiss in my logic/ architecture, kindly feel free to point it out.
Thanks!


title: Pytorch AttributeError: module 'torch' has no attribute 'set_grad_enabled'

body:
AttributeError: module 'torch' has no attribute 'set_grad_enabled'

Traceback (most recent call last):
  File
   "/home/thaqafi/PycharmProjects/transfer_learning/tutorial_transfer.py", line 280, in

model_ft = train_model(model_ft, criterion, optimizer_ft, exp_lr_scheduler, num_epochs=25)

File
 "/home/thaqafi/PycharmProjects/transfer_learning/tutorial_transfer.py", line 179, in train_model
    with torch.set_grad_enabled(True):

AttributeError: module 'torch' has no attribute 'set_grad_enabled'

using source code form Pytorch webiste.

http://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html


title: How can I turn a torch tensor into a list of numpy arrays in Pytorch?

body:
I have a torch tensor that looks like torch.Size([32, 3, 64, 64]).

I'm trying convert the tensor into something that can pass these assertions:

assert(type(images) == list)
assert(type(images[0]) == np.ndarray)
assert(len(images[0].shape) == 3)
assert(np.max(images[0]) > 10)
assert(np.min(images[0]) >= 0.0)

I'm currently doing this to convert the tensor:

# turn tensor into list of lists
imgs = imgs.tolist()

# iterate over list and turn each image into a numpy array with normalized values
for idx, img in enumerate(imgs):
  img = cv2.normalize(np.array(img), None,
  alpha = 0, beta = 255, norm_type = cv2.NORM_MINMAX )

and I get this error:

File "scripts/run_model.py", line 158, in get_inception_score
assert(type(images[0]) == np.ndarray)
AssertionError

How can I convert the tensor correctly so that type(images) is a list and type(images[0] is a np.ndarray)?
Any help would be greatly appreciated.
Thank you in advance.


title: Pytorch - add rows of a 2D tensor element-wise

body:
I have the following tensor :

ts = torch.tensor([[1,2,3],[4,6,7],[8,9,10]])
> tensor([[ 1,  2,  3],
    [ 4,  6,  7],
    [ 8,  9, 10]])

I am looking for a pytorch generic operation that adds all rows element-wise like that:

ts2 = ts[0]+ts[1]+ts[2]
print(ts2)
> tensor([13, 17, 20])

In reality, the number of rows corresponds to the batch size that vary.


title: How to print the "actual" learning rate in Adadelta in pytorch

body:
In short:

I can't draw lr/epoch curve when using adadelta optimizer in pytorch because optimizer.param_groups[0]['lr'] always return the same value.

In detail:

Adadelta can dynamically adapts over time using only first order information and
has minimal computational overhead beyond vanilla stochastic gradient descent [1].

In pytorch, the source code of Adadelta is here https://pytorch.org/docs/stable/_modules/torch/optim/adadelta.html#Adadelta

Since it requires no manual tuning of learning rate, in my knowledge, we don't have to set any schedular after declare the optimizer

self.optimizer = torch.optim.Adadelta(self.model.parameters(), lr=1)

The way to check learning rate is

current_lr = self.optimizer.param_groups[0]['lr']

The problem is it always return 1 (the initial lr).

Could anyone tell me how can I get the true learning rate so that can I draw a lr/epch curve?

[1] https://arxiv.org/pdf/1212.5701.pdf


title: PyTorch: cpp_extension.py:184: UserWarning: Error checking compiler version

body:
I got a problem regarding pytorch:
when i run python setup.py build_ext --inplace
I get:
...
running build_ext
C:\python36\lib\site-packages\torch\utils\cpp_extension.py:184: UserWarning: Error checking compiler version for cl: [WinError 2] Das System kann die
angegebene Datei nicht finden
warnings.warn('Error checking compiler version for {}: {}'.format(compiler, error))
building 'pointnet2._ext' extension
SETUP:
CUDA 9.0
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin>nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2017 NVIDIA Corporation
Built on Fri_Sep__1_21:08:32_Central_Daylight_Time_2017
Cuda compilation tools, release 9.0, V9.0.176
MSVC
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin>cl
Microsoft (R) C/C++ Optimizing Compiler Version 19.00.24215.1 for x86
Copyright (C) Microsoft Corporation.  All rights reserved.
Pytorch 1.0.1
(installed via PIP)
Additonal:
When I run:
python setup.py install
I get:
...

  File "C:\python36\lib\distutils\command\build_ext.py", line 448, in build_extensions
    self._build_extensions_serial()
  File "C:\python36\lib\distutils\command\build_ext.py", line 473, in _build_extensions_serial
    self.build_extension(ext)
  File "C:\python36\lib\site-packages\setuptools\command\build_ext.py", line 199, in build_extension
    _build_ext.build_extension(self, ext)
  File "C:\python36\lib\distutils\command\build_ext.py", line 558, in build_extension
    target_lang=language)
  File "C:\python36\lib\distutils\ccompiler.py", line 717, in link_shared_object
    extra_preargs, extra_postargs, build_temp, target_lang)
  File "C:\python36\lib\distutils\_msvccompiler.py", line 501, in link
    build_temp = os.path.dirname(objects[0])
IndexError: list index out of range

setup.py
setup(
...
install_requires=requirements,
    ext_modules=[
        CUDAExtension(
            name="pointnet2._ext",
            sources=_ext_sources,
            extra_compile_args={
                "cxx": ["-O2", "-I{}".format("{}/include".format(_ext_src_root))],
                "nvcc": ["-O2", "-I{}".format("{}/include".format(_ext_src_root))],
            },
        )
    ],
...
)


title: How to calculate Pixel wise accuracy in pytorch?

body:
My code looks like the following and I get accuracy from 0 to 9000, which means its clearly not working.

optimizer.zero_grad()
outputs = net(inputs)
loss = criterion(outputs, labels)
loss.backward()
optimizer.step()
running_loss += loss.item()

predicted = outputs.data
predicted = predicted.to('cpu')
predicted_img = predicted.numpy()

labels_data = labels.data
labels_data = labels_data.to('cpu')
labels_data = labels_data.numpy()
labels = labels.to(device)

_, predicted = torch.max(outputs.data, 1)
total = labels.size(0) * labels.size(1) * labels.size(2)
correct = (predicted_img == labels_data).sum().item()
accuracy += ( correct / total)
avg_accuracy = accuracy/(batch)

What am I doing wrong ?


title: how to upload and read a zip file containing training and testing images data from google colab from my pc

body:
I am new to google colab. I am implementing a pretrained vgg16 and resnet50 model using pytorch, but I am unable to load my file and read it as it returns an error of no directory found

I have uploaded the data through file also I have used to upload it using

from google.colab import files
uploaded = files.upload()

The file got uploaded but when I tried to unzip it because it is a zip file using
!unzip content/cropped_months
then it says

  no file found

import torch
import torch.nn as nn
import torch.optim as optim
from torchvision.transforms import *
from torch.optim import lr_scheduler
from torch.autograd import Variable
import numpy as np
import torchvision
from torchvision import datasets, models, transforms
import matplotlib.pyplot as plt
import time
import os
import copy

from google.colab import files
uploaded = files.upload()

!unzip content/cropped_months

data_dir = 'content/cropped_months'

#Define transforms for the training data and testing data
train_transforms = transforms.Compose([transforms.RandomRotation(30),transforms.RandomResizedCrop(224),transforms.RandomHorizontalFlip(),transforms.ToTensor(),transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])])

test_transforms = transforms.Compose([transforms.Resize(256),transforms.CenterCrop(224),transforms.ToTensor(),transforms.Normalize([0.485, 0.456, 0.406],[0.229, 0.224, 0.225])])

#pass transform here-in
train_data = datasets.ImageFolder(data_dir + '/train', transform=train_transforms)
test_data = datasets.ImageFolder(data_dir + '/test', transform=test_transforms)

#data loaders
trainloader = torch.utils.data.DataLoader(train_data, batch_size=8, shuffle=True)
testloader = torch.utils.data.DataLoader(test_data, batch_size=8, shuffle=True)

print("Classes: ")
class_names = train_data.classes
print(class_names)

first error

  unzip:  cannot find or open content/cropped_months,
  content/cropped_months.zip or content/cropped_months.ZIP.

second error

  --------------------------------------------------------------------------- FileNotFoundError                         Traceback (most recent call
  last)  in ()
       16
       17 #pass transform here-in
  ---> 18 train_data = datasets.ImageFolder(data_dir + '/train', transform=train_transforms)
       19 test_data = datasets.ImageFolder(data_dir + '/test', transform=test_transforms)
       20

  2 frames
  /usr/local/lib/python3.6/dist-packages/torchvision/datasets/folder.py
  in _find_classes(self, dir)
      114         if sys.version_info >= (3, 5):
      115             # Faster and available in Python 3.5 and above
  --> 116             classes = [d.name for d in os.scandir(dir) if d.is_dir()]
      117         else:
      118             classes = [d for d in os.listdir(dir) if os.path.isdir(os.path.join(dir, d))]

  FileNotFoundError: [Errno 2] No such file or directory:
  'content/cropped_months (1)/train'


title: What is the PyTorch equivalent to Numpy's linalg.solve?

body: I have looked it up and there are some posts, for example this one, that suggest to use torch.gesv but I can't seem to find it in the PyTorch documentation.


title: How to fix pytorch multi processing issue on cpu?

body: I'm doing inference of pytorch on CPU. I found pytorch is not utilizing all the cores of CPU for prediction. How to use all cores in pytorch?


title: What does torchvision.transforms.Resize(size, interpolation=2) actually do?

body: Does it add to the image if too small or crop if too big or just stretch the image to the desired size?


title: Pytorch default dataloader gets stuck for large image classification training set

body:
I am training image classification models in Pytorch and using their default data loader to load my training data. I have a very large training dataset, so usually a couple thousand sample images per class. I've trained models with about 200k images total without issues in the past. However I've found that when have over a million images in total, the Pytorch data loader get stuck.

I believe the code is hanging when I call datasets.ImageFolder(...). When I Ctrl-C, this is consistently the output:

Traceback (most recent call last):                                                                                │
  File "main.py", line 412, in <module>                                                                                │
    main()                                                                                │
  File "main.py", line 122, in main                                                                                │
    run_training(args.group, args.num_classes)                                                                                │
  File "main.py", line 203, in run_training                                                                                │
    train_loader = create_dataloader(traindir, tfm.train_trans, shuffle=True)                                                      │
  File "main.py", line 236, in create_dataloader                                                                                │
    dataset = datasets.ImageFolder(directory, trans)                                                                               │
  File "/home/username/.local/lib/python3.5/site-packages/torchvision/datasets/folder.py", line 209, in __init__     │
    is_valid_file=is_valid_file)                                                                                │
  File "/home/username/.local/lib/python3.5/site-packages/torchvision/datasets/folder.py", line 94, in __init__      │
    samples = make_dataset(self.root, class_to_idx, extensions, is_valid_file)                                                     │
  File "/home/username/.local/lib/python3.5/site-packages/torchvision/datasets/folder.py", line 47, in make_dataset  │
    for root, _, fnames in sorted(os.walk(d)):                                                                                │
  File "/usr/lib/python3.5/os.py", line 380, in walk                                                                               │
    is_dir = entry.is_dir()                                                                                │
Keyboard Interrupt

I thought there might be a deadlock somewhere, however based off the stack output from Ctrl-C it doesn't look like its waiting on a lock. So then I thought that the dataloader was just slow because I was trying to load a lot more data. I let it run for about 2 days and it didn't make any progress, and in the last 2 hours of loading I checked the amount of RAM usage stayed the same.  I also have been able to load training datasets with over 200k images in less than a couple hours in the past. I also tried upgrading my GCP machine to have 32 cores, 4 GPUs, and over 100GB in RAM, however it seems to be that after a certain amount of memory is loaded the data loader just gets stuck.

I'm confused how the data loader could be getting stuck while looping through the directory, and I'm still unsure if its stuck or just extremely slow. Is there some way I can change the Pytortch dataloader to be able to handle 1million+ images for training? Any debugging suggestions are also appreciated!

Thank you!


title: How to use multiprocessing in PyTorch?

body:
I'm trying to use PyTorch with complex loss function. In order to accelerate the code, I hope that I can use the PyTorch multiprocessing package.

The first trial, I put 10x1 features into the NN and get 10x4 output.

After that, I want to pass 10x4 parameters into a function to do some calculation. (The calculation will be complex in the future.)

After calculating, the function will return a 10x1 array in total. This array will be set as NN_energy and calculate loss function.

Besides, I also want to know if there is another method to create a backward-able array to store the NN_energy array, instead of using

NN_energy = net(Data_in)[0:10,0]

Thanks a lot.

Full Code:

import torch
import numpy as np
from torch.autograd import Variable
from torch import multiprocessing

def func(msg,BOP):
    ans = (BOP[msg][0]+BOP[msg][1]/BOP[msg][2])*BOP[msg][3]
    return ans

class Net(torch.nn.Module):
    def __init__(self, n_feature, n_hidden_1, n_hidden_2, n_output):
        super(Net, self).__init__()
        self.hidden_1 = torch.nn.Linear(n_feature , n_hidden_1)  # hidden layer
        self.hidden_2 = torch.nn.Linear(n_hidden_1, n_hidden_2)  # hidden layer
        self.predict  = torch.nn.Linear(n_hidden_2, n_output  )  # output layer

    def forward(self, x):
        x = torch.tanh(self.hidden_1(x))      # activation function for hidden layer
        x = torch.tanh(self.hidden_2(x))      # activation function for hidden layer
        x = self.predict(x)                   # linear output
        return x

if __name__ == '__main__': # apply_async
    Data_in      = Variable( torch.from_numpy( np.asarray(list(range( 0,10))).reshape(10,1) ).float() )
    Ground_truth = Variable( torch.from_numpy( np.asarray(list(range(20,30))).reshape(10,1) ).float() )

    net = Net( n_feature=1 , n_hidden_1=15 , n_hidden_2=15 , n_output=4 )     # define the network
    optimizer = torch.optim.Rprop( net.parameters() )
    loss_func = torch.nn.MSELoss()  # this is for regression mean squared loss
    NN_output = net(Data_in)
    args = range(0,10)
    pool = multiprocessing.Pool()
    return_data = pool.map( func, zip(args, NN_output) )
    pool.close()
    pool.join()

    NN_energy = net(Data_in)[0:10,0]
    for i in range(0,10):
        NN_energy[i] = return_data[i]

    loss = torch.sqrt( loss_func( NN_energy , Ground_truth ) )     # must be (1. nn output, 2. target)
    print(loss)

Error messages:

  File
  "C:\ProgramData\Anaconda3\lib\site-packages\torch\multiprocessing\reductions.py",
  line 126, in reduce_tensor
      raise RuntimeError("Cowardly refusing to serialize non-leaf tensor which requires_grad, "

  RuntimeError: Cowardly refusing to serialize non-leaf tensor which
  requires_grad, since autograd does not support crossing process
  boundaries.  If you just want to transfer the data, call detach() on
  the tensor before serializing (e.g., putting it on the queue).


title: Multiclass semantic segmentation model evaluation

body:
I am doing a project on multiclass semantic segmentation. I have formulated a model that outputs pretty descent segmented images by decreasing the loss value. However, I cannot evaluate the model performance in metrics, such as meanIoU or Dice coefficient.
In case of binary semantic segmentation it was easy just to set the threshold of 0.5, to classify the outputs as an object or background, but it does not work in the case of multiclass semantic segmentation. Could you please tell me how to obtain model performance on the aforementioned metrics? Any help will be highly appreciated!

By the way, I am using PyTorch framework and CamVid dataset.


title: How to create Datasets Like MNIST in Pytorch?

body:
I have looked Pytorch source code of MNIST dataset but it seems to read numpy array directly from binaries.
How can I just create train_data and train_labels like it? I have already prepared images and txt with labels.

I have learned how to read image and label and write get_item and len, what really confused me is how to make train_data and train_labels, which is torch.Tensor. I tried to arrange them into python lists and convert to torch.Tensor but failed:

for index in range(0,len(self.files)):
  fn, label = self.files[index]
  img = self.loader(fn)
  if self.transform is not None:
    img = self.transform(img)
  train_data.append(img)
self.train_data = torch.tensor(train_data)

ValueError: only one element tensors can be converted to Python scalars


title: Understanding PyTorch Tensor Shape

body:
I have a simple question regarding the shape of tensor we define in PyTorch. Let's say if I say:

input = torch.randn(32, 35)

This will create a matrix with 32 row and 35 columns.
Now when I define:

input2 = torch.randn(1,2,32, 35)

What can I say about the dimension of the new matrix input2?
How can I define the rows and columns here? I mean do I have two matrices with shapes 32*35 packed by the tensor?

I want to better understand the geometry behind this. Thanks.


title: Pytorch how to get the gradient of loss function twice

body:
Here is what I'm trying to implement:

We calculate loss based on F(X), as usual. But we also define "adversarial loss" which is a loss based on F(X + e). e is defined as dF(X)/dX multiplied by some constant. Both loss and adversarial loss are backpropagated for the total loss.

In tensorflow, this part (getting dF(X)/dX) can be coded like below:

  grad, = tf.gradients( loss, X )
  grad = tf.stop_gradient(grad)
  e = constant * grad

Below is my pytorch code:

class DocReaderModel(object):
    def __init__(self, embedding=None, state_dict=None):
        self.train_loss = AverageMeter()
        self.embedding = embedding
        self.network = DNetwork(opt, embedding)
        self.optimizer = optim.SGD(parameters)

    def adversarial_loss(self, batch, loss, embedding, y):
        self.optimizer.zero_grad()
        loss.backward(retain_graph=True)
        grad = embedding.grad
        grad.detach_()

        perturb = F.normalize(grad, p=2)* 0.5
        self.optimizer.zero_grad()
        adv_embedding = embedding + perturb
        network_temp = DNetwork(self.opt, adv_embedding) # This is how to get F(X)
        network_temp.training = False
        network_temp.cuda()
        start, end, _ = network_temp(batch) # This is how to get F(X)
        del network_temp # I even deleted this instance.
        return F.cross_entropy(start, y[0]) + F.cross_entropy(end, y[1])

    def update(self, batch):
        self.network.train()
        start, end, pred = self.network(batch)
        loss = F.cross_entropy(start, y[0]) + F.cross_entropy(end, y[1])
        loss_adv = self.adversarial_loss(batch, loss, self.network.lexicon_encoder.embedding.weight, y)
        loss_total = loss + loss_adv

        self.optimizer.zero_grad()
        loss_total.backward()
        self.optimizer.step()

I have few questions:

1) I substituted tf.stop_gradient with grad.detach_(). Is this correct?

2) I was getting "RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time." so I added retain_graph=True at the loss.backward. That specific error went away.
However now I'm getting a memory error after few epochs (RuntimeError: cuda runtime error (2) : out of memory at /opt/conda/conda-bld/pytorch_1525909934016/work/aten/src/THC/generic/THCStorage.cu:58
). I suspect I'm unnecessarily retaining graph.

Can someone let me know pytorch's best practice on this? Any hint / even short comment will be highly appreciated.


title: Gradient is none in pytorch when it shouldn't

body:
I am trying to get/trace the gradient of a variable using pytorch, where I have that variable, pass it to a first function that looks for some minimum value of some other variable, then the output of the first function is inputted to a second function, and the whole thing repeats multiple times.

Here is my code:

import torch

def myFirstFunction(parameter_current_here):
    optimalValue = 100000000000000
    Optimal = 100000000000000
    for j in range(2, 10):
        i = torch.ones(1, requires_grad=True)*j
        with torch.enable_grad():
            optimalValueNow = i*parameter_current_here.sum()
        if (optimalValueNow < optimalValue):
            optimalValue = optimalValueNow
            Optimal = i
    return optimalValue, Optimal

def mySecondFunction(Current):
    with torch.enable_grad():
        y = (20*Current)/2 + (Current**2)/10
    return y

counter = 0
while counter < 5:
    parameter_current = torch.randn(2, 2, requires_grad=True)

    outputMyFirstFunction = myFirstFunction(parameter_current)
    outputmySecondFunction = mySecondFunction(outputMyFirstFunction[1])
    outputmySecondFunction.backward()

    print("outputMyFirstFunction after backward:",
               outputMyFirstFunction)
    print("outputmySecondFunction after backward:",
               outputmySecondFunction)
    print("parameter_current Gradient after backward:",
               parameter_current.grad)

    counter = counter + 1

The parameter_current.grad is none for all iterations when it obviously shouldn't be none. What am I doing wrong? And how can I fix it?

Your help on this would be highly appreciated. Thanks a lot!

Aly


title: Pooling over channels in pytorch

body:
In tensorflow, I can pool over the depth dimension which would reduce the channels and leave the spatial dimensions unchanged. I'm trying to do the same in pytorch but the documentation seems to say pooling can only be done over the height and width dimensions. Is there a way I can pool over channels in pytorch?
I've a tensor of shape [1,512,50,50] I'm trying to use pooling to bring the number of channels down to 3.
I saw this question but did not find the answer helpful.


title: Pytorch dataloader Transforms tensor error

body:
[Unable to access the pytorch dataloader values for use
TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found object
Error can be reproduced by running https://jovian.ml/pravin-bnmit/cxr/ on kaggle


title: Application of nn.Linear layer in pytorch on additional dimentions

body: How is the fully-connected layer (nn.Linear) in pytorch applied on "additional dimensions"? The documentation says, that it can be applied to connect a tensor (N,*,in_features) to (N,*,out_features), where N in the number of examples in a batch, so it is irrelevant, and * are those "additional" dimensions. Does it mean that a single layer is trained using all possible slices in the additional dimensions or are separate layers trained for each slice or something yet different?


title: When to use individual optimizers in PyTorch?

body: The example given here uses two optimizers for encoder and decoder individually. Why? And when to do like that?


title: Difference between hidden dimension and n_layers in rnn using pytorch

body: I am stuck between hidden dimension and n_layers. What I understood so far, is that n_layers in the parameters of RNN using pytorch, is number of hidden layers. If n_layers represents the number if hidden layers than what is hidden dimension?


title: How to create variable names in loop for layers in pytorch neural network

body:
I am implementing a straightforward feedforward neural newtork in PyTorch. However I am wondern if theres a nicer way to add a flexible amount of layer to the network? Maybe by naming them during a loop, but i heard thats impossible?

Currently I am doing it like this

import torch
import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):

    def __init__(self, input_dim, output_dim, hidden_dim):
        super(Net, self).__init__()
        self.input_dim = input_dim
        self.output_dim = output_dim
        self.hidden_dim = hidden_dim
        self.layer_dim = len(hidden_dim)
        self.fc1 = nn.Linear(self.input_dim, self.hidden_dim[0])
        i = 1
        if self.layer_dim > i:
            self.fc2 = nn.Linear(self.hidden_dim[i-1], self.hidden_dim[i])
            i += 1
        if self.layer_dim > i:
            self.fc3 = nn.Linear(self.hidden_dim[i-1], self.hidden_dim[i])
            i += 1
        if self.layer_dim > i:
            self.fc4 = nn.Linear(self.hidden_dim[i-1], self.hidden_dim[i])
            i += 1
        if self.layer_dim > i:
            self.fc5 = nn.Linear(self.hidden_dim[i-1], self.hidden_dim[i])
            i += 1
        if self.layer_dim > i:
            self.fc6 = nn.Linear(self.hidden_dim[i-1], self.hidden_dim[i])
            i += 1
        if self.layer_dim > i:
            self.fc7 = nn.Linear(self.hidden_dim[i-1], self.hidden_dim[i])
            i += 1
        if self.layer_dim > i:
            self.fc8 = nn.Linear(self.hidden_dim[i-1], self.hidden_dim[i])
            i += 1
        self.fcn = nn.Linear(self.hidden_dim[-1], self.output_dim)

    def forward(self, x):
        # Max pooling over a (2, 2) window
        x = F.relu(self.fc1(x))
        i = 1
        if self.layer_dim > i:
            x = F.relu(self.fc2(x))
            i += 1
        if self.layer_dim > i:
            x = F.relu(self.fc3(x))
            i += 1
        if self.layer_dim > i:
            x = F.relu(self.fc4(x))
            i += 1
        if self.layer_dim > i:
            x = F.relu(self.fc5(x))
            i += 1
        if self.layer_dim > i:
            x = F.relu(self.fc6(x))
            i += 1
        if self.layer_dim > i:
            x = F.relu(self.fc7(x))
            i += 1
        if self.layer_dim > i:
            x = F.relu(self.fc8(x))
            i += 1
        x = F.softmax(self.fcn(x))
        return x


title: CUDNN_STATUS_MAPPING_ERROR when training with pose2body

body:
I'm trying to train https://github.com/NVIDIA/vid2vid. I'm...

...executing with pretty much the vanilla parametrization shown in the readme, I had to change the number of GPUs though and increased the number of threads for reading the dataset. Command:

python train.py \
--name pose2body_256p  \
--dataroot datasets/pose  \
--dataset_mode pose  \
--input_nc 6  \
--num_D 2  \
--resize_or_crop ScaleHeight_and_scaledCrop  \
--loadSize 384  \
--fineSize 256  \
--gpu_ids 0,1  \
--batchSize 1  \
--max_frames_per_gpu 3  \
--no_first_img  \
--n_frames_total 12  \
--max_t_step 4  \
--nThreads 6
...training on the supplied example datasets.
...running a docker container built with the scripts in vid2vid/docker, e. g. with CUDA 9.0 and CUDNN 7.
...using two NVIDIA V100 GPUs.

Whenever I start training the script crashes after a couple of minutes with the message RuntimeError: CUDNN_STATUS_MAPPING_ERROR. Full error message:

Traceback (most recent call last):
  File "train.py", line 329, in <module>
    train()
  File "train.py", line 104, in train
    fake_B, fake_B_raw, flow, weight, real_A, real_Bp, fake_B_last = modelG(input_A, input_B, inst_A, fake_B_last)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/parallel/data_parallel.py", line 114, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/parallel/data_parallel.py", line 124, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/parallel/parallel_apply.py", line 65, in parallel_apply
    raise output
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/parallel/parallel_apply.py", line 41, in _worker
    output = module(*input, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/vid2vid/models/vid2vid_model_G.py", line 130, in forward
    fake_B, fake_B_raw, flow, weight = self.generate_frame_train(netG, real_A_all, fake_B_prev, start_gpu, is_first_frame)
  File "/vid2vid/models/vid2vid_model_G.py", line 175, in generate_frame_train
    fake_B_feat, flow_feat, fake_B_fg_feat, use_raw_only)
  File "/vid2vid/models/networks.py", line 171, in forward
    downsample = self.model_down_seg(input) + self.model_down_img(img_prev)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/container.py", line 91, in forward
    input = module(input)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/conv.py", line 301, in forward
    self.padding, self.dilation, self.groups)
RuntimeError: CUDNN_STATUS_MAPPING_ERROR

From reading the issues in the vid2vid using two V100 should work with this setup. The error also occurs if CUDA 8/CUDNN 6 are used. I checked the flags but haven't found any indication of further necessary changes to the arguments supplied to train.py.

Any ideas on how to solve (or work around) this?


title: how to check whether a certain number is in the Pytorch tensor?

body:
for a Pytorch tensor A:

A = tensor([1,0,0],
           [0,0,0])

is there way I can check whether the number 1 is an element of the tensor A?
like is there a pytorch function that returns True is 1 is an element of A, and returns False if 1 is not an element of A?

Thank you,


title: BERT sentence embedding by summing last 4 layers

body:
I used Chris Mccormick tutorial on BERT using pytorch-pretained-bert to get a sentence embedding as follows:

tokenized_text = tokenizer.tokenize(marked_text)
indexed_tokens = tokenizer.convert_tokens_to_ids(tokenized_text)
segments_ids = [1] * len(tokenized_text)
tokens_tensor = torch.tensor([indexed_tokens])
segments_tensors = torch.tensor([segments_ids])
model = BertModel.from_pretrained('bert-base-uncased')
model.eval()

with torch.no_grad():
    encoded_layers, _ = model(tokens_tensor, segments_tensors)
    # Holds the list of 12 layer embeddings for each token
    # Will have the shape: [# tokens, # layers, # features]
    token_embeddings = []

    # For each token in the sentence...
    for token_i in range(len(tokenized_text)):
        # Holds 12 layers of hidden states for each token
        hidden_layers = []

        # For each of the 12 layers...
        for layer_i in range(len(encoded_layers)):

                # Lookup the vector for `token_i` in `layer_i`
                vec = encoded_layers[layer_i][batch_i][token_i]

                hidden_layers.append(vec)

        token_embeddings.append(hidden_layers)

Now, I am trying to get the final sentence embedding by summing the last 4 layers as follows:

summed_last_4_layers = [torch.sum(torch.stack(layer)[-4:], 0) for layer in token_embeddings]

But instead of getting a single torch vector of length 768 I get the following:

[tensor([-3.8930e+00, -3.2564e+00, -3.0373e-01,  2.6618e+00,  5.7803e-01,
-1.0007e+00, -2.3180e+00,  1.4215e+00,  2.6551e-01, -1.8784e+00,
-1.5268e+00,  3.6681e+00, ...., 3.9084e+00]), tensor([-2.0884e+00, -3.6244e-01,  ....2.5715e+00]), tensor([ 1.0816e+00,...-4.7801e+00]), tensor([ 1.2713e+00,.... 1.0275e+00]), tensor([-6.6105e+00,..., -2.9349e-01])]

What did I get here? How do I pool the sum of the last for layers?

Thank you!


title: how would someone install pytorch using pip on python 3.9?

body:
when i tried the command the website used pip install torch==1.7.0+cpu torchvision==0.8.1+cpu torchaudio===0.7.0 -f https://download.pytorch.org/whl/torch_stable.html
and it returned the error code
ERROR: Could not find a version that satisfies the requirement torch==1.7.0+cpu (from versions: 0.1.2,
0.1.2.post1, 0.1.2.post2)
ERROR: No matching distribution found for torch==1.7.0+cpu

even when i change the code to satisfy the requirement it comes up with more errors like
ModuleNotFoundError: No module named 'tools.setup_helpers'


title: PyTorch: The number of sizes provided (0) must be greater or equal to the number of dimensions in the tensor (1)

body:
I'm trying to convert a CPU model to GPU using Pytorch, but I'm running into issues. I'm running this on Colab and I'm sure that Pytorch detects a GPU. This is a deep Q network (RL).

I declare my network as: Q = Q_Network(input_size, hidden_size, output_size).to(device)

I ran into an issue when I tried to pass arguments through the network (It expected type cuda but got type cpu) so I add .to(device):

batch = np.array(shuffled_memory[i:i+batch_size])
b_pobs = np.array(batch[:, 0].tolist(), dtype=np.float32).reshape(batch_size, -1)
b_pact = np.array(batch[:, 1].tolist(), dtype=np.int32)
b_reward = np.array(batch[:, 2].tolist(), dtype=np.int32)
b_obs = np.array(batch[:, 3].tolist(), dtype=np.float32).reshape(batch_size, -1)
b_done = np.array(batch[:, 4].tolist(), dtype=np.bool)

q = Q(torch.from_numpy(b_pobs).to(device))
q_ = Q_ast(torch.from_numpy(b_obs).to(device))

maxq = torch.max(q_.data,axis=1)
target = copy.deepcopy(q.data)

for j in range(batch_size):
    print(target[j, b_pact[j]].shape) # torch.Size([])
    target[j, b_pact[j]] = b_reward[j]+gamma*maxq[j]*(not b_done[j]) #I run into issues here

Here is the error:

RuntimeError: expand(torch.cuda.FloatTensor{[50]}, size=[]): the number of sizes provided (0) must be greater or equal to the number of dimensions in the tensor (1)


I finetune a BERT model using hugging face transformer library and train it in GPU in the cloud. Then I save the model and tokenizer like below:

model.save_pretrained('/saved_model/')
torch.save(best_model.state_dict(), '/saved_model/model')
tokenizer.save_pretrained('/saved_model/')

I download the saved_model directory in my computer. Then I load the model/tokenizer like below in my computer

import torch
from transformers import *
tokenizer = BertTokenizer.from_pretrained('./saved_model/')
config = BertConfig('./saved_model/config.json')
model = BertModel(config)
model.load_state_dict(torch.load('./saved_model/pytorch_model.bin', map_location=torch.device('cpu')))
model.eval()

But it throws below error for the model.load_state_dict line

RuntimeError: Error(s) in loading state_dict for BertModel:
    Missing key(s) in state_dict:

It lists a bunch of keys that are apparently missing from the state_dict.

I am new to pytorch and not sure what is going on. Most likely I am not saving the model the right way.

Please suggest.


title: cannot import torch audio ' No audio backend is available.'

body:
import torchaudio

When I just try to import torch audio on Pycharm, I have this error
61: UserWarning: No audio backend is available.

warnings.warn('No audio backend is available.')


title: vgg16 in pytorch got size error

body:
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.autograd import Variable
from torchvision import datasets, transforms, models
import time
import argparse
import os

batch_size = 64

train_dataset = datasets.CIFAR10(root='./data/cifar10/',
                                 train=True,
                                 transform=transforms.ToTensor(),
                                 download=True)
test_dataset = datasets.CIFAR10(root='./data/cifar10/',
                                train=False,
                                transform=transforms.ToTensor())

train_loader = torch.utils.data.DataLoader(dataset=train_dataset,
                                           batch_size=batch_size,
                                           shuffle=True)

test_loader = torch.utils.data.DataLoader(dataset=test_dataset,
                                          batch_size=batch_size,
                                          shuffle=False)

classes = ('plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck')

class Vgg16(nn.Module):
    def __init__(self, num_classes=10):
        super(Vgg16, self).__init__()
        self.features = nn.Sequential(
            nn.Conv2d(3, 64, kernel_size=3, padding=1),
            nn.ReLU(True),
            nn.Conv2d(64, 64, kernel_size=3, padding=1),
            nn.ReLU(True),
            nn.MaxPool2d(kernel_size=2, stride=2, dilation=1),

            nn.Conv2d(64, 128, kernel_size=3, padding=1),
            nn.ReLU(True),
            nn.Conv2d(128, 128, kernel_size=3, padding=1),
            nn.ReLU(True),
            nn.MaxPool2d(kernel_size=2, stride=2, dilation=1),

            nn.Conv2d(128, 256, kernel_size=3, padding=1),
            nn.ReLU(True),
            nn.Conv2d(256, 256, kernel_size=3, padding=1),
            nn.ReLU(True),
            nn.Conv2d(256, 256, kernel_size=3, padding=1),
            nn.ReLU(True),
            nn.MaxPool2d(kernel_size=2, stride=2, dilation=1),

            nn.Conv2d(256, 512, kernel_size=3, padding=1),
            nn.ReLU(True),
            nn.Conv2d(512, 512, kernel_size=3, padding=1),
            nn.ReLU(True),
            nn.Conv2d(512, 512, kernel_size=3, padding=1),
            nn.ReLU(True),
            nn.MaxPool2d(kernel_size=2, stride=2, dilation=1),

            nn.Conv2d(512, 512, kernel_size=3, padding=1),
            nn.ReLU(True),
            nn.Conv2d(512, 512, kernel_size=3, padding=1),
            nn.ReLU(True),
            nn.Conv2d(512, 512, kernel_size=3, padding=1),
            nn.ReLU(True),
            nn.MaxPool2d(kernel_size=2, stride=2, dilation=1)
        )
        self.classifier = nn.Sequential(
            nn.Linear(25088, 4096),
            nn.ReLU(True),
            nn.Dropout(),
            nn.Linear(4096, 4096),
            nn.ReLU(True),
            nn.Dropout(),
            nn.Linear(4096, num_classes)
        )

    def forward(self, x):
        x = self.features(x)
        x = x.view(x.size(0), -1)
        x = self.classifier(x)
        return F.softmax(x)

model = Vgg16()

# print(model)
optimizer = optim.SGD(model.parameters(), lr=0.001, momentum=0.9)
# optimizer = optim.Adam(model.parameters(), lr=0.001)
criterion = nn.CrossEntropyLoss().cuda()

if torch.cuda.device_count() > 0:
    # os.environ["CUDA_VISIBLE_DEVICES"]= '0'
    print("USE", torch.cuda.device_count(), "GPUs!")
    model = nn.DataParallel(model)
else:
    print("USE ONLY CPU!")

if torch.cuda.is_available():
    model.cuda()

def train(epoch):
    model.train()
    for batch_idx, (data, target) in enumerate(train_loader):
        if torch.cuda.is_available():
            data, target = Variable(data.cuda()), Variable(target.cuda())
        else:
            data, target = Variable(data), Variable(target)
        optimizer.zero_grad()
        output = model(data)
        # loss = F.nll_loss(output, target)
        loss = criterion(output, target)
        loss.backward()
        optimizer.step()
        if batch_idx % 10 == 0:
            print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                epoch, batch_idx * len(data), len(train_loader.dataset),
                100. * batch_idx / len(train_loader), loss.data[0]))

def test():
    model.eval()
    test_loss = 0
    correct = 0
    for data, target in test_loader:
        if torch.cuda.is_available():
            data, target = Variable(data.cuda(), volatile=True), Variable(target.cuda())
        else:
            data, target = Variable(data, volatile=True), Variable(target)
        output = model(data)
        test_loss += F.nll_loss(output, target, size_average=False).data[0]
        pred = output.data.max(1, keepdim=True)[1]  # [0] : value, [1]: index
        correct += pred.eq(target.data.view_as(pred)).sum()
        test_loss /= len(test_loader.dataset)
        print('\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\n'.format(
            test_loss, correct, len(test_loader.dataset),
            100. * correct / len(test_loader.dataset)))

for epoch in range(0, 200):
    train(epoch)
    test()

When I run this codes.. this error occured.

  -> RuntimeError: size mismatch at /pytorch/torch/lib/THC/generic/THCTensorMathBlas.cu:243

When I print(model) in other vgg codes in pytorch, There is a 25088 input size of FC layer...
So, I tried to set this parameter 25088,There is size mismatch error.
When I change this input_size from 25088 to 512, there is no error but training is not works well.(Never changed loss in training process and Always have 10% accuracy in test process)
So I think this input size of FC layer is the problem.. What can I do in this situation?
Thanks in advance;


title: how to evaluate and get accuracy of a Feed forward neural network with pytorch

body:
I started using Pytorch and I'm currently working on a Project where I'm using a simple feed forward neural network for linear regression. The Problem is I didn't find anything in Pytorch that allows me to get the Accuracy of a linear regression Model as in Keras or in SKlearn. in keras it would be simple just by setting metrics=["accuracy"] inside the compile function. I searched in the docs and official website of Pytorch but I didn't find anything. seems that this API doesn't exist in Pytorch. I know that I can observe the loss during training or I can simply get the test loss and based on it I can know whether the loss decreased or not but I want to use that Keras Structure where I get the loss value and also an Accuracy value. the Keras way looks more clear. I also tried to implement an accuracy function using the r2_score from sklearn but it gave me wierd values:

criterion = nn.MSELoss()
optimizer = optim.SGD(model.parameters(), lr=1e-3, momentum=0.9)

def train(model, optimizer, loss_fn):
    def train_step(x, y):
        model.train()
        optimizer.zero_grad()
        out = model(x)
        loss = loss_fn(out, y)
        loss.backward()
        optimizer.step()
        return loss.item()
    return train_step

def fit(epochs=100):
    train_func = train(model, optimizer, criterion)
    count, total = 0, 0
    loss_list, accuracy_list, iters = [], [], []
    for e in range(epochs):
        for X, y in train_loader:
            loss = train_func(X, y)
            count += 1
            total += len(y)
            if count % 50 == 0:
                print("loss= ", loss)
                loss_list.append(loss)
                iters.append(total)

            if count % 100 == 0:
                model.eval()   # im not sure if we can do this in pytorch. I mean evaluating the model while training! it would be great if you tell me whether this is ok or not
                out = model(X)
                out = out.detach().numpy()
                y = y.detach().numpy()
                accuracy = r2_score(y, out)   # r2_score is the scikit learn r2 score function.
                print("accuracy = ", accuracy)   # here i get wierd values and it doesn't get better over time, in contrast the loss decreased over time
                accuracy_list.append(accuracy)

    return iters, loss_list, accuracy_list

I know how to implement an Accuracy function in case of Classification Problem because it is using discrete values. that is clear to me because the implementation is easy and clear. I must only look which correct prediction did the model made and then calculate accuracy. but in this Case I have continuous values so that's why I couldn't implement the function myself and it surprised me that Pytorch don't have a built in function for this. so could someone maybe tell me how to implement this or where to find an Implementation of it?

another thing is where to use the evaluation and where to set the model in evaluation mode by calling the eval function. should I use it during training like I did in my Code or should I train and then test after training and if I test during training should I call the eval function as I did there or it will affect the training when the loop goes back to training mode? another thing I didn't find it also in Pytorch which is Cross validation. how should I implement it in pytorch if there is no API for it like in Keras?


title: ValueError: expected 2D or 3D input (got 1D input) PyTorch

body:
class VAE(torch.nn.Module):

 def __init__(self, input_size, hidden_sizes, batch_size):

    super(VAE, self).__init__()

    self.input_size = input_size
    self.hidden_sizes = hidden_sizes
    self.batch_size = batch_size
    self.fc = torch.nn.Linear(input_size, hidden_sizes[0])
    self.BN = torch.nn.BatchNorm1d(hidden_sizes[0])
    self.fc1 = torch.nn.Linear(hidden_sizes[0], hidden_sizes[1])
    self.BN1 = torch.nn.BatchNorm1d(hidden_sizes[1])
    self.fc2 = torch.nn.Linear(hidden_sizes[1], hidden_sizes[2])
    self.BN2 = torch.nn.BatchNorm1d(hidden_sizes[2])
    self.fc3_mu = torch.nn.Linear(hidden_sizes[2], hidden_sizes[3])
    self.fc3_sig = torch.nn.Linear(hidden_sizes[2], hidden_sizes[3])

    self.fc4 = torch.nn.Linear(hidden_sizes[3], hidden_sizes[2])
    self.BN4 = torch.nn.BatchNorm1d(hidden_sizes[2])
    self.fc5 = torch.nn.Linear(hidden_sizes[2], hidden_sizes[1])
    self.BN5 = torch.nn.BatchNorm1d(hidden_sizes[1])
    self.fc6 = torch.nn.Linear(hidden_sizes[1], hidden_sizes[0])
    self.BN6 = torch.nn.BatchNorm1d(hidden_sizes[0])
    self.fc7 = torch.nn.Linear(hidden_sizes[0], input_size)

def sample_z(self, x_size, mu, log_var):

     eps = torch.randn(x_size, self.hidden_sizes[-1])
     return(mu + torch.exp(log_var/2) * eps)

 def forward(self, x):

    ###########
    # Encoder #
    ###########

    out1 = self.fc(x)
    out1 = nn.relu(self.BN(out1))
    out2 = self.fc1(out1)
    out2 = nn.relu(self.BN1(out2))
    out3 = self.fc2(out2)
    out3 = nn.relu(self.BN2(out3))
    mu = self.fc3_mu(out3)
    sig = nn.softplus(self.fc3_sig(out3))

    ###########
    # Decoder  #
    ###########

    # sample from the distro
    sample = self.sample_z(x.size(0), mu, sig)
    out4 = self.fc4(sample)
    out4 = nn.relu(self.BN4(out4))
    out5 = self.fc5(out4)
    out5 = nn.relu(self.BN5(out5))
    out6 = self.fc6(out5)
    out6 = nn.relu(self.BN6(out6))
    out7 = nn.sigmoid(self.fc7(out6))

    return(out7, mu, sig)

vae = VAE(input_size, hidden_sizes, batch_size)

vae.eval()

x_sample, z_mu, z_var = vae(X)

The error is:

File "VAE_LongTensor.py", line 200, in <module>
x_sample, z_mu, z_var = vae(X)
ValueError: expected 2D or 3D input (got 1D input)


title: pytorch model loading and prediction, AttributeError: 'dict' object has no attribute 'predict'

body:
model = torch.load('/home/ofsdms/san_mrc/checkpoint/best_v1_checkpoint.pt', map_location='cpu')
results, labels = predict_function(model, dev_data, version)

> /home/ofsdms/san_mrc/my_utils/data_utils.py(34)predict_squad()
-> phrase, spans, scores = model.predict(batch)
(Pdb) n
AttributeError: 'dict' object has no attribute 'predict'

How do I load  a saved checkpoint of pytorch model, and use the same for prediction. I have the model saved in .pt extension


title: Pytorch: join tensor and dictionary

body:
Using Pytorch.
Got the following tensor:

(tensor([[-0.0030, -7.6063, -7.6334, -7.7098, -8.3540]], device='cuda:0'),
 tensor([[ 14,  85,  45,  82,  15]], device='cuda:0'))

Need to join it with the following dictionary:

{'14': 'a',
 '100': 'b',
 '45': 'c',
 '33': 'd',
 '15': 'e'}

In order to get the following results:
'a','c','e'


title: Trouble Converting LSTM Pytorch Model to ONNX

body:
I am trying to export my LSTM Anomally-Detection Pytorch model to ONNX, but I'm experiencing errors. Please take a look at my code below.

Note: My data is shaped as [2685, 5, 6].
Here is where I define my model:

class Model(torch.nn.Module):
    def __init__(self, input_dim, hidden_dim, layer_dim):
        super(Model, self).__init__()
        self.hidden_dim = hidden_dim
        self.layer_dim = layer_dim
        self.lstm = nn.LSTM(input_dim, hidden_dim, layer_dim, batch_first=True)
        self.fc1 = torch.nn.Linear(hidden_dim, hidden_dim)
        self.fc2 = torch.nn.Linear(hidden_dim, input_dim)
    def forward(self, x):
        h0 = torch.zeros(self.layer_dim, x.size(0), self.hidden_dim).requires_grad_()
        c0 = torch.zeros(self.layer_dim, x.size(0), self.hidden_dim).requires_grad_()
        out, (hn, cn) = self.lstm(x, (h0.detach(), c0.detach()))
        out = self.fc1(out)
        out = self.fc2(out)
        return out

input_dim = 6
hidden_dim = 3
layer_dim = 2
model = Model(input_dim, hidden_dim, layer_dim)

I can train it and test with it fine. But the problem comes when exporting:

model.eval()
import torch.onnx
torch_out = torch.onnx.export(model,
                         torch.randn(2685, 5, 6),
                         "onnx_model.onnx",
                         export_params = True
                        )

But I have the following error:

LSTM(6, 3, num_layers=2, batch_first=True)
Linear(in_features=3, out_features=3, bias=True)
Linear(in_features=3, out_features=6, bias=True)
['input_1', 'l_lstm_LSTM', 'l_fc1_Linear', 'l_fc2_Linear', 'l_lstm_LSTM', 'l_fc1_Linear', 'l_fc2_Linear', 'l_lstm_LSTM', 'l_fc1_Linear', 'l_fc2_Linear', 'l_lstm_LSTM', 'l_fc1_Linear', 'l_fc2_Linear', 'l_lstm_LSTM', 'l_fc1_Linear', 'l_fc2_Linear', 'l_lstm_LSTM', 'l_fc1_Linear', 'l_fc2_Linear', 'l_lstm_LSTM', 'l_fc1_Linear', 'l_fc2_Linear', 'l_lstm_LSTM', 'l_fc1_Linear', 'l_fc2_Linear', 'l_lstm_LSTM', 'l_fc1_Linear', 'l_fc2_Linear', 'l_lstm_LSTM', 'l_fc1_Linear', 'l_fc2_Linear']

/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/onnx/symbolic.py:173: UserWarning: ONNX export failed on RNN/GRU/LSTM because batch_first not supported
  warnings.warn("ONNX export failed on " + op + " because " + msg + " not supported")

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-264-28c6c55537ab> in <module>()
     10                          torch.randn(2685, 5, 6),
     11                          "onnx_model.onnx",
---> 12                          export_params = True
     13                         )

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/onnx/__init__.py in export(*args, **kwargs)
     23 def export(*args, **kwargs):
     24     from torch.onnx import utils
---> 25     return utils.export(*args, **kwargs)
     26
     27

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/onnx/utils.py in export(model, args, f, export_params, verbose, training, input_names, output_names, aten, export_raw_ir, operator_export_type, opset_version, _retain_param_name, do_constant_folding, strip_doc_string)
    129             operator_export_type=operator_export_type, opset_version=opset_version,
    130             _retain_param_name=_retain_param_name, do_constant_folding=do_constant_folding,
--> 131             strip_doc_string=strip_doc_string)
    132
    133

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/onnx/utils.py in _export(model, args, f, export_params, verbose, training, input_names, output_names, operator_export_type, export_type, example_outputs, propagate, opset_version, _retain_param_name, do_constant_folding, strip_doc_string)
    367         if export_params:
    368             proto, export_map = graph._export_onnx(params_dict, opset_version, defer_weight_export, operator_export_type,
--> 369                                                    strip_doc_string)
    370         else:
    371             proto, export_map = graph._export_onnx({}, opset_version, False, operator_export_type, strip_doc_string)

RuntimeError: ONNX export failed: Couldn't export operator aten::lstm

Defined at:
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/modules/rnn.py(522): forward_impl
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/modules/rnn.py(539): forward_tensor
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/modules/rnn.py(559): forward
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/modules/module.py(481): _slow_forward
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/modules/module.py(491): __call__
<ipython-input-255-468cef410a2c>(14): forward
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/modules/module.py(481): _slow_forward
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/modules/module.py(491): __call__
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/jit/__init__.py(294): forward
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/modules/module.py(493): __call__
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/jit/__init__.py(231): get_trace_graph
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/onnx/utils.py(225): _trace_and_get_graph_from_model
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/onnx/utils.py(266): _model_to_graph
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/onnx/utils.py(363): _export
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/onnx/utils.py(131): export
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/onnx/__init__.py(25): export
<ipython-input-264-28c6c55537ab>(12): <module>
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/IPython/core/interactiveshell.py(2963): run_code
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/IPython/core/interactiveshell.py(2903): run_ast_nodes
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/IPython/core/interactiveshell.py(2785): _run_cell
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/IPython/core/interactiveshell.py(2662): run_cell
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/ipykernel/zmqshell.py(537): run_cell
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/ipykernel/ipkernel.py(208): do_execute
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/ipykernel/kernelbase.py(399): execute_request
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/ipykernel/kernelbase.py(233): dispatch_shell
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/ipykernel/kernelbase.py(283): dispatcher
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/tornado/stack_context.py(276): null_wrapper
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py(432): _run_callback
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py(480): _handle_recv
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py(450): _handle_events
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/tornado/stack_context.py(276): null_wrapper
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/tornado/platform/asyncio.py(117): _handle_events
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/asyncio/events.py(145): _run
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/asyncio/base_events.py(1432): _run_once
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/asyncio/base_events.py(422): run_forever
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/tornado/platform/asyncio.py(127): start
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/ipykernel/kernelapp.py(486): start
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/traitlets/config/application.py(658): launch_instance
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/ipykernel/__main__.py(3): <module>
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/runpy.py(85): _run_code
/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/runpy.py(193): _run_module_as_main

Graph we tried to export:
graph(%input.1 : Float(2685, 5, 6),
      %lstm.weight_ih_l0 : Float(12, 6),
      %lstm.weight_hh_l0 : Float(12, 3),
      %lstm.bias_ih_l0 : Float(12),
      %lstm.bias_hh_l0 : Float(12),
      %lstm.weight_ih_l1 : Float(12, 3),
      %lstm.weight_hh_l1 : Float(12, 3),
      %lstm.bias_ih_l1 : Float(12),
      %lstm.bias_hh_l1 : Float(12),
      %fc1.weight : Float(3, 3),
      %fc1.bias : Float(3),
      %fc2.weight : Float(6, 3),
      %fc2.bias : Float(6)):
  %13 : Long() = onnx::Constant[value={0}](), scope: Model
  %14 : Tensor = onnx::Shape(%input.1), scope: Model
  %15 : Long() = onnx::Gather[axis=0](%14, %13), scope: Model
  %16 : Long() = onnx::Constant[value={2}](), scope: Model
  %17 : Long() = onnx::Constant[value={3}](), scope: Model
  %18 : Tensor = onnx::Unsqueeze[axes=[0]](%16)
  %19 : Tensor = onnx::Unsqueeze[axes=[0]](%15)
  %20 : Tensor = onnx::Unsqueeze[axes=[0]](%17)
  %21 : Tensor = onnx::Concat[axis=0](%18, %19, %20)
  %22 : Float(2, 2685, 3) = onnx::ConstantOfShape[value={0}](%21), scope: Model
  %23 : Long() = onnx::Constant[value={0}](), scope: Model
  %24 : Tensor = onnx::Shape(%input.1), scope: Model
  %25 : Long() = onnx::Gather[axis=0](%24, %23), scope: Model
  %26 : Long() = onnx::Constant[value={2}](), scope: Model
  %27 : Long() = onnx::Constant[value={3}](), scope: Model
  %28 : Tensor = onnx::Unsqueeze[axes=[0]](%26)
  %29 : Tensor = onnx::Unsqueeze[axes=[0]](%25)
  %30 : Tensor = onnx::Unsqueeze[axes=[0]](%27)
  %31 : Tensor = onnx::Concat[axis=0](%28, %29, %30)
  %32 : Float(2, 2685, 3) = onnx::ConstantOfShape[value={0}](%31), scope: Model
  %33 : Long() = onnx::Constant[value={1}](), scope: Model/LSTM[lstm]
  %34 : Long() = onnx::Constant[value={2}](), scope: Model/LSTM[lstm]
  %35 : Double() = onnx::Constant[value={0}](), scope: Model/LSTM[lstm]
  %36 : Long() = onnx::Constant[value={0}](), scope: Model/LSTM[lstm]
  %37 : Long() = onnx::Constant[value={0}](), scope: Model/LSTM[lstm]
  %38 : Long() = onnx::Constant[value={1}](), scope: Model/LSTM[lstm]
  %input.2 : Float(2685!, 5!, 3), %40 : Float(2, 2685, 3), %41 : Float(2, 2685, 3) = aten::lstm(%input.1, %22, %32, %lstm.weight_ih_l0, %lstm.weight_hh_l0, %lstm.bias_ih_l0, %lstm.bias_hh_l0, %lstm.weight_ih_l1, %lstm.weight_hh_l1, %lstm.bias_ih_l1, %lstm.bias_hh_l1, %33, %34, %35, %36, %37, %38), scope: Model/LSTM[lstm]
  %42 : Float(3!, 3!) = onnx::Transpose[perm=[1, 0]](%fc1.weight), scope: Model/Linear[fc1]
  %43 : Float(2685, 5, 3) = onnx::MatMul(%input.2, %42), scope: Model/Linear[fc1]
  %44 : Float(2685, 5, 3) = onnx::Add(%43, %fc1.bias), scope: Model/Linear[fc1]
  %45 : Float(3!, 6!) = onnx::Transpose[perm=[1, 0]](%fc2.weight), scope: Model/Linear[fc2]
  %46 : Float(2685, 5, 6) = onnx::MatMul(%44, %45), scope: Model/Linear[fc2]
  %47 : Float(2685, 5, 6) = onnx::Add(%46, %fc2.bias), scope: Model/Linear[fc2]
  return (%47)

What does this mean? What should I do to export properly?


title: How to define several layers via a loop in __init__ for Pytorch?

body:
I am trying to define a multi-task model in Pytorch where I need a different set of layers for different tasks. I face problems in defining layers, especially if I use a for loop to store different layers in a list then I get an error from optimizer stating that model.parameters() is an empty list, which in fact it is.

Following is the code:

x_trains=[]
y_trains=[]

num_tasks=2
for i in range(num_tasks):
    x_trains.append(torch.from_numpy(np.random.rand(100,1,50)).float())
    y_trains.append(torch.from_numpy(np.array([np.random.randint(10) for i in range(100)])).long())
nb_classes=10

class Net(torch.nn.Module):
def __init__(self):
    super(Net, self).__init__()

    self.all_task_layers=[]
    for i in range(num_tasks):
        self.all_task_layers.append(nn.Conv1d(1, 128, 8))
        self.all_task_layers.append(nn.BatchNorm1d(128))
        self.all_task_layers.append(nn.Conv1d(128, 256, 5))
        self.all_task_layers.append(nn.BatchNorm1d(256))
        self.all_task_layers.append(nn.Conv1d(256, 128, 3))
        self.all_task_layers.append(nn.BatchNorm1d(128))
        self.all_task_layers.append(nn.Linear(128, nb_classes))
    #self.dict_layers_for_tasks[i][1]
    self.all_b1s=[]
    self.all_b2s=[]
    self.all_b3s=[]
    self.all_dense1s=[]

def forward(self, x_num_tasks):
    for i in range(0,len(self.all_task_layers),num_tasks):
        self.all_b1s.append(F.relu(self.all_task_layers[i+1](self.all_task_layers[i+0](x_num_tasks[i]))))

    for i in range(0,len(self.all_task_layers),num_tasks):
        self.all_b2s.append(F.relu(self.all_task_layers[i+3](self.all_task_layers[i+2](self.all_b1s[i]))))

    for i in range(0,len(self.all_task_layers),num_tasks):
        self.all_b3s.append(F.relu(self.all_task_layers[i+5](self.all_task_layers[i+4](self.all_b2s[i]))))

    for i in range(0,len(self.all_task_layers),num_tasks):
        self.all_dense1s.append(self.all_task_layers[i+6](self.all_b3s[i]))

    return self.all_dense1s

model = Net()
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)

losses=[]
for t in range(50):
    y_preds= model(x_trains)
    optimizer.zero_grad()
    for i in range(num_tasks):
        loss=criterion(y_preds[i], y_trains[i])
        losses.append(loss)
        print(t,i,loss.item())
        loss.backward(retain_graph=True)
        optimizer.step()

The same model if I try to initialize the layers for the two tasks with two sets of Conv->BatchNorm->Conv->BatchNorm->Conv->BatchNorm->GlobalAveragePooling->Linear then it would work. But if I have let’s say 5 tasks, then it can get pretty messy, and that is why I created a list of layers, and indexed them. For example, the first 7 layers are for task 1 and then the last 7 for task 2. But then the model.parameters() is giving me an empty list. How else could I do it? Or is there a simple fix, I am overlooking?


I'm getting the following error when training a model with PyTorch:

CuDNNError: 8: b'CUDNN_STATUS_EXECUTION_FAILED'

This happens after I get an "out of memory" error and I then restart the training script.

RuntimeError: cuda runtime error (2) : out of memory

The out of memory is because sometimes I sample a network configuration that doesn't fit on GPU (for hyperparameter optimization). This is fine. What I would like to do is simply re-start my training, but this triggers the first error.

I can get rid of this error by restarting jupyter lab and reloading the data and preprocessing. (The behavior is the same in jupyter notebook). If I just reload my data and re-do the processing, I still get the CUDNN_STATUS_EXECUTION_FAILED error.

While I can get rid of the error by restarting the notebook, this is tedious and hard to automate, so I would like to understand better that the CUDNN_STATUS_EXECUTION_FAILED error means so I can avoid it.

Edit: Sometimes, I can re-start the script after the out-of-memory error without triggering the CUDNN_STATUS_EXECUTION_FAILED error. I would guess the condition for this is that I wait a long time after the memory error (but I'm not sure, this is hard to reproduce). My point is that the CUDNN_STATUS_EXECUTION_FAILED does not happen systematically.

I'm running pytorch 0.3.1, cuda 8, python 3.5, pip 9.0.1 and ubuntu 16.04.


title: How to convert a list of different-sized tensors to a single tensor?

body:
I want to convert a list of tensors with different sizes to a single tensor.

I tried torch.stack, but it shows an error.

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-237-76c3ff6f157f> in <module>
----> 1 torch.stack(t)

RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 0. Got 5 and 6 in dimension 1 at C:\w\1\s\tmp_conda_3.7_105232\conda\conda-bld\pytorch_1579085620499\work\aten\src\TH/generic/THTensor.cpp:612

My list of tensors:

[tensor([-0.1873, -0.6180, -0.3918, -0.5849, -0.3607]),
 tensor([-0.6873, -0.3918, -0.5849, -0.9768, -0.7590, -0.6707]),
 tensor([-0.6686, -0.7022, -0.7436, -0.8231, -0.6348, -0.4040, -0.6074, -0.6921])]

I have also tried this in a different way, instead of tensors, I used a lists of these individual tensors and tried to make a tensor out of it. That also showed an error.

list: [[-0.18729999661445618, -0.6179999709129333, -0.3917999863624573, -0.5849000215530396, -0.36070001125335693], [-0.6873000264167786, -0.3917999863624573, -0.5849000215530396, -0.9768000245094299, -0.7590000033378601, -0.6707000136375427], [-0.6686000227928162, -0.7021999955177307, -0.7436000108718872, -0.8230999708175659, -0.6348000168800354, -0.40400001406669617, -0.6074000000953674, -0.6920999884605408]]

The error:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-245-489aea87f307> in <module>
----> 1 torch.FloatTensor(t)

ValueError: expected sequence of length 5 at dim 1 (got 6)

Apparently, it says, it is expecting the same length of lists if I am not wrong.

Can anyone help here?


title: How can I save PyTorch's DataLoader instance?

body: I want to save PyTorch's torch.utils.data.dataloader.DataLoader instance, so that I can continue training where I left off (keeping shuffle seed, states and everything).


title: torchtext BucketIterator minimum padding

body:
I'm trying to use the BucketIterator.splits function in torchtext to load data from csv files for use in a CNN. Everything works fine unless I have a batch that the longest sentence is shorter than the biggest filter size.

In my example I have filters of sizes 3, 4, and 5 so if the longest sentence doesn't have at least 5 words I get an error. Is there a way to let the BucketIterator dynamically set the padding for batches, but also set a minimum padding length?

This is my the code I am using for my BucketIterator:

train_iter, val_iter, test_iter = BucketIterator.splits((train, val, test), sort_key=lambda x: len(x.text), batch_size=batch_size, repeat=False, device=device)

I'm hoping there is a way to set a minimum length on the sort_key or something like that?

I tried this but it doesn't work:

FILTER_SIZES = [3,4,5]
train_iter, val_iter, test_iter = BucketIterator.splits((train, val, test), sort_key=lambda x: len(x.text) if len(x.text) >= FILTER_SIZES[-1] else FILTER_SIZES[-1], batch_size=batch_size, repeat=False, device=device)


title: How to convert a Tensorflow model checkpoint to Pytorch?

body:
I'm working with a Deep Learning model which has a ResNet-50 as backbone pretrained on ImageNet. The dataset that I'm using is the CUB-200, which is a set of 200 species of birds. For this reason I think that could be good to have a pretrained model on a dataset that has a similar domain and I found that the iNaturalist one could be the one that I'm looking for.

The problem is that I didn't find any pretrained model for Pytorch, but only a Tensorflow one here.

I tried to convert it using the MDNN library, but it needs also the '.ckpt.meta' file extend and I have only the '.ckpt'.

This is an example of how to use the MDNN library to convert a tf model to torch:

mmconvert -sf tensorflow -in imagenet_resnet_v2_152.ckpt.meta -iw imagenet_resnet_v2_152.ckpt --dstNode MMdnn_Output -df pytorch -om tf_to_pytorch_resnet_152.pth

Could anyone help me with it?


title: Pytorch dataloader, too many threads, too much cpu memory allocation

body:
I'm training a model using PyTorch. To load the data, I'm using torch.utils.data.DataLoader. The data loader is using a custom database I've implemented. A strange problem has occurred, every time the second for in the following code executes, the number of threads/processes increases and a huge amount of memory is allocated

    for epoch in range(start_epoch, opt.niter + opt.niter_decay + 1):
        epoch_start_time = time.time()
        if epoch != start_epoch:
            epoch_iter = epoch_iter % dataset_size
        for i, item in tqdm(enumerate(dataset, start=epoch_iter)):

I suspect the threads and memories of the previous iterators are not released after each __iter__() call to the data loader.
The allocated memory is close to the amount of memory allocated by the main thread/process when the threads are created. That is in the initial epoch the main thread is using 2GB of memory and so 2 threads of size 2GB are created. In the next epochs, 5GB of memory is allocated by the main thread and two 5GB threads are constructed (num_workers is 2).
I suspect that fork() function copies most of the context to the new threads.

The following is the Activity monitor showing the processes created by python, ZMQbg/1 are processes related to python.

My dataset used by the data loader has 100 sub-datasets, the __getitem__ call randomly selects one (ignoring the index). (the sub-datasets are AlignedDataset from pix2pixHD GitHub repository):


title: In PyTorch's "MaxPool2D", is padding added depending on "ceil_mode"?

body:
In MaxPool2D the padding is by default set to 0 and the ceil_mode is also set to False. Now, if I have an input of size 7x7 with kernel=2,stride=2 the output shape becomes 3x3, but when I use ceil_mode=True, it becomes 4x4, which makes sense because (if the following formula is correct), for 7x7 with output_shape would be 3.5x3.5 and depending on the ceil_mode it would be either 3x3 or 4x4.

Now, my question is, if the ceil_mode=True, does it change the default padding?

If it does, then how is it adding the padding i.e. is it adding the padding on left first or right, up first or down?


title: CUDA out of memory runtime error, anyway to delete pytorch "reserved memory"

body:
Like many othersm I'm getting a Runtime error of Cuda out of memory, but for some reason pytorch has reserved a large amount of it.
RuntimeError: CUDA out of memory. Tried to allocate 2.00 MiB (GPU 0; 6.00 GiB total capacity; 4.31 GiB already allocated; 844.80 KiB free; 4.71 GiB reserved in total by PyTorch)
I've tried the torch.cuda.empy_cache(), but this isn't working either and none of the other CUDA out of memory posts have helped me either.
When I've checked my gpu usage(nvidia-smi) before running my python program, it is plenty free.


title: 2d array as index in Pytorch

body:
I want to ‘grow’ a matrix using a set of rules.

Example of rules:

0->[[1,1,1],[0,0,0],[2,2,2]],
1->[[2,2,2],[2,2,2],[2,2,2]],
2->[[0,0,0],[0,0,0],[0,0,0]]

Example of growing a matrix:

[[0]]->[[1,1,1],[0,0,0],[2,2,2]]->
[[2,2,2,2,2,2,2,2,2],[2,2,2,2,2,2,2,2,2],[2,2,2,2,2,2,2,2,2],
[1,1,1,1,1,1,1,1,1],[0,0,0,0,0,0,0,0,0],[2,2,2,2,2,2,2,2,2],
[0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0]]

This is the code I’ve been trying to get to work in Pytorch

rules = np.random.randint(256,size=(10,256,3,3,3))
rules_tensor = torch.randint(256,size=(10,
            256, 3, 3, 3),
            dtype=torch.uint8, device = torch.device('cuda'))

rules = rules[0]
rules_tensor = rules_tensor[0]

seed = np.array([[128]])
seed_tensor = seed_tensor = torch.cuda.ByteTensor([[128]])

decode = np.empty((3**3, 3**3, 3))
decode_tensor = torch.empty((3**3,
                3**3, 3), dtype=torch.uint8,
                device = torch.device('cuda'))

for i in range(3):
    grow = seed
    grow_tensor = seed_tensor
    for j in range(1,4):
        grow = rules[grow,:,:,i].reshape(3**j,-1)
        grow_tensor = rules_tensor[grow_tensor,:,:,i].reshape(3**j,-1)

    decode[..., i] = grow
    decode_tensor[..., i] = grow_tensor

I can’t seem to select indices the same way as in Numpy in this line:

grow = rules[grow,:,:,i].reshape(3**j,-1)

Is there a way to do the following in Pytorch?


title: Passing tensorDataset or Dataloader to skorch

body:
I want to apply cross validation in Pytorch using skorch, so I prepared my model and my tensorDataset which returns (image,caption and captions_length) and so it has X and Y, so I'll not be able to set Y in the method

net.fit(dataset)

but when I tried that I got an error :

  ValueError: Stratified CV requires explicitly passing a suitable y

Here's part of my code:

start = time.time()
net = NeuralNetClassifier(
        decoder, criterion= nn.CrossEntropyLoss,
        max_epochs=args.epochs,
        lr=args.lr,
        optimizer=optim.SGD,
        device='cuda',  # uncomment this to train with CUDA
       )
net.fit(dataset, y=None)
end = time.time()


title: Need help regarding Transfer Learning a Faster RCNN ResNet50FPN in PyTorch

body:
I am new to PyTorch. I'm trying to use a pre-trained Faster RCNN torchvision.models.detection.fasterrcnn_resnet50_fpn() for object detection project. I have created a CustomDataset(Dataset) class to handle the custom dataset.

Here is the custom class implementation

class ToTensor(object):
    """Convert ndarrays in sample to Tensors."""

    def __call__(self, sample):
        image, landmarks = sample['image'], sample['meta_data']

        # swap color axis because
        # numpy image: H x W x C
        # torch image: C X H X W
        image = image.transpose((2, 0, 1))
        return {'image': torch.from_numpy(image),
                'meta_data': landmarks}

class CustomDataset(Dataset):
    """Custom Landmarks dataset."""
    def __init__(self, data_dir, root_dir, transform=None):
        """
        Args:
            data_dir (string): Directory with all the labels(json).
            root_dir (string): Directory with all the images.
            transform (callable, optional): Optional transform to be applied
                on a sample.
        """
        self.data_dir = data_dir
        self.root_dir = root_dir
        self.transform = transform

    def __len__(self):
        return len(os.listdir(self.data_dir))

    def __getitem__(self, idx):
        img_name = sorted(os.listdir(self.root_dir))[idx]
        image = io.imread(self.root_dir+'/'+img_name, plugin='matplotlib')
        json_file = sorted(os.listdir(self.data_dir))[idx]
        with open(self.data_dir+'/'+json_file) as f:
          meta_data = json.load(f)

        meta_data = meta_data['annotation']['object']
        sample = {'image': image, 'meta_data': meta_data}

        to_tensor = ToTensor()
        transformed_sample = to_tensor(sample)

        if self.transform:
            sample = self.transform(sample)

        return transformed_sample

Here is the train_model function

def train_model(model, criterion, optimizer, lr_scheduler, num_epochs=25):
    since = time.time()

    best_model = model
    best_acc = 0.0

    for epoch in range(num_epochs):
        print('Epoch {}/{}'.format(epoch, num_epochs - 1))
        print('-' * 10)

        # Each epoch has a training and validation phase
        for phase in ['train', 'test']:
            if phase == 'train':
                optimizer = lr_scheduler(optimizer, epoch)
                model.train()  # Set model to training mode
            else:
                model.eval()  # Set model to evaluate mode

            running_loss = 0.0
            running_corrects = 0

            for data in dset_loaders[phase]:
                # get the inputs

                inputs, labels = data['image'], data['meta_data']

                inputs= inputs.to(device) # ,

                # zero the parameter gradients
                optimizer.zero_grad()

                # forward
                outputs = model(inputs, labels)
                _, preds = torch.max(outputs.data, 1)
                loss = criterion(outputs, labels)

                # backward + optimize only if in training phase
                if phase == 'train':
                    loss.backward()
                    optimizer.step()

                # statistics
                running_loss += loss.item()
                running_corrects += torch.sum(preds == labels).item()

            epoch_loss = running_loss / dset_sizes[phase]
            epoch_acc = running_corrects / dset_sizes[phase]

            print('{} Loss: {:.4f} Acc: {:.4f}'.format(
                phase, epoch_loss, epoch_acc))

            # deep copy the model
            if phase == 'test' and epoch_acc > best_acc:
                best_acc = epoch_acc
                best_model = copy.deepcopy(model)

        print()

    time_elapsed = time.time() - since
    print('Training complete in {:.0f}m {:.0f}s'.format(
        time_elapsed // 60, time_elapsed % 60))
    print('Best val Acc: {:4f}'.format(best_acc))
    return best_model

While performing model_ft = train_model(model_ft, criterion, optimizer_ft, exp_lr_scheduler, num_epochs=25) I am getting "RuntimeError: _thnn_upsample_bilinear2d_forward not supported on CUDAType for Byte"


title: IndexError: Target -1 is out of bounds error in tabular learner fatai2

body:
Getting the below error when trying to fit a tabular_learner from fastai2 library.

used data loaders

learn = tabular_learner(dls, layers=[1000,500], metrics=accuracy)

learn.fit(30,1e-2)

IndexError                                Traceback (most recent call last)
<ipython-input-35-f0c57ab3748f> in <module>
----> 1 learn.fit(30,1e-2)

/mnt/c/fastai2/fastai2/learner.py in fit(self, n_epoch, lr, wd, cbs, reset_opt)
    191                         self.epoch=epoch;          self('begin_epoch')
    192                         self._do_epoch_train()
--> 193                         self._do_epoch_validate()
    194                     except CancelEpochException:   self('after_cancel_epoch')
    195                     finally:                       self('after_epoch')

/mnt/c/fastai2/fastai2/learner.py in _do_epoch_validate(self, ds_idx, dl)
    173             dl,old,has = change_attrs(dl, names, [False,False])
    174             self.dl = dl;                                    self('begin_validate')
--> 175             with torch.no_grad(): self.all_batches()
    176         except CancelValidException:                         self('after_cancel_validate')
    177         finally:

/mnt/c/fastai2/fastai2/learner.py in all_batches(self)
    141     def all_batches(self):
    142         self.n_iter = len(self.dl)
--> 143         for o in enumerate(self.dl): self.one_batch(*o)
    144
    145     def one_batch(self, i, b):

/mnt/c/fastai2/fastai2/learner.py in one_batch(self, i, b)
    149             self.pred = self.model(*self.xb);                self('after_pred')
    150             if len(self.yb) == 0: return
--> 151             self.loss = self.loss_func(self.pred, *self.yb); self('after_loss')
    152             if not self.training: return
    153             self.loss.backward();                            self('after_backward')

/mnt/c/fastai2/fastai2/layers.py in __call__(self, inp, targ, **kwargs)
    291         if targ.dtype in [torch.int8, torch.int16, torch.int32]: targ = targ.long()
    292         if self.flatten: inp = inp.view(-1,inp.shape[-1]) if self.is_2d else inp.view(-1)
--> 293         return self.func.__call__(inp, targ.view(-1) if self.flatten else targ, **kwargs)
    294
    295 # Cell

~/anaconda3/envs/py3/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--> 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

~/anaconda3/envs/py3/lib/python3.6/site-packages/torch/nn/modules/loss.py in forward(self, input, target)
    914     def forward(self, input, target):
    915         return F.cross_entropy(input, target, weight=self.weight,
--> 916                                ignore_index=self.ignore_index, reduction=self.reduction)
    917
    918

~/anaconda3/envs/py3/lib/python3.6/site-packages/torch/nn/functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction)
   2019     if size_average is not None or reduce is not None:
   2020         reduction = _Reduction.legacy_get_string(size_average, reduce)
-> 2021     return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
   2022
   2023

~/anaconda3/envs/py3/lib/python3.6/site-packages/torch/nn/functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)
   1836                          .format(input.size(0), target.size(0)))
   1837     if dim == 2:
-> 1838         ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
   1839     elif dim == 4:
   1840         ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)

IndexError: Target -1 is out of bounds.

Any clue would be much appreciated! Thanks


title: Libtorch: How to load an ONNX-model?

body: I need to load and run an ONNX-model in a C++ environment using Libtorch on Windows 10 (Visual Studio 2015, v140). Searching the web, there seem to be almost exclusivly instructions for how to do it in Python. Is there a well documented way/does anyone know how to this in C++?


title: shall I apply softmax before cross entropy?

body:
The pytorch tutorial (https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#sphx-glr-beginner-blitz-cifar10-tutorial-py) trains a convolutional neural network (CNN) on a CIFAR dataset.

    class Net(nn.Module):
        def __init__(self):
            super(Net, self).__init__()
            self.conv1 = nn.Conv2d(3, 6, 5)
            self.pool = nn.MaxPool2d(2, 2)
            self.conv2 = nn.Conv2d(6, 16, 5)
            self.fc1 = nn.Linear(16 * 5 * 5, 120)
            self.fc2 = nn.Linear(120, 84)
            self.fc3 = nn.Linear(84, 10)

        def forward(self, x):
            x = self.pool(F.relu(self.conv1(x)))
            x = self.pool(F.relu(self.conv2(x)))
            x = x.view(-1, 16 * 5 * 5)
            x = F.relu(self.fc1(x))
            x = F.relu(self.fc2(x))
            x = self.fc3(x)
            return x

The network looks good except that the very last layer fc3, which predicts the probability of belonging to 10 classes without a softmax. Shouldn't we apply a softmax first to make sure the output of the fc layer is between 0 and 1 and sum before calculating cross-entropy loss?

I tested this by applying the softmax and rerunning, butvthe accuracy dropped to around 35%. This seems counterintuitive. What is the explanation?


title: Is there a `tensor` operation or function in Pytorch that works like cv2.dilate in OpenCV?

body:
I built several masks through a network. These masks are stored in a torch.tensor variable. I would like to do a cv2.dilate like operation on every channel of the tensor.

I know there is a way that convert the tensor to numpy.ndarray and then apply cv2.dilate to every channel using a for loop. But since there are about 32 channels, this method might slow down the forward operation in the network.


title: Pytorch argsort ordered, with duplicate elements in the tensor

body:
I have a vector A = [0,1,2,3,0,0,1,1,2,2,3,3]. I need to sort it in an increasing matter such that it is listed in an ordered fashion and from that extract the argsort. To better explain this I need to sort A to such that it returns B = [0,4,5,1,6,7,2,8,9,3,10,11]. However, when I use pyotrch's torch.argsort(A) it returns B = [4,5,0,1,6,7,2,8,9,3,10,11].

I'm assuming the algorithm that does so cannot be controlled on my end. Is there anyway to approach this without introducing for loops? Such operation are part of my NN model and will cause performance issues if not done efficiently. Thanks!


title: How to Get n-Number of Images from torchvision.datasets.ImageFolder

body:
I am currently experimenting on CNN using PyTorch, and the task I want the model to accomplish is to classify images.

I understand that using torchvision.datasets.ImageFolder can help with loading all images from my training folder, according to each subfolders' names as the labels.

I plan on taking only n-number of images from the ImageFolder randomly, but as far as I know, there is no mechanism for ImageFolder to load n-number of images randomly, where n is any number ranging from 1 to all available images.

How can I do that? Thanks for any help


title: Computing gradients of intermediate nodes in PyTorch

body:
I'm trying to learn how autograd works in PyTorch. In the simple program below, I don't understand why gradients of loss w.r.t W1 and W2 are None.  As far as I understand from the documentation, W1 and W2 are volatile, therefore gradients cannot be computed.  Is that it? I mean, how I cannot take derivative of the loss w.r.t intermediate nodes? Can anyone explain me what I am missing here?

import torch
import torch.autograd as tau

W = tau.Variable(torch.FloatTensor([[0, 1]]), requires_grad=True)
a = tau.Variable(torch.FloatTensor([[2, 2]]), requires_grad=False)
b = tau.Variable(torch.FloatTensor([[3, 3]]), requires_grad=False)

W1 = W  + a * a
W2 = W1 - b * b * b
Z = W2 * W2

print 'W:', W
print 'W1:', W1
print 'W2:', W2
print 'Z:', Z

loss = torch.sum((Z - 3) * (Z - 3))
print 'loss:', loss

# free W gradient buffer in case you are running this cell more than 2 times
if W.grad is not None: W.grad.data.zero_()

loss.backward()
print 'W.grad:', W.grad

# all of them are None
print 'W1.grad:', W1.grad
print 'W2.grad:', W2.grad
print 'a.grad:', a.grad
print 'b.grad:', b.grad
print 'Z.grad:', Z.grad


title: How to asynchronously load and train batches to train a DeepLearning model?

body:
I have 3TB dataset and 64GB RAM and a 12 core CPU and one 12GB GPU. would like to train a deep learning model on this dataset. How do I have asynchronous load of batches and training of the model? I want to make sure disk load of data doesn't block training loop to be waiting for the new batch to load into memory.

I am not language dependent and the easiest library that can do this without friction wins but I prefer one of torch, pytorch, tensorflow.


title: volatile was removed and now had no effect use with.torch.no_grad() instread

body:
my torch program stopped at this point
I guess i can not use volatile=True
how should I change it and what is the reason to stop?
 and How should I change this code?

images = Variable(images.cuda())
targets = [Variable(ann.cuda(), volatile=True) for ann in targets]

  train.py:166: UserWarning: volatile was removed and now has no effect.
  Use with torch.no_grad(): instead.