title: How to Efficiently Find the Indices of Max Values in a Multidimensional Array of Matrices using Pytorch and/or Numpy

body:
Background
It is common in machine learning to deal with data of a high dimensionality. For example, in a Convolutional Neural Network (CNN) the dimensions of each input image may be 256x256, and each image may have 3 color channels (Red, Green, and Blue). If we assume that the model takes in a batch of 16 images at a time, the dimensionality of the input going into our CNN is [16,3,256,256]. Each individual convolutional layer expects data in the form [batch_size, in_channels, in_y, in_x], and all of these quantities often change layer-to-layer (except batch_size). The term we use for the matrix made up of the [in_y, in_x] values is feature map, and this question is concerned with finding the maximum value, and its index, in every feature map at a given layer.
Why do I want to do this? I want to apply a mask to every feature map, and I want to apply that mask centered at the max value in each feature map, and to do that I need to know where each max value is located. This mask application is done during both training and testing of the model, so efficiency is vitally important to keep computational times down. There are many Pytorch and Numpy solutions for finding singleton max values and indices, and for finding the maximum values or indices along a single dimension, but no (that I could find) dedicated and efficient built-in functions for finding the indices of maximum values along 2 or more dimensions at a time. Yes, we can nest functions that operate on a single dimension, but these are some of the least efficient approaches.
What I've Tried

I've looked at this Stackoverflow question, but the author is dealing with a special-case 4D array which is trivially squeezed to a 3D array. The accepted answer is specialized for this case, and the answer pointing to TopK is misguided because it not only operates on a single dimension, but would necessitate that k=1 given the question asked, thus devlolving to a regular torch.max call.
I've looked at this Stackoverflow question, but this question, and its answer, focus on looking through a single dimension.
I have looked at this Stackoverflow question, but I already know of the answer's approach as I independently formulated it in my own answer here (where I amended that the approach is very inefficient).
I have looked at this Stackoverflow question, but it does not satisfy the key part of this question, which is concerned with efficiency.
I have read many other Stackoverflow questions and answers, as well as the Numpy documentation, Pytorch documentation, and posts on the Pytorch forums.
I've tried implementing a LOT of varying approaches to this problem, enough that I have created this question so that I can answer it and give back to the community, and anyone who goes looking for a solution to this problem in the future.

Standard of Performance
If I am asking a question about efficiency I need to detail expectations clearly. I am trying to find a time-efficient solution (space is secondary) for the problem above without writing C code/extensions, and which is reasonably flexible (hyper specialized approaches aren't what I'm after). The approach must accept an [a,b,c,d] Torch tensor of datatype float32 or float64 as input, and output an array or tensor of the form [a,b,2] of datatype int32 or int64 (because we are using the output as indices).
Solutions should be benchmarked against the following typical solution:
max_indices = torch.stack([torch.stack([(x[k][j]==torch.max(x[k][j])).nonzero()[0] for j in range(x.size()[1])]) for k in range(x.size()[0])])


title: Pytorch pretrained model (VGG-19) same image gives slightly different class score in the final FC layer

body:
I am using pretrained vgg-19 from torch.vision module
I have the pre-processing of image data like below:

normalize = transforms.Normalize(
    mean=[0.485, 0.456, 0.406],
    std=[0.229, 0.224, 0.225]
)

preprocess = transforms.Compose([
   transforms.Scale(224),
   transforms.ToTensor(),
   normalize
])

The problem is if I pass an image like a tennis ball through the network and save all the 1000 class score from the final FC layer and pass the same image again after some time, the final FC layer i.e the class score is slightly changed. Although the image category detected by the network is correct. (its a tennis ball)

Is it normal to have the slightly different class score for the same image? I mean, does this suppose to happen? Or for the correctly implemented pre-trained model, the network should give you the exact same class score every time for the same image.


title: How can I convert numpy.ndarray having type object to torch.tensor?

body:
I'm trying to work on lstm in pytorch. It takes only tensors as the input. The data that I have is in the form of a numpy.object_ and if I convert this to a numpy.float, then it can be converted to tensor.

I checked the data type using print(type(array)) it gives class 'numpy.ndarray' as output and print(arr.dtype.type) gives class 'numpy.object_' as output.

Or is there any way to convert tuple directly to torch.tensor?


title: best_state changes with the model during training in pytorch

body:
I want to save the best model and then load it during the test. So I used the following method:

def train():
    #training steps …
    if acc > best_acc:
        best_state = model.state_dict()
        best_acc = acc
    return best_state

Then, in the main function I used:

model.load_state_dict(best_state)

to resume the model.

However, I found that best_state is always the same as the last state during training, not the best state. Is anyone know the reason and how to avoid it?

By the way, I know I can use torch.save(the_model.state_dict(), PATH) and then load the model by
the_model.load_state_dict(torch.load(PATH)).
However, I don’t want to save the parameters to file as train and test functions are in one file.


title: No matching distribution found for numpy (from torch==1.10)

body:
Working on Ubuntu 18.04 in a VM.

I have python3 installed at /usr/bin/python3.6 according to whereis python3.
When I run pip3 freeze, I see numpy installed, with version 1.15.4, and when I enter into python3 console, I can import numpy as np perfectly fine with no errors.

What I am trying to do now is to install PyTorch, specifically torch==1.1.0, without GPU / CUDA.
I got the following installation instructions from https://pytorch.org/get-started/locally/:

pip3 install https://download.pytorch.org/whl/cpu/torch-1.1.0-cp36-cp36m-linux_x86_64.whl
pip3 install https://download.pytorch.org/whl/cpu/torchvision-0.3.0-cp36-cp36m-linux_x86_64.whl

I downloaded these wheel files, and tried to run them from local directory as such:

/usr/bin/python3.6 -m pip install ~/Desktop/torch-1.1.0-cp36-cp36m-linux_x86_64.whl

and got the following error:

No matching distribution found for numpy (from torch==1.1.0)

I don't understand why this is the case, cause my numpy library is clearly already installed and working (READ ABOVE: I am able to import numpy in python3 console). Also, I had no issues installing PyTorch with GPU on my Windows machine, with torch 1.1.0 and numpy 1.15.4.

Alternatively, I tried to build PyTorch from source by moving into the PyTorch source directory and executing /usr/bin/python3 setup.py install, but it didn't work since I am not using a GPU, and CUDA is not installed. I am not able to find instructions on how I can build PyTorch from source without GPU in a Linux environment?

Can anyone kindly advise? Thank you!

Some additional information
(1) Tried to install PyTorch with the --no-dependencies flag as suggested by @Jan, and it installed, but I am not sure if Torch will really work perfectly.
(2) which pip3 points to /usr/bin/pip3
(3) pip3 show numpy tells me that the location is at /home/name/.local/lib/python3.6/site-packages


title: Getting error while trying to plot ROC for multiclass classification

body:
I am trying to plot ROC Curve for multiclass classification.I followed https://scikit-learn.org/stable/auto_examples/model_selection/plot_roc.html.
I used the below code to calculate y_test,and y_score

def test_epoch(net,test_loader):
 y_test =[]
 y_score =[]
 with torch.no_grad():
  for batch in test_loader:
    images, labels = batch['image'], batch['grade']
    images =Variable(images)
    labels= Variable(labels)
    target =F.one_hot(labels,5)
    outputs = net(images)
    _, predicted = torch.max(outputs.data, 1)
    c = (predicted == labels).squeeze().numpy()
    y_score.append(outputs.numpy())
    y_test.append(labels.numpy())
    return y_test,y_score

I seen that my y_test is array like below

y_test data>> [array([[0, 0, 1, 0, 0],
       [1, 0, 0, 0, 0],
       [0, 1, 0, 0, 0],
       [1, 0, 0, 0, 0],
       [1, 0, 0, 0, 0],
       [0, 0, 0, 1, 0],
       [0, 0, 1, 0, 0],
       [1, 0, 0, 0, 0],
       [0, 0, 1, 0, 0],
       [0, 0, 1, 0, 0],
       [0, 1, 0, 0, 0]

And y_score is like

[array([[ 0.30480504, -0.12213976,  0.09632117, -0.16465648, -0.44081157],[ 0.21797988, -0.09650452,  0.07616544, -0.12001953, -0.34972644],[ 0.3230184 , -0.13098559,  0.10277118, -0.17656785, -0.45888817],[ 0.38143447, -0.15880316,  0.12123139, -0.21719441, -0.5281661 ],[ 0.3427343 , -0.13945231,  0.11076729, -0.19657779, -0.4913683 ]

Whenever I called the function for plotting ROC curve

def plot_roc(y_test, y_score, N_classes):
    """
    compute ROC curve and ROC area for each class in each fold

    """

    fpr = dict()
    tpr = dict()
    roc_auc = dict()
    for i in range(N_classes):
        fpr[i], tpr[i], _ = roc_curve(np.array(y_test[:, i]),np.array(y_score[:, i]))
        roc_auc[i] = auc(fpr[i], tpr[i])
    # Compute micro-average ROC curve and ROC area
    fpr["micro"], tpr["micro"], _ = roc_curve(y_test.ravel(), y_score.ravel())
    roc_auc["micro"] = auc(fpr["micro"], tpr["micro"])

    # Compute macro-average ROC curve and ROC area

    # First aggregate all false positive rates
    all_fpr = np.unique(np.concatenate([fpr[i] for i in range(N_classes)]))

    # Then interpolate all ROC curves at this points
    mean_tpr = np.zeros_like(all_fpr)
    for i in range(N_classes):
        mean_tpr += interp(all_fpr, fpr[i], tpr[i])

    # Finally average it and compute AUC
    mean_tpr /= N_classes

    fpr["macro"] = all_fpr
    tpr["macro"] = mean_tpr
    roc_auc["macro"] = auc(fpr["macro"], tpr["macro"])

    # Plot all ROC curves
    plt.figure()
    plt.plot(fpr["micro"], tpr["micro"],
             label='micro-average ROC curve (area = {0:0.2f})'
                   ''.format(roc_auc["micro"]),
             color='deeppink', linestyle=':', linewidth=4)

    plt.plot(fpr["macro"], tpr["macro"],
             label='macro-average ROC curve (area = {0:0.2f})'
                   ''.format(roc_auc["macro"]),
             color='navy', linestyle=':', linewidth=4)

    colors = cycle(['aqua', 'darkorange', 'cornflowerblue'])
    for i, color in zip(range(N_classes), colors):
        plt.plot(fpr[i], tpr[i], color=color, lw=2,
                 label='ROC curve of class {0} (area = {1:0.2f})'
                       ''.format(i, roc_auc[i]))

    plt.plot([0, 1], [0, 1], 'k--', lw=2)
    plt.xlim([0.0, 1.0])
    plt.ylim([0.0, 1.05])
    plt.xlabel('False Positive Rate')
    plt.ylabel('True Positive Rate')
    plt.title('Some extension of Receiver operating characteristic to multi-class')
    plt.legend(loc="lower right")
    plt.show()

I am getting this error massages,

 Traceback (most recent call last):
  File "/home/Downloads/demo 3.py", line 405, in <module>
    plot_roc(y_test, y_score, 5)
  File "/home/Downloads/demo 3.py", line 225, in plot_roc
    fpr[i], tpr[i], _ = roc_curve(np.array(y_test[:, i]),np.array(y_score[:, i]))
TypeError: list indices must be integers or slices, not tuple

I could not understand how I will solve this problem.
I am highly appreciate any kind of help regarding this issue.


title: PyTorch RuntimeError: CUDA out of memory. Tried to allocate 14.12 GiB

body:
I have faced Cuda out of memory error for a simple fully connected layer model. I have tried torch.cuda.empty_cache() and gc.collect(). Also, I delete unnecessary variables by del and tried by reducing batch size. But the error does not resolved. Also, the error appears only for SUN datasets where 1440 test images are used to evaluate. But the code runs well for AWA2 datasets where no. of test images is 7913. I am using google colab here. I have used RTX 2060 too.
Here is the code snippet, where it gets error:
def euclidean_dist(x, y):
    # x: N x D
    # y: M x D
    torch.cuda.empty_cache()
    n = x.size(0)
    m = y.size(0)
    d = x.size(1)
    assert d == y.size(1)
    x = x.unsqueeze(1).expand(n, m, d)
    y = y.unsqueeze(0).expand(n, m, d)
    del n,m,d
    return torch.pow(x - y, 2).sum(2)

def compute_accuracy(test_att, test_visual, test_id, test_label):
    global s2v
    s2v.eval()
    with torch.no_grad():
        test_att = Variable(torch.from_numpy(test_att).float().to(device))
        test_visual = Variable(torch.from_numpy(test_visual).float().to(device))
        outpre = s2v(test_att, test_visual)
        del test_att, test_visual
        outpre = torch.argmax(torch.softmax(outpre, dim=1), dim=1)

    outpre = test_id[outpre.cpu().data.numpy()]

    #compute averaged per class accuracy
    test_label = np.squeeze(np.asarray(test_label))
    test_label = test_label.astype("float32")
    unique_labels = np.unique(test_label)
    acc = 0
    for l in unique_labels:
        idx = np.nonzero(test_label == l)[0]
        acc += accuracy_score(test_label[idx], outpre[idx])
    acc = acc / unique_labels.shape[0]
    return acc

The error is:
Traceback (most recent call last):   File "GBU_new_v2.py", line 234, in <module>
    acc_seen_gzsl = compute_accuracy(attribute, x_test_seen, np.arange(len(attribute)), test_label_seen)   File "GBU_new_v2.py", line 111, in compute_accuracy
    outpre = s2v(test_att, test_visual)   File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)   File "GBU_new_v2.py", line 80, in forward
    a1 = euclidean_dist(feat, a1)   File "GBU_new_v2.py", line 62, in euclidean_dist
    return torch.pow(x - y, 2).sum(2)#.sqrt() # return: N x M RuntimeError: CUDA out of memory. Tried to allocate 14.12 GiB (GPU 0;
15.90 GiB total capacity; 14.19 GiB already allocated; 669.88 MiB free; 14.55 GiB reserved in total by PyTorch)


title: Best way to copy weight from pretrained model to my newly defined model with skip branches in PyTorch?

body:
I want to fine-tuning the existing state-of-the-art models on ImageNet for my own task. My model is adapted from vgg network and it has several branches connected to different convolution layer output, kind of like the following:

But the definition of vgg model in torchvision is not so straight forward. It is composed of a feature module and a classifier module, both of them are nn.Sequential, which makes it hard to make a branch from the output of  intermediate convolution layers. So right now, I manually define the new model layers by layers and do not use nn.Sequential, just like this.

But there is problem. Now the newly defined model do share same weight with the original vgg network in some of the layers, however, the layer names do not correspond any more. Of course I can manually find the correspondence between the weight of the old and new models and copy the weight directly. I think it is cumbersome. Is there a simpler way to do copy the weight from old model to the new model?


title: Use pretrained pytorch model in C?

body:
So I have a pretrained pytorch model, I have saved both the model and the parameters just in case, but I need to use this model in C or C++ code, anybody know how I can do that?

Thanks a lot.


title: Pytorch no gradient calculated one for variable

body:
I defined 2 variables (D, ht) to received gradients. However after the loss function and backward() calculation, only D has gradients, ht does not have any gradients calculated. Can you please help me to understand why it is the case? thank you.

import torch.nn as nn
import torch
from torch.autograd import Variable, Function

x = Variable(torch.from_numpy(np.random.normal(0,1,(10,10))), requires_grad=False)  # original

# depends on size of the dictionary, number of atoms.
D = Variable(torch.from_numpy(np.random.normal(0,1,(500,10,10))), requires_grad=True)

# hx sparse representation
ht = Variable(torch.from_numpy(np.random.normal(0,1,(500,1,1))), requires_grad=True)
ld = 0.02
lr = 0.001 # learning rate

# optimizer
optimizer = torch.optim.SGD([ht,D], lr=lr, momentum=0.9)
optimizer.zero_grad()

loss_ht = 0.5*torch.norm((x-(D*ht).sum(dim=0)),p=2)**2
loss_ht.backward() # back propogation
optimizer.step() # update parameters

D has graidents, see below

D.grad.data

( 0 ,.,.) =
2.0307e+01 9.9208e+00 4.9194e+00 … -2.3104e+01 -2.6616e+01 -9.8070e-01
3.7742e+01 2.5255e+01 4.5286e+00 … -3.5697e+01 -1.1306e+01 1.9366e+01

ht does not have any gradients. but why? please help. see below.

ht.grad.data

AttributeError Traceback (most recent call last)
in ()
----> 1 ht.grad.data

AttributeError: ‘NoneType’ object has no attribute ‘data’


title: PyTorch model validation: The size of tensor a (32) must match the size of tensor b (13)

body:
I am a very beginner in case of machine learning. So for learning purpose I am trying to develop a simple CNN to classify chess pieces. The net already works and I can train it but I have a problem with my validation function.

I can't compare my prediction with my target_data because my prediction is only a tensor of size 13 while target.data is [batch_size]x13. I can't figure out where my mistake is. The PyTorch examples are almost all using this function to compare the prediction with the target data.

It would be really great if anybody could help me out here.

You can lookup the rest of the code here: https://github.com/michaelwolz/ChessML/blob/master/train.ipynb

def validate(model, validation_data, criterion):
    model.eval()
    loss = 0
    correct = 0

    for i in range(len(validation_data)):
        data, target = validation_data[i][0], validation_data[i][1]
        target = torch.Tensor(target)

        if torch.cuda.is_available():
            data = data.cuda()
            target = target.cuda()

        out = model(data)

        loss += criterion(out, target).item()

        _, prediction = torch.max(out.data, 1)
        correct += (prediction == target.data).sum().item()

    loss = loss / len(validation_data)
    print("###################################")
    print("Average loss:", loss)
    print("Accuracy:", 100. * correct / len(validation_data))
    print("###################################")

Error:

<ipython-input-6-6b21e2bfb8a6> in validate(model, validation_data,

criterion)
     17
     18         _, prediction = torch.max(out.data, 1)
---> 19         correct += (prediction == target.data).sum().item()
     20
     21     loss = loss / len(validation_data)

RuntimeError: The size of tensor a (32) must match the size of tensor b (13) at non-singleton dimension 1

Edit:
My labels look like this:

[1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

Each index represents one class.
The output of the torch.max() function seems to be the index of the class.
I don't understand how I could compare the index to the target_label. I mean I could just write a function which checks if there is a 1 at the predicted index but I think that my mistake is somewhere else.


title: BucketIterator not returning batches of correct size

body:
I'm implementing a simple LSTM language model in PyTorch, and wanted to check out the BucketIterator that is provided by torchtext.

It turns out that the batch that is returned has the size of my entire corpus, so I must be doing something wrong during its initialisation.

I've already got the BPTTIterator working, but as I want to be able to train on batches of complete sentences as well, I thought the BucketIterator should be the way to go.

I use the following setup, with my corpus a simple txt file containing sentences at each line.

field = Field(use_vocab=True, batch_first=True)
corpus = PennTreebank('project_2_data/train_lines.txt', field)
field.build_vocab(corpus)

iterator = BucketIterator(corpus,
                          batch_size=64,
                          repeat=False,
                          sort_key=lambda x: len(x.text),
                          sort_within_batch=True,
                          )

I expect a batch from this iterator to have the shape (batch_size, max_len), but it appends the entire corpus into 1 tensor of shape (1, corpus_size).

What am I missing in my setup?

Edit: it seems the PennTreebank object is not compatible with a BucketIterator (it contains only 1 Example as noted here http://mlexplained.com/2018/02/15/language-modeling-tutorial-in-torchtext-practical-torchtext-part-2/). Using a TabularDataset with only 1 Field got it working.

If someone has an idea how language modelling with padded sentence batches can be done in torchtext in a more elegant manner I'd love to hear it!


title: Add Training and Testing Accuracy to a Simple Neural Network in PyTorch

body:
I know this is a primitive question but what should I add in my code for it to output the training accuracy of the Neural Network in addition to the loss, I checked PyTorch tutorials and they show how to add training/testing accuracy in image classification but I do not know how to do that in my simple XOR solving NN, below is the code:

# Step 1: importing our dependencies
import torch
from torch.autograd import Variable
import numpy as np

# Our data
x = Variable(torch.Tensor([[0, 0, 1], [0, 1, 1], [1, 0, 1], [0, 1, 0], [1, 0, 0],
                           [1, 1, 1], [0, 0, 0]]))
y = Variable(torch.Tensor([[0], [1], [1], [1], [1], [0], [0]]))

# Step 2: building our class model
class NeuralNetwork(torch.nn.Module):
def __init__(self):
    super(NeuralNetwork, self).__init__()
    self.linear_ij = torch.nn.Linear(3, 4)
    self.linear_jk = torch.nn.Linear(4, 1)

def forward(self, x):
    matmul = self.linear_ij(x)
    activation = torch.sigmoid(matmul)
    matmul = self.linear_jk(activation)
    prediction = torch.sigmoid(matmul)
    return prediction

# Our model
model = NeuralNetwork()

# Constructing the loss function and the optimization algorithm
criterion = torch.nn.BCELoss(reduction='mean')
optimizer = torch.optim.SGD(model.parameters(), lr=1)

# Step 3: the training process
for epoch in range(10000):

    prediction = model(x)
    loss = criterion(prediction, y)

    if epoch % 1000 == 0 or epoch == 10000 - 1:
        print("epoch ", epoch, ",", "loss: ", loss.item())

    # Backpropagation process
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

and this is it what it gives as an output:

epoch  0 , loss:  0.6983293294906616
epoch  1000 , loss:  0.015215665102005005
epoch  2000 , loss:  0.0048239342868328094
epoch  3000 , loss:  0.00280318153090775
epoch  4000 , loss:  0.001963752554729581
epoch  5000 , loss:  0.0015071843517944217
epoch  6000 , loss:  0.0012211233843117952
epoch  7000 , loss:  0.0010254186345264316
epoch  8000 , loss:  0.000883264874573797
epoch  9000 , loss:  0.0007753585232421756
epoch  9999 , loss:  0.0006908221403136849

As for testing:

# Testing our model
model.eval()

x_test = Variable(torch.Tensor([[1, 1, 0], [0, 0, 1], [0, 1, 1]]))
y_test = Variable(torch.Tensor([[0], [0], [1]]))
y_pred = model(x_test)
print(model(x_test))

with the output:

tensor([[0.0026],
        [0.0011],
        [0.9991]], grad_fn=<SigmoidBackward>)


title: How to divide the dataset when it is distributed

body:
Now I want to divide a dataset into two parts: the train set and validation set. I know that on a single GPU I can do this using a sampler:

indices = list(range(len(train_data)))
train_loader = torch.utils.data.DataLoader(
      train_data, batch_size=args.batch_size,
      sampler=torch.utils.data.sampler.SubsetRandomSampler(indices[:split]),
      pin_memory=True, num_workers=2)

But when I want to train it in a parallel way using torch.distributed, I have to use another sampler, namely, sampler = torch.utils.data.distributed.DistributedSampler(train_data)

So how should I do to use the two samplers, so that I can divide the dataset and distribute it at the same time?

Thank you very much for any help!


title: Assign variable in pytorch

body:
I'd like to know if it is possble to the following code, but now using pytorch, where dtype = torch.cuda.FloatTensor. There's the code straight python (using numpy): Basically I want to get the value of x that produces the min value of fitness.

import numpy as np
import random as rand
xmax, xmin       = 5, -5
pop              = 30
x                = (xmax-xmin)*rand.random(pop,1)
y                = x**2
[minz, indexmin] = np.amin(y), np.argmin(y)
best             = x[indexmin]

This is my attempt to do it:

import torch
dtype = torch.cuda.FloatTensor
def fit (x):
    return  x**2
def main():
    pop              = 30
    xmax, xmin       = 5, -5
    x                = (xmax-xmin)*torch.rand(pop, 1).type(dtype)+xmin
    y                = fit(x)
    [miny, indexmin] = torch.min(y,0)
    best             = x[indexmin]
main()

The last part where I define the variable best as the value of x with index equal to indexmin it doesn't work. What am I doing wrong here.

The following messenge appears: RuntimeError:

expecting vector of indices at /opt/conda/conda-bld/pytorch_1501971235237/work/pytorch-0.1.12/torch/lib/THC/generic/THCTensorIndex.cu:405


title: Difference between Tensorflow's tf.keras.layers.Dense and PyTorch's torch.nn.Linear?

body:
I have a quick (and possibly silly) question about how Tensorflow defines its Linear layer. Within PyTorch, a Linear (or Dense) layer is defined as, y = x A^T + b where A and b are the weight matrix and bias vector for a Linear layer (see here).
However, I can't precisely find an equivalent equation for Tensorflow! Is it the same as PyTorch or is it just y = x A + b ?
Thank you in advance!


I am testing ZED Camera with the code on https://github.com/stereolabs/zed-pytorch. While running the final command: python zed_object_detection.py --config-file configs/caffe2/e2e_mask_rcnn_R_50_C4_1x_caffe2.yaml --min-image-size 256
I get the following error:
Traceback (most recent call last):
File "zed_object_detection.py", line 6, in
from predictor import COCODemo
File "/home/fypadmin/Desktop/23Apr_ZED/zed-pytorch/predictor.py", line 4, in
from torchvision import transforms as T
File "/home/fypadmin/anaconda3/envs/pytorch1/lib/python3.8/site-packages/torchvision/init.py", line 4, in
from torchvision import datasets
File "/home/fypadmin/anaconda3/envs/pytorch1/lib/python3.8/site-packages/torchvision/datasets/init.py", line 1, in
from .lsun import LSUN, LSUNClass
File "/home/fypadmin/anaconda3/envs/pytorch1/lib/python3.8/site-packages/torchvision/datasets/lsun.py", line 19, in
from .utils import verify_str_arg, iterable_to_str
File "/home/fypadmin/anaconda3/envs/pytorch1/lib/python3.8/site-packages/torchvision/datasets/utils.py", line 11, in
from torch._six import PY3
ImportError: cannot import name 'PY3' from 'torch._six' (/home/fypadmin/anaconda3/envs/pytorch1/lib/python3.8/site-packages/torch/_six.py)
I am new to ML and I am running pytorch 1.8.1. Looking forward to any help. Thanks


title: How to use torch.utils.tensorboard's SummaryWriter add_graph with dictionary outputs?

body:
I have a PyTorch model that inherits from nn.Module and that has a forward method which returns a dictionary containing multiple tensors (stored as values). When I try to log this model to TensorBoard using torch.utils.tensorboard's SummaryWriter.add_graph() method I receive the following error:

RuntimeError("Tracer cannot infer type of <output dictionary>

How can I use add_graph() when the model's forward() method returns a dictionary?

Edit 1: Complete stack trace up to error:

WARNING:root:This caffe2 python run does not have GPU support. Will run in CPU only mode.
Only tensors or tuples of tensors can be output from traced functions (getOutput at /pytorch/torch/csrc/jit/tracer.cpp:208)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x33 (0x7fc661bd4273 in /usr/local/lib/python3.6/dist-packages/torch/lib/libc10.so)
frame #1: torch::jit::tracer::TracingState::getOutput(c10::IValue const&) + 0x3e8 (0x7fc664cf77b8 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch.so)
frame #2: torch::jit::tracer::exit(std::vector<c10::IValue, std::allocator<c10::IValue> > const&) + 0x3c (0x7fc664cf7aec in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch.so)
frame #3: <unknown function> + 0x47b0f2 (0x7fc66e4820f2 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so)
frame #4: <unknown function> + 0x48f061 (0x7fc66e496061 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so)
frame #5: <unknown function> + 0x19da64 (0x7fc66e1a4a64 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so)
frame #6: _PyCFunction_FastCallDict + 0x35c (0x5674fc in /usr/bin/python3.6)
frame #7: /usr/bin/python3.6() [0x50abb3]
frame #8: _PyEval_EvalFrameDefault + 0x449 (0x50c5b9 in /usr/bin/python3.6)
frame #9: <unknown function> + 0x18d10 (0x7fc6965ded10 in /home/rylan/.PyCharmCE2019.3/system/cythonExtensions/_pydevd_frame_eval_ext/pydevd_frame_evaluator.cpython-36m-x86_64-linux-gnu.so)
frame #10: /usr/bin/python3.6() [0x508245]
frame #11: /usr/bin/python3.6() [0x50a080]
frame #12: /usr/bin/python3.6() [0x50aa7d]
frame #13: _PyEval_EvalFrameDefault + 0x449 (0x50c5b9 in /usr/bin/python3.6)
frame #14: <unknown function> + 0x18d10 (0x7fc6965ded10 in /home/rylan/.PyCharmCE2019.3/system/cythonExtensions/_pydevd_frame_eval_ext/pydevd_frame_evaluator.cpython-36m-x86_64-linux-gnu.so)
frame #15: /usr/bin/python3.6() [0x508245]
frame #16: /usr/bin/python3.6() [0x50a080]
frame #17: /usr/bin/python3.6() [0x50aa7d]
frame #18: _PyEval_EvalFrameDefault + 0x449 (0x50c5b9 in /usr/bin/python3.6)
frame #19: <unknown function> + 0x18d10 (0x7fc6965ded10 in /home/rylan/.PyCharmCE2019.3/system/cythonExtensions/_pydevd_frame_eval_ext/pydevd_frame_evaluator.cpython-36m-x86_64-linux-gnu.so)
frame #20: /usr/bin/python3.6() [0x508245]
frame #21: /usr/bin/python3.6() [0x50a080]
frame #22: /usr/bin/python3.6() [0x50aa7d]
frame #23: _PyEval_EvalFrameDefault + 0x449 (0x50c5b9 in /usr/bin/python3.6)
frame #24: <unknown function> + 0x18d10 (0x7fc6965ded10 in /home/rylan/.PyCharmCE2019.3/system/cythonExtensions/_pydevd_frame_eval_ext/pydevd_frame_evaluator.cpython-36m-x86_64-linux-gnu.so)
frame #25: /usr/bin/python3.6() [0x508245]
frame #26: /usr/bin/python3.6() [0x50a080]
frame #27: /usr/bin/python3.6() [0x50aa7d]
frame #28: _PyEval_EvalFrameDefault + 0x1220 (0x50d390 in /usr/bin/python3.6)
frame #29: <unknown function> + 0x18d10 (0x7fc6965ded10 in /home/rylan/.PyCharmCE2019.3/system/cythonExtensions/_pydevd_frame_eval_ext/pydevd_frame_evaluator.cpython-36m-x86_64-linux-gnu.so)
frame #30: /usr/bin/python3.6() [0x508245]
frame #31: _PyFunction_FastCallDict + 0x2e2 (0x509642 in /usr/bin/python3.6)
frame #32: /usr/bin/python3.6() [0x595311]
frame #33: /usr/bin/python3.6() [0x54a6ff]
frame #34: /usr/bin/python3.6() [0x551b81]
frame #35: _PyObject_FastCallKeywords + 0x19c (0x5aa6ec in /usr/bin/python3.6)
frame #36: /usr/bin/python3.6() [0x50abb3]
frame #37: _PyEval_EvalFrameDefault + 0x1220 (0x50d390 in /usr/bin/python3.6)
frame #38: <unknown function> + 0x18d10 (0x7fc6965ded10 in /home/rylan/.PyCharmCE2019.3/system/cythonExtensions/_pydevd_frame_eval_ext/pydevd_frame_evaluator.cpython-36m-x86_64-linux-gnu.so)
frame #39: /usr/bin/python3.6() [0x509d48]
frame #40: /usr/bin/python3.6() [0x50aa7d]
frame #41: _PyEval_EvalFrameDefault + 0x449 (0x50c5b9 in /usr/bin/python3.6)
frame #42: <unknown function> + 0x18d10 (0x7fc6965ded10 in /home/rylan/.PyCharmCE2019.3/system/cythonExtensions/_pydevd_frame_eval_ext/pydevd_frame_evaluator.cpython-36m-x86_64-linux-gnu.so)
frame #43: /usr/bin/python3.6() [0x508245]
frame #44: /usr/bin/python3.6() [0x516915]
frame #45: /usr/bin/python3.6() [0x50a8af]
frame #46: _PyEval_EvalFrameDefault + 0x449 (0x50c5b9 in /usr/bin/python3.6)
frame #47: <unknown function> + 0x18d10 (0x7fc6965ded10 in /home/rylan/.PyCharmCE2019.3/system/cythonExtensions/_pydevd_frame_eval_ext/pydevd_frame_evaluator.cpython-36m-x86_64-linux-gnu.so)
frame #48: /usr/bin/python3.6() [0x508245]
frame #49: /usr/bin/python3.6() [0x50a080]
frame #50: /usr/bin/python3.6() [0x50aa7d]
frame #51: _PyEval_EvalFrameDefault + 0x449 (0x50c5b9 in /usr/bin/python3.6)
frame #52: <unknown function> + 0x18d10 (0x7fc6965ded10 in /home/rylan/.PyCharmCE2019.3/system/cythonExtensions/_pydevd_frame_eval_ext/pydevd_frame_evaluator.cpython-36m-x86_64-linux-gnu.so)
frame #53: /usr/bin/python3.6() [0x509d48]
frame #54: /usr/bin/python3.6() [0x50aa7d]
frame #55: _PyEval_EvalFrameDefault + 0x449 (0x50c5b9 in /usr/bin/python3.6)
frame #56: /usr/bin/python3.6() [0x508245]
frame #57: /usr/bin/python3.6() [0x50a080]
frame #58: /usr/bin/python3.6() [0x50aa7d]
frame #59: _PyEval_EvalFrameDefault + 0x449 (0x50c5b9 in /usr/bin/python3.6)
frame #60: /usr/bin/python3.6() [0x509d48]
frame #61: /usr/bin/python3.6() [0x50aa7d]
frame #62: _PyEval_EvalFrameDefault + 0x449 (0x50c5b9 in /usr/bin/python3.6)
frame #63: /usr/bin/python3.6() [0x508245]

Error occurs, No graph saved
Traceback (most recent call last):
  File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.6/dist-packages/torch/onnx/utils.py", line 42, in set_training
    yield
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/tensorboard/_pytorch_graph.py", line 234, in graph
    raise e
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/tensorboard/_pytorch_graph.py", line 229, in graph
    trace = torch.jit.trace(model, args)
  File "/usr/local/lib/python3.6/dist-packages/torch/jit/__init__.py", line 772, in trace
    check_tolerance, _force_outplace, _module_class)
  File "/usr/local/lib/python3.6/dist-packages/torch/jit/__init__.py", line 904, in trace_module
    module._c._create_method_from_trace(method_name, func, example_inputs, var_lookup_fn, _force_outplace)
RuntimeError: Only tensors or tuples of tensors can be output from traced functions (getOutput at /pytorch/torch/csrc/jit/tracer.cpp:208)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x33 (0x7fc661bd4273 in /usr/local/lib/python3.6/dist-packages/torch/lib/libc10.so)
frame #1: torch::jit::tracer::TracingState::getOutput(c10::IValue const&) + 0x3e8 (0x7fc664cf77b8 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch.so)
frame #2: torch::jit::tracer::exit(std::vector<c10::IValue, std::allocator<c10::IValue> > const&) + 0x3c (0x7fc664cf7aec in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch.so)
frame #3: <unknown function> + 0x47b0f2 (0x7fc66e4820f2 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so)
frame #4: <unknown function> + 0x48f061 (0x7fc66e496061 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so)
frame #5: <unknown function> + 0x19da64 (0x7fc66e1a4a64 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so)
frame #6: _PyCFunction_FastCallDict + 0x35c (0x5674fc in /usr/bin/python3.6)
frame #7: /usr/bin/python3.6() [0x50abb3]
frame #8: _PyEval_EvalFrameDefault + 0x449 (0x50c5b9 in /usr/bin/python3.6)
frame #9: <unknown function> + 0x18d10 (0x7fc6965ded10 in /home/rylan/.PyCharmCE2019.3/system/cythonExtensions/_pydevd_frame_eval_ext/pydevd_frame_evaluator.cpython-36m-x86_64-linux-gnu.so)
frame #10: /usr/bin/python3.6() [0x508245]
frame #11: /usr/bin/python3.6() [0x50a080]
frame #12: /usr/bin/python3.6() [0x50aa7d]
frame #13: _PyEval_EvalFrameDefault + 0x449 (0x50c5b9 in /usr/bin/python3.6)
frame #14: <unknown function> + 0x18d10 (0x7fc6965ded10 in /home/rylan/.PyCharmCE2019.3/system/cythonExtensions/_pydevd_frame_eval_ext/pydevd_frame_evaluator.cpython-36m-x86_64-linux-gnu.so)
frame #15: /usr/bin/python3.6() [0x508245]
frame #16: /usr/bin/python3.6() [0x50a080]
frame #17: /usr/bin/python3.6() [0x50aa7d]
frame #18: _PyEval_EvalFrameDefault + 0x449 (0x50c5b9 in /usr/bin/python3.6)
frame #19: <unknown function> + 0x18d10 (0x7fc6965ded10 in /home/rylan/.PyCharmCE2019.3/system/cythonExtensions/_pydevd_frame_eval_ext/pydevd_frame_evaluator.cpython-36m-x86_64-linux-gnu.so)
frame #20: /usr/bin/python3.6() [0x508245]
frame #21: /usr/bin/python3.6() [0x50a080]
frame #22: /usr/bin/python3.6() [0x50aa7d]
frame #23: _PyEval_EvalFrameDefault + 0x449 (0x50c5b9 in /usr/bin/python3.6)
frame #24: <unknown function> + 0x18d10 (0x7fc6965ded10 in /home/rylan/.PyCharmCE2019.3/system/cythonExtensions/_pydevd_frame_eval_ext/pydevd_frame_evaluator.cpython-36m-x86_64-linux-gnu.so)
frame #25: /usr/bin/python3.6() [0x508245]
frame #26: /usr/bin/python3.6() [0x50a080]
frame #27: /usr/bin/python3.6() [0x50aa7d]
frame #28: _PyEval_EvalFrameDefault + 0x1220 (0x50d390 in /usr/bin/python3.6)
frame #29: <unknown function> + 0x18d10 (0x7fc6965ded10 in /home/rylan/.PyCharmCE2019.3/system/cythonExtensions/_pydevd_frame_eval_ext/pydevd_frame_evaluator.cpython-36m-x86_64-linux-gnu.so)
frame #30: /usr/bin/python3.6() [0x508245]
frame #31: _PyFunction_FastCallDict + 0x2e2 (0x509642 in /usr/bin/python3.6)
frame #32: /usr/bin/python3.6() [0x595311]
frame #33: /usr/bin/python3.6() [0x54a6ff]
frame #34: /usr/bin/python3.6() [0x551b81]
frame #35: _PyObject_FastCallKeywords + 0x19c (0x5aa6ec in /usr/bin/python3.6)
frame #36: /usr/bin/python3.6() [0x50abb3]
frame #37: _PyEval_EvalFrameDefault + 0x1220 (0x50d390 in /usr/bin/python3.6)
frame #38: <unknown function> + 0x18d10 (0x7fc6965ded10 in /home/rylan/.PyCharmCE2019.3/system/cythonExtensions/_pydevd_frame_eval_ext/pydevd_frame_evaluator.cpython-36m-x86_64-linux-gnu.so)
frame #39: /usr/bin/python3.6() [0x509d48]
frame #40: /usr/bin/python3.6() [0x50aa7d]
frame #41: _PyEval_EvalFrameDefault + 0x449 (0x50c5b9 in /usr/bin/python3.6)
frame #42: <unknown function> + 0x18d10 (0x7fc6965ded10 in /home/rylan/.PyCharmCE2019.3/system/cythonExtensions/_pydevd_frame_eval_ext/pydevd_frame_evaluator.cpython-36m-x86_64-linux-gnu.so)
frame #43: /usr/bin/python3.6() [0x508245]
frame #44: /usr/bin/python3.6() [0x516915]
frame #45: /usr/bin/python3.6() [0x50a8af]
frame #46: _PyEval_EvalFrameDefault + 0x449 (0x50c5b9 in /usr/bin/python3.6)
frame #47: <unknown function> + 0x18d10 (0x7fc6965ded10 in /home/rylan/.PyCharmCE2019.3/system/cythonExtensions/_pydevd_frame_eval_ext/pydevd_frame_evaluator.cpython-36m-x86_64-linux-gnu.so)
frame #48: /usr/bin/python3.6() [0x508245]
frame #49: /usr/bin/python3.6() [0x50a080]
frame #50: /usr/bin/python3.6() [0x50aa7d]
frame #51: _PyEval_EvalFrameDefault + 0x449 (0x50c5b9 in /usr/bin/python3.6)
frame #52: <unknown function> + 0x18d10 (0x7fc6965ded10 in /home/rylan/.PyCharmCE2019.3/system/cythonExtensions/_pydevd_frame_eval_ext/pydevd_frame_evaluator.cpython-36m-x86_64-linux-gnu.so)
frame #53: /usr/bin/python3.6() [0x509d48]
frame #54: /usr/bin/python3.6() [0x50aa7d]
frame #55: _PyEval_EvalFrameDefault + 0x449 (0x50c5b9 in /usr/bin/python3.6)
frame #56: /usr/bin/python3.6() [0x508245]
frame #57: /usr/bin/python3.6() [0x50a080]
frame #58: /usr/bin/python3.6() [0x50aa7d]
frame #59: _PyEval_EvalFrameDefault + 0x449 (0x50c5b9 in /usr/bin/python3.6)
frame #60: /usr/bin/python3.6() [0x509d48]
frame #61: /usr/bin/python3.6() [0x50aa7d]
frame #62: _PyEval_EvalFrameDefault + 0x449 (0x50c5b9 in /usr/bin/python3.6)
frame #63: /usr/bin/python3.6() [0x508245]


title: Implementing SmoothL1Loss for specific case

body: I have been experimenting with L1 and MSE losses in Pytorch and noticing that L1Loss performs better because it is more robust to outliers. I discovered SmoothL1Loss which seems to be the best of both worlds. I understand that it behaves like MSELoss for error<1 and like L1Loss otherwise. My dataset only contains values between 0 and 1. Therefore the largest possible error is 1. Does this mean the function behaves identical to MSELoss? Is it possible adjust the threshold in anyway to work better for my problem?


title: RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED in backward pass in Torch

body:
I am working on a Pytorch code of Autoendcoder. The training set contains images of shape batchsizex3xheightxwidth. The model's code is too big to put here. Some codes are:
model = Network()
model.cuda()
#loss = nn.CrossEntropyLoss()
loss = nn.BCELoss()
optim = torch.optim.Adam(model.parameters())

I got the output of the forward pass simply with the line:
y_pred = model( torch.tensor(train, dtype= torch.float).cuda())

This gives output of shape batchsizex3xheight1xwidth1. I made the groundtruths of the same shape.
The loss is called by this line individually:
l = loss(y_pred, my_list_resized.cuda())

And it is working perfectly so far...
However, when I try it in loops for epochs:
for epoch in range(1):
    y_pred = model( torch.tensor(train, dtype= torch.float).cuda())
    l = loss(y_pred, my_list_resized.cuda())
    optim.zero_grad()
    l.backward()
    optim.step()

this gives the error in line l.backward():
RuntimeError                              Traceback (most recent call last)
<ipython-input-31-99d4765682f1> in <module>
----> 1 l.backward()

~\anaconda3\lib\site-packages\torch\tensor.py in backward(self, gradient, retain_graph, create_graph)
    219                 retain_graph=retain_graph,
    220                 create_graph=create_graph)
--> 221         torch.autograd.backward(self, gradient, retain_graph, create_graph)
    222
    223     def register_hook(self, hook):

~\anaconda3\lib\site-packages\torch\autograd\__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
    128         retain_graph = create_graph
    129
--> 130     Variable._execution_engine.run_backward(
    131         tensors, grad_tensors_, retain_graph, create_graph,
    132         allow_unreachable=True)  # allow_unreachable flag

RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED

individually running l.backward() that is outside the epcoh also gives this error. ANy solution please?


title: NVidia 1080ti eGPU Ubuntu 16.04.5 LTS - PyTorch / Tensorflow without root permissions

body:
I have some trouble in setting up my system properly. My system consists of:

Intel NUC7i7BNH
ASUS ROG with a NVidia 1080ti

My GPU is correctly detected by lspci:

06:00.0 VGA compatible controller: NVIDIA Corporation Device 1b06 (rev a1) (prog-if 00 [VGA controller])
        Subsystem: ASUSTeK Computer Inc. Device 85ea
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 128 bytes
        Interrupt: pin A routed to IRQ 18
        Region 0: Memory at c4000000 (32-bit, non-prefetchable) [size=16M]
        Region 1: Memory at a0000000 (64-bit, prefetchable) [size=256M]
        Region 3: Memory at b0000000 (64-bit, prefetchable) [size=32M]
        Region 5: I/O ports at 2000 [size=128]
        Expansion ROM at c5000000 [disabled] [size=512K]
        Capabilities: <access denied>
        Kernel driver in use: nvidia
        Kernel modules: nvidiafb, nouveau, nvidia_384_drm, nvidia_384

and with the commands below I installed drivers for the GPU, CUDA and cuDNN:

# Show thunderbolt port / authorize eGPU
$ cat /sys/bus/thunderbolt/devices/0-1/device_name
$ echo 1 | sudo tee -a /sys/bus/thunderbolt/devices/0-1/authorized

# eGPU - on Ubuntu 16.04 - nvidia-384
$ sudo ubuntu-drivers devices
$ sudo ubuntu-drivers autoinstall
$ sudo apt-get install nvidia-modprobe

# CUDA - Download CUDA from Nvidia - http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64/
$ wget https://developer.nvidia.com/compute/cuda/9.0/Prod/local_installers/cuda_9.0.176_384.81_linux-run
$ chmod +x cuda_9.0.176_384.81_linux-run
$ ./cuda_9.0.176_384.81_linux-run --extract=$HOME
$ sudo ./cuda-linux.9.0.176-22781540.run
$ sudo ./cuda-samples.9.0.176-22781540-linux.run
$ sudo bash -c "echo /usr/local/cuda/lib64/ > /etc/ld.so.conf.d/cuda.conf"
$ sudo ldconfig
# add :/usr/local/cuda/bin (including the ":") at the end of the PATH="/blah:/blah/blah" string (inside the quotes).
$ sudo nano /etc/environments

# CUDA samples - Check installation
$ cd /usr/local/cuda-9.0/samples
$ sudo make
$ /usr/local/cuda/samples/bin/x86_64/linux/release/deviceQuery

# cuDNN
$ wget http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64/libcudnn7_7.0.5.15-1+cuda9.0_amd64.deb
$ dpkg -i libcudnn7_7.0.5.15-1+cuda9.0_amd64.deb

# Authorization (Security risk!)
$ sudo nano /etc/udev/rules.d/99-local.rules
# Add
ACTION=="add", SUBSYSTEM=="thunderbolt", ATTR{authorized}=="0", ATTR{authorized}="1"

With sudo prime-select nvidia I can switch between the eGPU and the Intel integrated one. After a logout + login the eGPU seems to work (nvidia-smi):

Fri Sep 21 09:25:18 2018
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 384.130                Driver Version: 384.130                   |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 108...  Off  | 00000000:06:00.0 Off |                  N/A |
|  0%   47C    P0    84W / 275W |    214MiB / 11172MiB |      4%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      6734      G   /usr/lib/xorg/Xorg                           138MiB |
|    0      7081      G   kwin_x11                                      23MiB |
|    0      7084      G   /usr/bin/krunner                               2MiB |
|    0      7092      G   /usr/bin/plasmashell                          47MiB |
+-----------------------------------------------------------------------------+

The CUDA samples work also:

$ /usr/local/cuda/samples/bin/x86_64/linux/release/deviceQuery
/usr/local/cuda/samples/bin/x86_64/linux/release/deviceQuery Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 1 CUDA Capable device(s)

Device 0: "GeForce GTX 1080 Ti"
  CUDA Driver Version / Runtime Version          9.0 / 9.0
  CUDA Capability Major/Minor version number:    6.1
  Total amount of global memory:                 11172 MBytes (11715084288 bytes)
  (28) Multiprocessors, (128) CUDA Cores/MP:     3584 CUDA Cores
  GPU Max Clock rate:                            1683 MHz (1.68 GHz)
  Memory Clock rate:                             5505 Mhz
  Memory Bus Width:                              352-bit
  L2 Cache Size:                                 2883584 bytes
  Maximum Texture Dimension Size (x,y,z)         1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
  Maximum Layered 1D Texture Size, (num) layers  1D=(32768), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(32768, 32768), 2048 layers
  Total amount of constant memory:               65536 bytes
  Total amount of shared memory per block:       49152 bytes
  Total number of registers available per block: 65536
  Warp size:                                     32
  Maximum number of threads per multiprocessor:  2048
  Maximum number of threads per block:           1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
  Maximum memory pitch:                          2147483647 bytes
  Texture alignment:                             512 bytes
  Concurrent copy and kernel execution:          Yes with 2 copy engine(s)
  Run time limit on kernels:                     Yes
  Integrated GPU sharing Host Memory:            No
  Support host page-locked memory mapping:       Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Disabled
  Device supports Unified Addressing (UVA):      Yes
  Supports Cooperative Kernel Launch:            Yes
  Supports MultiDevice Co-op Kernel Launch:      Yes
  Device PCI Domain ID / Bus ID / location ID:   0 / 6 / 0
  Compute Mode:
     < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 9.0, CUDA Runtime Version = 9.0, NumDevs = 1
Result = PASS

Let's come to the problem: The following snippet works only, if I execute python with root permissions.

$ python
Python 3.6.6 |Anaconda, Inc.| (default, Jun 28 2018, 17:14:51)
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.cuda.is_available()
True
>>> torch.Tensor([0]).cuda()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: CUDA error: unknown error

vs.

$ sudo ~/miniconda3/envs/PyTorch/bin/python
Python 3.6.6 |Anaconda, Inc.| (default, Jun 28 2018, 17:14:51)
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.cuda.is_available()
True
>>> torch.Tensor([0]).cuda()
tensor([0.], device='cuda:0')

TLDR:
My CUDA installation seems to work and my eGPU is correctly recognized by the system. However in PyTorch (and also Tensorflow) an 'Unknown Error' is thrown, when I try to move same data onto the eGPU, although it is recognized by the frameworks. When I execute the same code with root permissions everything works like a charm. Does somebody know, where I have to adopt some permissions, such that the code can be executed as standard user?


title: Why is the derivative of f(x) with respect to 'x' 'x' and not 1 in pytorch?

body:
I am trying to understand pytorch's autograd in full and I stumbled with this: let f(x)=x, from basic maths we know that f'(x)=1, however when I do that exercise in pytorch I get that f'(x) = x.

z = torch.linspace(-1, 1, steps=5, requires_grad=True)
y = z
y.backward(z)
print("Z tensor is: {} \n Gradient of y with respect to z is: {}".format(z, z.grad))

I would expect to get a tensor of size 5 full of 1 but instead I get:

Z tensor is: tensor([-1.0000, -0.5000,  0.0000,  0.5000,  1.0000], requires_grad=True)
 Gradient of y with respect to z is: tensor([-1.0000, -0.5000,  0.0000,  0.5000,  1.0000])

Why is this the behavior of pytorch?


title: Why can't I use pytorch to multi-cuda calculate distance? (Initialization Error for default and bad value(s) in fds_in_keep for spawn)

body:
I'm using pytorch not for network, but just for GPU distance matrix calculation. When I just use one GPU, everything goes perfectly; but when it goes to multi-GPU, Error occurs.

Firstly, I got RuntimeError: CUDA error: initialization error, and I googled this Error and found a solution: add mp.set_start_method('spawn'). But new error occurred, this time the error was ValueError: bad value(s) in fds_to_keep, and I've not found a way to solve this error.

Now I'm confused and don't know how to solve it.

I'm using Ubuntu 16.04, Python 3.6.8 and pytorch 1.0.0.

For no mp.set_start_method('spawn'), the total traceback is:

Traceback (most recent call last):
  File "/home/mc/anaconda3/envs/Lab/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/home/mc/anaconda3/envs/Lab/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/home/mc/nfs/Project/Polaris_Toolkit/src/density.py", line 288, in run_kernel
    d_ts = gpr.pairwise_distance(data_i, data_j, device=device)
  File "/home/mc/nfs/Project/Polaris_Toolkit/src/gpr.py", line 126, in pairwise_distance
    data1, data2 = data1.cuda(device), data2.cuda(device)
RuntimeError: CUDA error: initialization error

Here device is the index of GPU.

And for mp.set_start_method('spawn'), the traceback is:

~/nfs/Project/Polaris_Toolkit/src/density.py in multi_gpu_density(trans_matrix, mmd, device, workers, threshold_coe, p, batch_size)
    438             param = (data, indice, threshold, d, d_ma_row, d_ma_col, p)
    439             pr = mp.Process(target=run_kernel, args=param)
--> 440             pr.start()
    441             jobs.append(pr)
    442         for job in jobs:

~/anaconda3/envs/Lab/lib/python3.6/multiprocessing/process.py in start(self)
    103                'daemonic processes are not allowed to have children'
    104         _cleanup()
--> 105         self._popen = self._Popen(self)
    106         self._sentinel = self._popen.sentinel
    107         # Avoid a refcycle if the target function holds an indirect

~/anaconda3/envs/Lab/lib/python3.6/multiprocessing/context.py in _Popen(process_obj)
    221     @staticmethod
    222     def _Popen(process_obj):
--> 223         return _default_context.get_context().Process._Popen(process_obj)
    224
    225 class DefaultContext(BaseContext):

~/anaconda3/envs/Lab/lib/python3.6/multiprocessing/context.py in _Popen(process_obj)
    282         def _Popen(process_obj):
    283             from .popen_spawn_posix import Popen
--> 284             return Popen(process_obj)
    285
    286     class ForkServerProcess(process.BaseProcess):

~/anaconda3/envs/Lab/lib/python3.6/multiprocessing/popen_spawn_posix.py in __init__(self, process_obj)
     30     def __init__(self, process_obj):
     31         self._fds = []
---> 32         super().__init__(process_obj)
     33
     34     def duplicate_for_child(self, fd):

~/anaconda3/envs/Lab/lib/python3.6/multiprocessing/popen_fork.py in __init__(self, process_obj)
     17         util._flush_std_streams()
     18         self.returncode = None
---> 19         self._launch(process_obj)
     20
     21     def duplicate_for_child(self, fd):

~/anaconda3/envs/Lab/lib/python3.6/multiprocessing/popen_spawn_posix.py in _launch(self, process_obj)
     57             self._fds.extend([child_r, child_w])
     58             self.pid = util.spawnv_passfds(spawn.get_executable(),
---> 59                                            cmd, self._fds)
     60             self.sentinel = parent_r
     61             with open(parent_w, 'wb', closefd=False) as f:

~/anaconda3/envs/Lab/lib/python3.6/multiprocessing/util.py in spawnv_passfds(path, args, passfds)
    415             args, [os.fsencode(path)], True, passfds, None, None,
    416             -1, -1, -1, -1, -1, -1, errpipe_read, errpipe_write,
--> 417             False, False, None)
    418     finally:
    419         os.close(errpipe_read)

ValueError: bad value(s) in fds_to_keep

I now know that in context fork, cuda mustn't be initialized before multiprocessing, but I can't understand the error in spawn. In fact, I tried to look for fds_to_keep keyword in multiprocessing and torch.multiprocessing source code, but there is no fds_to_keep in both the two source codes.

So I'd like to know how to solve this problem, and why set_start_method('spawn') doesn't work, what is fds_to_keep, where is this keyword.

update

I have found fds_to_keep error in _posixsubprocess.c, and I looked _sanity_check_python_fd_sequence, I understood iter_fd might cause the error. So I went to multiprocessing/util.py, in spawnv_passfds function, I print variable passfds before and after sort.

original code:

# multiprocessing/util.py

def spawnv_passfds(path, args, passfds):
    import _posixsubprocess
    passfds = tuple(sorted(map(int, passfds)))
    errpipe_read, errpipe_write = os.pipe()
    try:
        return _posixsubprocess.fork_exec(
            args, [os.fsencode(path)], True, passfds, None, None,
            -1, -1, -1, -1, -1, -1, errpipe_read, errpipe_write,
            False, False, None)
    finally:
        os.close(errpipe_read)
        os.close(errpipe_write)

change:

def spawnv_passfds(path, args, passfds):
    print('before', passfds)
    import _posixsubprocess
    passfds = tuple(sorted(map(int, passfds)))
    print('after', passfds)
    errpipe_read, errpipe_write = os.pipe()
    ...

passfds didn't change, because it is a tuple which can't be sorted.

Then I changed passfds = tuple(sorted(map(int, passfds))) to passfds = tuple(sorted(map(int, list(passfds)))), this time the result of passfds is:

before [77]
after (77,)
before [78, 76, 80, 79]
after (76, 78, 79, 80)
before [78, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 80, 79]
after (75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 75, 78, 79, 80)

It is obvious that there are too many 75s in the 3rd passfds, so there would appear iter_fd <= prev_fd, and error occurs.

So now my question is: why does this phenomenon appear and why there would be nothing wrong when using fork (if I don't use any gpu before multiprocessing)?


title: How to fix the error `Process finished with exit code -1073741819 (0xC0000005)`

body:
My problem is when I running FC network the code works well in both CPU and GPU. But when it comes to CNN, I can only train it on CPU. It raises an error when I try to train it on GPU.

Like that:

  Process finished with exit code -1073741819 (0xC0000005)

I find the error raised when the code goes to loss.backword. The error happened when I use the first column instead of the second.

device = torch.device("cuda:0")

device = torch.device("cuda:0" if opt.cuda else "cpu")

My environment is Python 3.6.9, Windows 10, Torch 1.2.0, Cuda 9.2.


title: Loading and correctly displaying an image dataset using pytorch Dataloader

body:
I'm trying to load a custom dataset for training a neural network, but before I load them in, I would like to verify that they've been loaded correctly. So far it looks like they are not being loaded correctly, but I can't figure out what gives the images the format that they get.

This is the code that I'm loading the images with, and then displaying them.

f, axarr = plt.subplots(2,2, figsize=(20,20))

def load_dataset():
    data_path = 'processedData/HE/train/'
    train_dataset = torchvision.datasets.ImageFolder(
        root=data_path,
        transform=torchvision.transforms.ToTensor()
    )
    train_loader = DataLoader(
        train_dataset, batch_size=64
    )
    return train_loader
x_train = load_dataset()

datathing = next(iter(x_train))

for i, ax in enumerate(axarr.flat):
    ax.imshow(datathing[0][i].view(128,128,3))
    ax.axis('off')
plt.show()

When running this with the images, the output looks like this.

It is suppose to look something like these images

I have been trying with different image datasets, but all the sets return the same format, so my question is:

How to load images and then displaying them in their real format using pytorch's dataloader?


title: Great performance gap between PyTorch and Tensorflow

body:
PyTorch was my go to framework for deep learning for quite some time, but I decided to give Tensorflow a shot and I experimented a bit how the frameworks compare performance wise. I used the Mnist example from Tensorflow's tutorial site and created same network in Pytorch. I ran both models for 30 epoch on full 60k training examples from Mnist with batch size of 1000.
What I found out is that GPU and CPU utilisation in Tensorflow is much better than that of Pytorch and consequently the train times are much shorter in the former. GPU and CPU utilisation stats as well as corresponding code for both frameworks is found below.
From nvidia-smi utility it is visible that Pytorch uses only about 800MB of GPU memory, while Tensorflow essentially uses whole memory.
Train times under above mentioned conditions:
TensorFlow: 7.44318 s
PyTorch: 27.94735 s
I am wondering wha they did in TensorFlow to be so much more efficient, and if there is any way to achieve comparable performance in Pytorch? Or is there just some mistake in Pytorch version of the code?
Environment settings:
PyTorch:

Pytorch 1.5.1
cuda 10.2

TensorFlow:

tensorflow 2.20
cuda 10.2

Tensorflow code
import tensorflow as tf
import time

start_time = time.time()

print("Num GPUs Available: ", len(tf.config.experimental.list_physical_devices('GPU')))
tf.debugging.set_log_device_placement(True)
print(tf.test.is_gpu_available(
    cuda_only=False, min_cuda_compute_capability=None
))

with tf.device("/device:CPU:0"):

  mnist = tf.keras.datasets.mnist

  (x_train, y_train), (x_test, y_test) = mnist.load_data()
  x_train, x_test = x_train / 255.0, x_test / 255.0

  model = tf.keras.models.Sequential([
    tf.keras.layers.Flatten(input_shape=(28, 28)),
    tf.keras.layers.Dense(128, activation='relu'),
    tf.keras.layers.Dropout(0.2),
    tf.keras.layers.Dense(10)
  ])

  loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
  model.compile(optimizer='adam',
                loss=loss_fn,
                metrics=['accuracy'])
  model.fit(x_train, y_train, epochs=30,batch_size=1000,max_queue_size=10)
  model.evaluate(x_test,  y_test, verbose=2)

  probability_model = tf.keras.Sequential([
    model,
    tf.keras.layers.Softmax()
  ])

  probability_model(x_test[:5])

end_time = time.time()

print(f"TF model took:{end_time-start_time}")

PyTorch code
#Imports from external libraries
import torch
import torchvision
import torchvision.transforms as transforms
import torch.nn as nn
import torch.nn.functional as F
from torch.utils.data import DataLoader
import matplotlib.pyplot as plt
import numpy as np
import torch.optim as optim
import time

#Imports from internal libraries

class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.hidden1 = nn.Linear(784, 128)
        self.hidden2 = nn.Linear(128,10)
        self.dropout = nn.Dropout(0.2)

    def forward(self, x):
        x = F.relu(self.hidden1(x))
        x = self.dropout(x)
        x = self.hidden2(x)
        x = F.softmax(x, dim=1)
        return x
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
# device = torch.device("cpu")
model = MyModel()
model.to(device)
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)

train_dataset = torchvision.datasets.MNIST(
    root='./data_mnist',
    train=True,
    download=True,
    transform=transforms.ToTensor()
)

val_dataset = torchvision.datasets.MNIST(
    root='./data_mnist',
    train=False,
    download=True,
    transform=transforms.ToTensor()
)

train_loader = DataLoader(
    train_dataset,
    batch_size=1000,
    shuffle=True,
    num_workers=24
)

val_loader = DataLoader(
    val_dataset,
    batch_size=100,
    shuffle=False,
    num_workers=20
)

start_time = time.time()
torch.backends.cudnn.benchmark = True

for epoch in range(30):
    train_loss = 0.
    val_loss = 0.
    train_acc = 0.
    val_acc = 0.

    num_samples = 0
    for data, target in train_loader:
        data = data.to(device)
        target = target.to(device)
        optimizer.zero_grad()
        num_samples += data.size(0)
        output = model(data.view(data.size(0), -1))
        loss = criterion(output, target)
        loss.backward()
        optimizer.step()

        train_loss += loss.item()
        train_acc += (torch.argmax(output, 1) == target).float().sum()

    print(f"Num datapoints in trainset: {num_samples}")

    with torch.no_grad():
        for data, target in val_loader:
            data = data.to(device)
            target = target.to(device)
            output = model(data.view(data.size(0), -1))
            loss = criterion(output, target)
            val_loss += loss.item()
            val_acc += (torch.argmax(output, 1) == target).float().sum()

    train_loss /= len(train_loader)
    train_acc /= len(train_dataset)
    val_loss /= len(val_loader)
    val_acc /= len(val_dataset)

    print('Epoch {}, train_loss {}, val_loss {}, train_acc {}, val_acc {}'.format(
        epoch, train_loss, val_loss, train_acc, val_acc))
end_time = time.time()

print(f"pytorch training took: {end_time-start_time} s")

Tensorflow GPU utilisation

Pytorch GPU utilisation

EDIT:
As pointed out in the comments I changed the number of workers in PyTorch implementation to 8 since I found out that there is no performance improvement with more than 8 workers for this example. I commented out the validation code which was giving about 10 sec overhead, and I removed the softmax function in the forward method of the network.
This gave PyTorch new running time of about 16 seconds which is still about twice the time of the TF implementation of 7.5 seconds.
Edited code:
#Imports from external libraries
import torch
import torchvision
import torchvision.transforms as transforms
import torch.nn as nn
import torch.nn.functional as F
from torch.utils.data import DataLoader
import matplotlib.pyplot as plt
import numpy as np
import torch.optim as optim
import time

#Imports from internal libraries

class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.hidden1 = nn.Linear(784, 128)
        self.hidden2 = nn.Linear(128,10)
        self.dropout = nn.Dropout(0.2)

    def forward(self, x):
        x = F.relu(self.hidden1(x))
        x = self.dropout(x)
        x = self.hidden2(x)
        return x
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
# device = torch.device("cpu")
model = MyModel()
model.to(device)
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)

train_dataset = torchvision.datasets.MNIST(
    root='./data_mnist',
    train=True,
    download=True,
    transform=transforms.ToTensor()
)

val_dataset = torchvision.datasets.MNIST(
    root='./data_mnist',
    train=False,
    download=True,
    transform=transforms.ToTensor()
)

train_loader = DataLoader(
    train_dataset,
    batch_size=1000,
    shuffle=True,
    num_workers=8
)

val_loader = DataLoader(
    val_dataset,
    batch_size=100,
    shuffle=False,
    num_workers=20
)

start_time = time.time()
torch.backends.cudnn.benchmark = True

for epoch in range(30):
    train_loss = 0.
    val_loss = 0.
    train_acc = 0.
    val_acc = 0.

    num_samples = 0
    for data, target in train_loader:
        data = data.to(device)
        target = target.to(device)
        optimizer.zero_grad()
        num_samples += data.size(0)
        output = model(data.view(data.size(0), -1))
        loss = criterion(output, target)
        loss.backward()
        optimizer.step()

        train_loss += loss.item()
        train_acc += (torch.argmax(output, 1) == target).float().sum()

    print(f"Num datapoints in trainset: {num_samples}")

    # with torch.no_grad():
    #     for data, target in val_loader:
    #         data = data.to(device)
    #         target = target.to(device)
    #         output = model(data.view(data.size(0), -1))
    #         loss = criterion(output, target)
    #         val_loss += loss.item()
    #         val_acc += (torch.argmax(output, 1) == target).float().sum()

    # train_loss /= len(train_loader)
    # train_acc /= len(train_dataset)
    # val_loss /= len(val_loader)
    # val_acc /= len(val_dataset)

    print('Epoch {}, train_loss {}, val_loss {}, train_acc {}, val_acc {}'.format(
        epoch, train_loss, val_loss, train_acc, val_acc))
end_time = time.time()

print(f"pytorch training took: {end_time-start_time} s")


title: About pytorch reduction mean

body:
I want use L1loss and BCELoss with reduction='mean' in vae reconstruction loss
but it produce same result for all different input i.e. result for landmark
so i use reduction='sum' it produce correct result that different output for different input.
how can i use mean reduction??
L1Loss = nn.L1Loss(reduction='mean').to(device)
BCELoss = nn.BCELoss(reduction='mean').to(device)
kld_criterion = KLDLoss(reduction='mean').to(device)

in training
rec_m, (rec_f, mean_f, logvar_f), (rec_l, mean_l, logvar_l) = model(origin)

lm_loss = CELoss(rec_l, lm)
f_loss = L1Loss(rec_f, f)
m_loss = CELoss(rec_m, m)

lm_kld_loss = kld_criterion(mean_l, logvar_l)
f_kld_loss = kld_criterion(mean_f, logvar_f)

loss = 4000*(f_loss + m_loss) + 30 * (lm_kld_loss + f_kld_loss) + 2000 * lm_loss

and model code
class VAE_NET(nn.Module):
    def __init__(self, nc=3, ndf=32, nef=32, nz=128, isize=128, device=torch.device("cuda:0"), is_train=True):
        super(VAE_NET, self).__init__()

        self.nz = nz
        # Encoder
        self.l_encoder = Encoder(nc=nc, nef=nef, nz=nz, isize=isize, device=device)
        self.f_encoder = Encoder(nc=nc, nef=nef, nz=nz, isize=isize, device=device)

        # Decoder
        self.l_decoder = Decoder(nc=nc, ndf=ndf, nz=nz, isize=isize)
        self.m_decoder = Decoder(nc = nc, ndf = ndf, nz = nz * 2, isize = isize)
        self.f_decoder = Decoder(nc = nc, ndf = ndf, nz = nz * 2, isize = isize)

        if is_train == False:
            for param in self.encoder.parameters():
                param.requires_grad = False
            for param in self.decoder.parameters():
                param.requires_grad = False

    def forward(self, x):
        latent_l, mean_l, logvar_l = self.l_encoder(x)
        latent_f, mean_f, logvar_f = self.f_encoder(x)
        concat_latent = torch.cat((latent_l, latent_f), 1)
        rec_l = self.l_decoder(latent_l)
        rec_m = self.m_decoder(concat_latent)
        rec_f = self.f_decoder(concat_latent)
        return rec_m, (rec_f, mean_f, latent_f), (rec_l, mean_l, latent_l)

l is for face landmark
m is for face mask
f is for face part


title: Using torch.nn.DataParallel with a custom CUDA extension

body:
To my understanding, the built-in PyTorch operations all automatically handle batches through implicit vectorization, allowing parallelism across multiple GPUs.

However, when writing a custom operation in CUDA as per the Documentation, the LLTM example given performs operations that are batch invariant, for example computing the gradient of the Sigmoid function elementwise.

However, I have a use case that is not batch element invariant and not vectorizable. Running on a single GPU, I currently (inefficiently) loop over each element in the batch, performing a kernel launch for each, like so (written in the browser, just to demonstrate):

std::vector<at::Tensor> op_cuda_forward(at::Tensor input,
                                        at::Tensor elementSpecificParam) {

    auto output = at::zeros(torch::CUDA(/* TYPE */), {/* DIMENSIONS */});

    const size_t blockDim = //
    const size_t gridDim = //
    const size_t = numBatches = //

    for (size_t i = 0; i < numBatches; i++) {
        op_cuda_forward_kernel<T><<<gridDim, blockDim>>>(input[i],
                                                         elementSpecificParam[i],
                                                         output[i]);
    }

    return {output};
}

However, I wish to split this operation over multiple GPUs by batch element.

How would the allocation of the output Tensor work in a multi-GPU scenario?

Of course, one may create intermediate Tensors on each GPU before launching the appropriate kernel, however, the overhead of copying the input data to each GPU and back again would be problematic.

Is there a simpler way to launch the kernels without first probing the environment for GPU information (# GPU's etc)?

The end goal is to have a CUDA operation that works with torch.nn.DataParallel.


title: Getting the value at the intersection of a row and column in a pytorch tensor matrix

body:
I am new to pytorch and am looking to get a value at an index from a matrix. There is a matrix called psfm_s that has been initialized with
psfm_s=Var(torch.randn(12,20),requires_grad=True) For example, I would like to to get the number in the first column (out of 12 columns)  and the number in the first row (out of 20 rows).

I have tried doing something like index=torch.tensor([0,0])
num_at_index=psfm_s[index] to get the desired number but that just gets me a tensor with a bunch of numbers in it, I'm not really sure what happens with this method.

I just want the one number at the desired index, how can I go about doing this if it's even possible? Thanks for the help!


title: PyTorch: initializing weight with numpy array + create a constant tensor

body:
I have the following code :

self.wi = nn.Embedding(num_embeddings, embedding_dim)
self.wj = nn.Embedding(num_embeddings1, embedding_dim)
self.bi = nn.Embedding(num_embeddings, 1)
self.bj = nn.Embedding(num_embeddings1, 1)

self.wi.weight.data.uniform_(-1, 1)
self.wj.weight.data.uniform_(-1, 1)
self.bi.weight.data.zero_()
self.bj.weight.data.zero_()

I want to initialize the weights with numpy array, and I want to create a constant tensor, which is also a numpy array.
I am new to PyTorch, and I appreciate any help.


title: LSTM Autoencoder problems

body:
TLDR:
Autoencoder underfits timeseries reconstruction and just predicts average value.
Question Set-up:
Here is a summary of my attempt at a sequence-to-sequence autoencoder. This image was taken from this paper: https://arxiv.org/pdf/1607.00148.pdf

Encoder: Standard LSTM layer. Input sequence is encoded in the final hidden state.
Decoder: LSTM Cell (I think!). Reconstruct the sequence one element at a time, starting with the last element x[N].
Decoder algorithm is as follows for a sequence of length N:

Get Decoder initial hidden state hs[N]: Just use encoder final hidden state.
Reconstruct last element in the sequence: x[N]= w.dot(hs[N]) + b.
Same pattern for other elements: x[i]= w.dot(hs[i]) + b
use x[i] and hs[i] as inputs to LSTMCell to get x[i-1] and hs[i-1]

Minimum Working Example:
Here is my implementation, starting with the encoder:
class SeqEncoderLSTM(nn.Module):
    def __init__(self, n_features, latent_size):
        super(SeqEncoderLSTM, self).__init__()

        self.lstm = nn.LSTM(
            n_features,
            latent_size,
            batch_first=True)

    def forward(self, x):
        _, hs = self.lstm(x)
        return hs

Decoder class:
class SeqDecoderLSTM(nn.Module):
    def __init__(self, emb_size, n_features):
        super(SeqDecoderLSTM, self).__init__()

        self.cell = nn.LSTMCell(n_features, emb_size)
        self.dense = nn.Linear(emb_size, n_features)

    def forward(self, hs_0, seq_len):

        x = torch.tensor([])

        # Final hidden and cell state from encoder
        hs_i, cs_i = hs_0

        # reconstruct first element with encoder output
        x_i = self.dense(hs_i)
        x = torch.cat([x, x_i])

        # reconstruct remaining elements
        for i in range(1, seq_len):
            hs_i, cs_i = self.cell(x_i, (hs_i, cs_i))
            x_i = self.dense(hs_i)
            x = torch.cat([x, x_i])
        return x

Bringing the two together:
class LSTMEncoderDecoder(nn.Module):
    def __init__(self, n_features, emb_size):
        super(LSTMEncoderDecoder, self).__init__()
        self.n_features = n_features
        self.hidden_size = emb_size

        self.encoder = SeqEncoderLSTM(n_features, emb_size)
        self.decoder = SeqDecoderLSTM(emb_size, n_features)

    def forward(self, x):
        seq_len = x.shape[1]
        hs = self.encoder(x)
        hs = tuple([h.squeeze(0) for h in hs])
        out = self.decoder(hs, seq_len)
        return out.unsqueeze(0)

And here's my training function:
def train_encoder(model, epochs, trainload, testload=None, criterion=nn.MSELoss(), optimizer=optim.Adam, lr=1e-6,  reverse=False):

    device = 'cuda' if torch.cuda.is_available() else 'cpu'
    print(f'Training model on {device}')
    model = model.to(device)
    opt = optimizer(model.parameters(), lr)

    train_loss = []
    valid_loss = []

    for e in tqdm(range(epochs)):
        running_tl = 0
        running_vl = 0
        for x in trainload:
            x = x.to(device).float()
            opt.zero_grad()
            x_hat = model(x)
            if reverse:
                x = torch.flip(x, [1])
            loss = criterion(x_hat, x)
            loss.backward()
            opt.step()
            running_tl += loss.item()

        if testload is not None:
            model.eval()
            with torch.no_grad():
                for x in testload:
                    x = x.to(device).float()
                    loss = criterion(model(x), x)
                    running_vl += loss.item()
                valid_loss.append(running_vl / len(testload))
            model.train()

        train_loss.append(running_tl / len(trainload))

    return train_loss, valid_loss

Data:
Large dataset of events scraped from the news (ICEWS). Various categories exist that describe each event. I initially one-hot encoded these variables, expanding the data to 274 dimensions. However, in order to debug the model, I've cut it down to a single sequence that is 14 timesteps long and only contains 5 variables. Here is the sequence I'm trying to overfit:
tensor([[0.5122, 0.0360, 0.7027, 0.0721, 0.1892],
        [0.5177, 0.0833, 0.6574, 0.1204, 0.1389],
        [0.4643, 0.0364, 0.6242, 0.1576, 0.1818],
        [0.4375, 0.0133, 0.5733, 0.1867, 0.2267],
        [0.4838, 0.0625, 0.6042, 0.1771, 0.1562],
        [0.4804, 0.0175, 0.6798, 0.1053, 0.1974],
        [0.5030, 0.0445, 0.6712, 0.1438, 0.1404],
        [0.4987, 0.0490, 0.6699, 0.1536, 0.1275],
        [0.4898, 0.0388, 0.6704, 0.1330, 0.1579],
        [0.4711, 0.0390, 0.5877, 0.1532, 0.2201],
        [0.4627, 0.0484, 0.5269, 0.1882, 0.2366],
        [0.5043, 0.0807, 0.6646, 0.1429, 0.1118],
        [0.4852, 0.0606, 0.6364, 0.1515, 0.1515],
        [0.5279, 0.0629, 0.6886, 0.1514, 0.0971]], dtype=torch.float64)

And here is the custom Dataset class:
class TimeseriesDataSet(Dataset):
    def __init__(self, data, window, n_features, overlap=0):
        super().__init__()
        if isinstance(data, (np.ndarray)):
            data = torch.tensor(data)
        elif isinstance(data, (pd.Series, pd.DataFrame)):
            data = torch.tensor(data.copy().to_numpy())
        else:
            raise TypeError(f"Data should be ndarray, series or dataframe. Found {type(data)}.")

        self.n_features = n_features
        self.seqs = torch.split(data, window)

    def __len__(self):
        return len(self.seqs)

    def __getitem__(self, idx):
        try:
            return self.seqs[idx].view(-1, self.n_features)
        except TypeError:
            raise TypeError("Dataset only accepts integer index/slices, not lists/arrays.")

Problem:
The model only learns the average, no matter how complex I make the model or now long I train it.
Predicted/Reconstruction:

Actual:

My research:
This problem is identical to the one discussed in this question: LSTM autoencoder always returns the average of the input sequence
The problem in that case ended up being that the objective function was averaging the target timeseries before calculating loss. This was due to some broadcasting errors because the author didn't have the right sized inputs to the objective function.
In my case, I do not see this being the issue. I have checked and double checked that all of my dimensions/sizes line up. I am at a loss.
Other Things I've Tried

I've tried this with varied sequence lengths from 7 timesteps to 100 time steps.
I've tried with varied number of variables in the time series. I've tried with univariate all the way to all 274 variables that the data contains.
I've tried with various reduction parameters on the nn.MSELoss module. The paper calls for sum, but I've tried both sum and mean. No difference.
The paper calls for reconstructing the sequence in reverse order (see graphic above). I have tried this method using the flipud on the original input (after training but before calculating loss). This makes no difference.
I tried making the model more complex by adding an extra LSTM layer in the encoder.
I've tried playing with the latent space. I've tried from 50% of the input number of features to 150%.
I've tried overfitting a single sequence (provided in the Data section above).

Question:
What is causing my model to predict the average and how do I fix it?


title: pytorch where is Embedding "max_norm" implemented?

body:
The "embedding" class documentation https://pytorch.org/docs/stable/nn.html says

max_norm (float, optional) – If given, will renormalize the embedding vectors to have a norm lesser than this before extracting.

1) In my model, I use this embedding class as a parameter, not just as an input (the model learns the embedding.) In this case, I assume every time when updates happen, the embedding gets renormalized, not only when it's initialized. Is my understanding correct?

2) I wanted to confirm 1) by looking at the source, but I couldn't find the implementation in pytorch embedding class. https://pytorch.org/docs/stable/_modules/torch/nn/modules/sparse.html
Can someone point me to the max_norm implementation?


title: How to extend a Loss Function Pytorch

body:
I would like to create my own custom Loss function as a weighted combination of 3 Loss Function, something similar to:

criterion = torch.nn.CrossEntropyLoss(out1, lbl1) + \
            torch.nn.CrossEntropyLoss(out2, lbl2) + \
            torch.nn.CrossEntropyLoss(out3, lbl3)

I am doing it to address a multi-class multi-label classification problem.
Does it make sense? How to implement correctly such Loss Function in Pytorch?

Thanks


title: UnboundLocalError: local variable 'labels' referenced before assignment white training for object detection using torch and yolov3

body:
I'm training on custom objects for object detection by using yolo-v3 and PyTorch . labeling and every other task completed. while run the train command I'm getting the following error

(base) C:\Users\murali\Desktop\yolov3>python train.py --data coco.data --cfg cfg/yolov3.cfg
Namespace(accumulate=2, adam=False, arc='defaultpw', batch_size=32, bucket='', cache_images=False, cfg='cfg/yolov3.cfg', data='coco.data', device='', epochs=273, evolve=False, img_size=416, img_weights=False, multi_scale=False, name='', nosave=False, notest=False, prebias=False, rect=False, resume=False, transfer=False, var=None, weights='')
Using CPU

WARNING:root:This caffe2 python run does not have GPU support. Will run in CPU only mode.
Reading labels (357 found, 0 missing, 4 empty for 361 images): 100%|███████████████| 361/361 [00:00<00:00, 6489.34it/s]
Model Summary: 222 layers, 6.19491e+07 parameters, 6.19491e+07 gradients
Starting training for 273 epochs...

 Epoch   gpu_mem      GIoU       obj       cls     total   targets  img_size

Corrupt JPEG data: 2 extraneous bytes before marker 0xd9
0%| | 0/12 [00:00<?, ?it/s]Corrupt JPEG data: 2 extraneous bytes before marker 0xd9
Corrupt JPEG data: 1 extraneous bytes before marker 0xd9
Corrupt JPEG data: 1 extraneous bytes before marker 0xd9
Corrupt JPEG data: 2 extraneous bytes before marker 0xd9
Corrupt JPEG data: 1 extraneous bytes before marker 0xd9
Corrupt JPEG data: 2 extraneous bytes before marker 0xd9
Corrupt JPEG data: 1 extraneous bytes before marker 0xd9
Corrupt JPEG data: 1 extraneous bytes before marker 0xd9
Corrupt JPEG data: 1 extraneous bytes before marker 0xd9
Corrupt JPEG data: 2 extraneous bytes before marker 0xd9
Corrupt JPEG data: 2 extraneous bytes before marker 0xd9
Corrupt JPEG data: 1 extraneous bytes before marker 0xd9
Corrupt JPEG data: 1 extraneous bytes before marker 0xd9
Corrupt JPEG data: 2 extraneous bytes before marker 0xd9
Traceback (most recent call last):
File "train.py", line 426, in
train() # train normally
File "train.py", line 235, in train
for i, (imgs, targets, paths, _) in pbar: # batch -------------------------------------------------------------
File "C:\Users\murali\AppData\Local\Continuum\anaconda3\lib\site-packages\tqdm_tqdm.py", line 1005, in iter
for obj in iterable:
File "C:\Users\murali\AppData\Local\Continuum\anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 819, in next
return self._process_data(data)
File "C:\Users\murali\AppData\Local\Continuum\anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 846, in _process_data
data.reraise()
File "C:\Users\murali\AppData\Local\Continuum\anaconda3\lib\site-packages\torch_utils.py", line 369, in reraise
raise self.exc_type(msg)
UnboundLocalError: Caught UnboundLocalError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "C:\Users\murali\AppData\Local\Continuum\anaconda3\lib\site-packages\torch\utils\data_utils\worker.py", line 178, in _worker_loop
data = fetcher.fetch(index)
File "C:\Users\murali\AppData\Local\Continuum\anaconda3\lib\site-packages\torch\utils\data_utils\fetch.py", line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "C:\Users\murali\AppData\Local\Continuum\anaconda3\lib\site-packages\torch\utils\data_utils\fetch.py", line 44, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "C:\Users\murali\Desktop\yolov3\utils\datasets.py", line 416, in getitem
img, labels = load_mosaic(self, index)
File "C:\Users\murali\Desktop\yolov3\utils\datasets.py", line 590, in load_mosaic
labels4.append(labels)
UnboundLocalError: local variable 'labels' referenced before assignment

The lines mentioned the error is following two part of code in
dataset.py

416 line is in following code

    if mosaic:
        # Load mosaic
    img, labels = load_mosaic(self, index)
        h, w, _ = img.shape

590 line in the following code (labels4.append(labels))

    # Load labels
    label_path = self.label_files[index]
    if os.path.isfile(label_path):
        x = self.labels[index]
        if x is None:  # labels not preloaded
            with open(label_path, 'r') as f:
                x = np.array([x.split() for x in f.read().splitlines()], dtype=np.float32)

        if x.size > 0:
            # Normalized xywh to pixel xyxy format
            labels = x.copy()
            labels[:, 1] = w * (x[:, 1] - x[:, 3] / 2) + padw
            labels[:, 2] = h * (x[:, 2] - x[:, 4] / 2) + padh
            labels[:, 3] = w * (x[:, 1] + x[:, 3] / 2) + padw
            labels[:, 4] = h * (x[:, 2] + x[:, 4] / 2) + padh

        labels4.append(labels)
labels4 = np.concatenate(labels4, 0)

How to solve this?

The full code is here

`


title: Why is PyTorch 2x slower than Keras for an identical model and hyperparameters?

body:
I've experienced this with custom made modules as well, but for this example I'm specifically using one of the official PyTorch examples and the MNIST dataset.

I've ported the exact architecture in Keras and TF2 with eager mode like so:

model = keras.models.Sequential([ keras.layers.Conv2D(32, (3, 3) , input_shape=(28,28,1), activation='relu'),
                                 keras.layers.Conv2D(64, (3, 3)),
                                 keras.layers.MaxPool2D((2, 2)),
                                 keras.layers.Dropout(0.25),
                                 keras.layers.Flatten(),
                                 keras.layers.Dense(128, activation='relu'),
                                 keras.layers.Dropout(0.5),
                                 keras.layers.Dense(10, activation='softmax')]
                                )

model.summary()

model.compile(optimizer=keras.optimizers.Adadelta(), loss=keras.losses.sparse_categorical_crossentropy, metrics=['accuracy'])

model.fit(train_data,train_labels,batch_size=64,epochs=30,shuffle=True, max_queue_size=1)

The training loop in PyTorch is:

def train(args, model, device, train_loader, optimizer, epoch):
    model.train()
    for batch_idx, (data, target) in enumerate(train_loader):
        data, target = data.to(device), target.to(device)
        optimizer.zero_grad()
        output = model(data)
        loss = F.nll_loss(output, target)
        loss.backward()
        optimizer.step()

With me timing every epoch like so:

for epoch in range(1, args.epochs + 1):
    since = time.time()
    train(args, model, device, train_loader, optimizer, epoch)
    # test(args, model, device, test_loader)
    # scheduler.step()
    time_elapsed = time.time() - since
    print('Training complete in {:.0f}m {:.0f}s'.format(
        time_elapsed // 60, time_elapsed % 60))

I have verified that:

Both versions use the same Optimizer (AdaDelta)
Both versions have around the same number of trainable parameters (1.2 million)
I removed the normalization in dataLoader, leaving it to just a toTensor() call.
pin_memory is set to True, and num_workers is set to 1 for the PyTorch code.
Per the suggestion of Timbus Calin I set max_queue_size to 1 and the results are identical.

The Keras version runs at around 4-5 seconds per epoch while the PyTorch version runs at around 9-10 seconds per epoch.

Why is this and how can I improve this time?


title: torchvision installation problem: avcodec-58.dll not found

body:
I am having an odd issue with torchvision installation.  I just installed torch/torchvision with the folowing pip command:

pip install torch===1.4.0 torchvision===0.5.0 -f https://download.pytorch.org/whl/torch_stable.html

system:
OS: win 7, python 3.6.8, with cuda

the error: when I do:
import torchvision

I get a pop up "python.exe - System Error" "The program can't start because avcodec-58.dll is missing from your computer..."

I have torch/torchvision running on another win machine (10, cpu only) without problems.

Someone suggested installing ffmpeg, which did not help.  The thing is that my other win PC never had ffmpeg and torchvision installation works fine there.

Does anyone have suggestions on how to fix this?

and what is avcodec-58.dll anyway?

EDIT:
It turns out there are two versions of ffmpeg static and shared.  Apparently chocolatey installs the static one and the shared one is needed by torchvision as it is the one with the dlls including avcodec-58.dll.


title: Torchscripting a module with _ConvNd in forward

body:
I am using PyTorch 1.4 and need to export a model with convolutions inside a loop in forward:

class MyCell(torch.nn.Module):
    def __init__(self):
        super(MyCell, self).__init__()

    def forward(self, x):
        for i in range(5):
            conv = torch.nn.Conv1d(1, 1, 2*i+3)
            x = torch.nn.Relu()(conv(x))
        return x

torch.jit.script(MyCell())

This gives the following error:

RuntimeError:
Arguments for call are not valid.
The following variants are available:

  _single(float[1] x) -> (float[]):
  Expected a value of type 'List[float]' for argument 'x' but instead found type 'Tensor'.

  _single(int[1] x) -> (int[]):
  Expected a value of type 'List[int]' for argument 'x' but instead found type 'Tensor'.

The original call is:
  File "***/torch/nn/modules/conv.py", line 187
                 padding=0, dilation=1, groups=1,
                 bias=True, padding_mode='zeros'):
        kernel_size = _single(kernel_size)
                      ~~~~~~~ <--- HERE
        stride = _single(stride)
        padding = _single(padding)
'Conv1d.__init__' is being compiled since it was called from 'Conv1d'
  File "***", line ***
    def forward(self, x):
        for _ in range(5):
            conv = torch.nn.Conv1d(1, 1, 2*i+3)
                   ~~~~~~~~~~~~~~~ <--- HERE
            x = torch.nn.Relu()(conv(x))
        return x
'Conv1d' is being compiled since it was called from 'MyCell.forward'
  File "***", line ***
    def forward(self, x, h):
        for _ in range(5):
            conv = torch.nn.Conv1d(1, 1, 2*i+3)
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
            x = torch.nn.Relu()(conv(x))
        return x

I have also tried pre-defining the conv's then putting them in a list inside __init__, but such a type is not allowed by TorchScript:

class MyCell(torch.nn.Module):
    def __init__(self):
        super(MyCell, self).__init__()
        self.conv = [torch.nn.Conv1d(1, 1, 2*i+3) for i in range(5)]

    def forward(self, x):
        for i in range(len(self.conv)):
            x = torch.nn.Relu()(self.conv[i](x))
        return x

torch.jit.script(MyCell())

This instead gives:

RuntimeError:
Module 'MyCell' has no attribute 'conv' (This attribute exists on the Python module, but we failed to convert Python type: 'list' to a TorchScript type.):
  File "***", line ***
    def forward(self, x):
        for i in range(len(self.conv)):
                           ~~~~~~~~~ <--- HERE
            x = torch.nn.Relu()(self.conv[i](x))
        return x

So how to export this module? Background: I am exporting Mixed-scale Dense Networks (source) to TorchScript; while nn.Sequential may work for this simplified case, practically I need to convolve with all the historical convolution outputs in each iteration, which is more than chaining the layers.