title: Difference between logloss in sklearn and BCEloss in Pytorch?

body:
Looking at the documentation for logloss in Sklearn and BCEloss in Pytorch, these should be the same, i.e. just the normal log loss with weights applied. However, they behave differently - both with and without weights applied. Can anyone explain it to me? I could not find the source code for BCEloss (which refers to binary_cross_entropy internally).

input = torch.randn((3, 1), requires_grad=True)
target = torch.ones((3, 1), requires_grad=False)
w = torch.randn((3, 1), requires_grad=False)

# ----- With weights
w = F.sigmoid(w)
criterion_test = nn.BCELoss(weight=w)
print(criterion_test(input=F.sigmoid(input), target=F.sigmoid(target)))
print(log_loss(y_true=target.detach().numpy(),
               y_pred=F.sigmoid(input).detach().numpy(), sample_weight=w.detach().numpy().reshape(-1), labels=np.array([0.,1.])))
print("")
print("")
# ----- Without weights
criterion_test = nn.BCELoss()
print(criterion_test(input=F.sigmoid(input),target=F.sigmoid(target)))
print(log_loss(y_true=target.detach().numpy(),
               y_pred=F.sigmoid(input).detach().numpy(), labels=np.array([0.,1.])))


title: Image segmentation training set labeling

body:
I am new to pytorch and Deep learning. I am trying to do image segmentation.
But , I am stuck at how to label training set images.

Can anyone please help me ?

This is one of my training image

I have two kinds of plants here - one is weed and another one is a good crop. I need to label them.

Can anyone tell me how can I do this ?

I am going to use deep neural network models ( like ResNet ) on the labelled data.


title: CUDA out of memory when fine-tuning a large model

body:
I have previously trained a VGG mode(say model1), and a two layer model(say model2) separately, now I have to train a new model which combines those two models together, and each part of the new model is initialized with the learned weights of model1 and model2, which I implemented as follows:

class TransferModel(nn.Module):
    def __init__(self, VGG, TwoLayer):
        super(TransferModel, self).__init__()
        self.vgg_layer=VGG
        self.linear = TwoLayer
        for param in self.vgg_layer.parameters():
            param.requires_grad = True
    def forward(self, x):
        h1_vgg = self.vgg_layer(x)
        y_pred = self.linear(h1_vgg)
        return y_pred
# for image_id in train_ids[0:1]:
#     img = load_image(train_id_to_file[image_id])
new_model=TransferModel(trained_vgg_instance, trained_twolayer_instance)
new_model.linear.load_state_dict(trained_twolayer_instance.state_dict())
new_model.vgg_layer.load_state_dict(trained_vgg_instance.state_dict())
new_model.cuda()

And when training, I try:

def train(model, learning_rate=0.001, batch_size=50, epochs=2):
    optimizer=optim.Adam(model.parameters(), lr=learning_rate)
    criterion = torch.nn.MultiLabelSoftMarginLoss()
    x = torch.zeros([batch_size, 3, img_size, img_size])
    y_true = torch.zeros([batch_size, 4096])
    for epoch in range(epochs):  # loop over the dataset multiple times
        running_loss = 0.0
        shuffled_indcs=torch.randperm(20000)
        for i in range(20000):
        for batch_num in range(int(20000/batch_size)):
            optimizer.zero_grad()
            for j in range(batch_size):
                # ... some code to load batches of images into x....
            x_batch=Variable(x).cuda()
            print(batch_num)
            y_true_batch=Variable(train_labels[batch_num*batch_size:(batch_num+1)*batch_size, :]).cuda()
            y_pred =model(x_batch)
            loss = criterion(y_pred, y_true_batch)
            loss.backward()
            optimizer.step()
            running_loss += loss
            del x_batch, y_true_batch, y_pred
            torch.cuda.empty_cache()
        print("in epoch[%d] = %.8f " % (epoch, running_loss /(batch_num+1)))
        running_loss = 0.0

    print('Finished Training')
train(new_model)

In the second iteration(batch_num=1) of the first epoch, I get this error:

  CUDA out of memory. Tried to allocate 153.12 MiB (GPU 0; 5.93 GiB
  total capacity; 4.83 GiB already allocated; 66.94 MiB free; 374.12 MiB
  cached)

Although I have explicitly used 'del' in my training, by running nvidia-smi it looks like it doesn't do anything and the memory isn't being freed.

What should I do?


title: How to get the full Jacobian of a derivative in PyTorch?

body:
Lets consider a simple tensor x and lets define another one which depends on x and have multiple dimension : y = (x, 2x, x^2).

How can I have the full gradient dy/dx = (1,2,x) ?

For example lets take the code :

import torch
from torch.autograd import grad

x = 2 * torch.ones(1)
x.requires_grad = True
y = torch.cat((x, 2*x, x*x))
# dy_dx = ???

This is what I have I unsuccessfuly tried so far :

>>> dy_dx = grad(y, x, grad_outputs=torch.ones_like(y), create_graph=True)
(tensor([7.], grad_fn=<AddBackward0>),)
>>> dy_dx = grad(y, x, grad_outputs=torch.Tensor([1,0,0]), create_graph=True)
(tensor([1.], grad_fn=<AddBackward0>),)
>>> dy_dx = grad(y, [x,x,x], grad_outputs=torch.eye(3), create_graph=True)
(tensor([7.], grad_fn=<AddBackward0>),)

Each times I got only part of the gradient or an accumulated version...

I know I could use a for loop using the second expression like

dy_dx = torch.zeros_like(y)
coord = torch.zeros_like(y)
for i in range (y.size(0)):
    coord[i] = 1
    dy_dx[i], = grad(y, x, grad_outputs=coord, create_graph=True)
    coord[i] = 0

However, as I am handeling with high dimensions tensors, this for loop could take too much time to compute. Moreover, there must be a way to perform the full jacobian without acuumulating the gradient...

Does anyone has the solution ? Or an alternative ?


title: Recommended way to replace several values in a tensor at once?

body:
Is there a batch way to replace several particular values in a pytorch tensor at once without a for loop?

Example:

old_values = torch.Tensor([1, 2, 3, 4, 5, 5, 2, 3, 3, 2])
old_new_value = [[2,22], [3,33], [6, 66]]

old_new_value = [[2,22], [3,33], [6, 66]], which means 2 should be replaced by 22, and 3 should be replaced by 33 and 6 to 66

Can I have an efficient way to achieve the following end_result?

end_result = torch.Tensor([1, 22, 33, 4, 5, 5, 22, 33, 33, 22])

Note that old_values is not unique. Also, it is possible that old_new_value has a pair here(6, 66) that does not exist in the old_values.
Also, the old_new_values includes unique rows,


title: Pytorch: Why loss functions are implemented both in nn.modules.loss and nn.functional module?

body:
Many loss functions in Pytorch are implemented both in nn.modules.loss and nn.functional.

For example, the two lines of the below return same results.

import torch.nn as nn
import torch.functional as F
nn.L1Loss()(x,y)
F.l1_loss(x,y)

Why are there two implementations?

Consistency for other parametric loss functions
Instantiation of loss function brings something good
otherwise


title: How does pytorch's parallel method and distributed method work?

body:
I'm not an expert in distributed system and CUDA. But there is one really interesting feature that PyTorch support which is nn.DataParallel and nn.DistributedDataParallel. How are they actually implemented? How do they separate common embeddings and synchronize data?
Here is a basic example of DataParallel.
import torch.nn as nn
from torch.autograd.variable import Variable
import numpy as np

class Model(nn.Module):
    def __init__(self):
        super().__init__(
            embedding=nn.Embedding(1000, 10),
            rnn=nn.Linear(10, 10),
        )

    def forward(self, x):
        x = self.embedding(x)
        x = self.rnn(x)
        return x

model = nn.DataParallel(Model())
model.forward(Variable.from_numpy(np.array([1,2,3,4,5,6], dtype=np.int64)).cuda()).cpu()

PyTorch can split the input and send them to many GPUs and merge the results back.
How does it manage embeddings and synchronization for a parallel model or a distributed model?
I wandered around PyTorch's code but it's very hard to know how the fundamentals work.


title: How can I resize a PyTorch tensor with a sliding window?

body: I have a tensor with size: torch.Size([118160, 1]). What I want to do is split it up into n tensors with 100 elements each, sliding by 50 elements at a time. What's the best way to achieve this with PyTorch?


title: PyTorch : How to apply the same random transformation to multiple image?

body:
I am writing a simple transformation for a dataset which contains many pairs of images. As a data augmentation, I want to apply some random transformation for each pair but the images in that pair should be transformed in the same way.
For example, given a pair of two images A and B, if A is flipped horizontally, B must be flipped horizontally as A. Then the next pair C and D should be differently transformed from A and B but C and D are transformed in the same way. I am trying that in the way below
import random
import numpy as np
import torchvision.transforms as transforms
from PIL import Image

img_a = Image.open("sample_ajpg") # note that two images have the same size
img_b = Image.open("sample_b.png")
img_c, img_d = Image.open("sample_c.jpg"), Image.open("sample_d.png")

transform = transforms.RandomChoice(
    [transforms.RandomHorizontalFlip(),
     transforms.RandomVerticalFlip()]
)
random.seed(0)
display(transform(img_a))
display(transform(img_b))

random.seed(1)
display(transform(img_c))
display(transform(img_d))

Yet、 the above code does not choose the same transformation and as I tested, it is dependent on the number of times transform is called.
Is there any way to force transforms.RandomChoice to use the same transform when specified?


title: Applying any optimization causes values to be NaN

body:
I am working on a classifier model for this data set:
https://archive.ics.uci.edu/ml/datasets/ILPD+%28Indian+Liver+Patient+Dataset%29
and I have come up with this code in pytorch:

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from sklearn import preprocessing
from sklearn.model_selection import train_test_split
import pandas as pd
import numpy as np

ILPDataset = pd.read_csv('preprocessed-ilpd.csv')
ILPDataset["Age"] = pd.to_numeric(ILPDataset["Age"])
LabelEncoder = preprocessing.LabelEncoder()
LabelEncoder.fit(ILPDataset["Gender"])
ILPDataset["Gender"] = LabelEncoder.transform(ILPDataset["Gender"])
print(ILPDataset["Gender"].describe())
ILPDataset["AAP"] = preprocessing.scale(ILPDataset["AAP"])
ILPDataset["SgAlAm"] = preprocessing.scale(ILPDataset["SgAlAm"])
ILPDataset["SgApAm"] = preprocessing.scale(ILPDataset["SgApAm"])
Features = ["Age","Gender","TB","DB","AAP","SgAlAm","SgApAm","TP","ALB","A/G"]
ILPDFeatures = ILPDataset[Features]
ILPDTarget = ILPDataset["Selector"]

X_Train, X_Test, Y_Train, Y_Test = train_test_split(ILPDFeatures,ILPDTarget,test_size=0.2,random_state=0)

print(X_Train.shape)
print(Y_Train.shape)
print(X_Test.shape)
print(Y_Test.shape)
torch.set_default_tensor_type(torch.DoubleTensor)
TrainX = torch.from_numpy(X_Train.values).double()
TestX = torch.from_numpy(X_Test.values).double()

TrainY = torch.from_numpy(Y_Train.values).long().view(1,-1)[0]
TestY = torch.from_numpy(Y_Test.values).long().view(1,-1)[0]
TrainY.reshape(TrainY.shape[0],1)
TestY.reshape(TestY.shape[0],1)
print(X_Train.shape[1])
input_layers = X_Train.shape[1]
output_layers = 2
hidden = 100

class Net(nn.Module):
    def __init__(self):
        super(Net,self).__init__()
        self.fc1 = nn.Linear(input_layers,hidden)
        self.fc2 = nn.Linear(hidden,hidden)
        self.fc3 = nn.Linear(hidden,output_layers)

    def forward(self,x):
        x = torch.sigmoid(self.fc1(x))
        x = torch.sigmoid(self.fc2(x))
        x = self.fc3(x)
        return F.softmax(x,dim=-1)

model = Net()
optimizer = torch.optim.SGD(model.parameters(),lr = 0.0001,momentum = 0.9)
loss_fn = nn.NLLLoss()
epochs_data = []
epochs = 601
print(TrainX.shape)
print(TestY.shape)
for epoch in range(1,3):
    optimizer.zero_grad()
    Ypred = model(TrainX)
    loss = loss_fn(Ypred,TrainY)
    loss.backward()
    optimizer.step()
    print(Ypred)
    if epoch%100==0:
        print("Epoch - %d, (%d%%) "%(epoch,epoch/epochs*100))

YPred_Test = model(TestX)
Loss_Test = loss_fn(YPred_Test,TestY)
print(Loss_Test.item())
print(YPred_Test)
print(YPred_Test)
print(TestY)

I have used different optimizers with different LRs and Momentum but on applying each one of them after loss.backward(), it turns the values in the tensor to NaN. I looked up the different answers on SO but even after trying them out I couldn't solve the bug.


title: Differences between F.relu(X) and torch.max(X, 0)

body:
I am trying to implement the following loss function

To me, the most straight forword implementation would be using torch.max

losses = torch.max(ap_distances - an_distances + margin, torch.Tensor([0]))

However, I saw other implementations on github using F.relu

losses = F.relu(ap_distances - an_distances + margin)

They give essential the same output, but I wonder if there's any fundamental difference between the two methods.


what is torch's unsqueeze  equivalence with tensorflow?

#tensorflow auto-broadcasts singleton dimensions
lower_bounds = tf.argmax(set_1[:, :2].unsqueeze(1), set_2[:, :2].unsqueeze(0))  # (n1, n2, 2)
upper_bounds = tf.argmin(set_1[:, 2:].unsqueeze(1), set_2[:, 2:].unsqueeze(0))  # (n1, n2, 2)


title: Pytorch: Size Mismatch during running a test image through a trained CNN

body:
I am going through tutorials to train/test a convolutional neural network(CNN), and I am having an issue with prepping a test image to run it through the trained network. My initial guess is that it has something to do with having a correct format of the tensor input for the net.

Here is the code for the Net.

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.nn.init as I

class Net(nn.Module):

def __init__(self):
    super(Net, self).__init__()

    ## 1. This network takes in a square (same width and height), grayscale image as input
    ## 2. It ends with a linear layer that represents the keypoints
    ## this last layer output 136 values, 2 for each of the 68 keypoint (x, y) pairs

    # input size 224 x 224
    # after the first conv layer, (W-F)/S + 1 = (224-5)/1 + 1 = 220
    # after one pool layer, this becomes (32, 110, 110)
    self.conv1 = nn.Conv2d(1, 32, 5)

    # maxpool layer
    # pool with kernel_size = 2, stride = 2
    self.pool = nn.MaxPool2d(2,2)

    # second conv layer: 32 inputs, 64 outputs , 3x3 conv
    ## output size = (W-F)/S + 1 = (110-3)/1 + 1 = 108
    ## output dimension: (64, 108, 108)
    ## after another pool layer, this becomes (64, 54, 54)
    self.conv2 = nn.Conv2d(32, 64, 3)

    # third conv layer: 64 inputs, 128 outputs , 3x3 conv
    ## output size = (W-F)/S + 1 = (54-3)/1 + 1 = 52
    ## output dimension: (128, 52, 52)
    ## after another pool layer, this becomes (128, 26, 26)
    self.conv3 = nn.Conv2d(64,128,3)

    self.conv_drop = nn.Dropout(p = 0.2)
    self.fc_drop = nn.Dropout(p = 0.4)

    # 64 outputs * 5x5 filtered/pooled map  = 186624
    self.fc1 = nn.Linear(128*26*26, 1000)
    #
    self.fc2 = nn.Linear(1000, 1000)

    self.fc3 = nn.Linear(1000, 136)

def forward(self, x):
    x = self.pool(F.relu(self.conv1(x)))
    x = self.pool(F.relu(self.conv2(x)))
    x = self.pool(F.relu(self.conv3(x)))
    x = self.conv_drop(x)

    # prep for linear layer
    # flattening
    x = x.view(x.size(0), -1)

    # two linear layers with dropout in between
    x = F.relu(self.fc1(x))
    x = self.fc_drop(x)
    x = self.fc2(x)
    x = self.fc_drop(x)
    x = self.fc3(x)

    return x

Maybe my calculation in layer inputs is wrong?

And here is the test-running code block: (you can think of 'roi' as a standard numpy image.)

# loop over the detected faces from your haar cascade
for i, (x,y,w,h) in enumerate(faces):

    plt.figure(figsize=(10,5))
    ax = plt.subplot(1, len(faces), i+1)
    # Select the region of interest that is the face in the image
    roi = image_copy[y:y+h, x:x+w]

    ## TODO: Convert the face region from RGB to grayscale
    roi = cv2.cvtColor(roi, cv2.COLOR_RGB2GRAY)

    ## TODO: Normalize the grayscale image so that its color range falls in [0,1] instead of [0,255]
    roi = np.multiply(roi, 1/255)

    ## TODO: Rescale the detected face to be the expected square size for your CNN (224x224, suggested)
    roi = cv2.resize(roi, (244,244))

    roi = roi.reshape(roi.shape[0], roi.shape[1], 1)
    roi = roi.transpose((2, 0, 1))

    ## TODO: Change to tensor
    roi = torch.from_numpy(roi)
    roi = roi.type(torch.FloatTensor)
    roi = roi.unsqueeze(0)
    print (roi.shape)

    ## TODO: run it through the net
    output_pts = net(roi)

And I get the error message saying:

RuntimeError: size mismatch, m1: [1 x 100352], m2: [86528 x 1000] at /opt/conda/conda-bld/pytorch_1524584710464/work/aten/src/TH/generic/THTensorMath.c:2033

The caveat is if I run my trained network in the provided test suite (where the tensor inputs are already prepped), it gives no errors there and runs as it is supposed to. I think that means there's nothing wrong with the design of the network architecture itself. I think there's something wrong with the way that I am prepping the image.

The output of the 'roi.shape' is:

torch.Size([1, 1, 244, 244])

Which should be okay because ([batch_size, color_channel, x, y]).

UPDATE: I have printed out the shape of the layers during running through the net. It turns out the matching input dimensions for FC are different for the test image for the task and the given test images from the test suite. Then I'm almost 80% sure that my prepping the input image for the net is wrong. But how can they have different matching dimensions if the input tensor for both has the exact same dimension ([1,1,244,244])?

when using the provided test suite (where it runs fine):

input: torch.Size([1, 1, 224, 224])
layer before 1st CV: torch.Size([1, 1, 224, 224])
layer after 1st CV pool: torch.Size([1, 32, 110, 110])
layer after 2nd CV pool: torch.Size([1, 64, 54, 54])
layer after 3rd CV pool: torch.Size([1, 128, 26, 26])
flattend layer for the 1st FC: torch.Size([1, 86528])

When prepping/running the test image:

input: torch.Size([1, 1, 244, 244])
layer before 1st CV: torch.Size([1, 1, 244, 244])
layer after 1st CV pool: torch.Size([1, 32, 120, 120]) #<- what happened here??
layer after 2nd CV pool: torch.Size([1, 64, 59, 59])
layer after 3rd CV pool: torch.Size([1, 128, 28, 28])
flattend layer for the 1st FC: torch.Size([1, 100352])


title: PyTorch most efficient Jacobian/Hessian calculation

body:
I am looking for the most efficient way to get the Jacobian of a function through Pytorch and have so far come up with the following solutions:
# Setup
def func(X):
    return torch.stack((X.pow(2).sum(1),
                        X.pow(3).sum(1),
                        X.pow(4).sum(1)),1)

X = Variable(torch.ones(1,int(1e5))*2.00094, requires_grad=True).cuda()

# Solution 1:
t = time()
Y = func(X)
J = torch.zeros(3, int(1e5))

for i in range(3):
    J[i] = grad(Y[0][i], X, create_graph=True, retain_graph=True, allow_unused=True)[0]

print(time()-t)
>>> Output: 0.002 s

# Solution 2:
def Jacobian(f,X):
    X_batch = Variable(X.repeat(3,1), requires_grad=True)
    f(X_batch).backward(torch.eye(3).cuda(), retain_graph=True)
    return X_batch.grad

t = time()
J2 = Jacobian(func,X)
print(time()-t)
>>> Output: 0.001 s

Since there seem to be not a big difference between using a loop in the first solution than the second one, I wanted to ask if there might still be be a faster way to calculate a Jacobian in pytorch.
My other question is then also about what might be the most efficient way to calculate the Hessian.
Finally, does anyone know if something like this can be done easier or more efficient in TensorFlow?


title: Concatenation step of U-Net for unequal number of channels

body:
I am trying to implement the U-NET architecture for image segmentation while implementing the crop and concatenation step in the expansive path, I am unable to understand how the unequal number of channels are concatenated.

According to the architecture, the input from the first upsampling step has to be concatenated from the corresponding output from contracting path but the problem is number of channels in contracting path is 512 while after upsampling step they are 1024, how they are supposed to be concatenated.My code for the crop and concatenate is -

def crop_and_concat(self, upsampled, bypass, crop=False):
    if crop:
        c = (bypass.size()[2] - upsampled.size()[2]) // 2
        bypass = F.pad(bypass, (-c, -c, -c, -c))
    return torch.cat((upsampled, bypass), 1)

The error I am receiving-
RuntimeError: Given groups=1, weight of size 128 256 5 5, expected input[4, 384, 64, 64] to have 256 channels, but got 384 channels instead
Where I am doing wrong?


title: Error in Tensorboard's(PyTorch) add_graph

body:
I'm following this Pytorch's Tensorboard documentation.
I have the following code:
model = torchvision.models.resnet50(False)
writer.add_graph(model)

It throws the following error:

_ = model(*args)  # don't catch, just print the error message
TypeError: ResNet object argument after * must be an iterable, not NoneType

I don't know what I'm doing wrong here!


I am trying to switch the training of my network from cpu to gpu but keep getting the following error.

I am getting the following error

Expected object of device type cuda but got device type cpu for argument #1 'self' in call to _thnn_conv2d_forward
Error occurs, No graph saved
Traceback (most recent call last):

  File "<ipython-input-6-2720a5ea768d>", line 12, in <module>
    tb.add_graph(network, images)

  File "E:\Anaconda\lib\site-packages\torch\utils\tensorboard\writer.py", line 707, in add_graph
    self._get_file_writer().add_graph(graph(model, input_to_model, verbose))

  File "E:\Anaconda\lib\site-packages\torch\utils\tensorboard\_pytorch_graph.py", line 291, in graph
    raise e

  File "E:\Anaconda\lib\site-packages\torch\utils\tensorboard\_pytorch_graph.py", line 285, in graph
    trace = torch.jit.trace(model, args)

  File "E:\Anaconda\lib\site-packages\torch\jit\__init__.py", line 882, in trace
    check_tolerance, _force_outplace, _module_class)

  File "E:\Anaconda\lib\site-packages\torch\jit\__init__.py", line 1034, in trace_module
    module._c._create_method_from_trace(method_name, func, example_inputs, var_lookup_fn, _force_outplace)

  File "E:\Anaconda\lib\site-packages\torch\nn\modules\module.py", line 530, in __call__
    result = self._slow_forward(*input, **kwargs)

  File "E:\Anaconda\lib\site-packages\torch\nn\modules\module.py", line 516, in _slow_forward
    result = self.forward(*input, **kwargs)

  File "<ipython-input-5-cd44a4e4fb73>", line 52, in forward
    t = F.relu(self.conv1(t))

  File "E:\Anaconda\lib\site-packages\torch\nn\modules\module.py", line 530, in __call__
    result = self._slow_forward(*input, **kwargs)

  File "E:\Anaconda\lib\site-packages\torch\nn\modules\module.py", line 516, in _slow_forward
    result = self.forward(*input, **kwargs)

  File "E:\Anaconda\lib\site-packages\torch\nn\modules\conv.py", line 345, in forward
    return self.conv2d_forward(input, self.weight)

  File "E:\Anaconda\lib\site-packages\torch\nn\modules\conv.py", line 342, in conv2d_forward
    self.padding, self.dilation, self.groups)

RuntimeError: Expected object of device type cuda but got device type cpu for argument #1 'self' in call to _thnn_conv2d_forward```**

I think it says that argument is in type cpu but I changed it in the training part.

I have the following code

Conv-neural network

class Network(nn.Module):
    def __init__(self):
    super(Network, self).__init__()
    self.conv1 = nn.Conv2d( in_channels= 1, out_channels= 6, kernel_size=5 )
    self.conv2 = nn.Conv2d( in_channels= 6, out_channels= 12, kernel_size=5 )

    self.fc1 = nn.Linear( in_features = 12*4*4, out_features = 120 )
    self.fc2 = nn.Linear( in_features = 120, out_features = 60 )
    self.out = nn.Linear( in_features = 60, out_features = 10 )

def forward(self, t):
    t = F.relu(self.conv1(t))
    t = F.max_pool2d(t, kernel_size=2, stride=2)

    t = F.relu(self.conv2(t))
    t = F.max_pool2d(t, kernel_size=2, stride=2)

    t = F.relu(self.fc1(t.reshape(-1, 12*4*4)))
    t = F.relu(self.fc2(t))
    t = self.out(t)

    return t

The training part

parameters = dict(
    lr = [.01, .001]
    , batch_size = [10, 100, 1000]
    , shuffle = [True, False]
    )

param_values = [v for v in parameters.values()]
param_values

for lr, batch_size, shuffle in product(*param_values):
network = Network()
network.to(device)
train_loader = torch.utils.data.DataLoader(train_set, batch_size=batch_size, shuffle = shuffle)
optimizer = optim.Adam(network.parameters(), lr=lr)
images, labels = next(iter(train_loader))
grid = torchvision.utils.make_grid(images)

comment = f' batch_size={batch_size} lr={lr} shuffle={shuffle}'
tb = SummaryWriter(comment = comment)
tb.add_image('images', grid)
tb.add_graph(network, images)

for epoch in range(10):

    total_loss = 0
    total_correct = 0

    for batch in train_loader: # Get batch
        images, labels = batch
        images = images.to(device)  # Changing data to gpu
        preds = network(images)
        loss = F.cross_entropy(preds, labels)

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        total_loss += loss.item() * batch_size
        total_correct += get_num_correct(preds, labels)

    tb.add_scalar('Loss:', total_loss, epoch)
    tb.add_scalar('Number Correct:', total_correct, epoch)
    tb.add_scalar('Accuracy:', total_correct/len(train_set), epoch)

    #tb.add_histogram('conv1.bias', network.conv1.bias, epoch)
    #tb.add_histogram('conv1.weight', network.conv1.weight, epoch)
    #tb.add_histogram('conv1.weight.grap', network.conv1.weight.grad, epoch)

    for name, weight in network.named_parameters():
        tb.add_histogram(name, weight, epoch)
        tb.add_histogram(f'{name}.grad', weight.grad, epoch)

    print("epoch:", epoch, "total_correct:", total_correct, "loss:",total_loss)

tb.close()

I am new to deep learning so any help will be highly appreciated. Thanks


title: Transferring pretrained pytorch model to onnx

body:
I am trying to convert pytorch model to ONNX, in order to use it later for TensorRT. I followed the following tutorial https://pytorch.org/tutorials/advanced/super_resolution_with_caffe2.html, but my kernel dies all the time.

This is the code that I implemented.

 # Some standard imports
import io
import numpy as np

from torch import nn
import torch.onnx
from deepformer.nets.quicknat import quickNAT
param = {
    'num_channels': 64,
    'num_filters': 64,
    'kernel_h': 5,
    'kernel_w': 5,
    'kernel_c': 1,
    'stride_conv': 1,
    'pool': 2,
    'stride_pool': 2,
    'num_classes': 1,
    'padding': 'reflection'
}

net = quickNAT(param)
checkpoint_path = 'checkpoint_epoch36_loss0.78.t7'
checkpoints=torch.load(checkpoint_path)
map_location = lambda storage, loc: storage
if torch.cuda.is_available():
    map_location = None
net.load_state_dict(checkpoints['net'])
net.train(False)
# Input to the modelvcdfx
x = torch.rand(1, 64, 256, 1600, requires_grad=True)

# Export the model
torch_out = torch.onnx._export(net,             # model being run
                               x,                       # model input (or a tuple for multiple inputs)
                               "quicknat.onnx", # where to save the model (can be a file or file-like object)
                               export_params=True)      # store the trained parameter weights inside the model file


I'm using the Soft Actor-Critic implementation available here for one of my projects. But when I try to run it, I get the following error:

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [256, 1]], which is output 0 of TBackward, is at version 3; expected version 2 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

The error arises from the gradient computation in the sac.py file. I can't see the operation that might be inplace. Any help?

The traceback:

Traceback (most recent call last)
<ipython-input-10-c124add9a61d> in <module>()
     22             for i in range(updates_per_step):
     23                 # Update parameters of all the networks
---> 24                 critic_1_loss, critic_2_loss, policy_loss, ent_loss, alpha = agent.update_parameters(memory, batch_size, updates)
     25                 updates += 1
     26

2 frames
<ipython-input-7-a2432c4c3767> in update_parameters(self, memory, batch_size, updates)
     87
     88         self.policy_optim.zero_grad()
---> 89         policy_loss.backward()
     90         self.policy_optim.step()
     91

/usr/local/lib/python3.6/dist-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph)
    196                 products. Defaults to ``False``.
    197         """
--> 198         torch.autograd.backward(self, gradient, retain_graph, create_graph)
    199
    200     def register_hook(self, hook):

/usr/local/lib/python3.6/dist-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
     98     Variable._execution_engine.run_backward(
     99         tensors, grad_tensors, retain_graph, create_graph,
--> 100         allow_unreachable=True)  # allow_unreachable flag
    101
    102


title: Different output from Libtorch C++ and pytorch

body:
I'm using the same traced model in pytorch and libtorch but I'm getting different outputs.
Python Code:
import cv2
import numpy as np
import torch
import torchvision
from torchvision import transforms as trans

# device for pytorch
device = torch.device('cuda:0')

torch.set_default_tensor_type('torch.cuda.FloatTensor')

model = torch.jit.load("traced_facelearner_model_new.pt")
model.eval()

# read the example image used for tracing
image=cv2.imread("videos/example.jpg")

test_transform = trans.Compose([
        trans.ToTensor(),
        trans.Normalize([0.5, 0.5, 0.5], [0.5, 0.5, 0.5])
    ])

resized_image = cv2.resize(image, (112, 112))

tens = test_transform(resized_image).to(device).unsqueeze(0)
output = model(tens)
print(output)

C++ Code:
#include <iostream>
#include <algorithm>
#include <opencv2/opencv.hpp>
#include <torch/script.h>

int main()
{
    try
    {
        torch::jit::script::Module model = torch::jit::load("traced_facelearner_model_new.pt");
        model.to(torch::kCUDA);
        model.eval();

        cv::Mat visibleFrame = cv::imread("example.jpg");

        cv::resize(visibleFrame, visibleFrame, cv::Size(112, 112));
        at::Tensor tensor_image = torch::from_blob(visibleFrame.data, { 1, visibleFrame.rows,
                                                    visibleFrame.cols, 3 }, at::kByte);
        tensor_image = tensor_image.permute({ 0, 3, 1, 2 });
        tensor_image = tensor_image.to(at::kFloat);

        tensor_image[0][0] = tensor_image[0][0].sub(0.5).div(0.5);
        tensor_image[0][1] = tensor_image[0][1].sub(0.5).div(0.5);
        tensor_image[0][2] = tensor_image[0][2].sub(0.5).div(0.5);

        tensor_image = tensor_image.to(torch::kCUDA);
        std::vector<torch::jit::IValue> input;
        input.emplace_back(tensor_image);
        // Execute the model and turn its output into a tensor.
        auto output = model.forward(input).toTensor();
        output = output.to(torch::kCPU);
        std::cout << "Embds: " << output << std::endl;

        std::cout << "Done!\n";
    }
    catch (std::exception e)
    {
        std::cout << "exception" << e.what() << std::endl;
    }
}

The model gives (1x512) size output tensor as shown below.
Python output
tensor([[-1.6270e+00, -7.8417e-02, -3.4403e-01, -1.5171e+00, -1.3259e+00,

-1.1877e+00, -2.0234e-01, -1.0677e+00, 8.8365e-01, 7.2514e-01,

2.3642e+00, -1.4473e+00, -1.6696e+00, -1.2191e+00, 6.7770e-01,

...

-7.1650e-01, 1.7661e-01]], device=‘cuda:0’,
grad_fn=)

C++ output
Embds: Columns 1 to 8 -84.6285 -14.7203 17.7419 47.0915 31.8170 57.6813 3.6089 -38.0543

Columns 9 to 16 3.3444 -95.5730 90.3788 -10.8355 2.8831 -14.3861 0.8706 -60.7844

...

Columns 505 to 512 36.8830 -31.1061 51.6818 8.2866 1.7214 -2.9263 -37.4330 48.5854

[ CPUFloatType{1,512} ]

Using

Pytorch 1.6.0
Libtorch 1.6.0
Visual studio 2019
Windows 10
Cuda 10.1


title: Loss is not converging in Pytorch but does in Tensorflow

body:
Epoch: 1    Training Loss: 0.816370     Validation Loss: 0.696534
Validation loss decreased (inf --> 0.696534).  Saving model ...
Epoch: 2    Training Loss: 0.507756     Validation Loss: 0.594713
Validation loss decreased (0.696534 --> 0.594713).  Saving model ...
Epoch: 3    Training Loss: 0.216438     Validation Loss: 1.119294
Epoch: 4    Training Loss: 0.191799     Validation Loss: 0.801231
Epoch: 5    Training Loss: 0.111334     Validation Loss: 1.753786
Epoch: 6    Training Loss: 0.064309     Validation Loss: 1.348847
Epoch: 7    Training Loss: 0.058158     Validation Loss: 1.839139
Epoch: 8    Training Loss: 0.015489     Validation Loss: 1.370469
Epoch: 9    Training Loss: 0.082856     Validation Loss: 1.701200
Epoch: 10   Training Loss: 0.003859     Validation Loss: 2.657933
Epoch: 11   Training Loss: 0.018133     Validation Loss: 0.593986
Validation loss decreased (0.594713 --> 0.593986).  Saving model ...
Epoch: 12   Training Loss: 0.160197     Validation Loss: 1.499911
Epoch: 13   Training Loss: 0.012942     Validation Loss: 1.879732
Epoch: 14   Training Loss: 0.002037     Validation Loss: 2.399405
Epoch: 15   Training Loss: 0.035908     Validation Loss: 1.960887
Epoch: 16   Training Loss: 0.051137     Validation Loss: 2.226335
Epoch: 17   Training Loss: 0.003953     Validation Loss: 2.619108
Epoch: 18   Training Loss: 0.000381     Validation Loss: 2.746541
Epoch: 19   Training Loss: 0.094646     Validation Loss: 3.555713
Epoch: 20   Training Loss: 0.022620     Validation Loss: 2.833098
Epoch: 21   Training Loss: 0.004800     Validation Loss: 4.181845
Epoch: 22   Training Loss: 0.014128     Validation Loss: 1.933705
Epoch: 23   Training Loss: 0.026109     Validation Loss: 2.888344
Epoch: 24   Training Loss: 0.000768     Validation Loss: 3.029443
Epoch: 25   Training Loss: 0.000327     Validation Loss: 3.079959
Epoch: 26   Training Loss: 0.000121     Validation Loss: 3.578420
Epoch: 27   Training Loss: 0.148478     Validation Loss: 3.297387
Epoch: 28   Training Loss: 0.030328     Validation Loss: 2.218535
Epoch: 29   Training Loss: 0.001673     Validation Loss: 2.934132
Epoch: 30   Training Loss: 0.000253     Validation Loss: 3.215722

My loss is not converging. I am working on Horses vs humans dataset. There is an official notebook in tensorflow for that and it worked like a charm. When I am trying to replicate the same with pytorch, loss is not converging. Can you please have a look?

I am using criterion = nn.BCEWithLogitsLoss() and optimizer = optim.RMSprop(model.parameters(), lr=0.001). Although it seems to have some effect on Training Loss, but Validation losses look like random numbers and not forming any pattern. What could be the possible reasons for loss not converging?

This is my CNN architecture:

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        # convolutional layer (sees 298x298x3 image tensor)
        self.conv1 = nn.Conv2d(3, 16, 3)
        # convolutional layer (sees 147x147x16 tensor)
        self.conv2 = nn.Conv2d(16, 32, 3)
        # convolutional layer (sees 71x71x32 tensor)
        self.conv3 = nn.Conv2d(32, 64, 3)
        # convolutional layer (sees 33x33x64 tensor)
        self.conv4 = nn.Conv2d(64, 64, 3)
        # convolutional layer (sees 14x14x64 tensor)
        self.conv5 = nn.Conv2d(64, 64, 3)
        # max pooling layer
        self.pool = nn.MaxPool2d(2, 2)
        # linear layer (64 * 7 * 7 -> 500)
        self.fc1 = nn.Linear(3136, 512)
        # linear layer (512 -> 1)
        self.fc2 = nn.Linear(512, 1)
        # dropout layer (p=0.25)
        self.dropout = nn.Dropout(0.25)

    def forward(self, x):
        # add sequence of convolutional and max pooling layers
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = self.pool(F.relu(self.conv3(x)))
        x = self.pool(F.relu(self.conv4(x)))
        x = self.pool(F.relu(self.conv5(x)))

        # flatten image input
        x = x.view(-1, 64 * 7 * 7)
        # add dropout layer
        x = self.dropout(x)
        # add 1st hidden layer, with relu activation function
        x = F.relu(self.fc1(x))
        # add dropout layer
        x = self.dropout(x)
        # add 2nd hidden layer
        x = self.fc2(x)
        return x

This is the complete jupyter notebook. Apologies for not being able to create a minimal reproduce-able example code.


title: I cant install torch-sparse in Google Colab

body:
I am trying to install torch-sparse in Google Colab using ! pip install  torch-sparse, but i am getting the following erorr:

Collecting torch-sparse

      Using cached https://files.pythonhosted.org/packages/0e/bf/6242893c898621e7e4756e1ad298e903df6dfae208aec1c32adf8cfd1f7f/torch_sparse-0.4.4.tar.gz
    Requirement already satisfied: scipy in /usr/local/lib/python3.6/dist-packages (from torch-sparse) (1.4.1)
    Requirement already satisfied: numpy>=1.13.3 in /usr/local/lib/python3.6/dist-packages (from scipy->torch-sparse) (1.17.5)
    Building wheels for collected packages: torch-sparse
      Building wheel for torch-sparse (setup.py) ... error
      ERROR: Failed building wheel for torch-sparse
      Running setup.py clean for torch-sparse
    Failed to build torch-sparse
    Installing collected packages: torch-sparse
        Running setup.py install for torch-sparse ... error
    ERROR: Command errored out with exit status 1: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-h3_oysnr/torch-sparse/setup.py'"'"'; __file__='"'"'/tmp/pip-install-h3_oysnr/torch-sparse/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-0xvimdk5/install-record.txt --single-version-externally-managed --compile Check the logs for full command output.

How can I fix it?


title: What are the numbers in torch.transforms.normalize and how to select them?

body:
I am following some tutorials and I keep seeing different numbers that seem quite arbitrary to me in the transforms section
namely,
transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (0.5,))])

or
transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,))])

or
transform = transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])

or others.
I wonder where these numbers arise, and how to know to select the correct ones?
I am about to use MNIST for sanity, but very soon to use my own unique dataset and will probably need my own normaliztion.


title: Indexing a 3d tensor using a 2d tensor

body:
I have a 3d tensor, source of shape (bsz x slen1 x nhd) and a 2d tensor, index of shape (bsz x slen2). More specifically, I have:

source = 32 x 20 x 768
index  = 32 x 16

Each value in the index tensor is in between [0, 19] which is the index of the desired vector according to the 2nd dim of the source tensor.

After indexing, I am expecting an output tensor of shape, 32 x 16 x 768.

Currently I am doing this:

bsz, _, nhid = source.size()
_, slen = index.size()

source = source.reshape(-1, nhid)
source = source[index.reshape(-1), :]
source = source.reshape(bsz, slen, nhid)

So, I am converting the 3d source tensor to a 2d tensor and 2d indexing tensor to a 1d tensor and then perform the indexing. Is this correct?

Is there any better way to do it?

Update

I checked that my code is not giving the expected result. To explain what I want, I am providing the following code snippet.

source = torch.FloatTensor([
    [[ 0.2413, -0.6667,  0.2621],
     [-0.4216,  0.3722, -1.2258],
     [-0.2436, -1.5746, -0.1270],
     [ 1.6962, -1.3637,  0.8820],
     [ 0.3490, -0.0198,  0.7928]],

    [[-0.0973,  2.3106, -1.8358],
     [-1.9674,  0.5381,  0.2406],
     [ 3.0731,  0.3826, -0.7279],
     [-0.6262,  0.3478, -0.5112],
     [-0.4147, -1.8988, -0.0092]]
     ])

index = torch.LongTensor([[0, 1, 2, 3],
                          [1, 2, 3, 4]])

And I want the output tensor as:

torch.FloatTensor([
    [[ 0.2413, -0.6667,  0.2621],
     [-0.4216,  0.3722, -1.2258],
     [-0.2436, -1.5746, -0.1270],
     [ 1.6962, -1.3637,  0.8820]],

    [[-1.9674,  0.5381,  0.2406],
     [ 3.0731,  0.3826, -0.7279],
     [-0.6262,  0.3478, -0.5112],
     [-0.4147, -1.8988, -0.0092]]
     ])


title: Understanding Feature Maps in Convolutional Layers (PyTorch)

body:
I've got this segment of code in a discriminator network for MNIST:

nn.Conv2d(1, 64, 4, 2, 1),

From my understanding, there is 1 input channel (the MNIST image), then we apply a 4x4 kernel to the image in strides of 2 to produce 64 feature maps. Does this mean that we actually have 64 kernels at this layer? Because in order to get 64 different feature maps, we would need 64 separate kernels to convolve over the image?

Then after some ReLu, we have another convolution:

nn.Conv2d(64, 128, 4, 2, 1),

How do we get from 64 to 128? From my understanding of the first example, we have 64 seperate kernels that can produce 64 seperate feature maps. But here we go from 64 feature maps to 128 feature maps? Does that mean that we only have two kernels?

I hope someone can shine some light on whether my understanding is correct!


title: Trouble converting pyTorch model to ONNX

body:
I want to convert my pyTorch model to ONNX. However, I get an error saying that

  RuntimeError: number of input names provided (9) exceeded number of inputs (7)
  However, my code runs perfectly if I take out the two Dropout layers from my model.

Why is this?

Here is my code:

# Define the model
model = torch.nn.Sequential(
    torch.nn.Linear(D_in, H),
    torch.nn.ReLU(),
    torch.nn.Dropout(0.2),  # problem with dropout layer
    torch.nn.Linear(H, H),
    torch.nn.LeakyReLU(),
    torch.nn.Dropout(0.2),  # problem with dropout layer
    torch.nn.Linear(H, D_out),
    torch.nn.Sigmoid()
)

checkpoint = torch.load("./saved_pytorch_model.pth")  # load pyTorch model
model.load_state_dict(checkpoint['state_dict'])
features = torch.Tensor(df_X.values[0])

# Convert pyTorch model to ONNX
input_names = ['input_1']
output_names = ['output_1']
for key, module in model._modules.items():
    input_names.append("l_{}_".format(key) + module._get_name())
torch_out = torch.onnx.export(model,
                         features,
                         "onnx_model.onnx",
                         export_params = True,
                         verbose = True,
                         input_names = input_names,
                         output_names = output_names,
                        )

What can I do to make it export to ONNX with Dropout included?


title: How are PyTorch's tensors implemented?

body:
I am building my own Tensor class in Rust, and I am trying to make it like PyTorch's implementation.

What is the most efficient way to store tensors programmatically, but, specifically, in a strongly typed language like Rust? Are there any resources that provide good insights into how this is done?

I am currently building a contiguous array, so that, given dimensions of 3 x 3 x 3, my array would just have 3^3 elements in it, which would represent the tensor. However, this does make some of the mathematical operations and manipulations of the array harder.

The dimension of the tensor should be dynamic, so that I could have a tensor with n dimensions.


title: Some parameters are not getting saved when saving a model in pytorch

body:
I have built an encoder-decoder model with attention for morph inflection generation. I am able to train the model and predict on test data but I am getting wrong predicting after loading a saved model
I am not getting any error during saving or loading but
When I load a saved model its predictions are completely wrong. It looks like some parameters are not getting saved.

I have tried to load and save the model using both techniques

using state_dict() eg. torch.save(encoder.state_dict(),'path')
saving complete model eg.torch.save(encoder,'path')

I have tried to save different classes one by one and also making a superclass that initiates all those class and then saving just superclass

but nothing seems to be working

Encoder class

class Encoder(nn.Module):
    def __init__(self,vocab_size,embedding_size, encoder_hid_dem,decoder_hid_dem,bidirectional,dropout):
        super().__init__()

        self.encoder_hid_dem  = encoder_hid_dem
        self.encoder_n_direction=1;
        self.bias = False
        self.dropout=dropout
        if(bidirectional==True):
            self.encoder_n_direction=2;

        self.embedding_layer  = nn.Embedding(num_embeddings=vocab_size, embedding_dim=embedding_size, padding_idx=0)
        self.GRU_layer        = nn.GRU(input_size=embedding_size, hidden_size=encoder_hid_dem, batch_first=True, bidirectional=bidirectional)
        self.fc               = nn.Linear(encoder_hid_dem*self.encoder_n_direction,decoder_hid_dem)
        self.dropout          = nn.Dropout(dropout)

    def forward(self, input_word):
        # print(input_word.size())
        #[batch_size    src_sent_lent]

        embed_out = self.embedding_layer(input_word)
        #[BATCH_SIZE    src_sent_lent   embedding_dim]

        embed_out = F.relu(embed_out)
        embed_out = self.dropout(embed_out)

        self.batch = embed_out.size()[0]

        # hidden =  self.init_hidden()
        GRU_out,hidden = self.GRU_layer(embed_out)

        # print(GRU_out.size())
        # print(hidd.size())

        #[BATCH_SIZE    sec_sent_len    n_direction*hid_dem]
        #[n_layer*n_direction   batch_size    hid_dem]

        #where the first hid_dim elements in the third axis are the hidden states from the top layer forward RNN, and the last hid_dim elements are hidden states from the top layer backward RNN

        #hidden is stacked [forward_1, backward_1, forward_2, backward_2, ...]
        #hidden [-2, :, : ] is the last of the forwards RNN
        #hidden [-1, :, : ] is the last of the backwards RNN

        GRU_out = F.relu(GRU_out)
        hidden = torch.tanh(self.fc(torch.cat((hidden[-2,:,:],hidden[-1,:,:]),dim=1)))

        # print(GRU_out.size())
        # print(hidden.size())

        #outputs = [batch_size    src sent len, encoder_hid_dim * n_direction]
        #hidden = [batch size, dec hid dim]
        return GRU_out,hidden

    def init_hidden(self):
        return (Variable(torch.eye(1, self.encoder_hid_dem)).unsqueeze(1).repeat(2, self.batch, 1).to(self.device))

Attention class

class Attention(nn.Module):
    def __init__(self,encoder_hid_dem,decoder_hid_dem,bidirectional):
        super().__init__()
        self.enc_hid_dim = encoder_hid_dem
        self.dec_hid_dim = decoder_hid_dem
        self.encoder_n_direction=1;
        if(bidirectional==True):
            self.encoder_n_direction=2;

        self.attn = nn.Linear((encoder_hid_dem * self.encoder_n_direction) + decoder_hid_dem, decoder_hid_dem)
        self.v = nn.Parameter(torch.rand(decoder_hid_dem))

    def forward(self, hidden, encoder_outputs):

        #hidden = [batch size, dec hid dim]
        #encoder_outputs = [batch_size    ,src sent len, enc hid dim * encoder_n_direction]

        batch_size = encoder_outputs.shape[0]
        src_len    = encoder_outputs.shape[1]

        hidden = hidden.unsqueeze(1).repeat(1, src_len, 1)

        #hidden          = [batch size, src sent len, dec hid dim]
        #encoder_outputs = [batch size, src sent len, enc hid dim * encoder_n_direction]

        energy = torch.tanh(self.attn(torch.cat((hidden, encoder_outputs), dim=2)))
        #energy = [batch size, src sent len, dec hid dim]

        energy = energy.permute(0, 2, 1)
        #energy = [batch size, dec hid dim, src sent len]

        #v = [dec hid dim]
        v = self.v.repeat(batch_size, 1).unsqueeze(1)
        #v = [batch size, 1, dec hid dim]

        attention = torch.bmm(v, energy).squeeze(1)
        #attention= [batch size, src len]

        return F.softmax(attention, dim=1)

Decoder class

class Decoder(nn.Module):
    def __init__(self, decoder_hid_dem, encoder_hid_dem, vocab_size,embedding_dim,attention,decoder_input_size,linear_input_size,bidirectional,dropout):
        super().__init__()
        self.encoder_hid_dem=encoder_hid_dem
        self.decoder_hid_dem=decoder_hid_dem
        self.attention=attention
        self.dropout = dropout
        self.output_dim = vocab_size

        self.decoder_n_direction=1;
        if(bidirectional==True):
            self.decoder_n_direction=2;

        self.GRU_layer_out = nn.GRU(decoder_input_size,decoder_hid_dem)
        self.out_layer = nn.Linear(in_features=linear_input_size, out_features=vocab_size)
        self.dropout = nn.Dropout(dropout)
        #self.GRU_layer_out.bias = torch.nn.Parameter(torch.zeros(decoder_input_size))

    def forward(self, feature, hidden,actual_word,encoder_outputs):

        feature = feature.unsqueeze(1)
        # print('decoder')
        # print(feature.size())
        #[batch_size    src_sent_lent=1   feat_size=6]

        # print(hidden.size())
        # [batch_size     dec_hid_dim]

        # print(actual_word.size())
        # [batch_size    src_sent_lent=1   embedding_dim]

        # print(encoder_outputs.size())
        # outputs = [batch_size    src sent len, encoder_hid_dim * encoder_n_directional]

        a = self.attention(hidden,encoder_outputs)
        #  print(a.size())
        # [batch_size    src_sent_len]

        a = a.unsqueeze(1)
        #a = [batch size, 1, src len]

        weighted = torch.bmm(a,encoder_outputs)
        # print(weighted.size())
        # weighted = [batch size, 1, enc_hid_dim * encoder_n_direction]
        # if len(actual_word.size()) != 0:
        input_char = torch.cat((actual_word,feature,weighted),2)
        # else:
        #     input_char = torch.cat((feature,weighted),2)

        input_char=input_char.permute(1,0,2)
        #  print(input_char.size())
        # [1    BATCH_SIZE      decoder_input_size]

        hidden = hidden.unsqueeze(0)
        # print(hidden.size())
        #[1 batch_size decoder_hid_dem]

        output, hidden = self.GRU_layer_out(input_char, hidden)

        # print(output.size())
        # [sent_len=1   batch_size  decoder_n_direction*decoder_hid_dem]
        # print(hidden.size())
        # [n_layer*n_direction    BATCH_SIZE      hid_dem]

        output = F.leaky_relu(output)
        output = self.dropout(output)

        output = torch.cat((output.squeeze(0),weighted.squeeze(1),actual_word.squeeze(1)),dim=1)
        pre_out = self.out_layer(output)
        predicted_output = F.log_softmax(pre_out, dim=1)

        # print(predicted_output.size())
        # [ batch_size vacab_size ]
        return predicted_output, hidden.squeeze(0)

    def init_hidden(self, batch):
        return (Variable(torch.eye(1, self.decoder_hid_dem)).unsqueeze(1).repeat(1, batch, 1).to(self.device),Variable(torch.eye(1, self.decoder_hid_dem)).unsqueeze(1).repeat(1, batch, 1).to(self.device))

seq2seq class

class Seq2Seq(nn.Module):
    def __init__(self,encoder,decoder,device):
        super().__init__()
        self.encoder = encoder
        self.decoder = decoder
        self.device = device

    def forward(self,input_word,output_word,features_word,teaching_forcing_ratio,limit):
        #print(input_word)
        #print(input_word.size())
        input_word = input_word.to(self.device)
        output_word = output_word.to(self.device)
        features_word = features_word.to(self.device)

        batch_size= input_word.size()[0]
        if(limit==0):
            max_len   = input_word.size()[1]
        else:
            max_len   = limit
        vocabsize = self.decoder.output_dim

        actual_word = self.encoder.embedding_layer(torch.tensor(char_to_index['<sos>']).view(1, -1).to(self.device)).repeat(batch_size, 1, 1)
        encoder_outputs,hidden = self.encoder(input_word)
        features=features_word[:,:]

        predicted_word = torch.zeros(max_len,batch_size,vocabsize).to(self.device)

        for t in range(1,max_len):
            output,hidden=self.decoder(features, hidden,actual_word,encoder_outputs)
            #print(output.size())
            predicted_word[t] = output
            topv, topi = output.topk(1)
            bs = topi.size()[0]
            temp2 = torch.zeros(0,1,300).to(self.device)
            for row in range(bs):
                index = topi[row][0].item()
                temp = self.encoder.embedding_layer(torch.tensor(index).view(1, -1).to(self.device))
                temp2 = torch.cat((temp2,temp))

            teacher_force = random.random() < teaching_forcing_ratio
            if teacher_force == 1:
                actual_word = self.encoder.embedding_layer(output_word[:,t]).unsqueeze(1)
            else:
                actual_word = temp2

        return predicted_word

and this code is used to save and load model

torch.save(model.state_dict(), 'model.pt')
model.load_state_dict(torch.load('model.pt'))

I want that when I run my model on pre-trained weights, it should predict correctly acc to those weights


title: Why does the parameters saved in the checkpoint are different from the ones in the fused model?

body:
I trained a QAT (Quantization Aware Training) based model in Pytorch, the training went on smoothly. However when I tried to load the weights into the fused model and run a test on widerface dataset I faced lots of errors:

(base) marian@u04-2:/mnt/s3user/Pytorch_Retinaface_quantized# python test_widerface.py --trained_model ./weights/mobilenet0.25_Final_quantized.pth --network mobile0.25layers:
Loading pretrained model from ./weights/mobilenet0.25_Final_quantized.pth
remove prefix 'module.'
Missing keys:235
Unused checkpoint keys:171
Used keys:65
Traceback (most recent call last):
  File "/root/.vscode/extensions/ms-python.python-2020.1.58038/pythonFiles/ptvsd_launcher.py", line 43, in <module>
    main(ptvsdArgs)
  File "/root/.vscode/extensions/ms-python.python-2020.1.58038/pythonFiles/lib/python/old_ptvsd/ptvsd/__main__.py", line 432, in main
    run()
  File "/root/.vscode/extensions/ms-python.python-2020.1.58038/pythonFiles/lib/python/old_ptvsd/ptvsd/__main__.py", line 316, in run_file
    runpy.run_path(target, run_name='__main__')
  File "/root/anaconda3/lib/python3.7/runpy.py", line 263, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "/root/anaconda3/lib/python3.7/runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "/root/anaconda3/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/mnt/f3user/Pytorch_Retinaface_quantized/test_widerface.py", line 114, in <module>
    net = load_model(net, args.trained_model, args.cpu)
  File "/mnt/f3user/Pytorch_Retinaface_quantized/test_widerface.py", line 95, in load_model
    model.load_state_dict(pretrained_dict, strict=False)
  File "/root/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 830, in load_state_dict
    self.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for RetinaFace:
        While copying the parameter named "ssh1.conv3X3.0.weight", whose dimensions in the model are torch.Size([32, 64, 3, 3]) and whose dimensions in the checkpoint are torch.Size([32, 64, 3, 3]).
        While copying the parameter named "ssh1.conv5X5_2.0.weight", whose dimensions in the model are torch.Size([16, 16, 3, 3]) and whose dimensions in the checkpoint are torch.Size([16, 16, 3, 3]).
        While copying the parameter named "ssh1.conv7x7_3.0.weight", whose dimensions in the model are torch.Size([16, 16, 3, 3]) and whose dimensions in the checkpoint are torch.Size([16, 16, 3, 3]).
        While copying the parameter named "ssh2.conv3X3.0.weight", whose dimensions in the model are torch.Size([32, 64, 3, 3]) and whose dimensions in the checkpoint are torch.Size([32, 64, 3, 3]).
        While copying the parameter named "ssh2.conv5X5_2.0.weight", whose dimensions in the model are torch.Size([16, 16, 3, 3]) and whose dimensions in the checkpoint are torch.Size([16, 16, 3, 3]).
.....

The full list can be found here.
basically the weights cant be found. plus the scale and zero_point which are missing from the fused model.

in case it matters, the following snippet is the actual training loop which was used to train and save the model :

if __name__ == '__main__':
    # train()
    ...
    net = RetinaFace(cfg=cfg)
    print("Printing net...")
    print(net)

    net.fuse_model()
    ...

    net.qconfig = torch.quantization.get_default_qat_qconfig('fbgemm')
    torch.quantization.prepare_qat(net, inplace=True)
    print(f'quantization preparation done.')

    ...

    quantized_model = net
    for i in range(max_epoch):
        net = net.to(device)
        train_one_epoch(net, data_loader, optimizer, criterion, cfg, gamma, i, step_index, device)
        if i in stepvalues:
            step_index += 1
        if i > 3 :
            net.apply(torch.quantization.disable_observer)
        if i > 2 :
            net.apply(torch.nn.intrinsic.qat.freeze_bn_stats)
        net=net.cpu()
        quantized_model = torch.quantization.convert(net.eval(), inplace=False)
        quantized_model.eval()
        # evaluate on test set ?!

    torch.save(net.state_dict(), save_folder + cfg['name'] + '_Final.pth')
    torch.save(quantized_model.state_dict(), save_folder + cfg['name'] + '_Final_quantized.pth')
    #torch.jit.save(torch.jit.script(quantized_model), save_folder + cfg['name'] + '_Final_quantized_jit.pth')

for testing the test_widerface.py is used which can be accessed here
You can view the keys here

Why has this happened? How should this be taken care of?

Update

I checked the name, and created a new state_dict dictionary and inserted the 112 keys that were in both checkpoint and model using the snippet below :

new_state_dict  = {}
checkpoint_state_dict = torch.load(checkpoint_path, map_location=lambda storage, loc: storage)
for (ck, cp) in checkpoint_state_dict.items():
    for (mk, mp) in model.state_dict().items():
        kname,kext = os.path.splitext(ck)
        mname,mext = os.path.splitext(mk)
        # check the two parameter and see if they are the same
        # then use models key naming scheme and use checkpoints weights
        if kname+kext == mname+mext or kname+'.0'+kext == mname+mext:
            new_state_dict[mname+mext] = cp
        else:
             if kext in ('.scale','.zero_point'):
                 new_state_dict[ck] = cp

and then use  this new state_dict! yet I'm getting  the ver same exact errors!
meaning errors like  this :

RuntimeError: Error(s) in loading state_dict for RetinaFace:
        While copying the parameter named "ssh1.conv3X3.0.weight", whose dimensions in the model are torch.Size([32, 64, 3, 3]) and whose dimensions in the checkpoint are torch.Size([32, 64, 3, 3]).

This is really frustrating and there is no documentation concerning this! I'm completely clueless here.


I know this seems to be a common problem but I wasn't able to find a solution. I'm running a multi-label classification model and having issues with tensor sizing.

My full code looks like this:

from transformers import DistilBertTokenizerFast, DistilBertForSequenceClassification
import torch

# Instantiating tokenizer and model
tokenizer = DistilBertTokenizerFast.from_pretrained('distilbert-base-cased')
model = DistilBertForSequenceClassification.from_pretrained('distilbert-base-cased')

# Instantiating quantized model
quantized_model = torch.quantization.quantize_dynamic(model, {torch.nn.Linear}, dtype=torch.qint8)

# Forming data tensors
input_ids = torch.tensor(tokenizer.encode(x_train[0], add_special_tokens=True)).unsqueeze(0)
labels = torch.tensor(Y[0]).unsqueeze(0)

# Train model
outputs = quantized_model(input_ids, labels=labels)
loss, logits = outputs[:2]

Which yields the error:

ValueError: Expected input batch_size (1) to match target batch_size (11)

Input_ids looks like:

tensor([[  101,   789,   160,  1766,  1616,  1110,   170,  1205,  7727,  1113,
           170,  2463,  1128,  1336,  1309,  1138,   112,   119, 11882, 11545,
           119,   108, 15710,   108,  3645,   108,  3994,   102]])

with shape:

torch.Size([1, 28])

and labels looks like:

tensor([[0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1]])

with shape:

torch.Size([1, 11])

The size of input_ids will vary as the strings to be encoded vary in size.

I also noticed that when feeding in 5 values of Y to produce 5 labels, it yields the error:

ValueError: Expected input batch_size (1) to match target batch_size (55).

with labels shape:

torch.Size([1, 5, 11])

(Note that I didn't feed 5 input_ids, which is presumably why input size remains constant)

I've tried a few different approaches to getting these to work, but I'm currently at a loss. I'd really appreciate some guidance. Thanks!


title: How do you generate positive definite matrix in pytorch?

body:
I am trying to define Multivariate Gaussian distribution with randomly generated covariance matrix:

psi = torch.zeros(512).normal_(0., 1.).requires_grad_(True)

# Generate random matrix
Sigma_k = torch.rand(512, 512)
# Make it symmetric positive
Sigma_k = Sigma_k * Sigma_k.t()
# Make it definite
Sigma_k.add_(0.001, torch.eye(512)).requires_grad_(True)

multivariate_normal.MultivariateNormal(psi, Sigma_k)

But I end up with getting an exception:

  RuntimeError: Lapack Error in potrf : the leading minor of order 2 is not positive definite at /Users/soumith/mc3build/conda-bld/pytorch_1549597882250/work/aten/src/TH/generic/THTensorLapack.cpp:658

What is the proper way to generate positive definite square matrix?


title: how to increase number of images with data augmentation

body:
I'm trying to apply data augmentation with pytorch. In particular, I have a dataset of 150 images and I want to apply 5 transformations (horizontal flip, 3 random rotation ad vertical flip) to every single image to have 750 images, but with my code I always have 150 images.

'train': transforms.Compose([
    transforms.Resize(224),
    transforms.RandomHorizontalFlip(),
    transforms.RandomRotation(degrees = (90,90)),
    transforms.RandomRotation(degrees = (180,180)),
    transforms.RandomRotation(degrees = (270,270)),
    transforms.RandomVerticalFlip(p=1),
    transforms.ToTensor(),
    transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
])


title: How To Fix: RuntimeError: size mismatch in pyTorch

body:
I am new to pyTorch and getting following Size Mismatch error:

RuntimeError: size mismatch, m1: [7 x 2092500], m2: [180 x 120] at ..\aten\src\TH/generic/THTensorMath.cpp:961

Model:

class Net(nn.Module):
def __init__(self):
    super(Net, self).__init__()
    self.conv1 = nn.Conv2d(3, 200, 5)
    self.pool = nn.MaxPool2d(2, 2)
    self.conv2 = nn.Conv2d(200, 180, 5)
    self.fc1 = nn.Linear(180, 120)
    self.fc2 = nn.Linear(120, 84)
    self.fc3 = nn.Linear(84,5)

def forward(self, x):
    x = self.pool(F.relu(self.conv1(x)))
    x = self.pool(F.relu(self.conv2(x)))
    x = x.view(x.shape[0], -1)
    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(x))
    x = self.fc3(x)
    return x

How ever I tried changing x = x.view(x.shape[0], -1) to x = x.view(x.size(0), -1) but that also did'nt work. Dimension of images is 512x384. and have used following transformation:

def load_dataset():
data_path = './dataset/training'

transform = transforms.Compose(
               [transforms.Resize((512,384)),
                transforms.ToTensor(),
                transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

train_dataset = torchvision.datasets.ImageFolder(root=data_path,transform=transform)
train_loader = torch.utils.data.DataLoader(train_dataset,batch_size=7,num_workers=0,shuffle=True)

return train_loader


title: How to specify pytorch / cuda version in pipenv

body:
I am trying to install a specific version of pytorch that is compatible with a specific cuda driver version with pipenv. The pytorch website shows how to to this with pip:

pip3 install torch==1.3.1+cu92 torchvision==0.4.2+cu92 -f https://download.pytorch.org/whl/torch_stable.html

I tried to convert this into an entry in my Pipfile like this:

[[source]]
name = "pytorch"
url = "https://download.pytorch.org/whl/torch_stable.html"
verify_ssl = false

pytorch = {version="==1.3.1+cu92", index="pytorch"}
torchvision = {version="==0.4.2+cu92", index="pytorch"}

However, this does not work. The dependency with this version can not be resolved. I am not sure if the url that is listed with the -f parameter in the pip3 command is even a valid source for pipenv.

I could install both libraries by just passing the command through to pip like this:

pipenv run pip install torch==1.3.1+cu92 torchvision==0.4.2+cu92 -f https://download.pytorch.org/whl/torch_stable.html

but I am not really satisfied with that solution since the dependencies are not in the Pipfile and I have to manually document the usage of this command.


title: Pytorch, TypeError: object() takes no parameters

body:
This is probably a beginner question, but nevertheless: When running an image classifier build with pytorch, I get this error:

Traceback (most recent call last):
File "/pytorch/kanji_torch.py", line 47, in <module>
    network = Network()
  File "/pytorch/kanji_torch.py", line 113, in __init__
    self.conv1 = nn.Conv2d(1, 32, 5)
  File "/python3.5/site-packages/torch/nn/modules/conv.py", line 233, in __init__
    False, _pair(0), groups, bias)
  File "/python3.5/site-packages/torch/nn/modules/conv.py", line 32, in __init__
    out_channels, in_channels // groups, *kernel_size))
TypeError: object() takes no parameters

I define the network class like this:

class Network(torch.nn.Module):
    def __init__(self):
        super(Network, self).__init__()
        self.conv1 = nn.Conv2d(1, 32, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(32, 64, 5)
        self.pool2 = nn.MaxPool2d(2, 2)
        self.conv3 = nn.Conv2d(64, 64, 5)
        self.pool2 = nn.MaxPool2d(2, 2)
        self.fc1 = nn.Linear(64 * 5 * 5, 512)
        self.fc2 = nn.Linear(512, 640)
        self.fc3 = nn.Linear(640, 3756)

Pretty sure that I imported all the relevant pytorch library modules correctly.
(import torch.nn as nn and
import torch)

Any ideas of what I'm doing wrong?

Thank you!


title: too many arguments in a Pytorch custom nn module forward function

body:
I'm trying to make a neural network in pytorch that has a variable number of layers. My problem is that apparently I am passing some sort of iterable with more than one item to a linear layer which can only take one argument. I just can't see why.

So here is some code. First I created my own module and import it to my notebook later

import torch

class NNet(torch.nn.Module):
    def __init__(self, layer_shapes, activation_functions):
        super(NNet, self).__init__()
        assert len(layer_shapes) == len(activation_functions) + 1
        self.layer_shapes = layer_shapes
        self.activation_functions = activation_functions

        linear_functions = list()
        for i in range(len(self.layer_shapes)-1):
            linear_functions.append(torch.nn.Linear(
                    self.layer_shapes[i], self.layer_shapes[i+1]))

        self.linear_functions = linear_functions

    def parameters(self):
        parameters = list()
        for function in self.linear_functions:
            parameters = parameters+list(function.parameters())

        return parameters

    def forward(self, x):
        assert x.shape[1] == self.layer_shapes[0]
        y = x
        for i in range(len(self.layer_shapes)-1):
            lin = self.linear_functions[i](y)
            y = self.activation_functions[i](lin)
        return y

In the notebook, the error is in the forward function at y = self.activation_functions[i](self.linear_functions[i](y))

Now I try to use the MNIST dataset supplied by torchvision and use my own module.

batch_size = 100
epochs = 500
learning_rate = 0.001

train_set = torchvision.datasets.MNIST(root =
                                       '../../data',
                                        train=True,
                          transform=torchvision.transforms.ToTensor(),
                                        download=True)

test_set = torchvision.datasets.MNIST(root =
                                      '../../data',
                                      train=False,
                          transform=torchvision.transforms.ToTensor(),
                                   download=True)

train_loader = torch.utils.data.DataLoader(dataset=train_set,
                                           batch_size=batch_size,
                                           shuffle=True)

test_loader = torch.utils.data.DataLoader(dataset=test_set,
                                          batch_size=batch_size,
                                          shuffle=False)

model = nnet.NNet([784, 16, 10], [torch.nn.Tanh,
                                  torch.nn.Softmax(dim=1)])

loss_function = torch.nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)

loss_items = list()

for t in range(epochs):
    for i, (images, labels) in enumerate(train_loader):
        images = images.reshape(-1,28*28)

        outputs = model(images)
        loss = loss_function(outputs, labels)
        loss_items.append(loss.item())

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

This last for loop yields the error:

TypeError                      Traceback (most recent call last)
<ipython-input-6-4ccb4b105a41> in <module>()
      5         images = images.reshape(-1,28*28)
      6
----> 7         outputs = model(images)
      8         loss = loss_function(outputs, labels)
      9         loss_items.append(loss.item())

~/.local/lib/python3.6/site-packages/torch/nn/modules/module.py in
__call__(self, *input, **kwargs)
    475             result = self._slow_forward(*input, **kwargs)
    476         else:
--> 477             result = self.forward(*input, **kwargs)
    478         for hook in self._forward_hooks.values():
    479             hook_result = hook(self, input, result)

~/Desktop/Archive/Computing/Projects/Python/ai/neural_network.py in
forward(self, x)
     28         for i in range(len(self.layer_shapes)-1):
     29             lin = self.linear_functions[i](y)
---> 30             y = self.activation_functions[i](lin)
     31         return y
     32

TypeError: __init__() takes 1 positional argument but 2 were given

I'm sure someone could tell me why this is happening, but could someone please give me a helpful strategy for debugging here? I'm new to pytorch and I doubt this will be the last trouble that I have. So a strategy for investigating these things would be helpful.

I would appreciate any input.


title: Does PyTorch seed affect dropout layers?

body:
I came across the idea of seeding my neural network for reproducible results, and was wondering if pytorch seeding affects dropout layers and what is the proper way to seed my training/testing?

I'm reading the documentation here, and wondering if just placing these lines will be enough?

torch.manual_seed(1)
torch.cuda.manual_seed(1)


I have a seq2seq network (a class) which is trained without any problem. I have just a constructor and a forward function in that class.

During testing, I am loading the model states and trying to call a test function of which executes testing but getting the following error.

if model.config.model == 'LSTM':
  File "/if5/wua4nw/anaconda3/lib/python3.5/site-packages/torch/nn/modules/module.py", line 237, in __getattr__
    return object.__getattr__(self, name)
AttributeError: type object 'object' has no attribute '__getattr__'

What I am doing:

model = Sequence2Sequence(dictionary, embeddings_index, args.max_length, args)
if args.cuda:
    model = torch.nn.DataParallel(model).cuda()
helper.load_model_states(model, filename)
test(model, input_sentence)

When program attempts to execute the last line, it produces the above mentioned error. Any idea why it is not working?

model.test function:

if model.config.model == 'LSTM':
    encoder_hidden, encoder_cell = model.encoder.init_weights(batch_sentence.size(0))
    output, hidden = model.encoder(batch_sentence, (encoder_hidden, encoder_cell))
else:
    encoder_hidden = model.encoder.init_weights(batch_sentence.size(0))
    output, hidden = model.encoder(batch_sentence, encoder_hidden)

# rest of the code goes here...


Returned iterator of BucketIterator is giving an error.

train_iterator = data.BucketIterator.splits(
    train_data,
    batch_size=BATCH_SIZE,
    device=device
)

Value of train_data also looks good.

print(vars(train_data.examples[10]))
{'text': ['plus',
  "you've",
  'added',
  'commercials',
  'to',
  'the',
  'experience...',
  'tacky.'],
 'label': 'positive'}

Stack trace:

print([x for x in train_iterator])

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-50-f14edc400c88> in <module>()
----> 1 [x for x in train_iter]

<ipython-input-50-f14edc400c88> in <listcomp>(.0)
----> 1 [x for x in train_iter]

~/anaconda3/envs/env/lib/python3.6/site-packages/torchtext/data/iterator.py in __iter__(self)
    154                     else:
    155                         minibatch.sort(key=self.sort_key, reverse=True)
--> 156                 yield Batch(minibatch, self.dataset, self.device)
    157             if not self.repeat:
    158                 return

~/anaconda3/envs/env/lib/python3.6/site-packages/torchtext/data/batch.py in __init__(self, data, dataset, device)
     32                 if field is not None:
     33                     batch = [getattr(x, name) for x in data]
---> 34                     setattr(self, name, field.process(batch, device=device))
     35
     36     @classmethod

~/anaconda3/envs/env/lib/python3.6/site-packages/torchtext/data/field.py in process(self, batch, device)
    235         """
    236         padded = self.pad(batch)
--> 237         tensor = self.numericalize(padded, device=device)
    238         return tensor
    239

~/anaconda3/envs/env/lib/python3.6/site-packages/torchtext/data/field.py in numericalize(self, arr, device)
    334         if self.use_vocab:
    335             if self.sequential:
--> 336                 arr = [[self.vocab.stoi[x] for x in ex] for ex in arr]
    337             else:
    338                 arr = [self.vocab.stoi[x] for x in arr]

~/anaconda3/envs/env/lib/python3.6/site-packages/torchtext/data/field.py in <listcomp>(.0)
    334         if self.use_vocab:
    335             if self.sequential:
--> 336                 arr = [[self.vocab.stoi[x] for x in ex] for ex in arr]
    337             else:
    338                 arr = [self.vocab.stoi[x] for x in arr]

~/anaconda3/envs/env/lib/python3.6/site-packages/torchtext/data/field.py in <listcomp>(.0)
    334         if self.use_vocab:
    335             if self.sequential:
--> 336                 arr = [[self.vocab.stoi[x] for x in ex] for ex in arr]
    337             else:
    338                 arr = [self.vocab.stoi[x] for x in arr]

AttributeError: 'Field' object has no attribute 'vocab'

May I know what is it that we are missing here?


title: PyTorch C++ - how to know the recommended version of cuDNN?

body:
I've previously inferenced TensorFlow graphs from C++.  Now I'm embarking on working out how to inference PyTorch graphs via C++.

My first question is, how can I know the recommended version of cuDNN to use with LibTorch, or if I'm doing my own PyTorch compile?

Determining the recommended CUDA version is easy.  Upon going to https://pytorch.org/ and choosing the options under Quick Start Locally (PyTorch Build, Your OS, etc.) the site makes it pretty clear that CUDA 10.1 is recommended, but there is no mention of cuDNN version and upon Googling I'm unable to find a definitive answer for this.

From what I understand about PyTorch on ubuntu, if you use the Python version you have to install the CUDA driver (ex. so nvidia-smi works, version 440 currently), but the CUDA and cuDNN install are not actually required beyond the driver because they are included in the pip3 package, is this correct?  If so, then is there a command I can run in a Python script that shows the version of CUDA (expected to be 10.1) and cuDNN that the pip pre-compiled .whl uses?  I suspect there is such a command but I'm not familiar enough with PyTorch yet to know what that may be or how to look it up.

I've ran into compile and inferencing errors using C++ with TensorFlow when I was not using the specific recommended version of cuDNN for a certain version of TensorFlow and CUDA so I'm aware these version can be sensitive and I have to make the right choices from the get-go.  If anybody can assist in determining the recommended version of cuDNN for a certain version of PyTorch that would be great.


title: Pytorch: Add input normalization to model (division layer)

body:
I want to add the image normalization to an existing pytorch model, so that I don't have to normalize the input image anymore.
Say I have an existing model
model = torch.hub.load('pytorch/vision:v0.6.0', 'mobilenet_v2', pretrained=True)
model.eval()

Now I can add new layers (for example a relu) using torch.nn.Sequential:
new_model = nn.Sequential(
    model,
    nn.ReLU()
)

However I couldn't find a layer to do perform just a division or subtraction as needed for the input normalization here shown in numpy:
import cv2
import numpy as np
img = cv2.imread("my_img.jpg")
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
img = img.astype(np.float32)
mean = np.array([0.485, 0.456, 0.406], dtype=np.float32)
std = np.array([0.229, 0.224, 0.225], dtype=np.float32)
img = img / 255.0
img = img - mean
img = img / std
img = np.transpose(img, (2, 0, 1))
img = np.expand_dims(img, axis=0)

The goal is that normalization is eventually done on GPU to save time during inference. Also I cannot use torchvision transforms as those operation are not stored inside the model itself. For example, if I want to save the model to disk (in order to convert it to tflite using onnx) the torchvision transform operations will not be saved along with the model. Is there an elegant way of doing this?
(preferably without using a linear layer, which would fix my model input size, which should be flexible as my real model is fully convolutional)


title: How does one make sure that the parameters are update manually in pytorch using modules?

body:
I wanted to update the parameters of a model manually with pytorch. I made a super simple standard sequential model (full code here) but whenever I try to train my model it does not train unless I create the actual variables explicitly (code for model variables explicitly). So with the sequential model the code looks as follow:

mdl_sgd = torch.nn.Sequential( torch.nn.Linear(D_sgd,1,bias=False) )
...
for i in range(nb_iter):
    # Forward pass: compute predicted Y using operations on Variables
    batch_xs, batch_ys = get_batch2(X,Y,M,dtype) # [M, D], [M, 1]
    ## FORWARD PASS
    y_pred = mdl_sgd.forward(X)
    ## LOSS
    loss = (1/N)*(y_pred - batch_ys).pow(2).sum()
    ## Manually zero the gradients after updating weights
    mdl_sgd.zero_grad()
    ## BACKARD PASS
    loss.backward() # Use autograd to compute the backward pass. Now w will have gradients
    ## SGD update
    for W in mdl_sgd.parameters():
        #print(W.grad.data)
        W.data = W.data - eta*W.grad.data

when I train it it seems that nothing happens. I've tried many things to make this work like wrapping it in a class and putting explicit require_grads=True or change the locations where I make the zero out the gradients etc but nothing seems to work. What I really want/need is to be able to explicitly be able to do the update rule myself (not with optimum). Not sure if thats the reason it doesn't work but the following does work for some reason:

X = poly_kernel_matrix(x_true,Degree_mdl) # maps to the feature space of the model
X = Variable(torch.FloatTensor(X).type(dtype), requires_grad=False)
Y = Variable(torch.FloatTensor(Y).type(dtype), requires_grad=False)
w_init=torch.randn(D_sgd,1).type(dtype)
W = Variable( w_init, requires_grad=True)
...
for i in range(nb_iter):
        # Forward pass: compute predicted Y using operations on Variables
        batch_xs, batch_ys = get_batch2(X,Y,M,dtype) # [M, D], [M, 1]
        ## FORWARD PASS
        #y_pred = mdl_sgd.forward(X)
        y_pred = batch_xs.mm(W)
        ## LOSS
        loss = (1/N)*(y_pred - batch_ys).pow(2).sum()
        ## BACKARD PASS
        loss.backward() # Use autograd to compute the backward pass. Now w will have gradients
        ## SGD update
        W.data = W.data - eta*W.grad.data
        ## Manually zero the gradients after updating weights
        #mdl_sgd.zero_grad()
        W.grad.data.zero_()

the reason I know this is because the plot of the regression lines look sensible:

while when I use the torch.nn.Sequential I get:

I am sure its a really newbie question but I am not sure why I can't update the parameters. Does someone know why? I want to be able to update the parameters manually (however I want) and in this case I decided to use SGD to see if I could even update the parameters.

Note I also tried subclassing modules and registering params but it didn't work either. This is the class I built:

class regression_NN(torch.nn.Module):

def __init__(self,w_init):
    """
    """
    super(type(self), self).__init__()
    # mdl
    #self.W = Variable(w_init, requires_grad=True)
    #self.W = torch.nn.Parameter( Variable(w_init, requires_grad=True) )
    #self.W = torch.nn.Parameter( w_init )
    self.W = torch.nn.Parameter( w_init,requires_grad=True )
    #self.mod_list = torch.nn.ModuleList([self.W])

def forward(self, x):
    """
    """
    y_pred = x.mm(self.W)
    return y_pred

All code is:

https://github.com/brando90/simple_regression

I'm relatively new at pytorch so I might have many bad practice...you can correct them if u want but Im mostly concerned that my paremters are not updating even when I try to explicitly register them in a class that inherits from torch.nn.Module.

I also linked to the question from the pytorch official forum: https://discuss.pytorch.org/t/how-does-one-make-sure-that-the-parameters-are-update-manually-in-pytorch-using-modules/6076


title: Implementing FFT with Pytorch

body:
I am trying to implement FFT by using the conv1d function provided in Pytorch.

Generating artifical signal

import numpy as np
import torch
from torch.autograd import Variable
from torch.nn.functional import conv1d

from scipy import fft, fftpack

import matplotlib.pyplot as plt

%matplotlib inline

# Creating filters

d = 4096 # size of windows

def create_filters(d):
    x = np.arange(0, d, 1)
    wsin = np.empty((d,1,d), dtype=np.float32)
    wcos = np.empty((d,1,d), dtype=np.float32)
    window_mask = 1.0-1.0*np.cos(x)
    for ind in range(d):
        wsin[ind,0,:] = np.sin(2*np.pi*((ind+1)/d)*x)
        wcos[ind,0,:] = np.cos(2*np.pi*((ind+1)/d)*x)

    return wsin,wcos

wsin, wcos = create_filters(d)
wsin_var = Variable(torch.from_numpy(wsin), requires_grad=False)
wcos_var = Variable(torch.from_numpy(wcos),requires_grad=False)

# Creating signal

t = np.linspace(0,1,4096)
x = np.sin(2*np.pi*100*t)+np.sin(2*np.pi*200*t)+np.random.normal(scale=5,size=(4096))

plt.plot(x)

FFT with Pytorch

signal_input = torch.from_numpy(x.reshape(1,-1),)[:,None,:4096]

signal_input = signal_input.float()

zx = conv1d(signal_input, wsin_var, stride=1).pow(2)+conv1d(signal_input, wcos_var, stride=1).pow(2)

FFT with Scipy

fig = plt.figure(figsize=(20,5))
plt.plot(np.abs(fft(x).reshape(-1))[:500])

My Question

As you can see the two outputs are quite similar in terms of the peaks characteristics. That means my implementation is not totally wrong.
However, there are also some subtleties, such as the scale of the spectrum, and the signal to noise ratio. I am unable to figure out what's missing here to get the exact same result.


title: Pass user specified parameters to DataLoader

body:
I am using U - Net and implementing the weighting technique described in the papers from 2015 (U-Net: Convolutional Networks for Biomedical
Image Segmentation) and 2019 (U-Net – Deep Learning for Cell Counting, Detection, and Morphometry). In that technique there is a variance σ and a weight w_0. I would like, especially the σ, to be a learnable parameter instead of guessing which value is best from dataset to dataset.

From what I found, I can do this using nn.Parameter.
To use the learned σ from epoch to epoch, I need somehow to pass this new value to the get_item function of the DataSet through the DataLoader.

My current take on this, is to extend torch.utils.data.DataLoader where the new init has an extra parameter accepting the user specified/learnable parameters. Given the source code of torch.utils.data.DataLoader, I do not understand where and how the DataLoader calls the DataSet instance and hence to pass these parameters.
Code wise, in the DataSet definition there is the function
def __getitem__(self, index):

that I can change as
def __getitem__(self, index, sigma):

and to make use of the updated, newly learned σ.
My problem is that during training, I iterate through training dataset as
for epoch in range( checkpoint[ 'epoch'], num_epochs):
....
    for ii, ( X, y, y_weight, fname) in enumerate( dataLoader[ phase]):

In that enumeration of DataLoader, how can I pass the new σ to the DataLoader such that the DataLoader will pass it to the DataSet getitem function mentioned above?
EDIT
Currently, I define inside the DataSet class a parameter sigma
class MedicalImageDataset( Dataset):
      def __init__(self, fname, img_transform = None, mask_transform = None, weight_transform = None, sigma = 8):
            ...
            self.sigma = sigma

      def __getitem__(self, index):
            sigma = self.sigma
            ...

which I update through the DataLoader as
dataLoader[ 'train'].dataset.sigma = model.sigma

where,
model.sigma

is a custom parameter defined as
model.register_parameter( name = 'sigma', param = torch.nn.Parameter( torch.tensor( 16, dtype = torch.float16), requires_grad = True))

after creating the model.
My problem is, that model.sigma doesn't look being updated from epoch to epoch. Specifically, is the same as the initial value. Why is this?
Having a look at optimizer.state_dict() I couldn't find any parameter named 'sigma', whereas I can find one in model.named_parameters().
Finally, this parameter sigma is not attached to any layer, it's kinda "free".


title: Very simple torch.tensor().to("cuda") gives CUDA error: device-side assert triggered

body:
All I do is,
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
torch.tensor([123,123]).to(device)

And I get:
RuntimeError: CUDA error: device-side assert triggered

I really cannot see why.
Edit: Weirdly enough, I see that this happens only I after I run some code before hand. Restarting the kernel solves it. But why, some code can cause a problem like this? I cannot share the code because it's not mine but I still welcome any guesses. Thanks!


title: What exactly is label for image segmentation task in computer vision

body:
I have been working on a some image segmentation tasks lately and would like to apply one from scratch.

Segmentation as I have understood is the per pixel prediction to where it belongs - to an object instance(things), to a background segment instance(stuff).

As per the COCO dataset on which the latest algorithm Mask RCNN is based :

  things are countable objects such as people, animals, tools. Stuff classes are amorphous regions of similar texture or material such as grass, sky, road.

As per the Mask Rcnn paper the final classification is a binary cross entropy loss function taking per pixel sigmoid (to avoid intra-class race). This pipeline is based on top of the FRCNN object detection pipeline from where it gets the Region-Of-Interest (roi) and passes them through a ROI-align class to keep the spatial information intact.

What I'm confused with is the following. Given a very simple code snippet below, for applying Binary Cross Entropy loss to separate 3 fully connected layers( some random experiment with scales):

class ModelMain(nn.Module):
    def __init__(self, config, is_training=True):
        super(ModelMain, self).__init__()
        self.fc_1 = torch.nn.Linear(incoming_size_1, outgoing_size_1)
        self.fc_2 = torch.nn.Linear(incoming_size_2, outgoing_size_2)
        self.fc_3 = torch.nn.Linear(incoming_size_3, outgoing_size_3)

    def forward(self, x):
        y_1 = F.sigmoid(self.fc_1(x))
        y_2 = F.sigmoid(self.fc_2(x))
        y_3 = F.sigmoid(self.fc_3(x))

        return y_1, y_2, y_3

model = ModelMain()
criterion = torch.nn.BCELoss(size_average = True)
optimizer = torch.optim.SGD(model.parameters(), lr = 0.01)

def run_epoch():
    batchsize = 10
    for epoch in range(batchsize):
        # Find image segment predicted by running forward pass:
        y_predicted_1, y_predicted_2, y_predicted_3  = model(batch_data_x)

        # Compute and print loss :
        loss_1 = criterion(y_predicted_1, batch_data_y)
        loss_2 = criterion(y_predicted_2, batch_data_y)
        loss_3 = criterion(y_predicted_3, batch_data_y)

        print( "Epoch ", epoch, "Loss : ", loss_1, loss_2, loss_3)

        # Perform Backward pass :
        optimizer.zero_grad()
        loss_1.backward()
        loss_2.backward()
        loss_3.backward()
        optimizer.step()

... what exactly do we provide here as label?

From the dataset :

Formatted JSON Data

image :

 {
       "license":2,
       "file_name":"000000000139.jpg",
       "coco_url":"http://images.cocodataset.org/val2017/000000000139.jpg",
       "height":426,
       "width":640,
       "date_captured":"2013-11-21 01:34:01",
       "flickr_url":"http://farm9.staticflickr.com/8035/8024364858_9c41dc1666_z.jpg",
       "id":139
    }

Segment info :

{
   "segments_info":[
      {
         "id":3226956,
         "category_id":1,
         "iscrowd":0,
         "bbox":[
            413,
            158,
            53,
            138
         ],
         "area":2840
      },
      {
         "id":6979964,
         "category_id":1,
         "iscrowd":0,
         "bbox":[
            384,
            172,
            16,
            36
         ],
         "area":439
      },
      {
         "id":3103374,
         "category_id":62,
         "iscrowd":0,
         "bbox":[
            413,
            223,
            30,
            81
         ],
         "area":1250
      },
      {
         "id":2831194,
         "category_id":62,
         "iscrowd":0,
         "bbox":[
            291,
            218,
            62,
            98
         ],
         "area":1848
      },
      {
         "id":3496593,
         "category_id":62,
         "iscrowd":0,
         "bbox":[
            412,
            219,
            10,
            13
         ],
         "area":90
      },
      {
         "id":2633066,
         "category_id":62,
         "iscrowd":0,
         "bbox":[
            317,
            219,
            22,
            12
         ],
         "area":212
      },
      {
         "id":3165572,
         "category_id":62,
         "iscrowd":0,
         "bbox":[
            359,
            218,
            56,
            103
         ],
         "area":2251
      },
      {
         "id":8824489,
         "category_id":64,
         "iscrowd":0,
         "bbox":[
            237,
            149,
            24,
            62
         ],
         "area":369
      },
      {
         "id":3032951,
         "category_id":67,
         "iscrowd":0,
         "bbox":[
            321,
            231,
            126,
            89
         ],
         "area":2134
      },
      {
         "id":2038814,
         "category_id":72,
         "iscrowd":0,
         "bbox":[
            7,
            168,
            149,
            95
         ],
         "area":13247
      },
      {
         "id":3289671,
         "category_id":72,
         "iscrowd":0,
         "bbox":[
            557,
            209,
            82,
            79
         ],
         "area":5846
      },
      {
         "id":2437710,
         "category_id":78,
         "iscrowd":0,
         "bbox":[
            512,
            206,
            15,
            16
         ],
         "area":224
      },
      {
         "id":4159376,
         "category_id":82,
         "iscrowd":0,
         "bbox":[
            493,
            174,
            20,
            108
         ],
         "area":2056
      },
      {
         "id":3423599,
         "category_id":84,
         "iscrowd":0,
         "bbox":[
            613,
            308,
            13,
            46
         ],
         "area":324
      },
      {
         "id":3094634,
         "category_id":84,
         "iscrowd":0,
         "bbox":[
            605,
            306,
            14,
            45
         ],
         "area":331
      },
      {
         "id":3296100,
         "category_id":85,
         "iscrowd":0,
         "bbox":[
            448,
            121,
            14,
            22
         ],
         "area":227
      },
      {
         "id":6054280,
         "category_id":86,
         "iscrowd":0,
         "bbox":[
            241,
            195,
            14,
            18
         ],
         "area":187
      },
      {
         "id":5942189,
         "category_id":86,
         "iscrowd":0,
         "bbox":[
            549,
            309,
            36,
            90
         ],
         "area":2171
      },
      {
         "id":4086154,
         "category_id":86,
         "iscrowd":0,
         "bbox":[
            351,
            209,
            11,
            22
         ],
         "area":178
      },
      {
         "id":7438777,
         "category_id":86,
         "iscrowd":0,
         "bbox":[
            337,
            200,
            10,
            16
         ],
         "area":120
      },
      {
         "id":3031159,
         "category_id":118,
         "iscrowd":0,
         "bbox":[
            0,
            269,
            564,
            157
         ],
         "area":49754
      },
      {
         "id":9284267,
         "category_id":119,
         "iscrowd":0,
         "bbox":[
            338,
            166,
            29,
            50
         ],
         "area":842
      },
      {
         "id":6068135,
         "category_id":130,
         "iscrowd":0,
         "bbox":[
            212,
            11,
            321,
            127
         ],
         "area":3391
      },
      {
         "id":2567230,
         "category_id":156,
         "iscrowd":0,
         "bbox":[
            129,
            168,
            351,
            162
         ],
         "area":5699
      },
      {
         "id":10334639,
         "category_id":181,
         "iscrowd":0,
         "bbox":[
            204,
            63,
            234,
            174
         ],
         "area":15587
      },
      {
         "id":6266027,
         "category_id":186,
         "iscrowd":0,
         "bbox":[
            136,
            0,
            473,
            116
         ],
         "area":20106
      },
      {
         "id":5274512,
         "category_id":188,
         "iscrowd":0,
         "bbox":[
            0,
            38,
            549,
            297
         ],
         "area":25483
      },
      {
         "id":7238567,
         "category_id":189,
         "iscrowd":0,
         "bbox":[
            457,
            350,
            183,
            76
         ],
         "area":9421
      },
      {
         "id":4224910,
         "category_id":199,
         "iscrowd":0,
         "bbox":[
            0,
            0,
            640,
            358
         ],
         "area":83201
      },
      {
         "id":6391959,
         "category_id":200,
         "iscrowd":0,
         "bbox":[
            135,
            359,
            336,
            67
         ],
         "area":12618
      }
   ],
   "file_name":"000000000139.png",
   "image_id":139
}

The Mask image :

The Original image :

For the object detection task we have bounding box, but for image segmentation I need to calculate loss with the mask provided.
So what should be the value for the  batch_data_y in the above code.
Will it be the vector for the mask image.  But doesn't that train my network as to what color some segment is ?  Or am I missing some other segment annotation ?


title: Numpy / PyTorch - how to assign value with indices in different dimensions?

body:
Suppose I have a matrix and some indices

a = np.array([[1, 2, 3], [4, 5, 6]])
a_indices = np.array([[0,2], [1,2]])

Is there any efficient way to achieve following operation?

for i in range(2):
    a[i, a_indices[i]] = 100

# a: np.array([[100, 2, 100], [4, 100, 100]])