title: How do loss functions know for which model to compute gradients in PyTorch?

body:
I am unsure how PyTorch manges to link the loss function to the model I want it to be computed for. There is never an explicit reference between the loss and the model, such as the one between the model's parameters and the optimizer.

Say for example I want to train 2 networks on the same dataset, so I want to utilize a single pass through the dataset. How would PyTorch link the appropriate loss functions to the appropriate models. Here's code for reference:

import torch
from torch import nn, optim
import torch.nn.functional as F
from torchvision import datasets, transforms
import shap

# Define a transform to normalize the data
transform = transforms.Compose([transforms.ToTensor(),
                                transforms.Normalize((0.5,), (0.5,)),
                              ])
# Download and load the training data
trainset = datasets.MNIST('~/.pytorch/MNIST_data/', download=True, train=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=64, shuffle=True)

model = nn.Sequential(nn.Linear(784, 128),
                      nn.ReLU(),
                      nn.Linear(128, 64),
                      nn.ReLU(),
                      nn.Linear(64, 10),
                      nn.LogSoftmax(dim=1))

model2 = nn.Sequential(nn.Linear(784, 128),
                      nn.ReLU(),
                      nn.Linear(128, 10),
                      nn.LogSoftmax(dim=1))

# Define the loss
criterion = nn.NLLLoss()
criterion2 = nn.NLLLoss()

optimizer = optim.SGD(model.parameters(), lr=0.003)
optimizer2 = optim.SGD(model2.parameters(), lr=0.003)

epochs = 5
for e in range(epochs):
    running_loss = 0
    running_loss_2 = 0
    for images, labels in trainloader:
        # Flatten MNIST images into a 784 long vector
        images = images.view(images.shape[0], -1) # batch_size x total_pixels

        # Training pass
        optimizer.zero_grad()
        optimizer2.zero_grad()

        output = model(images)
        loss = criterion(output, labels)
        loss.backward()
        optimizer.step()

        output2 = model2(images)
        loss2 = criterion2(output2, labels)
        loss2.backward()
        optimizer2.step()

        running_loss += loss.item()
        running_loss_2 += loss2.item()

    print(f"Training loss 1: {running_loss/len(trainloader)}")
    print(f"Training loss 2: {running_loss_2/len(trainloader)}")
    print()

So once again, how does pytorch know how to compute the appropriate gradients for the appropriate models when loss.backward() and loss2.backward() are called?


title: Trouble using openCV to load a net from ONNX (python/pytorch)

body:
I'm trying to load a trained .onnx model (from a neural-style-transfer algorithm) into cv2. I've seen that there is a

cv.dnn.readNetFromONNX()

function, but there is no such function in cv2. I can't seem to import or load opencv as cv, and as such can't seem to load my model in cv2. Does anyone know a solution?

I've basically trained a model with
https://github.com/pytorch/examples/blob/master/fast_neural_style/neural_style/neural_style.py#L122-L150
this script, and made an export of an onnx model by adding

torch.onnx.export(style_model, dummy_input, "chipsoft_mod.onnx", verbose=True)

Now I want to run the trained model trough the cv2 reader, but I fail spectacularly.


title: Convolutional Autoencoder in Pytorch for Dummies

body:
I am here to ask some more general questions about Pytorch and Convolutional Autoencoders.

If I only use Convolutional Layers (FCN), do I even have to care about the input shape? And then how do I choose the number of featuremaps best?
Does a ConvTranspose2d Layer automatically unpool?
Can you spot any errors or unconventional code in my example?

By the way, I want to make a symmetrical Convolutional Autoencoder to colorize black and white images with different image sizes.

    self.encoder = nn.Sequential (
        # conv 1
        nn.Conv2d(in_channels=3, out_channels=512, kernel_size=3, stride=1, padding=1),
        nn.ReLU,
        nn.MaxPool2d(kernel_size=2, stride=2),  # 1/2
        nn.BatchNorm2d(512),

        # conv 2
        nn.Conv2d(in_channels=512, out_channels=256, kernel_size=3, stride=1, padding=1),
        nn.ReLU,
        nn.MaxPool2d(kernel_size=2, stride=2),  # 1/4
        nn.BatchNorm2d(256),

        # conv 3
        nn.Conv2d(in_channels=256, out_channels=128, kernel_size=3, stride=1, padding=1),
        nn.ReLU,
        nn.MaxPool2d(kernel_size=2, stride=2),  # 1/8
        nn.BatchNorm2d(128),

        # conv 4
        nn.Conv2d(in_channels=128, out_channels=64, kernel_size=3, stride=1, padding=1),
        nn.ReLU,
        nn.MaxPool2d(kernel_size=2, stride=2),  #1/16
        nn.BatchNorm2d(64)
    )

    self.encoder = nn.Sequential (
        # conv 5
        nn.ConvTranspose2d(in_channels=64, out_channels=128, kernel_size=3, stride=1, padding=1),
        nn.ReLU,
        nn.BatchNorm2d(128),

        # conv 6
        nn.ConvTranspose2d(in_channels=128, out_channels=256, kernel_size=3, stride=1, padding=1),
        nn.ReLU,
        nn.BatchNorm2d(256),

        # conv 7
        nn.ConvTranspose2d(in_channels=256, out_channels=512, kernel_size=3, stride=1, padding=1),
        nn.ReLU,
        nn.BatchNorm2d(512),

        # conv 8
        nn.ConvTranspose2d(in_channels=512, out_channels=512, kernel_size=3, stride=1, padding=1),
        nn.Softmax()
    )

    def forward(self, x):
        h = x
        h = self.encoder(h)
        h = self.decoder(h)
        return h


title: Best loss function for multi-class classification when the dataset is imbalance?

body:
I'm currently using the Cross Entropy Loss function but with the imbalance data-set the performance is not great.

Is there better lost function?


I am new to Deep-Learning, and I try to recurrence the code in https://github.com/swamiviv/LSD-seg, which is used for image semantic segmentation, but a problem happened in Pytorch, I really need your help.

Thanks very much.

Here's the environment:

Ubuntu 14.04.1
Python 3.6.4
Pytorch 0.4.0

The code is for downloading the pretrained model but my server can't visit googledrive, so I downloaded the model from other method and updated the model file to the server, and changed the related code, but it didn't work.

The original code:

import os.path as osp
import fcn
import torch
import torchvision

def VGG16(pretrained=False):
    model = torchvision.models.vgg16(pretrained=False)
    if not pretrained:
        return model

    model_file = _get_vgg16_pretrained_model()
    state_dict = torch.load(model_file)
    model.load_state_dict(state_dict)
    return model

def _get_vgg16_pretrained_model():
    return fcn.data.cached_download(url='http://drive.google.com/uc?id=0B9P1L--7Wd2vLTJZMXpIRkVVRFk',path=osp.expanduser('/home/sunyuze/LSD/data/vgg16_from_caffe.pth'),md5='aa75b158f4181e7f6230029eb96c1b13',)

After I downloaded the model file by other method and put it in the specific position, I modified the code:

import os.path as osp
import fcn
import torch
import torchvision

def VGG16(pretrained=False):
    model = torchvision.models.vgg16(pretrained=False)
    if not pretrained:
        return model

    state_dict = torch.load('/home/sunyuze/LSD/data/vgg16_from_caffe.pth')
    model.load_state_dict(state_dict)
    return model

The error alert is :

Traceback (most recent call last):
  File "train.py", line 157, in <module>
    main()
  File "train.py", line 76, in main
    vgg16 = torchfcn.models.VGG16(pretrained=True)
  File "/home/sunyuze/LSD/code/torchfcn/models/vgg.py", line 17, in VGG16
    state_dict = torch.load('/home/sunyuze/LSD/data/vgg16_from_caffe.pth')
  File "/home/sunyuze/anaconda3/lib/python3.6/site-packages/torch/serialization.py", line 303, in load
    return _load(f, map_location, pickle_module)
  File "/home/sunyuze/anaconda3/lib/python3.6/site-packages/torch/serialization.py", line 459, in _load
    magic_number = pickle_module.load(f)
_pickle.UnpicklingError: unpickling stack underflow


title: What is the default batch size of pytorch SGD?

body:
What does pytorch SGD do if I feed the whole data and do not specify the batch size? I don't see any "stochastic" or "randomness" in the case.
For example, in the following simple code, I feed the whole data (x,y) into a model.
optimizer = torch.optim.SGD(model.parameters(), lr=0.1)
for epoch in range(5):
    y_pred = model(x_data)
    loss = criterion(y_pred, y_data)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

Suppose there are 100 data pairs (x,y), i.e. x_data and y_data each has 100 elements.
Question:  It seems to me that all the 100 gradients are calculated before one update of parameters. Size of a "mini_batch" is 100, not 1. So there is no randomness, am I right? At first, I think SGD means randomly choose 1 data point and calculate its gradient, which will be used as an approximation of the true gradient from all data.


title: what is uninitialized data in pytorch.empty function

body:
i was going through pytorch tutorial and came across pytorch.empty function. it was mentioned that empty can be used for uninitialized data. But, when i printed it, i got a value. what is the difference between this and pytorch.rand which also generates data(i know that rand generates between 0 and 1). Below is the code i tried

a = torch.empty(3,4)
print(a)

Output:

tensor([[ 8.4135e-38,  0.0000e+00,  6.2579e-41,  5.4592e-39],
        [-5.6345e-08,  2.5353e+30,  5.0447e-44,  1.7020e-41],
        [ 1.4000e-38,  5.7697e-05,  2.5353e+30,  2.1580e-43]])

b = torch.rand(3,4)
print(b)

Output:

tensor([[ 0.1514,  0.8406,  0.2708,  0.3422],
        [ 0.7196,  0.6120,  0.4476,  0.6705],
        [ 0.6989,  0.2086,  0.5100,  0.8285]])

Here is the link to official documentation


title: undefined symbol: THPVariableClaload_textures.cpython-37m-x86_64-linux-gnu.so: undefined symbol: THPVariableClass

body:
Do you know how I could fix this? I am trying to use https://github.com/benjiebob/SMALViewer/issues/3 repo however I get error on neural_renderer port:
$ python smal_viewer.py
Traceback (most recent call last):
  File "smal_viewer.py", line 2, in <module>
    import pyqt_viewer
  File "/home/mona/research/3danimals/SMALViewer/pyqt_viewer.py", line 13, in <module>
    from smal.smal3d_renderer import SMAL3DRenderer
  File "/home/mona/research/3danimals/SMALViewer/smal/smal3d_renderer.py", line 6, in <module>
    import neural_renderer as nr
  File "/home/mona/anaconda3/lib/python3.7/site-packages/neural_renderer/__init__.py", line 3, in <module>
    from .load_obj import load_obj
  File "/home/mona/anaconda3/lib/python3.7/site-packages/neural_renderer/load_obj.py", line 8, in <module>
    import neural_renderer.cuda.load_textures as load_textures_cuda
ImportError: /home/mona/anaconda3/lib/python3.7/site-packages/neural_renderer/cuda/load_textures.cpython-37m-x86_64-linux-gnu.so: undefined symbol: THPVariableClass

Here is some details:
$ python
Python 3.7.6 (default, Jan  8 2020, 19:59:22)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.__version__
'1.6.0'
>>> torch.version.cuda
'10.1'
>>> torch.cuda.is_available()
True

$ gcc --version
gcc (Ubuntu 9.3.0-10ubuntu2) 9.3.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

$ lsb_release -a
LSB Version:    core-11.1.0ubuntu2-noarch:security-11.1.0ubuntu2-noarch
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.1 LTS
Release:    20.04
Codename:   focal

Here is the neural renderer git repo: https://github.com/daniilidis-group/neural_renderer
I installed the neural renderer using pip install neural_renderer_pytorch


title: PyTorch - sparse tensors do not have strides

body:
I am building my first sentiment analysis model for a small dataset of 1000 reviews using TF-IDF approach along with LSTM using the below code. I am preparing the train data by preprocessing it and feeding to the Vectorizer as below

def tfidf_features(X_train, X_val, X_test):
tfidf_vectorizer = TfidfVectorizer(analyzer='word', token_pattern = '(\S+)', min_df = 5, max_df =
0.9, ngram_range=(1,2))
X_train=tfidf_vectorizer.fit_transform(X_train)
X_val=tfidf_vectorizer.transform(X_val)
X_test=tfidf_vectorizer.transform(X_test)

return X_train, X_val, X_test, tfidf_vectorizer.vocabulary_

I am converting my csr_matrix to a pytorch tensor using the below code

def spy_sparse2torch_sparse(data):
samples=data.shape[0]
features=data.shape[1]
values=data.data
coo_data=data.tocoo()
indices=torch.LongTensor([coo_data.row,coo_data.col])
t=torch.sparse.FloatTensor(indices,torch.from_numpy(values).float(),[samples,features])
return t

And I am getting the training sentences tensor as this

   tensor(indices=tensor([[  0,   0,   1,  ..., 599, 599, 599],
                   [ 97, 131,  49,  ..., 109,  65,  49]]),
   values=tensor([0.6759, 0.7370, 0.6076,  ..., 0.3288, 0.3927, 0.3288]),
   size=(600, 145), nnz=1607, layout=torch.sparse_coo)

I am creating a TensorDataSet using the below code wherein I am also converting my label data from bumpy to a torch tensor

train_data = TensorDataset(train_x, torch.from_numpy(train_y))

I have defined my LSTM network and calling it with the following parameters

n_vocab = len(vocabulary)
n_embed = 100
n_hidden = 256
n_output = 1   # 1 ("positive") or 0 ("negative")
n_layers = 2

net = Sentiment_Lstm(n_vocab, n_embed, n_hidden, n_output, n_layers)

I have also defined the loss and optimizer. Now I am training my model using the below code

print_every = 100
step = 0
n_epochs = 4  # validation loss increases from ~ epoch 3 or 4
clip = 5  # for gradient clip to prevent exploding gradient problem in LSTM/RNN

for epoch in range(n_epochs):
h = net.init_hidden(batch_size)

for inputs, labels in train_loader:
    step += 1

    # making requires_grad = False for the latest set of h
    h = tuple([each.data for each in h])

    net.zero_grad()
    output, h = net(inputs)
    loss = criterion(output.squeeze(), labels.float())
    loss.backward()
    nn.utils.clip_grad_norm(net.parameters(), clip)
    optimizer.step()

    if (step % print_every) == 0:
        net.eval()
        valid_losses = []
        v_h = net.init_hidden(batch_size)

        for v_inputs, v_labels in valid_loader:
            v_inputs, v_labels = inputs.to(device), labels.to(device)

            v_h = tuple([each.data for each in v_h])

            v_output, v_h = net(v_inputs)
            v_loss = criterion(v_output.squeeze(), v_labels.float())
            valid_losses.append(v_loss.item())

        print("Epoch: {}/{}".format((epoch+1), n_epochs),
              "Step: {}".format(step),
              "Training Loss: {:.4f}".format(loss.item()),
              "Validation Loss: {:.4f}".format(np.mean(valid_losses)))
        net.train()

However, I am getting a major error on the line output, h = net(inputs) as RuntimeError: sparse tensors do not have strides

The workarounds given on other websites are not understandable. I am expecting an exact code change I need to make in order to fix this issue.


title: How to get input tensor shape of an unknown PyTorch model

body:
I am writing a python script, which converts any deep learning models from popular frameworks (TensorFlow, Keras, PyTorch) to ONNX format. Currently I have used tf2onnx for tensorflow and keras2onnx for keras to ONNX conversion, and those work.
Now PyTorch has integrated ONNX support, so I can save ONNX models from PyTorch directly. But the problem is I will need input tensor shape for that model, in order to save it in ONNX format. As you already might have guessed, I am writing this script to convert unknown deep learning models.
Here is PyTorch's tutorial for ONNX conversion. There it's written:

Limitations¶
The ONNX exporter is a trace-based exporter, which means that it operates by executing your model once, and exporting the operators which were actually run during this run. This means that if your model is dynamic, e.g., changes behavior depending on input data, the export won’t be accurate.

Similarly, a trace is might be valid only for a specific input size (which is one reason why we require explicit inputs on tracing). Most of the operators export size-agnostic versions and should work on different batch sizes or input sizes. We recommend examining the model trace and making sure the traced operators look reasonable.

The code snippet I am using is this:
import torch

def convert_pytorch2onnx(self):
    """pytorch -> onnx"""

    model = torch.load(self._model_file_path)

    # Don't know how to get this INPUT_SHAPE
    dummy_input = torch.randn(INPUT_SHAPE)
    torch.onnx.export(model, dummy_input, self._onnx_file_path)
    return

So how do I know the INPUT_SHAPE of the input tensor of that unknown PyTorch model? Or is there any other way to convert the PyTorch model to ONNX?


title: pytorch KLDivLoss loss is negative

body:
my target is training a span prediction model
which can predict the position in the BERT output sequence
my input's shape is (batch_size, max_sequence_len(512),embedding_size(768))
output's shape will be (batch_size , max_sequence_len , 1) and the third dim is stand for kind a probability , then I will reshape output to (batch_size,max_sequence_len)
my label's shape is (batch_size , max_sequence_len), and in the max_sequence_len(512), only one position will be 1 and the others will be zero
and I've already check this
(batch_size is 2)
start_pos_labels.sum(dim=1)
output >>
tensor([1.0000, 1.0000], device='cuda:0', dtype=torch.float64)

start_pred.sum(dim=1)
tensor([1., 1.], device='cuda:0', dtype=torch.float64, grad_fn=<SumBackward1>)

but when I use nn.KLDivLoss() , output still be negative, I really dont know why
can somebody help me? thanks!
Here is my code
Model Code
class posClassfication_new(nn.Module):
    def __init__(self):
        super(posClassfication_new, self).__init__()
        self.start_task = nn.Sequential(
            nn.Linear(768, 1),
#             nn.ReLU(),
#             nn.Linear(256, 128),
#             nn.ReLU(),
#             nn.Linear(128, 1)
        )
        self.end_task = nn.Sequential(
            nn.Linear(768, 1),
#             nn.ReLU(),
#             nn.Linear(256, 128),
#             nn.ReLU(),
#             nn.Linear(128, 1)
        )
#
    def forward(self, start_x,end_x):
        start_x = start_x.double()
        end_x = end_x.double()

        start_out = self.start_task(start_x)
        end_out = self.end_task(end_x)

        return start_out,end_out

training code
BATCH_SIZE = 8
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
print("device:", device)

class PosTrainDataset(Dataset):
    def __init__(self, x, start_y,end_y):
        self.x = x
        self.start_y = start_y
        self.end_y = end_y

    def __getitem__(self,idx):
        x = self.x[idx]
        start_y = self.start_y[idx]
        end_y = self.end_y[idx]
        return x, start_y, end_y

    def __len__(self):
        return len(self.x)

trainset = PosTrainDataset(pos_train_x , start_pos_labels_train , end_pos_labels_train)
trainloader = DataLoader(trainset, batch_size=BATCH_SIZE)

pos_model = posClassfication_new()
pos_model = pos_model.to(device)
pos_model = pos_model.double()
pos_model.train()

pos_loss = nn.KLDivLoss()
# pos_loss = nn.CrossEntropyLoss()

optimizer = torch.optim.Adam(pos_model.parameters(), lr=1e-5)

EPOCHS = 5
for epoch in range(EPOCHS):
    running_loss = 0.0
    for data in trainloader:
        x, start_pos_labels, end_pos_labels = [t.to(device) for t in data]
        mini_batch = x.size()[0]
        optimizer.zero_grad()

        start_pred , end_pred = pos_model(x,x)

        start_pred = start_pred.reshape((mini_batch,512))
        end_pred = end_pred.reshape((mini_batch,512))

        start_pred = torch.nn.functional.softmax(start_pred,dim=1)
        end_pred = torch.nn.functional.softmax(end_pred,dim=1)

        start_pos_labels = start_pos_labels + 0.0001
        start_pos_labels = torch.nn.functional.softmax(start_pos_labels,dim=1)

        end_pos_labels = end_pos_labels  + 0.0001
        end_pos_labels = torch.nn.functional.softmax(end_pos_labels,dim=1)

#         start_pos_labels = torch.argmax(start_pos_labels,dim=1)
#         end_pos_labels = torch.argmax(end_pos_labels,dim=1)

        start_loss = pos_loss(start_pred,start_pos_labels)
        end_loss = pos_loss(end_pred,end_pos_labels)

        loss = start_loss + end_loss
        loss.backward()
        optimizer.step()
        running_loss += loss.item()

    torch.save(pos_model,'pos_model_single_task.pkl')
    print('[epoch %d] loss: %.3f' %(epoch + 1, running_loss))


I am trying to do text classification using pretrained BERT model. I trained the model on my dataset, and in the phase of testing; I know that BERT can only take to 512 tokens, so I wrote if condition to check the length of the test senetence in my dataframe. If it is longer than 512 I split the sentence into sequences each sequence has 512 token. And then do tokenizer encode. The length of the seqience is 512, however, after doing tokenize encode the length becomes 707 and I get this error.
The size of tensor a (707) must match the size of tensor b (512) at non-singleton dimension 1

Here is the code I used to do the preivous steps:
tokenizer = BertTokenizer.from_pretrained('bert-base-cased', do_lower_case=False)
import math

pred=[]
if (len(test_sentence_in_df.split())>512):

  n=math.ceil(len(test_sentence_in_df.split())/512)
  for i in range(n):
    if (i==(n-1)):
      print(i)
      test_sentence=' '.join(test_sentence_in_df.split()[i*512::])
    else:
      print("i in else",str(i))
      test_sentence=' '.join(test_sentence_in_df.split()[i*512:(i+1)*512])

      #print(len(test_sentence.split()))  ##here's the length is 512
    tokenized_sentence = tokenizer.encode(test_sentence)
    input_ids = torch.tensor([tokenized_sentence]).cuda()
    print(len(tokenized_sentence)) #### here's the length is 707
    with torch.no_grad():
      output = model(input_ids)
      label_indices = np.argmax(output[0].to('cpu').numpy(), axis=2)
    pred.append(label_indices)

print(pred)


title: How is spectral norm of a parameter calculated?

body:
when I do,

import torch, torch.nn as nn
x = nn.Linear(3, 3)
y = torch.nn.utils.spectral_norm(x)

then it gives four different weight matrices,

y.weight_u

tensor([ 0.6534, -0.1644,  0.7390])

y.weight_orig

Parameter containing:
tensor([[ 0.2538,  0.3196,  0.3380],
        [ 0.4946,  0.0519,  0.1022],
        [-0.5549, -0.0401,  0.1654]], requires_grad=True)

y.weight_v

tensor([-0.3650,  0.2870,  0.8857])

y.weight

tensor([[ 0.5556,  0.6997,  0.7399],
        [ 1.0827,  0.1137,  0.2237],
        [-1.2149, -0.0878,  0.3622]], grad_fn=<DivBackward0>)

how are these four matrices calculated?


title: My custom loss function in Pytorch does not train

body:
My custom loss function in Pytorch does not update during training. The loss stays exactly the same. I am trying to write this custom loss function based on the false positive and negative rates. I am giving you a simplified version of the code. Any idea what could be happening? Does the backpropagation turns to 0? Is this not the correct way of defining a custom loss function?

I have already checked that during backpropagation the Gradient always stays TRUE (assert requires_grad). I have also tried to make a class (torch.nn.module) of the function false_pos_neg_rate, but that did not work. The Assert Requires_grad turned out to be negative and I left it out afterwards.
There is no error, the training does continue.

def false_pos_neg_rate(outputs, truths):
    y = truths
    y_predicted = outputs
    cut_off= torch.tensor(0.5, requires_grad=True)
    y_predicted =torch.where(y_predicted <= cut_off, zeros, ones)
    tp, fp, tn, fn = confusion_matrix(y_predicted, y)
    fp_rate = fp / (fp+tn).float()
    fn_rate = fn / (fn+tp).float()
    loss = fn_rate + fp_rate
    return loss

for i, (samples, truths) in enumerate(train_loader):
    samples = Variable(samples)
    truths = Variable(truths)
    outputs = model(samples)
    loss = false_pos_neg_rate_torch(outputs, truths)
    loss.backward()
    optimizer.step()

I expect the loss function to update the model and be smaller every training step. Instead the loss stays exactly the same and nothing happens.

Please help me, what happens? Why does the model not train during training steps?


title: Pytorch doesn't find a CUDA device

body:
I tried using Cuda in Pytorch in my set up but it can't be detected and I am puzzled as to why.

torch.cuda.is_available()

return False. Digging deeper,

torch._C._cuda_getDeviceCount()

returns 0. Using version 1.5, e.g.

$ pip freeze | grep torch
torch==1.5.0

I tried to write a small C program to do the same, e.g.

#include <stdio.h>
#include <cuda_runtime_api.h>

int main() {
   int count = 0;
   cudaGetDeviceCount(&count);
   printf("Device count: %d\n", count);
   return 0;
}

prints 1, so the Cuda runtime can obviously find a device. Also, running nvidia-smi:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 435.21       Driver Version: 435.21       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 106...  Off  | 00000000:02:00.0  On |                  N/A |
|  0%   41C    P8     9W / 200W |    219MiB /  6075MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

So where did my Cuda device disappear in Python?


title: Shape of pytorch model.parameter is inconsistent with how it's defined in the model

body:
I'm attempting to extract the weights and biases from a simple network built in PyTorch. My entire network is composed of nn.Linear layers. When I create a layer by calling nn.Linear(in_dim, out_dim), I expect the parameters that I get from calling model.parameters() for that model to be of shape (in_dim, out_dim) for the weight and (out_dim) for the bias. However, the weights that come out of model.parameters() are instead of shape (out_dim, in_dim).

The intention of my code is to be able to use matrix multiplication to perform a forward pass using only numpy, not any PyTorch. Because of the shape inconsistency, matrix multiplications throw an error. How can I fix this?

Here is my exact code:

class RNN(nn.Module):

    def __init__(self, dim_input, dim_recurrent, dim_output):

        super(RNN, self).__init__()

        self.dim_input = dim_input
        self.dim_recurrent = dim_recurrent
        self.dim_output = dim_output

        self.dense1 = nn.Linear(self.dim_input, self.dim_recurrent)
        self.dense2 = nn.Linear(self.dim_recurrent, self.dim_recurrent, bias = False)
        self.dense3 = nn.Linear(self.dim_input, self.dim_recurrent)
        self.dense4 = nn.Linear(self.dim_recurrent, self.dim_recurrent, bias = False)
        self.dense5 = nn.Linear(self.dim_recurrent, self.dim_output)

#There is a defined forward pass

model = RNN(12, 100, 6)

for i in model.parameters():
    print(i.shape())

The output is:

torch.Size([100, 12])
torch.Size([100])
torch.Size([100, 100])
torch.Size([100, 12])
torch.Size([100])
torch.Size([100, 100])
torch.Size([6, 100])
torch.Size([6])

The output should, if I'm correct, be:

torch.Size([12, 100])
torch.Size([100])
torch.Size([100, 100])
torch.Size([12, 100])
torch.Size([100])
torch.Size([100, 100])
torch.Size([100, 6])
torch.Size([6])

What is my issue?


title: Why we need image.to('CUDA') when we have model.to('CUDA')

body:
I'm taking a course on PyTorch.
And I'm wondering why we need to separately tell to torch.utils.data.DataLoader output on what device it's running on. If the model is already on CUDA why doesn't it automatically change the inputs accordingly? This pattern seems funny to me:

model.to(device)

for ii, (inputs, labels) in enumerate(trainloader):

    # Move input and label tensors to the GPU
    inputs, labels = inputs.to(device), labels.to(device)

Is there a use case where I'd like to have model running on GPU but my inputs to be on CPU mode or vice versa?


title: Derivative of ReLU

body:
I'm learning PyTorch. Here is the first example in official tutorial. I got two questions, as shown in the block below,

a) I understand that derivative of a ReLU function is 0 when x < 0 and 1 when x > 0. Is that right? But the code seems to keep the x > 0 part unchanged and set x < 0 part to 0. Why is that?

b) Why transpose, i.e. x.T.mm(grad_h)? A transpose does't seem needed to me. I'm just confused. Thanks,

# -*- coding: utf-8 -*-

import torch

dtype = torch.float
device = torch.device("cpu")
# device = torch.device("cuda:0") # Uncomment this to run on GPU

# N is batch size; D_in is input dimension;
# H is hidden dimension; D_out is output dimension.
N, D_in, H, D_out = 64, 1000, 100, 10

# Create random input and output data
x = torch.randn(N, D_in, device=device, dtype=dtype)
y = torch.randn(N, D_out, device=device, dtype=dtype)

# Randomly initialize weights
w1 = torch.randn(D_in, H, device=device, dtype=dtype)
w2 = torch.randn(H, D_out, device=device, dtype=dtype)

learning_rate = 1e-6
for t in range(500):
    # Forward pass: compute predicted y
    h = x.mm(w1)
    h_relu = h.clamp(min=0)
    y_pred = h_relu.mm(w2)

    # Compute and print loss
    loss = (y_pred - y).pow(2).sum().item()
    print(t, loss)

    # Backprop to compute gradients of w1 and w2 with respect to loss
    grad_y_pred = 2.0 * (y_pred - y)
    grad_w2 = h_relu.t().mm(grad_y_pred)
    grad_h_relu = grad_y_pred.mm(w2.t())

    grad_h = grad_h_relu.clone()
    grad_h[h < 0] = 0
    grad_w1 = x.t().mm(grad_h)

    # Update weights using gradient descent
    w1 -= learning_rate * grad_w1
    w2 -= learning_rate * grad_w2


title: Input Tensors not being moved to GPU in pytorch

body:
When running my code, I get the error:

Input and parameter tensors are not at the same device, found input tensor at cpu and parameter tensor at cuda:0

even though I'm using .cuda() on my inputs.

Google Colab link

Code:

use_cuda = True
if use_cuda and torch.cuda.is_available():
   model.cuda()

def test():
model.eval()
avgLoss = 0
for dataPoint in range(len(testData)):
    lstmInput = testData[dataPoint][0]
    lstmInput = torch.Tensor(lstmInput)
    lstmInput = lstmInput.view(len(testData[dataPoint][0]), 1, 5)
    label = testData[dataPoint][1]
    label = torch.Tensor(label)
    lstmInput = Variable(lstmInput)
    label = Variable(label)

    if use_cuda and torch.cuda.is_available():
          lstmInput.cuda()
          label.cuda()

    pred_label = model(lstmInput)
    loss = loss_fn(label, pred_label)
    avgLoss += loss.item()
return avgLoss / len(testData)

def train(num_epochs):
model.train()
for epoch in range(num_epochs):
    avgLoss = 0.0
    for datapoint in range(len(trainData)):
        model.hidden = model.init_hidden()
        optimizer.zero_grad()

        lstmInput = trainData[datapoint][0]
        lstmInput = torch.Tensor(lstmInput)
        lstmInput = lstmInput.view(len(trainData[datapoint][0]), 1, 5)
        label = torch.Tensor(trainData[datapoint][1])
        label = label.view(1, 5)
        lstmInput = Variable(lstmInput)
        label = Variable(label)

        if use_cuda and torch.cuda.is_available():
          print("happens")
          lstmInput.cuda()
          label.cuda()

        pred_label = model(lstmInput)
        loss = loss_fn(pred_label, label)
        # print(label, pred_label)
        avgLoss += loss.item()
        loss.backward()
        optimizer.step()
    print("Epoch: ", epoch, "MSELoss: ", avgLoss / len(trainData), "Test Acc: ", test())


title: Can you accelerate torch DL training on anything other than "cuda" like "hip" or "OpenCL"?

body:
I've noticed that torch.device can accept a range of arguments, precisely cpu, cuda, mkldnn, opengl, opencl, ideep, hip, msnpu.
However, when training deep learning models, I've only ever seen cuda or cpu being used. Very often the code looks something like this
if torch.cuda.is_available():
    device = torch.device("cuda")
else:
    device = torch.device("cpu")

I've never seen any of the others being used, and was wondering if they can be used and how. The latest MacBooks with an AMD graphic card I believe should be able to use "hip", but is that true? And will the training speed be similar to that of using one CUDA GPU? If not, what is the point in torch.device accepting so many options if they cannot actually be used?


title: How to make a Truncated normal distribution in pytorch?

body:
I want to create a Truncated normal distribution(that is Gaussian distribution with a range) in PyTorch.
I want to be able to change the mean, std, and range.
Is there a PyTorch method for that?


title: When setting the floating point type in PyTorch, what is the difference between tensor type and dtype, and when should I set one over the other?

body:
I'm using doubles as my model inputs and outputs, so I am trying to set torch to use float64 instead of float32. What exactly is the difference between

torch.set_default_tensor_type(torch.DoubleTensor)

  Sets the default torch.Tensor type to floating point tensor type t. This type will also be used as default floating point type for type inference in torch.tensor().

torch.set_default_dtype(torch.float64)

  Sets the default floating point dtype to d. This type will be used as default floating point type for type inference in torch.tensor().

The documentation reads to me that setting the tensor type also sets the dtype, but I am not sure when I would use one over the other.

I should mention that either statement fixes the error that I was seeing after changing from floats to doubles:

Traceback (most recent call last):
  File "train.py", line 122, in train_model
    output = net(action)
  File "/opt/anaconda3/lib/python3.7/site- packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "models.py", line 25, in forward
    return self.fc2(F.relu(self.fc1(x)))
  File "/opt/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/anaconda3/lib/python3.7/site-packages/torch/nn/modules/linear.py", line 67, in forward
    return F.linear(input, self.weight, self.bias)
  File "/opt/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py", line 1352, in linear
    ret = torch.addmm(torch.jit._unwrap_optional(bias), input, weight.t())
RuntimeError: Expected object of scalar type Float but got scalar type Double for argument #4 'mat1'


title: Why is the input size of the MultiheadAttention in Pytorch Transformer module 1536?

body:
When using the torch.nn.modules.transformer.Transformer module/object, the first layer is the encoder.layers.0.self_attn layer that is a MultiheadAttention layer, i.e.

from torch.nn.modules.transformer import Transformer
bumblebee = Transformer()

bumblee.parameters

[out]:

<bound method Module.parameters of Transformer(
  (encoder): TransformerEncoder(
    (layers): ModuleList(
      (0): TransformerEncoderLayer(
        (self_attn): MultiheadAttention(
          (out_proj): Linear(in_features=512, out_features=512, bias=True)
        )
        (linear1): Linear(in_features=512, out_features=2048, bias=True)
        (dropout): Dropout(p=0.1, inplace=False)
        (linear2): Linear(in_features=2048, out_features=512, bias=True)
        (norm1): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
        (norm2): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
        (dropout1): Dropout(p=0.1, inplace=False)
        (dropout2): Dropout(p=0.1, inplace=False)
      )

And if we print out the size of the layer, we see:

for name in bumblebee.encoder.state_dict():
    print(name, '\t', bumblebee.encoder.state_dict()[name].shape)

[out]:

layers.0.self_attn.in_proj_weight    torch.Size([1536, 512])
layers.0.self_attn.in_proj_bias      torch.Size([1536])
layers.0.self_attn.out_proj.weight   torch.Size([512, 512])
layers.0.self_attn.out_proj.bias     torch.Size([512])
layers.0.linear1.weight      torch.Size([2048, 512])
layers.0.linear1.bias    torch.Size([2048])
layers.0.linear2.weight      torch.Size([512, 2048])
layers.0.linear2.bias    torch.Size([512])
layers.0.norm1.weight    torch.Size([512])
layers.0.norm1.bias      torch.Size([512])
layers.0.norm2.weight    torch.Size([512])
layers.0.norm2.bias      torch.Size([512])

It seems like 1536 is 512 * 3 and somehow the layers.0.self_attn.in_proj_weight parameter might be storing all three QKV tensors in the transformer architecture in one matrix.

From https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/activation.py#L649

class MultiheadAttention(Module):
    def __init__(self, embed_dim, num_heads, dropout=0., bias=True, add_bias_kv=False, add_zero_attn=False, kdim=None, vdim=None):
        super(MultiheadAttention, self).__init__()
        self.embed_dim = embed_dim
        self.kdim = kdim if kdim is not None else embed_dim
        self.vdim = vdim if vdim is not None else embed_dim
        self._qkv_same_embed_dim = self.kdim == embed_dim and self.vdim == embed_dim

        self.num_heads = num_heads
        self.dropout = dropout
        self.head_dim = embed_dim // num_heads
        assert self.head_dim * num_heads == self.embed_dim, "embed_dim must be divisible by num_heads"

        if self._qkv_same_embed_dim is False:
            self.q_proj_weight = Parameter(torch.Tensor(embed_dim, embed_dim))
            self.k_proj_weight = Parameter(torch.Tensor(embed_dim, self.kdim))
            self.v_proj_weight = Parameter(torch.Tensor(embed_dim, self.vdim))
        else:
            self.in_proj_weight = Parameter(torch.empty(3 * embed_dim, embed_dim))

And the note in the docstring of the MultiheadAttention says:

  Note: if kdim and vdim are None, they will be set to embed_dim such that
  query, key, and value have the same number of features.

Is that correct?


title: Why bilinear scaling of images with PIL and pytorch produces different results?

body:
In order to feed an image to the pytorch network I first need to downscale it to some fixed size. At first I've done it using PIL.Image.resize() method, with interpolation mode set to BILINEAR. Then I though it would be more convenient to first convert a batch of images to pytorch tensor and then use torch.nn.functional.interpolate() function to scale the whole tensor at once on a GPU ('bilinear' interpolation mode as well). This lead to a decrease of the model accuracy because now during inference a type of scaling (torch) was different from the one used during training (PIL). After that, I compared two methods of downscaling visually and found out that they produce different results. Pillow downscaling seems more smooth. Do these methods perform different operations under the hood though both being bilinear? If so, I am also curious if there is a way to achieve the same result as Pillow image scaling with torch tensor scaling?

Original image (the well-known Lenna image)

Pillow scaled image:

Torch scaled image:

Mean channel absolute difference map:

Demo code:

import numpy as np
from PIL import Image
import torch
import torch.nn.functional as F
from torchvision import transforms
import matplotlib.pyplot as plt

pil_to_torch = transforms.ToTensor()
res_shape = (128, 128)

pil_img = Image.open('Lenna.png')
torch_img = pil_to_torch(pil_img)

pil_image_scaled = pil_img.resize(res_shape, Image.BILINEAR)
torch_img_scaled = F.interpolate(torch_img.unsqueeze(0), res_shape, mode='bilinear').squeeze(0)

pil_image_scaled_on_torch = pil_to_torch(pil_image_scaled)
relative_diff = torch.abs((pil_image_scaled_on_torch - torch_img_scaled) / pil_image_scaled_on_torch).mean().item()
print('relative pixel diff:', relative_diff)

pil_image_scaled_numpy = pil_image_scaled_on_torch.cpu().numpy().transpose([1, 2, 0])
torch_img_scaled_numpy = torch_img_scaled.cpu().numpy().transpose([1, 2, 0])
plt.imsave('pil_scaled.png', pil_image_scaled_numpy)
plt.imsave('torch_scaled.png', torch_img_scaled_numpy)
plt.imsave('mean_diff.png', np.abs(pil_image_scaled_numpy - torch_img_scaled_numpy).mean(-1))

Python 3.6.6, requirements:

cycler==0.10.0
kiwisolver==1.1.0
matplotlib==3.2.1
numpy==1.18.2
Pillow==7.0.0
pyparsing==2.4.6
python-dateutil==2.8.1
six==1.14.0
torch==1.4.0
torchvision==0.5.0


title: Pytorch / device problem(cpu, gpu) when load state dict for optimizer

body:
Hi i`m student who studies pytorch since last summer.

state = torch.load('drive/My Drive/MODEL/4 CBAM classifier55')

model = MyResNet()
model.load_state_dict(state['state_dict'])

criterion = nn.CrossEntropyLoss()

optimizer = optim.Adam(model.parameters(), lr=0.0003,betas=(0.5,0.999))
optimizer.load_state_dict(state['optimizer'])

model.to(device)

i wrote code like above.

RuntimeError                              Traceback (most recent call last)
<ipython-input-26-507493db387a> in <module>()
     56     new_loss.backward()
     57
---> 58     optimizer.step()
     59
     60     running_loss += loss.item()

/usr/local/lib/python3.6/dist-packages/torch/autograd/grad_mode.py in decorate_context(*args, **kwargs)
     13         def decorate_context(*args, **kwargs):
     14             with self:
---> 15                 return func(*args, **kwargs)
     16         return decorate_context
     17

/usr/local/lib/python3.6/dist-packages/torch/optim/adam.py in step(self, closure)
     97
     98                 # Decay the first and second moment running average coefficient
---> 99                 exp_avg.mul_(beta1).add_(grad, alpha=1 - beta1)
    100                 exp_avg_sq.mul_(beta2).addcmul_(grad, grad, value=1 - beta2)
    101                 if amsgrad:

RuntimeError: expected device cpu but got device cuda:0

And when i implement training code, then i got this kind of error. When i comment out 'optimizer.load_state_dict', it works well. How can i solve this problem? Thank you for your answer. :)


title: CUDNN Error in backprop for big batches

body:
I implemented a combination of MLP, RNN, CNN. With a batch size of 420, everything seems to work fine (aka I dont get any errors). However as soon as I increase the batch to 840, I receive the following error:

Traceback (most recent call last):
  File "train_cnn_rnn.py", line 152, in <module>
    loss.backward()
  File "/home/tbaumgae/.local/lib/python3.5/site-packages/torch/autograd/variable.py", line 146, in backward
    self._execution_engine.run_backward((self,), (gradient,), retain_variables)
RuntimeError: CUDNN_STATUS_NOT_SUPPORTED. This error may appear if you passed in a non-contiguous input.

The forward pass seems to work fine. I check all the variables whether they are contiguous and they are. Also my prediction and target for the loss calculation are contiguous and also the returned loss. But then this error occurs when calling backward(). Any ideas why this would occur?

CUDA Version 8.0.61

Python 3.5.2

Comment Summary:

There are 210 images in one sequence, therefore, my batch size is in steps of 210. Each image as a shape of [3, 250, 250].
I'm using the PyTorch backward, haven't implemented any backward method myself.


title: Explanation for slicing in Pytorch

body:
why is the output same every time?

a = torch.tensor([0, 1, 2, 3, 4])
a[-2:] = torch.tensor([[[5, 6]]])
a

tensor([0, 1, 2, 5, 6])

a = torch.tensor([0, 1, 2, 3, 4])
a[-2:] = torch.tensor([[5, 6]])
a

tensor([0, 1, 2, 5, 6])

a = torch.tensor([0, 1, 2, 3, 4])
a[-2:] = torch.tensor([5, 6])
a

tensor([0, 1, 2, 5, 6])


title: Dot Product of Tensors with Different Shapes in PyTorch

body:
p=torch.randn(2,3)
q=torch.randn(3,4,5)

I want to perform dot product, to obtain a result of shape (2,4,5).

How can this be done with PyTorch?


title: Is there a version of sparse categorical cross entropy in pytorch?

body: I saw a sudoku solver CNN uses a sparse categorical cross-entropy as a loss function using the TensorFlow framework, I am wondering if there is a similar function for Pytorch? if not could how could I potentially calculate the loss of a 2d array using Pytorch?


title: Sequence Labelling with BERT

body:
I am using a model consisting of an embedding layer and an LSTM to perform sequence labelling, in pytorch + torchtext. I have already tokenised the sentences.

If I use self-trained or other pre-trained word embedding vectors, this is straightforward.

But if I use the Huggingface transformers BertTokenizer.from_pretrained and BertModel.from_pretrained there is a '[CLS]' and '[SEP]' token added to the beginning and end of the sentence, respectively. So the output of the model becomes a sequence that is two elements longer than the label/target sequence.

What I am unsure of is:

Are these two tags needed for the BertModel to embed each token of a sentence "correctly"?
If they are needed, can I take them out after the BERT embedding layer, before the input to the LSTM, so that the lengths are correct in the output?


title: How does max_length, padding and truncation arguments work in HuggingFace' BertTokenizerFast.from_pretrained('bert-base-uncased') work??

body:
I am working with Text Classification problem where I want to use the BERT model as the base followed by Dense layers. I want to know how does the 3 arguments work? For example, if I have 3 sentences as:
'My name is slim shade and I am an aspiring AI Engineer',
'I am an aspiring AI Engineer',
'My name is Slim'

SO what will these 3 arguments do? What I think is as follows:

max_length=5 will keep all the sentences as of length 5 strictly
padding=max_length will add a padding of 1 to the third sentence
truncate=True will truncate the first and second sentence so that their length will be strictly 5.

Please correct me if I am wrong.
Below is my code which I have used.
! pip install transformers==3.5.1

from transformers import BertTokenizerFast

tokenizer = BertTokenizerFast.from_pretrained('bert-base-uncased')

tokens = tokenizer.batch_encode_plus(text,max_length=5,padding='max_length', truncation=True)

text_seq = torch.tensor(tokens['input_ids'])
text_mask = torch.tensor(tokens['attention_mask'])


title: In language modeling, why do I have to init_hidden weights before every new epoch of training? (pytorch)

body:
I have a question about the following code in pytorch language modeling:

print("Training and generating...")
    for epoch in range(1, config.num_epochs + 1):
        total_loss = 0.0
        model.train()
        hidden = model.init_hidden(config.batch_size)

        for ibatch, i in enumerate(range(0, train_len - 1, seq_len)):
            data, targets = get_batch(train_data, i, seq_len)
            hidden = repackage_hidden(hidden)
            model.zero_grad()

            output, hidden = model(data, hidden)
            loss = criterion(output.view(-1, config.vocab_size), targets)
            loss.backward()

Please check line 5.

And the init_hidden function is as follows:

def init_hidden(self, bsz):
    weight = next(self.parameters()).data
    if self.rnn_type == 'LSTM':  # lstm：(h0, c0)
        return (Variable(weight.new(self.n_layers, bsz, self.hi_dim).zero_()),
                Variable(weight.new(self.n_layers, bsz, self.hi_dim).zero_()))
    else:  # gru & rnn：h0
        return Variable(weight.new(self.n_layers, bsz, self.hi_dim).zero_())

My question is:

Why do we need to init_hidden every epoch? Shouldn't it be that the model inherit the hidden parameters from last epoch and continue training on them.


title: Where can I see the source code for pytorch's MSELoss?

body:
I use the U-NET network to train my data.
But I need to modify its loss function to reduce the loss of pixels below 1 to reduce the impact of negative cases on network weights. But I opened the source code in pycharm MSELOSS, see this:

class MSELoss(_Loss):
    r"""Creates a criterion that measures the mean squared error between
    `n` elements in the input `x` and target `y`:

    :math:`{loss}(x, y)  = 1/n \sum |x_i - y_i|^2`

    `x` and `y` arbitrary shapes with a total of `n` elements each.

    The sum operation still operates over all the elements, and divides by `n`.

    The division by `n` can be avoided if one sets the internal variable
    `size_average` to `False`.

    """
    pass

I can't get anything useful.


title: Masking and Instance Normalization in PyTorch

body:
Assume I have a PyTorch tensor, arranged as shape [N, C, L] where N is the batch size, C is the number of channels or features, and L is the length.  In this case, if one wishes to perform instance normalization, one does something like:

N = 20
C = 100
L = 40
m = nn.InstanceNorm1d(C, affine=True)
input = torch.randn(N, C, L)
output = m(input)

This will perform a normalization in the L-wise dimension for each N*C = 2000 slices of data, subtracting 2000 means, scaling by 2000 standard deviations, and re-scaling by 100 learnable weight and bias parameters (one per channel).  The unspoken assumption here is that all of these values exist and are meaningful.

But I have a situation where, for the slice N=1, I would like to exclude all data after (say) L=35.  For the slice N=2 (say) all the data are valid.  For the slice N=3, exclude all data after L=30, etc.  This mimics data which are one dimensional time sequences, having multiple features, but which are not the same length.

How can I perform an instance norm on such data, get correct statistics, and maintain differentiability/AutoGrad information in PyTorch?

Update:  While maintaining GPU performance, or at least not killing it dead.

I cannot...

...Mask with zero values, as this destroys the computer means and variances giving erroneous results
...Mask with np.nan or np.inf, as PyTorch tensors do not ignore such values, but treat them as errors.  They are sticky, and lead to garbage results.  PyTorch currently lacks the equivalent of np.nanmean and np.nanvar.
...Permute or transpose to an amenable arrangement of data; no such approach gives me what I need
...Use a pack_padded_sequence; instance normalization does not operate on that data structure, and one cannot import data into that structure as far as I know.  Also, data re-arrangement would still be necessary, see 3 above.

Am I missing an approach which would give me what I need?  Or perhaps am I missing a method of data re-arrangement which would allow 3 or 4 above to work?

This is an issue faced by recurrent neural networks all the time, hence the pack_padded_sequence functionality, but it isn't quite applicable here.


title: Use of torchvision.utils.save_image twice on the same image tensor makes the second save not work. What's going on?

body:
(Fast Gradient Sign Attack method detailed here: https://pytorch.org/tutorials/beginner/fgsm_tutorial.html)

I have a trained classifier with >90% accuracy which I am using to create these adversarial examples, then I am using torchvision.utils.save_image to save the images to different folders.

The folder hierarchy is as follows:

FOLDER_1

original_image.jpg (1)
perturbed_image.jpg (2)

FOLDER_2

perturbed_image.jpg (3)

Here (2) and (3) are the same image tensor, which is the sum of the original image tensor and a perturbation image tensor---I just want to save images that fooled the classifier twice. What I'm finding is that (1) and (2) print O.K., but (3) only prints the perturbation image tensor (it subtracted the original image tensor!). So when I open up (2) I see my original picture with all the noise on top (random RGB pixel switches from the FGSM attack), but when I open up the (3) I see a blank canvas with those random RGB pixel switches ONLY.

Since I am printing the same variable (perturbed_data) twice, I don't understand why torchvision.utils.save_image is choosing to subtract the perturbation image tensor the second time I call it. The code for what I'm describing is below, and data is the original image tensor.

epsilon = 0.5
# Collect datagrad
data_grad = data.grad.data

# Call FGSM Attack
perturbed_data = fgsm_attack(data, epsilon, data_grad)

# Re-classify the perturbed image
perturbed_output = model(perturbed_data)
perturbed_output = torch.sigmoid(perturbed_output)
perturbed_output = perturbed_output.max(1, keepdim=True)[1]
max_pred  = perturbed_output.item()
final_pred = torch.tensor([0, 0]).to(device)
final_pred[max_pred] = 1

# Store all original and perturbed images, regardless of model prediction
torchvision.utils.save_image(data, "./FOLDER_1/original.jpg")
torchvision.utils.save_image(perturbed_data, "./FOLDER_1/perturbed_image.jpg")

# If the perturbed image fools our classifier, put a copy of it in FOLDER_2
if !torch.all(torch.eq(final_pred, target)):
    torchvision.utils.save_image(perturbed_data, "./FOLDER_2/perturbed_image.jpg")

I'm almost sure that this is a torchvision bug, but I thought I would ask here before submitting a bug report. Maybe someone sees something I don't. I've also attached an example of (2) and (3) for visualization. The first image is in the correct format, but the second one prints without the original image tensor.


title: RuntimeError: Found dtype Long but expected Float: When using criterion

body:
I realized this question has been asked many times but I cannot find any solution which fixes my issue. I found this code online and tried to run it to understand how it actually works. I found the error is RuntimeError: Found dtype Long but expected Float. It happens at errD_real = criterion(output, label). I tried to add .float() to change the type, but it does not work. I use Pytorch 1.7.0, Python 3.8.3, Anaconda 1.9.12, Cudnn 7.6.0. The error and code are below.
runfile('C:/Summer2020/computer_vision/final_project/copy_of_chest_x_ray_shenanigans_(dcgan).py', wdir='C:/Summer2020/computer_vision/final_project')
Generator(
  (main): Sequential(
    (0): ConvTranspose2d(100, 512, kernel_size=(4, 4), stride=(1, 1), bias=False)
    (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (2): ReLU(inplace=True)
    (3): ConvTranspose2d(512, 256, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1), bias=False)
    (4): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (5): ReLU(inplace=True)
    (6): ConvTranspose2d(256, 128, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1), bias=False)
    (7): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (8): ReLU(inplace=True)
    (9): ConvTranspose2d(128, 64, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1), bias=False)
    (10): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (11): ReLU(inplace=True)
    (12): ConvTranspose2d(64, 3, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1), bias=False)
    (13): Tanh()
  )
)
Discriminator(
  (main): Sequential(
    (0): Conv2d(3, 64, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1), bias=False)
    (1): LeakyReLU(negative_slope=0.2, inplace=True)
    (2): Conv2d(64, 128, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1), bias=False)
    (3): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (4): LeakyReLU(negative_slope=0.2, inplace=True)
    (5): Conv2d(128, 256, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1), bias=False)
    (6): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (7): LeakyReLU(negative_slope=0.2, inplace=True)
    (8): Conv2d(256, 512, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1), bias=False)
    (9): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (10): LeakyReLU(negative_slope=0.2, inplace=True)
    (11): Conv2d(512, 1, kernel_size=(4, 4), stride=(1, 1), bias=False)
    (12): Sigmoid()
  )
)
Starting Training Loop...
Type of output is  <class 'torch.Tensor'>
Type of label is  <class 'torch.Tensor'>
Traceback (most recent call last):

  File "C:\Summer2020\computer_vision\final_project\copy_of_chest_x_ray_shenanigans_(dcgan).py", line 337, in <module>
    main()

  File "C:\Summer2020\computer_vision\final_project\copy_of_chest_x_ray_shenanigans_(dcgan).py", line 243, in main
    errD_real = criterion(output, label)

  File "C:\Users\binhd\anaconda3\envs\t2\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)

  File "C:\Users\binhd\anaconda3\envs\t2\lib\site-packages\torch\nn\modules\loss.py", line 530, in forward
    return F.binary_cross_entropy(input, target, weight=self.weight, reduction=self.reduction)

  File "C:\Users\binhd\anaconda3\envs\t2\lib\site-packages\torch\nn\functional.py", line 2525, in binary_cross_entropy
    return torch._C._nn.binary_cross_entropy(

RuntimeError: Found dtype Long but expected Float

from __future__ import print_function
import argparse
import os
import random
import torch
import torch.nn as nn
import torch.nn.parallel
import torch.backends.cudnn as cudnn
import torch.optim as optim
import torch.utils.data
import torchvision.datasets as dset
import torchvision.transforms as transforms
import torchvision.utils as vutils
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.animation as animation
from IPython.display import HTML

def main():
    # Set random seed for reproducibility
    manualSeed = 999
    #manualSeed = random.randint(1, 10000) # use if you want new results
    random.seed(manualSeed)
    torch.manual_seed(manualSeed)

    # Load dataset

    dataroot = 'C:\Spring 2020/Machine Learning and Computer Vision/Project Database/chest xray/chest_xray/chest_xray/train/'

    # Number of workers for dataloader
    workers = 4

    # Batch size during training
    batch_size = 128

    # Spatial size of training images. All images will be resized to this
    #   size using a transformer.
    image_size = 64

    # Number of channels in the training images. For color images this is 3
    nc = 3

    # Size of z latent vector (i.e. size of generator input)
    nz = 100

    # Size of feature maps in generator
    ngf = 64

    # Size of feature maps in discriminator
    ndf = 64

    # Number of training epochs
    num_epochs = 5

    # Learning rate for optimizers
    lr = 0.0002

    # Beta1 hyperparam for Adam optimizers
    beta1 = 0.5

    # Number of GPUs available. Use 0 for CPU mode.
    ngpu = 1

    # Create the dataset
    dataset = dset.ImageFolder(root='C:\Spring 2020/Machine Learning and Computer Vision/Project Database/chest xray/chest_xray/chest_xray/train/',
                               transform=transforms.Compose([
                                   transforms.Resize(image_size),
                                  #  transforms.Grayscale(1),
                                   transforms.CenterCrop(image_size),
                                   transforms.ToTensor(),
                                   transforms.Normalize((0.5), (0.5)),
                               ]))
    # Create the dataloader
    dataloader = torch.utils.data.DataLoader(dataset, batch_size=batch_size,
                                             shuffle=True, num_workers=workers)

    # Decide which device we want to run on
    device = torch.device("cuda:0" if (torch.cuda.is_available() and ngpu > 0) else "cpu")

    # Plot some training images
    real_batch = next(iter(dataloader))
    plt.figure(figsize=(8,8))
    plt.axis("off")
    plt.title("Training Images")
    plt.imshow(np.transpose(vutils.make_grid(real_batch[0].to(device)[:64], padding=2, normalize=True).cpu(),(1,2,0)))

    # Custom weights initialization called on netG and netD
    def weights_init(m):
        classname = m.__class__.__name__
        if classname.find('Conv') != -1:
            nn.init.normal_(m.weight.data, 0.0, 0.02)
        elif classname.find('BatchNorm') != -1:
            nn.init.normal_(m.weight.data, 1.0, 0.02)
            nn.init.constant_(m.bias.data, 0)

    """Generator Code"""

    class Generator(nn.Module):
        def __init__(self, ngpu):
            super(Generator, self).__init__()
            self.ngpu = ngpu
            self.main = nn.Sequential(
                # input is Z, going into a convolution
                nn.ConvTranspose2d( nz, ngf * 8, 4, 1, 0, bias=False),
                nn.BatchNorm2d(ngf * 8),
                nn.ReLU(True),
                # state size. (ngf*8) x 4 x 4
                nn.ConvTranspose2d(ngf * 8, ngf * 4, 4, 2, 1, bias=False),
                nn.BatchNorm2d(ngf * 4),
                nn.ReLU(True),
                # state size. (ngf*4) x 8 x 8
                nn.ConvTranspose2d( ngf * 4, ngf * 2, 4, 2, 1, bias=False),
                nn.BatchNorm2d(ngf * 2),
                nn.ReLU(True),
                # state size. (ngf*2) x 16 x 16
                nn.ConvTranspose2d( ngf * 2, ngf, 4, 2, 1, bias=False),
                nn.BatchNorm2d(ngf),
                nn.ReLU(True),
                # state size. (ngf) x 32 x 32
                nn.ConvTranspose2d( ngf, nc, 4, 2, 1, bias=False),
                nn.Tanh()
                # state size. (nc) x 64 x 64
            )

        def forward(self, input):
            return self.main(input)

    # Create the generator
    netG = Generator(ngpu).to(device)

    # Handle multi-gpu if desired
    if (device.type == 'cuda') and (ngpu > 1):
        netG = nn.DataParallel(netG, list(range(ngpu)))

    # Apply the weights_init function to randomly initialize all weights
    #  to mean=0, stdev=0.2.
    netG.apply(weights_init)

    # Print the model
    print(netG)

    """Discriminator Code

    """

    class Discriminator(nn.Module):
        def __init__(self, ngpu):
            super(Discriminator, self).__init__()
            self.ngpu = ngpu
            self.main = nn.Sequential(
                # input is (nc) x 64 x 64
                nn.Conv2d(nc, ndf, 4, 2, 1, bias=False),
                nn.LeakyReLU(0.2, inplace=True),
                # state size. (ndf) x 32 x 32
                nn.Conv2d(ndf, ndf * 2, 4, 2, 1, bias=False),
                nn.BatchNorm2d(ndf * 2),
                nn.LeakyReLU(0.2, inplace=True),
                # state size. (ndf*2) x 16 x 16
                nn.Conv2d(ndf * 2, ndf * 4, 4, 2, 1, bias=False),
                nn.BatchNorm2d(ndf * 4),
                nn.LeakyReLU(0.2, inplace=True),
                # state size. (ndf*4) x 8 x 8
                nn.Conv2d(ndf * 4, ndf * 8, 4, 2, 1, bias=False),
                nn.BatchNorm2d(ndf * 8),
                nn.LeakyReLU(0.2, inplace=True),
                # state size. (ndf*8) x 4 x 4
                nn.Conv2d(ndf * 8, 1, 4, 1, 0, bias=False),
                nn.Sigmoid()
            )

        def forward(self, input):
            return self.main(input)

    # Create the Discriminator
    netD = Discriminator(ngpu).to(device)

    # Handle multi-gpu if desired
    if (device.type == 'cuda') and (ngpu > 1):
        netD = nn.DataParallel(netD, list(range(ngpu)))

    # Apply the weights_init function to randomly initialize all weights
    #  to mean=0, stdev=0.2.
    netD.apply(weights_init)

    # Print the model
    print(netD)

    """Train the GAN!

    """

    # Initialize BCELoss function
    criterion = nn.BCELoss()

    # Create batch of latent vectors that we will use to visualize
    #  the progression of the generator
    fixed_noise = torch.randn(64, nz, 1, 1, device=device)

    # Establish convention for real and fake labels during training
    real_label = 1
    fake_label = 0

    # Setup Adam optimizers for both G and D
    optimizerD = optim.Adam(netD.parameters(), lr=0.0002, betas=(beta1, 0.999))
    optimizerG = optim.Adam(netG.parameters(), lr=0.001, betas=(beta1, 0.999))

    # Commented out IPython magic to ensure Python compatibility.
    # Training Loop

    # Lists to keep track of progress
    img_list = []
    G_losses = []
    D_losses = []
    iters = 0
    num_epochs = 35

    print("Starting Training Loop...")
    # For each epoch
    for epoch in range(num_epochs):
        # For each batch in the dataloader
        for i, data in enumerate(dataloader, 0):

            ############################
            # (1) Update D network: maximize log(D(x)) + log(1 - D(G(z)))
            ###########################
            ## Train with all-real batch
            netD.zero_grad()
            # Format batch
            real_cpu = data[0].to(device)
            b_size = real_cpu.size(0)
            label = torch.full((b_size,), real_label, device=device) # 64 = batchSize
            # print("label " + str(label.size()))
            # Forward pass real batch through D
            output = netD(real_cpu).view(-1) # 256??
            print('Type of output is ', type(output))
            print('Type of label is ', type(label))
            # print("output " + str(output.size()))
            # Calculate loss on all-real batch
            errD_real = criterion(output, label)
            # Calculate gradients for D in backward pass
            errD_real.backward()
            D_x = output.mean().item()

            ## Train with all-fake batch
            # Generate batch of latent vectors
            noise = torch.randn(b_size, nz, 1, 1, device=device)
            # Generate fake image batch with G
            fake = netG(noise)
            # print("fake " + str(fake.size()))
            label.fill_(fake_label)
            # Classify all fake batch with D
            output = netD(fake.detach()).view(-1)
            # Calculate D's loss on the all-fake batch
            errD_fake = criterion(output, label)
            # Calculate the gradients for this batch
            errD_fake.backward()
            D_G_z1 = output.mean().item()
            # Add the gradients from the all-real and all-fake batches
            errD = errD_real + errD_fake
            # Update D
            optimizerD.step()

            ############################
            # (2) Update G network: maximize log(D(G(z)))
            ###########################
            netG.zero_grad()
            label.fill_(real_label)  # fake labels are real for generator cost
            # Since we just updated D, perform another forward pass of all-fake batch through D
            output = netD(fake).view(-1)
            # Calculate G's loss based on this output
            errG = criterion(output, label)
            # Calculate gradients for G
            errG.backward()
            D_G_z2 = output.mean().item()
            # Update G
            optimizerG.step()

            # Output training stats
            if i % 50 == 0:
                print('[%d/%d][%d/%d]\tLoss_D: %.4f\tLoss_G: %.4f\tD(x): %.4f\tD(G(z)): %.4f / %.4f'
                       % (epoch, num_epochs, i, len(dataloader),
                         errD.item(), errG.item(), D_x, D_G_z1, D_G_z2))

            # Save Losses for plotting later
            G_losses.append(errG.item())
            D_losses.append(errD.item())

            # Check how the generator is doing by saving G's output on fixed_noise
            if (iters % 500 == 0) or ((epoch == num_epochs-1) and (i == len(dataloader)-1)):
                with torch.no_grad():
                    fake = netG(fixed_noise).detach().cpu()
                img_list.append(vutils.make_grid(fake, padding=2, normalize=True))

            iters += 1

    plt.figure(figsize=(10,5))
    plt.title("Generator and Discriminator Loss During Training")
    plt.plot(G_losses,label="G")
    plt.plot(D_losses,label="D")
    plt.xlabel("iterations")
    plt.ylabel("Loss")
    plt.legend()
    plt.show()

    fig = plt.figure(figsize=(8,8))
    plt.axis("off")
    ims = [[plt.imshow(np.transpose(i,(1,2,0)), animated=True)] for i in img_list]
    ani = animation.ArtistAnimation(fig, ims, interval=1000, repeat_delay=1000, blit=True)

    HTML(ani.to_jshtml())

    # Grab a batch of real images from the dataloader
    real_batch = next(iter(dataloader))

    # Plot the real images
    plt.figure(figsize=(80,80))
    plt.subplot(1,2,1)
    plt.axis("off")
    plt.title("Real Images")
    plt.imshow(np.transpose(vutils.make_grid(real_batch[0].to(device)[:64], padding=5, normalize=True).cpu(),(1,2,0)))

    # Plot the fake images from the last epoch
    plt.subplot(1,2,2)
    plt.axis("off")
    plt.title("Fake Images")
    plt.imshow(np.transpose(img_list[-1],(1,2,0)))
    plt.show()

if __name__ == '__main__':
    main()


title: Bert-multilingual in pytorch

body:
I am using bert embedding for french text data. and I have problem with loading model and vocabulary.

I used the following code for tokenization that works well, but to get the vocabulary, it gives me Chinese words!!

tokenizer = BertTokenizer.from_pretrained('bert-base-multilingual-cased')
text = "La Banque Nationale du Canada fête cette année le 110e anniversaire de son bureau de Paris."
marked_text = "[CLS] " + text + " [SEP]"
tokenized_text = tokenizer.tokenize(marked_text)
list(tokenizer.vocab.keys())[5000:5020]

I expected french words in vocabulary but i get chinese words, should i specify the language somewhere in the code?


title: After finetuning Faster RCNN object detection model, how to visualize bbox prediction?

body:
I finetuned pytorch torchvision model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True) on my own custom dataset.

I followed this guide https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html#torchvision-object-detection-finetuning-tutorial but only trained Faster RCNN, not Mask RCNN.

I successfully finished training with no error, and the model returned a dict containing predicted boxes, labels, and scores.

In the guide I followed, they show how to visualize masks predicted by the model trained.
Is there a similar method to visualize bounding box? I'm having a lot of trouble figuring this out.

Thank you


title: Pytorch RuntimeError: mat1 dim 1 must match mat2 dim 0

body:
I tried to run a NeRF Neural Radiance Fields PyTorch program on Google Colab, and I got an error(RuntimeError: mat1 dim 1 must match mat2 dim 0) at line 113 in the last cell(below Run training / validation cell). It seems like the source code is missing two arguments in nerf_forward_pass function, the nerf_forward_pass function has 19 arguments and it inputs only 17, but I still got the same error after adding the two missing arguments.
I contacted the author, he said he enabled the public editing of his Google Colab, so most likely someone mistakenly edited it and made it not runnable. Recently the author is unavailable to come back and fix it. If we want the author to fix, we have to wait for long.
Can anyone provide solutions?
Google Colab: https://colab.research.google.com/drive/1L6QExI2lw5xhJ-MLlIwpbgf7rxW7fcz3
Github: https://github.com/krrish94/nerf-pytorch


title: How to rotate a Torch Tensor by a random number of degrees

body:
as part of training a CNN, I am working with an array inputs that contain <class 'torch.Tensor'> objects. I want to rotate an individual <class 'torch.Tensor'> object by some random number of degrees x, as shown here:
def rotate(inputs, x):
    # Rotate inputs[0] by x degrees, x can take on any value from 0 - 180 degrees

How can I do this? For existing implementations, I can only find that torch has a rot90 function, but that limits me to multiples of 90 degrees which does not help my scenario.
Thanks, Vinny


title: Gradient clipping in pytorch has no effect (Gradient exploding still happens)

body:
I have an exploding gradient problem when train the minibatch for 150-200 epochs with batch size = 256 and there’s about 30-60 minibatch (This depends on my specific config). But I have an exploding gradient issues even if I add the code below.

As you can see this below images, notice that in step about 40k there’s the swing of gradients between ± 20k, 40k and 60k respectively. I don’t know why this happens because i use the clip_grad_value_ above. Also Using the learning rate decay from 0.01 to about 0.008 at step 40k.

Or do I need to update the weight parameters by myself something like this
image
But i think optimizer.step() should do the job and the clip_grad_value_ is an inplace operation so i don’t need to take the return value from function. Please correct if i did anything wrong. Thank you very much

Best regards,
Mint


title: Pytorch Inner Product of 3D tensor with 1D Tensor to generate 2D Tensor

body:
Operation : I have pytorch tensor A of dimension [n x m x c] and B of dimension [1 x 1 x c]. I want to take inner product of each of 1 x 1 x c vector from A with B and hence generate a tensor C of dimension [n x m].

Inside forward function of my network at a specific step I receive tensor of dimension [N, channels, Height, Width] where N is number of images, channels is number of channels in the feature map, and height and width are of the current feature map. I also have an [N x channels] feature map from some other subnetwork. In the next step I want to carry out the above mentioned operation.

Can somebody explain the best way and functions available in pytorch to achieve such step.

I am new to pytorch and was unable to find a proper way. Tensorflow supports NHWC format but I think pytorch doesn't, so one of the way is to reshape it to [N, Height, Width, channels] and then iterate like :

# if img is reshaped to [N, H, W, C]
img
# tensor of dimension [N, C]
aud

ans = torch.empty(N, H, W, dtype=torch.double)

for batches in range(img.shape[0]):
    for i in range(img.shape[1]):
        for j in range(img.shape[2]):
            ans[batches][i][j] = torch.dot(img[batches][i][j], aud[batches])

Any other cleaner API ?

PS : This step is required in DeepMind's paper "Object That Sound" for sound localization step.


title: Fix random seed for torchvision transforms

body:
I use some code similar to the following - for data augmentation:

    from torchvision import transforms

    #...

    augmentation = transforms.Compose([
        transforms.RandomApply([
            transforms.RandomRotation([-30, 30])
        ], p=0.5),
        transforms.RandomHorizontalFlip(p=0.5),
    ])

During my testing I want to fix random values to reproduce the same random parameters each time I change the model training settings. How can I do it?

I want to do something similar to np.random.seed(0) so each time I call random function with probability for the first time, it will run with the same rotation angle and probability. In other words, if I do not change the code at all, it must reproduce the same result when I rerun it.

Alternatively I can separate transforms, use p=1, fix the angle min and max to a particular value and use numpy random numbers to generate results, but my question if I can do it keeping the code above unchanged.


title: RuntimeError: expected stride to be a single integer value

body:
I am new at Pytorch sorry for the basic question. The model gives me dimension mismatch error how to solve this ?
Maybe more than one problems in it.
Any help would be appriciated.
Thanks

class PR(nn.Module):
        def __init__(self):
            super(PR, self).__init__()
            self.conv1     = nn.Conv2d(3,6,kernel_size=5)
            self.conv2     = nn.Conv2d(6,1,kernel_size=2)
            self.dens1     = nn.Linear(300, 256)
            self.dens2     = nn.Linear(256, 256)
            self.dens3     = nn.Linear(512, 24)
            self.drop      = nn.Dropout()

        def forward(self, x):
            out = self.conv1(x)
            out = self.conv2(x)
            out = self.dens1(x)
            out = self.dens2(x)
            out = self.dens3(x)
            return out

model = PR()
input = torch.rand(28,28,3)
output = model(input)


I have a similar problem to this post. However, the opposed solution does not work in my case.

I checked the dtypes of my inputs and outputs and they are all float32

>>>y_train.dtype
torch.float32
>>>y_test.dtype
torch.float32
>>>X_train.dtype
torch.float32
>>>X_test.dtype
torch.float32

However, I receive the error message:

RuntimeError: Expected object of scalar type Float but got scalar type Double for argument #4 'mat1'

Here is my code - The error occurs at grid_NN.fit():

def create_NN_model(neurons):
    hidden_n = neurons
    model_NN = torch.nn.Sequential(
        torch.nn.Linear(X.shape[1], hidden_n),
        torch.nn.Dropout(0.5),
        torch.nn.ReLU(),
        torch.nn.Linear(hidden_n, hidden_n),
        torch.nn.Dropout(0.5),
        torch.nn.ReLU(),
        torch.nn.Linear(hidden_n, 1),
        )
    return model_NN

model_NN = create_NN_model(300)
net = NeuralNetRegressor(model_NN,
                         max_epochs=100,
                         lr=0.001,
                         verbose=1)

steps_NN = [('scaler', StandardScaler()), ('Net', net)]
pipeline_NN = Pipeline(steps_NN)
parameters_NN = {'Net__lr': [0.001,0.005],'Net__max_epochs': list(range(400,500))}

grid_NN = GridSearchCV(pipeline_NN, param_grid= parameters_NN, cv=5, verbose=1, scoring='neg_median_absolute_error')

X_train = torch.from_numpy(np.array(X_train)).float()
X_test = torch.from_numpy(np.array(X_test)).float()

y_train = torch.from_numpy(np.array(y_train).reshape(-1,1)).float()
y_test = torch.from_numpy(np.array(y_test).reshape(-1,1)).float()

grid_NN.fit(X_train, y_train)
print(grid_NN.score(X_test, y_test))
pred = grid_NN.best_estimator_.predict(X_test)

Full Traceback:

Traceback (most recent call last):
  File "C:/Users/Maximilian/maxAnalytics/LisbonHousing/venv/models.py", line 115, in <module>
    grid_NN.fit(X_train, y_train)
  File "C:\Users\Maximilian\Anaconda3\envs\LisbonHousing\lib\site-packages\sklearn\model_selection\_search.py", line 739, in fit
    self.best_estimator_.fit(X, y, **fit_params)
  File "C:\Users\Maximilian\Anaconda3\envs\LisbonHousing\lib\site-packages\sklearn\pipeline.py", line 354, in fit
    self._final_estimator.fit(Xt, y, **fit_params)
  File "C:\Users\Maximilian\Anaconda3\envs\LisbonHousing\lib\site-packages\skorch\regressor.py", line 91, in fit
    return super(NeuralNetRegressor, self).fit(X, y, **fit_params)
  File "C:\Users\Maximilian\Anaconda3\envs\LisbonHousing\lib\site-packages\skorch\net.py", line 848, in fit
    self.partial_fit(X, y, **fit_params)
  File "C:\Users\Maximilian\Anaconda3\envs\LisbonHousing\lib\site-packages\skorch\net.py", line 807, in partial_fit
    self.fit_loop(X, y, **fit_params)
  File "C:\Users\Maximilian\Anaconda3\envs\LisbonHousing\lib\site-packages\skorch\net.py", line 739, in fit_loop
    step = self.train_step(Xi, yi, **fit_params)
  File "C:\Users\Maximilian\Anaconda3\envs\LisbonHousing\lib\site-packages\skorch\net.py", line 664, in train_step
    self.optimizer_.step(step_fn)
  File "C:\Users\Maximilian\Anaconda3\envs\LisbonHousing\lib\site-packages\torch\optim\sgd.py", line 80, in step
    loss = closure()
  File "C:\Users\Maximilian\Anaconda3\envs\LisbonHousing\lib\site-packages\skorch\net.py", line 661, in step_fn
    step = self.train_step_single(Xi, yi, **fit_params)
  File "C:\Users\Maximilian\Anaconda3\envs\LisbonHousing\lib\site-packages\skorch\net.py", line 603, in train_step_single
    y_pred = self.infer(Xi, **fit_params)
  File "C:\Users\Maximilian\Anaconda3\envs\LisbonHousing\lib\site-packages\skorch\net.py", line 996, in infer
    return self.module_(x, **fit_params)
  File "C:\Users\Maximilian\Anaconda3\envs\LisbonHousing\lib\site-packages\torch\nn\modules\module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\Users\Maximilian\Anaconda3\envs\LisbonHousing\lib\site-packages\torch\nn\modules\container.py", line 92, in forward
    input = module(input)
  File "C:\Users\Maximilian\Anaconda3\envs\LisbonHousing\lib\site-packages\torch\nn\modules\module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\Users\Maximilian\Anaconda3\envs\LisbonHousing\lib\site-packages\torch\nn\modules\linear.py", line 92, in forward
    return F.linear(input, self.weight, self.bias)
  File "C:\Users\Maximilian\Anaconda3\envs\LisbonHousing\lib\site-packages\torch\nn\functional.py", line 1406, in linear
    ret = torch.addmm(bias, input, weight.t())
RuntimeError: Expected object of scalar type Float but got scalar type Double for argument #4 'mat1'


title: Parallelization strategies for deep learning

body:
What strategies and forms of parallelization are feasible and available for training and serving a neural network?:

inside a machine across cores (e.g. GPU / TPU / CPU)
across machines on a network or a rack

I'm also looking for evidence for how they may also be used in e.g. TensorFlow, PyTorch or MXNet.
Training
To my knowledge, when training large neural networks on large datasets, one could at least have:

Different cores or machines operate on different parts of the graph ("graph splitting"). E.g. backpropagation through the graph itself can be parallelized e.g. by having different layers hosted on different machines since (I think?) the autodiff graph is always a DAG.
Different cores or machines operate on different samples of data ("data splitting"). In SGD, the computation of gradients across batches or samples can also be parallelized (e.g. the gradients can be combined after computing them independently on different batches). I believe this is also called gradient accumulation (?).

When is each strategy better for what type of problem or neural network? Which modes are supported by modern libraries? and can one combine all four (2x2) strategies?
On top of that, I have read about:

Asynchronous training
Synchronous training

but I don't know what exactly that refers to, e.g. is it the computation of gradients on different data batches or the computation of gradients on different subgraphs? Or perhaps it refers to something else altogether?
Serving
If the network is huge, prediction / inference may also be slow, and the model may not fit on a single machine in memory at serving time. Are there any known multi-core and multi-node prediction solutions that work that can handle such models?


I am new to AI and python, I'm trying to build an architecture to train a set of images. and later to aim to overfit. but up till now, I couldn't understand how to get the inputs and outputs correctly. I keep seeing the error whenever I try to train the network:
mat1 and mat2 shapes cannot be multiplied (48x13456 and 16x64)
my network:
net2 = nn.Sequential(

nn.Conv2d(3,8, kernel_size=5, padding=0),
nn.ReLU(),
nn.MaxPool2d(kernel_size=2, stride=2),

nn.Conv2d(8,16, kernel_size=5, padding=0),
nn.ReLU(),
nn.MaxPool2d(kernel_size=2, stride=2),

nn.Flatten(),
nn.Linear(16,64),
nn.ReLU(),
nn.Linear(64,10)
)

this is a part of a task I'm working on and I really don't get why it's not running. any hints!


title: "TypeError: tensor is not a torch image" appears sometimes when code modified, and disappears after I run the code again

body:
I am learning how to use Pytorch following the official tutorial: https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html

Everything went smoothly until I ran the CIFAR10 classification example.(I didn't modify the code at all)

The error message:

Traceback (most recent call last):
  File "/tmp/pycharm_project_331/main.py", line 90, in <module>
    for i, data in enumerate(trainloader, 0):
  File "/usr/local/lib/python3.5/dist-packages/torch/utils/data/dataloader.py", line 286, in __next__
    return self._process_next_batch(batch)
  File "/usr/local/lib/python3.5/dist-packages/torch/utils/data/dataloader.py", line 307, in _process_next_batch
    raise batch.exc_type(batch.exc_msg)
TypeError: Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/torch/utils/data/dataloader.py", line 57, in _worker_loop
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "/usr/local/lib/python3.5/dist-packages/torch/utils/data/dataloader.py", line 57, in <listcomp>
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "/usr/local/lib/python3.5/dist-packages/torchvision/datasets/cifar.py", line 121, in __getitem__
    img = self.transform(img)
  File "/usr/local/lib/python3.5/dist-packages/torchvision/transforms/transforms.py", line 49, in __call__
    img = t(img)
  File "/usr/local/lib/python3.5/dist-packages/torchvision/transforms/transforms.py", line 143, in __call__
    return F.normalize(tensor, self.mean, self.std)
  File "/usr/local/lib/python3.5/dist-packages/torchvision/transforms/functional.py", line 165, in normalize
    raise TypeError('tensor is not a torch image.')
TypeError: tensor is not a torch image.

I thought maybe I miss some codes, but the code is fine. So I ran the code again for a few more times. The error message disappear somehow, and the training went smoothly.

I can't find any pattern to replicate the error. It just pops up whenever I change the code, and fixed itself without any code modification.

I configured PyCharm to connect to a remote interpreters via SSH(an Ubuntu server), but the same thing occurred when I'm using a local interpreter, and running the code via python console.

This is so frustrating since I have no idea if my code is wrong or it's just the same thing happening.


I was trying to run a python file named api.py. In this file, I'm loading the pickle file of the Deep Learning model that was built and trained using PyTorch.
api.py
In api.py the below-given functions are the most important ones.
def load_model_weights(model_architecture, weights_path):
  if os.path.isfile(weights_path):
      cherrypy.log("CHERRYPYLOG Loading model from: {}".format(weights_path))
      model_architecture.load_state_dict(torch.load(weights_path))
  else:
      raise ValueError("Path not found {}".format(weights_path))

def load_recommender(vector_dim, hidden, activation, dropout, weights_path):

    rencoder_api = model.AutoEncoder(layer_sizes=[vector_dim] + [int(l) for l in hidden.split(',')],
                               nl_type=activation,
                               is_constrained=False,
                               dp_drop_prob=dropout,
                               last_layer_activations=False)
    load_model_weights(rencoder_api, weights_path)
    rencoder_api.eval()
    rencoder_api = rencoder_api.cuda()
    return rencoder_api

The directory structure
MP1
 ┣ .ipynb_checkpoints
 ┃ ┗ RS_netflix3months_100epochs_64,128,128-checkpoint.ipynb
 ┣ data
 ┃ ┣ AutoEncoder.png
 ┃ ┣ collaborative_filtering.gif
 ┃ ┣ movie_titles.txt
 ┃ ┗ shut_up.gif
 ┣ DeepRecommender
 ┃ ┣ data_utils
 ┃ ┃ ┣ movielens_data_convert.py
 ┃ ┃ ┗ netflix_data_convert.py
 ┃ ┣ reco_encoder
 ┃ ┃ ┣ data
 ┃ ┃ ┃ ┣ __pycache__
 ┃ ┃ ┃ ┃ ┣ input_layer.cpython-37.pyc
 ┃ ┃ ┃ ┃ ┣ input_layer_api.cpython-37.pyc
 ┃ ┃ ┃ ┃ ┗ __init__.cpython-37.pyc
 ┃ ┃ ┃ ┣ input_layer.py
 ┃ ┃ ┃ ┣ input_layer_api.py
 ┃ ┃ ┃ ┗ __init__.py
 ┃ ┃ ┣ model
 ┃ ┃ ┃ ┣ __pycache__
 ┃ ┃ ┃ ┃ ┣ model.cpython-37.pyc
 ┃ ┃ ┃ ┃ ┗ __init__.cpython-37.pyc
 ┃ ┃ ┃ ┣ model.py
 ┃ ┃ ┃ ┗ __init__.py
 ┃ ┃ ┣ __pycache__
 ┃ ┃ ┃ ┗ __init__.cpython-37.pyc
 ┃ ┃ ┗ __init__.py
 ┃ ┣ __pycache__
 ┃ ┃ ┗ __init__.cpython-37.pyc
 ┃ ┣ compute_RMSE.py
 ┃ ┣ infer.py
 ┃ ┣ run.py
 ┃ ┗ __init__.py
 ┣ model_save
 ┃ ┣ model.epoch_99
 ┃ ┃ ┗ archive
 ┃ ┃ ┃ ┣ data
 ┃ ┃ ┃ ┃ ┣ 92901648
 ┃ ┃ ┃ ┃ ┣ 92901728
 ┃ ┃ ┃ ┃ ┣ 92901808
 ┃ ┃ ┃ ┃ ┣ 92901888
 ┃ ┃ ┃ ┃ ┣ 92901968
 ┃ ┃ ┃ ┃ ┣ 92902048
 ┃ ┃ ┃ ┃ ┣ 92902128
 ┃ ┃ ┃ ┃ ┣ 92902208
 ┃ ┃ ┃ ┃ ┣ 92902288
 ┃ ┃ ┃ ┃ ┣ 92902368
 ┃ ┃ ┃ ┃ ┣ 92902448
 ┃ ┃ ┃ ┃ ┗ 92902608
 ┃ ┃ ┃ ┣ data.pkl
 ┃ ┃ ┃ ┗ version
 ┃ ┣ model.epoch_99.zip
 ┃ ┗ model.onnx
 ┣ Netflix
 ┃ ┣ N1Y_TEST
 ┃ ┃ ┗ n1y.test.txt
 ┃ ┣ N1Y_TRAIN
 ┃ ┃ ┗ n1y.train.txt
 ┃ ┣ N1Y_VALID
 ┃ ┃ ┗ n1y.valid.txt
 ┃ ┣ N3M_TEST
 ┃ ┃ ┗ n3m.test.txt
 ┃ ┣ N3M_TRAIN
 ┃ ┃ ┗ n3m.train.txt
 ┃ ┣ N3M_VALID
 ┃ ┃ ┗ n3m.valid.txt
 ┃ ┣ N6M_TEST
 ┃ ┃ ┗ n6m.test.txt
 ┃ ┣ N6M_TRAIN
 ┃ ┃ ┗ n6m.train.txt
 ┃ ┣ N6M_VALID
 ┃ ┃ ┗ n6m.valid.txt
 ┃ ┣ NF_TEST
 ┃ ┃ ┗ nf.test.txt
 ┃ ┣ NF_TRAIN
 ┃ ┃ ┗ nf.train.txt
 ┃ ┗ NF_VALID
 ┃ ┃ ┗ nf.valid.txt
 ┣ test
 ┃ ┣ testData_iRec
 ┃ ┃ ┣ .part-00199-f683aa3b-8840-4835-b8bc-a8d1eaa11c78.txt.crc
 ┃ ┃ ┣ part-00000-f683aa3b-8840-4835-b8bc-a8d1eaa11c78.txt
 ┃ ┃ ┣ part-00003-f683aa3b-8840-4835-b8bc-a8d1eaa11c78.txt
 ┃ ┃ ┗ _SUCCESS
 ┃ ┣ testData_uRec
 ┃ ┃ ┣ .part-00000-4a844096-8dd9-425e-9d9d-bd9062cc6940.txt.crc
 ┃ ┃ ┣ ._SUCCESS.crc
 ┃ ┃ ┣ part-00161-4a844096-8dd9-425e-9d9d-bd9062cc6940.txt
 ┃ ┃ ┣ part-00196-4a844096-8dd9-425e-9d9d-bd9062cc6940.txt
 ┃ ┃ ┗ part-00199-4a844096-8dd9-425e-9d9d-bd9062cc6940.txt
 ┃ ┣ data_layer_tests.py
 ┃ ┣ test_model.py
 ┃ ┗ __init__.py
 ┣ __pycache__
 ┃ ┣ api.cpython-37.pyc
 ┃ ┣ load_test.cpython-37.pyc
 ┃ ┣ parameters.cpython-37.pyc
 ┃ ┗ utils.cpython-37.pyc
 ┣ api.py
 ┣ compute_RMSE.py
 ┣ load_test.py
 ┣ logger.py
 ┣ netflix_1y_test.csv
 ┣ netflix_1y_train.csv
 ┣ netflix_1y_valid.csv
 ┣ netflix_3m_test.csv
 ┣ netflix_3m_train.csv
 ┣ netflix_3m_valid.csv
 ┣ netflix_6m_test.csv
 ┣ netflix_6m_train.csv
 ┣ netflix_6m_valid.csv
 ┣ netflix_full_test.csv
 ┣ netflix_full_train.csv
 ┣ netflix_full_valid.csv
 ┣ parameters.py
 ┣ preds.txt
 ┣ RS_netflix3months_100epochs_64,128,128.ipynb
 ┗ utils.py

I am getting such an error (serialization.py). Can someone help me with this error?
D:\Anaconda\envs\practise\lib\site-packages\torch\serialization.py in _legacy_load(f, map_location, pickle_module, **pickle_load_args)
    762             "functionality.")
    763
--> 764     magic_number = pickle_module.load(f, **pickle_load_args)
    765     if magic_number != MAGIC_NUMBER:
    766         raise RuntimeError("Invalid magic number; corrupt file?")

UnpicklingError: A load persistent id instruction was encountered,
but no persistent_load function was specified.


title: Custom Loss in Pytorch where object does not have attribute backward()

body:
I am new to pytorch and I tried creating my own custom loss. This has been really challenging. Below is what I have for my loss.

class CustomLoss(nn.Module):

    def __init__(self,  size_average=True, reduce=True):
        """

        Args:
            size_average (bool, optional): By default, the losses are averaged
               over observations for each minibatch. However, if the field
               size_average is set to ``False``, the losses are instead summed for
               each minibatch. Only applies when reduce is ``True``. Default: ``True``
            reduce (bool, optional): By default, the losses are averaged
               over observations for each minibatch, or summed, depending on
               size_average. When reduce is ``False``, returns a loss per input/target
               element instead and ignores size_average. Default: ``True``
        """
        super(CustomLoss, self).__init__()

    def forward(self, S, N, M, type='softmax',):

        return self.loss_cal(S, N, M, type)

    ### new loss cal
    def loss_cal(self, S, N, M, type="softmax",):
        """ calculate loss with similarity matrix(S) eq.(6) (7)
        :type: "softmax" or "contrast"
        :return: loss
        """

        self.A = torch.cat([S[i * M:(i + 1) * M, i:(i + 1)]
                               for i in range(N)], dim=0)
        self.A = torch.autograd.Variable(self.A)

        if type == "softmax":
            self.B = torch.log(torch.sum(torch.exp(S.float()), dim=1, keepdim=True) + 1e-8)
            self.B = torch.autograd.Variable(self.B)
            total = torch.abs(torch.sum(self.A - self.B))
        else:
            raise AssertionError("loss type should be softmax or contrast !")
        return total

When I run the following:

loss = CustomLoss()
(loss.loss_cal(S=S,N=N,M=M))
loss.backward()

I get the following error:

C:\Program Files\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py in run_cell_magic(self, magic_name, line, cell)
   2113             magic_arg_s = self.var_expand(line, stack_depth)
   2114             with self.builtin_trap:
-> 2115                 result = fn(magic_arg_s, cell)
   2116             return result
   2117

<decorator-gen-60> in time(self, line, cell, local_ns)

C:\Program Files\Anaconda3\lib\site-packages\IPython\core\magic.py in <lambda>(f, *a, **k)
    186     # but it's overkill for just that one bit of state.
    187     def magic_deco(arg):
--> 188         call = lambda f, *a, **k: f(*a, **k)
    189
    190         if callable(arg):

C:\Program Files\Anaconda3\lib\site-packages\IPython\core\magics\execution.py in time(self, line, cell, local_ns)
   1178         else:
   1179             st = clock2()
-> 1180             exec(code, glob, local_ns)
   1181             end = clock2()
   1182             out = None

<timed exec> in <module>()

C:\Program Files\Anaconda3\lib\site-packages\torch\nn\modules\module.py in __getattr__(self, name)
    530                 return modules[name]
    531         raise AttributeError("'{}' object has no attribute '{}'".format(
--> 532             type(self).__name__, name))
    533
    534     def __setattr__(self, name, value):

AttributeError: 'CustomLoss' object has no attribute 'backward'

Why am I getting this error? I did not face this error with TF. My understanding is, it has to do with the autograd? If someone can explain why I am facing this error, I can figure the rest out.


title: Why does this error pop up while working with Deep Q learning?

body:
I have been working with Deep Q Learning on Windows 10 Machine. I have version 0.4.1 of pytorch with NVIDA graphics card.

def select_action(self, state):
    probs = F.softmax(self.model(Variable(state, volatile = True))*7)
    action = probs.multinomial()
    return action.data[0,0]

From this section of the code, I keep getting this error:

TypeError: multinomial() missing 1 required positional arguments: "num_samples"

If any other information is needed, It will be very quickly provided.


title: How to implement time-distributed dense (TDD) layer in PyTorch

body:
In some deep learning models which analyse temporal data (e.g. audio, or video), we use a "time-distributed dense" (TDD) layer. What this creates is a fully-connected (dense) layer which is applied separately to every time-step.

In Keras this can be done using the TimeDistributed wrapper, which is actually slightly more general. In PyTorch it's been an open feature request for a couple of years.

How can we implement time-distributed dense manually in PyTorch?


title: Pytorch could not find module

body:
I have installed pytorch with command:

conda install pytorch torchvision cudatoolkit=10.2 -c pytorch -y

Python complains regarding line import torch with message:

Could not find module 'C:\ProgramData\Anaconda3\envs\edx\lib\site-packages\torch\lib\caffe2_nvrtc.dll' (or one of its dependencies). Try using the full path with constructor syntax

This directory contains library caffe2_nvrtc.dll. What might be wrong and how to fix this error?


title: how to modify rnn cells in pytorch?

body:
If I want to change the compute rules in a RNN cell (e.g. GRU cell), what should I do?
I do not want to implement it via for or while loop considering the issue of efficiency.
I have viewed the source code of pytorch, but it seems that the major components of rnn cells are implement in c code which I cannot find and modify.
You can answer this question through an example: implement GRU cell without the existing version.

thank you ~


title: Why pytorch training on CUDA works much slower than in CPU?

body:
I guess i have made something in folowing simple neural network with PyTorch, because this runs much slower with CUDA then in CPU, can you find the mistake pls. The using function like

    def backward(ctx, input):

        return backward_sigm(ctx, input)

seems have no real impact on preformance

import torch
import torch.nn as nn
import torch.nn.functional as f

dname = 'cuda:0'
dname = 'cpu'

device = torch.device(dname)

print(torch.version.cuda)

def forward_sigm(ctx, input):

    sigm = 1 / (1 + torch.exp(-input))

    ctx.save_for_backward(sigm)

    return sigm

def forward_step(ctx, input):

    return  torch.tensor(input > 0.5, dtype = torch.float32, device = device)

def backward_sigm(ctx, grad_output):

    sigm, = ctx.saved_tensors

    return grad_output * sigm * (1-sigm)

def backward_step(ctx, grad_output):

    return grad_output

class StepAF(torch.autograd.Function):
    @staticmethod
    def forward(ctx, input):
        return forward_sigm(ctx, input)

    @staticmethod
    def backward(ctx, input):

        return backward_sigm(ctx, input)
    #else return grad_output

class StepNN(torch.nn.Module):

    def __init__(self, input_size, hidden_size, output_size):
        super(StepNN, self).__init__()
        self.linear1 = torch.nn.Linear(input_size, hidden_size)
        #self.linear1.cuda()
        self.linear2 = torch.nn.Linear(hidden_size, output_size)
        #self.linear2.cuda()

        #self.StepAF = StepAF.apply

    def forward(self,x):

        h_line_1 = self.linear1(x)

        h_thrash_1 = StepAF.apply(h_line_1)

        h_line_2 = self.linear2(h_thrash_1)

        output = StepAF.apply(h_line_2)

        return output

inputs = torch.tensor( [[1,0,1,0],[1,0,0,1],[0,1,0,1],[0,1,1,0],[1,0,0,0],[0,0,0,1],[1,1,0,1],[0,1,0,0],], dtype = torch.float32, device = device)

expected = torch.tensor( [[1,0,0],[1,0,0],[0,1,0],[0,1,0],[1,0,0],[0,0,1],[0,1,0],[0,0,1],], dtype = torch.float32, device = device)

nn = StepNN(4,8,3)

#print(*(x for x in nn.parameters()))

criterion = torch.nn.MSELoss(reduction='sum')
optimizer = torch.optim.SGD(nn.parameters(), lr=1e-3)

steps = 50000

print_steps = steps // 20

good_loss = 1e-5

for t in range(steps):

    output = nn(inputs)
    loss = criterion(output, expected)

    if t % print_steps == 0:
        print('step ',t, ', loss :' , loss.item())

    if loss < good_loss:
        print('step ',t, ', loss :' , loss.item())
        break

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

test = torch.tensor( [[0,1,0,1],[0,1,1,0],[1,0,1,0],[1,1,0,1],], dtype = torch.float32, device=device)

print(nn(test))


title: Withou onnx, how to convert a pytorch model into a tensorflow model manually?

body:
Since ONNX supports limited models, I tried to do this conversion by assigning parameters directly, but the gained tensorflow model failed to show the desired accuracy. Details are described as follows:

The source model is Lenet trained on MNIST dataset.
I firstly extracted each module and its parameters by model.named_parameters() and save them into a dictionary where the key is the module's name and the value is the parameters
Then, I built and initiated a tensorflow model with the same architecture
Finally, I assign each layer's parameters of pytroch model to the tensorflow model

However, the accuracy of gained tensorflow model is only about 20%. Thus, my question is that is it possible to convert the pytorch model by this method?. If yes, what's the possible issue causing the bad result? If no, then please kindly explain the reasons.

PS: assume the assignment procedure is right.