title: Why is the decoder in an autoencoder uses a sigmoid on the last layer?

body:
I am looking at this working variational auto encoder.
The main class
class VAE(nn.Module):
    def __init__(self):
        super(VAE, self).__init__()

        self.fc1 = nn.Linear(784, 400)
        self.fc21 = nn.Linear(400, 20)
        self.fc22 = nn.Linear(400, 20)
        self.fc3 = nn.Linear(20, 400)
        self.fc4 = nn.Linear(400, 784)

    def encode(self, x):
        h1 = F.relu(self.fc1(x))
        return self.fc21(h1), self.fc22(h1)

    def reparametrize(self, mu, logvar):
        std = logvar.mul(0.5).exp_()
        if torch.cuda.is_available():
            eps = torch.cuda.FloatTensor(std.size()).normal_()
        else:
            eps = torch.FloatTensor(std.size()).normal_()
        eps = Variable(eps)
        return eps.mul(std).add_(mu)

    def decode(self, z):
        h3 = F.relu(self.fc3(z))
        return F.sigmoid(self.fc4(h3))

    def forward(self, x):
        mu, logvar = self.encode(x)
        z = self.reparametrize(mu, logvar)
        return self.decode(z), mu, logvar

has
    def decode(self, z):
        h3 = F.relu(self.fc3(z))
        return F.sigmoid(self.fc4(h3))

I can't explain to myself why the last layer should be passed through a sigmoid before returning.
Please explain.

EDIT:
I just checked without the sigmoid. Results are still nice.
Now I am not sure if it is needed or not.


title: Is there a way to use an external loss function in pytorch?

body:
A typical skeleton of pytorch neural network has a forward() method, then we compute loss based on outputs of forward pass, and call backward() on that loss to update the gradients. What if my loss is determined externally (e.g. by running simulation in some RL environment). Can I still leverage this typical structure this way?

This might be a bit dumb as we no longer know exactly how much each element of output influences the loss, but perhaps there is some trickery I'm not aware of. Otherwise I'm not sure how neural nets can be used in combination with other RL algorithms.

Thank you!


title: Why does this semantic segmentation network have no softmax classification layer in Pytorch?

body:
I am trying to use the following CNN architecture for semantic pixel classification. The code I am using is here

However, from my understanding this type of semantic segmentation network typically should have a softmax output layer for producing the classification result.

I could not find softmax used anywhere within the script. Here is the paper I am reading on this segmentation architecture. From Figure 2, I am seeing softmax being used. Hence I would like to find out why this is missing in the script. Any insight is welcome.


title: Implementation of VGG16 on Pytorch giving size mismatch error

body:
Snippet of my code implementation on PyTorch is:

model = models.vgg16(pretrained = False)
classifier = nn.Sequential(
        nn.Linear(25088, 128),
        nn.ReLU(True),
        nn.Dropout(),
        nn.Linear(128, 128),
        nn.ReLU(True),
        nn.Dropout(),
        nn.Linear(128, 20)
)
model.classifier = classifier

I'm feeding images of input size (60x60x3) and batch_size = 30.

When I run the code from Linux (Ubuntu) Terminal (with PyTorch Version:  1.0.0, Torchvision Version:  0.2.1) it gives me, the following error message:

  RuntimeError: size mismatch, m1: [30 x 512], m2: [25088 x 128]

While, when I run it from Spyder (Anaconda) on Windows (with PyTorch Version:  1.0.1, Torchvision Version:  0.2.2), it runs perfectly.

Am I missing something or is this because of some version mismatch in Pytorch and Torchvision? Both, I'm running on Python 3.6.
Please suggest.

[UPDATE: Mistakenly interchanged the version numbers for the error-case and error-free case. Thanks @Manoj Mohan for pointing it out]


title: Modifying manually pytorch state_dict file and retrain

body:
I save and load pytorch state_dict file and then evaluate and retrain the model. That works.

torch.save(net.state_dict(), path)
net.load_state_dict(torch.load(path))

However, when I modify the state_dict file (manually changing the values) after loading it and evaluating it, I receive the error:

RuntimeError: a leaf Variable that requires grad has been used in an in-place operation.

How to safely modify the state_dict and then retrain it without the error?


title: How to use masked select for these kind of tensors?

body:
Suppose, I got a tensor a and a tensor b

import torch
a = torch.tensor([[[ 0.8856,  0.1411, -0.1856, -0.1425],
    [-0.0971,  0.1251,  0.1608, -0.1302],
    [-0.0901,  0.3215,  0.1763, -0.0412]],
    [[ 0.8856,  0.1411, -0.1856, -0.1425],
    [-0.0971,  0.1251,  0.1608, -0.1302],
    [-0.0901,  0.3215,  0.1763, -0.0412]]])
b = torch.tensor([[0,
    2,
    1],
    [0,
    2,
    1]])

Now, I would like to select indices from tensor a, where the value of tensor b is not 0.

pred_masks = ( b != 0 )
c = torch.masked_select( a, (pred_masks == 1))

And of course, I get an expected error.

----> 1 c = torch.masked_select( a, (pred_masks == 1))

RuntimeError: The size of tensor a (4) must match the size of tensor b (3) at non-singleton dimension 2

This is caused by the nested list containing 4 items. However, it is required to select all the values of the nested list at index x in tensor a, corresponding to the index x in tensor b.

I will be grateful for any hint or answer.


title: not able to install easyocr (pytorch error)

body:
I am trying to install easyocr for python using pip
pip install easyocr

but it's not installing
its givig me this in the terminal
ERROR: torchvision 0.5.0 has requirement torch==1.4.0, but you'll have torch 0.1.2.post2 which is incompatible.
Installing collected packages: torch, torchvision, python-bidi, scipy, cycler, kiwisolver, python-dateutil, certifi, pyparsing, matplotlib, PyWavelets, imageio, decorator, networkx, tifffile, scikit-image, easyocr
  Running setup.py install for torch ... error
    ERROR: Command errored out with exit status 1:
     command: 'c:\users\murari\appdata\local\programs\python\python37\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\Murari\\AppData\\Local\\Temp\\pip-install-jl3470hv\\torch\\setup.py'"'"'; __file__='"'"'C:\\Users\\Murari\\AppData\\Local\\Temp\\pip-install-jl3470hv\\torch\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\Murari\AppData\Local\Temp\pip-record-8j183bzh\install-record.txt' --single-version-externally-managed --compile
         cwd: C:\Users\Murari\AppData\Local\Temp\pip-install-jl3470hv\torch\
    Complete output (23 lines):
    running install
    running build_deps
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\Users\Murari\AppData\Local\Temp\pip-install-jl3470hv\torch\setup.py", line 265, in <module>
        description="Tensors and Dynamic neural networks in Python with strong GPU acceleration",
      File "c:\users\murari\appdata\local\programs\python\python37\lib\site-packages\setuptools\__init__.py", line 145, in setup
        return distutils.core.setup(**attrs)
      File "c:\users\murari\appdata\local\programs\python\python37\lib\distutils\core.py", line 148, in setup
        dist.run_commands()
      File "c:\users\murari\appdata\local\programs\python\python37\lib\distutils\dist.py", line 966, in run_commands
        self.run_command(cmd)
      File "c:\users\murari\appdata\local\programs\python\python37\lib\distutils\dist.py", line 985, in run_command
        cmd_obj.run()
      File "C:\Users\Murari\AppData\Local\Temp\pip-install-jl3470hv\torch\setup.py", line 99, in run
        self.run_command('build_deps')
      File "c:\users\murari\appdata\local\programs\python\python37\lib\distutils\cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "c:\users\murari\appdata\local\programs\python\python37\lib\distutils\dist.py", line 985, in run_command
        cmd_obj.run()
      File "C:\Users\Murari\AppData\Local\Temp\pip-install-jl3470hv\torch\setup.py", line 51, in run
        from tools.nnwrap import generate_wrappers as generate_nn_wrappers
    ModuleNotFoundError: No module named 'tools.nnwrap'
    ----------------------------------------
ERROR: Command errored out with exit status 1: 'c:\users\murari\appdata\local\programs\python\python37\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\Murari\\AppData\\Local\\Temp\\pip-install-jl3470hv\\torch\\setup.py'"'"'; __file__='"'"'C:\\Users\\Murari\\AppData\\Local\\Temp\\pip-install-jl3470hv\\torch\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\Murari\AppData\Local\Temp\pip-record-8j183bzh\install-record.txt' --single-version-externally-managed --compile Check the logs for full command output.
WARNING: You are using pip version 19.2.3, however version 20.2.4 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.


title: I don't understand the code for training a classifier in pytorch

body:
I don't understand the line labels.size(0). I'm new to Pytorch and been quite confused about the data structure.

correct = 0
total = 0
with torch.no_grad():
   for data in testloader:
      images, labels = data
      outputs = net(images)
      _, predicted = torch.max(outputs.data, 1)
      total += labels.size(0)
      correct += (predicted == labels).sum().item()

print('Accuracy of the network on the 10000 test images: %d %%' % (
    100 * correct / total))`


title: Text classification with torchnlp

body:
i'm trying to build a neural network using pytorch-nlp (https://pytorchnlp.readthedocs.io/en/latest/).
My intent is to build a network like this:

Embedding layer (uses pytorch standard layer and from_pretrained method)
Encoder with LSTM (also uses standard nn.LSTM)
Attention mechanism (uses torchnlp.nn.Attention)
Decoder siwth LSTM (as encoder)
Linear layer standard

I'm encountering a major problem with the dimensions of the input sentences (each word is a vector) but most importantly with the attention layer : I don't know how to declare it because i need the exact dimensions of the output from the encoder, but the sequences have varying dimensions (corresponding to the fact that sentences have different number of words).
I've tried to look at torch.nn.utils.rnn.pad_packed_sequence and torch.nn.utils.rnn.pack_padded_sequence since they're supported by LSTM, but i cannot find the solution.
Can anyone help me?
EDIT
I thought about padding all sequences to a specific dimension, but I don't want to truncate longer sequences because I want to keep all the information.


title: Pytorch on Google VM (Linux) does not recognize GPU

body:
I created a Google VM instance using this available image:

c1-deeplearning-common-cu100-20191226

Description

Google, Deep Learning Image: Base, m39 (with CUDA 10.0), A Debian based image with CUDA 10.0

I then installed Anaconda onto this VM, then installed Pytorch using the following command line as recommended by the Pytorch website:

conda install pytorch torchvision cudatoolkit=10.1 -c pytorch

(this corresponds to Linux, Python 3.7, CUDA 10.1)

From Python, I ran this code to check the GPU detection:

import torch
torch.cuda.is_available()
False

From the nvidia-smi tool, this is the result even after the main body of code is running the training:

(base) redexces.bf@tensorflow-1x-2x:~$ nvidia-smi
Thu Jan  2 01:33:10 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.104      Driver Version: 410.104      CUDA Version: 10.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla P4            Off  | 00000000:00:04.0 Off |                    0 |
| N/A   37C    P0    22W /  75W |      0MiB /  7611MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

Clearly, there are no running processes nor any memory allocated.

This problem appears to be related to Pytorch only; the same VM also has Tensorflow-gpu installed in a separate conda environment which recognizes the GPU and utilizes it as I would expect.

Am I missing any pieces?  Again the same CUDA driver and image are working fine for tensorflow.


title: How to get columns from 2D tensor list in Pytorch

body:
I have a one 2D list that contains tensors like:
[
 [tensor([-0.0705,  1.2019]), tensor([[0]]), tensor([-1.3865], dtype=torch.float64), tensor([-0.0744,  1.1880]), tensor([False])],
 [tensor([-0.0187,  1.3574]), tensor([[2]]), tensor([0.3373], dtype=torch.float64), tensor([-0.0221,  1.3473]), tensor([False])],
 [....] ]

The outer list contains 64 little list. One little list contains 5 different tensor elements.
And I want to get first elements of inner lists like tensor([-0.0705,  1.2019]) and tensor([-0.0187,  1.3574]) and create tensor like 64x2 to feed my neural  net.
How can I do this in the fastest way?
Thanks


title: How to add LSTM layer on top of Huggingface BERT model

body:
I am working on a binary classification task and would like to try adding lstm layer on top of the last hidden layer of huggingface BERT model, however, I couldn't reach the last hidden layer. Is it possible to combine BERT with LSTM?
tokenizer = BertTokenizer.from_pretrained(model_path)
tain_inputs, train_labels, train_masks = data_prepare_BERT(
    train_file, lab2ind, tokenizer, content_col, label_col,
    max_seq_length)
validation_inputs, validation_labels, validation_masks = data_prepare_BERT(
    dev_file, lab2ind, tokenizer, content_col, label_col,max_seq_length)

# Load BertForSequenceClassification, the pretrained BERT model with a single linear classification layer on top.
model = BertForSequenceClassification.from_pretrained(
    model_path, num_labels=len(lab2ind))


title: Neural networks pytorch

body:
I am very new in pytorch and implementing my own network of image classifier. However I see for each epoch training accuracy is very good but validation accuracy is 0.i noted till 5th epoch. I am using Adam optimizer and have learning rate .001. also resampling the whole data set after each epoch into training n validation set. Please help where I am going wrong.

Here is my code:

### where is data?
data_dir_train = '/home/sup/PycharmProjects/deep_learning/CNN_Data/training_set'
data_dir_test = '/home/sup/PycharmProjects/deep_learning/CNN_Data/test_set'

# Define your batch_size
batch_size = 64

allData = datasets.ImageFolder(root=data_dir_train,transform=transformArr)

# We need to further split our training dataset into training and validation sets.
def split_train_validation():
    # Define the indices
    num_train = len(allData)
    indices = list(range(num_train)) # start with all the indices in training set
    split = int(np.floor(0.2 * num_train)) # define the split size
    #train_idx, valid_idx = indices[split:], indices[:split]

    # Random, non-contiguous split
    validation_idx = np.random.choice(indices, size=split, replace=False)
    train_idx = list(set(indices) - set(validation_idx))
    # define our samplers -- we use a SubsetRandomSampler because it will return
    # a random subset of the split defined by the given indices without replacement
    train_sampler = SubsetRandomSampler(train_idx)
    validation_sampler = SubsetRandomSampler(validation_idx)

    #train_loader = DataLoader(allData,batch_size=batch_size,sampler=train_sampler,shuffle=False,num_workers=4)
    #validation_loader = DataLoader(dataset=allData,batch_size=1, sampler=validation_sampler)

    return (train_sampler,validation_sampler)

Training

from torch.optim import Adam
import torch
import createNN
import torch.nn as nn
import loadData as ld
from torch.autograd import  Variable
from torch.utils.data import DataLoader

# check if cuda - GPU support available
cuda = torch.cuda.is_available()

#create model, optimizer and loss function
model = createNN.ConvNet(class_num=2)
optimizer = Adam(model.parameters(),lr=.001,weight_decay=.0001)
loss_func = nn.CrossEntropyLoss()

if cuda:
    model.cuda()

# function to save model
def save_model(epoch):
    torch.save(model.load_state_dict(),'imageClassifier_{}.model'.format(epoch))
    print('saved model at epoch',epoch)

def exp_lr_scheduler (  epoch , init_lr = args.lr, weight_decay = args.weight_decay, lr_decay_epoch = cf.lr_decay_epoch):
 lr = init_lr * ( 0.5 ** (epoch // lr_decay_epoch))

def train(num_epochs):
    best_acc = 0.0

    for epoch in range(num_epochs):
        print('\n\nEpoch {}'.format(epoch))
        train_sampler, validation_sampler = ld.split_train_validation()
        train_loader = DataLoader(ld.allData, batch_size=30, sampler=train_sampler, shuffle=False)
        validation_loader = DataLoader(dataset=ld.allData, batch_size=1, sampler=validation_sampler)
        model.train()
        acc = 0.0
        loss = 0.0
        total = 0
        # train model with training data
        for i,(images,labels) in enumerate(train_loader):
            # if cuda then move to GPU
            if cuda:
                images = images.cuda()
                labels = labels.cuda()
            # Variable class wraps a tensor and we can calculate grad
            images = Variable(images)
            labels = Variable(labels)
            # reset accumulated gradients for each batch
            optimizer.zero_grad()
            # pass images to model which returns preiction
            output = model(images)
            #calculate the loss based on prediction and actual
            loss = loss_func(output,labels)
            # backpropagate the loss and compute gradient
            loss.backward()
            # update weights as per the computed gradients
            optimizer.step()

            # prediction class
            predVal , predClass = torch.max(output.data, 1)
            acc += torch.sum(predClass == labels.data)
            loss += loss.cpu().data[0]
            total += labels.size(0)
        # print the statistics
        train_acc = acc/total
        train_loss = loss / total
        print('Mean train acc = {} over epoch = {}'.format(epoch,acc))
        print('Mean train loss = {} over epoch = {}'.format(epoch, loss))

        # Valid model with validataion data
        model.eval()
        acc = 0.0
        loss = 0.0
        total = 0
        for i,(images,labels) in enumerate(validation_loader):
            # if cuda then move to GPU
            if cuda:
                images = images.cuda()
                labels = labels.cuda()
            # Variable class wraps a tensor and we can calculate grad
            images = Variable(images)
            labels = Variable(labels)
            # reset accumulated gradients for each batch
            optimizer.zero_grad()
            # pass images to model which returns preiction
            output = model(images)
            #calculate the loss based on prediction and actual
            loss = loss_func(output,labels)
            # backpropagate the loss and compute gradient
            loss.backward()
            # update weights as per the computed gradients
            optimizer.step()

            # prediction class
            predVal, predClass = torch.max(output.data, 1)
            acc += torch.sum(predClass == labels.data)
            loss += loss.cpu().data[0]
            total += labels.size(0)
        # print the statistics
        valid_acc = acc / total
        valid_loss = loss / total
        print('Mean train acc = {} over epoch = {}'.format(epoch, valid_acc))
        print('Mean train loss = {} over epoch = {}'.format(epoch, valid_loss))

        if(best_acc<valid_acc):
            best_acc = valid_acc
            save_model(epoch)

        # at 30th epoch we save the model
        if (epoch == 30):
            save_model(epoch)

train(20)


title: How to fix Process finished with exit code -1073741819 (0xC0000005) when using Convolutional layers in pytorch, error on backward()

body:
When I use Conv1d or Conv2d layers on pytorch, the process is killed unexpectedly. I am getting the error in the following line:
loss.backward()

My set up:

Windows 10
cuda 10.2
cudnn 7.6.5
RTX 2060 Super
Nvidia driver 451.67
Pycharm 2020.04

Error:
Process finished with exit code -1073741819 (0xC0000005)

In comparison, when I replace the conv layer with a dense layer, the problem doesn't occur.
For more comparison, the same project and the same code was run on Ubuntu 20.04 as well, and it worked quite well.


title: GCP AI Platform Notebook driver too old?

body:
I am trying to run the following Hugging Face Transformers tutorial on GCP's AI Platform Notebook with 32 vCPUs, 208 GB RAM, and 2 NVIDIA Tesla T4s.
However, when I try to run the part
model = DistillBERTClass()
model.to(device)
I get the following Assertion Error:
AssertionError: The NVIDIA driver on your system is too old (found version 10010).
Please update your GPU driver by downloading and installing a new
version from the URL: http://www.nvidia.com/Download/index.aspx
Alternatively, go to: https://pytorch.org to install
a PyTorch version that has been compiled with your version
of the CUDA driver.

However, when I run
!nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.87.01    Driver Version: 418.87.01    CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |
| N/A   38C    P0    22W /  70W |     10MiB / 15079MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla T4            Off  | 00000000:00:05.0 Off |                    0 |
| N/A   39C    P8    10W /  70W |     10MiB / 15079MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |

The version on the NVIDIA driver is compatible with the latest PyTorch version, which I am using.
Has anyone else ran into this error, and is there a way around it?


title: PyTorch network produces constant output

body:
I am trying to train a simple MLP to approximate y=f(a,b,c).
My code is as below.
import torch
import torch.nn as nn
from torch.autograd import Variable

# hyper parameters
input_size = 3
output_size = 1
num_epochs = 50
learning_rate = 0.001

# Network definition
class FeedForwardNet(nn.Module):
    def __init__(self, l1_size, l2_size):
      super(FeedForwardNet, self).__init__()
      self.fc1 = nn.Linear(input_size, l1_size)
      self.relu1 = nn.ReLU()
      self.fc2 = nn.Linear(l1_size, l2_size)
      self.relu2 = nn.ReLU()
      self.fc3 = nn.Linear(l2_size, output_size)

    def forward(self, x):
      out = self.fc1(x)
      out = self.relu1(out)
      out = self.fc2(out)
      out = self.relu2(out)
      out = self.fc3(out)
      return out

model = FeedForwardNet(5 , 3)

# sgd optimizer
optimizer = torch.optim.SGD(model.parameters(), learning_rate, momentum=0.9)

for epoch in range(11):
    print ('Epoch ', epoch)
    for i in range(trainX_light.shape[0]):
    X = Variable( torch.from_numpy(trainX_light[i]).view(-1, 3) )
    Y = Variable( torch.from_numpy(trainY_light[i]).view(-1, 1) )
    # forward
    optimizer.zero_grad()
    output = model(X)

    loss = (Y - output).pow(2).sum()
    print (output.data[0,0])
    loss.backward()
    optimizer.step()
    totalnorm = 0
    for p in model.parameters():
        modulenorm = p.grad.data.norm()
        totalnorm += modulenorm ** 2
        totalnorm = math.sqrt(totalnorm)

     print (totalnorm)

    # validation code
    if (epoch + 1) % 5 == 0:
    print (' test points',testX_light.shape[0])
    total_loss = 0
    for t in range(testX_light.shape[0]):
        X = Variable( torch.from_numpy(testX_light[t]).view(-1, 3) )
        Y = Variable( torch.from_numpy(testY_light[t]).view(-1, 1) )
        output = model(X)
        loss = (Y - output).pow(2).sum()
        print (output.data[0,0])
        total_loss += loss
    print ('epoch ', epoch, 'avg_loss ', total_loss.data[0] / testX_light.shape[0])

print ('Done')

The problem that I have now is, the validation code

output = model(X)

is always producing an exact same output value (I guess this value is some sort of garbage). I am not sure what mistake I am doing in this part. Could some help me figure out the mistake in my code?


title: define torch.cuda.LongTensor instead of torch.LongTensor

body:
When I run the following code the variable type turns into torch.LongTensor.  How can I have this create a torch.cuda.LongTensor instead?

# Turn string into list of longs
def char_tensor(string):
   tensor = torch.zeros(len(string)).long()
   for c in range(len(string)):
      tensor[c] = all_characters.index(string[c])
   return Variable(tensor)

print(char_tensor('abcDEF'))

output:

Variable containing:
 10
 11
 12
 39
 40
 41
[torch.LongTensor of size 6]


title: Missing Weight Vectors when converting from PyTorch to CoreML via ONNX

body:
I am trying to convert a PyTorch model to CoreML via ONNX, but the ONNX-->CoreML conversion is missing weight vectors?

I am following the tutorial here which makes this statement:

  Step 3: Converting the model to CoreML

  It's as easy as running the convert function. The resulting object is a coremltools MLModel object that you can save to a file and import in XCode later.
  cml = onnx_coreml.convert(model)

Unfortunately when I try to do this it fails horribly.

Here's my code:

# convert.py
import torch
import torch.onnx
from torch.autograd import Variable

import onnx
from onnx_coreml import convert

from hourglass_model import Hourglass

model_no = 1
torch_model = Hourglass(joint_count=14, size=256)
state_dict = torch.load("hourglass_model_{}.model".format(model_no))
torch_model.load_state_dict(state_dict)
torch_model.train(False)
torch_model.eval()

# Dummy Input to the model
x = Variable(torch.randn(1,3,256,256,dtype=torch.float32))

# Export the model
onnx_filename = "test_hourglass.onnx"
torch_out = torch.onnx.export(torch_model, x, onnx_filename, export_params=False)

# Load back in ONNX model
onnx_model = onnx.load(onnx_filename)

# Check that the IR is well formed
onnx.checker.check_model(onnx_model)

# Print a human readable representation of the graph
graph = onnx.helper.printable_graph(onnx_model.graph)
print(graph)

coreml_model = convert(onnx_model,
    add_custom_layers=True,
    image_input_names=["input"],
    image_output_names=["output"])
coreml_model.save('test_hourglass.mlmodel')

Here's what the print(graph) line gives.

graph torch-jit-export (
  %0[FLOAT, 1x3x256x256]
  %1[FLOAT, 64x3x5x5]
  %2[FLOAT, 64]
  %3[FLOAT, 64x64x5x5]
  %4[FLOAT, 64]
  %5[FLOAT, 64x64x5x5]
  %6[FLOAT, 64]
  %7[FLOAT, 64x64x5x5]
  %8[FLOAT, 64]
  %9[FLOAT, 64x64x5x5]
  %10[FLOAT, 64]
  %11[FLOAT, 64x64x5x5]
  %12[FLOAT, 64]
  %13[FLOAT, 64x64x5x5]
  %14[FLOAT, 64]
  %15[FLOAT, 64x64x1x1]
  %16[FLOAT, 64]
  %17[FLOAT, 14x64x1x1]
  %18[FLOAT, 14]
) {
  %19 = Conv[dilations = [1, 1], group = 1, kernel_shape = [5, 5], pads = [2, 2, 2, 2], strides = [1, 1]](%0, %1, %2)
  %20 = Relu(%19)
  %21 = MaxPool[kernel_shape = [4, 4], pads = [0, 0, 0, 0], strides = [4, 4]](%20)
  %22 = Conv[dilations = [1, 1], group = 1, kernel_shape = [5, 5], pads = [2, 2, 2, 2], strides = [1, 1]](%21, %3, %4)
  %23 = Relu(%22)
  %24 = MaxPool[kernel_shape = [4, 4], pads = [0, 0, 0, 0], strides = [4, 4]](%23)
  %25 = Conv[dilations = [1, 1], group = 1, kernel_shape = [5, 5], pads = [2, 2, 2, 2], strides = [1, 1]](%24, %5, %6)
  %26 = Relu(%25)
  %27 = Conv[dilations = [1, 1], group = 1, kernel_shape = [5, 5], pads = [2, 2, 2, 2], strides = [1, 1]](%26, %7, %8)
  %28 = Relu(%27)
  %29 = Conv[dilations = [1, 1], group = 1, kernel_shape = [5, 5], pads = [2, 2, 2, 2], strides = [1, 1]](%28, %9, %10)
  %30 = Relu(%29)
  %31 = Upsample[height_scale = 4, mode = 'nearest', width_scale = 4](%30)
  %32 = Add(%31, %23)
  %33 = Conv[dilations = [1, 1], group = 1, kernel_shape = [5, 5], pads = [2, 2, 2, 2], strides = [1, 1]](%32, %11, %12)
  %34 = Relu(%33)
  %35 = Upsample[height_scale = 4, mode = 'nearest', width_scale = 4](%34)
  %36 = Add(%35, %20)
  %37 = Conv[dilations = [1, 1], group = 1, kernel_shape = [5, 5], pads = [2, 2, 2, 2], strides = [1, 1]](%36, %13, %14)
  %38 = Relu(%37)
  %39 = Conv[dilations = [1, 1], group = 1, kernel_shape = [1, 1], pads = [0, 0, 0, 0], strides = [1, 1]](%38, %15, %16)
  %40 = Relu(%39)
  %41 = Conv[dilations = [1, 1], group = 1, kernel_shape = [1, 1], pads = [0, 0, 0, 0], strides = [1, 1]](%40, %17, %18)
  %42 = Relu(%41)
  return %42
}

And this is the error message:

1/24: Converting Node Type Conv
Traceback (most recent call last):
  File "convert.py", line 38, in <module>
    image_output_names=["output"])
  File "/Users/stephenf/Developer/miniconda3/envs/pytorch/lib/python3.6/site-packages/onnx_coreml/converter.py", line 396, in convert
    _convert_node(builder, node, graph, err)
  File "/Users/stephenf/Developer/miniconda3/envs/pytorch/lib/python3.6/site-packages/onnx_coreml/_operators.py", line 994, in _convert_node
    return converter_fn(builder, node, graph, err)
  File "/Users/stephenf/Developer/miniconda3/envs/pytorch/lib/python3.6/site-packages/onnx_coreml/_operators.py", line 31, in _convert_conv
    "Weight tensor: {} not found in the graph initializer".format(weight_name,))
  File "/Users/stephenf/Developer/miniconda3/envs/pytorch/lib/python3.6/site-packages/onnx_coreml/_error_utils.py", line 71, in missing_initializer
    format(node.op_type, node.inputs[0], node.outputs[0], err_message)
ValueError: Missing initializer error in op of type Conv, with input name = 0, output name = 19. Error message: Weight tensor: 1 not found in the graph initializer

From what I can gather, it says the weight tensor %1[FLOAT, 64x3x5x5] is missing. This is how I'm saving the model:

torch.save(model.state_dict(), "hourglass_model_{}.model".format(epoch))

ONNX loads it fine - it's just the step where I'm converting from ONNX to CoreML.

Any help in figuring this out would be greatly appreciated. I'm sure I've done a bunch of other things wrong, but I just need this thing to export for now.

Thanks,


title: How to visualize 3d joints of a SMPL model based on pose params

body:
I am trying to use demo.py in nkolot
/
GraphCMR | GitHub. I am interested in obtaining joints from the inferred SMPL image and visualize it similar to described in README of this project: gulvarol
/
smplpytorch | GitHub.
I also posted the issue here: https://github.com/nkolot/GraphCMR/issues/36.
What I tried that didn't work.
I changed https://github.com/nkolot/GraphCMR/blob/4e57dca4e9da305df99383ea6312e2b3de78c321/demo.py#L118 to
pred_vertices, pred_vertices_smpl, pred_camera, smpl_pose, smpl_shape = model(...) to get smpl_pose (of shape torch.Size([1, 24, 3, 3])). Then I just flattened it by doing smpl_pose.cpu().data.numpy()[:, :, :, -1].flatten('C').reshape(1, -1) and used the resulting (1, 72) pose params as input in pose_params variable of smplpytorch demo.
The resulting visualization doesn't look correct to me. Is this the right approach? Perhaps there is an easier way to do what I am doing.
How to get 3d joints from demo.py and visualize it | nkolot
/
GraphCMR


title: BertForTokenClassification Not loading

body:
I tried to load a Bert model from local directory and it was showing an error
I am using cuda 10.0 version and pytorch 1.6.0
Code to load model:-
output_dir = './ner_model/'
model = BertForTokenClassification.from_pretrained(output_dir)
tokenizer = BertTokenizer.from_pretrained(output_dir)
model.to(device)

Any help would be appreicated
ReadError: invalid header

During handling of the above exception, another exception occurred:

RuntimeError                              Traceback (most recent call last)
~\anaconda3\envs\env\lib\site-packages\transformers\modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    511             try:
--> 512                 state_dict = torch.load(resolved_archive_file, map_location="cpu")
    513             except Exception:

~\anaconda3\envs\env\lib\site-packages\torch\serialization.py in load(f, map_location, pickle_module, **pickle_load_args)
    385     try:
--> 386         return _load(f, map_location, pickle_module, **pickle_load_args)
    387     finally:

~\anaconda3\envs\env\lib\site-packages\torch\serialization.py in _load(f, map_location, pickle_module, **pickle_load_args)
    558                 # .zip is used for torch.jit.save and will throw an un-pickling error here
--> 559                 raise RuntimeError("{} is a zip archive (did you mean to use torch.jit.load()?)".format(f.name))
    560             # if not a tarfile, reset file offset and proceed

RuntimeError: ./ner_model/pytorch_model.bin is a zip archive (did you mean to use torch.jit.load()?)

During handling of the above exception, another exception occurred:

OSError                                   Traceback (most recent call last)
<ipython-input-13-770da388c2c8> in <module>
     23
     24 output_dir = './ner_model/'
---> 25 model = BertForTokenClassification.from_pretrained(output_dir)
     26 tokenizer = BertTokenizer.from_pretrained(output_dir)
     27 model.to(device)

~\anaconda3\envs\env\lib\site-packages\transformers\modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    513             except Exception:
    514                 raise OSError(
--> 515                     "Unable to load weights from pytorch checkpoint file. "
    516                     "If you tried to load a PyTorch model from a TF 2.0 checkpoint, please set from_tf=True. "
    517                 )

OSError: Unable to load weights from pytorch checkpoint file. If you tried to load a PyTorch model from a TF 2.0 checkpoint, pleas

e set from_tf=True.


title: What do the logits and probabilities from RobertaForSequenceClassification represent?

body:
Being new to the "Natural Language Processing" scene, I am experimentally learning and have implemented the following segment of code:
from transformers import RobertaTokenizer, RobertaForSequenceClassification
import torch

path = "D:/LM/rb/"
tokenizer = RobertaTokenizer.from_pretrained(path)
model = RobertaForSequenceClassification.from_pretrained(path)

inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
outputs = model(**inputs)
pred_logits = outputs.logits
print(pred_logits)
probs = pred_logits.softmax(dim=-1).detach().cpu().flatten().numpy().tolist()
print(probs)

I understand that applying the model returns a "torch.FloatTensor comprising various elements depending on the configuration (RobertaConfig) and inputs", and that the logits are accessible using .logits. As demonstrated I have applied the .softmax function to the tensor to return normalised probabilities and have converted the result into a list. I am outputted with the following:
[0.5022980570793152, 0.49770188331604004]

Do these probabilities represent some kind of overall "masked" probability?
What do the first and second index represent in context of the input?

EDIT:
model.num_labels

Output:
2

@cronoik explains that the model "tries to classify if a sequence belongs to one class or another"
Am I to assume that because there are no trained output layers these classes don't mean anything yet?
For example, I can assume that the probability that the sentence, post analysis, belongs to class 1 is 0.5. However, what is class 1?
Additionally, model cards with pre-trained output layers such as the open-ai detector help differentiate between what is "real" and "fake", and so I can assume the class that a sentence belongs to. However, how can I confirm these "labels" without some type of "mapping.txt" file?


title: Cannot compare tensor to scalar anymore in new pytorch

body:
I was working with a self compiled version of pytorch 0.4.0 for a while and hence my code has the following line for multidimensional tensors:

myTensor3 = torch.arange(torch.numel(myTensor0) # this is a 1 dimensional Tensor
outTensor = myTensor3[(myTensor1 == 1) & (myTensor2 != 1)] # myTensor1 and myTensor2 are multidimensional

Whenever I try to switch to another (newer) version of pytorch, the above line throws the following error:

IndexError: too many indices for tensor of dimension 1

I assume in the newer version of pytorch the index array returned by the comparison operation has dimension 1, whereas before it had the same size as myTensor3.
How can I change back to the old behaviour in the newer version of pytorch?

Edit:
A simple fix was to linearize myTensor1 and 2:

outTensor = myTensor3[(myTensor1.view(-1) == 1) & (myTensor2.view(-1) != 1)]

for some strange reason this was already working in the self compiled pytorch version


title: How to call a function batchwise in Pytorch

body:
I have a function defined and I just want to know if it is possible to perform it batchwise. For instance,

def function():
    Some processes here
    return x

def forward():
    encode = self._encoding(embedded_premises,premises_lengths)

Now since, the encode will be 3D tensor, which will be batch size, seq_length, hidden size I want to perform function() batchwise and return x also batchwise.

Is there any other way than looping over all batches?


title: nvcc fatal : Could not open output file

body:
I use pytorch to object detection algorithms on windows:

When I run ext/build.py build_ext develop script
from https://github.com/lufficc/SSD repository
or when I run python support/setup.py develop
from https://github.com/potterhsu/easy-faster-rcnn.pytorch repository

import glob
import os

import torch
from setuptools import setup
from torch.utils.cpp_extension import CUDA_HOME
from torch.utils.cpp_extension import CppExtension
from torch.utils.cpp_extension import CUDAExtension

requirements = ["torch", "torchvision"]

def get_extensions():
    this_dir = os.path.dirname(os.path.abspath(__file__))
    extensions_dir = os.path.join(this_dir, "src")

    main_file = glob.glob(os.path.join(extensions_dir, "*.cpp"))
    source_cpu = glob.glob(os.path.join(extensions_dir, "cpu", "*.cpp"))
    source_cuda = glob.glob(os.path.join(extensions_dir, "cuda", "*.cu"))

    sources = main_file + source_cpu
    extension = CppExtension

    extra_compile_args = {"cxx": []}
    define_macros = []

    if torch.cuda.is_available() and CUDA_HOME is not None:
        extension = CUDAExtension
        sources += source_cuda
        define_macros += [("WITH_CUDA", None)]
        extra_compile_args["nvcc"] = [
            "-DCUDA_HAS_FP16=1",
            "-D__CUDA_NO_HALF_OPERATORS__",
            "-D__CUDA_NO_HALF_CONVERSIONS__",
            "-D__CUDA_NO_HALF2_OPERATORS__",
        ]

    sources = [os.path.join(extensions_dir, s) for s in sources]

    include_dirs = [extensions_dir]

    ext_modules = [
        extension(
            "support._C",
            sources,
            include_dirs=include_dirs,
            define_macros=define_macros,
            extra_compile_args=extra_compile_args,
        )
    ]

    return ext_modules

setup(
    name="support",
    version="0.1",
    ext_modules=get_extensions(),
    cmdclass={"build_ext": torch.utils.cpp_extension.BuildExtension},
)

I get the same error:

  C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2\bin\nvcc.exe
  -c C:\Users\along\PycharmProjects\SSD\ext\cuda\nms.cu -o build\temp.win-amd64-3.7\Release\Users\along\PycharmProjects\SSD\ext\cuda\nms.obj
  -IC:\Users\along\PycharmProjects\SSD\ext -IC:\Users\along\Anaconda3\lib\site-packages\torch\include -IC:\Users\along\Anaconda3\lib\site-packages\torch\include\torch\csrc\api\include
  -IC:\Users\along\Anaconda3\lib\site-packages\torch\include\TH -IC:\Users\along\Anaconda3\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2\include"
  -IC:\Users\along\Anaconda3\include -IC:\Users\along\Anaconda3\include "-IC:\Program Files (x86)\Microsoft Visual
  Studio\2017\Community\VC\Tools\MSVC\14.16.27023\ATLMFC\include"
  "-IC:\Program Files (x86)\Microsoft Visual
  Studio\2017\Community\VC\Tools\MSVC\14.16.27023\include" "-IC:\Program
  Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt" "-IC:\Program
  Files (x86)\Windows Kits\10\include\10.0.17763.0\shared" "-IC:\Program
  Files (x86)\Windows Kits\10\include\10.0.17763.0\um" "-IC:\Program
  Files (x86)\Windows Kits\10\include\10.0.17763.0\winrt" "-IC:\Program
  Files (x86)\Windows Kits\10\include\10.0.17763.0\cppwinrt" -Xcompiler
  /EHsc -Xcompiler /wd4819 -Xcompiler /MD -D__CUDA_NO_HALF_OPERATORS__
  -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -DCUDA_HAS_FP16=1 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=torch_extension -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_75,code=sm_75 nvcc fatal   : Could not open output file 'along\AppData\Local\Temp/tmpxft_00005b70_00000000' error:
  command 'C:\Program Files\NVIDIA GPU Computing
  Toolkit\CUDA\v10.2\bin\nvcc.exe' failed with exit status 1

I don't know what is the error, I have a GPU device on the computer

I think that the problem is on CMakeLists.txt file on Anaconda3\pkgs\pytorch-1.3.1-py3.7_cuda92_cudnn7_0\info\test\test\custom_operator directory:

# Basic CMake setup
cmake_minimum_required(VERSION 3.1 FATAL_ERROR)
project(custom_ops)

find_package(Torch REQUIRED)

add_library(custom_ops SHARED op.cpp)

add_subdirectory(${CMAKE_CURRENT_SOURCE_DIR}/../../third_party/pybind11/ ./pybind11)
pybind11_add_module(custom_class SHARED classes.cpp)
target_link_libraries(custom_class PRIVATE "${TORCH_LIBRARIES}")

target_compile_features(custom_ops PUBLIC cxx_range_for)
target_link_libraries(custom_ops "${TORCH_LIBRARIES}")
target_compile_definitions(custom_ops PRIVATE custom_ops_EXPORTS)

add_executable(test_custom_ops test_custom_ops.cpp)
target_link_libraries(test_custom_ops custom_ops)


title: PyTorch ToTensor Changes C x H x W (5 x 600 x 900) to H x C x W (900 x 5 x 600)

body:
Here is my DataLoader. When I use ToTensor, it changes the dimensions of the image to H x C x W. Is permute okay to fix this or this might change some orientation?

class DPWHDataset(Dataset):
  def __init__(self, mean=None, std=None, phase=None, dataset=None):
    self.data = dataset
    self.mean = mean
    self.std = std
    self.phase = phase
    self.transforms = get_transforms(phase, mean, std)

  def __len__(self):
    return len(self.data)

  def __getitem__(self, idx):
    image_name = self.data[idx]

    image_path = image_prefix + image_name + ".jpg"
    mask_path = binary_mask_prefix + image_name + "_mask.png"

    mask = cv2.imread(mask_path, 0)
    print(image_path)

    # image = np.array(Image.open(image_path))
    # mask = np.array(Image.open(mask_path))
    image = cv2.imread(image_path)
    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)

    mask = create_channel_mask(mask)
    # augmented = self.transforms(image=image, mask=mask)
    # image = augmented['image']
    # mask = augmented['mask']

    image = torchvision.transforms.ToTensor()(image)
    image = torchvision.transforms.Normalize(mean=self.mean, std=self.std)(image)
    mask = torchvision.transforms.ToTensor()(mask)
    return image, mask


title: PyTorch - Element-wise signed min/max?

body:
I may be missing something obvious, but I can't find a way to compute this.
Given two tensors, I want to keep the minimum elements in each one of them as well as the sign.
I thought about
sign_x = torch.sign(x)
sign_y = torch.sign(y)
min = torch.min(torch.abs(x), torch.abs(y))

in order to eventually multiply the signs with the obtained minimums, but then I have no method to multiply the correct sign to each element that was kept and must choose one of the two tensors.


title: Is there an equivalent function of pytorch named "index_select" in tensorflow

body: I tried to translate pytorch code to tensorflow. So I wanna know is there an equivalent function of pytorch named "index_select" in tensorflow


title: Pytorch LSTM text-generator repeats same words

body:
UPDATE: It was a mistake in the logic generating new characters. See answer below.

ORIGINAL QUESTION: I built an LSTM for character-level text generation with Pytorch. The model trains well (loss decreases reasonably etc.) but the trained model ends up outputting the last handful of words of the input repeated over and over again (e.g. Input: "She told her to come back later, but she never did"; Output: ", but she never did, but she never did, but she never did" and so on).

I have played around with the hyperparameters a bit, and the problem persists. I'm currently using:

Loss function: BCE
Optimizer: Adam
Learning rate: 0.001
Sequence length: 64
Batch size: 32
Embedding dim: 128
Hidden dim: 512
LSTM layers: 2

I also tried not always choosing the top choice, but this only introduces incorrect words and doesn't break the loop. I've been looking at countless tutorials, and I can't quite figure out what I'm doing differently/wrong.

The following is the code for training the model. training_data is one long string and I'm looping over it predicting the next character for each substring of length SEQ_LEN. I'm not sure if my mistake is here or elsewhere but any comment or direction is highly appreciated!

loss_dict = dict()
for e in range(EPOCHS):
    print("------ EPOCH {} OF {} ------".format(e+1, EPOCHS))

    lstm.reset_cell()

    for i in range(0, DATA_LEN, BATCH_SIZE):

        if i % 50000 == 0:
            print(i/float(DATA_LEN))

        optimizer.zero_grad()

        input_vector = torch.tensor([[
            vocab.get(char, len(vocab))
            for char in training_data[i+b:i+b+SEQ_LEN]
        ] for b in range(BATCH_SIZE)])

        if USE_CUDA and torch.cuda.is_available():
            input_vector = input_vector.cuda()

        output_vector = lstm(input_vector)

        target_vector = torch.zeros(output_vector.shape)

        if USE_CUDA and torch.cuda.is_available():
            target_vector = target_vector.cuda()

        for b in range(BATCH_SIZE):
            target_vector[b][vocab.get(training_data[i+b+SEQ_LEN])] = 1

        error = loss(output_vector, target_vector)

        error.backward()
        optimizer.step()

        loss_dict[(e, int(i/BATCH_SIZE))] = error.detach().item()


title: How to copy `grad_fn` in pytorch?

body:
>>> print(foo.grad_fn)
<AddBackward0 object at 0x7f7f9f450710>

I want to copy from foo.grad_fn to bar.grad_fn. For reference, no foo.data is required. I want to copy only the gradient.

Is this possible? I tried the following and it failed.

>>> bar.grad_fn = foo.grad_fn
AttributeError: attribute 'grad_fn' of 'torch._C._TensorBase' objects is not writable

thank you.


title: When using Conv2D in PyTorch, does padding or dilation happen first?

body:
Consider the following bit of code:

torch.nn.Conv2d(1, 1, 2, padding = 1, dilation = 2)

Which of the following two cases is a correct interpretation?


title: How to compute hessian matrix for all parameters in a network in pytorch?

body:
Suppose vector \theta is all the parameters in a neural network, I wonder how to compute hessian matrix for \theta in pytorch.
Suppose the network is as follows:
class Net(Module):
    def __init__(self, h, w):
        super(Net, self).__init__()
        self.c1 = torch.nn.Conv2d(1, 32, 3, 1, 1)
        self.f2 = torch.nn.Linear(32 * h * w, 5)

    def forward(self, x):
        x = self.c1(x)
        x = x.view(x.size(0), -1)
        x = self.f2(x)
        return x

I know the second derivative can be calculated by calling torch.autograd.grad() twice, but the parameters in pytorch is organized by net.parameters(), and I don't know how to compute the hessian for all parameters.
I have tried to use torch.autograd.functional.hessian() in pytorch 1.5 as follows:
import torch
import numpy as np
from torch.nn import Module
import torch.nn.functional as F

class Net(Module):
    def __init__(self, h, w):
        super(Net, self).__init__()
        self.c1 = torch.nn.Conv2d(1, 32, 3, 1, 1)
        self.f2 = torch.nn.Linear(32 * h * w, 5)

    def forward(self, x):
        x = self.c1(x)
        x = x.view(x.size(0), -1)
        x = self.f2(x)
        return x

def func_(a, b c, d):
    p = [a, b, c, d]
    x = torch.randn(size=[8, 1, 12, 12], dtype=torch.float32)
    y = torch.randint(0, 5, [8])
    x = F.conv2d(x, p[0], p[1], 1, 1)
    x = x.view(x.size(0), -1)
    x = F.linear(x, p[2], p[3])
    loss = F.cross_entropy(x, y)
    return loss

if __name__ == '__main__':
    net = Net(12, 12)

    h = torch.autograd.functional.hessian(func_, tuple([_ for _ in net.parameters()]))
    print(type(h), len(h))

h is a tuple, and the results are in strange shape. For example, the shape of \frac{\delta Loss^2}{\delta c1.weight^2} is [32,1,3,3,32,1,3,3]. It seems like I can combine them into a complete H, but I don't know which part it is in the whole Hessian Matrix and the corresponding order.


title: Non-reproducible results in pytorch after saving and loading the model

body:
I am unable to reproduce my results in PyTorch after saving and loading the model whereas the in-memory model works as expected. Just for context, I am seeding my libraries, using model.eval to turn off the dropouts but still, results are not reproducible. Any suggestions if I am missing something.  Thanks in advance.

Libraries that I am using:-

import matplotlib.pyplot as plt
import torch
from torch import nn
from torch import optim
import torch.nn.functional as F
from torchvision import datasets, transforms
import numpy as np
import random

Libraries that I am seeding

manualSeed = 1
np.random.seed(manualSeed)
random.seed(manualSeed)
torch.manual_seed(manualSeed)
random.seed(manualSeed)

Below are the results on in memory and loaded model.

In Memory model Loss : 1.596395881312668, In Memory model Accuracy : tensor(0.3989)
Loaded model Loss : 1.597083057567572, Loaded model Accuracy : tensor(0.3983)


title: How to create a linspace like pytorch tensor which ensures that all values are between start and stop?

body:
I want to create a 1d tensor of values similar to torch.linspace or torch.arange in pytorch, which ensures that all values of the created tensor are between start and stop.

Example values which break my assertion:

>>> start = torch.tensor(3.197054147720337, dtype=torch.float)
>>> stop = torch.tensor(11, dtype=torch.float)
>>> num_samples = 8

So the result_tensor should pass this test:

>>> assert (start <= result_tensor).all() and (result_tensor <= stop).all()

But unfortunately, neither the linspace solution fulfills this requirement:

>>> result_tensor = torch.linspace(start.item(), stop.item(), num_samples)
>>> result_tensor[-1] - stop
torch.tensor(9.5367e-07)

Nor does the arange solution:

>>> step = (stop - start) / (num_samples - 1)
>>> result_tensor = torch.arange(num_samples, dtype=torch.float) * step + start
>>> result_tensor[-1] - stop
torch.tensor(9.5367e-07)

In both possibilties, the biggest value of the result_tensor is slightly higher than stop.
Is there a way to ensure this requirement for all numbers without doing explicit value clipping?

(Gradientes are not required. A vectorizeable solution like the one with torch.arange is preferable, but not a must.start, stop, between, step could be matrices where the last dimension is of size 1 and is broadcasted to num_samples)


title: Pytorch Concatenate rows in alternate order

body:
I am trying to code up the positional encoding in the transformers paper. In order to do so I need to do an operation similar to the following:

a = torch.arange(20).reshape(4,5)
b = a * 2
c = torch.cat([torch.stack([a_row,b_row]) for a_row, b_row in zip(a,b)])

I feel like there might be a faster way to do the above? perhaps by adding a dimension on to a and b?


title: pytorch-yolov3 train RuntimeError: Unable to find a valid cuDNN algorithm to run convolution

body:
https://github.com/ultralytics/yolov3/
When I started training yolov3 with 'python train.py'  this happened, what is the reason?
Traceback (most recent call last):
    File "train.py", line 415, in <module>
        train()  # train normally
    File "train.py", line 279, in train
        scaled_loss.backward()
    File "C:\Users\zst\anaconda3\lib\site-packages\torch\tensor.py", line 198, in backward
        torch.autograd.backward(self, gradient, retain_graph, create_graph)
    File "C:\Users\zst\anaconda3\lib\site-packages\torch\autograd\__init__.py", line 100, in backward
        allow_unreachable=True)  # allow_unreachable flag
RuntimeError: Unable to find a valid cuDNN algorithm to run convolution (try_all at ..\aten\src\ATen\native\cudnn\Conv.cpp:693)
(no backtrace available)

win10 2080ti  cuda10.2 cudnn7604  python 3.7 torch1.5 torchvision0.6
Trace screenshot:

Environment screenshot:


title: Is this a right way to descrease size of my docker images?

body:
I am running a deep learning model in Docker container which needs pytorch and Azure ML service.
AML requirement is Ubuntu 18.04 (which by default has only python3.6 and only way to install python3.7+ is from the source from what i was able to find)
transformers in pytorch have a requirement of python 3.7+
and i need pytorch with Cuda so i choose abinali/pytorch https://github.com/anibali/docker-pytorch/blob/master/dockerfiles/1.5.0-cuda10.2-ubuntu18.04/Dockerfile
The issue is that the size of the image is around 4GB +. so I wanted to move the PyTorch installation out of the image and run install when the docker is running the container(which increases container running uptime).
Error when running basic torch commands
File "test.py", line 1, in <module>
    import torch
  File "/home/user/miniconda/lib/python3.8/site-packages/torch/__init__.py", line 135, in <module>
    _load_global_deps()
  File "/home/user/miniconda/lib/python3.8/site-packages/torch/__init__.py", line 93, in _load_global_deps
    ctypes.CDLL(lib_path, mode=ctypes.RTLD_GLOBAL)
  File "/home/user/miniconda/lib/python3.8/ctypes/__init__.py", line 373, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /home/user/miniconda/lib/python3.8/site-packages/torch/lib/../../../../libnvToolsExt.so.1: invalid ELF header

so my current docker file is :
From nvidia/cuda:10.2-base-ubuntu18.04

RUN apt-get update && apt-get install -y \
    curl \
    ca-certificates \
    sudo \
    git \
    bzip2 \
    libx11-6 \
 && rm -rf /var/lib/apt/lists/*

# Create a working directory

# COPY . /app
# WORKDIR /app/
RUN mkdir /app
COPY . /app
WORKDIR /app

# Create a non-root user and switch to it
RUN adduser --disabled-password --gecos '' --shell /bin/bash user \
 && chown -R user:user /app
RUN echo "user ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/90-user
USER user

# All users can use /home/user as their home directory
ENV HOME=/home/user
RUN chmod 777 /home/user

# Install Miniconda and Python 3.8
ENV CONDA_AUTO_UPDATE_CONDA=false
ENV PATH=/home/user/miniconda/bin:$PATH
RUN curl -sLo ~/miniconda.sh https://repo.continuum.io/miniconda/Miniconda3-py38_4.8.2-Linux-x86_64.sh \
 && chmod +x ~/miniconda.sh \
 && ~/miniconda.sh -b -p ~/miniconda \
 && rm ~/miniconda.sh \
 && conda install -y python==3.8.1 \
 && conda clean -ya

CMD ["sh", "-c", "conda install -y -c pytorch cudatoolkit=10.2 \"pytorch=1.5.0=py3.8_cuda10.2.89_cudnn7.6.5_0\" \"torchvision=0.6.0=py38_cu102\" && conda clean -ya && python test.py"]


title: Pytorch parameters won't update with custom loss function (Pytorch)

body:
I am trying to use the optimizer to tune a set of parameters for a cost function that includes, among other things, a forward pass across a neural network. The parameters specify the means and variances of the weights of this neural network. However, when updating the parameters at every iteration of the optimization process, all the terms of the cost function except the one belonging to the forward pass contribute to the parameter update. That is, if all other terms are commented out, no parameters will update. Are there any ways of fixing this issue?

EDIT: I added a contrived example below.

import torch

class TestNN(torch.nn.Module):
    def __init__(self):
        super(TestNN, self).__init__()
        self.fc1 = torch.nn.Linear(10, 1)

    def forward(self, x):
        x = self.fc1(x)
        return x

    def getParameters(self):
        return [self.fc1.weight.transpose(0, 1), self.fc1.bias]

    def setParameters(self, parameters):
        # Can anything be done here to keep parameters in the graph?
        weight, bias = parameters
        self.fc1.weight = torch.nn.Parameter(weight.transpose(0, 1))
        self.fc1.bias = torch.nn.Parameter(bias)

def computeCost(parameters, input):
    testNN = TestNN()
    testNN.setParameters(parameters)
    cost = testNN(input) ** 2
    print(cost) # Cost stays the same :(
    return cost

def minimizeLoss(maxIter, optimizer, lossFunc, lossFuncArgs):
    for i in range(maxIter):
        optimizer.zero_grad()
        loss = lossFunc(*lossFuncArgs)
        loss.backward(retain_graph = True)
        optimizer.step()
        if i % 100 == 0:
            print(loss)

input = torch.randn(1, 10)
weight = torch.ones(10, 1)
bias = torch.ones(1, 1)

parameters = (weight, bias)
lossArgs = (parameters, input)
optimizer = torch.optim.Adam(parameters, lr = 0.01)

minimizeLoss(10, optimizer, computeCost, lossArgs)


title: Training model in eval() mode gives better result in PyTorch?

body:
I have a model with Dropout layers (with p=0.6). I ended up training the model in .eval() mode and again trained the model in .train() mode, I find that the training .eval() mode gave me better accuracy and quicker loss reduction on training data,

train(): Train loss : 0.832, Validation Loss : 0.821
eval(): Train loss : 0.323, Validation Loss : 0.251

Why is this so?


title: free up the memory allocation cuda pytorch?

body:
RuntimeError: CUDA out of memory. Tried to allocate 12.00 MiB (GPU 1;
  11.91 GiB total capacity; 10.12 GiB already allocated; 21.75 MiB free; 56.79 MiB cached)

I encountered the preceding error during pytorch training.
I'm using pytorch on jupyter notebook. Is there a way to free up the gpu memory in jupyter notebook?


title: How do I do masking in PyTorch / Numpy with different dimensions?

body:
I have a mask with a size of torch.Size([20, 1, 199]) and a tensor, reconstruct_output and inputs both with a size of torch.Size([20, 1, 161, 199]).

I want to set reconstruct_output to inputs where the mask is 0. I tried:

reconstruct_output[mask == 0] = inputs[mask == 0]

But I get an error:

IndexError: The shape of the mask [20, 1, 199] at index 2 does not match the shape of the indexed tensor [20, 1, 161, 199] at index 2


title: BERT Fine tuning: High loss and low accuracy in multiclass classification

body:
while binary classification with a finetuned Bert worked well, I am stuck with the multiclass classification. My dataset (german news articles, with 10 classes) contains roughly 10.000 samples.
Although, training loss and average evaluation loss circle around 2.2.
Some NLP config variables:
DEBUG=True
VERSION = 1
MAX_LEN = 200 #Set the maximum length according to the diagram above
BATCH_SIZE = 16
EPOCHS = 3
LEARNING_RATE = 2e-4
MOMENTUM = 0.9
TRAIN_SIZE = 0.7
NUM_LABELS = len(df_data.Labels.unique())
MODEL_NAME = "dbmdz/bert-base-german-cased"
params = {"debug": DEBUG, "max_len": MAX_LEN, "batch_size": BATCH_SIZE, "epochs": EPOCHS, "lr": LEARNING_RATE, "momentum": MOMENTUM, "model": MODEL_NAME, "loss": "BCEWithLogitsLoss", "optimizer": "SGD"}
tokenizer = transformers.AutoTokenizer.from_pretrained(MODEL_NAME)

The DataLoader:
class NLPDataset(th.utils.data.Dataset):

  def __init__(self, dataframe, tokenizer, max_len):
    self.tokenizer = tokenizer
    self.max_len = max_len
    self.data = dataframe
    self.text = dataframe.Text
    self.targets = dataframe.Labels
    self.len = len(self.text)

  def __getitem__(self,idx):
    text = str(self.text[idx])
    text = " ".join(text.split())

    inputs = self.tokenizer(
        text,
        None,
        add_special_tokens=True,
        max_length=self.max_len,
        pad_to_max_length=True,
        return_token_type_ids=True,
        truncation=True,
        padding='max_length',
    )
    input_ids = inputs['input_ids']
    mask = inputs['attention_mask']
    token_type_ids = inputs["token_type_ids"]

    return {
        'input_ids': th.tensor(input_ids, dtype=th.long),
        'mask': th.tensor(mask, dtype=th.float),
        'token_type_ids': th.tensor(token_type_ids, dtype=th.long),
        'targets': th.tensor(self.targets[idx], dtype=th.long)
    }

  def __len__(self):
    return self.len

The Pytorch_lightning Module:
class NLPClassifier(pl.LightningModule):

  def __init__(self):
      super().__init__()

      #changing the configuration to 10 lables instead of 2
      config = transformers.AutoConfig.from_pretrained(MODEL_NAME)
      config.num_labels = NUM_LABELS
      self.model = transformers.AutoModelForSequenceClassification.from_config(config)
      self.loss = th.nn.CrossEntropyLoss(reduction="none")

  def prepare_data(self):
      # train/val split
      train_dataset = df_data.sample(frac=TRAIN_SIZE)
      val_dataset=df_data.drop(train_dataset.index).sample(frac=1).reset_index(drop=True)
      train_dataset = train_dataset.reset_index(drop=True)

      # Assign CustomDataset Class
      train_set = NLPDataset(train_dataset, tokenizer, MAX_LEN)
      val_set = NLPDataset(val_dataset, tokenizer, MAX_LEN)

      print("FULL Dataset: {}".format(df_data.shape))
      print("TRAIN Dataset: {}".format(train_dataset.shape))
      print("VAL Dataset: {}".format(val_dataset.shape))

      # assign to use in dataloaders
      self.train_ds = train_set
      self.val_ds = val_set
      #self.test_dataset = mnist_test DO TO

  def forward(self, input_ids, mask):
      logits, = self.model(input_ids, mask)
      # logits.shape: (16, 10)
      return logits

  def training_step(self, batch, batch_idx):
    logits = self.forward(batch['input_ids'], batch['mask']).squeeze()
    loss = self.loss(logits, batch['targets']).mean()
    run.log(name='train_loss', value=loss.tolist())
    return {'loss': loss, 'log': {'train_loss': loss}}

  def validation_step(self,batch, batch_idx):
    logits = self.forward(batch['input_ids'], batch['mask']).squeeze()
    print(logits.shape)
    acc = (logits.argmax(-1) == batch['targets']).float()
    loss = self.loss(logits, batch['targets'])
    run.log_list('loss', loss.tolist())
    run.log_list('acc', acc.tolist())
    return {'loss': loss, 'acc': acc}

  def validation_epoch_end(self, outputs):
    loss = th.cat([o['loss'] for o in outputs], 0).mean()
    acc = th.cat([o['acc'] for o in outputs], 0).mean()
    out = {'val_loss': loss, 'val_acc': acc}
    run.log('val_loss', loss.tolist())
    run.log('val_acc', acc.tolist())
    return {**out, 'log': {'val_loss': loss, 'val_acc': acc}}

  def train_dataloader(self):
      return th.utils.data.DataLoader(
          self.train_ds,
          batch_size=BATCH_SIZE,
          num_workers=8,
          drop_last=True,
          shuffle=False,
      )

  def val_dataloader(self):
      return th.utils.data.DataLoader(
          self.val_ds,
          batch_size=BATCH_SIZE,
          num_workers=8,
          drop_last=False,
          shuffle=False,
      )

  def configure_optimizers(self):
      return transformers.AdamW(
            self.model.parameters(),
            lr=LEARNING_RATE,
            #momentum=MOMENTUM,
        )

The trainer:
model = NLPClassifier()
trainer = pl.Trainer(
   gpus=(1 if th.cuda.is_available() else 0),
   default_root_dir = f"./models/version_{VERSION}",
   max_epochs=EPOCHS,
   fast_dev_run=DEBUG,
   limit_train_batches=1.0,
   val_check_interval=0.5,
   limit_val_batches=1.0,
   profiler=True,
   #logger=wandb_logger
   )

trainer.fit(model)
This is a sample loss curve.
Sample loss curve
My central questions are:

Is CrossEntropyLoss correctly used?
Does the optimizer work, as the predictions fairly quickly become the same for every sample.
The learning rate issue did not solve the problem. I tried a range from 1e-2 to 1e-6

Thanks for your help. :)


title: model.to(device) for Pytorch Lighting

body:
I currently train my model using GPUs using Pytorch Lightning
trainer = pl.Trainer( gpus=[0,1],
        distributed_backend='ddp',
        resume_from_checkpoint=hparams["resume_from_checkpoint"])

    trainer.fit(model, train_dataloader=train_loader, val_dataloaders=val_loader)

Instructions are also clear for how to run test samples with trainer defined to use GPU
trainer.test(test_dataloader=test_dataloader)

and also how to load a model and use it interactively
model = transformer.Model.load_from_checkpoint('/checkpoints/run_300_epoch_217.ckpt')
results = model(in_data,

I use the later to interface with an interactive system via sockets in a docker container.
Is there a proper way to make this Pytorch Lightning model run on GPU?
Lightning instructions say not to use model.to(device), but it appears to work just like Pytorch. Reason for instructions to avoid a side effect?
I started reading about ONNX, but would rather just have an easy way to specify GPU since the interactive setup works perfectly with cpu.


title: How can i get all outputs of the last transformer encoder in bert pretrained model and not just the cls token output?

body:
I'm using pytorch and this is the model from huggingface transformers link:
from transformers import BertTokenizerFast, BertForSequenceClassification
bert = BertForSequenceClassification.from_pretrained("bert-base-uncased",
                                                     num_labels=int(data['class'].nunique()),
                                                     output_attentions=False,
                                                     output_hidden_states=False)

and in the forward function I'm building, I'm calling x1, x2 = self.bert(sent_id, attention_mask=mask)
Now, as far as I know, x2 is the cls output(which is the output of the first transformer encoder) but yet again, I don't think I understand the output of the model.
but I want the output of all the 12 last transformer encoders.
How can I do that in pytorch ?


title: PyTorch RuntimeError : Gradients are not CUDA tensors

body:
I am getting the following error while doing seq to seq on characters and feeding to LSTM, and decoding to words using attention. The forward propagation is fine but while computing loss.backward() I am getting the following error.

  RuntimeError: Gradients aren't CUDA tensors

My train() function is as followed.

def train(input_batch, input_batch_length, target_batch, target_batch_length, batch_size):

    # Zero gradients of both optimizers
    encoderchar_optimizer.zero_grad()
    encoder_optimizer.zero_grad()
    decoder_optimizer.zero_grad()

    encoder_input = Variable(torch.FloatTensor(len(input_batch), batch_size, 500))

    for ix , w in enumerate(input_batch):
        w = w.contiguous().view(15, batch_size)
        reshaped_input_length = [x[ix] for x in input_batch_length] # [15 ,.. 30 times] * 128
        if USE_CUDA:
            w = w.cuda()
            #reshaped_input_length =  Variable(torch.LongTensor(reshaped_input_length)).cuda()
        hidden_all , output = encoderchar(w, reshaped_input_length)
        encoder_input[ix] = output.transpose(0,1).contiguous().view(batch_size, -1)
        if USE_CUDA:
            encoder_input = encoder_input.cuda()

    temporary_target_batch_length = [15] * batch_size

    encoder_hidden_all, encoder_output = encoder(encoder_input, target_batch_length)
    decoder_input = Variable(torch.LongTensor([SOS_token] * batch_size))
    decoder_hidden = encoder_output

    max_target_length = max(temporary_target_batch_length)
    all_decoder_outputs = Variable(torch.zeros(max_target_length, batch_size, decoder.output_size))

    # Move new Variables to CUDA
    if USE_CUDA:
        decoder_input = decoder_input.cuda()
        all_decoder_outputs = all_decoder_outputs.cuda()
        target_batch =  target_batch.cuda()

    # Run through decoder one time step at a time
    for t in range(max_target_length):
        decoder_output, decoder_hidden, decoder_attn = decoder(
            decoder_input, decoder_hidden, encoder_hidden_all
        )

        all_decoder_outputs[t] = decoder_output
        decoder_input = target_batch[t] # Next input is current target
        if USE_CUDA:
            decoder_input = decoder_input.cuda()

    # Loss calculation and backpropagation
    loss = masked_cross_entropy(
        all_decoder_outputs.transpose(0, 1).contiguous(), # -> batch x seq
        target_batch.transpose(0, 1).contiguous(), # -> batch x seq
        target_batch_length
    )
    loss.backward()

    # Clip gradient norms
    ecc = torch.nn.utils.clip_grad_norm(encoderchar.parameters(), clip)
    ec = torch.nn.utils.clip_grad_norm(encoder.parameters(), clip)
    dc = torch.nn.utils.clip_grad_norm(decoder.parameters(), clip)

    # Update parameters with optimizers
    encoderchar_optimizer.step()
    encoder_optimizer.step()
    decoder_optimizer.step()

    return loss.data[0], ec, dc

Full Stack Trace is here.

RuntimeError                              Traceback (most recent call last)
<ipython-input-10-9778e12ded02> in <module>()
     11         data_target_batch_index= Variable(torch.LongTensor(data_target_batch_index)).transpose(0,1)
     12         # Send the data for training
---> 13         loss, ar1, ar2 = train(data_input_batch_index, data_input_batch_length, data_target_batch_index, data_target_batch_length, batch_size)
     14
     15         # Keep track of loss

<ipython-input-8-9c71c385f8cd> in train(input_batch, input_batch_length, target_batch, target_batch_length, batch_size)
     54         target_batch_length
     55     )
---> 56     loss.backward()
     57
     58     # Clip gradient norms

/home/ubuntu/anaconda3/envs/tensorflow/lib/python3.6/site-packages/torch/autograd/variable.py in backward(self, gradient, retain_variables)
    144                     'or with gradient w.r.t. the variable')
    145             gradient = self.data.new().resize_as_(self.data).fill_(1)
--> 146         self._execution_engine.run_backward((self,), (gradient,), retain_variables)
    147
    148     def register_hook(self, hook):

/home/ubuntu/anaconda3/envs/tensorflow/lib/python3.6/site-packages/torch/autograd/function.py in _do_backward(self, gradients, retain_variables)
    207     def _do_backward(self, gradients, retain_variables):
    208         self.retain_variables = retain_variables
--> 209         result = super(NestedIOFunction, self)._do_backward(gradients, retain_variables)
    210         if not retain_variables:
    211             del self._nested_output

/home/ubuntu/anaconda3/envs/tensorflow/lib/python3.6/site-packages/torch/autograd/function.py in backward(self, *gradients)
    215     def backward(self, *gradients):
    216         nested_gradients = _unflatten(gradients, self._nested_output)
--> 217         result = self.backward_extended(*nested_gradients)
    218         return tuple(_iter_None_tensors(result))
    219

/home/ubuntu/anaconda3/envs/tensorflow/lib/python3.6/site-packages/torch/nn/_functions/rnn.py in backward_extended(self, grad_output, grad_hy)
    314             grad_hy,
    315             grad_input,
--> 316             grad_hx)
    317
    318         if any(self.needs_input_grad[1:]):

/home/ubuntu/anaconda3/envs/tensorflow/lib/python3.6/site-packages/torch/backends/cudnn/rnn.py in backward_grad(fn, input, hx, weight, output, grad_output, grad_hy, grad_input, grad_hx)
    371                 hidden_size, dcy.size()))
    372         if not dhy.is_cuda or not dy.is_cuda or (dcy is not None and not dcy.is_cuda):
--> 373             raise RuntimeError('Gradients aren\'t CUDA tensors')
    374
    375         check_error(cudnn.lib.cudnnRNNBackwardData(

RuntimeError: Gradients aren't CUDA tensors

any suggestions about why I am doing wrong?


title: Hidden units saturate in a seq2seq model in PyTorch

body:
I'm trying to write a very simple machine translation toy example in PyTorch. To simply the question, I turn the machine translation task into this one:

  Given a random sequence ([4, 8, 9 ...]), predict a sequence whose elements its elements plus 1 ([5, 9, 10, ...]). Id: 0, 1, 2 will be used as pad, bos, eos, respectively.

I observed the same problem in this toy task in my machine translation task. To debug, I use a very small data size n_data = 50, and find that the model can not even overfit these data. Looking into the model, I find that, the hidden state of the encoder/decoder soon becomes saturated, namely, all units in the hidden state become very close to 1/-1 due to the tanh.

-0.8987  0.9634  0.9993  ...  -0.8930 -0.4822 -0.9960
-0.9673  1.0000 -0.8007  ...   0.9929 -0.9992  0.9990
-0.9457  0.9290 -0.9260  ...  -0.9932  0.9851  0.9980
          ...             ⋱             ...
-0.9995  0.9997 -0.9350  ...  -0.9820 -0.9942 -0.9913
-0.9951  0.9488 -0.8894  ...  -0.9842 -0.9895 -0.9116
-0.9991  0.9769 -0.5871  ...   0.7557  0.9049  0.9881

Also, no matter how I adjust the learning rate, or switch the units to RNN/LSTM/GRU unit, the loss value seems to have a low bound even with 50 test samples. With more data, the model seems not converge at all.

step: 0, loss: 2.313938
step: 10, loss: 1.435780
step: 20, loss: 0.779704
step: 30, loss: 0.395590
step: 40, loss: 0.281261
...
step: 480, loss: 0.231419
step: 490, loss: 0.231410

When I use tensorflow, I can overfit such a dataset using a seq2seq model easily, and have a very small loss value.

Here are what I've tried:

Manually initialize the embedding to very small numbers;
Clipping the gradient to fixed norm like 1e-2, 2, 3, 5, 10;
Excluding the padding index (by adding ignore_index to NLLLoss) when computing loss.

All of what I've tried did nothing help to the problem.

How can I get rid of this? Any help will be appreciated.

Here's the code, for better reading experience, it's on gist.

#!/usr/bin/env python3
# -*- coding: utf-8 -*-

import numpy as np
import torch
import torch.nn.functional as F
from torch import nn
from torch.autograd import Variable

np.random.seed(0)
torch.manual_seed(0)

_RECURRENT_FN_MAPPING = {
    'rnn': torch.nn.RNN,
    'gru': torch.nn.GRU,
    'lstm': torch.nn.LSTM,
}

def get_recurrent_cell(n_inputs,
                       num_units,
                       num_layers,
                       type_,
                       dropout=0.0,
                       bidirectional=False):
    cls = _RECURRENT_FN_MAPPING.get(type_)

    return cls(
        n_inputs,
        num_units,
        num_layers,
        dropout=dropout,
        bidirectional=bidirectional)

class Recurrent(nn.Module):

    def __init__(self,
                 num_units,
                 num_layers=1,
                 unit_type='gru',
                 bidirectional=False,
                 dropout=0.0,
                 embedding=None,
                 attn_type='general'):
        super(Recurrent, self).__init__()

        num_inputs = embedding.weight.size(1)
        self._num_inputs = num_inputs
        self._num_units = num_units
        self._num_layers = num_layers
        self._unit_type = unit_type
        self._bidirectional = bidirectional
        self._dropout = dropout
        self._embedding = embedding
        self._attn_type = attn_type
        self._cell_fn = get_recurrent_cell(num_inputs, num_units, num_layers,
                                           unit_type, dropout, bidirectional)

    def init_hidden(self, batch_size):
        direction = 1 if not self._bidirectional else 2
        h = Variable(
            torch.zeros(direction * self._num_layers, batch_size,
                        self._num_units))
        if self._unit_type == 'lstm':
            return (h, h.clone())
        else:
            return h

    def forward(self, x, h, len_x):
        # Sort by sequence lengths
        sorted_indices = np.argsort(-len_x).tolist()
        unsorted_indices = np.argsort(sorted_indices).tolist()
        x = x[:, sorted_indices]
        h = h[:, sorted_indices, :]
        len_x = len_x[sorted_indices].tolist()

        embedded = self._embedding(x)
        packed = torch.nn.utils.rnn.pack_padded_sequence(embedded, len_x)

        if self._unit_type == 'lstm':
            o, (h, c) = self._cell_fn(packed, h)
            o, _ = torch.nn.utils.rnn.pad_packed_sequence(o)
            return (o[:, unsorted_indices, :], (h[:, unsorted_indices, :],
                                                c[:, unsorted_indices, :]))
        else:
            o, hh = self._cell_fn(packed, h)
            o, _ = torch.nn.utils.rnn.pad_packed_sequence(o)
            return (o[:, unsorted_indices, :], hh[:, unsorted_indices, :])

class Encoder(Recurrent):
    pass

class Decoder(Recurrent):
    pass

class Seq2Seq(nn.Module):

    def __init__(self, encoder, decoder, num_outputs):
        super(Seq2Seq, self).__init__()
        self._encoder = encoder
        self._decoder = decoder
        self._out = nn.Linear(decoder._num_units, num_outputs)

    def forward(self, x, y, h, len_x, len_y):
        # Encode
        _, h = self._encoder(x, h, len_x)
        # Decode
        o, h = self._decoder(y, h, len_y)
        # Project
        o = self._out(o)

        return F.log_softmax(o)

def load_data(size,
              min_len=5,
              max_len=15,
              min_word=3,
              max_word=100,
              epoch=10,
              batch_size=64,
              pad=0,
              bos=1,
              eos=2):
    src = [
        np.random.randint(min_word, max_word - 1,
                          np.random.randint(min_len, max_len)).tolist()
        for _ in range(size)
    ]
    tgt_in = [[bos] + [xi + 1 for xi in x] for x in src]
    tgt_out = [[xi + 1 for xi in x] + [eos] for x in src]

    def _pad(batch):
        max_len = max(len(x) for x in batch)
        return np.asarray(
            [
                np.pad(
                    x, (0, max_len - len(x)),
                    mode='constant',
                    constant_values=pad) for x in batch
            ],
            dtype=np.int64)

    def _len(batch):
        return np.asarray([len(x) for x in batch], dtype=np.int64)

    for e in range(epoch):
        batch_start = 0

        while batch_start < size:
            batch_end = batch_start + batch_size

            s, ti, to = (src[batch_start:batch_end],
                         tgt_in[batch_start:batch_end],
                         tgt_out[batch_start:batch_end])
            lens, lent = _len(s), _len(ti)

            s, ti, to = _pad(s).T, _pad(ti).T, _pad(to).T

            yield (Variable(torch.LongTensor(s)),
                   Variable(torch.LongTensor(ti)),
                   Variable(torch.LongTensor(to)), lens, lent)

            batch_start += batch_size

def print_sample(x, y, yy):
    x = x.data.numpy().T
    y = y.data.numpy().T
    yy = yy.data.numpy().T

    for u, v, w in zip(x, y, yy):
        print('--------')
        print('S: ', u)
        print('T: ', v)
        print('P: ', w)

n_data = 50
min_len = 5
max_len = 10
vocab_size = 101
n_samples = 5

epoch = 100000
batch_size = 32
lr = 1e-2
clip = 3

emb_size = 50
hidden_size = 50
num_layers = 1
max_length = 15

src_embed = torch.nn.Embedding(vocab_size, emb_size)
tgt_embed = torch.nn.Embedding(vocab_size, emb_size)

eps = 1e-3
src_embed.weight.data.uniform_(-eps, eps)
tgt_embed.weight.data.uniform_(-eps, eps)

enc = Encoder(hidden_size, num_layers, embedding=src_embed)
dec = Decoder(hidden_size, num_layers, embedding=tgt_embed)
net = Seq2Seq(enc, dec, vocab_size)

optimizer = torch.optim.Adam(net.parameters(), lr=lr)
criterion = torch.nn.NLLLoss()

loader = load_data(
    n_data,
    min_len=min_len,
    max_len=max_len,
    max_word=vocab_size,
    epoch=epoch,
    batch_size=batch_size)

for i, (x, yin, yout, lenx, leny) in enumerate(loader):
    net.train()
    optimizer.zero_grad()

    logits = net(x, yin, enc.init_hidden(x.size()[1]), lenx, leny)
    loss = criterion(logits.view(-1, vocab_size), yout.contiguous().view(-1))

    loss.backward()

    torch.nn.utils.clip_grad_norm(net.parameters(), clip)
    optimizer.step()

    if i % 10 == 0:
        print('step: {}, loss: {:.6f}'.format(i, loss.data[0]))

    if i % 200 == 0 and i > 0:
        net.eval()
        x, yin, yout, lenx, leny = (x[:, :n_samples], yin[:, :n_samples],
                                    yout[:, :n_samples], lenx[:n_samples],
                                    leny[:n_samples])
        outputs = net(x, yin, enc.init_hidden(x.size()[1]), lenx, leny)
        _, preds = torch.max(outputs, 2)
        print_sample(x, yout, preds)


title: How to release temporarily consumed GPU memory after each forward?

body:
I have a class like this:

class Stem(nn.Module):

    def __init__(self):
        super(Stem, self).__init__()
        self.out_1 = BasicConv2D(3, 32, kernelSize = 3, stride = 2)
        self.out_2 = BasicConv2D(32, 32, kernelSize = 3, stride = 1)
        self.out_3 = BasicConv2D(32, 64, kernelSize = 3, stride = 1, padding = 1)

    def forward(self, x):
        x = self.out_1(x)
        x = self.out_2(x)
        x = self.out_3(x)

        return x

and the attributes out_1,2,3 of Stem are instances of below class:

class BasicConv2D(nn.Module):

    def __init__(self, inChannels, outChannels, kernelSize, stride, padding = 0):
        super(BasicConv2D, self).__init__()
        self.conv = nn.Conv2d(inChannels, outChannels,
                            kernel_size = kernelSize,
                            stride = stride,
                            padding = padding, bias = False)
        self.bn = nn.BatchNorm2d(outChannels,
                                    eps = 0.001,
                                    momentum = 0.1,
                                    affine = True)
        self.relu = nn.ReLU(inplace = False)

    def forward(self, x):
        x = self.conv(x)
        x = self.bn(x)
        y = self.relu(x)
        return y

when training, within Stem.forward(), nvidia-smi tells that each line will consume x MBs GPU memory, but after Stem.forward() finished, the memory won't be released, causing the training to quickly crashed, out of GPU memory.

The question is thus:
how to release the temporarily consumed GPU memory?


title: Python 3 unable to pickle lambda

body:
I was following a tutorial online about Deep Learning for Self-Driving Cars with pytorch. But then I encounter some problem when python trying to pickle lambda.

I've tried to define my own function but then the result is still the same

Below is where I define my dataset for training.

params = {'batch_size': 32, 'shuffle': True, 'num_workers':4}
training_set = Dataset(train_samples, transform=transdorms.Compose([transforms.Lambda(lambda x: (x/255)-0.5)])
training_generator = DataLoader(training_set, **params)

Then the problem is at the following code.

for local_batch, (centers, lefts, rights) in enumerate(training generator):
        ... Some code

The following error is shown

  _pickle.PicklingError: Can't pickle <function <lambda> at 0x000001B915587948>: attribute lookup  on main failed


title: Gradient Computation broken by Sigmoid function in Pytorch

body:
Hey I have been struggling with this weird problem. Here is my code for the Neural Net:

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv_3d_=nn.Sequential(
            nn.Conv3d(1,1,9,1,4),
            nn.LeakyReLU(),
            nn.Conv3d(1,1,9,1,4),
            nn.LeakyReLU(),
            nn.Conv3d(1,1,9,1,4),
            nn.LeakyReLU()
        )

        self.linear_layers_ = nn.Sequential(

            nn.Linear(batch_size*32*32*32,batch_size*32*32*3),
            nn.LeakyReLU(),
            nn.Linear(batch_size*32*32*3,batch_size*32*32*3),
            nn.Sigmoid()
        )

    def forward(self,x,y,z):
        conv_layer = x + y + z
        conv_layer = self.conv_3d_(conv_layer)
        conv_layer = torch.flatten(conv_layer)
        conv_layer = self.linear_layers_(conv_layer)
        conv_layer = conv_layer.view((batch_size,3,input_sizes,input_sizes))
        return conv_layer

The weird problem I am facing is that running this NN gives me an error

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [3072]], which is output 0 of SigmoidBackward, is at version 1; expected version 0 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

The stack trace shows that the issue is in line

conv_layer = self.linear_layers_(conv_layer)

However, if I replace the last activation function of my FCN from nn.Sigmoid() to nn.LeakyRelu(), the NN executes properly.

Can anyone tell me why Sigmoid activation function is causing my backward computation to break?


title: Batch normalization: converting tensorflow trained weights to pytorch

body:
I'm trying to convert an old tensorflow/keras network I have to pytorch and I'm confused as to the values I obtain of the batch_normalization (BN) weights. For example, the first BN layer of my pytorch network has two sets of elements (weight and bias) of shape [8]. In contrast, the same BN layer in tensorflow returns 4 sets of elements of shape [8].

Since I'm not planning on retraining the network once I've converted it to pytorch, I'd like to know which two sets of elements of the tensorflow BN layer are the corresponding for the pytorch BN layer.


title: Deserialize variable saved on GPU on GPU

body:
I saved the parameter variable on my computer (equipped with GPU and CUDA). The weights were saved with the GPU mode. Then when attempting to load the weights on the same computer, I still get an error:

RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location='cpu' to map your storages to the CPU.

The issue is related to the fact that torch.cuda.is_available() returns False.

The issue disappears after restarting the computer but then reappears after some time.


title: Resizing PyTorch tensor with grad to smaller size

body:
I am trying to downsize a tensor from let's say (3,3) to (1, 1), but I want to keep the original tensor:

import torch

a = torch.rand(3, 3)
a_copy = a.clone()
a_copy.resize_(1, 1)

I need requires_grad=True in my initial tensor but PyTorch forbids this me from trying to resize the copy:

a = torch.rand(3, 3, requires_grad=True)
a_copy = a.clone()
a_copy.resize_(1, 1)

Throws an error:

Traceback (most recent call last):
  File "pytorch_test.py", line 7, in <module>
    a_copy.resize_(1, 1)
RuntimeError: cannot resize variables that require grad

Clone and Detach

I tried to .clone() and .detach() as well:

a = torch.rand(3, 3, requires_grad=True)
a_copy = a.clone().detach()

with torch.no_grad():
    a_copy.resize_(1, 1)

which gives this error instead:

Traceback (most recent call last):
  File "pytorch_test.py", line 14, in <module>
    a_copy.resize_(1, 1)
RuntimeError: set_sizes_contiguous is not allowed on a Tensor created from .data or .detach().
If your intent is to change the metadata of a Tensor (such as sizes / strides / storage / storage_offset)
without autograd tracking the change, remove the .data / .detach() call and wrap the change in a `with torch.no_grad():` block.
For example, change:
    x.data.set_(y)
to:
    with torch.no_grad():
        x.set_(y)

This behaviour had been stated in the docs and #15070.

With no_grad()

So, following what they said in the error message, I removed .detach() and used no_grad() instead:

a = torch.rand(3, 3, requires_grad=True)
a_copy = a.clone()

with torch.no_grad():
    a_copy.resize_(1, 1)

But it still gives me an error about grad:

Traceback (most recent call last):
  File "pytorch_test.py", line 21, in <module>
    a_copy.resize_(1, 1)
RuntimeError: cannot resize variables that require grad

Similar questions

I have looked at Resize PyTorch Tensor but it the tensor in that example retains all original values.
I have also looked at Pytorch preferred way to copy a tensor which is the method I am using to copy the tensor.

I am using PyTorch version 1.4.0


I am using this github repo: https://github.com/vchoutas/smplify-x and I am running a script. I get the following error. How can I fix it? I understand I might have to convert - to ~ however not sure where it exactly is. I don't want to downgrade my PyTorch.
$ cat fit.sh
export CUDA_VISIBLE_DEVICES=0
python smplifyx/main.py --config cfg_files/fit_smplx.yaml --data_folder ../../data/smplify-x/DATA_FOLDER --output_folder ../../data/smplify-x/RESULTS --visualize="False" --model_folder ../../data/smplify-x/models_smplx_v1_1/models/smplx/SMPLX_NEUTRAL.npz --vposer_ckpt ../../data/smplify-x/vposer_v1_0 --part_segm_fn ../../data/smplify-x/smplx_parts_segm.pkl

--------------------------------------

(smplifyx) mona@ubuntu:~/mona/code/smplify-x$ ./fit.sh
Processing: ../../data/smplify-x/DATA_FOLDER/images/woman_bike.jpg
Found Trained Model: ../../data/smplify-x/vposer_v1_0/snapshots/TR00_E096.pt
Traceback (most recent call last):
  File "smplifyx/main.py", line 272, in <module>
    main(**args)
  File "smplifyx/main.py", line 262, in main
    **args)
  File "~/mona/code/smplify-x/smplifyx/fit_single_frame.py", line 274, in fit_single_frame
    focal_length=focal_length, dtype=dtype)
  File "/home/mona/venv/smplifyx/lib/python3.6/site-packages/torch/autograd/grad_mode.py", line 26, in decorate_context
    return func(*args, **kwargs)
  File "~/mona/code/smplify-x/smplifyx/fitting.py", line 75, in guess_init
    pose_embedding, output_type='aa').view(1, -1) if use_vposer else None
  File "../../data/smplify-x/vposer_v1_0/vposer_smpl.py", line 114, in decode
    if output_type == 'aa': return VPoser.matrot2aa(Xout)
  File "../../data/smplify-x/vposer_v1_0/vposer_smpl.py", line 152, in matrot2aa
    pose = tgm.rotation_matrix_to_angle_axis(homogen_matrot).view(batch_size, 1, -1, 3).contiguous()
  File "/home/mona/venv/smplifyx/lib/python3.6/site-packages/torchgeometry/core/conversions.py", line 233, in rotation_matrix_to_angle_axis
    quaternion = rotation_matrix_to_quaternion(rotation_matrix)
  File "/home/mona/venv/smplifyx/lib/python3.6/site-packages/torchgeometry/core/conversions.py", line 302, in rotation_matrix_to_quaternion
    mask_c1 = mask_d2 * (1 - mask_d0_d1)
  File "/home/mona/venv/smplifyx/lib/python3.6/site-packages/torch/tensor.py", line 511, in __rsub__
    return _C._VariableFunctions.rsub(self, other)
RuntimeError: Subtraction, the `-` operator, with a bool tensor is not supported. If you are trying to invert a mask, use the `~` or `logical_not()` operator instead.

I also have:
(smplifyx) mona@ubuntu:~/mona/code/smplify-x$ python
Python 3.6.9 (default, Oct  8 2020, 12:12:24)
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.__version__
'1.7.1'
>>> import torchgeometry
>>> torchgeometry.__version__
'0.1.2'


title: Why Tensorflow multi-GPU training so slow?

body:
Recently I modified my model to train on 2-GPUs on the same machine. But the speed was same as that of one GPU.

The question is:

1.Why backward covers so long time range? Is this the normal case?

2.It seems that there was no performance bottleneck in forward, but why the brown region in the image is so sparse, but the gray region above showed that there was indeed computation on GPU?

3.There are a lot peer-to-peer memcpy after forward, why? What's the operation?BTW, there are not much BN in my model.


title: How to extract position input-output indeces from huggingface transformer text tokenizator?

body:
I want to solve stress prediction task with pretrained russian bert.

Input data looks like this:

  граммов сверху|000100000001000

Zeros mean no stress. Ones represent stress position character.

I want to map it as word -> vowel number index

So it will be like
граммов -> 1
сверху -> 1

So, for each token, it should be a linear layer with softmax.

I understand this part, but it's hard for me to deal with text preprocessing because text tokenizator can split a word into subword tokens.

Tokenizator takes an input string and returns tokens like this

bert_tokenizer.encode('граммов сверху')
->
[101, 44505, 26656, 102]

So, how to get position mapping between input chars and words?

The desired output should be like [[0, 7], [8, 14]]

I tried to read docs
https://huggingface.co/transformers/main_classes/tokenizer.html

And found convert_ids_to_tokens function
It works like

encoded = bert_tokenizer.encode('граммов сверху')
bert_tokenizer.convert_ids_to_tokens(encoded)
->
['[CLS]', 'граммов', 'сверху', '[SEP]']

But I'm not sure how to use original string and stress indices to calculate stress position number for token.


title: Understanding PyTorch implementation

body:
So I came across this code:
import torch.nn.functional as F
loss_cls = F.binary_cross_entropy_with_logits(input, target)

I wanted to see more about the binary_cross_entropy_with_logits function which is a sum of logs, so I head over to the documentation here which leads me to the source code here
All this does is return torch.binary_cross_entropy_with_logits(input, target, weight, pos_weight, reduction_enum)
I want to see the actual code where the sum of logs is being performed. Where can I see the source code for torch.binary_cross_entropy_with_logits