title: Pytorch Siamese Network not converging

body:
Good morning everyone
Below is my implementation of a pytorch siamese network. I am using 32 batch size, MSE loss and SGD with 0.9 momentum as optimizer.
class SiameseCNN(nn.Module):
    def __init__(self):
        super(SiameseCNN, self).__init__()                                      # 1, 40, 50
        self.convnet = nn.Sequential(nn.Conv2d(1, 8, 7), nn.ReLU(),             # 8, 34, 44
                                    nn.Conv2d(8, 16, 5), nn.ReLU(),             # 16, 30, 40
                                    nn.MaxPool2d(2, 2),                         # 16, 15, 20
                                    nn.Conv2d(16, 32, 3, padding=1), nn.ReLU(), # 32, 15, 20
                                    nn.Conv2d(32, 64, 3, padding=1), nn.ReLU()) # 64, 15, 20
        self.linear1 = nn.Sequential(nn.Linear(64 * 15 * 20, 100), nn.ReLU())
        self.linear2 = nn.Sequential(nn.Linear(100, 2), nn.ReLU())

    def forward(self, data):
        res = []
        for j in range(2):
            x = self.convnet(data[:, j, :, :])
            x = x.view(-1, 64 * 15 * 20)
            res.append(self.linear1(x))
        fres = abs(res[1] - res[0])
        return self.linear2(fres)

Each batch contains alternating pairs, i.e [pos, pos], [pos, neg], [pos, pos] etc... However, the network doesn't converge, and the problem seems that fres in the network is the same for each pair (regardless of whether it is a positive or negative pair), and the output of self.linear2(fres) is always approximately equal to [0.0531, 0.0770]. This is in contrast with what I am expecting, which is that the first value of [0.0531, 0.0770] would get closer to 1 for a positive pair as the network learns, and the second value would get closer to 1 for a negative pair. These two values also need to sum up to 1.
I have tested exactly the same setup and same input images for a 2 channel network architecture, where, instead of feeding in [pos, pos] you would stack those 2 images in a depth-wise fashion, for example numpy.stack([pos, pos], -1). The dimension of nn.Conv2d(1, 8, 7) also changes to nn.Conv2d(2, 8, 7) in this setup. This works perfectly fine.
I have also tested exactly the same setup and input images for a traditional CNN approach, where I just pass in single positive and negative grey scale images into the network, instead of stacking them (as with the 2-CH approach) or passing them in as image pairs (as with the Siamese approach). This also works perfectly, but the results are not so good as with the 2 channel approach.
EDIT (Solutions I've tried):

I have tried a number of different loss functions, including HingeEmbeddingLoss and CrossEntropyLoss, all resulting in more or less the same problem. So I think it is safe to say that the problem is not caused by the employed loss function; MSELoss.
Different batch sizes also seem to have no effect on the issue.
I tried increasing the number of trainable parameters as suggested in
Keras Model for Siamese Network not Learning and always predicting the same ouput
Also doesn't work.
Tried to change the network architecture as implemented here: https://github.com/benmyara/pytorch-examples/blob/master/notebooks/1_NeuralNetworks/9_siamese_nn.ipynb. In other words, changed the forward pass to the following code. Also changed the loss to CrossEntropy, and the optimizer to Adam. Still no luck:

def forward(self, data):
    res = []
    for j in range(2):
        x = self.convnet(data[:, j, :, :])
        x = x.view(-1, 64 * 15 * 20)
        res.append(x)
    fres = self.linear2(self.linear1(abs(res[1] - res[0]))))
    return fres

I also tried to change the whole network from a CNN to a linear network as implemented here: https://github.com/benmyara/pytorch-examples/blob/master/notebooks/1_NeuralNetworks/9_siamese_nn.ipynb. Still doesn't work.
Tried to use a lot more data as suggested here: Keras Model for Siamese Network not Learning and always predicting the same ouput. No luck...
Tried to use torch.nn.PairwiseDistance between the outputs of convnet. Made some sort of improvement; the network starts to converge for the first few epochs, and then hits the same plateau everytime:

def forward(self, data):
    res = []
    for j in range(2):
        x = self.convnet(data[:, j, :, :])
        res.append(x)
    pdist = nn.PairwiseDistance(p=2)
    diff = pdist(res[1], res[0])
    diff = diff.view(-1, 64 * 15 * 10)
    fres = self.linear2(self.linear1(diff))
    return fres

Another thing to note perhaps is that, within the context of my research, a Siamese network is trained for each object. So the first class is associated with the images containing the object in question, and the second class is associated with images containing other objects. Don't know if this might be the cause of the problem. It is however not a problem within the context of the Traditional CNN and 2-Channel CNN approaches.
As per request, here is my training code:
model = SiameseCNN().cuda()
ls_fn = torch.nn.BCELoss()
optim = torch.optim.SGD(model.parameters(),  lr=1e-6, momentum=0.9)
epochs = np.arange(100)
eloss = []
for epoch in epochs:
    model.train()
    train_loss = []
    for x_batch, y_batch in dp.train_set:
        x_var, y_var = Variable(x_batch.cuda()), Variable(y_batch.cuda())
        y_pred = model(x_var)
        loss = ls_fn(y_pred, y_var)
        train_loss.append(abs(loss.item()))
        optim.zero_grad()
        loss.backward()
        optim.step()
    eloss.append(np.mean(train_loss))
    print(epoch, np.mean(train_loss))

Note dp in dp.train_set is a class with attributes train_set, valid_set, test_set, where each set is created as follows:
DataLoader(TensorDataset(torch.Tensor(x), torch.Tensor(y)), batch_size=bs)

As per request, here is an example of the predicted probabilities vs true label, where you can see the model doesn't seem to be learning:
Predicted:  0.5030623078346252 Label:  1.0
Predicted:  0.5030624270439148 Label:  0.0
Predicted:  0.5030624270439148 Label:  1.0
Predicted:  0.5030625462532043 Label:  0.0
Predicted:  0.5030625462532043 Label:  1.0
Predicted:  0.5030626654624939 Label:  0.0
Predicted:  0.5030626058578491 Label:  1.0
Predicted:  0.5030627250671387 Label:  0.0
Predicted:  0.5030626654624939 Label:  1.0
Predicted:  0.5030627846717834 Label:  0.0
Predicted:  0.5030627250671387 Label:  1.0
Predicted:  0.5030627846717834 Label:  0.0
Predicted:  0.5030627250671387 Label:  1.0
Predicted:  0.5030628442764282 Label:  0.0
Predicted:  0.5030627846717834 Label:  1.0
Predicted:  0.5030628442764282 Label:  0.0


title: Graph Neural Network with Nodes as Input and Edges as Output in DGL

body:
I would like to adapt the example DGL GATLayer such that instead of learning node representations, the network can learn the edge weights. That is, I want to to build a network that takes a set of node features as input and outputs the edges. The labels will be a set of "truth edges", which represent which nodes come from a common source, such that I can learn to cluster unseen data in the same way.
I am using as a starting point the code from the following DGL example:
https://www.dgl.ai/blog/2019/02/17/gat.html
import torch.nn as nn
import torch.nn.functional as F

class GATLayer(nn.Module):
    def __init__(self, g, in_dim, out_dim):
        super(GATLayer, self).__init__()
        self.g = g
        # equation (1)
        self.fc = nn.Linear(in_dim, out_dim, bias=False)
        # equation (2)
        self.attn_fc = nn.Linear(2 * out_dim, 1, bias=False)

    def edge_attention(self, edges):
        # edge UDF for equation (2)
        z2 = torch.cat([edges.src['z'], edges.dst['z']], dim=1)
        a = self.attn_fc(z2)
        return {'e' : F.leaky_relu(a)}

    def message_func(self, edges):
        # message UDF for equation (3) & (4)
        return {'z' : edges.src['z'], 'e' : edges.data['e']}

    def reduce_func(self, nodes):
        # reduce UDF for equation (3) & (4)
        # equation (3)
        alpha = F.softmax(nodes.mailbox['e'], dim=1)
        # equation (4)
        h = torch.sum(alpha * nodes.mailbox['z'], dim=1)
        return {'h' : h}

    def forward(self, h):
        # equation (1)
        z = self.fc(h)
        self.g.ndata['z'] = z
        # equation (2)
        self.g.apply_edges(self.edge_attention)
        # equation (3) & (4)
        self.g.update_all(self.message_func, self.reduce_func)
        return self.g.ndata.pop('h')

class MultiHeadGATLayer(nn.Module):
    def __init__(self, g, in_dim, out_dim, num_heads, merge='cat'):
        super(MultiHeadGATLayer, self).__init__()
        self.heads = nn.ModuleList()
        for i in range(num_heads):
            self.heads.append(GATLayer(g, in_dim, out_dim))
        self.merge = merge

    def forward(self, h):
        head_outs = [attn_head(h) for attn_head in self.heads]
        if self.merge == 'cat':
            # concat on the output feature dimension (dim=1)
            return torch.cat(head_outs, dim=1)
        else:
            # merge using average
            return torch.mean(torch.stack(head_outs))

class GAT(nn.Module):
    def __init__(self, g, in_dim, hidden_dim, out_dim, num_heads):
        super(GAT, self).__init__()
        self.layer1 = MultiHeadGATLayer(g, in_dim, hidden_dim, num_heads)
        # Be aware that the input dimension is hidden_dim*num_heads since
        #   multiple head outputs are concatenated together. Also, only
        #   one attention head in the output layer.
        self.layer2 = MultiHeadGATLayer(g, hidden_dim * num_heads, out_dim, 1)

    def forward(self, h):
        h = self.layer1(h)
        h = F.elu(h)
        h = self.layer2(h)
        return h

I had hoped I could adapt this to simply return the edges instead of the nodes, eg by replacing the line
return self.g.ndata.pop('h')
with
return self.e.ndata.pop('e')
But it seems it is not this simple. I managed to get something to run, but the loss jumped around all over the place and no learning occurred.
I am new to graph networks, though not to deep learning in general. Is what I am trying to do a reasonable thing? Am I missing something crucial in my understanding of how this works? I have been unable to find any easy to understand examples of graph networks where the edges themselves are the learning objective, so I'm a bit muddled at the moment. I appreciate any help that anyone can give!


title: How can I load a dataset into a pytorch torchtext.data.TabularDataset from a json having list of dicts?

body:
I have a list of dicts as follows:
[{'text': ['The', 'Fulton', 'County', 'Grand', ...], 'tags': ['AT', 'NP-TL', 'NN-TL', 'JJ-TL', ...]},
 {'text': ['The', 'jury', 'further', 'said', ...], 'tags': ['AT', 'NN', 'RBR', 'VBD', ...]},
 ...]

Each value of each dict is a list of sentence-words/tags. This is directly from the Brown corpus of the NLTK dataset, loaded using:
from nltk.corpus import brown
data = brown.tagged_sents()
data = {'text': [[word for word, tag in sent] for sent in data], 'tags': [[tag for word, tag in sent] for sent in data]}

import pandas as pd
df = pd.DataFrame(training_data, columns=["text", "tags"])

from sklearn.model_selection import train_test_split
train, val = train_test_split(df, test_size=0.2)
train.to_json("train.json", orient='records')
val.to_json("val.json", orient='records')

I want to load this json into a torchtext.data.TabularDataset using:
TEXT = data.Field(lower=True)
TAGS = data.Field(unk_token=None)

data_fields = [('text', TEXT), ('tags', TAGS)]
train, val = data.TabularDataset.splits(path='./', train='train.json', validation='val.json', format='json', fields=data_fields)

But it gives me this error:
/usr/local/lib/python3.6/dist-packages/torchtext/data/example.py in fromdict(cls, data, fields)
     17     def fromdict(cls, data, fields):
     18         ex = cls()
---> 19         for key, vals in fields.items():
     20             if key not in data:
     21                 raise ValueError("Specified key {} was not found in "

AttributeError: 'list' object has no attribute 'items'

Note that I don't want TabularDataset to tokenize the sentence for me as it is already tokenized by nltk. How do I approach this? (I cannot switch corpuses to something I can directly load from torchtext.dataset, I have to use the Brown Corpus)


title: Channels dimension index in the input shape while porting Pytorch models to Tensorflow

body:
One of the major problems I've encountered when converting PyTorch models to TensorFlow through ONNX, is slowness, which appears to be related to the input shape, even though I was able to get bit-exact outputs with the two frameworks.

While the PyTorch input shape is B,C,H,W, the Tensorflow input shape is B,H,W,C, where B,C,H,W stand for batch size, channels, height and width, respectively. Technically, I solve the input shape problem easily when working in Tensorflow, using two calls to np.swapaxes:

# Single image, no batch size here yet
image = np.swapaxes(image, 0, 2)   # Swapping C and H dimensions - result: C,W,H
image = np.swapaxes(image, 1, 2)   # Swapping H and W dimensions - result: C,H,W (like Pytorch)

The slowness problem seems to be related to the differences in the ways the convolutional operations are implemented in PyTorch vs Tensorflow. While PyTorch expects channels first, Tensorflow expects channels last.

As a result, when I visualize the models using Netron, the ONNX model looks abstract and making sense (first image), whereas the Tensorflow .pb formatted model looks like a big mess (second image).

Note: It appears that this problem has already concerned the writers of the onnx2keras library, which supports an experimental feature of changing the C,H,W ordering originated in Pytorch, into H,W,C.

Any idea how to overcome this limitation? Are there other options for more abstractly exporting PyTorch models into Tensorflow?

ONNX (from PyTorch) - you can see the straight flow and the residual blocks:

Tensorflow (imported from the ONNX model) - almost nothing looks like a series of predefined operations:


title: Pytorch transformers memory leak in inference

body:
I am using Pytorch and a pretrained model from the transformers library. However, while finetuning it runs out of GPU memory very quickly and I wonder why.
I've found out that there is a memory leak in forward pass.
show_memory('before call')
outputs = model(inputs, masked_lm_labels=labels) if args.mlm else model(inputs, labels=labels)
show_memory('after call')

del outputs
gc.collect()
model.zero_grad()
torch.cuda.empty_cache()
show_memory('after clearing')

Here show_memory function is defined as following:
def show_memory(text=''):
  t = torch.cuda.get_device_properties(0).total_memory
  c = torch.cuda.memory_cached(0)
  a = torch.cuda.memory_allocated(0)
  f = c-a  # free inside cache
  print(f'\n\n{text}\nTotal: {t}\nCached: {c} \nAllocated: {a} \nFree in cache: {f}\n\n')

And the output is
before call
Total: 17071734784
Cached: 3441426432
Allocated: 3324039680
Free in cache: 117386752

after call
Total: 17071734784
Cached: 6830424064
Allocated: 6720267264
Free in cache: 110156800

after clearing
Total: 17071734784
Cached: 6109003776
Allocated: 4876882944
Free in cache: 1232120832

So despite I clear everything, there are still almost two gigabytes more data in memory than there was originally. I wonder why it happens and if there is a way to inspect such cases in detail.


title: Pytorch errors when given numpy integer types only in python 3 (not in python 2)

body:
For example, the torch.randn function, among others, gets mad when given a numpy.int64 type:

Python 3.5.5 |Anaconda custom (64-bit)| (default, Mar 12 2018, 23:12:44)
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> import numpy
>>> torch.randn(numpy.int64(4))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: torch.randn received an invalid combination of arguments - got (numpy.int64), but expected one of:
 * (int ... size)
      didn't match because some of the arguments have invalid types: (numpy.int64)
 * (torch.Size size)
      didn't match because some of the arguments have invalid types: (numpy.int64)
 * (torch.Generator generator, int ... size)
 * (torch.Generator generator, torch.Size size)

But in python 2, this works just fine:

Python 2.7.14 |Anaconda, Inc.| (default, Dec  7 2017, 17:05:42)
[GCC 7.2.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> import numpy
>>> torch.randn(numpy.int64(3))

-2.0513
 0.5409
-0.0814
[torch.FloatTensor of size 3]

I couldn't find anyone else running into this issue. Is this known? Is this something about my setup? Is there any way to work around this without completely dropping numpy?

I'm using version 0.3.1 of pytorch and version 1.14.2 of numpy.


title: What is happening with torch.Tensor.add_?

body:
I'm looking at this implementation of SGD for PyTorch: https://pytorch.org/docs/stable/_modules/torch/optim/sgd.html#SGD

And I see some strange calculations which I don't understand.

For instance, take a look at p.data.add_(-group['lr'], d_p). It makes sense to think that there is a multiplication of the two parameters, right? (It's how SGD works, -lr * grads)

But the documentation of the function doesn't say anything about this.

And what is more confusing, although this SGD code actually works (I tested by copying the code and calling prints below the add_), I can't simply use add_ with two arguments as it does:

#this returns an error about using too many arguments
import torch

a = torch.tensor([1,2,3])
b = torch.tensor([6,10,15])
c = torch.tensor([100,100,100])
a.add_(b, c)
print(a)

What's going on here? What am I missing?


title: How to use pytorch to construct multi-task DNN, e.g., for more than 100 tasks?

body:
Below is the example code to use pytorch to construct DNN for two regression tasks. The forward function returns two outputs (x1, x2). How about the network for lots of regression/classification tasks? e.g., 100 or 1000 outputs. It definitely not a good idea to hardcode all the outputs (e.g., x1, x2, ..., x100). Is there an simple method to do that? Thank you.

import torch
from torch import nn
import torch.nn.functional as F

class mynet(nn.Module):
    def __init__(self):
        super(mynet, self).__init__()
        self.lin1 = nn.Linear(5, 10)
        self.lin2 = nn.Linear(10, 3)
        self.lin3 = nn.Linear(10, 4)

    def forward(self, x):
        x = self.lin1(x)
        x1 = self.lin2(x)
        x2 = self.lin3(x)
        return x1, x2

if __name__ == '__main__':
    x = torch.randn(1000, 5)
    y1 = torch.randn(1000, 3)
    y2 = torch.randn(1000,  4)
    model = mynet()
    optimizer = torch.optim.Adam(model.parameters(), lr=0.001, weight_decay=1e-4)
    for epoch in range(100):
        model.train()
        optimizer.zero_grad()
        out1, out2 = model(x)
        loss = 0.2 * F.mse_loss(out1, y1) + 0.8 * F.mse_loss(out2, y2)
        loss.backward()
        optimizer.step()


title: How to specify the input dimension of pytorch nn.Linear?

body:
For example, I have defined  a model shown below:
class Net(nn.module):
   def __init__():
      self.conv11 = nn.Conv2d(in_channel,out1_channel,3)
      self.conv12 = nn.Conv2d(...)
      self.conv13 = nn.Conv2d(...)
      self.conv14 = nn.Conv2d(...)
      ...
      #Here is the point
      flat = nn.Flatten()
      #I don't want to compute the size of data after flatten, but I need a linear layer.

      fc_out = nn.Linear(???,out_dim)

The problem is the Linear layer, I don't want to compute the size of the input to the Linear layer, but the defining model needs to specify it. How can I solve this problem?


title: Trying to download the CIFAR10 dataset using 'torchvision.datasets' on my computer

body:
i am trying to download the CIFAR10 dataset on my computer using the last two lines of the following code:
import torch
import torchvision
import torchvision.transforms as transforms

transform = transforms.Compose(
[transforms.ToTensor(),
 transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])
trainset = torchvision.datasets.CIFAR10(root='./Users/Sohrab/Downloads', train=True,
                                    download=True, transform=transform)

I execute it in Google Colab,but after that,when i look in my Downloads file on my computer,i do not see
the CIFAR10 dataset.
Is it actually stored on my computer or does Colab store it somewhere else(in a cloud architecture for instance)?
Thank you in advance.


title: Make sure BERT model does not load pretrained weights?

body:
I want to make sure my BertModel does not loads pre-trained weights. I am using auto class (hugging face) which loads model automatically.
My question is how do I load bert model without pretrained weights?


title: Setup Egg Error when trying to install Torchvision | Python and Anaconda

body:
I'm trying to install PyTorch from: https://pytorch.org/

At this point, I've tried so many things and don't even remember, but I believe that I was finally able to get the first command

pip3 install http://download.pytorch.org/whl/cu80/torch-0.4.1-cp36-cp36m-win_amd64.whl

(I'm not sure if this is the one I eventually used) to work. I tried literally every windows combination and went through Github problems/comments to try and figure out what my issues were and how to fix it.

Then I tried to do the

pip3 install torchvision

which errored out and can be viewed below:

I searched many different links:

https://github.com/GoogleCloudPlatform/google-cloud-python/issues/3884

https://github.com/facebook/prophet/issues/418

https://github.com/donnemartin/gitsome/issues/4

as some examples, regarding the issue. All suggested I use some variation of

pip install --upgrade setuptools

which I have done. This hasn't helped my issue.

I'm using Windows 10, 64-bit

Python Version: 3.6.1

Anaconda Version: 4.2.0

I have updated my pip, numpy, and a lot of other things. At this point, I'm very confused. I appreciate the help.

Edit: At this point, like I said I don't even remember what I did to try and get the first command to work. running them all now I get the following:

Windows | Pip | 3.6 | 8 : torch-0.4.1-cp36-cp36m-win_amd64.whl is not a supported wheel on this platform.

Windows | Pip | 3.6 | 9.0 : same

Windows | Pip | 3.6 | 9.2 : same

Windows | Pip | 3.6 | None : same

Running any of the conda commands gives me the same:

This is solved, I uninstalled my Anaconda and redownloaded it. Possible that I had a 32-bit, and it's now a corrected 64-bit.


title: Add channel to MNIST via transform?

body:
I'm trying to use the MNIST dataset from torchvision.datasets.It seems to be provided as an N x H x W (uint8) (batch dimension, height, width) tensor. All the pytorch classes for working on images (for instance Conv2d) however require a N x C x H x W (float32) tensor where C is the number of colour channels. I've tried to add add the ToTensor transform but that didn't add a color channel.

Is there a way using torchvision.transforms to add this additional dimension? For a raw tensor we could just do .unsqueeze(1) but that doesn't look like a very elegant solution. I'm just trying to do it the "proper" way.

Here is the failed conversion.

import torchvision
dataset = torchvision.datasets.MNIST("~/PyTorchDatasets/MNIST/", train=True, transform=torchvision.transforms.ToTensor(), download=True)
print(dataset.train_data[0])


title: PyTorch multiprocessing error with Hogwild

body:
I've encountered a mysterious bug while trying to implement Hogwild with torch.multiprocessing. In particular, one version of the code runs fine, but when I add in a seemingly unrelated bit of code before the multiprocessing step, this somehow causes an error during the multiprocessing step: RuntimeError: Unable to handle autograd's threading in combination with fork-based multiprocessing. See https://github.com/pytorch/pytorch/wiki/Autograd-and-Fork
I reproduced the error in a minimal code sample, pasted below. If I comment out the two lines of code m0 = Model(); train(m0) which carry out a non-parallel training run on a separate model instance, then everything runs fine. I can't figure out how these lines could be causing a problem.
I'm running PyTorch 1.5.1 and Python 3.7.6 on a Linux machine, training on CPU only.
import torch
import torch.multiprocessing as mp
from torch import nn

def train(model):
    opt = torch.optim.Adam(model.parameters(), lr=1e-5)
    for _ in range(10000):
        opt.zero_grad()
        # We train the model to output the value 4 (arbitrarily)
        loss = (model(0) - 4)**2
        loss.backward()
        opt.step()

# Toy model with one parameter tensor of size 3.
# Output is always the sum of the elements in the tensor,
# independent of the input
class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.x = nn.Parameter(torch.ones(3))

    def forward(self, x):
        return torch.sum(self.x)

############################################
# Create a separate Model instance and run
# a non-parallel training run.
# For some reason, this code causes the
# subsequent parallel run to fail.
m0 = Model()
train(m0)
print ('Done with preliminary run')
############################################

num_processes = 2
model = Model()
model.share_memory()
processes = []
for rank in range(num_processes):
    p = mp.Process(target=train, args=(model,))
    p.start()
    processes.append(p)
for p in processes:
    p.join()

print(model.x)


title: Detectron2- How to log validation loss during training?

body:
I copied the idea from mnslarcher and wrote the following two functions for my keypoint detector (resnet50 backbone) algorithm.
def build_valid_loader(cfg):
    _cfg = cfg.clone()
    _cfg.defrost()  # make this cfg mutable.
    _cfg.DATASETS.TRAIN = cfg.DATASETS.TEST
    return build_detection_train_loader(_cfg)

def store_valid_loss(model, data, storage):
    training_mode = model.training
    with torch.no_grad():
        loss_dict = model(data)
        losses = sum(loss_dict.values())
        assert torch.isfinite(losses).all(), loss_dict

        loss_dict_reduced = {k: v.item()
                                for k, v in comm.reduce_dict(loss_dict).items()}
        losses_reduced = sum(loss for loss in loss_dict_reduced.values())

        if comm.is_main_process():
            storage.put_scalars(val_loss=losses_reduced, **loss_dict_reduced)
    model.train(training_mode)

then in plain_train_net.py I am calling them as bellow.
    val_data_loader = build_valid_loader(cfg)
    logger.info("Starting training from iteration {}".format(start_iter))
    with EventStorage(start_iter) as storage:
        for data, val_data, iteration in zip(data_loader, val_data_loader, range(start_iter, max_iter)):
            iteration = iteration + 1
            ..
            ..
           #At the end of the for loop.
           # Calculate and log validation loss.
            store_valid_loss(model, val_data, storage)

after 1k iteration, loss_keypoint is increasing, but total_loss is same compared to without store_valid_loss call. What am I missing? Can anyone please help to understand?
I am using 4 GeForce RTX 2080 Ti.


title: Pytorch register_hook to Keras implementation

body:
Im trying to implement the following project into Tensorflow/Keras.
https://github.com/jacobgil/pytorch-pruning

Im having a hard time understanding what register_hook does? It can be found in finetune.py, row 66.
x.register_hook(self.compute_rank)

I've searched for clear explanations regarding this function and tried to find Keras-equivalents, without any luck. Do you have answers to these questions?


title: How to convert binary classifier output/loss tensor in LSTM to multiclass

body:
I am trying to train an LSTM model to predict what year a song was written given its lyrics using word-level association in Pytorch. There are 51 potential classes/labels (1965-2015) - however I was working off of a template that used a binary classifier for a different problem. I have been trying to figure out how to change the model to predict multiple classes (1965, 1966, etc).

I understand that you are supposed to provide a tensor of size C=num_classes as the output. However, I did that by making output_size=51 but I am getting an error, which makes me think there is something related to defining or operating on the criterion class I am defining that I am not doing correctly.

Here is the model:

class LyricLSTM(nn.Module):
    def __init__(self, vocab_size, output_size, embedding_dim, hidden_dim, n_layers, drop_prob=0.5):
        super().__init__()

        self.output_size = output_size
        self.n_layers = n_layers
        self.hidden_dim = hidden_dim

        # embedding and LSTM layers
        self.embedding = nn.Embedding(vocab_size, embedding_dim)
        self.lstm = nn.LSTM(embedding_dim, hidden_dim, n_layers,
                            dropout=drop_prob, batch_first=True)

        # dropout layer
        self.dropout = nn.Dropout(0.3)

        # linear and sigmoid layers
        self.fc = nn.Linear(hidden_dim, output_size)
        self.sig = nn.Sigmoid()
        #self.softmax = nn.LogSoftmax(dim=1)

    def forward(self, x, hidden):
        batch_size = x.size(0)

        # embeddings and lstm_out
        embeds = self.embedding(x)
        lstm_out, hidden = self.lstm(embeds, hidden)

        # stack up lstm outputs
        lstm_out = lstm_out.contiguous().view(-1, self.hidden_dim)

        # dropout and fully-connected layer
        out = self.dropout(lstm_out)
        out = self.fc(out)
        # sigmoid function
        sig_out = self.sig(out)
        #sig_out = self.softmax(out)

        # reshape to be batch_size first
        sig_out = sig_out.view(batch_size, -1)
        sig_out = sig_out[:, -1]  # get last batch of labels

        # return last sigmoid output and hidden state
        return sig_out, hidden

    def init_hidden(self, batch_size):
        ''' Initializes hidden state '''
        # Create two new tensors with sizes n_layers x batch_size x hidden_dim,
        # initialized to zero, for hidden state and cell state of LSTM
        weight = next(self.parameters()).data

        hidden = (weight.new(self.n_layers, batch_size, self.hidden_dim).zero_(),
                  weight.new(self.n_layers, batch_size, self.hidden_dim).zero_())

        return hidden

And the training loop:

n_epochs = 10
batch_size = 16 #100  # 11 batches of size 337 so iters = 11 (11 * 337 = 3707)

# Split into training, validation, testing - train= 80% | valid = 10% | test = 10%
split_frac = 0.8
train_x = encoded_lyrics[0:int(split_frac * len(encoded_lyrics))] # 3707 training samples
train_y = encoded_years[0:int(split_frac * len(encoded_lyrics))]  # 3707 training samples

# Dataloaders and batching
# create Tensor datasets
train_data = TensorDataset(torch.from_numpy(train_x), torch.from_numpy(train_y))

# make sure to SHUFFLE your data
train_loader = DataLoader(train_data, shuffle=True, batch_size=batch_size, drop_last=True)

output_size = 51
embedding_dim = 400
hidden_dim = 128 #256
n_layers = 2
lstmc = lstm.LyricLSTM(vocab_len, output_size, embedding_dim, hidden_dim, n_layers)

# Loss function + accuracy reporting
current_loss = 0
losses = np.zeros(n_epochs)  # For plotting
accuracy = np.zeros(n_epochs)

lr = 0.001
criterion = nn.CrossEntropyLoss() #nn.BCELoss()
optimizer = torch.optim.Adam(lstmc.parameters(), lr=lr)
counter = 0
print_every = 1
clip = 5  # gradient clipping

# Main training loop
start = time.time()
lstmc.train()
for epoch in range(0, n_epochs):
    # initialize hidden state
    h = lstmc.init_hidden(batch_size)

    # batch loop
    for inputs, labels in train_loader:
        counter += 1

        # Creating new variables for the hidden state, otherwise
        # we'd backprop through the entire training history
        h = tuple([each.data for each in h])

        # zero accumulated gradients
        lstmc.zero_grad()

        # get the output from the model
        inputs = inputs.type(torch.LongTensor)
        output, h = lstmc(inputs, h)

        # calculate the loss and perform backprop
        loss = criterion(output.squeeze(), labels.float())
        loss.backward()

        nn.utils.clip_grad_norm_(lstmc.parameters(), clip)
        optimizer.step()

I am getting this error when I run the code

File "main.py", line 182, in main
    loss = criterion(output.squeeze(), labels.float())
/venv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
/venv/lib/python3.7/site-packages/torch/nn/modules/loss.py", line 904, in forward
    ignore_index=self.ignore_index, reduction=self.reduction)
/venv/lib/python3.7/site-packages/torch/nn/functional.py", line 1970, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
/venv/lib/python3.7/site-packages/torch/nn/functional.py", line 1295, in log_softmax
    ret = input.log_softmax(dim)
RuntimeError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

This is the output I am getting and the labels (for batch size 16):

Output: tensor([0.4962, 0.5025, 0.4963, 0.4936, 0.5058, 0.4872, 0.4995, 0.4852, 0.4840,
        0.4791, 0.4984, 0.5034, 0.4796, 0.4826, 0.4811, 0.4859],
       grad_fn=<SqueezeBackward0>)

Labels: tensor([1994., 1965., 1981., 1986., 1973., 1981., 1975., 1968., 1981., 1968.,
        1989., 1981., 1988., 1991., 1983., 1982.])

I was expecting for the output to be a tensor of length 51 where each element contained the likelihood that that year was the correct answer (ex: output[0] = first year / 1965, output[1] = 1966, etc).


title: using huggingface's pytorch- transformers GPT-2 for classifcation tasks

body:
I want to use GPT-2 to make a text classifier model. I am not really sure what head should I add after I extracted features through the GPT-2. for eample I have a sequence.

import pytorch_transformers as pt
import torch
text=test.iloc[1,1]
text
'If a fire wanted fanning, it could readily be fanned with a newspaper, and as the government grew weaker, I have no doubt that leather and iron acquired durability in proportion, for, in a very short time, there was not a pair of bellows in all Rotterdam that ever stood in need of a stitch or required the assistance of a hammer.'
len(text)

74
tokenizer = pt.GPT2Tokenizer.from_pretrained('gpt2')
model = pt.GPT2Model.from_pretrained('gpt2')
zz = tokenizer.tokenize(text)
z1=torch.tensor([tokenizer.convert_tokens_to_ids(zz)])
z1
tensor([[ 1532,   257,  2046,  2227,  4336,   768,    11,   340,   714, 14704,
           307,   277,  3577,   351,   257,  7533,    11,   290,   355,   262,
          1230,  6348, 17642,    11,   314,   423,   645,  4719,   326, 11620,
           290,  6953,  9477, 26578,   287,  9823,    11,   329,    11,   287,
           257,   845,  1790,   640,    11,   612,   373,   407,   257,  5166,
           286,  8966,  1666,   287,   477, 18481,   353, 11043,   326,  1683,
          6204,   287,   761,   286,   257, 24695,   393,  2672,   262,  6829,
           286,   257, 15554,    13]])
output,hidden=model(z1)
ouput.shape
torch.Size([1, 74, 768])

the output of GPT2 is n x m x 768 for me, which n is the batch size,m is the number of tokens in the seqence(for example I can pad/truncate to 128.), so I can not do what as the paper said for a classification task just add a fully connected layer in the tail.And I searched on google, few GPT-2 classification task is mensioned.
I am not sure what is correct. Should I do flatten/max pooling/average pooling before the  fully connected layer or something else?


title: Splitting ImageFolder into train and validation datasets

body:
I have loaded my dataset as follows :

full_dataset = ImageFolder(root = os.path.join(root, 'train'), transform=train_transforms)

Now to split my dataset into training and validation sets, I used the following code :

train_size = int(0.8 * len(full_dataset))
validation_size = len(full_dataset) - train_size
train_dataset, validation_dataset = random_split(full_dataset, [train_size, validation_size])

Both train_dataset and validation_dataset are of type : torch.utils.data.dataset.Subset. Is there any way to convert these datasets into torchvision.datasets.folder.ImageFolder.
I need to do this as I am not able to iterate through datasets of type torch.utils.data.dataset.Subset


title: Training 1D CNN in Pytorch

body:
I want to train the model given below. I am developing 1D CNN model in PyTorch. Usually we use dataloaders in PyTorch. But I am not using dataloaders for my implementation. I need guidance on how i can train my model in pytorch.

import torch
import torch.nn as nn
import torch.nn.functional as F

class  CharCNN(nn.Module):
def __init__(self,num_labels=11):
    super(CharCNN, self).__init__()

    self.conv1 = nn.Sequential(
        nn.Conv1d(num_channels, depth_1, kernel_size=kernel_size_1, stride=stride_size),
        nn.ReLU(),
        nn.MaxPool1d(kernel_size=kernel_size_1, stride=stride_size),
        nn.Dropout(0.1),
    )

    self.conv2 = nn.Sequential(
        nn.Conv1d(depth_1, depth_2, kernel_size=kernel_size_2, stride=stride_size),
        nn.ReLU(),
        nn.MaxPool1d(kernel_size=kernel_size_2, stride=stride_size),
        nn.Dropout(0.25)
    )

    self.fc1 = nn.Sequential(
        nn.Linear(depth_2*kernel_size_2, num_hidden),
        nn.ReLU(),
        nn.Dropout(0.5)
    )

    self.fc2 = nn.Sequential(
        nn.Linear(num_hidden, num_labels),
        nn.ReLU(),
        nn.Dropout(0.5)
    )

def forward(self, x):
    out = self.conv1(x)
    out = self.conv2(out)

    # collapse
    out = x.view(x.size(0), -1)
    # linear layer
    out = self.fc1(out)
    # output layer
    out = self.fc2(out)
    #out = self.log_softmax(x,dim=1)

    return out

I am training my network like this:

criterion = nn.CrossEntropyLoss()
opt = torch.optim.Adam(model.parameters(),lr=learning_rate)

for e in range(training_epochs):
  if(train_on_gpu):
     net.cuda()
    train_losses = []

   for batch in iterate_minibatches(train_x, train_y, batch_size):
      x, y = batch
      inputs, targets = torch.from_numpy(x), torch.from_numpy(y)
      if(train_on_gpu):
        inputs, targets = inputs.cuda(), targets.cuda()
      opt.zero_grad()
      output = model(inputs, batch_size)

    loss = criterion(output, targets.long())
    train_losses.append(loss.item())
    loss.backward()
    opt.step()
val_losses = []
accuracy=0
f1score=0
print("Epoch: {}/{}...".format(e+1, training_epochs),
              "Train Loss: {:.4f}...".format(np.mean(train_losses)))

But i am getting the following error

          TypeError                                 Traceback (most recent call last)
          <ipython-input-60-3a3df06ef2f8> in <module>
   14             inputs, targets = inputs.cuda(), targets.cuda()
   15         opt.zero_grad()
   ---> 16         output = model(inputs, batch_size)
   17
   18         loss = criterion(output, targets.long())

   ~\AppData\Local\Continuum\anaconda3\lib\site-packages\torch\nn\modules\module.py in __call__(self,
   * input, **kwargs)
   530             result = self._slow_forward(*input, **kwargs)
   531         else:
   --> 532             result = self.forward(*input, **kwargs)
   533         for hook in self._forward_hooks.values():
   534             hook_result = hook(self, input, result)

   TypeError: forward() takes 2 positional arguments but 3 were given

Please guide me how i can resolve this issue.


title: Pytorch Facenet MTCNN Image output

body:
I am doing a face recognition app using facenet pytorch (https://github.com/timesler/facenet-pytorch) in python using two methods.
1st method
Code -
resnet = InceptionResnetV1(pretrained='vggface2').eval()
mtcnn = MTCNN(image_size=96)

img = Image.open(image_path)
image_prep = mtcnn(img)
plt.figure()
plt.axis('off')
plt.imshow(image_prep.permute(1, 2, 0))
if image_prep is not None:
  image_embedding = resnet(image_prep.unsqueeze(0))

In this code, I extract the face from the given images, and get the 512 encodings for recognizing faces.
In this example, I used two different faces, and plot the distance between the faces
        a.jpg       b.jpg
a.jpg   0.000000    1.142466
b.jpg   1.142466    0.000000

It works well...
2nd method
Code-
 img = Image.open(image)
 boxes, probs = mtcnn.detect(img) # Gives the coordinates of the face in the given image
 face = img.crop((boxes[0][0], boxes[0][1], boxes[0][2], boxes[0][3])) # Cropping the face
 plt.figure()
 plt.imshow(face)
 pil_to_tensor = transforms.ToTensor()(face).unsqueeze_(0) # Converting to tensor type
 image_embedding = resnet(pil_to_tensor)

In this code, I used to get the coordinates of the face, first and then the embeddings. The distance between the two faces -
        a.jpg       b.jpg
a.jpg   0.000000    0.631094
b.jpg   0.631094    0.000000

In 1st method, I directly feed the image to the mtcnn and gets better result, the distance between the two faces are more than 1.0.
In, 2nd method, I gets the coordinates of the faces using mtcnn.detect(), cropped the face from the given image, and feed to resnet. This method gives less distance between the two different faces.
Then, I find the reason for why the 1st method performs well than the 2nd method by plotting the result(face) before feeding to resnet.
In 2nd method, I feed the faces same as given in the input image(clear image) by cropping the faces using mtcnn.detect().
But, In 1st method, I directly gives the input to mtcnn(img) which returns the tensor of the faces in dark than in input image. This darker image is not a clear image(area around eyes are darker, I tested with many photos), Can't able to see the eyes clearly. This is the reason, the 1st method shows higher distance between two faces.
My doubt is, Why mtcnn returns the tensor in dark, How to solve it, Help me in this issue,
Thanks


title: Gradient disappearing after first epoch in manual linear regression

body:
I'm new to Pytorch and I've been working through the tutorials and playing around with toy examples. I wanted to just make a super simple model to get a better handle on autograd, but I'm running into issues.

I'm trying to train a linear regression model but I keep running into the following error,

----------------------------------------------------------------------

RuntimeError                              Traceback (most recent call last)
<ipython-input-80-ba5ca34a3a54> in <module>()
      9   loss = torch.dot(delta, delta)
     10
---> 11   loss.backward()
     12   with torch.no_grad():
     13     w, b = w - learning_rate*w.grad.data, b - learning_rate*b.grad.data

/usr/local/lib/python3.6/dist-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph)
     91                 products. Defaults to ``False``.
     92         """
---> 93         torch.autograd.backward(self, gradient, retain_graph, create_graph)
     94
     95     def register_hook(self, hook):

/usr/local/lib/python3.6/dist-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
     87     Variable._execution_engine.run_backward(
     88         tensors, grad_tensors, retain_graph, create_graph,
---> 89         allow_unreachable=True)  # allow_unreachable flag
     90
     91

RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

And for reference, the code is here,

# dataset for training
X = torch.randn(100, 3)
y = -3*X[:,0] + 2.2*X[:,1] + 0.002*X[:,2] + 1

w = torch.randn(3, requires_grad=True, dtype=torch.float)  # model weights
b = torch.randn(1, requires_grad=True, dtype=torch.float)  # model bias

num_epochs = 10
learning_rate = 1e-4

for i in range(num_epochs):

  y_hat = torch.mv(X, w) + b
  delta = y_hat - y
  loss = torch.dot(delta, delta)

  loss.backward()
  with torch.no_grad():
    w, b = w - learning_rate*w.grad, b - learning_rate*b.grad

The issue seems to be that after the first epoch the gradient attribute is set to None, but I'm a little confused why this would be the case.

If I try to zero the gradient after updating the weights, then I get a similar error.


title: Load Pre-trained model in Pytorch

body:
I am currently working on GANS, I have downloaded the code and models from http://www.cs.columbia.edu/~vondrick/tinyvideo/, where I just need to run to get output. I have given paths and all correctly, and the code where i am receiving an error is shown below. The have wrote the lines correctly but still getting syntax error. Please help me!

import torch

import torch.legacy.nn

import torchfile

from skimage import io, transform

import torch.nn as nn

import torch.backends.cudnn as cudnn

cudnn.enabled = True

opt = {
  model = 'models/beach/iter63000_net.t7',
  batchSize = 128,
  gpu = 1,
  cudnn = 1,
}


title: Error when training CNN: "RuntimeError: The size of tensor a (10) must match the size of tensor b (64) at non-singleton dimension 1"

body:
I'm new to Pytorch and I'm trying to implemente a simple CNN to recognize MNIST images.

I'm training the network using MSE Loss as loss function and SGD as optimizer. When I get to the training it gives me the following

  warning: " UserWarning: Using a target size (torch.Size([64])) that is different to the input size (torch.Size([64, 10])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size."

And then I get the following

error: "RuntimeError: The size of tensor a (10) must match the size of tensor b
       (64) at non-singleton dimension 1".

I've tried to solve it using some solutions I've found in other questions but nothing seems to work. Here's the code of how I load the dataset:

transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,),(0.5,))])

trainset = torchvision.datasets.MNIST(root='./data', train = True, transform = transform, download = True)
trainloader = torch.utils.data.DataLoader(trainset, batch_size = 64, shuffle = True)

testset = torchvision.datasets.MNIST(root='./data', train = False, transform = transform, download = True)
testloader = torch.utils.data.DataLoader(testset, batch_size = 64, shuffle = False)

The code to define my network:

    class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        #Convolutional layers
        self.conv1 = nn.Conv2d(1, 6, 5)
        self.conv2 = nn.Conv2d(6, 12, 5)
        #Fully connected layers
        self.fc1 = nn.Linear(12*4*4, 120)
        self.fc2 = nn.Linear(120, 60)
        self.out = nn.Linear(60,10)

    def forward(self, x):
        x = F.max_pool2d(F.relu(self.conv1(x)), (2,2))
        x = F.max_pool2d(F.relu(self.conv2(x)), (2,2))
        x = x.reshape(-1, 12*4*4)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.out(x)
        return x

And this is the training:

net = Net()
print(net)

criterion = nn.MSELoss()
optimizer = optim.SGD(net.parameters(), lr=0.001)
epochs = 3

for epoch in range(epochs):
    running_loss = 0;
    for images, labels in trainloader:
        optimizer.zero_grad()
        output = net(images)
        loss = criterion(output, labels)
        loss.backward()
        optimizer.step()

        running_loss += loss.item()
    else:
        print(f"Training loss: {running_loss/len(trainloader)}")

print('Finished training')

Thank you!


title: How does PyTorch's loss.backward() work when "retain_graph=True" is specified?

body:
I'm a newbie with PyTorch and adversarial networks. I've tried to look for an answer on the PyTorch documentation and from previous discussions both in the PyTorch and StackOverflow forums, but I couldn't find anything useful.

I'm trying to train a GAN with a Generator and a Discriminator, but I cannot understand if the whole process is working or not. As far as I'm concerned, I should train the Generator first and, then, updating the Discriminator's weights (similarly as this). My code for updating the weights of both models is:

# computing loss_g and loss_d...
optim_g.zero_grad()
loss_g.backward()
optim_g.step()

optim_d.zero_grad()
loss_d.backward()
optim_d.step()

where loss_g is the generator loss, loss_d is the discriminator loss, optim_g is the optimizer referring to the generator's parameters and optim_d is the discriminator optimizer.
If I run the code like this, I get an error:

RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.

So I specify loss_g.backward(retain_graph=True), and here comes my doubt: why should I specify retain_graph=True if there are two networks with two different graphs? Am I getting something wrong?


title: How to change the learning rate in PyTorch (1.6)

body:
I am using PyTorch and I want to change the learning rate after some epochs.
However, the code that is provided on most documentations, which is:
  optimizer = torch.optim.Adam([
        dict(params=model.parameters(), lr=learning_rate),
    ])
   #This line specifically
   optimizer.params_group[0]['lr'] = learning_rate

does not work.
Actually PyCharm hints at it:

Unresolved attribute reference 'params_group' for class 'Adam'

As a result, the error thrown is:

AttributeError: 'Adam' object has no attribute 'params_group'

How should one manually change the learning rate in PyTorch (1.6)?


title: What is the difference between model.training = False and model.param.require_grad = False

body:
What is the difference between these two:

model.training = False

and

for param in model.parameters():
    param.require_grad = False


title: PyTorch: Sigmoid of weights?

body:
I'm new to neural networks/PyTorch. I'm trying to make a net that takes in a vector x, first layer is h_j = w_j^T * x + b_j, output is max_j{h_j}. The only thing is that I want the w_j to be restricted between 0 and 1, by having w_j = S(k*a_j), where S is the sigmoid function, k is some constant, and a_j are the actual weight variables (w_j is just a function of a_j). How do I do this in PyTorch? I can't just use a torch.nn.Linear layer, there has to be something else/additional to add in the sigmoid function on the weights?

Side question, for that last output layer, can I just use torch.max to get the max of the previous layer's outputs? Does that behave nicely, or is there some torch.nn.Max or some pooling stuff that I don't understand that needs to happen?

Thanks!


title: How to fix pytorch 'RuntimeError: Expected object of type torch.cuda.LongTensor but found type torch.LongTensor'

body:
I'm trying to get this code running using FloydHub's GPU.
When I run the train.py script under train_model folder, I'm getting the RuntimeError mentioned.

Here is the full traceback:

  Traceback (most recent call last):
  File "./train_model/train.py", line 79, in
  answer = model(batch)
  File "/usr/local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in call
  result = self.forward(*input, **kwargs)
  File "/floyd/home/train_model/model.py", line 29, in forward
  vecs = self.embed(batch.text)
  File "/usr/local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in call
  result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/torch/nn/modules/sparse.py", line 110, in forward
  self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/usr/local/lib/python3.6/site-packages/torch/nn/functional.py", line 1110, in embedding
  return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
  RuntimeError: Expected object of type torch.cuda.LongTensor but found type torch.LongTensor for argument #3 'index'

I understand that parts of code are using GPU and other parts aren't but not knowing how to identify these and get all of the code to run on GPU.

Please help!


title: TypeError: 'NoneType' object is not callable (Pytorch)

body:
I am trying to complete an implementation of a neural network class that uses pytorch.
But the upgrade step is causing the error to pop up related to None Type.
I am using Pytorch Pkg with Python 3.73 using Jupyter Notebook.
The problem is in the step where I have to ake weight update step, then zero the gradient values.

        class NNet(torch.nn.Module):

def __init__(self, n_inputs, n_hiddens_per_layer, n_outputs, act_func='tanh'):
    super().__init__()  # call parent class (torch.nn.Module) constructor

    # Set self.n_hiddens_per_layer to [] if argument is 0, [], or [0]
    if n_hiddens_per_layer == 0 or n_hiddens_per_layer == [] or n_hiddens_per_layer == [0]:
        self.n_hiddens_per_layer = []
    else:
        self.n_hiddens_per_layer = n_hiddens_per_layer

    self.hidden_layers = torch.nn.ModuleList()  # necessary for model.to('cuda')

    for nh in self.n_hiddens_per_layer:
        self.hidden_layers.append( torch.nn.Sequential(
            torch.nn.Linear(n_inputs, nh),
            torch.nn.Tanh() if act_func == 'tanh' else torch.nn.ReLU()))

        n_inputs = nh

    self.output_layer = torch.nn.Linear(n_inputs, n_outputs)

    self.Xmeans = None
    self.Xstds = None
    self.Tmeans = None
    self.Tstds = None

    self.error_trace = []

def forward(self, X):
    Y = X
    for hidden_layer in self.hidden_layers:
        Y = hidden_layer(Y)
    Y = self.output_layer(Y)
    return Y

def train(self, X, T, n_epochs, learning_rate, verbose=True):

    # Set data matrices to torch.tensors if not already.
    if not isinstance(X, torch.Tensor):
        X = torch.from_numpy(X).float()
    if not isinstance(T, torch.Tensor):
        T = torch.from_numpy(T).float()
    W = torch.zeros((2, 1), requires_grad=True)
    print(W.requires_grad)

    # Calculate standardization parameters if not already calculated
    if self.Xmeans is None:
        self.Xmeans = X.mean(0)
        self.Xstds = X.std(0)
        self.Xstds[self.Xstds == 0] = 1
        self.Tmeans = T.mean(0)
        self.Tstds = T.std(0)
        self.Tstds[self.Tstds == 0] = 1

    # Standardize inputs and targets
    X = (X - self.Xmeans) / self.Xstds
    T = (T - self.Tmeans) / self.Tstds

    # Set optimizer to Adam and loss functions to MSELoss
    optimizer = torch.optim.Adam(self.parameters(), lr=learning_rate)
    mse_func = torch.nn.MSELoss()

    # For each epoch:
    #   Do forward pass to calculate output Y.
    #   Calculate mean squared error loss, mse.
    #   Calculate gradient of mse with respect to all weights by calling mse.backward().
    #   Take weight update step, then zero the gradient values.
    #   Unstandardize the mse error and save in self.error_trace
    #   Print epoch+1 and unstandardized error if verbose is True and
    #             (epoch+1 is n_epochs or epoch+1 % (n_epochs // 10) == 0)

    for epoch in range(n_epochs):
        #   Do forward pass to calculate output Y.
        Y = self.forward(X)
        print("Y = \n",Y)
        #   Calculate mean squared error loss, mse.
        mse = ((T - Y)**2).mean()
        #mse = torch.mean((T - Y[-1]) ** 2)
        print("Y shape = \n",Y.shape)
        print("Tshape = \n",T.shape)
        print("MSE = \n",mse)

        #   Calculate gradient of mse with respect to all weights by calling mse.backward().
        #W.retain_grad()
        mse.backward(torch.ones(100))
        #print("mse.backward(torch.ones(100))",mse.backward(torch.ones(100)))
        #   Take weight update step, then zero the gradient values.
        #print("W.grad = ",W.grad())
        with torch.no_grad():
            W = learning_rate*W.grad()
            print("kuttu",W.requires_grad)
            W -= learning_rate * W.grad()
            W.grad.zero_()

        #   Unstandardize the mse error and save in self.error_trace
        self.error_trace = mse * self.Tstds

        #. . .

def use(self, X):

   # Set input matrix to torch.tensors if not already.
     if not isinstance(X, torch.Tensor):
        X = torch.from_numpy(X).float()

    # Standardize X
        print("here=\n",type(X))
        X = (X - torch.mean(X)) / self.Xstds

    # Do forward pass and unstandardize resulting output. Assign to variable Y.

    # Return output Y after detaching from computation graph and converting to numpy
        return Y.detach().numpy()

*<ipython-input-20-6e1e577f866d> in train(self, X, T, n_epochs, learning_rate, verbose)
     86             #   Take weight update step, then zero the gradient values.
     87             with torch.no_grad():
---> 88                 W = learning_rate*W.grad()
     89                 print("w",W.requires_grad)
     90                 W -= learning_rate * W.grad()*

TypeError: 'NoneType' object is not callable


title: How to 4 dimension PyTorch tensor multiply by 1 dimension tensor?

body:
I'm trying to write function for mixup training. On this site i found some code and adapted to my previous code. But in original code only one random variable is generated for batch (64). But i want random value for every picture in batch.
Code with one variable for batch:

def mixup_data(x, y, alpha=1.0):
    lam = np.random.beta(alpha, alpha)
    batch_size = x.size()[0]
    index = torch.randperm(batch_size)

    mixed_x = lam * x + (1 - lam) * x[index,:]
    mixed_y = lam * y + (1 - lam) * y[index,:]

    return mixed_x, mixed_y

x and y for input come from pytorch DataLoader.
x input size: torch.Size([64, 3, 256, 256])
y input size: torch.Size([64, 3474])

This code works good. Then I changed it to this:

def mixup_data(x, y):
    batch_size = x.size()[0]
    lam = torch.rand(batch_size)
    index = torch.randperm(batch_size)

    mixed_x = lam[index] * x + (1 - lam[index]) * x[index,:]
    mixed_y = lam[index] * y + (1 - lam[index]) * y[index,:]

    return mixed_x, mixed_y

But it gives an error: RuntimeError: The size of tensor a (64) must match the size of tensor b (256) at non-singleton dimension 3

How i understand how the code works is it takes first image in batch and multiply by first value in lam tensor (64 values long). How can i do it?


title: Inputs to the nn.MultiheadAttention?

body: I have n-vectors which need to be influenced by each other and output n vectors with same dimensionality d. I believe this is what torch.nn.MultiheadAttention does. But the forward function expects query, key and value as inputs. According to this blog, I need to initialize a random weight matrix of shape (d x d) for each of q, k and v and multiply each of my vectors with these weight matrices and get 3 (n x d) matrices. Now are the q, k and v expected by torch.nn.MultiheadAttention just these three matrices or do I have it mistaken?


title: Rosenbrock function with D dimension using PyTorch

body:
How can I implement the Rosenbrock function with D dimension, using PyTorch?

  Create the variables, where D is the number of dimensions and N is the number of elements.

x = (xmax - xmin)*torch.rand(N,D).type(dtype) + xmin

  Function :

  Using straight Python I'd do something like this:

fit = 0
for i in range(D-1):
    term1 = x[i + 1] - x[i]**2
    term2 = 1 - x[i]
    fit = fit + 100 * term1**2 + term2**2

  My attempt using Pytorch:

def Rosenbrock(x):
    return torch.sum(100*(x - x**2)**2 + (x-1)**2)

  I just do not know how to do the x[i+1] without using a for loop.
  How can I deal with it?
  Thank you!


What is the reason for this error and how can I fix it? I am running the code from this repo: https://github.com/facebookresearch/frankmocap
(frank) mona@goku:~/research/code/frankmocap$ python -m demo.demo_frankmocap --input_path ./sample_data/han_short.mp4 --out_dir ./mocap_output
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/mona/research/code/frankmocap/demo/demo_frankmocap.py", line 25, in <module>
    from handmocap.hand_bbox_detector import HandBboxDetector
  File "/home/mona/research/code/frankmocap/handmocap/hand_bbox_detector.py", line 33, in <module>
    from detectors.hand_object_detector.lib.model.roi_layers import nms # might raise segmentation fault at the end of program
  File "/home/mona/research/code/frankmocap/detectors/hand_object_detector/lib/model/roi_layers/__init__.py", line 3, in <module>
    from .nms import nms
  File "/home/mona/research/code/frankmocap/detectors/hand_object_detector/lib/model/roi_layers/nms.py", line 3, in <module>
    from model import _C
ImportError: /home/mona/research/code/frankmocap/detectors/hand_object_detector/lib/model/_C.cpython-38-x86_64-linux-gnu.so: undefined symbol: _ZN6caffe28TypeMeta21_typeMetaDataInstanceIdEEPKNS_6detail12TypeMetaDataEv

I have:
$ lsb_release -a
LSB Version:    core-11.1.0ubuntu2-noarch:security-11.1.0ubuntu2-noarch
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.2 LTS
Release:    20.04
Codename:   focal

and
$ python
Python 3.8.5 (default, Jan 27 2021, 15:41:15)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.__version__
'1.8.1+cu111'
>>> import detectron2
>>> detectron2.__version__
'0.4'
>>> from detectron2 import _C

and:
$ python -m detectron2.utils.collect_env
/home/mona/venv/frank/lib/python3.8/site-packages/torch/cuda/__init__.py:52: UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 804: forward compatibility was attempted on non supported HW (Triggered internally at  /pytorch/c10/cuda/CUDAFunctions.cpp:109.)
  return torch._C._cuda_getDeviceCount() > 0
No CUDA runtime is found, using CUDA_HOME='/usr'
---------------------  --------------------------------------------------------------------------
sys.platform           linux
Python                 3.8.5 (default, Jan 27 2021, 15:41:15) [GCC 9.3.0]
numpy                  1.19.5
detectron2             0.4 @/home/mona/venv/frank/lib/python3.8/site-packages/detectron2
Compiler               GCC 7.3
CUDA compiler          CUDA 11.1
DETECTRON2_ENV_MODULE  <not set>
PyTorch                1.8.1+cu111 @/home/mona/venv/frank/lib/python3.8/site-packages/torch
PyTorch debug build    False
GPU available          False
Pillow                 8.1.0
torchvision            0.9.1+cu111 @/home/mona/venv/frank/lib/python3.8/site-packages/torchvision
fvcore                 0.1.3.post20210311
cv2                    4.5.1
---------------------  --------------------------------------------------------------------------
PyTorch built with:
  - GCC 7.3
  - C++ Version: 201402
  - Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v1.7.0 (Git Hash 7aed236906b1f7a05c0917e5257a1af05e9ff683)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.1, CUDNN_VERSION=8.0.5, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.8.1, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON,


title: Visdom shows nothing

body:
After starting the visdom server and opening it in the browser i only get a blue screen and the following errors:

ERROR:tornado.general:Could not open static file 'C:\Users\Username\Anaconda3\lib\site-packages\visdom\static\js\react-grid-layout.min.js'
                                                                                ERROR:tornado.general:Could not open static file 'C:\Users\Username\Anaconda3\lib\site-packages\visdom\static\js\plotly.min.js'

Any idea why is this happening and how can i solve it?


title: Visualize the output of Vgg16 model by TSNE plot?

body:
I need to visualize the output of Vgg16 model which classify 14 different classes.
I load the trained model and I did replace the classifier layer with the identity() layer but it doesn't categorize the output.

Here is the snippet:
the number of samples here is 1000 images.

epoch = 800
PATH = 'vgg16_epoch{}.pth'.format(epoch)
checkpoint = torch.load(PATH)
model.load_state_dict(checkpoint['model_state_dict'])
optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
epoch = checkpoint['epoch']

class Identity(nn.Module):
    def __init__(self):
        super(Identity, self).__init__()

    def forward(self, x):
        return x

model.classifier._modules['6'] = Identity()
model.eval()
logits_list = numpy.empty((0,4096))
targets = []

with torch.no_grad():
    for step, (t_image, target, classess, image_path) in enumerate(test_loader):

        t_image = t_image.cuda()
        target = target.cuda()
        target = target.data.cpu().numpy()
        targets.append(target)

        logits = model(t_image)
        print(logits.shape)

        logits = logits.data.cpu().numpy()
        print(logits.shape)
        logits_list = numpy.append(logits_list, logits, axis=0)
        print(logits_list.shape)

tsne = TSNE(n_components=2, verbose=1, perplexity=10, n_iter=1000)
tsne_results = tsne.fit_transform(logits_list)

 target_ids = range(len(targets))

plt.scatter(tsne_results[:,0],tsne_results[:,1],c = target_ids ,cmap=plt.cm.get_cmap("jet", 14))
plt.colorbar(ticks=range(14))
plt.legend()
plt.show()

here is what this script has been produced: I am not sure why I have all colors for each cluster!


title: GPU memory leakage when creating objects from sentence-transformers

body:
Description
I am creating a function in R that embeds sentences using the sentence_transformers library from Python.
For some unknown reason, creating the object multiple times under the same variable name ends up in insufficient memory space to allocate the transformer. To reproduce:
sentence_transformers <- reticulate::import("sentence_transformers")
for (i in 1:10) {
  print(i)
  bert_encoder <- sentence_transformers$SentenceTransformer("bert-large-nli-stsb-mean-tokens")
}

However, doing the same operation directly on Python does not produce an error
from sentence_transformers import SentenceTransformer
for i in range(10):
    print(i)
    bert_encoder = SentenceTransformer("bert-large-nli-stsb-mean-tokens")
}

This happens with any model that is allocated in GPU. On my NVIDIA GTX 1060 it reaches the 4th cycle, but on smaller GPUs it crashes earlier. One temporal solution is to create the model outside only once, and then pass the model as a parameter to the function as many times as wanted, but I would rather avoid that because it adds an extra step and in any case calling multiple models might just make it crash as well.
Expected behaviour
The for loop finishes without an error
Observed behaviour
Error in py_call_impl(callable, dots$args, dots$keywords) :
RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 6.00 GiB total capacity; 2.95 GiB already allocated; 16.11 MiB free; 238.68 MiB cached)
Unsuccesful attemps at solving it

The solutions proposed here
Using numba as suggested here
Declaring the variable explicitely on Python via reticulate::py_run_string() and then doing del bert_encoder and calling the garbage collector

Details
Windows 10 Home
Python 3.7.4
R 4.0.1
Reticulate 1.16
Torch 1.3.1
Tensorflow 2.2.0
Transformers 2.11.0
sentence_transformers 0.2.6


title: torch.jit.script(module) vs @torch.jit.script decorator

body:
Why is adding the decorator "@torch.jit.script" results in an error, while I can call torch.jit.script on that module, e.g. this fails:
import torch

@torch.jit.script
class MyCell(torch.nn.Module):
    def __init__(self):
        super(MyCell, self).__init__()
        self.linear = torch.nn.Linear(4, 4)

    def forward(self, x, h):
        new_h = torch.tanh(self.linear(x) + h)
        return new_h, new_h

my_cell = MyCell()
x, h = torch.rand(3, 4), torch.rand(3, 4)
traced_cell = torch.jit.script(my_cell, (x, h))
print(traced_cell)
traced_cell(x, h)

"C:\Users\Administrator\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\torch\jit\__init__.py", line 1262, in script
    raise RuntimeError("Type '{}' cannot be compiled since it inherits"
RuntimeError: Type '<class '__main__.MyCell'>' cannot be compiled since it inherits from nn.Module, pass an instance instead

While the following code works well:
class MyCell(torch.nn.Module):
    def __init__(self):
        super(MyCell, self).__init__()
        self.linear = torch.nn.Linear(4, 4)

    def forward(self, x, h):
        new_h = torch.tanh(self.linear(x) + h)
        return new_h, new_h

my_cell = MyCell()
x, h = torch.rand(3, 4), torch.rand(3, 4)
traced_cell = torch.jit.script(my_cell, (x, h))
print(traced_cell)
traced_cell(x, h)

This question is also featured on PyTorch forums.


title: Is there analog of theano.tensor.switch in pytorch?

body:
I'd like to force to zero all elements of a vector which are below a certain threshold. And I'd like to do it so that I can still propagate gradient through non-zero ones.

For example, in theano I could write:

B = theano.tensor.switch(A < .1, 0, A)

Is there a solution for that in pytorch?


title: Jupyterhub with custom singleuser image raises 'no such file or directory' for '/home/jovyan/work'

body:
I tried to setup Jupyterhub to serve Jupyter pytorch notebook using this repo as image https://github.com/stepankuzmin/pytorch-notebook, which is also available as a Docker image on dockerhub as https://hub.docker.com/r/stepankuzmin/pytorch-notebook/.

This involves modifying the config.yml file to point to the image, e.g.:

singleuser:
  image:
    name: stepankuzmin/pytorch-notebook
    tag: latest

I am getting:

oci runtime error: container_linux.go:247: starting container process caused \"chdir to cwd (\\\"/home/jovyan/work\\\") set in config.json failed: no such file or directory


title: BatchNormalization on 2D data in pytorch

body:
I'm really confused about using batch normalization. For example, i have (256,256) image , and i train my network with batch_size = 4. I need (4,64,64) feature map for each batch, so I have following model:

Conv1d(1,1,kernel_size = 1 , stride = 2, padding = 0)
BatchNorm1d(x)
Conv1d(1,1,kernel_size = 1, stride = 2,padding = 0)
BatchNorm1(x)

For above model, what should be proper number for x?

BatchNorm is performed, generally, on channel axis. However, desired feature map should have (4,64,64) size. Setting x to 1 is not make sense. Am I wrong?

Also I have one question. What is the difference between performing batchNorm on channel axis and other axis?


title: Problem with Installing and Running PyTorch

body:
I tried to use Anaconda to install PyTorch. I used

conda install -c peterjc123 pytorch

Then I use

conda install pytorch torchvision cudatoolkit=9.0 -c pytorch

When I run

import torch

I get the following error

NameError                                 Traceback (most recent call last)
 in
----> 1 import torch

~\Anaconda3\lib\site-packages\torch__init__.py in

84 from torch._C import *

85

---> 86 all += [name for name in dir(_C)

87             if name[0] != '_' and

88             not name.endswith('Base')]

NameError: name '_C' is not defined


title: Training loss not changing at all (PyTorch)

body:
I am trying to solve a text classification problem. My training data has input as a sequence of 80 numbers in which each represent a word and target value is just a number between 1 and 3.
I pass it through this model:

class Model(nn.Module):
    def __init__(self, tokenize_vocab_count):
        super().__init__()
        self.embd = nn.Embedding(tokenize_vocab_count+1, 300)
        self.embd_dropout = nn.Dropout(0.3)
        self.LSTM = nn.LSTM(input_size=300, hidden_size=100, dropout=0.3, batch_first=True)
        self.lin1 = nn.Linear(100, 1024)
        self.lin2 = nn.Linear(1024, 512)
        self.lin_dropout = nn.Dropout(0.8)
        self.lin3 = nn.Linear(512, 3)

    def forward(self, inp):
        inp = self.embd_dropout(self.embd(inp))
        inp, (h_t, h_o) = self.LSTM(inp)
        h_t = F.relu(self.lin_dropout(self.lin1(h_t)))
        h_t = F.relu(self.lin_dropout(self.lin2(h_t)))
        out = F.softmax(self.lin3(h_t))
        return out

My training loop is as follows:

model = Model(tokenizer_obj.count+1).to('cuda')

optimizer = optim.AdamW(model.parameters(), lr=1e-2)
loss_fn = nn.CrossEntropyLoss()

EPOCH = 10

for epoch in range(0, EPOCH):
     for feature, target in tqdm(author_dataloader):
         train_loss = loss_fn(model(feature.to('cuda')).view(-1,  3), target.to('cuda'))
         optimizer.zero_grad()
         train_loss.backward()
         optimizer.step()
      print(f"epoch: {epoch + 1}\tTrain Loss : {train_loss}")

I printed out the feature and target dimension and it is as follows:

torch.Size([64, 80]) torch.Size([64])

Here 64 is the batch_size.
I am not doing any validation as of now.
When I train I am getting a constant loss value and no change

/home/koushik/Software/miniconda3/envs/fastai/lib/python3.7/site-packages/torch/nn/modules/rnn.py:50: UserWarning: dropout option adds dropout after all but last recurrent layer, so non-zero dropout expects num_layers greater than 1, but got dropout=0.3 and num_layers=1
  "num_layers={}".format(dropout, num_layers))
  0%|                                                                                | 0/306 [00:00<?, ?it/s]/media/koushik/Backup Plus/Code/Machine Deep Learning/NLP/src/Deep Learning/model.py:20: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.
  out = F.softmax(self.lin3(h_t))
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 306/306 [00:03<00:00, 89.36it/s]
epoch: 1        Train Loss : 1.0986120700836182
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 306/306 [00:03<00:00, 89.97it/s]
epoch: 2        Train Loss : 1.0986120700836182
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 306/306 [00:03<00:00, 89.35it/s]
epoch: 3        Train Loss : 1.0986120700836182
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 306/306 [00:03<00:00, 89.17it/s]
epoch: 4        Train Loss : 1.0986120700836182
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 306/306 [00:03<00:00, 88.72it/s]
epoch: 5        Train Loss : 1.0986120700836182
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 306/306 [00:03<00:00, 87.75it/s]
epoch: 6        Train Loss : 1.0986120700836182
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 306/306 [00:03<00:00, 85.67it/s]
epoch: 7        Train Loss : 1.0986120700836182
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 306/306 [00:03<00:00, 85.40it/s]
epoch: 8        Train Loss : 1.0986120700836182
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 306/306 [00:03<00:00, 84.49it/s]
epoch: 9        Train Loss : 1.0986120700836182
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 306/306 [00:03<00:00, 84.21it/s]
epoch: 10       Train Loss : 1.0986120700836182

Can anyone please help


title: Pytorch Data Wont Fit in Memory - Example?

body:
I am trying to find an example of training in Pytorch in batch from data on disk - akin to the Keras fit_generator. How would I alter the code below to read the csv from disk instead of loading it to memory?

I have found that one can iterate over a custom data loader like below, but I am unsure how to do this without loading all the data in memory.

I would like to:

Train and validate the model with data held on disk
Use mini-batches of the full data on disk
Repeat x epochs

class testLoader(Dataset):

      def __init__(self):

            #regular old numpy
            boston = load_boston()
            x=boston.data
            y=boston.target

            self.x = torch.from_numpy(x)
            self.y = torch.from_numpy(y)

            self.length = x.shape[0]
            self.vars =x.shape[1]

      def __getitem__(self, index):

          return self.x[index], self.y[index]

      def __len__(self):
          return self.length

training_samples=testLoader()
train_loader = utils_data.DataLoader(training_samples, batch_size=64, shuffle=True)


title: Copying data from one tensor to another using bit masking

body:
import numpy as np
import torch
a = torch.zeros(5)
b = torch.tensor(tuple((0,1,0,1,0)),dtype=torch.uint8)
c= torch.tensor([7.,9.])
print(a[b].size())
a[b]=c
print(a)

  torch.Size([2])tensor([0., 7., 0., 9., 0.])

I am struggling to understand how this works. I initially thought the above code was using Fancy indexing but I realised that values from c tensors are getting copied corresponding to the indices marked 1. Also, if I don't specify dtype of b as uint8 then the above code does not work. Can someone please explain me the mechanism of the above code.


title: Pytorch warning displays despite being filtered

body:
I am fitting a dataset using the fastai library. The code causing the issue is here:

learn = language_model_learner(data_lm, TransformerXL, path='training_data')

with warnings.catch_warnings():

    warnings.simplefilter("ignore")
    warnings.filterwarnings("ignore")

    learn.fit_one_cycle(epochs, 1e-2)

It throws a warning every step, despite the filter:

Warning: masked_fill_ received a mask with dtype torch.uint8, this behavior is now deprecated,please use a mask with dtype torch.bool instead. (masked_fill__cuda at /pytorch/aten/src/ATen/native/cuda/LegacyDefinitions.cpp:19)

The code works, and I don't use pytorch directly, so I'd rather ignore the warning. In particular, the warning fills the console, preventing the epoch loss statistics from displaying. It looks like it's being handled by the standard library, so why doesn't the filter work? I have tried with and without the catch and both filters.


title: PyTorch training with dropout and/or batch-normalization

body:
A model should be set in the evaluation mode for inference by calling model.eval().
Do we need to also do this during training before getting the model outputs? Like within a training epoch if the network contains one or more dropout and/or batch-normalization layers.
If this is not done then the output of the forward pass in the training epoch might be affected by the randomness in the dropout?
Many example codes do not do this and something along these lines is the common approach:
for t in range(num_epochs):
    # forward pass
    yhat = model(x)

    # get the loss
    loss = criterion(yhat , y)

    # backward pass, optimizer step
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

For example here is an example code to look at : convolutional_neural_network/main.py
Should this instead be?
for t in range(num_epochs):
    # forward pass
    model.eval() # disable dropout etc
    yhat = model(x)

    # get the loss
    loss = criterion(yhat , y)

    # backward pass, optimizer step
    model.train()
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()


title: TextLMDataBunch Memory issue Language Model Fastai

body:
I have a dataset with 45 million rows of data. I have three 6gb ram gpu. I am trying to train a language model on the data.

For that, I am trying to load the data as the fastai data bunch. But this part always fails because of the memory issue.

data_lm = TextLMDataBunch.from_df('./', train_df=df_trn,
valid_df=df_val, bs=10)

How do I handle this issue?


title: Memory leak in Pytorch: object detection

body:
I am working on the object detection tutorial on PyTorch. The original tutorial works fine with the few epochs given. I expanded it to large epochs and encounter out of memory error.

I tried to debug it and find something interesting. This is the tool I am using:

def debug_gpu():
    # Debug out of memory bugs.
    tensor_list = []
    for obj in gc.get_objects():
        try:
            if torch.is_tensor(obj) or (hasattr(obj, 'data') and torch.is_tensor(obj.data)):
                tensor_list.append(obj)
        except:
            pass
    print(f'Count of tensors = {len(tensor_list)}.')

And I used it to monitor the memory of training one epoch:

def train_one_epoch(model, optimizer, data_loader, device, epoch, print_freq):
    ...
    for images, targets in metric_logger.log_every(data_loader, print_freq, header):
        # inference + backward + optimization
        debug_gpu()

The output is something like this:

Count of tensors = 414.
Count of tensors = 419.
Count of tensors = 424.
Count of tensors = 429.
Count of tensors = 434.
Count of tensors = 439.
Count of tensors = 439.
Count of tensors = 444.
Count of tensors = 449.
Count of tensors = 449.
Count of tensors = 454.

As you can see, the count of tensors tracked by garbage collector increases constantly.

Relevant files to execute can be found here.

I have two questions:
1. What is holding up the garbage collector to release these tensors?
2. What should I do with the out of memory error?


title: Input dimension for CrossEntropy Loss in PyTorch

body:
For a binary classification problem with batch_size = 1, I have logit and label values using which I need to calculate loss.

logit: tensor([0.1198, 0.1911], device='cuda:0', grad_fn=<AddBackward0>)
label: tensor(1], device='cuda:0')
# calculate loss
loss_criterion = nn.CrossEntropyLoss()
loss_criterion.cuda()
loss = loss_criterion( b_logits, b_labels )

However, this always results in the following error,

IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

What input dimensions is the CrossEntropyLoss actually asking for?


title: pytorch dataset map-style vs iterable-style

body:
A map-style dataset in Pytorch has the __getitem__() and __len__() and iterable-style datasets has __iter__() protocol. If we use map-style, we can access the data with dataset[idx] which is great, however with the iterable dataset we can't.
My question is why this distinction was necessary? What makes the data random read so expensive or even improbable?


title: BERT-based NER model giving inconsistent prediction when deserialized

body:
I am trying to train an NER model using the HuggingFace transformers library on Colab cloud GPUs, pickle it and load the model on my own CPU to make predictions.
Code
The model is the following:
from transformers import BertForTokenClassification

model = BertForTokenClassification.from_pretrained(
    "bert-base-cased",
    num_labels=NUM_LABELS,
    output_attentions = False,
    output_hidden_states = False
)

I am using this snippet to save the model on Colab
import torch

torch.save(model.state_dict(), FILENAME)

Then load it on my local CPU using
# Initiating an instance of the model type

model_reload = BertForTokenClassification.from_pretrained(
    "bert-base-cased",
    num_labels=len(tag2idx),
    output_attentions = False,
    output_hidden_states = False
)

# Loading the model
model_reload.load_state_dict(torch.load(FILENAME, map_location='cpu'))
model_reload.eval()

The code snippet used to tokenize the text and make actual predictions is the same both on the Colab GPU notebook instance and my CPU notebook instance.
Expected Behavior
The GPU-trained model behaves correctly and classifies the following tokens perfectly:
O       [CLS]
O       Good
O       morning
O       ,
O       my
O       name
O       is
B-per   John
I-per   Kennedy
O       and
O       I
O       am
O       working
O       at
B-org   Apple
O       in
O       the
O       headquarters
O       of
B-geo   Cupertino
O       [SEP]

Actual Behavior
When loading the model and use it to make predictions on my CPU, the predictions are totally wrong:
I-eve   [CLS]
I-eve   Good
I-eve   morning
I-eve   ,
I-eve   my
I-eve   name
I-eve   is
I-geo   John
B-eve   Kennedy
I-eve   and
I-eve   I
I-eve   am
I-eve   working
I-eve   at
I-gpe   Apple
I-eve   in
I-eve   the
I-eve   headquarters
I-eve   of
B-org   Cupertino
I-eve   [SEP]

Does anyone have ideas why it doesn't work? Did I miss something?


title: Pytorch autograd: Make gradient of a parameter a function of another parameter

body:
In Pytorch, how can I make the gradient of a parameter a function itself?
Here is a simple code snippet:
import torch

def fun(q):

    def result(w):
        l = w * q
        l.backward()
        return w.grad

    return result

w = torch.tensor((2.), requires_grad=True)
q = torch.tensor((3.), requires_grad=True)

f = fun(q)

print(f(w))

In the code above, how can I make f(w) have gradient with respect to q?
EDIT: based on the accepted answer I was able to write a code that works. Essentially I am alternating between 2 optimization steps. For dim == 1 it works and for dim == 2 it does not. I get the error "RuntimeError: Trying to backward through the graph a second time, but the saved intermediate results have already been freed. Specify retain_graph=True when calling backward the first time."
import torch

class f_class():
    def __init__(self, dim):
        self.dim = dim
        if self.dim == 1:
            self.w = torch.tensor((3.), requires_grad=True)
        elif self.dim == 2:
            self.w = [torch.tensor((3.), requires_grad=True), torch.tensor((5.), requires_grad=True)]
        else:
            raise ValueError("dim 1 or 2")

    def forward(self, x):
        if self.dim == 1:
            return torch.mul(self.w, x)
        elif self.dim == 2:
            return torch.mul(torch.mul(self.w[0], self.w[1]), x)

    def set_w(self, w):
        self.w = w

    def get_w(self):
        return self.w

class g_class():
    def __init__(self):
        self.q = torch.tensor((4.), requires_grad=True)

    def forward(self, f):
        return torch.mul(self.q, f)

    def set_q(self, q):
        self.q = q

    def get_q(self):
        return self.q

def w_new(f, g, dim):

    loss_g = g.forward(f.forward(xd))

    if dim == 1:
        grads = torch.autograd.grad(loss_g, f.get_w(), create_graph=True, only_inputs=True)[0]

        temp = f.get_w().detach() + grads
    else:
        grads = torch.autograd.grad(loss_g, f.get_w(), create_graph=True, only_inputs=True)

        temp = [wi.detach() + gi for wi, gi in zip(f.get_w(), grads)]

    return temp

def q_new(f, g):

    loss_f = 2 * f.forward(xd)

    loss_f.backward()

    temp = g.get_q().detach() + g.get_q().grad

    temp.requires_grad = True

    return temp

dim = 1

xd = torch.tensor((2.))

f = f_class(dim)
g = g_class()

for _ in range(3):

    print(f.get_w(), g.get_q())

    wnew = w_new(f, g, dim)

    f.set_w(wnew)
    print(f.get_w(), g.get_q())

    qnew = q_new(f, g)

    g.set_q(qnew)

print(f.get_w(), g.get_q())


title: Dimension out of range when applying l2 normalization in Pytorch

body:
I'm getting a runtime error:

  RuntimeError: Dimension out of range (expected to be in range of [-1, 0], but got 1)`

and can't figure out how to fix it.

The error appears to refer to the line:

i_enc = F.normalize(input =i_batch, p=2, dim=1, eps=1e-12)  # (batch, K, feat_dim)

I'm trying to encode image features (batch x 36 x 2038) by applying a L2 norm. Below is the full code for the section.

def forward(self, q_batch, i_batch):

    # batch size = 512
    # q -> 512(batch)x14(length)
    # i -> 512(batch)x36(K)x2048(f_dim)
    # one-hot -> glove
    emb = self.embed(q_batch)
    output, hn = self.gru(emb.permute(1, 0, 2))
    q_enc = hn.view(-1,self.h_dim)

    # image encoding with l2 norm
    i_enc = F.normalize(input =i_batch, p=2, dim=1, eps=1e-12)  # (batch, K, feat_dim)

    q_enc_copy = q_enc.repeat(1, self.K).view(-1, self.K, self.h_dim)

    q_i_concat = torch.cat((i_enc, q_enc_copy), -1)
    q_i_concat = self.non_linear(q_i_concat, self.td_W, self.td_W2 )#512 x 36 x 512
    i_attention = self.att_w(q_i_concat)  #512x36x1
    i_attention = F.softmax(i_attention.squeeze(),1)
    #weighted sum
    i_enc = torch.bmm(i_attention.unsqueeze(1), i_enc).squeeze()  # (batch, feat_dim)

    # element-wise multiplication
    q = self.non_linear(q_enc, self.q_W, self.q_W2)
    i = self.non_linear(i_enc, self.i_W, self.i_W2)
    h = torch.mul(q, i)  # (batch, hid_dim)

    # output classifier
    # BCE with logitsloss
    score = self.c_Wo(self.non_linear(h, self.c_W, self.c_W2))

    return score

I would appreciate any help.
Thanks


title: pytorch loading model not same softmax probabilities

body:
I cannot reproduce the same results after loading a model using pytorch.
I am training a model 'net' and in the same file, after training (kfold) then the model is saved and also tested in 1 specific testing file:

class model(nn.Module):
    def __init__(self,size_net):
        print('Initialize net with size: ',size_net)
        self.T = size_net

        # Layer 1
        self.conv1 = nn.Conv2d(1, 16, (1,16), padding = 0)
        self.batchnorm1 = nn.BatchNorm2d(16, False)

        # Layer 2
        self.padding1 = nn.ZeroPad2d((16, 17, 0, 1))
        self.conv2 = nn.Conv2d(1, 4, (2, 32))
        self.batchnorm2 = nn.BatchNorm2d(4, False)
        self.pooling2 = nn.MaxPool2d(2, 4)

        # Layer 3
        self.padding2 = nn.ZeroPad2d((2, 1, 4, 3))
        self.conv3 = nn.Conv2d(4, 4, (8, 4))
        self.batchnorm3 = nn.BatchNorm2d(4, False)
        self.pooling3 = nn.MaxPool2d((2, 4))

        # FC Layer
        # NOTE: This dimension will depend on the number of timestamps per sample in your data.
        # I have 120 timepoints.

        self.fc1 = nn.Linear(int(self.T/2), 2)

    def forward(self, x):
        # Layer 1
        x = F.elu(self.conv1(x))
        x = self.batchnorm1(x)
        x = F.dropout(x, 0.25)
        x = x.permute(0, 3, 1, 2)
        #print "layer 1"
        # Layer 2
        x = self.padding1(x)
        x = F.elu(self.conv2(x))
        x = self.batchnorm2(x)
        x = F.dropout(x, 0.25)
        x = self.pooling2(x)

        #print "layer 2"

        # Layer 3
        x = self.padding2(x)
        x = F.elu(self.conv3(x))
        x = self.batchnorm3(x)
        x = F.dropout(x, 0.25)
        x = self.pooling3(x)

        #print "layer 3"

        # FC Layer
        #print ('view:',x.shape)
        x = x.view(-1, int(self.T/2))
        #x = torch.sigmoid(self.fc1(x))
        x= torch.softmax(self.fc1(x),1)

        #print "layer 4"

        return x

#now call the model and train

net = model(SIZE_NET)

....

eval.train_Kfold_validation(n_epochs=25)

## save models state

"""
net = EEGNet(SIZE_NET)
save_path = './eeg_net_{}.pt'.format(date.today().strftime("%Y%m%d"))
torch.save(net.state_dict(), save_path)

'''
TEST
'''
testfile = '1_testonline_1_20190202-163051.csv'
kun_1 = np.genfromtxt( '../'+ testfile, delimiter=',').astype('float32')[:-1, :]
kun_1 = kun_1[:, :SIZE_NET]
X, y = prep.list_2darrays_to_3d([kun_1], -1)
print(X.shape)
array_dstack = np.array(X)
array_dstack_reshaped = np.reshape(array_dstack,(1, 1, SIZE_NET, 16))
inputs = Variable(torch.from_numpy(array_dstack_reshaped))
pred = net(inputs)
print('prob: '+str(pred)) #Converted to probabilities

For example for this file I got: pred=tensor([[0.5912, 0.4088]], grad_fn=)

When instead I load the saved model in a new script and I attempt inference again on the same testfile:

prep= Data_prep()
fileName = '1_testonline_1_20190202-163051.csv'

kun_1 = np.genfromtxt(file_dir+fileName, delimiter=',').astype('float32')[:-1,:]
kun_1 = kun_1[:,:SIZE_NET]

X , y = prep.list_2darrays_to_3d([kun_1],[-1])

# Load pre-trained model
net = model(SIZE_NET)
load_path = file_dir+'/model_colors/model_20190205.pt'

net.load_state_dict(torch.load(load_path))
net.eval()

array_dstack = np.array(X)
print(X.shape)

# (#samples, 1, #timepoints, #channels)
array_dstack_reshaped = np.reshape(array_dstack,(1, 1, SIZE_NET, 16))
inputs = Variable(torch.from_numpy(array_dstack_reshaped))
pred = net(inputs)

print(pred)

When I run the test script the prob values are different and even worse NOT stable: running multiple times give different predictions... Any help appreciated


title: How can I get hidden_states from BertForSequenceClassification?

body:
I read the official tutorial(https://huggingface.co/transformers/model_doc/bert.html) and tried to set config, but it doesn't work.

from transformers import PretrainedConfig
model = BertForSequenceClassification.from_pretrained("bert-base-uncased", num_labels=2)
model.config.output_hidden_states = True
model.load_state_dict(torch.load('../parameter.pkl'))
model.cuda()
output = model(input)


title: Is there a way to remove 1-2 (or more) specific neuron connections between the layers in the NN using Tensorflow (with Keras) or PyTorch?

body:
I am in the process of making a GUI for creating different unique Neural Networks. I am down for using TensorFlow 2.0 (with Keras API) or PyTorch as the back-end. But I am lacking information on those topics, and would be super grateful if anyone can answer these:

Language: Python

1) How to remove specific neuron connections between the layers in NN using any of these frameworks?
2) How to set the specific learning rule for some of the neurons in the layer?
3) How to set the specific activating function for some of the neurons in the layer?

Huge thanks, please don't hesitate to answer, any information is useful.

P.S. If anyone wants to contribute to the project, It would be awesome.


title: How to convert pytorch tensor to onnx tensor in custom layer?

body:
I'm trying to make onnx realization of pytorch block. I made custom block with custom forward function.

class MyConvBlockFunction(Function):

    @staticmethod
    def symbolic(g, input, conv1):
        from torch.onnx.symbolic_opset9 import _shape_as_tensor, _convolution, relu

        conv = _convolution(g, input, conv1.weight, False, 1, 1, 1, False, (), 1, None, None, None)
        output = relu(g, conv)

        return output

    @staticmethod
    def forward(self, input, conv1):
        conv = conv1(input)
        relu1 = nn.ReLU()
        res = relu1(conv)
        return res

class MyConvBlock(nn.Module):

    def __init__(self):
        super(MyConvBlock, self).__init__()
        self.conv1 = nn.Conv2d(in_channels = 3, out_channels = 2, kernel_size=3, stride=1, padding=1, bias=False)
        self.relu = nn.ReLU()
        #self.weight = torch.tensor(self.conv1.weight, requires_grad=False)

    def forward(self, input):
        return MyConvBlockFunction.apply(input, self.conv1)

But when I run export to onnx, I get an error that weight tensor is wrong tensor.

weight_size = weight.type().sizes()
AttributeError: 'str' object has no attribute 'sizes'

I found that my input tensor is a good onnx tensor, and 'conv.weight' is pytorch tensor, not onnx tensor.

input.1 defined in (%input.1 : Float(1, 3, 4, 4), %conv1.weight : Float(2, 3, 3, 3) = prim::Param()
2 defined in (%2 : Tensor = onnx::Constant[value=<Tensor>](), scope: MyConvBlock)

How can I send weights to onnx _convolution operation?

UPD: Maybe problem in my export code?

torch.onnx.export(model,               # model being run
                  x,                         # model input (or a tuple for multiple inputs)
                  "convblock.onnx",   # where to save the model (can be a file or file-like object)
                  export_params=True,        # store the trained parameter weights inside the model file
                  opset_version=OPSET_VER,          # the ONNX version to export the model to
                  do_constant_folding=True,  # whether to execute constant folding for optimization
                  input_names = ['input'],   # the model's input names
                  output_names = ['output'])


title: Expect FloatTensors but got LongTensors in MNIST-like task

body:
I am performing a MNIST-like task, the input is 10-class images, and the expected output is the predicted class of the images.

But now the output is like [-2.3274, -2.2723, ...], which the length is the batch_size. And the target is [4., 2., 2., 8., ...]

Error message: RuntimeError: expected object for scalar type Long but got scalar type float for argument #2 'target'

class Net(nn.Module):
    ...
    ...
    def forward(self, x):
        ...
        ...
        return F.log_softmax(x, dim = 1)

criterion = torch.nn.NLLLoss()

Can anyone give me some advice? Thanks.


title: Is it possible to make a max-pooling on dynamic length sentences without padding in Pytorch?

body:
Let's assume I have three set of different tensors:

x1 = torch.randn(3,16)
x2 = torch.randn(5,16)
x3 = torch.randn(2,16)

For each of them, I want to apply a max-pooling and get 1x16 dimensional tensor as output. Is there a way to do that without padding input tensors ?
I mean, I think I can do what I want by first padding x1 and x3 tensors so that they also become 5x16 tensors. and then I can apply maxpool1d() operator to get what I want.


title: Pytorch: How to load csv files in several folder

body:
My data like:
folder 1
  part0001.csv
  part0002.csv
  ...
  part0199.csv
folder 2
  part0001.csv
  part0002.csv
  ...
  part0199.csv
folder 3
  part0001.csv
  part0002.csv
  ...
  part0199.csv

Update:
Each .csv file is about 100Mb. Both features and label are in the same .csv file. Each .csv file is like as follows.
  feat1 feat2 label
1 1     3     0
2 3     4     1
3 2     5     0
...

I want to load the samples in the .csv file in batch.