title: pytorch TypeError: '<' not supported between instances of 'Example' and 'Example' when referring to iterator

body:
I am trying to use my own dataset to classify text according to https://github.com/bentrevett/pytorch-sentiment-analysis/blob/master/5%20-%20Multi-class%20Sentiment%20Analysis.ipynb. My dataset is a csv of sentences and a class associated with it. there are 6 different classes:

sent                      class
'the fox is brown'        animal
'the house is big'        object
'one water is drinkable'  water
...

When running:

N_EPOCHS = 5

best_valid_loss = float('inf')

for epoch in range(N_EPOCHS):

    start_time = time.time()
    print(start_time)
    train_loss, train_acc = train(model, train_iterator, optimizer, criterion)
    #print(train_loss.type())
    #print(train_acc.type())
    valid_loss, valid_acc = evaluate(model, valid_iterator, criterion)

    end_time = time.time()

    epoch_mins, epoch_secs = epoch_time(start_time, end_time)

    if valid_loss < best_valid_loss:
        best_valid_loss = valid_loss
        torch.save(model.state_dict(), 'tut5-model.pt')

    print(f'Epoch: {epoch+1:02} | Epoch Time: {epoch_mins}m {epoch_secs}s')
    print(f'\tTrain Loss: {train_loss:.3f} | Train Acc: {train_acc*100:.2f}%')
    print(f'\t Val. Loss: {valid_loss:.3f} |  Val. Acc: {valid_acc*100:.2f}%')

, I obtained the following error

1566460706.977012
epoch_loss
<torchtext.data.iterator.BucketIterator object at 0x000001FABE907E80>
TypeError: '<' not supported between instances of 'Example' and 'Example'

pointing to:

TypeError                                 Traceback (most recent call last)
<ipython-input-22-19e8a7eb204e> in <module>()
     10     #print(train_loss.type())
     11     #print(train_acc.type())
---> 12     valid_loss, valid_acc = evaluate(model, valid_iterator, criterion)
     13
     14     end_time = time.time()

<ipython-input-21-83b02f99bca7> in evaluate(model, iterator, criterion)
      9         print('epoch_loss')
     10         print(iterator)
---> 11         for batch in iterator:
     12             print('batch')
     13             predictions = model(batch.text)

I am new to pytorch, so only added a line to idenify iterator data type and got:

<torchtext.data.iterator.BucketIterator object at 0x000001FABE907E80>

I tried to determine specific attributes following  https://github.com/pytorch/text/blob/master/torchtext/data/iterator.py to no avail.

Any suggestions are appreciated.

The code for the evaluate (where the iterator is) method is:

def evaluate(model, iterator, criterion):

    epoch_loss = 0
    epoch_acc = 0

    model.eval()

    with torch.no_grad():
        print('epoch_loss')
        print(iterator)
        for batch in iterator:
            print('batch')
            predictions = model(batch.text)

            loss = criterion(predictions, batch.label)

            acc = categorical_accuracy(predictions, batch.label)

            epoch_loss += loss.item()
            epoch_acc += acc.item()

    return epoch_loss / len(iterator), epoch_acc / len(iterator)


title: masked kernel in convolution in pytorch

body:
Let's say:
x = torch.arange(16, dtype=torch.float).reshape(1, 1, 4, 4)

and a 2d convolution layer is:
layer = torch.nn.Conv2d(in_channels=1, out_channels=1, kernel_size=2, stride=2)
layer.weight.data[:] = 1.
layer.bias.data[:] = 0.

Normally, passing x to layer will give:
>>layer(x)
tensor([[[[10., 18.],
              [42., 50.]]]], grad_fn=<MkldnnConvolutionBackward>)

Considering having 4 mask filters, how it is done to mask kernel in each step?
for example the following picture indicates 4 filters(white: True, black: False)

The output should be:
tensor([[[[5., 15.],
          [30., 40.]]]], grad_fn=<MkldnnConvolutionBackward>)

P.S: all masks are obtained by missing pixels in 2d input array. So 4 masks above are actually a matrix with the same shape as input.


title: How is a 3-d tensor indexed by two 2d tensors?

body:
Here's a snapshot from line 15-20 in DIM

def random_permute(X):
    X = X.transpose(1, 2)
    b = torch.rand((X.size(0), X.size(1))).cuda()
    idx = b.sort(0)[1]
    adx = torch.range(0, X.size(1) - 1).long()
    X = X[idx, adx[None, :]].transpose(1, 2)

    return X

where X is a tensor of size [64, 64, 128], idx a tensor of size [64, 64], adx  a tensor of size [64].
How does X = X[idx, adx[None, :]] work? How can we use two 2d tensors to index a 3d tensor? What really happens to X after this indexing?


title: How should I understand the nn.Embeddings arguments num_embeddings and embedding_dim?

body:
I'm trying to get used to the Embedding class in the PyTorch nn module.

I've noticed that quite a few other people have had the same problem as myself, and therefore posted questions on the PyTorch discussion forum and on Stack Overflow, but I'm still having some confusion.

According to the official documentation, the arguments that are passed are num_embeddings and embedding_dim which each refer to how large our dictionary (or vocabulary) is and how many dimensions we want our embeddings to be, respectively.

What I'm confused about is how exactly I should interpret those. For example, the small practice code that I ran:

import torch
import torch.nn as nn

embedding = nn.Embedding(num_embeddings=10, embedding_dim=3)

a = torch.LongTensor([[1, 2, 3, 4], [4, 3, 2, 1]]) # (2, 4)

b = torch.LongTensor([[1, 2, 3], [2, 3, 1], [4, 5, 6], [3, 3, 3], [2, 1, 2],
                      [6, 7, 8], [2, 5, 2], [3, 5, 8], [2, 3, 6], [8, 9, 6],
                      [2, 6, 3], [6, 5, 4], [2, 6, 5]]) # (13, 3)

c = torch.LongTensor([[1, 2, 3, 2, 1, 2, 3, 3, 3, 3, 3],
                      [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2]]) # (2, 11)

When I run a, b, and c through the embedding variable, I get embedded results of shapes (2, 4, 3), (13, 3, 3), (2, 11, 3).

What's confusing me is that I thought of the number of samples we have exceeds the predefined number of embeddings, we should get an error? Since the embedding I've defined has 10 embeddings, shouldn't b give me an error since it is a tensor containing 13 words of dimension 3?


title: How to prevent inf while working with exponential

body:
I'm trying to create  a function in a network with trainable parameters. In my function I have an exponential that for large tensor values goes to infinity. What would the best way to avoid this be?

The function is as follows:

step1 = Pss-(k*Pvv)
step2 = step1*s
step3 = torch.exp(step2)
step4 = torch.log10(1+step3)
step5 = step4/s

#or equivalently
# train_curve = torch.log(1+torch.exp((Pss-k*Pvv)*s))/s

If it makes it easier to understand, the basic function is log10(1+e^(x-const)*10)/10. The exponential inside the log gets too big and goes to inf.

I think I might have to normalize my tensor x, and this would mean normalizing the constants and the rest of the function also. Would someone have any thoughts on the best way to go about this?

Thanks so much.


title: how to set values for layers in pytorch nn.module?

body:
I have a model that I am trying to get working.  I am working through the errors, but now I think it has come down to the values in my layers.  I get this error:

RuntimeError: Given groups=1, weight of size 24 1 3 3, expected input[512, 50, 50, 3] to have 1 channels,
but got 50 channels instead

My parameters are:

LR = 5e-2
N_EPOCHS = 30
BATCH_SIZE = 512
DROPOUT = 0.5

My image information is:

depth=24
channels=3
original height = 1600
original width = 1200
resized to 50x50

This is the size of my data:

Train shape (743, 50, 50, 3) (743, 7)
Test shape (186, 50, 50, 3) (186, 7)
Train pixels 0 255 188.12228712427097 61.49539262385051
Test pixels 0 255 189.35559211469533 60.688278787628775

I looked here to try to understand what values each layer is expecting, but when I put in what it says here, https://towardsdatascience.com/pytorch-layer-dimensions-what-sizes-should-they-be-and-why-4265a41e01fd, it gives me errors about wrong channels and kernels.

I found torch_summary to give me more understanding about the outputs, but it only poses more questions.

This is my torch_summary code:

from torchvision import models
from torchsummary import summary
import torch
import torch.nn as nn

class CNN(nn.Module):
    def __init__(self):
        super(CNN, self).__init__()
        self.conv1 = nn.Conv2d(1,24, kernel_size=5)  # output (n_examples, 16, 26, 26)
        self.convnorm1 = nn.BatchNorm2d(24) # channels from prev layer
        self.pool1 = nn.MaxPool2d((2, 2))  # output (n_examples, 16, 13, 13)
        self.conv2 = nn.Conv2d(24,48,kernel_size=5)  # output (n_examples, 32, 11, 11)
        self.convnorm2 = nn.BatchNorm2d(48) # 2*channels?
        self.pool2 = nn.AvgPool2d((2, 2))  # output (n_examples, 32, 5, 5)
        self.linear1 = nn.Linear(400,120)  # input will be flattened to (n_examples, 32 * 5 * 5)
        self.linear1_bn = nn.BatchNorm1d(400) # features?
        self.drop = nn.Dropout(DROPOUT)
        self.linear2 = nn.Linear(400, 10)
        self.act = torch.relu

    def forward(self, x):
        x = self.pool1(self.convnorm1(self.act(self.conv1(x))))
        x = self.pool2(self.convnorm2(self.act(self.conv2(x))))
        x = self.drop(self.linear1_bn(self.act(self.linear1(x.view(len(x), -1)))))
        return self.linear2(x)

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
model=CNN().to(device)
summary(model, (3, 50, 50))

This is what it gave me:

  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py", line 342, in conv2d_forward
self.padding, self.dilation, self.groups)
RuntimeError: Given groups=1, weight of size 24 1 5 5, expected input[2, 3, 50, 50] to have 1 channels, but got 3 channels instead

When I run my whole code, and unsqueeze_(0) my data, like so....x_train = torch.from_numpy(x_train).unsqueeze_(0)  I get this error:

 File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py", line 342, in conv2d_forward
self.padding, self.dilation, self.groups)
RuntimeError: Expected 4-dimensional input for 4-dimensional weight 24 1 5 5, but got 5-dimensional input of size [1, 743, 50, 50, 3] instead

I don't know how to figure out how to fill in the proper values in the layers.  Will someone please help me find the correct values and understand how to understand this?  I do know that the output of one layer should be the input of another layer.  Nothing is matching up with what I thought I knew.
Thanks in advance!!


title: GPU support for TensorFlow & PyTorch

body:
Okay, so I've worked on a bunch of Deep Learning projects and internships now and I've never had to do heavy training. But lately I've been thinking of doing some Transfer Learning for which I'll need to run my code on a GPU. Now I have a system with Windows 10 and a dedicated NVIDIA GeForce 940M GPU. I've been doing a lot of research online, but I'm still a bit confused. I haven't installed the NVIDIA Cuda Toolkit or cuDNN or tensorflow-gpu on my system yet. I currently use tensorflow and pytorch to train my DL models. Here are my queries -

When I define a tensor in tf or pytorch, it is a cpu tensor by default. So, all the training I've been doing so far has been on the CPU. So, if I make sure to install the correct versions of Cuda and cuDNN and tensorflow-gpu (specifically for tensorflow), I can run my models on my GPU using tf-gpu and pytorch and that's it? (I'm aware of the torch.cuda.is_available() in pytorch to ensure pytorch can access my GPU and the device_lib module in tf to check if my gpu is visible to tensorflow)(I'm also aware of the fact that tf doesnt support all Nvidia GPUs)

Why does tf have a separate module for GPU support? PyTorch doesnt seem to have that and all you need to do is cast your tensor from cpu() to cuda() to switch between them.

Why install cuDNN? I know it is a high-level API CUDA built for support to train Deep Neural Nets on the GPU. But do tf-gpu and torch use these in the backend while training on the gpu?

After tf == 1.15, did they combine CPU and GPU support all into one package?


title: Pytorch issue: torch.load() does not correctly load a saved model from file after closing and reopening Spyder IDE

body:
I followed the most basic code procedure for saving and loading neural network model parameters and it works perfectly fine. After training the network, it is saved to a specified file in a specified folder in the package using the standard torch.save(model.state_dict(), file) method; when I need to rerun the program to evaluate instead of train, it is loaded using the standard model.load_state_dict(torch.load(file)) method. However, as soon as I close the Spyder IDE application and reopen the IDE and project, the torch.load does not result in the desired saved model. Note that I am testing the 'correctness' of the model by feeding inputs and checking the outputs. I am not sure if this is a problem with Spyder or with Pytorch, though I'm learning towards Spyder because the loading does work.
In short, the program works if I run it consecutively while Spyder is open, but as soon as Spyder closes and reopens, it ceases to work properly. Does anyone have a clue?


title: How to vectorize computation of mean over specific set of indices given as matrix rows?

body:
I have a problem vectorizing some code in pytorch.
A numpy solution would also help, but a pytorch solution would be better.
I'm going to use array and Tensor interchangeably.

The problem I am facing is this:

Given an 2D float array X of size (n, x), and a boolean 2D array A of size (n, n), compute the mean over rows in X indexed by rows in A.
The problem is that the rows in A contain a variable number of True indices.

Example (numpy):

import numpy as np
A = np.array([[0, 1, 0, 0, 0, 0],
              [1, 0, 1, 0, 0, 0],
              [0, 1, 0, 0, 0, 0],
              [0, 0, 0, 0, 1, 0],
              [0, 0, 0, 1, 0, 0],
              [0, 1, 1, 1, 0, 0]])
X = np.arange(6 * 3, dtype=np.float32).reshape(6, 3)

# Compute the mean in numpy with a for loop
means_np = np.array([X[A.astype(np.bool)[i]].mean(axis=0) for i in np.arange(len(A)])

So this examples works, but this formulation has 3 problems:

The for loop is slow for larger A and X. I need to loop over a few 10 thousand indices.
It can happen that A[i] contains no True indices. This results in np.mean(np.array([])), which is NaN. I want this to be 0 instead.
Implementing it this way in pytorch results in SIGFPE (Floating point error) during the backwards pass of backpropagation through this function. The cause is when nothing being selected.

The workaround that I am using now is (also see code below):

Set the diagonal elements of A to True so that there is always at least one element to select
sum of all selected elements, subtract the values in X from that sum (the diagonal is guaranteed to be False in the beginning), and divide by the number of True elements - 1 clamped to at least 1 in each row.

This works, is differentiable in pytorch and does not produce NaN, but I still need a loop over all indices.
How can I get rid of this loop?

This is my current pytorch code:

 import torch
 A = torch.from_numpy(A).bytes()
 X = torch.from_numpy(X)
 A[np.diag_indices(len(A)] = 1  # Set the diagonal to 1
 means = [(X[A[i]].sum(dim=0) - X[i]) / torch.clamp(A[i].sum() - 1, min=1.)  # Compute the mean safely
          for i in range(len(A))]  # Get rid of the loop somehow
 means = torch.stack(means)

I don't mind if your version looks completely different, as long as it is differentiable and produces the same result.


I'm having trouble with visualizing the weights and bias of my model using tensorboardX.
Here is my model (it's pretty simple anyway):

    self.pipe = nn.Sequential(nn.Linear(9, 128),
                             nn.ReLU(),
                             nn.Linear(128, 256),
                             nn.ReLU(),
                             nn.Linear(256,2),
                             nn.Softmax()
                             )

     def forward(self, x):
     return self.pipe(x)

And here is where I get error in pytorch

for name, param in net.named_parameters():
    writer.add_histogram(name, param, epoch_size, bins='auto')

and the error is

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-70-d060d2df4423> in <module>()
      1 for name, param in net.named_parameters():
----> 2     writer.add_histogram(name, param, epoch_size, bins='auto')

~\Anaconda3\lib\site-packages\tensorboardX\writer.py in add_histogram(self, tag, values, global_step, bins, walltime)
    403         if isinstance(bins, six.string_types) and bins == 'tensorflow':
    404             bins = self.default_bins
--> 405         self.file_writer.add_summary(
    406             histogram(tag, values, bins), global_step, walltime)
    407

AttributeError: 'NoneType' object has no attribute 'add_summary'

but I really have to see the histogram where the weights stuck in suboptimal.
so I changed code little bit to proceed step by step

param = np.array(list(net.parameters()))
print(param[0].data)
writer.add_histogram('weight', param[0].data)

BOOM!  still same error, maybe that doesn't change at all.


title: Pytorch RuntimeError: size mismatch

body:
I get the following error when I try to run this code for training a GAN for prediction:

  RuntimeError: size mismatch, m1: [128 x 1], m2: [1392 x 2784] at C:\w\1\s\tmp_conda_3.7_021303\conda\conda-bld\pytorch_1565316900252\work\aten\src\TH/generic/THTensorMath.cpp:752

Please comment if you see any other mistakes in the code or if you have some tips in general.

# Sample indices
def sample_idx(m, n):
    A = np.random.permutation(m)
    idx = A[:n]
    return idx

class Discriminator(nn.Module):
    def __init__(self):
        super(Discriminator, self).__init__()
        self.fc1 = nn.Linear(in_features = dim, out_features = dim*2)
        self.fc2 = nn.Linear(in_features = dim*2, out_features = dim)
        self.fc3 = nn.Linear(in_features = dim, out_features = int(dim/2))
        self.fc4 = nn.Linear(in_features = int(dim/2), out_features = 1)
        self.relu = nn.LeakyReLU(0.2)
        self.sigmoid = nn.Sigmoid()
        self.init_weight()

    def init_weight(self):
        layers = [self.fc1, self.fc2, self.fc3]
        [nn.init.xavier_normal_(layer.weight) for layer in layers]

    def forward(self, x):
        x = self.relu(self.fc1(x))
        x = self.relu(self.fc2(x))
        x = self.relu(self.fc3(x))
        x = self.sigmoid(self.fc4(x))
        return x

class Generator(nn.Module):
    def __init__(self):
        super(Generator, self).__init__()
        self.fc1 = nn.Linear(in_features = dim, out_features = dim*2)
        self.fc2 = nn.Linear(in_features = dim*2, out_features = dim)
        self.fc3 = nn.Linear(in_features = dim, out_features = int(dim/2))
        self.fc4 = nn.Linear(in_features = int(dim/2), out_features = 1)
        self.relu = nn.LeakyReLU(0.2)
        self.init_weight()

    def init_weight(self):
        layers = [self.fc1, self.fc2, self.fc3]
        [nn.init.xavier_normal_(layer.weight) for layer in layers]

    def forward(self, x):
        x = self.relu(self.fc1(x))
        x = self.relu(self.fc2(x))
        x = self.relu(self.fc3(x))
        x = self.fc4(x)
        return x

# 1. Mini batch size
mb_size = 128

# 4. Loss Hyperparameters
alpha = 10

# 5. Input Dim (Fixed)
dim = data.shape[1] - 1

# 6. Train Test Split
train_n = int(199476 * 0.8)
test_n = 199476 - int(199476 * 0.8)

X_train = data[:train_n,:-1]
y_train = data[:train_n, -1]

X_test = data[train_n:,:-1]
y_test = data[train_n:, -1]

# Nets
D = Discriminator()
G = Generator()

# Optimizers
optimD = torch.optim.Adam(D.parameters(), lr = 0.001)
optimG = torch.optim.Adam(G.parameters(), lr = 0.001)

# Loss
mse_loss = nn.MSELoss(reduction = 'elementwise_mean')

i = 1
for it in tqdm(range(10000)):
    mb_idx = sample_idx(train_n, mb_size)
    X_mb = X_train[mb_idx,:]
    X_mb = torch.tensor(X_mb).float()
    y_mb = y_train[mb_idx]
    y_mb = torch.tensor(y_mb).float()

    # Train D
    G_sample = G(X_mb)
    D_loss1 = ((D(X_mb) + 1e-8).log()).mean() + ((1 - D(G_sample) + 1e-8).log()).mean()
    D_loss = - D_loss1
    D_loss.backward()
    optimD.step()
    optimD.zero_grad()

    # Train G
    G_sample = G(X_mb)
    D_prob.detach()
    G_loss1 = ((1 - D(G_sample) + 1e-8).log()).mean()
    G_mse_loss = mse_loss(G_sample, y_mb)
    G_loss = G_loss1 + alpha * G_mse_loss
    G_loss.backward()
    optimG.step()
    optimG.zero_grad()

    if it % 100 == 0:
        print("Iter: {}".format(it))
        print("D_loss: {:.4}".format(D_loss))
        print("Train loss: {:.4}".format(G_mse_loss))
        print()


title: How to use pretrained weights of a model for initializing the weights in next iteration?

body:
I have a model architecture. I have saved the entire model using torch.save() for some n number of iterations. I want to run another iteration of my code by using the pre-trained weights of the model I saved previously.

Edit: I want the weight initialization for the new iteration be done from the weights of the pretrained model

Edit 2: Just to add, I don't plan to resume training. I intend to save the model and use it for a separate training with same parameters. Think of it like using a saved model with weights etc. for a larger run and more samples (i.e. a complete new training job)

Right now, I do something like:

# default_lr = 5
# default_weight_decay = 0.001
# model_io = the pretrained model
model = torch.load(model_io)
optim = torch.optim.Adam(model.parameters(),lr=default_lr, weight_decay=default_weight_decay)
loss_new = BCELoss()
epochs = default_epoch
.
.
training_loop():
....
outputs = model(input)
....
.
#similarly for test loop

Am I missing something? I have to run for a very long epoch for a huge number of sample so can not afford to wait to see the results then figure out things.

Thank you!


title: expected object of backend CPU but got backend CUDA for argument #4 'mat1'

body:
I am trying to re_train FC layer of resnet50 on my dataset. After loading the data set and pre-train model, I move both to the device but I am getting

    error:
    Expected object of backend CPU but got backend CUDA for argument #4 'mat1'

I searched a lot online found that mainly this kind of error occurs due to if the model is not moved or input to the model is not moved to GPU. I did both but still, no luck. Here are code and error.

    if steps % print_every == 0:
        test_loss = 0
        accuracy = 0
        resnet50.eval()
        with torch.no_grad():
            for inputs, labels in testloader:
                inputs, labels = inputs.to(device),labels.to(device)
                logps = resnet50.forward(inputs)
                batch_loss = criterion(logps, labels)
                test_loss += batch_loss.item()

                ps = torch.exp(logps)
                top_p, top_class = ps.topk(1, dim=1)
                equals = top_class == labels.view(*top_class.shape)

               accuracy+=torch.mean(equals.type(torch.FloatTensor)).item()
        train_losses.append(running_loss/len(trainloader))
        test_losses.append(test_loss/len(testloader))
        print(f"Epoch {epoch+1}/{Epochs}.. "
              f"Train loss: {running_loss/print_every:.3f}.. "
              f"Test loss: {test_loss/len(testloader):.3f}.. "
              f"Test accuracy: {accuracy/len(testloader):.3f}")
        running_loss = 0
        resnet50.train()


title: embedding layer outputs nan

body:
I am trying to learn a seq2seq model.
An embedding layer is located in the encoder and it sometimes outputs nan value after some iterations.
I cannot identify the reason.
How can I solve this??
The problem is the first emb_layer in the forward function in the code below.

class TransformerEncoder(nn.Module):
    def __init__(self, vocab_size, hidden_size=1024, num_layers=6, dropout=0.2, input_pad=1, batch_first=False, embedder=None, init_weight=0.1):
        super(TransformerEncoder, self).__init__()
        self.input_pad = input_pad
        self.vocab_size = vocab_size
        self.num_layers = num_layers
        self.embedder = embedder

        if embedder is not None:
            self.emb_layer = embedder
        else:
            self.emb_layer = nn.Embedding(vocab_size, hidden_size, padding_idx=1)

        self.positional_encoder = PositionalEncoder()
        self.transformer_layers = nn.ModuleList()
        for _ in range(num_layers):
            self.transformer_layers.append(
                    TransformerEncoderBlock(num_heads=8, embedding_dim=1024, dropout=dropout))

    def set_mask(self, inputs):
        self.input_mask = (inputs == self.input_pad).unsqueeze(1)

    def forward(self, inputs):
        x = self.emb_layer(inputs)
        x = self.positional_encoder(x)


title: How to optimize the inference script to get a faster prediction of the classifier?

body:
I have written the following prediction code that predicts from a trained classifier model. Now the prediction time is around 40s which I want to reduce as much as possible.

Can I do any optimization to my inference script or should I look for developments in training script?

import torch
import torch.nn as nn
from torchvision.models import resnet18
from torchvision.transforms import transforms
import matplotlib.pyplot as plt
import numpy as np
from torch.autograd import Variable
import torch.functional as F
from PIL import Image
import os
import sys
import argparse
import time
import json

parser = argparse.ArgumentParser(description = 'To Predict from a trained model')

parser.add_argument('-i','--image', dest = 'image_name', required = True, help='Path to the image file')
args = parser.parse_args()

def predict_image(image_path):
    print("prediciton in progress")
    image = Image.open(image_path)

    transformation = transforms.Compose([
        transforms.RandomResizedCrop(224),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
        ])

    image_tensor = transformation(image).float()
    image_tensor = image_tensor.unsqueeze_(0)

    if cuda:
        image_tensor.cuda()

    input = Variable(image_tensor)
    output = model(input)

    index = output.data.numpy().argmax()
    return index

def parameters():
    hyp_param = open('param_predict.txt','r')
    param = {}
    for line in hyp_param:
        l = line.strip('\n').split(':')

def class_mapping(index):
    with open("class_mapping.json") as cm:
        data = json.load(cm)
    if index == -1:
        return len(data)
    else:
        return data[str(index)]

def segregate():
    with open("class_mapping.json") as cm:
        data = json.load(cm)
    try:
        os.mkdir(seg_dir)
        print("Directory " , seg_dir ,  " Created ")
    except OSError:
        print("Directory " , seg_dir ,  " already created")
    for x in range (0,len(data)):
        dir_path="./"+seg_dir+"/"+data[str(x)]
        try:
            os.mkdir(dir_path)
            print("Directory " , dir_path ,  " Created ")
        except OSError:
            print("Directory " , dir_path ,  " already created")

path_to_model = "./models/"+'trained.model'
checkpoint = torch.load(path_to_model)
seg_dir="segregation_folder"

cuda = torch.cuda.is_available()

num_class = class_mapping(index=-1)
print num_class
model = resnet18(num_classes = num_class)

if cuda:
    model.load_state_dict(checkpoint)
else:
    model.load_state_dict(checkpoint, map_location = 'cpu')

model.eval()

if __name__ == "__main__":

    imagepath = "./Predict_Image/"+args.image_name
    since = time.time()
    img = Image.open(imagepath)
    prediction = predict_image(imagepath)
    name = class_mapping(prediction)
    print("Time taken = ",time.time()-since)

    print("Predicted Class: ",name)

The entire project can be found at
https://github.com/amrit-das/custom_image_classifier_pytorch/


title: How to read in .txt file (corpus) into torchtext in pytorch?

body:
How to read in .txt file (corpus) into torchtext in pytorrch?

I only see data.Dataset for example datasets and data.TabularData for csv, json, and tsv.

https://github.com/pytorch/text#data

https://torchtext.readthedocs.io/en/latest/data.html#dataset

It still works if I read it in using a Tabular dataset like this:
    test_file = data.TabularDataset(path=input_filepath, format='csv', fields=[('text', data.Field())])

But my dataset is not tabular, so I wanted to check to see if there was a better option.


title: Pytorch Subclass of nn.Module has no attribute 'parameters'

body:
Python Version: Python 3.8.5
Pytorch Version: '1.6.0'
I am defining LSTM, a subclass of nn.Module. I am trying to create an optimizer but I am getting the following error: torch.nn.modules.module.ModuleAttributeError: 'LSTM' object has no attribute 'paramters'
I have two code files, train.py and lstm_class.py (contain the LSTM class). I will try to produce a minimum working example, let me know if any other information is helpful.

The code in lstm_class.py:
import torch.nn as nn

class LSTM(nn.Module):

    def __init__(self, vocab_size, embedding_dim, hidden_dim, n_layers, drop_prob=0.2):
        super(LSTM, self).__init__()

        # network size parameters
        self.n_layers = n_layers
        self.hidden_dim = hidden_dim
        self.vocab_size = vocab_size
        self.embedding_dim = embedding_dim

        # the layers of the network
        self.embedding = nn.Embedding(self.vocab_size, self.embedding_dim)
        self.lstm = nn.LSTM(self.embedding_dim, self.hidden_dim, self.n_layers, dropout=drop_prob, batch_first=True)
        self.dropout = nn.Dropout(drop_prob)
        self.fc = nn.Linear(self.hidden_dim, self.vocab_size)

    def forward(self, input, hidden):
        # Defines forward pass, probably isn't relevant

    def init_hidden(self, batch_size):
        #Initializes hidden state, probably isn't relevant

The code in train.py
import torch
import torch.optim
import torch.nn as nn
import lstm_class

vocab_size = 1000
embedding_dim = 256
hidden_dim = 256
n_layers = 2

net = lstm_class.LSTM(vocab_size, embedding_dim, hidden_dim, n_layers)
optimizer = torch.optim.Adam(net.paramters(), lr=learning_rate)

I am getting the error on the last line written above. The full error message:
Traceback (most recent call last):
  File "train.py", line 58, in <module>
    optimizer = torch.optim.Adam(net.paramters(), lr=learning_rate)
  File "/usr/local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 771, in __getattr__
    raise ModuleAttributeError("'{}' object has no attribute '{}'".format(
torch.nn.modules.module.ModuleAttributeError: 'LSTM' object has no attribute 'paramters'

Any tips for how to fix this would be appreciated. Also as written above, let me know if anything else would be relevant. Thanks


title: Efficiently Create HDF5 Image Dataset for Neural Network Training with Memory Limitations

body:
I am having big image datasets to train CNN's on. Since I cannot load all the images into my RAM I plan to dump them into a HDF5 file (with h5py) and then iterate over the set batchwise, as suggested in

Most efficient way to use a large data set for PyTorch?

I tried creating an own dataset for every picture, located in the same group, which is very fast. But I could not figure out to iterate over all datasets in the group, except for accessing the set by its name.
As an alternative I tried putting all the images itereatively into one dataset by extending its shape, according to

How to append data to one specific dataset in a hdf5 file with h5py
and

incremental writes to hdf5 with h5py

but this is very slow. Is there a faster way to create a HDF5 dataset to iterate over?


title: Pytorch nn embeddings dimension size?

body:
What is the correct dimension size for nn embeddings in Pytorch? I'm doing batch training.

I'm just a little confused with what the dimensions of "self.embeddings" in the code below are supposed to be when I get "shape"?

self.embeddings = nn.Embedding(vocab_size, embedding_dim)


title: How to get probability of each image belonging to a class

body:
I am new to Pytorch. I trained and tested a linear classifier (nn.Linear) with an image data set that has 8 categories with the batch_size = 35.
While testing, I wanted to see what is the probability of the given image belonging to any of these 8 classes. That is why I printed output.data variable. But these numbers are above 1 and they are not summing into 1. (I attached the testing code)
So, my question is what does those numbers mean?

Thanks!

correct = 0
total = 0
with torch.no_grad():
    for data in dataloaders['test']:
        images, labels = data[0].to(device), data[1].to(device)
        outputs = model(images)
        print(outputs.data)

        _, predicted = torch.max(outputs.data, 1)
        print(predicted)
        total += labels.size(0)
        correct += (predicted == labels).sum().item()
print('Accuracy of the network on the 4000 test images: %d %%' % (
100 * correct / total))


title: Why doesn't converting the dtype of a Tensor fix "RuntimeError: expected scalar type Double but found Float"?

body:
The important part of my code looks like this:
def forward(self, x):
        x = T.tensor(x).to(self.device)
        x = x.type(T.DoubleTensor)
        x = self.conv1(x)
        ...

Yet I still get the error expected scalar type Double but found Float on the last line of that snippet. The line x = x.type(T.DoubleTensor) should fix that, right? I've also tried x = x.double() and  x = T.tensor(x, dtype = T.double).to(self.device) for the earlier line, and I still get the error. I'm at a loss, what am I doing wrong?


title: Freezing Individual Weights in Pytorch

body:
The following question is not a duplicate of How to apply layer-wise learning rate in Pytorch? because this question aims at freezing a subset of a tensor from training rather than the entire layer.

I am trying out a PyTorch implementation of Lottery Ticket Hypothesis.

For that, I want to freeze the weights in a model that are zero. Is the following a correct way to implement it?

for name, p in model.named_parameters():
            if 'weight' in name:
                tensor = p.data.cpu().numpy()
                grad_tensor = p.grad.data.cpu().numpy()
                grad_tensor = np.where(tensor == 0, 0, grad_tensor)
                p.grad.data = torch.from_numpy(grad_tensor).to(device)


title: Installing PyTorch3D fails with anaconda and pip

body:
I saw that more people seem to have the same issue, but it was not resolved. I am trying to install Pytorch3D with Anaconda and got the following PackageNotFound error.

Collecting package metadata (current_repodata.json): done Solving environment: failed with initial frozen solve. Retrying with flexible solve. Collecting package metadata (repodata.json): done Solving environment: failed with initial frozen solve. Retrying with flexible solve.
PackagesNotFoundError: The following packages are not available from current channels:
  - pytorch3d
Current channels:
  - https://conda.anaconda.org/pytorch3d/win-64
  - https://conda.anaconda.org/pytorch3d/noarch
  - https://repo.anaconda.com/pkgs/main/win-64
  - https://repo.anaconda.com/pkgs/main/noarch
  - https://repo.anaconda.com/pkgs/r/win-64
  - https://repo.anaconda.com/pkgs/r/noarch
  - https://repo.anaconda.com/pkgs/msys2/win-64
  - https://repo.anaconda.com/pkgs/msys2/noarch
To search for alternate channels that may provide the conda package you're looking for, navigate to
    https://anaconda.org
and use the search bar at the top of the page.

enter image description here
enter image description here

I have also tried using pip install 'git+https://github.com/facebookresearch/pytorch3d.git'
and get the following:

C:\Users\Alexandra>pip install 'git+https://github.com/facebookresearch/pytorch3d.git'
ERROR: Invalid requirement: "'git+https://github.com/facebookresearch/pytorch3d.git'"

C:\Users\Alexandra>pip install git+https://github.com/facebookresearch/pytorch3d.git
Collecting git+https://github.com/facebookresearch/pytorch3d.git
  Cloning https://github.com/facebookresearch/pytorch3d.git to c:\users\alexan~1\appdata\local\temp\pip-req-build-uspo7an4
  Running command git clone -q https://github.com/facebookresearch/pytorch3d.git 'C:\Users\ALEXAN~1\AppData\Local\Temp\pip-req-build-uspo7an4'
  ERROR: Error [WinError 2] The system cannot find the file specified while executing command git clone -q https://github.com/facebookresearch/pytorch3d.git 'C:\Users\ALEXAN~1\AppData\Local\Temp\pip-req-build-uspo7an4'
ERROR: Cannot find command 'git' - do you have 'git' installed and in your PATH?

I am on Windows 10, using python 3.8, PyTorch 1.5 and CUDA 10.2.
I am very new to python, so I have no idea how to fix this (you can tell, that I've never installed from git before..) (please be lenient!)
Thank you!

EDIT: Thank you for your answers. I did install Git and it got me a bit further, but still not completeing the build..screenshot
Also, interesting enough, when I run the commands

!pip install torch torchvision
!pip install 'git+https://github.com/facebookresearch/pytorch3d.git@stable'

in Google Collab it seems to work, but I cannot run it , let's say, in jupyter.
Any more ideas?


title: Sagemaker Training Job Not Uploading/Saving Training Model to S3 Output Path

body:
Ok I've been dealing with this issue in Sagemaker for almost a week and I'm ready to pull my hair out. I've got a custom training script paired with a data processing script in a BYO algorithm Docker deployment type scenario. It's a Pytorch model built with Python 3.x, and the BYO Docker file was originally built for Python 2, but I can't see an issue with the problem that I am having.....which is that after a successful training run Sagemaker doesn't save the model to the target S3 bucket.
I've searched far and wide and can't seem to find an applicable answer anywhere. This is all done inside a Notebook instance. Note: I am using this as a contractor and don't have full permissions to the rest of AWS, including downloading the Docker image.
Dockerfile:
FROM ubuntu:18.04

MAINTAINER Amazon AI <sage-learner@amazon.com>

RUN apt-get -y update && apt-get install -y --no-install-recommends \
         wget \
         python-pip \
         python3-pip3
         nginx \
         ca-certificates \
    && rm -rf /var/lib/apt/lists/*

RUN wget https://bootstrap.pypa.io/get-pip.py && python3 get-pip.py && \
    pip3 install future numpy torch scipy scikit-learn pandas flask gevent gunicorn && \
        rm -rf /root/.cache

ENV PYTHONUNBUFFERED=TRUE
ENV PYTHONDONTWRITEBYTECODE=TRUE
ENV PATH="/opt/program:${PATH}"

COPY decision_trees /opt/program
WORKDIR /opt/program

Docker Image Build:
%%sh

algorithm_name="name-this-algo"

cd container

chmod +x decision_trees/train
chmod +x decision_trees/serve

account=$(aws sts get-caller-identity --query Account --output text)

region=$(aws configure get region)
region=${region:-us-east-2}

fullname="${account}.dkr.ecr.${region}.amazonaws.com/${algorithm_name}:latest"

aws ecr describe-repositories --repository-names "${algorithm_name}" > /dev/null 2>&1

if [ $? -ne 0 ]
then
    aws ecr create-repository --repository-name "${algorithm_name}" > /dev/null
fi

# Get the login command from ECR and execute it directly
$(aws ecr get-login --region ${region} --no-include-email)

# Build the docker image locally with the image name and then push it to ECR
# with the full name.

docker build  -t ${algorithm_name} .
docker tag ${algorithm_name} ${fullname}

docker push ${fullname}

Env setup and session start:
common_prefix = "pytorch-lstm"
training_input_prefix = common_prefix + "/training-input-data"
batch_inference_input_prefix = common_prefix + "/batch-inference-input-data"

import os
from sagemaker import get_execution_role
import sagemaker as sage

sess = sage.Session()

role = get_execution_role()
print(role)

Training Directory, Image, and Estimator Setup, then a fit call:
TRAINING_WORKDIR = "a/local/directory"

training_input = sess.upload_data(TRAINING_WORKDIR, key_prefix=training_input_prefix)
print ("Training Data Location " + training_input)

account = sess.boto_session.client('sts').get_caller_identity()['Account']
region = sess.boto_session.region_name
image = '{}.dkr.ecr.{}.amazonaws.com/image-that-works:working'.format(account, region)

tree = sage.estimator.Estimator(image,
                       role, 1, 'ml.p2.xlarge',
                       output_path="s3://sagemaker-directory-that-definitely/exists",
                       sagemaker_session=sess)

tree.fit(training_input)

The above script is working, for sure. I have print statements in my script and they are printing the expected results to the console. This runs as it's supposed to, finishes up, and says that it's deploying model artifacts when IT DEFINITELY DOES NOT.
Model Deployment:
model = tree.create_model()
predictor = tree.deploy(1, 'ml.m4.xlarge')

This throws an error that the model can't be found. A call to aws sagemaker describe-training-job shows that the training was completed but I found that the time it took to upload the model was super fast, so obviously there's an error somewhere and it's not telling me. Thankfully it's not just uploading it to the aether.
{
            "Status": "Uploading",
            "StartTime": 1595982984.068,
            "EndTime": 1595982989.994,
            "StatusMessage": "Uploading generated training model"
        },

Here's what I've tried so far:

I've tried uploading it to a different bucket. I figured my permissions were the problem so I pointed it to one that I new allowed me to upload as I had done it before to that bucket. No dice.
I tried backporting the script to Python 2.x, but that caused more problems than it probably would have solved, and I don't really see how that would be the problem anyways.
I made sure the Notebook's IAM role has sufficient permissions, and it does have a SagemakerFullAccess policy

What bothers me is that there's no error log I can see. If I could be directed to that I would be happy too, but if there's some hidden Sagemaker kungfu that I don't know about I would be forever grateful.

EDIT
The training job runs and prints to both the Jupyter cell and CloudWatch as expected. I've since lost the cell output in the notebook but below is the last few lines in CloudWatch. The first number is the epoch and the rest are various custom model metrics.


title: PyTorch's torch.nn.functional.interpolate behaving strangely

body:
I'm having issues with PyTorch's tensor-resizing options. In the following code, x is a dataset of 888 64x64 RGB images of pokemon. xs is to be a dictionary of the same dataset at different resolutions.
def load_data():
    pokemon = []
    for png in os.listdir("pokemon"):
        pokemon.append(imageio.imread("pokemon/" + png))
    pokemon = np.array(pokemon)
    s = pokemon.shape
    pokemon = pokemon.reshape((s[0], s[-1], s[1], s[2]))
    pokemon = (pokemon.astype(np.float32) - 127.5)/127.5
    return(pokemon)
x = load_data()

ss = [2, 1, 1/2, 1/4, 1/8, 1/16]
xs = {s : nn.functional.interpolate(torch.from_numpy(x), scale_factor = s, mode='nearest') for s in ss}

As expected, the output shapes include all 888 RGB images with new heights and widths.
print("Data shape:")
for x in xs.values():
    print(" ", x.shape)

> Data shape:
>  torch.Size([888, 3, 128, 128])
>  torch.Size([888, 3, 64, 64])
>  torch.Size([888, 3, 32, 32])
>  torch.Size([888, 3, 16, 16])
>  torch.Size([888, 3, 8, 8])
>  torch.Size([888, 3, 4, 4])

However, the images weren't sensibly resized.

When s = 1, the pokemon are in their ordinary 64x64 format, as expected:

When s = 2, the pokemon become Pollocks, overlapping in different colors:

When s = 1/2, the pokemon are cut into pieces and rearranged:

Is there an alternative PyTorch tool for resizing? I encounter roughly the same problem using AvgPool2d and Upsample instead. I will be resizing non-image tensors, too, so PIL isn't always an option. In any case, using the GPU instead of CPU would be helpful.


title: Multi-class for sentence classification with pytorch (Using nn.LSTM)

body:
I have this network, that I took from this tutorial, and I want to have sentences as input (Which is already done) and just a one line tensor as a result.

From the tutorial, this sentence “John’s dog likes food”, gets a 1 column tensor returned:

tensor([[-3.0462, -4.0106, -0.6096],
[-4.8205, -0.0286, -3.9045],
[-3.7876, -4.1355, -0.0394],
[-0.0185, -4.7874, -4.6013]])

...and class list:

tag_list[ “name”, “verb”, “noun”]

Each line has the probability of a tag being associated with the word. (The first word has [-3.0462, -4.0106, -0.6096] vector where the last element corresponds to the maximum scoring tag, "noun")

The tutorial’s dataset looks like this:

training_data = [
    ("The dog ate the apple".split(), ["DET", "NN", "V", "DET", "NN"]),
    ("Everybody read that book".split(), ["NN", "V", "DET", "NN"])
]

And I want mine to be of this format:

training_data = [
    ("Hello world".split(), ["ONE"]),
    ("I am dog".split(), ["TWO"]),
    ("It's Britney glitch".split(), ["THREE"])
]

The parameters are defined as:

class LSTMTagger(nn.Module):
    def __init__(self, embedding_dim, hidden_dim, vocab_size, tagset_size):
        super(LSTMTagger, self).__init__()
        self.hidden_dim = hidden_dim
        self.word_embeddings = nn.Embedding(vocab_size, embedding_dim)
        self.lstm = nn.LSTM(embedding_dim, hidden_dim)
        self.hidden2tag = nn.Linear(hidden_dim, tagset_size)

    def forward(self, sentence):
        embeds      = self.word_embeddings(sentence)
        lstm_out, _ = self.lstm(embeds.view(len(sentence), 1, -1))
        tag_space   = self.hidden2tag(lstm_out.view(len(sentence), -1))
        tag_scores  = F.log_softmax(tag_space, dim=1)
        return tag_scores

As of now, the sizes from input and output are not matching and i get:
ValueError: Expected input batch_size (2) to match target batch_size (1).

The criterion function doesn't accept the input due to size missmatch it seems:

loss        = criterion(tag_scores, targets)

I've read the last layer could be defined as nn.Linear in order to squash the outputs but i can't seem to get any results. Tried other loss functions

How can I change it in order for the model to classify the sentence , and not each word, as in the original tutorial?


title: Why grad_output requires_grad is False in pytorch?

body:
Here is the customLayer.py.

I am quite confused about the following things:

The input of the inner layer is not a Variable. Then in backward it becomes a Variable and requires gradient. Why?
grad_output is a Variable yet requires_grad is False. Why is not true?
In my custom layer, I need customize forward and backward operations. It is quite complicated. See the same link. I have posted questions in it.


title: Incorrect results obtained on running a model in LibTorch that was trained and exported from PyTorch

body:
I am trying to export a trained model along with weights for inference in C++ using LibTorch. However, the output tensor results do not match.

The shape of the output tensor is the same.

model = FCN()
state_dict = torch.load('/content/gdrive/My Drive/model/trained_model.pth')
model.load_state_dict(state_dict)
example = torch.randn(1, 3, 768, 1024)
traced_script_module = torch.jit.trace(model, example)
traced_script_module.save('/content/gdrive/My Drive/model/mymodel.pt')

However some warnings are generated which I think maybe causing the incorrect results to be generated.

  /usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:137:
  TracerWarning: Converting a tensor to a Python index might cause the
  trace to be incorrect. We can't record the data flow of Python values,
  so this value will be treated as a constant in the future. This means
  that the trace might not generalize to other inputs!
  /usr/local/lib/python3.6/dist-packages/torch/tensor.py:435:
  RuntimeWarning: Iterating over a tensor might cause the trace to be
  incorrect. Passing a tensor of different shape won't change the number
  of iterations executed (and might lead to errors or silently give
  incorrect results).'incorrect results).', category=RuntimeWarning)

Following is the LibTorch code to generate the output tensor

at::Tensor predict(std::shared_ptr<torch::jit::script::Module> model, at::Tensor &image_tensor) {
    std::vector<torch::jit::IValue> inputs;
    inputs.push_back(image_tensor);

    at::Tensor result = model->forward(inputs).toTensor();

    return result;
}

Has anyone tried using a trained PyTorch model in LibTorch?


title: Pytorch RuntimeError:CUDA out of memory when training graph attention network

body:
I am trying to reproduce the results of a model proposed in a paper with pytorch. This model uses the atttion mechanism to achieve the purpose of relationship prediction in the knowledge graph. It is mainly divided into two parts: graph attention network(GAT) and convolution network. The code for training GAT is as follows:

def train_gat(args):

    # Creating the gat model here.
    ####################################

    current_batch_2hop_indices = torch.tensor([])
    if(args.use_2hop):
        current_batch_2hop_indices = Corpus_.get_batch_nhop_neighbors_all(args,
                                                                          Corpus_.unique_entities_train, node_neighbors_2hop)

    if CUDA:
        current_batch_2hop_indices = Variable(
            torch.LongTensor(current_batch_2hop_indices)).cuda()
    else:
        current_batch_2hop_indices = Variable(
            torch.LongTensor(current_batch_2hop_indices))

    print("Defining model")

    print(
        "\nModel type -> GAT layer with {} heads used , Initital Embeddings training".format(args.nheads_GAT[0]))
    model_gat = SpKBGATModified(entity_embeddings, relation_embeddings, args.entity_out_dim, args.entity_out_dim,
                                args.drop_GAT, args.alpha, args.nheads_GAT)

    if CUDA:
        model_gat.cuda()
        model_gat=torch.nn.DataParallel(model_gat,device_ids=device_ids)

    optimizer = torch.optim.Adam(
        model_gat.parameters(), lr=args.lr, weight_decay=args.weight_decay_gat)

    scheduler = torch.optim.lr_scheduler.StepLR(
        optimizer, step_size=500, gamma=0.5, last_epoch=-1)

    gat_loss_func = nn.MarginRankingLoss(margin=args.margin)

    epoch_losses = []   # losses of all epochs
    print("Number of epochs {}".format(args.epochs_gat))

    for epoch in range(args.epochs_gat):
        print("\nepoch-> ", epoch)
        random.shuffle(Corpus_.train_triples)
        Corpus_.train_indices = np.array(
            list(Corpus_.train_triples)).astype(np.int32)

        model_gat.train()  # getting in training mode
        start_time = time.time()
        epoch_loss = []

        if len(Corpus_.train_indices) % args.batch_size_gat == 0:
            num_iters_per_epoch = len(
                Corpus_.train_indices) // args.batch_size_gat
        else:
            num_iters_per_epoch = (
                len(Corpus_.train_indices) // args.batch_size_gat) + 1

        for iters in range(num_iters_per_epoch):
            start_time_iter = time.time()
            train_indices, train_values = Corpus_.get_iteration_batch(iters)
            if CUDA:
                train_indices = Variable(
                    torch.LongTensor(train_indices)).cuda()
                train_values = Variable(torch.FloatTensor(train_values)).cuda()

            else:
                train_indices = Variable(torch.LongTensor(train_indices))
                train_values = Variable(torch.FloatTensor(train_values))

            # forward pass

            entity_embed, relation_embed = model_gat(
                Corpus_, Corpus_.train_adj_matrix, train_indices, current_batch_2hop_indices)

            optimizer.zero_grad()

            loss = batch_gat_loss(
                gat_loss_func, train_indices, entity_embed, relation_embed)

            loss.backward()
            optimizer.step()

            epoch_loss.append(loss.data.item())

            end_time_iter = time.time()

            print("Iteration-> {0}  , Iteration_time-> {1:.4f} , Iteration_loss {2:.4f}".format(
                iters, end_time_iter - start_time_iter, loss.data.item()))

        scheduler.step()
        print("Epoch {} , average loss {} , epoch_time {}".format(
            epoch, sum(epoch_loss) / len(epoch_loss), time.time() - start_time))
        epoch_losses.append(sum(epoch_loss) / len(epoch_loss))

        save_model(model_gat, args.data, epoch,
                   args.output_folder)

The code to calculate the loss of the GAT is as follows:

def batch_gat_loss(gat_loss_func, train_indices, entity_embed, relation_embed):
    len_pos_triples = int(
        train_indices.shape[0] / (int(args.valid_invalid_ratio_gat) + 1))

    pos_triples = train_indices[:len_pos_triples]
    neg_triples = train_indices[len_pos_triples:]

    pos_triples = pos_triples.repeat(int(args.valid_invalid_ratio_gat), 1)

    source_embeds = entity_embed[pos_triples[:, 0]]
    relation_embeds = relation_embed[pos_triples[:, 1]]
    tail_embeds = entity_embed[pos_triples[:, 2]]

    x = source_embeds + relation_embeds - tail_embeds
    pos_norm = torch.norm(x, p=1, dim=1)

    source_embeds = entity_embed[neg_triples[:, 0]]
    relation_embeds = relation_embed[neg_triples[:, 1]]
    tail_embeds = entity_embed[neg_triples[:, 2]]

    x = source_embeds + relation_embeds - tail_embeds
    neg_norm = torch.norm(x, p=1, dim=1)

    y = -torch.ones(int(args.valid_invalid_ratio_gat) * len_pos_triples).cuda()

    loss = gat_loss_func(pos_norm, neg_norm, y)
    return loss

The datasets used to train the model in this paper are FB15k-237, NELL-995 and umls. The numbers of triples in their training set are: 272115 ,149678 and 5216. I successfully reproduced the results on all three datasets. However, when I tried to use a new dataset YAGO3-10 (with 1079040 training data) to train the model, I got a GPU memory overflow error. The error message is as follows：

Traceback (most recent call last):
  File "main.py", line 366, in <module>
    train_gat(args)
  File "main.py", line 240, in train_gat
    gat_loss_func, train_indices, entity_embed, relation_embed)
  File "main.py", line 149, in batch_gat_loss
    x = source_embeds + relation_embeds - tail_embeds
RuntimeError: CUDA out of memory. Tried to allocate 1.61 GiB (GPU 0; 15.77 GiB total capacity; 14.24 GiB already allocated; 120.25 MiB free; 440.29 MiB cached)

The methods I have tried are as follows:

Reduce the batch size. After reduction, it will overflow during backpropagation. The error message is as follows:

Traceback (most recent call last):
  File "main.py", line 366, in <module>
    train_gat(args)
  File "main.py", line 242, in train_gat
    loss.backward()
  File "/home/user1/anaconda3/envs/stw/lib/python3.6/site-packages/torch/tensor.py", line 107, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/user1/anaconda3/envs/stw/lib/python3.6/site-packages/torch/autograd/__init__.py", line 93, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: CUDA out of memory. Tried to allocate 2.60 GiB (GPU 0; 15.77 GiB total capacity; 11.86 GiB already allocated; 2.42 GiB free; 511.96 MiB cached)
Use torch.cuda.empty_cache () to remove useless variables before backpropagation. However, the maximum memory usage does not seem to change.
Use dataparallel to train model on two GPUs, but did not run successfully, it seems because some model parameters are on different GPUs. The error message is as follows:

Traceback (most recent call last):
  File "main.py", line 377, in <module>
    train_gat(args)
  File "main.py", line 244, in train_gat
    Corpus_, Corpus_.train_adj_matrix_dim1, Corpus_.train_adj_matrix_dim00, Corpus_.train_adj_matrix_dim01 ,train_indices, current_batch_2hop_indices)
  File "/home/user1/anaconda3/envs/stw/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/user1/anaconda3/envs/stw/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 152, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/home/user1/anaconda3/envs/stw/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 162, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/home/user1/anaconda3/envs/stw/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 83, in parallel_apply
    raise output
  File "/home/user1/anaconda3/envs/stw/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 59, in _worker
    output = module(*input, **kwargs)
  File "/home/user1/anaconda3/envs/stw/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/user1/cxy/KBAT-2/models.py", line 148, in forward
    edge_list, edge_type, edge_embed, edge_list_nhop, edge_type_nhop)
  File "/home/user1/anaconda3/envs/stw/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/user1/cxy/KBAT-2/models.py", line 55, in forward
    for att in self.attentions], dim=1)
  File "/home/user1/cxy/KBAT-2/models.py", line 55, in <listcomp>
    for att in self.attentions], dim=1)
  File "/home/user1/anaconda3/envs/stw/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/user1/cxy/KBAT-2/layers.py", line 140, in forward
    edge_m = self.a.mm(edge_h)
RuntimeError: arguments are located on different GPUs at /opt/conda/conda-bld/pytorch_1556653183467/work/aten/src/THC/generic/THCTensorMathBlas.cu:255
Use apex package for float16 calculation. However, some parameters will report nan values, which may be the cause of using float16.

The latter two methods may work, but I haven't run the code successfully. I would like to ask if anyone has some experience using the above method, or has better suggestions.

Due to space limitations, I cannot show all the code. The code link of the model is as follows:
https://github.com/deepakn97/relationPrediction


title: Very weird behaviour when running the same deep learning code in two different GPUs

body:
I am training networks using pytorch framework. I had K40 GPU in my computer. Last week, I added 1080 to the same computer.

In my first experiment, I observed identical results in both GPU. Then, I tried my second code on both GPUs. In this case, I "constantly" got good results in K40 while getting "constantly" awful results in 1080 for "exactly the same code".

First, I thought the only reason for getting such diverse outputs would be the random seeds in the codes. So, I fixed the seeds like this:

torch.manual_seed(3)
torch.cuda.manual_seed_all(3)
numpy.random.seed(3)

But, this did not solve the issue. I believe issue cannot be randomness because I was "constantly" getting good results in K40 and "constantly" getting bad results in 1080. Moreover, I tried exactly the same code in 2 other computers and 4 other 1080 GPUs and always achieved good results. So, problem has to be about the 1080 I recently plugged in.

I suspect problem might be about driver, or the way I installed pytorch. But, it is still weird that I only get bad results for "some" of the experiments. For the other experiments, I had the identical results.

Can anyone help me on this?


title: Pytorch passing PackSequence argument to LSTM

body:
As per my understanding, pack_sequence and pack_padded_sequence returns a PackedSequence, for which its data attribute should always be 1 dimension.

However, the following code gives the error:
RuntimeError: input must have 2 dimensions, got 1.

import torch
import torch.nn.utils.rnn as rnn_utils
a = torch.Tensor([1, 2, 3])
b = torch.Tensor([4, 5])
c = torch.Tensor([6])
seq = rnn_utils.pack_sequence([a, b, c])
lstm = torch.nn.LSTM(
    input_size=3,
    hidden_size=3
)
lstm(seq)

I've tried batch_first=True, that did not work.

The source file of error code: https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/rnn.py#L127


title: What's the difference between tf.nn.ctc_loss with pytorch.nn.CTCLoss

body:
For the same input and label:

the output of pytorch.nn.CTCLoss is 5.74,
the output of tf.nn.ctc_loss is 129.69,
but the output of math.log(tf ctc loss) is 4.86

So what's the difference between pytorch.nn.CTCLoss with tf.nn.ctc_loss?

tf: 1.13.1
pytorch: 1.1.0

I had try to these:

log_softmax the input, and then send it to pytorch.nn.CTCLoss,
tf.nn.log_softmax the input, and then send it to tf.nn.ctc_loss
directly send the input to tf.nn.ctc_loss
directly send the input to tf.nn.ctc_loss, and then math.log(output of tf.nn.ctc_loss)

In the case 2, case 3, and case 4, the result of calculation is difference from pytorch.nn.CTCLoss

from torch import nn
import torch
import tensorflow as tf
import math

time_step = 50  # Input sequence length
vocab_size = 20  # Number of classes
batch_size = 16  # Batch size
target_sequence_length = 30  # Target sequence length

def dense_to_sparse(dense_tensor, sequence_length):
    indices = tf.where(tf.sequence_mask(sequence_length))
    values = tf.gather_nd(dense_tensor, indices)
    shape = tf.shape(dense_tensor, out_type=tf.int64)
    return tf.SparseTensor(indices, values, shape)

def compute_loss(x, y, x_len):
    ctclosses = tf.nn.ctc_loss(
        y,
        tf.cast(x, dtype=tf.float32),
        x_len,
        preprocess_collapse_repeated=False,
        ctc_merge_repeated=False,
        ignore_longer_outputs_than_inputs=False
    )
    ctclosses = tf.reduce_mean(ctclosses)

    with tf.Session() as sess:
        ctclosses = sess.run(ctclosses)
        print(f"tf ctc loss: {ctclosses}")
        print(f"tf log(ctc loss): {math.log(ctclosses)}")

minimum_target_length = 10

ctc_loss = nn.CTCLoss(blank=vocab_size - 1)
x = torch.randn(time_step, batch_size, vocab_size)  # [size] = T,N,C
y = torch.randint(0, vocab_size - 2, (batch_size, target_sequence_length), dtype=torch.long)  # low, high, [size]

x_lengths = torch.full((batch_size,), time_step, dtype=torch.long)  # Length of inputs
y_lengths = torch.randint(minimum_target_length, target_sequence_length, (batch_size,),
                          dtype=torch.long)  # Length of targets can be variable (even if target sequences are constant length)

loss = ctc_loss(x.log_softmax(2).detach(), y, x_lengths, y_lengths)
print(f"torch ctc loss: {loss}")

x = x.numpy()
y = y.numpy()
x_lengths = x_lengths.numpy()
y_lengths = y_lengths.numpy()
x = tf.cast(x, dtype=tf.float32)
y = tf.cast(dense_to_sparse(y, y_lengths), dtype=tf.int32)
compute_loss(x, y, x_lengths)

I expect the output of tf.nn.ctc_loss is same with the output of pytorch.nn.CTCLoss, but actually they are not, but how can i make them same?


title: PyTorch regression is producing the same numbers as prediction

body:
I have applied a simple NN for regression to the traditional boston housing price dataset. The problem I’m getting is, when I make predictions using the trained model it always predicts the same numbers. Here is my code:

import numpy as  np
import pandas as pd
from sklearn import datasets
data = datasets.load_boston()

X = pd.DataFrame(data.data, columns=data.feature_names)
Y = pd.DataFrame(data.target, columns=["MEDV"])

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.20, random_state=1234)

import torch

x_train = torch.tensor(X_train.values, dtype=torch.float)
y_train = torch.tensor(y_train.values, dtype=torch.float)
x_test = torch.tensor(X_test.values, dtype=torch.float)
y_test = torch.tensor(y_test.values, dtype=torch.float)

import torch.nn.functional as F

class Net(torch.nn.Module):
    def __init__(self, n_feature, n_hidden, n_output):
        super(Net, self).__init__()
        self.hidden = torch.nn.Linear(n_feature, n_hidden)
        self.predict = torch.nn.Linear(n_hidden, n_output)

    def forward(self, x):
        x = F.relu(self.hidden(x))
        x = self.predict(x)
        return x

net = Net(n_feature=13, n_hidden=50, n_output=1)
optimizer = torch.optim.SGD(net.parameters(), lr=0.2)
loss_func = torch.nn.MSELoss()

for t in range(200):
    prediction = net(x_train)

    loss = loss_func(prediction, y_train)
    print(t, loss.item())
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

Here is the loss per epoch log:

0 6414.029296875
1 3.883837028532696e+19
2 5.963952279684907e+18
3 2.1470207536047063e+18
4 7.729279495852524e+17
5 2.7825424945486234e+17
6 1.001715136546734e+17
7 3.606170935335322e+16
8 1.2982211888283648e+16
9 4673591211720704.0
10 1682493577101312.0
11 605696890503168.0
12 218050926215168.0
13 78498328739840.0
14 28259370663936.0
15 10173364043776.0
16 3662410940416.0
17 1318467796992.0
18 474648543232.0
19 170873470976.0
20 61514457088.0
21 22145208320.0
22 7972275200.0
23 2870019072.0
24 1033206912.0
25 371954528.0
26 133903616.0
27 48205360.0
28 17353982.0
29 6247483.5
30 2249145.25
31 809743.25
32 291558.65625
33 105012.1171875
34 37855.3984375
35 13678.982421875
36 4975.46826171875
37 1842.20361328125
38 714.2283325195312
39 308.15716552734375
40 161.97158813476562
41 109.34477233886719
42 90.39911651611328
43 83.57867431640625
44 81.12332153320312
45 80.23938751220703
46 79.92117309570312
47 79.8066177368164
48 79.76537322998047
49 79.75053405761719
50 79.74518585205078
51 79.7432632446289
52 79.74256896972656
53 79.74231719970703
54 79.74223327636719
55 79.74219512939453
56 79.7421875
57 79.74217987060547
58 79.74217987060547
59 79.74217987060547
60 79.74217987060547
61 79.74217987060547
62 79.74217987060547
63 79.74217987060547
64 79.74217987060547
65 79.74217987060547
66 79.74217987060547
67 79.74217987060547
68 79.74217987060547
69 79.74217987060547
70 79.74217987060547
71 79.74217987060547
72 79.74217987060547
73 79.74217987060547
74 79.74217987060547
75 79.74217987060547
76 79.74217987060547
77 79.74217987060547
78 79.74217987060547
79 79.74217987060547
80 79.74217987060547
81 79.74217987060547
82 79.74217987060547
83 79.74217987060547
84 79.74217987060547
85 79.74217987060547
86 79.74217987060547
87 79.74217987060547
88 79.74217987060547
89 79.74217987060547
90 79.74217987060547
91 79.74217987060547
92 79.74217987060547
93 79.74217987060547
94 79.74217987060547
95 79.74217987060547
96 79.74217987060547
97 79.74217987060547
98 79.74217987060547
99 79.74217987060547
100 79.74217987060547
101 79.74217987060547
102 79.74217987060547
103 79.74217987060547
104 79.74217987060547
105 79.74217987060547
106 79.74217987060547
107 79.74217987060547
108 79.74217987060547
109 79.74217987060547
110 79.74217987060547
111 79.74217987060547
112 79.74217987060547
113 79.74217987060547
114 79.74217987060547
115 79.74217987060547
116 79.74217987060547
117 79.74217987060547
118 79.74217987060547
119 79.74217987060547
120 79.74217987060547
121 79.74217987060547
122 79.74217987060547
123 79.74217987060547
124 79.74217987060547
125 79.74217987060547
126 79.74217987060547
127 79.74217987060547
128 79.74217987060547
129 79.74217987060547
130 79.74217987060547
131 79.74217987060547
132 79.74217987060547
133 79.74217987060547
134 79.74217987060547
135 79.74217987060547
136 79.74217987060547
137 79.74217987060547
138 79.74217987060547
139 79.74217987060547
140 79.74217987060547
141 79.74217987060547
142 79.74217987060547
143 79.74217987060547
144 79.74217987060547
145 79.74217987060547
146 79.74217987060547
147 79.74217987060547
148 79.74217987060547
149 79.74217987060547
150 79.74217987060547
151 79.74217987060547
152 79.74217987060547
153 79.74217987060547
154 79.74217987060547
155 79.74217987060547
156 79.74217987060547
157 79.74217987060547
158 79.74217987060547
159 79.74217987060547
160 79.74217987060547
161 79.74217987060547
162 79.74217987060547
163 79.74217987060547
164 79.74217987060547
165 79.74217987060547
166 79.74217987060547
167 79.74217987060547
168 79.74217987060547
169 79.74217987060547
170 79.74217987060547
171 79.74217987060547
172 79.74217987060547
173 79.74217987060547
174 79.74217987060547
175 79.74217987060547
176 79.74217987060547
177 79.74217987060547
178 79.74217987060547
179 79.74217987060547
180 79.74217987060547
181 79.74217987060547
182 79.74217987060547
183 79.74217987060547
184 79.74217987060547
185 79.74217987060547
186 79.74217987060547
187 79.74217987060547
188 79.74217987060547
189 79.74217987060547
190 79.74217987060547
191 79.74217987060547
192 79.74217987060547
193 79.74217987060547
194 79.74217987060547
195 79.74217987060547
196 79.74217987060547
197 79.74217987060547
198 79.74217987060547
199 79.74217987060547

After training the model, I’ve used the following code to make predictions:

with torch.no_grad():
    y_val = net(x_test)

When I print the predictions I get the following results:

tensor([[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099]])


title: pytorch : Runtime error using Distributed with gloo

body:
I am trying to use the PyTorch distributed package and gloo backend.

But, I get the following error.

Master:

Traceback (most recent call last):
  File "Distributed.py", line 169, in <module>
    init_processes(args.rank, size, run)
  File "Distributed.py", line 80, in init_processes
    dist.init_process_group(backend=backend, rank=rank, world_size=size)
  File "/usr/local/lib/python3.5/dist-packages/torch/distributed/__init__.py", line 49, in init_process_group
    group_name, rank)
RuntimeError: [enforce fail at /pytorch/torch/lib/gloo/gloo/transport/tcp/device.cc:127] rp != nullptr. Unable to find address for: <My specified Ip> at /pytorch/torch/lib/THD/process_group/General.cpp:17

And otrher worker get:

RuntimeError: [enforce fail at /pytorch/torch/lib/gloo/gloo/transport/tcp/device.cc:127] rp != nullptr. Unable to find address for: 10.37.0.1 at /pytorch/torch/lib/THD/process_group/General.cpp:17

Below is how I initialize:

def init_processes(rank, size, fn, backend='gloo'):
    """ Initialize the distributed environment. """

    os.environ['MASTER_ADDR'] = <My specified Ip>
    os.environ['MASTER_PORT'] = '8888' if rank == 0 else '31566'

    print("Init Processes ->", 'backend:', backend, 'rank:', rank, 'MASTER_ADDR:', os.environ['MASTER_ADDR'],
          'MASTER_PORT:', os.environ['MASTER_PORT'])
    dist.init_process_group(backend=backend, rank=rank, world_size=size)
    fn(rank, size)

Any thoughts on what may be causing this or how I can fix it ? Thx

ps. I use docker env with python 3.5 and PyTorch 0.3.1 with cuda9.0.


title: Error executing pytorch code AttributeError: 'function' object has no attribute 'Variable'

body:
Trying to run pytorch code from github from following repo

https://github.com/jalola/improved-wgan-pytorch

The error is at the following statement in the file gan_train.py

   writer.add_scalar('data/disc_cost', disc_cost, iteration)

The error is as follows

File "/home/anaconda/new_anaconda_3.5/lib/python3.6/site-packages/tensorboardX/writer.py", line 272, in add_scalar
    self.file_writer.add_summary(scalar(tag, scalar_value), global_step)

  File "/home/anaconda/new_anaconda_3.5/lib/python3.6/site-packages/tensorboardX/summary.py", line 86, in scalar
    scalar = makenp(scalar)

  File "/home/anaconda/new_anaconda_3.5/lib/python3.6/site-packages/tensorboardX/x2num.py", line 15, in makenp
    return pytorch_np(x, modality)

  File "/home/anaconda/new_anaconda_3.5/lib/python3.6/site-packages/tensorboardX/x2num.py", line 24, in pytorch_np
    if isinstance(x, torch.autograd.variable.Variable):
AttributeError: 'function' object has no attribute 'Variable'

Any ideas, I am not very proficient in pytorch. It seems something is wrong with the tensorboard API.

Thanks a lot.

Shan


title: Converting a list of lists and scalars to a list of PyTorch tensors throws warning

body:
I was converting a list of lists to a PyTorch tensor and got a warning message. The conversion itself isn't difficult. For example:
>>> import torch
>>> thing = [[1, 2, 3, 4, 5], [2, 3], 2, 3]
>>> thing_tensor = list(map(torch.tensor, thing))

I get the warning:
home/user1/files/module.py:1: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).

and am wondering what the reason may be. Is there any other way to convert the data into a tensor that I'm not aware of? Thanks.


title: Can't replace classifier on Densenet121 in pytorch

body:
I am trying to do some transfer learning using this github DenseNet121 model (https://github.com/gaetandi/cheXpert.git). I'm running into issues resizing the classification layer from 14 to 2 outputs.

Relevant part of the github code is:

class DenseNet121(nn.Module):
    """Model modified.
    The architecture of our model is the same as standard DenseNet121
    except the classifier layer which has an additional sigmoid function.
    """
    def __init__(self, out_size):
        super(DenseNet121, self).__init__()
        self.densenet121 = torchvision.models.densenet121(pretrained=True)
        num_ftrs = self.densenet121.classifier.in_features
        self.densenet121.classifier = nn.Sequential(
            nn.Linear(num_ftrs, out_size),
            nn.Sigmoid()
        )
def forward(self, x):
    x = self.densenet121(x)
    return x

I load and init with:

# initialize and load the model
model = DenseNet121(nnClassCount).cuda()
model = torch.nn.DataParallel(model).cuda()
modeldict = torch.load("model_ones_3epoch_densenet.tar")
model.load_state_dict(modeldict['state_dict'])

It looks like DenseNet doesn't split layers up into children so model = nn.Sequential(*list(modelRes.children())[:-1]) won't work.

model.classifier = nn.Linear(1024, 2) seems to work on default DenseNets, but with the modified classifier (additional sigmoid function) here it ends up just adding an additional classifier layer without replacing the original.

I've tried

model.classifier = nn.Sequential(
    nn.Linear(1024, dset_classes_number),
    nn.Sigmoid()
)

But am having the same added instead of replaced classifier issue:

...
      )
      (classifier): Sequential(
        (0): Linear(in_features=1024, out_features=14, bias=True)
        (1): Sigmoid()
      )
    )
  )
  (classifier): Sequential(
    (0): Linear(in_features=1024, out_features=2, bias=True)
    (1): Sigmoid()
  )
)


I am trying to run Image Point augmentation using fastaiv2. When the developers run it in their environment on a fresh install, it works fine, but when I do in colab I cannot get it to work.

I've looked into the aug_transforms function to see just where this problem is coming up in and it has to do with augmenting the image points.

Here is the full steps:

!pip install git+https://github.com/fastai/fastai_dev > /dev/null
!pip install Pillow --upgrade # to get libraries

from fastai2.basics import *
from fastai2.callback.all import *
from fastai2.vision.all import *

path = untar_data(URLs.BIWI_HEAD_POSE)
cal = np.genfromtxt(path/'01'/'rgb.cal', skip_footer=6)
def img2txt_name(f): return path/f'{str(f)[:-7]}pose.txt'

def convert_biwi(coords:array):
  x = coords[0] * cal[0][0]/coords[2] + cal[0][2]
  y = coords[1] * cal[1][1]/coords[2] + cal[1][2]
  return tensor([x,y])

def get_ctr(f:Path):
  ctr = np.genfromtxt(img2txt_name(f), skip_header=3)
  return convert_biwi(ctr)

def get_ip(img:PILImage, pts:array): return TensorPoint.create(pts, sz=img.size)

splitter = FuncSplitter(lambda o: o.parent.name=='13')
dblock = DataBlock(blocks=(ImageBlock, PointBlock),
                   get_items=get_image_files,
                   splitter=splitter,
                   get_y=get_ctr)

batch_tfms = [*aug_transforms(size=(120,160), max_warp=0), Normalize(*imagenet_stats)]

dbunch = dblock.databunch(path, path=path, bs=64, batch_tfms=batch_tfms)
dbunch.show_batch(max_n=9, figsize=(9,6)) # breaks here

The expected result was to show a batch of augmented images, instead I get a stack trace with the following RuntimeError:

/usr/local/lib/python3.6/dist-packages/fastai2/vision/augment.py in affine_coord(x, mat, coord_tfm, sz, mode, pad_mode)
    260     if sz is None: sz = x._meta.get('sz', None)
    261     if coord_tfm is not None: x = coord_tfm(x, invert=True)
--> 262     if mat is not None: x = (x - mat[:,:,2].unsqueeze(1)) @ torch.inverse(mat[:,:,:2].transpose(1,2))
    263     return TensorPoint(x, sz=sz)
    264

RuntimeError: inverse_cuda: For batch 0: U(19536384,19536384) is zero, singular U.

Any thoughts on how to go about this? As I also tried this on my fresh machine and it worked fine.

Thanks!


title: one of the variables needed for gradient computation has been modified by an inplace operation

body:
Here is my LossFunction, when I use this function, it will make this error.
And I have tested that using the nn.L1Loss() instead of my LossFunction, and the network is ok.
what should I do? Thanks for your help!

class LossV1(nn.Module):
    def __init__(self,weight=1,pos_weight=1,scale_factor=2.5):
        super(LossV1,self).__init__()
        self.weight = weight
        self.pos_weight = pos_weight
        self.scale_factor = scale_factor

    def forward(self,pred,truth):
        objmask = torch.tensor(truth[:,:6,:,:],dtype=torch.float32,requires_grad=False)
        #没有物体的Boxes，置信度损失*0.4
        objmask[objmask<0.65] = 0.4
        #辅助Boxes，系数0.8
        objmask[(objmask>0.649)*(objmask<0.949)] = 0.8
        objLoss = torch.sum(objmask*self.myBCEWithLogitsLoss(pred[:,:6,:,:],truth[:,:6,:,:]))
        #没有物体的Boxes，只计算置信度损失
        objmask[objmask<0.41] = 0
        personLoss = torch.sum(objmask*self.myBCEWithLogitsLoss(pred[:,6:12,:,:],truth[:,6:12,:,:]))
        carLoss = torch.sum(objmask*self.myBCEWithLogitsLoss(pred[:,12:18,:,:],truth[:,12:18,:,:]))
        wLoss = torch.sum(objmask*self.myL2Loss(pred[:,18:24,:,:],truth[:,18:24,:,:]))
        hLoss = torch.sum(objmask*self.myL2Loss(pred[:,24:,:,:],truth[:,24:,:,:]))
        return objLoss+personLoss+carLoss+wLoss+hLoss

    def myBCEWithLogitsLoss(self,x,y):
        #pos_weight>1增加召回,pos_weight<1提高精度
        return -self.weight*(self.pos_weight*y*torch.log(torch.sigmoid(x))+(1-y)*torch.log(1-torch.sigmoid(x)))

    def myL2Loss(self,x,y):
        return torch.pow(self.scale_factor*torch.sigmoid(x/self.scale_factor) - y,2)


title: Why does loss decrease but accuracy decreases too (Pytorch, LSTM)?

body:
I have built a model with LSTM - Linear modules in Pytorch for a classification problem (10 classes). I am training the model and for each epoch I output the loss and accuracy in the training set. The ouput is as follows:

epoch: 0 start!
  Loss: 2.301875352859497
  Acc: 0.11388888888888889
  epoch: 1 start!
  Loss: 2.2759320735931396
  Acc: 0.29
  epoch: 2 start!
  Loss: 2.2510263919830322
  Acc: 0.4872222222222222
  epoch: 3 start!
  Loss: 2.225804567337036
  Acc: 0.6066666666666667
  epoch: 4 start!
  Loss: 2.199286699295044
  Acc: 0.6511111111111111
  epoch: 5 start!
  Loss: 2.1704766750335693
  Acc: 0.6855555555555556
  epoch: 6 start!
  Loss: 2.1381614208221436
  Acc: 0.7038888888888889
  epoch: 7 start!
  Loss: 2.1007182598114014
  Acc: 0.7194444444444444
  epoch: 8 start!
  Loss: 2.0557992458343506
  Acc: 0.7283333333333334
  epoch: 9 start!
  Loss: 1.9998993873596191
  Acc: 0.7427777777777778
  epoch: 10 start!
  Loss: 1.9277743101119995
  Acc: 0.7527777777777778
  epoch: 11 start!
  Loss: 1.8325848579406738
  Acc: 0.7483333333333333
  epoch: 12 start!
  Loss: 1.712520718574524
  Acc: 0.7077777777777777
  epoch: 13 start!
  Loss: 1.6056485176086426
  Acc: 0.6305555555555555
  epoch: 14 start!
  Loss: 1.5910680294036865
  Acc: 0.4938888888888889
  epoch: 15 start!
  Loss: 1.6259561777114868
  Acc: 0.41555555555555557
  epoch: 16 start!
  Loss: 1.892195224761963
  Acc: 0.3655555555555556
  epoch: 17 start!
  Loss: 1.4949012994766235
  Acc: 0.47944444444444445
  epoch: 18 start!
  Loss: 1.4332982301712036
  Acc: 0.48833333333333334

For loss function I have used nn.CrossEntropyLoss and Adam Optimizer.
Although the loss is constantly decreasing, the accuracy increases until epoch 10 and then begins for some reason to decrease.

Why is this happening ?

Even if my model is overfitting, doesn't that mean that the accuracy should be high ?? (always speaking for accuracy and loss measured on the training set, not the validation set)


title: How is KL-divergence in pytorch code related to the formula?

body:
In VAE tutorial, kl-divergence of two Normal Distributions is defined by:

And in many code, such as here, hereand here, the code is implemented as:

 KL_loss = -0.5 * torch.sum(1 + logv - mean.pow(2) - logv.exp())

or

def latent_loss(z_mean, z_stddev):
    mean_sq = z_mean * z_mean
    stddev_sq = z_stddev * z_stddev
    return 0.5 * torch.mean(mean_sq + stddev_sq - torch.log(stddev_sq) - 1)

How are they related? why there is not any "tr" or ".transpose()" in code?


title: How to create custom neural network with custom weight initialization in tensorflow or pytorch

body: I'm trying to create a small neural network with custom connections between neurons. The connections should exist over several layers and not be fully connected (sparse) as shown in the picture. I would also like to do the weight initialization manually and not completely randomly. My goal is to determine whether a connection is positive or negative. Is it possible to create such a neural net in tensorflow (python/js) or pytorch?


title: Pytorch Quantization RuntimeError: Trying to create tensor with negative dimension

body:
I am trying out the pytorch quantization module. When doing static post training quantization I follow the
next procedure detailed in the documentation:

adding QuantStub and DeQuantStub modules
Fuse operations
Specify qauntization config
torch.quantization.prepare()
Calibrate the model by running inference against a calibration dataset
torch.quantization.convert()

However, when calibrating the model after preparing it the program breaks.

The error appears at the last fully connected layers. It seems that the observers introduced in the graph are trying to create an histogram of negative dimension.

Here is the error:

    x = self.fc(x)
  File "/home/juan/miniconda3/envs/sparse/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/juan/miniconda3/envs/sparse/lib/python3.6/site-packages/torch/nn/modules/container.py", line 100, in forward
    input = module(input)
  File "/home/juan/miniconda3/envs/sparse/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/juan/miniconda3/envs/sparse/lib/python3.6/site-packages/torch/nn/modules/container.py", line 100, in forward
    input = module(input)
  File "/home/juan/miniconda3/envs/sparse/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/juan/miniconda3/envs/sparse/lib/python3.6/site-packages/torch/nn/modules/container.py", line 100, in forward
    input = module(input)
  File "/home/juan/miniconda3/envs/sparse/lib/python3.6/site-packages/torch/nn/modules/module.py", line 552, in __call__
    hook_result = hook(self, input, result)
  File "/home/juan/miniconda3/envs/sparse/lib/python3.6/site-packages/torch/quantization/quantize.py", line 74, in _observer_forward_hook
    return self.activation_post_process(output)
  File "/home/juan/miniconda3/envs/sparse/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/juan/miniconda3/envs/sparse/lib/python3.6/site-packages/torch/quantization/observer.py", line 805, in forward
    self.bins)
  File "/home/juan/miniconda3/envs/sparse/lib/python3.6/site-packages/torch/quantization/observer.py", line 761, in _combine_histograms
    histogram_with_output_range = torch.zeros((Nbins * downsample_rate))
RuntimeError: Trying to create tensor with negative dimension -4398046511104: [-4398046511104]

The fully connected are built as

class LinearReLU(nn.Sequential):
    def __init__(self, in_neurons, out_neurons):
        super(LinearReLU, self).__init__(
            nn.Linear(in_neurons, out_neurons),
            nn.ReLU(inplace=False)
        )

They are appended in the fc(x) as fc = nn.Sequential(*([LinearReLU, LinearReLU, ...]).

However, I suspect that it has something to do with the reshape between the convolutions and the fully connected layers.

x = x.reshape(-1, size)

Until now I have not been able to solve this error.

Thanks in advance