title: How to solve the OSError while Importing torch in Python

body:
Information:

OS: Windows 8.1 64-bit
Python: 3.7.3 64-bit
Anaconda: 64-bit

Agenda:

To make sure python -c "import torch" works without any errors.

Process:

I have created my environment using conda create -n myenv

After which with great difficulty I have further installed conda install pytorch torchvision cudatoolkit -c pytorch

When the above command was executed, it was interrupted midway due to HTTPRequest failure

After several such failures, I have installed pytorch using conda install pytorch -c pytorch

When I type python -c "import torch" I get OSError like the one below

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\sanpj\.conda\envs\myenv\lib\site-packages\torch\__init__.py", line 42, in <module>
    import numpy as _np  # noqa: F401
  File "C:\Users\sanpj\AppData\Roaming\Python\Python37\site-packages\numpy\__init__.py", line 142, in <module>
    from . import core
  File "C:\Users\sanpj\AppData\Roaming\Python\Python37\site-packages\numpy\core\__init__.py", line 23, in <module>
    WinDLL(os.path.abspath(filename))
  File "C:\Users\sanpj\.conda\envs\myenv\lib\ctypes\__init__.py", line 356, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: [WinError 193] %1 is not a valid Win32 application

But the output should not be as such.

When I type in python, I get some information as mentioned below

Python 3.7.3 (default, Apr 24 2019, 15:29:51) [MSC v.1915 64 bit (AMD64)] :: Anaconda, Inc. on win32
Type "help", "copyright", "credits" or "license" for more information.


title: Compare the example of Pytorch and Keras on Cifar10 data

body:
I use CIFAR10 dataset to learn how to code using Keras and PyTorch.

The environment is Python 3.6.7, Torch 1.0.0, Keras 2.2.4, Tensorflow 1.14.0.
I use the same batch size, number of epochs, learning rate and optimizer.
I use DenseNet121 as the model.

After training, Keras get 69% accuracy in test data.
PyTorch just get 54% in test data.

I know the results are different, but why is the result so bad in PyTorch?

Here is the Keras code:

import os, keras
from keras.datasets import cifar10
from keras.applications.densenet import DenseNet121
batch_size = 32
num_classes = 10
epochs = 20
# The data, split between train and test sets:
(x_train, y_train), (x_test, y_test) = cifar10.load_data()
print('x_train shape:', x_train.shape)
print(x_train.shape[0], 'train samples')
print(x_test.shape[0], 'test samples')

# Convert class vectors to binary class matrices.
y_train = keras.utils.to_categorical(y_train, num_classes)
y_test = keras.utils.to_categorical(y_test, num_classes)

# model
model = DenseNet121(include_top=True, weights=None, input_shape=(32,32,3), classes=10)

# initiate RMSprop optimizer
opt = keras.optimizers.SGD(lr=0.001, momentum=0.9)

model.compile(loss='categorical_crossentropy', optimizer=opt, metrics=['accuracy'])

x_train = x_train.astype('float32')
x_test = x_test.astype('float32')
x_train /= 255
x_test /= 255

model.fit(x_train, y_train,
          batch_size=batch_size,
          epochs=epochs,
          validation_data=(x_test, y_test),
          shuffle=True)

# Score trained model.
scores = model.evaluate(x_test, y_test, verbose=1)
print('Test loss:', scores[0])
print('Test accuracy:', scores[1])

Here is the Pytorch code:

import torch
import torchvision
import torchvision.transforms as transforms
from torch import flatten
import torch.optim as optim
from torchvision import transforms, models
from torch.nn import Linear, Softmax, Module, Sequential, CrossEntropyLoss
import numpy as np
from tqdm import tqdm
classes = ('plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck')
transform = transforms.Compose([transforms.ToTensor()])

trainset = torchvision.datasets.CIFAR10(root='./DataSet', train=True, download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=32, shuffle=True, num_workers=0)

testset = torchvision.datasets.CIFAR10(root='./DataSet', train=False, download=True, transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=4, shuffle=False, num_workers=0)

import torch.nn as nn
import torch.nn.functional as F

class Net(Module):
    def __init__(self):
        super(Net, self).__init__()
        self.funFeatExtra   = Sequential(*[i for i in list(models.densenet121().children())[:-1]])
        self.funFlatten     = flatten
        self.funOutputLayer = Linear(1024, 10)
        self.funSoftmax     = Softmax(dim=1)
    def forward(self, x):
        x = self.funFeatExtra(x)
        x = self.funFlatten(x, 1)
        x = self.funOutputLayer(x)
        x = self.funSoftmax(x)
        return x

net = Net()

criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)

for epoch in range(20):  # loop over the dataset multiple times

    running_loss = 0.0
    for i, data in tqdm(enumerate(trainloader, 0)):
        # get the inputs; data is a list of [inputs, labels]
        inputs, labels = data

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = net.cuda()(inputs.cuda())
        loss = criterion(outputs, labels.cuda())
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()

        # if i % 2000 == 1999:    # print every 2000 mini-batches
        #     print('[%d, %5d] loss: %.3f' % (epoch + 1, i + 1, running_loss / 2000))
        #     running_loss = 0.0

print('Finished Training')

########################################################################
# The results seem pretty good.
#
# Let us look at how the network performs on the whole dataset.

correct = 0
total = 0
with torch.no_grad():
    for data in tqdm(testloader):
        images, labels = data
        outputs = net.cpu()(images.cpu())
        _, predicted = torch.max(outputs.data, 1)
        total += labels.size(0)
        correct += (predicted == labels).sum().item()

print('Accuracy of the network on the 10000 test images: %d %%' % (100 * correct / total))


title: Very simple optim.SGD training loop not working as expected - PyTorch

body:
I've been reading through the PyTorch documentation and I've been trying to figure out MSELoss and autograd. I tried creating a very simple training loop that takes two random tensors and updates the values in each tensor so that the sum all values in tensor1 plus the sum of all values in tensor2 add up to some target number. In my example I used 100. Here is how I implemented this:

import torch.nn as nn
import torch
import torch.optim as optim

loss = nn.MSELoss()
tensor1 = torch.randn(3, 5, requires_grad=True)
tensor2 = torch.randn(3, 5, requires_grad=True)
target_sum = torch.FloatTensor(torch.Tensor([100]))
optimizer = optim.SGD([nn.Parameter(tensor1), nn.Parameter(tensor2)], lr=1e-4)
print(str(tensor1.sum().item()) + str(tensor2.sum().item()))
for i in range(100):
    pred = tensor1.sum() + tensor2.sum()
    optimizer.zero_grad()
    loss(pred, target_sum).backward
    optimizer.step()
print(str(tensor1.sum().item()) + str(tensor2.sum().item()))

The sums of the tensors before the training loop and after the training loop are the same, but I should be seeing an increase in the sum and have it approach 100. I'm not sure what I'm missing here. I think it  might have something to do with my optimizer as the training loop is almost straight out of the documentation example. I'm new to PyTorch and I appreciate any help!


title: coremltools: how to properly use NeuralNetworkMultiArrayShapeRange?

body:
I have a PyTorch network and I want to deploy it to iOS devices. In short, I fail to add flexibility to the input tensor shape in CoreML.

The network is a convnet that takes an RGB image (stored as a tensor) as an input and returns an RGB image of the same size. Using PyTorch, I can input images of any size I want, for instance a tensor of size (1, 3, 300, 300) for a 300x300 image.

To convert the PyTorch model to a CoreML model, I first convert it to an ONNX model using torch.onnx.export. This function requires to pass a dummy input so that it can execute the graph. So I did using:

input = torch.rand(1, 3, 300, 300)

My guess is that the ONNX model only accepts images / tensors of size (1, 3, 300, 300). Now, I can use the onnx_coreml.convert function to convert the ONNX model to a CoreML model. By printing the CoreML model's spec description using Python, I get something like:

input {
  name: "my_image"
  type {
    multiArrayType {
      shape: 1
      shape: 3
      shape: 300
      shape: 300
      dataType: FLOAT32
    }
  }
}
output {
  name: "my_output"
  type {
    multiArrayType {
      shape: 1
      shape: 3
      shape: 300
      shape: 300
      dataType: FLOAT32
    }
  }
}
metadata {
  userDefined {
    key: "coremltoolsVersion"
    value: "3.1"
  }
}

The model's input must be a multiArrayType of size (1, 3, 300, 300). By copying this model to XCode, I can see while inspecting the model that my_name is listed under the "Inputs" section and it is expected to be a MultiArray (Float32 1 x 3 x 300 x 300). So far, everything is coherent.

My problem is to add flexibility to the input shape. I tried to use coremltools with no luck. This is my problem. Here is my code:

import coremltools
from coremltools.models.neural_network import flexible_shape_utils

spec = coremltools.utils.load_spec('my_model.mlmodel')

shape_range = flexible_shape_utils.NeuralNetworkMultiArrayShapeRange()
shape_range.add_channel_range((3,3))
shape_range.add_height_range((64, 5000))
shape_range.add_width_range((64, 5000))
flexible_shape_utils.update_multiarray_shape_range(spec, feature_name='my_image', shape_range=shape_range)
coremltools.models.utils.save_spec(spec, 'my_flexible_model.mlmodel')

I get the following spec description using Python:

input {
  name: "my_image"
  type {
    multiArrayType {
      shape: 1
      shape: 3
      shape: 300
      shape: 300
      dataType: FLOAT32
      shapeRange {
        sizeRanges {
          lowerBound: 3
          upperBound: 3
        }
        sizeRanges {
          lowerBound: 64
          upperBound: 5000
        }
        sizeRanges {
          lowerBound: 64
          upperBound: 5000
        }
      }
    }
  }
}

Only 3 ranges as specified, which makes sense since I only defined a range for channel, height and width, but not for the batch size. In XCode, I get the following error when inspecting the flexible CoreML model:

There was a problem decoding this CoreML document
validator error: Description of multiarray feature 'my_image' has a default 4-d shape but a 3-d shape range

I'm pretty sure it was working on another project when I was on macOS X Mojave, but at this point I'm sure of nothing.

I'm using:

macOS X Catalina
conda 4.7.12
python 3.7.5
pytorch 1.3.1
onnx 1.6.0
onnx-coreml 1.1
coremltools 3.1

Thanks for the help


title: File not found: archive/constants.pkl at Python Pytorch

body:
I'm getting error while preparing List of operators of your serialized torchscript model. What is the issue here? Is it coming when loading it?
Python code
# Dump list of operators used by MobileNetV2:
import torch, yaml

root = '/content/drive/My Drive/Monitoring/'
model = torch.jit.load(root+'model.pt')
ops = torch.jit.export_opnames(model)
with open('MobileNetV2.yaml', 'w') as output:
    yaml.dump(ops, output)

Stack Trace
    ---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-39-8b61c35fb898> in <module>()
      3
      4 root = '/content/drive/My Drive/Monitoring/'
----> 5 model = torch.jit.load(root+'model.pt')
      6 ops = torch.jit.export_opnames(model)
      7 with open('MobileNetV2.yaml', 'w') as output:

/usr/local/lib/python3.6/dist-packages/torch/jit/_serialization.py in load(f, map_location, _extra_files)
    159     cu = torch._C.CompilationUnit()
    160     if isinstance(f, str) or isinstance(f, pathlib.Path):
--> 161         cpp_module = torch._C.import_ir_module(cu, f, map_location, _extra_files)
    162     else:
    163         cpp_module = torch._C.import_ir_module_from_buffer(

RuntimeError: [enforce fail at inline_container.cc:222] . file not found: archive/constants.pkl


title: Fixing a subset of weights in Neural network during training

body:
I am considering creating a customized neural network. The basic structure is the same as usual, but I want to truncate the connections between layers. For example, if I construct a network with two hidden layers, I would like to delete some weights and keep the others, like so:

This is not conventional dropout (to avoid overfitting), since the remaining weights (connections) should be specified and fixed.
Are there any ways in python to do it? Tensorflow, pytorch, theano or any other modules?


title: Glove Pytorch speed up

body:
I am trying to implement the GloVe algorithm in pytorch. This is the first time I am using pytorch and I think my implementation might not be very efficient. Apart from the obvious (vectorizing the for loop that is run every batch) is there anything that would give me some speed up? I found that the self.optimizer.step() is particularly expensive.

from torch.autograd import Variable
import numpy as np
import torch
import torch.optim as optim
import torch.nn as nn
from scipy import sparse
from collections import Counter
import gensim.utils as utils
import pandas as pd
from collections import deque
import time
import gensim.utils as utils

def tokenize_sentences(sentences,enforce_lower = True):
    return [list(utils.tokenize(sentence,lowercase=enforce_lower)) for sentence in sentences]

class GloVe(nn.Module):

def __init__(self, window = 3, size = 100, xmax = 2, alpha = 0.75, batch_size = 10,
    l_rate = 0.01, num_epochs = 100, min_count=2, sentence_representation={'name':'mean_of_words'},
    verbose=True, retrain_first=False, pretrained_model=None):

    super(GloVe, self).__init__()
    self.window = window
    self.size = size
    self.xmax = xmax
    self.alpha = alpha
    self.batch_size = batch_size
    self.l_rate = l_rate
    self.num_epochs = num_epochs
    self.min_count = min_count
    self.sentence_representation = sentence_representation
    self.verbose = verbose

def build_model(self):

    self.vocab, self.reverse_vocab = self.build_vocab(self.sentence_tokens)
    self.comat, self.coocs = self.build_cooccur(self.vocab, self.sentence_tokens, self.min_count)
    self.l_embed = nn.Embedding(self.vocab_size, self.size, sparse=True)
    self.r_embed = nn.Embedding(self.vocab_size, self.size, sparse=True)

    # If we use other optimizers such as Adam, we can't use sparse matrices
    # self.l_embed = nn.Embedding(self.vocab_size, self.size)
    # self.r_embed = nn.Embedding(self.vocab_size, self.size)

    self.l_biases = Variable(torch.from_numpy(np.random.normal(0, 0.01,
        (self.vocab_size, 1))).float(),requires_grad = True)
    self.r_biases = Variable(torch.from_numpy(np.random.normal(0, 0.01,
        (self.vocab_size, 1))).float(),requires_grad = True)
    self.optimizer = optim.SGD(self.parameters(), lr = self.l_rate)

def build_vocab(self, sentence_tokens):

    vocab = Counter()
    for sentence in sentence_tokens:
        vocab.update(sentence)
    normal_vocab = {word: (i, freq) for i, (word, freq) in enumerate(vocab.items())}
    reverse_vocab = {i: (word, freq) for i, (word, freq) in enumerate(vocab.items())}
    return normal_vocab, reverse_vocab

def build_cooccur(self, vocab, corpus, window_size=10, min_count=None):

    self.vocab_size = len(vocab)
    id2word = dict((i, word) for word, (i, _) in vocab.items())

    comat = np.zeros((self.vocab_size, self.vocab_size))
    for i, sentence in enumerate(corpus):
        token_ids = [vocab[word][0] for word in sentence]

        for center_i, center_id in enumerate(token_ids):
            context_ids = token_ids[max(0, center_i - window_size) : center_i]
            contexts_len = len(context_ids)

            for left_i, left_id in enumerate(context_ids):
                distance = contexts_len - left_i
                increment = 1.0 / float(distance)
                comat[center_id, left_id] += increment
                comat[left_id, center_id] += increment
    coocs = np.transpose(np.nonzero(comat))
    return comat, coocs

def gen_batch(self):
    sample = np.random.choice(np.arange(len(self.coocs)), size=self.batch_size, replace=False).astype(int)
    indices = np.array([self.coocs[ind] for ind in sample])
    l_vecs = self.l_embed(Variable(torch.LongTensor(indices[:,0])))
    r_vecs = self.r_embed(Variable(torch.LongTensor(indices[:,1])))
    l_v_bias = self.l_biases[torch.LongTensor(indices[:,0])]
    r_v_bias = self.r_biases[torch.LongTensor(indices[:,1])]
    covals = torch.from_numpy(np.array([[self.comat[tuple(self.coocs[chosen])]] for chosen in sample])).float()
    return l_vecs, r_vecs, covals, l_v_bias, r_v_bias

def wf(self, x):
    return torch.min((x/self.xmax)**self.alpha, torch.from_numpy(np.array([1])).float())

def train_model(self):
    w_list_size = np.sum([len(s) for s in self.sentence_tokens])

    continue_training = True
    last_losses = deque(maxlen=5)
    last_losses.append(np.inf)
    moving_avg = np.mean(last_losses)
    epoch = 0
    while continue_training:
        num_batches = int(w_list_size/self.batch_size)
        avg_loss = 0.0
        for batch in range(num_batches):
            if self.verbose:
                print('Batch {}/{}'.format(batch, num_batches))
                start_time = time.time()
            loss = 0.0

            l_vecs, r_vecs, covals, l_v_bias, r_v_bias = self.gen_batch()

            # TODO: vectorize this
            loop_time = time.time()
            for i in range(self.batch_size):

                loss +=  Variable(self.wf(covals[i]), requires_grad=False) * torch.pow((torch.dot(l_vecs[i], r_vecs[i])
                     + l_v_bias[i] + r_v_bias[i] - Variable(torch.log(covals[i]), requires_grad=False)),2)/self.batch_size
            print('loop time: ', time.time()-loop_time)

            avg_loss += loss/num_batches

            self.optimizer.zero_grad() # reset gradient vector
            loss.backward() # calculate gradients
            self.optimizer.step() # do optimization step

            if self.verbose:
                print('batch time: ', time.time()-start_time)

        last_losses.append(avg_loss.data.numpy())
        if (np.mean(last_losses)>0.98*moving_avg and epoch>1) or (epoch == self.num_epochs):
            continue_training = False
        else:
            moving_avg = np.mean(last_losses)
        if self.verbose:
            print("Average loss for epoch {}: {}, Moving average: {}".format(epoch, avg_loss.data.numpy()[0], moving_avg))
        epoch += 1

def get_embeddings(self):
    embeddings = (self.l_embed.weight.data + self.r_embed.weight.data).numpy()/2
    embeddings_dict = {}
    for k,v in self.vocab.items():
        embeddings_dict[k] = embeddings[v[0]]
    return embeddings, embeddings_dict

def get_mean_sentence_embeddings(self, tokenised_sentences):
    sentence_embeddings = []
    sentence_embeddings = np.zeros((len(tokenised_sentences), self.size))
    ct = 0
    for tokens in tokenised_sentences:

        vecs = []
        for word in tokens:
            try:
                vecs.append(self.embeddings_dict[word])
            except:
                pass
        if len(vecs) > 0:
            arr = np.array(vecs)
            sentence_embeddings[ct] = list(arr.mean(axis=0))

        ct +=1
    return sentence_embeddings

def fit_transform(self, sentences):

    self.sentence_tokens = tokenize_sentences(sentences)
    self.build_model()
    self.train_model()
    self.get_embeddings()

    if self.sentence_representation['name'] == 'mean_of_words':
        return self.get_mean_sentence_embeddings(self.sentence_tokens)

def transform(self,sentences):
    sentence_tokens = tokenize_sentences(sentences)
    if self.sentence_representation['name'] == 'mean_of_words':
        return self.get_mean_sentence_embeddings(sentence_tokens)


title: Top K indices of a multi-dimensional tensor

body:
I have a 2D tensor and I want to get the indices of the top k values. I know about pytorch's topk function. The problem with pytorch's topk function is, it computes the topk values over some dimension. I want to get topk values over both dimensions.
For example for the following tensor
a = torch.tensor([[4, 9, 7, 4, 0],
        [8, 1, 3, 1, 0],
        [9, 8, 4, 4, 8],
        [0, 9, 4, 7, 8],
        [8, 8, 0, 1, 4]])

pytorch's topk function will give me the following.
values, indices = torch.topk(a, 3)

print(indices)
# tensor([[1, 2, 0],
#        [0, 2, 1],
#        [0, 1, 4],
#        [1, 4, 3],
#        [1, 0, 4]])

But I want to get the following
tensor([[0, 1],
        [2, 0],
        [3, 1]])

This is the indices of 9 in the 2D tensor.
Is there any approach to achieve this using pytorch?


title: HTTP error when installing pytorch

body:
I use ubuntu14.04, without GPU.
when I install pytorch in anaconda(I type command just as pytorch.org says),things always get wrong,like this:

------------------------------------------------------------
Fetching package metadata ...........
Solving package specifications: .

Package plan for installation in environment /home/zhanglu/anaconda3:

The following NEW packages will be INSTALLED:

    pytorch:     0.1.9-py36_2 soumith
    torchvision: 0.1.7-py36_1 soumith

The following packages will be UPDATED:

    conda:       4.3.8-py36_0         --> 4.3.13-py36_0

Proceed ([y]/n)? y

pytorch-0.1.9- 100% |################################| Time: 0:02:49   1.44 MB/s
pytorch-0.1.9- 100% |################################| Time: 0:05:52 692.97 kB/s
pytorch-0.1.9- 100% |################################| Time: 0:01:15   3.23 MB/s

CondaError: CondaHTTPError: HTTP None None for url <None>
Elapsed: None

An HTTP error occurred when trying to retrieve this URL.
ConnectionError(ReadTimeoutError("HTTPSConnectionPool(host='binstar-cio-packages-prod.s3.amazonaws.com', port=443): Read timed out.",),)

CondaError: CondaHTTPError: HTTP None None for url <None>
Elapsed: None

An HTTP error occurred when trying to retrieve this URL.
ConnectionError(ReadTimeoutError("HTTPSConnectionPool(host='binstar-cio-packages-prod.s3.amazonaws.com', port=443): Read timed out.",),)

CondaError: CondaHTTPError: HTTP None None for url <None>
Elapsed: None

An HTTP error occurred when trying to retrieve this URL.
ConnectionError(ReadTimeoutError("HTTPSConnectionPool(host='binstar-cio-packages-prod.s3.amazonaws.com', port=443): Read timed out.",),)

----------------------------------------------------------------

I've tried many times, but every time error occurs in the same step.
I'm looking forward your answers,thanks very very much for your help!!!


title: Duplicate layers when reusing pytorch model

body:
I am trying to reuse some of the resnet layers for a custom architecture and ran into a issue I can't figure out. Here is a simplified example; when I run:

import torch
from torchvision import models
from torchsummary import summary

def convrelu(in_channels, out_channels, kernel, padding):
    return nn.Sequential(
        nn.Conv2d(in_channels, out_channels, kernel, padding=padding),
        nn.ReLU(inplace=True),
    )

class ResNetUNet(nn.Module):
    def __init__(self):
        super().__init__()

        self.base_model = models.resnet18(pretrained=False)
        self.base_layers = list(self.base_model.children())

        self.layer0 = nn.Sequential(*self.base_layers[:3])

    def forward(self, x):
        print(x.shape)

        output = self.layer0(x)

        return output

base_model = ResNetUNet().cuda()
summary(base_model,(3,224,224))

Is giving me:

----------------------------------------------------------------
        Layer (type)               Output Shape         Param #
================================================================
            Conv2d-1         [-1, 64, 112, 112]           9,408
            Conv2d-2         [-1, 64, 112, 112]           9,408
       BatchNorm2d-3         [-1, 64, 112, 112]             128
       BatchNorm2d-4         [-1, 64, 112, 112]             128
              ReLU-5         [-1, 64, 112, 112]               0
              ReLU-6         [-1, 64, 112, 112]               0
================================================================
Total params: 19,072
Trainable params: 19,072
Non-trainable params: 0
----------------------------------------------------------------
Input size (MB): 0.57
Forward/backward pass size (MB): 36.75
Params size (MB): 0.07
Estimated Total Size (MB): 37.40
----------------------------------------------------------------

This is duplicating each layer (there are 2 convs, 2 batchnorms, 2 relu's) as opposed to giving one layer each. If I print out self.base_layers[:3] I get:

[Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False), BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True), ReLU(inplace=True)]

which shows just three layers without duplicates. Why is it duplicating my layers?

I am using pytorch version 1.4.0


title: Is indexing with a bool array or index array faster in numpy/pytorch?

body: I can index my numpy array / pytorch tensor with a boolean array/tensor of the same shape or an array/tensor containing integer indexes of the elements I'm after. Which is faster?


title: Not Able to Import PyTorch in Conda Env

body:
Last week I had a working conda env I was using for a project. I have not touched the project in a week. I just went to run a python file (python file.py) that had been running with no errors.

Now I get the following error:

Traceback (most recent call last):
  File "file.py", line 2, in <module>
    from torch.utils.data import Dataset, DataLoader
ModuleNotFoundError: No module named 'torch'

In an attempt to troubleshoot, I opened a python console and ran the following code:

>>> import torch

The result was the following error message:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'torch'

If I check all the installed packages using conda list -n <env_name>, I can see that PyTorch is in fact installed, just as it was last week.

...
pytorch                   1.2.0           py3.7_cuda9.2.148_cudnn7.6.2_0    pytorch
...
torchvision               0.4.0                 py37_cu92    pytorch
...

Here is what I see when I start python console using python:

Python 3.7.4 (default, Aug 13 2019, 20:35:49)
[GCC 7.3.0] :: Anaconda, Inc. on linux

The output of python -c 'import sys; print(sys.path) in the base env is:

['', '/home/<name>/anaconda3/lib/python37.zip', '/home/<name>/anaconda3/lib/python3.7', '/home/<name>/anaconda3/lib/python3.7/lib-dynload', '/home/<name>/anaconda3/lib/python3.7/site-packages'] I have not personally made any chages to PYTHONPATH.

If I run python -c 'import sys; print(sys.path)' with my conda env (non-base) active, I get:

['', '/home/<name>/anaconda3/envs/<env_name>/lib/python37.zip', '/home/<name>/anaconda3/envs/<env_name>/lib/python3.7', '/home/<name>/anaconda3/envs/<env_name>/lib/python3.7/lib-dynload', '/home/<name>/anaconda3/envs/<env_name>/lib/python3.7/site-packages']

This is totally bizarre, can't figure out what is going on, and what could have happened, over the course of the last week, without me touching the code or making any changes to Anaconda.


title: Where is '_DataLoaderIter' in pytorch 1.3.1?

body:
When I use pytorch 1.3.1 with python3.7.4, like this

import torch

from torch.utils.data.dataloader import _DataLoaderIter

Here is an errer : cannot import name '_DataLoaderIter' from 'torch.utils.data.dataloader'

How should I solve that?

Should I uninstall 1.3.1?

I found _DataLoaderIter is in dataloader.pyi :

class _DataLoaderIter:
    def __init__(self, loader: DataLoader) -> None:...
    def __len__(self) -> int: ...
    def __iter__(self) -> _DataLoaderIter: ...
    def __next__(self) -> Any: ...

But I can't find it in dataloader.py.


title: LibTorch, convert deeplabv3_resnet101 to c++

body:
I am trying to use this example code from the PyTorch website to convert a python model for use in the PyTorch c++ api (LibTorch).

Converting to Torch Script via Tracing
To convert a PyTorch model to Torch Script via tracing, you must pass an instance of your model along with an example input to the torch.jit.trace function. This will produce a torch.jit.ScriptModule object with the trace of your model evaluation embedded in the module’s forward method:

import torch
import torchvision

# An instance of your model.
model = torchvision.models.resnet18()

# An example input you would normally provide to your model's forward() method.
example = torch.rand(1, 3, 224, 224)

# Use torch.jit.trace to generate a torch.jit.ScriptModule via tracing.
traced_script_module = torch.jit.trace(model, example)
traced_script_module.save("model.pt")

This example works fine, and saves out the file as expected.
When i switch to this model:

model = models.segmentation.deeplabv3_resnet101(pretrained=True)

It gives me the following error:

File "convert.py", line 14, in <module>
    traced_script_module = torch.jit.trace(model, example)
  File "C:\Python37\lib\site-packages\torch\jit\__init__.py", line 636, in trace
          raise ValueError('Expected more than 1 value per channel when training, got input size {}'.format(size))
ValueError: Expected more than 1 value per channel when training, got input size torch.Size([1, 256, 1, 1])

I assume this is because the example format is wrong, but how can I get the correct one?

Based on the comments below, my new code is:

import torch
import torchvision
from torchvision import models

model = models.segmentation.deeplabv3_resnet101(pretrained=True)
model.eval()

# An example input you would normally provide to your model's forward() method.
example = torch.rand(1, 3, 224, 224)

# Use torch.jit.trace to generate a torch.jit.ScriptModule via tracing.
traced_script_module = torch.jit.trace(model, example)

traced_script_module.save("model.pt")

And i now get the error:

File "convert.py", line 15, in <module>
    traced_script_module = torch.jit.trace(model, example)
  File "C:\Python37\lib\site-packages\torch\jit\__init__.py", line 636, in trace
    var_lookup_fn, _force_outplace)
RuntimeError: Only tensors and (possibly nested) tuples of tensors are supported as inputs or outputs of traced functions (toIValue at C:\a\w\1\s\windows\pytorch\torch/csrc/jit/pybind_utils.h:91)
(no backtrace available)


I am trying to build a model for learning assigned scores (real numbers) to some sentences in a data set. I use RNNs (in PyTorch) for this purpose.
I have defined a model:

class RNNModel1(nn.Module):

    def forward(self, input ,hidden_0):

        embedded = self.embedding(input)
        output, hidden = self.rnn(embedded, hidden_0)
        output=self.linear(hidden)
        return output , hidden

Train function is as:

def train(model,optimizer,criterion,BATCH_SIZE,train_loader,clip):
    model.train(True)
    total_loss = 0
    hidden = model._init_hidden(BATCH_SIZE)

    for i, (batch_of_data, batch_of_labels) in enumerate(train_loader, 1):
        hidden=hidden.detach()
        model.zero_grad()
        output,hidden= model(batch_of_data,hidden)
        loss = criterion(output, sorted_batch_target_scores)
        total_loss += loss.item()
        loss.backward()
        torch.nn.utils.clip_grad_norm(model.parameters(), clip)
        optimizer.step()
    return total_loss/len(train_loader.dataset)

when I run the code I receive this error:

  RuntimeError: Expected hidden size (2, 24, 50), got (2, 30, 50)

Batch size=30, Hidden size=50, Number of Layers=1, Bidirectional=True.

I receive that error in the last batch of data.
I checked the description of RNNs in PyTorch to solve this problem. RNNs in PyTorch have two input arguments and two output arguments. The input arguments are input and h_0. h_0 is a tensor includes initial hidden state for each element in batch of size(num_layers*num_directions, batch, hidden size).
The output arguments are output ans h_n. h_n is a tensor includes hidden state for t=seq_len of size (num_layers*num_directions, batch, hidden size).

in all batches (except the last batch) the size of h_0 and h_n is the same. but in the last batch, perhaps number of elements is less than batch size. Thesefore the size of h_n is (num_layersnum_directions, remained_elements_in_last_batch, hidden size) but the size of h_0 is still (num_layersnum_directions, batch_size, hidden size).

So I receive that error in the last batch of data.

How can I solve this problem and handle the situation in which the size of h_0 and h_n is different?

Thanks in advance.


title: Adding dropout between time-steps in pytorch RNN

body:
I am training built-in pytorch rnn modules (eg torch.nn.LSTM) and would like to add fixed-per-minibatch dropout between each time step (Gal dropout, if I understand correctly).

Most simply, I could unroll the network and compute my forward computation on a single batch something like this:

dropout = get_fixed_dropout()
for sequence in batch:
    state = initial_state
    for token in sequence:
        state, output = rnn(token,state)
        state, output = dropout(state, output)
        outputs.append(output)
    loss += loss(outputs,sequence)
loss.backward()
optimiser.step()

However, assuming that looping in python is pretty slow, I would much rather take advantage of pytorch's ability to fully process a batch of several sequences, all in one call (as in rnn(batch,state) for regular forward computations).

i.e., I would prefer something that looks like this:

rnn_with_dropout = drop_neurons(rnn)
outputs, states = rnn_with_dropout(batch,initial_state)
loss = loss(outputs,batch)
loss.backward()
optimiser.step()
rnn = return_dropped(rnn_with_dropout)

(note: Pytorch's rnns do have a dropout parameter, but it is for dropout between layers and not between time steps, and so not what I want).

My questions are:

Am I even understanding Gal dropout correctly?
Is an interface like the one I want available?
If not, would simply implementing some drop_neurons(rnn), return_dropped(rnn) functions that zero random rows in the rnn's weight matrices and bias vectors and then return their previous values after the update step be equivalent? (This imposes the same dropout between the layers as in between the steps, i.e. completely removes some neurons for the whole minibatch, and I'm not sure that doing this is 'correct').


title: How to see the adapted learning rate for Adam in pytorch?

body:
There are many different optimizers with adaptive learning rate methods. Is it possible to see the adapted value of the initial learning rate for Adam?

Here is a similar question about Adadelta and the answer was to search for ["acc_delta"] key, but Adam has no that key.


title: There is an error with LSTM Hidden state dimension: RuntimeError: Expected hidden[0] size (4, 1, 256), got (1, 256)

body:
I'm experimenting with seq2seq_tutorial in PyTorch. There appears to be a dimension error with the encoder's lstm hidden state size.

With bidirectional=True and num_layers = 2, the hidden state's shape is supposed to be (num_layers*2, batch_size, hidden_size).

However, an error occurs with the following message:

RuntimeError: Expected hidden[0] size (4, 1, 256), got (1, 256)

I have tried reshaping the hidden state to initializing the hidden state with a different shape, to begin with, but nothing seems to work.

Here is the train method of my code:

def train(self, input, target, encoder, decoder, encoder_optim, decoder_optim, criterion):
    enc_optimizer = encoder_optim
    dec_optimizer = decoder_optim
    enc_optimizer.zero_grad()
    dec_optimizer.zero_grad()

    pair = (input, target)
    input_len = input.size(0)
    target_len = target.size(0)
    enc_output_tensor = torch.zeros(self.opt['max_seq_len'], encoder.hidden_size, device=device)
    enc_hidden = encoder.cuda().initHidden(device)

    for word_idx in range(input_len):
        print('Input:', input[word_idx], '\nHidden shape:', enc_hidden.size())
        enc_output, enc_hidden = encoder(input[word_idx], enc_hidden)
        enc_output_tensor[word_idx] = enc_output[0,0]

Here is the encoder method of my code:

class EncoderBRNN(nn.Module):
    # A bidirectional rnn based encoder
    def __init__(self, input_size, hidden_size, emb_size, batch_size=1, num_layers=2, bidir=True):
        super(EncoderBRNN, self).__init__()
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.batch_size = batch_size
        self.embedding_dim = emb_size
        self.num_layers = num_layers
        self.bidir = bidir
        self.embedding_layer = nn.Embedding(self.input_size, self.embedding_dim)
        self.enc_layer = nn.LSTM(self.embedding_dim, self.hidden_size, num_layers=self.num_layers, bidirectional=self.bidir)

    def forward(self, input, hidden):
        embed = self.embedding_layer(input).view(1, 1, -1)
        output, hidden = self.enc_layer(embed, hidden)
        return output, hidden

    def initHidden(self, device):
        if self.bidir:
            num_stacks = self.num_layers * 2
        else:
            num_stacks = self.num_layers
        return torch.zeros(num_stacks, self.batch_size, self.hidden_size, device=device)


title: How to load torch pretrained models in Python?

body:
I have a torch .t7 file of a binarized pretrained model of the AlexNet. I need to convert this .t7 file to a npz/h5 file so I can import it comfortably.

It is the alexnet_XNOR by allenAI.

This is the code being used to read the file: https://github.com/bshillingford/python-torchfile/blob/master/torchfile.py

import torchfile as T
o = T.load('alexnet_XNOR.t7')

This is the error that shows up:

File "<ipython-input-5-7ef550de1de7>", line 1, in <module>
runfile('C:/Users/yash/Desktop/importer.py', wdir='C:/Users/yash/Desktop')

File "C:\Program Files\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 866, in runfile
execfile(filename, namespace)

File "C:\Program Files\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 102, in execfile
exec(compile(f.read(), filename, 'exec'), namespace)

File "C:/Users/yash/Desktop/importer.py", line 10, in <module>
o = T.load('alexnet_XNOR.t7')

File "C:\Users\yash\Desktop\torchreader.py", line 430, in load
return reader.read_obj()

File "C:\Users\yash\Desktop\torchreader.py", line 377, in read_obj
obj._obj = self.read_obj()

File "C:\Users\yash\Desktop\torchreader.py", line 394, in read_obj
obj[k] = v

TypeError: unhashable type: 'numpy.ndarray'

My attempt:
I modified the torchreader and ran the main code (added print statements):

for _ in range(size):
            k = self.read_obj()
            v = self.read_obj()
            print(obj.shape)
            print(v.shape)
            obj[k] = v
            if self.use_list_heuristic:
                if not isinstance(k, int) or k <= 0:
                    keys_natural = False
                elif isinstance(k, int):
                    key_sum += k

It said None for obj.shape, and for v.shape, it said:

    AttributeError: 'bytes' object has no attribute 'shape'

print(obj)
print("===========")
print(v)

When i simply print obj and v using the above snippet code, I get this followed by simply the error i got initially:

{}
=================
b'torch.CudaTensor'
{b'_type': b'torch.CudaTensor'}
=================
[]
{b'output': array([], dtype=float32), b'_type': b'torch.CudaTensor'}
=================
b'gradInput'
{None: b'gradInput', b'output': array([], dtype=float32), b'_type': b'torch.CudaTensor'}
=================
[ 0.]

Can someone explain why this error is happening? I cannot understand the codebase itself as I have not worked with torch before.

Thank you!


title: Static graph is fast. Dynamic graph is slow. Is there any specific benchmark demonstrating this?

body:
I've see some benchmark about tensorflow and pytorch. Tensorflow maybe faster but seems not that faster even sometimes slower.

Is there any benchmark on specifically testing on static graph and dynamic graph demonstrating that static graph is much faster that dynamic graph?


title: How to read pictures from a big folder and split it into train, validation and test sets?

body:
I am working on sign language gesture classifier using pytorch, I have pictures resembling each letter residing in a folder titled with that specific letter. E.g. folder "A" has "1_A_1.jpg", "1_A_2.jpg", "21_A_3.jpg".. etc.

I am trying to build a function that:

Iterates through the different folders
Splits the data into training, validation, and test sets
Labels those pictures with their respective folder name (i.e. letter label)
Returns 3 created folders that are train, test and validation

All the online code shows examples of splitting data coming from torchvision data sets (built in data sets), nothing from scratch.

I found the following on stackoverflow:

import os
import numpy as np
import argparse

def get_files_from_folder(path):

    files = os.listdir(path)
    return np.asarray(files)

def main(path_to_data, path_to_test_data, train_ratio):
    # get dirs
    _, dirs, _ = next(os.walk(path_to_data))

    # calculates how many train data per class
    data_counter_per_class = np.zeros((len(dirs)))

    for i in range(len(dirs)):
        path = os.path.join(path_to_data, dirs[i])
        files = get_files_from_folder(path)
        data_counter_per_class[i] = len(files)
    test_counter = np.round(data_counter_per_class * (1 - train_ratio))

    # transfers files
    for i in range(len(dirs)):
        path_to_original = os.path.join(path_to_data, dirs[i])
        path_to_save = os.path.join(path_to_test_data, dirs[i])

        #creates dir
        if not os.path.exists(path_to_save):
            os.makedirs(path_to_save)
        files = get_files_from_folder(path_to_original)
        # moves data
        for j in range(int(test_counter[i])):
            dst = os.path.join(path_to_save, files[j])
            src = os.path.join(path_to_original, files[j])
            shutil.move(src, dst)

and when I tried doing the following:

path_to_data= r'path\A'
path_to_test_data=r"path\test"
train_ratio=0.8

main(path_to_data,path_to_test_data,train_ratio)

Nothing really happened..

If I can get this working for train and test, I can easily extend it for validation.


title: Augmenting only the training set in K-folds cross validation

body:
I am trying to create a binary CNN classifier for an unbalanced dataset (class 0 = 4000 images, class 1 = around 250 images), which I want to perform 5-fold cross validation on. Currently I am loading my training set into an ImageLoader that applies my transformations/augmentations(?) and loads it into a DataLoader. However, this results in both my training splits and validation splits containing the augmented data.
I originally applied transformations offline (offline augmentation?) to balance my dataset, but from this thread (https://stats.stackexchange.com/questions/175504/how-to-do-data-augmentation-and-train-validate-split), it seems it would be ideal to only augment the training set. I would also prefer to train my model on solely augmented training data and then validate it on non-augmented data in a 5-fold cross validation
My data is organized as root/label/images, where there are 2 label folders (0 and 1) and images sorted into the respective labels.
My Code So Far
total_set = datasets.ImageFolder(ROOT, transform = data_transforms['my_transforms'])

//Eventually I plan to run cross-validation as such:
splits = KFold(cv = 5, shuffle = True, random_state = 42)

for train_idx, valid_idx in splits.split(total_set):
    train_sampler = SubsetRandomSampler(train_idx)
    valid_sampler = SubsetRandomSampler(valid_idx)

    train_loader = torch.utils.data.DataLoader(total_set, batch_size=32, sampler=train_sampler)
    val_loader = torch.utils.data.DataLoader(total_set, batch_size=32, sampler=valid_sampler)

model.train()
//Model train/eval works but may be overpredict

I'm sure I'm doing something sub-optimally or wrong in this code, but I can't seem to find any documentation on specifically augmenting only the training splits in cross-validation!
Any help would be appreciated!


title: FastAI throwing a runtime error when using custom train & test sets

body:
I’m working on the Food-101 dataset and as you may know, the dataset comes with both train and test parts. Because the dataset could no longer be found on the ETH Zurich link, I had to divide them into partitions < 1GB each and clone them into Colab and reassemble. Its very tedious work but I got it working. I will omit the Python code but the file structure looks like this:

Food-101
      images
            train
               ...75750 train images
            test
               ...25250 test images
      meta
            classes.txt
            labes.txt
            test.json
            test.txt
            train.json
            train.txt
      README.txt
      license_agreement.txt

The following code is what’s throwing the runtime error

train_image_path = Path('images/train/')
test_image_path = Path('images/test/')
path = Path('../Food-101')

food_names = get_image_files(train_image_path)

file_parse = r'/([^/]+)_\d+\.(png|jpg|jpeg)'

data = ImageDataBunch.from_folder(train_image_path, test_image_path, valid_pct=0.2, ds_tfms=get_transforms(), size=224)
data.normalize(imagenet_stats)

My guess is that  ImageDataBunch.from_folder()  is what’s throwing the error but I don’t know why its getting caught up on the data types as (I don’t think) I’m supplying it with any data that has a specific type.

/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py:2854: UserWarning: The default behavior for interpolate/upsample with float scale_factor will change in 1.6.0 to align with other frameworks/libraries, and use scale_factor directly, instead of relying on the computed output size. If you wish to keep the old behavior, please set recompute_scale_factor=True. See the documentation of nn.Upsample for details.
  warnings.warn("The default behavior for interpolate/upsample with float scale_factor will change "
/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py:2854: UserWarning: The default behavior for interpolate/upsample with float scale_factor will change in 1.6.0 to align with other frameworks/libraries, and use scale_factor directly, instead of relying on the computed output size. If you wish to keep the old behavior, please set recompute_scale_factor=True. See the documentation of nn.Upsample for details.
  warnings.warn("The default behavior for interpolate/upsample with float scale_factor will change "
/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py:2854: UserWarning: The default behavior for interpolate/upsample with float scale_factor will change in 1.6.0 to align with other frameworks/libraries, and use scale_factor directly, instead of relying on the computed output size. If you wish to keep the old behavior, please set recompute_scale_factor=True. See the documentation of nn.Upsample for details.
  warnings.warn("The default behavior for interpolate/upsample with float scale_factor will change "
/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py:2854: UserWarning: The default behavior for interpolate/upsample with float scale_factor will change in 1.6.0 to align with other frameworks/libraries, and use scale_factor directly, instead of relying on the computed output size. If you wish to keep the old behavior, please set recompute_scale_factor=True. See the documentation of nn.Upsample for details.
  warnings.warn("The default behavior for interpolate/upsample with float scale_factor will change "
/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py:2854: UserWarning: The default behavior for interpolate/upsample with float scale_factor will change in 1.6.0 to align with other frameworks/libraries, and use scale_factor directly, instead of relying on the computed output size. If you wish to keep the old behavior, please set recompute_scale_factor=True. See the documentation of nn.Upsample for details.
  warnings.warn("The default behavior for interpolate/upsample with float scale_factor will change "
/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py:2854: UserWarning: The default behavior for interpolate/upsample with float scale_factor will change in 1.6.0 to align with other frameworks/libraries, and use scale_factor directly, instead of relying on the computed output size. If you wish to keep the old behavior, please set recompute_scale_factor=True. See the documentation of nn.Upsample for details.
  warnings.warn("The default behavior for interpolate/upsample with float scale_factor will change "
/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py:2854: UserWarning: The default behavior for interpolate/upsample with float scale_factor will change in 1.6.0 to align with other frameworks/libraries, and use scale_factor directly, instead of relying on the computed output size. If you wish to keep the old behavior, please set recompute_scale_factor=True. See the documentation of nn.Upsample for details.
  warnings.warn("The default behavior for interpolate/upsample with float scale_factor will change "
You can deactivate this warning by passing `no_check=True`.
/usr/local/lib/python3.6/dist-packages/fastai/basic_data.py:262: UserWarning: There seems to be something wrong with your dataset, for example, in the first batch can't access these elements in self.train_ds: 9600,37233,16116,38249,1826...
  warn(warn_msg)
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
/usr/local/lib/python3.6/dist-packages/IPython/core/formatters.py in __call__(self, obj)
    697                 type_pprinters=self.type_printers,
    698                 deferred_pprinters=self.deferred_printers)
--> 699             printer.pretty(obj)
    700             printer.flush()
    701             return stream.getvalue()

11 frames
/usr/local/lib/python3.6/dist-packages/fastai/vision/image.py in affine(self, func, *args, **kwargs)
    181         "Equivalent to `image.affine_mat = image.affine_mat @ func()`."
    182         m = tensor(func(*args, **kwargs)).to(self.device)
--> 183         self.affine_mat = self.affine_mat @ m
    184         return self
    185

RuntimeError: Expected object of scalar type Float but got scalar type Double for argument #3 'mat2' in call to _th_addmm_out


title: Is torchvision.datasets.cifar.CIFAR10 a list or not?

body:
When I try to figure out what is inside torchvision.datasets.cifar.CIFAR10, I did some simple code

trainset = torchvision.datasets.CIFAR10(root='./data', train=True,
                                    download=True, transform=transform)
print(trainset[1])
print(trainset[:10])
print(type(trainset))

However, I got some error when I try

print(trainset[:10])

The error info is

TypeError: Cannot handle this data type

I am wondering why I can use trainset[1], but not trainset[:10]?


title: An analog of weighted_cross_entropy_with_logits in PyTorch

body:
I'm trying to train a model with PyTorch. Is there any simple way to create a loss like weighted_cross_entropy_with_logits from Tensorflow?

There are pos_weight argument in weighted_cross_entropy_with_logits that can help with balancing. But there are only weights for labels in the list of arguments in BCEWithLogitsLoss.


title: loading librispeech in pytorch for ASR

body:
I'm newly working to train an automatic speech recognition machine using neural network and CTC loss. But the first thing I'm supposed to do is to prepare the data for training the model. Since the Librispeech contains huge amounts of data, initially I am going to use a subset of it called "Mini LibriSpeech ASR corpus". (http://www.openslr.org/31/). Also I am using SeanNaren Pytorch bindings for Warp-ctc (https://github.com/SeanNaren/warp-ctc).

After reading the audio files and their corresponding transcripts, I'm using Spicy package to calculate the spectrogram of each audio file. The problem arises when I'm going to feed the spectrograms to a convolutional layer for feature extraction. The length of each spectrogram differs from the other ones. After searching more on the issue, I figured out I should probably pass a specific number of frames to the network, but in order to reach this I need to tag each frame of the sound file with the corresponding character(also containing blank symbol).

Is there a way to do that in python?


title: Cuda Out of Memory Error after few successful batches

body:
Here is my code. I am receiving CUDA out of memory error after executing 140 batches successfully.
I have used item method to avoid storing tensor. Used empty_cache, gc collect, retain_graph as False while calling backward. But all in vain. Kindly suggest.

for epoch in range(1, n_epochs+1):
    train_loss = 0.0
    model.train()
    for batch_idx, (image,label) in enumerate(loaders['train']):
    # move to GPU
    if use_cuda:
        image,label = image.cuda(), label.cuda()
    if (batch_idx +1) % 20 == 0:
        print('Batch Id '+ str(batch_idx +1))
    ## find the loss and update the model parameters accordingly
    ## record the average training loss, using something like
    ## train_loss = train_loss + ((1 / (batch_idx + 1)) * (loss.data -
    ## train_loss))
    optimizer_scratch.zero_grad()
    output = model_scratch(image)
    loss = criterion_scratch(output,label)
    ##loss.backward(retain_graph=False)
    loss.backward()

    optimizer_scratch.step()
    train_loss = train_loss + ((1 / (batch_idx + 1)) * (loss.item() -
    train_loss))
    torch.cuda.empty_cache()
    ##gc.collect()


title: How to put datasets created by torchvision.datasets in GPU in one operation?

body:
I’m dealing with CIFAR10 and I use torchvision.datasets to create it. I’m in need of GPU to accelerate the calculation but I can’t find a way to put the whole dataset into GPU at one time. My model need to use mini-batches and it is really time-consuming to deal with each batch separately.

I've tried to put each mini-batch into GPU separately but it seems really time-consuming.


title: How to multiply a dense matrix by a sparse matrix element-wise in pytorch

body: I can use torch.sparse.mm() or torch.spmm() to do multiplication between sparse matrix and dense matrix directly, but which function should I choose to do element-wise multiplication?


title: Display PyTorch model with multiple outputs using torchviz make_dots

body:
I have a model with multiple outputs, 4 to be exact:

 def forward(self, x):
      outputs = []
      for conv, act in zip(self.Convolutions, self.Activations):
           y = conv(x)
           outputs.append(act(y))
      return outputs

I wanted to display it using make_dot from torchviz:

 from torchviz import make_dot
 generator = ...
 batch = next(iter(generator))
 input, output = batch["input"].to(device, dtype=torch.float), batch["output"].to(device, dtype=torch.float)
 dot = make_dot(model(input), params=dict(model.named_parameters()))

But I get the following error:

 File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/torchviz/dot.py", line 37, in make_dot
 output_nodes = (var.grad_fn,) if not isinstance(var, tuple) else tuple(v.grad_fn for v in var)
 AttributeError: 'list' object has no attribute 'grad_fn'

Obviously a list does not have a grad_fn function, but according to this discussion, I can return a list of outputs.

What am I doing wrong?


title: Unsupported ONNX opset version: 11

body:
I'm following this guide to convert darknet to onnx. However, I'm facing the following error:

"C:\Users\Scott\Anaconda3\envs\pytorch_yolov4\lib\site-packages\torch\onnx\symbolic_helper.py",
line 253, in _set_opset_version
raise ValueError("Unsupported ONNX opset version: " + str(opset_version)) ValueError: Unsupported ONNX opset version: 11

What does this error mean and how to deal with it?


title: Expected parameters of Conv2d

body:
Below code :

import torch
import torch.nn as nn
import torchvision
import torchvision.transforms as transforms
import torch.utils.data as data_utils
import numpy as np

train_dataset = []
mu, sigma = 0, 0.1 # mean and standard deviation
num_instances = 20
batch_size_value = 10
for i in range(num_instances) :
    image = []
    image_x = np.random.normal(mu, sigma, 1000).reshape((1 , 100, 10))
    train_dataset.append(image_x)
labels = [1 for i in range(num_instances)]
x2 = torch.tensor(train_dataset).float()
y2 = torch.tensor(labels).long()
my_train2 = data_utils.TensorDataset(x2, y2)
train_loader2 = data_utils.DataLoader(my_train2, batch_size=batch_size_value, shuffle=False)

# Device configuration
device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')

# Hyper parameters
num_epochs = 5
num_classes = 1
batch_size = 5
learning_rate = 0.001

# Convolutional neural network (two convolutional layers)
class ConvNet(nn.Module):
    def __init__(self, num_classes=1):
        super(ConvNet, self).__init__()
        self.layer1 = nn.Sequential(
            nn.Conv2d(1, 16, kernel_size=5, stride=1, padding=2),
            nn.BatchNorm2d(16),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2))
        self.layer2 = nn.Sequential(
            nn.Conv2d(16, 32, kernel_size=5, stride=1, padding=2),
            nn.BatchNorm2d(32),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2))
        self.fc = nn.Linear(7*7*32, num_classes)

    def forward(self, x):
        out = self.layer1(x)
        out = self.layer2(out)
        out = out.reshape(out.size(0), -1)
        out = self.fc(out)
        return out

model = ConvNet(num_classes).to(device)

# Loss and optimizer
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

# Train the model
total_step = len(train_loader2)
for epoch in range(num_epochs):
    for i, (images, labels) in enumerate(train_loader2):
        images = images.to(device)
        labels = labels.to(device)

        # Forward pass
        outputs = model(images)
        loss = criterion(outputs, labels)

        # Backward and optimize
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if (i+1) % 100 == 0:
            print ('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}'
                   .format(epoch+1, num_epochs, i+1, total_step, loss.item()))

returns error :

RuntimeError: size mismatch, m1: [10 x 1600], m2: [1568 x 1] at /pytorch/aten/src/THC/generic/THCTensorMathBlas.cu:249

Reading the documentation for conv2d,  I tried to change the first parameter to 10X100 to match

  input – input tensor of shape (minibatch×in_channels×iH×iW)

from https://pytorch.org/docs/stable/nn.html#torch.nn.functional.conv2d

but then received the error :

RuntimeError: Given groups=1, weight[16, 1000, 5, 5], so expected input[10, 1, 100, 10] to have 1000 channels, but got 1 channels instead

So I'm not sure if I've corrected the original error or just caused a new one?

How should Conv2d be set in order to match image shape of (10,100) ?


title: How to cover a label list under the multi-label classification context into one-hot encoding with pytorch?

body:
I have a list of one batch data with multi-label for every sample. So how to covert it into torch.Tensor in one-hot encoding?
For example, with batch_size=5 and class_num=6,

label =[
[1,2,3],
[4,6],
[1],
[1,4,5],
[4]
]

how to make it into one-hot encoding in pytorch?

label_tensor=tensor([
[1,1,1,0,0,0],
[0,0,0,1,0,1],
[1,0,0,0,0,0],
[1,0,0,1,1,0],
[0,0,0,1,0,0]
])


title: Plotting a torch.Tensor into an image

body:
same_digit_imgs = torch.empty(0, 1, 28, 28)
for digit in range(10):
  similar_img = create_interpolates(similar_pairs[digit][0], similar_pairs[digit][1], encoder, decoder)
  print(type(similar_img), similar_img.shape)
  similar_img = similar_img.view(similar_img.shape[1], similar_img.shape[2], similar_img.shape[0])
  plt.imshow(similar_img)
  plt.show()

I am trying to plot the similar_img Tensor in pytorch.
I am new to pytorch and I am not sure how to plot the image.
The type and the shape is <class 'torch.Tensor'> torch.Size([9, 1, 28, 28])

and I am getting an Error:
RuntimeError: shape '[1, 28, 9]' is invalid for input of size 7056 when I did similar_img.view(similar_img.shape[1], similar_img.shape[2], similar_img.shape[0]).

Any help?


title: PyTorch Circular Padding in one Dimension

body:
for a convolution i want to apply a circular padding in one dimension and a zero padding in all other dimension. How can i do this?
For the convolution there are 28 channels and fore the the data is described in spherical bins. There are 20 bins for radius times 20 bins for polar times 20 bins for inclination.
The circular padding should only be applied for the inclination.
Small Example
# Example:
x = torch.tensor([[1,2,3],
                  [4,5,6],
                  [7,8,9]])
y = sphere_pad(x, pad=(0, 1))

# y is now tensor([[3, 1, 2, 3, 1],
#                  [6, 4, 5, 6, 4],
#                  [9, 7, 8, 9, 7]])

I have tried to apply
def sphere_pad(x, pad=(1,1)):
    return x.repeat(*x.shape)[
        (x.shape[0]-pad[0]):(2*x.shape[0]+pad[0]),
        (x.shape[1]-pad[1]):(2*x.shape[1]+pad[1])]

and then apply a convolution with a normal zero padding ( and no padding in the last dimension).
This works for a small example but this method exceeds the GPU memory for the actual problem size.
Are there other ways?


title: RuntimeError: cuda runtime error (30) : unknown error at /opt/conda/conda-bld/pytorch_1501972792122/work/pytorch-0.1.12/torch/lib/THC/THCGeneral.c:66

body:
I'm getting a runtime error everytime I try to debug pytorch code for the second time. It seems like after restarting the ipython console some of the cuda programs are not canceled properly so that this error occurs while trying to rerun the code.
Has anyone had the same experience / knows how to solve it?
Is there a way to manually flush all cuda activities from my GPU?

I'm using pytorch version 0.1.12_1, ipython 4.2.0 and as IDE spyder 3.2.4.


title: Mask values in segmentation task are either 0 or 255. How do I resolve this?

body:
I must be getting something terribly wrong with the fast-ai library, since I seem to be the only one having this problem. Everytime I try the learning rate finder or training the network, it gives me an error. It took me a week to produce this specific error message, which made me check the mask values. It turns out they are either 0 for background pixels and 255 for foreground ones. This is a problem as I only have two classes. How can I change the 255 values to 1 within my Databunch Object? Is there a way to divide each mask value by 255 or do I need to do it beforehand somehow? I am kinda lost within the process.

Here's the error message i'm getting:

LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.

---------------------------------------------------------------------------

    RuntimeError                              Traceback (most recent call last)

    <ipython-input-20-c7a9c29f9dd1> in <module>()
    ----> 1 learn.lr_find()
          2 learn.recorder.plot()

    8 frames

    /usr/local/lib/python3.6/dist-packages/fastai/train.py in lr_find(learn, start_lr, end_lr, num_it, stop_div, wd)
         39     cb = LRFinder(learn, start_lr, end_lr, num_it, stop_div)
         40     epochs = int(np.ceil(num_it/len(learn.data.train_dl)))
    ---> 41     learn.fit(epochs, start_lr, callbacks=[cb], wd=wd)
         42
         43 def to_fp16(learn:Learner, loss_scale:float=None, max_noskip:int=1000, dynamic:bool=True, clip:float=None,

    /usr/local/lib/python3.6/dist-packages/fastai/basic_train.py in fit(self, epochs, lr, wd, callbacks)
        198         else: self.opt.lr,self.opt.wd = lr,wd
        199         callbacks = [cb(self) for cb in self.callback_fns + listify(defaults.extra_callback_fns)] + listify(callbacks)
    --> 200         fit(epochs, self, metrics=self.metrics, callbacks=self.callbacks+callbacks)
        201
        202     def create_opt(self, lr:Floats, wd:Floats=0.)->None:

    /usr/local/lib/python3.6/dist-packages/fastai/basic_train.py in fit(epochs, learn, callbacks, metrics)
         99             for xb,yb in progress_bar(learn.data.train_dl, parent=pbar):
        100                 xb, yb = cb_handler.on_batch_begin(xb, yb)
    --> 101                 loss = loss_batch(learn.model, xb, yb, learn.loss_func, learn.opt, cb_handler)
        102                 if cb_handler.on_batch_end(loss): break
        103

    /usr/local/lib/python3.6/dist-packages/fastai/basic_train.py in loss_batch(model, xb, yb, loss_func, opt, cb_handler)
         28
         29     if not loss_func: return to_detach(out), to_detach(yb[0])
    ---> 30     loss = loss_func(out, *yb)
         31
         32     if opt is not None:

    /usr/local/lib/python3.6/dist-packages/fastai/layers.py in __call__(self, input, target, **kwargs)
        241         if self.floatify: target = target.float()
        242         input = input.view(-1,input.shape[-1]) if self.is_2d else input.view(-1)
    --> 243         return self.func.__call__(input, target.view(-1), **kwargs)
        244
        245 def CrossEntropyFlat(*args, axis:int=-1, **kwargs):

    /usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
        539             result = self._slow_forward(*input, **kwargs)
        540         else:
    --> 541             result = self.forward(*input, **kwargs)
        542         for hook in self._forward_hooks.values():
        543             hook_result = hook(self, input, result)

    /usr/local/lib/python3.6/dist-packages/torch/nn/modules/loss.py in forward(self, input, target)
        914     def forward(self, input, target):
        915         return F.cross_entropy(input, target, weight=self.weight,
    --> 916                                ignore_index=self.ignore_index, reduction=self.reduction)
        917
        918

    /usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction)
       2007     if size_average is not None or reduce is not None:
       2008         reduction = _Reduction.legacy_get_string(size_average, reduce)
    -> 2009     return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
       2010
       2011

    /usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)
       1836                          .format(input.size(0), target.size(0)))
       1837     if dim == 2:
    -> 1838         ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
       1839     elif dim == 4:
       1840         ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)

    RuntimeError: Assertion `cur_target >= 0 && cur_target < n_classes' failed.  at /pytorch/aten/src/THNN/generic/ClassNLLCriterion.c:97

Also this is how I set up my data:

    data = (SegmentationItemList.from_df(img_df,IMAGE_PATH)
          # import from df in greyscale ('L')
          .split_by_rand_pct(valid_pct=0.15)
          # 1/15 train/validation split
          .label_from_func(get_mask, classes = array(['background','cell']))
          # segmentation mask and classes
          .transform(tfms, tfm_y=True, size=TILE_SHAPE)
          # apply data augmentation
          .databunch(bs=BATCH_SIZE)
          # set batchsize
          .normalize()
    )

Please tell me if you need any more information. I already tried adding an 'after_open' function, which should have divided all by 255 to the 'label_from_func' part. I also know there is a div attribute within the 'open_image' function of fast-ai, which is supposed to normalize RGB values between 0 and 1, but I couldn't find one for 'label_from_func'.

Edit:

I found this post in the fastai community. However even with these answers i was not able to solve my problem. I tried adding this snippet to pass div=True into the open_mask function, but it did not work:

    src.train.y.create_func = partial(open_mask, div=True)
    src.valid.y.create_func = partial(open_mask, div=True)

I also tried .set_attr(mask_opener=partial(open_mask, div=True)) after .label_from_func(), but then it throws this attribute error: AttributeError: setattr

Still need help


title: Pytorch ValueError: Expected target size (2, 13), got torch.Size([2]) when calling CrossEntropyLoss

body:
I am trying to train a Pytorch LSTM network, but I'm getting ValueError: Expected target size (2, 13), got torch.Size([2]) when I try to calculate CrossEntropyLoss. I think I need to change the shape somewhere, but I can't figure out where.
Here is my network definition:
class LSTM(nn.Module):

    def __init__(self, vocab_size, embedding_dim, hidden_dim, n_layers, drop_prob=0.2):
        super(LSTM, self).__init__()

        # network size parameters
        self.n_layers = n_layers
        self.hidden_dim = hidden_dim
        self.vocab_size = vocab_size
        self.embedding_dim = embedding_dim

        # the layers of the network
        self.embedding = nn.Embedding(self.vocab_size, self.embedding_dim)
        self.lstm = nn.LSTM(self.embedding_dim, self.hidden_dim, self.n_layers, dropout=drop_prob, batch_first=True)
        self.dropout = nn.Dropout(drop_prob)
        self.fc = nn.Linear(self.hidden_dim, self.vocab_size)

    def forward(self, input, hidden):
        # Perform a forward pass of the model on some input and hidden state.
        batch_size = input.size(0)
        print(f'batch_size: {batch_size}')

        print(Input shape: {input.shape}')

        # pass through embeddings layer
        embeddings_out = self.embedding(input)
        print(f'Shape after Embedding: {embeddings_out.shape}')

        # pass through LSTM layers
        lstm_out, hidden = self.lstm(embeddings_out, hidden)
        print(f'Shape after LSTM: {lstm_out.shape}')

        # pass through dropout layer
        dropout_out = self.dropout(lstm_out)
        print(f'Shape after Dropout: {dropout_out.shape}')

        #pass through fully connected layer
        fc_out = self.fc(dropout_out)
        print(f'Shape after FC: {fc_out.shape}')

        # return output and hidden state
        return fc_out, hidden

    def init_hidden(self, batch_size):
        #Initializes hidden state
        # Create two new tensors `with sizes n_layers x batch_size x hidden_dim,
        # initialized to zero, for hidden state and cell state of LSTM

        hidden = (torch.zeros(self.n_layers, batch_size, self.hidden_dim), torch.zeros(self.n_layers, batch_size, self.hidden_dim))
        return hidden

I added comments stating the shape of the network at each spot. My data is in a TensorDataset called training_dataset with two attributes, features and labels. Features has shape torch.Size([97, 3]), and
labels has shape: torch.Size([97]).
This is the code for the network training:
# Size parameters
vocab_size = 13
embedding_dim = 256
hidden_dim = 256
n_layers = 2

# Training parameters
epochs = 3
learning_rate = 0.001
clip = 1
batch_size = 2

training_loader = DataLoader(training_dataset, batch_size=batch_size, drop_last=True, shuffle=True)

net = LSTM(vocab_size, embedding_dim, hidden_dim, n_layers)
optimizer = torch.optim.Adam(net.parameters(), lr=learning_rate)
loss_func = torch.nn.CrossEntropyLoss()

net.train()
for e in range(epochs):
    print(f'Epoch {e}')
    print(batch_size)
    hidden = net.init_hidden(batch_size)

    # loops through each batch
    for features, labels in training_loader:

        # resets training history
        hidden = tuple([each.data for each in hidden])
        net.zero_grad()

        # computes gradient of loss from backprop
        output, hidden = net.forward(features, hidden)
        loss = loss_func(output, labels)
        loss.backward()

        # using clipping to avoid exploding gradient
        nn.utils.clip_grad_norm_(net.parameters(), clip)
        optimizer.step()

When I try to run the training I get the following error:
Traceback (most recent call last):
  File "train.py", line 75, in <module>
    loss = loss_func(output, labels)
  File "/usr/local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/torch/nn/modules/loss.py", line 947, in forward
    return F.cross_entropy(input, target, weight=self.weight,
  File "/usr/local/lib/python3.8/site-packages/torch/nn/functional.py", line 2422, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
  File "/usr/local/lib/python3.8/site-packages/torch/nn/functional.py", line 2227, in nll_loss
    raise ValueError('Expected target size {}, got {}'.format(
ValueError: Expected target size (2, 13), got torch.Size([2])

Also here is the result of the print statements:
batch_size: 2
Input shape: torch.Size([2, 3])
Shape after Embedding: torch.Size([2, 3, 256])
Shape after LSTM: torch.Size([2, 3, 256])
Shape after Dropout: torch.Size([2, 3, 256])
Shape after FC: torch.Size([2, 3, 13])

There is some kind of shape error happening, but I can't figure out where. Any help would be appreciated. If relevant I'm using Python 3.8.5 and Pytorch 1.6.0.


title: Invalid syntax error when passing a list of modules in Pytorch

body:
I have two blocks in my deep models which are defined as follows:

def make_conv_bn_relu(in_channels, out_channels, kernel_size=3, stride=1, padding=1):
    return [
        nn.Conv2d(in_channels, out_channels, kernel_size=kernel_size, stride=stride, padding=padding, bias=False),
        nn.BatchNorm2d(out_channels),
        nn.ReLU(inplace=True),
    ]

def make_conv_relu(in_channels, out_channels, kernel_size=3, stride=1, padding=1):
    return [
        nn.Conv2d(in_channels, out_channels, kernel_size=kernel_size, stride=stride, padding=padding, bias=True),
        nn.ReLU(inplace=True),
    ]

Now, I want to pass it in nn.Sequential.

self.down1 = nn.Sequential(*make_conv_bn_relu(in_channels, 16, kernel_size=3, stride=1, padding=1 ), *make_conv_bn_relu(16, 32, kernel_size=3, stride=2, padding=1 ),)

But I am getting the following error:

Traceback (most recent call last):
  File "train_unet.py", line 17, in <module>
    from net.model.unet1 import UNet256_3x3 as Net
  File "/home/avijit.d/Kaggle/Pytorch/source/dummy-01/net/model/unet1.py", line 40
    self.down1 = nn.Sequential(*make_conv_bn_relu(in_channels, 16, kernel_size=3, stride=1, padding=1 ), *make_conv_bn_relu(16, 32, kernel_size=3, stride=2, padding=1 ),)
                                                                                ^
SyntaxError: invalid syntax

How to get rid of this? I am using Python 2.7


title: Pytorch List object has no attribute ‘to’

body:
I am new to Pytorch. I am using a pre-trained model (RESNET-50). And i am trying to train this model on MS-COCO dataset using cocoapi.
I have loaded my dataset images and annotations in train loader.
for images,labels in train_loader:
    print(type(images))
    print(type(labels))
    print(images.size())
    print(images.dim())
    steps+=1
    images, labels = images.to(device), labels.to(device)

i got the following output:
class 'torch.Tensor'
class 'list'
torch.Size([64, 3, 224, 224])
4
And i got this Error: I think it is because labels is not a Tensor?


title: Conda Runtime error when installing PyTorch using anaconda

body:
When installing pytorch using anaconda I am continiously getting Conda Runtime Error

Traceback (most recent call last):
  File "/home/codehead/anaconda2/lib/python2.7/site-packages/conda/exceptions.py", line 479, in conda_exception_handler
    return_value = func(*args, **kwargs)
  File "/home/codehead/anaconda2/lib/python2.7/site-packages/conda/cli/main.py", line 145, in _main
    exit_code = args.func(args, p)
  File "/home/codehead/anaconda2/lib/python2.7/site-packages/conda/cli/main_install.py", line 80, in execute
    install(args, parser, 'install')
  File "/home/codehead/anaconda2/lib/python2.7/site-packages/conda/cli/install.py", line 420, in install
    raise CondaRuntimeError('RuntimeError: %s' % e)
CondaRuntimeError: Runtime error: RuntimeError: Runtime error: Could not open u'/home/codehead/anaconda2/pkgs/pytorch-0.1.11-py27_5.tar.bz2.part' for writing (HTTPSConnectionPool(host='binstar-cio-packages-prod.s3.amazonaws.com', port=443): Read timed out.).

The command I am using to install PyTorch is

 conda install -c soumith pytorch=0.1.11


title: Running a PyTorch dataloader/Dataset on multiple distributed CPUs

body:
I wonder if there is a way to distributed the dataloader/Dataset to many CPUs, even when using a single GPU.
Specifically, I would like to have have a Dataset class, and the __getitem__ function will be distributed across many different CPU's (using mpi maybe? but any other way is also good).

Thanks

EDIT
My title was erroneously edited, I am not trying to distribute the model itslef, I only want to distribute the data loading/parsing of the model


title: Function AddBackward0 returned an invalid gradient at index 1 - expected type torch.FloatTensor but got torch.cuda.FloatTensor

body:
My Model:

class myNet(nn.Module):
    def __init__(self):
        super(myNet,self).__init__()
        self.act1=Dynamic_relu_b(64)
        self.conv1=nn.Conv2d(3,64,3)
        self.pool=nn.AdaptiveAvgPool2d(1)
        self.fc=nn.Linear(128,20)
    def forward(self,x):
        x=self.conv1(x)
        x=self.act1(x)
        x=self.pool(x)
        x=x.view(x.shape[0],-1)
        x=self.fc(x)

        return x

A code that replicates the experiment is provided：

def one_hot_smooth_label(x,num_class,smooth=0.1):
    num=x.shape[0]
    labels=torch.zeros((num,20))
    for i in range(num):
        labels[i][x[i]]=1
    labels=(1-(num_class-1)/num_class*smooth)*labels+smooth/num_class
    return labels
images=torch.rand((4,3,300,300))
images=images.cuda()
labels=torch.from_numpy(np.array([1,0,0,1]))
model=myNet()
model=model.cuda()

output=model(images)
labels=one_hot_smooth_label(labels,20)
labels = labels.cuda()
criterion=nn.BCEWithLogitsLoss()

loss=criterion(output,labels)
loss.backward()

The error:

    RuntimeError                              Traceback (most recent call last)
<ipython-input-42-1268777e87e6> in <module>()
     21
     22 loss=criterion(output,labels)
---> 23 loss.backward()

1 frames
/usr/local/lib/python3.6/dist-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
     98     Variable._execution_engine.run_backward(
     99         tensors, grad_tensors, retain_graph, create_graph,
--> 100         allow_unreachable=True)  # allow_unreachable flag
    101
    102

RuntimeError: Function AddBackward0 returned an invalid gradient at index 1 - expected type TensorOptions(dtype=float, device=cpu, layout=Strided, requires_grad=false) but got TensorOptions(dtype=float, device=cuda:0, layout=Strided, requires_grad=false) (validate_outputs at /pytorch/torch/csrc/autograd/engine.cpp:484)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x46 (0x7fcf7711b536 in /usr/local/lib/python3.6/dist-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x2d84224 (0x7fcfb1bad224 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so)
frame #2: torch::autograd::Engine::evaluate_function(std::shared_ptr<torch::autograd::GraphTask>&, torch::autograd::Node*, torch::autograd::InputBuffer&) + 0x548 (0x7fcfb1baed58 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so)
frame #3: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&, bool) + 0x3d2 (0x7fcfb1bb0ce2 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so)
frame #4: torch::autograd::Engine::thread_init(int) + 0x39 (0x7fcfb1ba9359 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so)
frame #5: torch::autograd::python::PythonEngine::thread_init(int) + 0x38 (0x7fcfbe2e8378 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so)
frame #6: <unknown function> + 0xbd6df (0x7fcfe23416df in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
frame #7: <unknown function> + 0x76db (0x7fcfe34236db in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #8: clone + 0x3f (0x7fcfe375c88f in /lib/x86_64-linux-gnu/libc.so.6)

After many experiments, I found that act1 in the model was the problem. If you delete act1, the error will not appear！

But I don't know why act1 has this problem.

What seems to be the wrong part of the error is requiers_grad=False, and I don't know which part set this.

This is the code about act1(Dynamic_relu_b):

class Residual(nn.Module):
    def __init__(self, in_channel, R=8, k=2):
        super(Residual, self).__init__()
        self.avg = nn.AdaptiveAvgPool2d((1, 1))
        self.relu = nn.ReLU(inplace=True)
        self.R = R
        self.k = k
        out_channel = int(in_channel / R)
        self.fc1 = nn.Linear(in_channel, out_channel)
        fc_list = []
        for i in range(k):
            fc_list.append(nn.Linear(out_channel, 2 * in_channel))
        self.fc2 = nn.ModuleList(fc_list)

    def forward(self, x):
        x = self.avg(x)
        x = torch.squeeze(x)
        x = self.fc1(x)
        x = self.relu(x)
        result_list = []
        for i in range(self.k):
            result = self.fc2[i](x)
            result = 2 * torch.sigmoid(result) - 1
            result_list.append(result)
        return result_list

class Dynamic_relu_b(nn.Module):
    def __init__(self, inchannel, R=8, k=2):
        super(Dynamic_relu_b, self).__init__()
        self.lambda_alpha = 1
        self.lambda_beta = 0.5
        self.R = R
        self.k = k
        self.init_alpha = torch.zeros(self.k)
        self.init_beta = torch.zeros(self.k)
        self.init_alpha[0] = 1
        self.init_beta[0] = 1
        for i in range(1, k):
            self.init_alpha[i] = 0
            self.init_beta[i] = 0

        self.residual = Residual(inchannel)

    def forward(self, input):
        delta = self.residual(input)
        in_channel = input.shape[1]
        bs = input.shape[0]
        alpha = torch.zeros((self.k, bs, in_channel))
        beta = torch.zeros((self.k, bs, in_channel))
        for i in range(self.k):
            for j, c in enumerate(range(0, in_channel * 2, 2)):
                alpha[i, :, j] = delta[i][:, c]
                beta[i, :, j] = delta[i][:, c + 1]
        alpha1 = alpha[0]
        beta1 = beta[0]
        max_result = self.dynamic_function(alpha1, beta1, input, 0)
        for i in range(1, self.k):
            alphai = alpha[i]
            betai = beta[i]
            result = self.dynamic_function(alphai, betai, input, i)
            max_result = torch.max(max_result, result)
        return max_result

    def dynamic_function(self, alpha, beta, x, k):
        init_alpha = self.init_alpha[k]
        init_beta = self.init_beta[k]
        alpha = init_alpha + self.lambda_alpha * alpha
        beta = init_beta + self.lambda_beta * beta
        bs = x.shape[0]
        channel = x.shape[1]
        results = torch.zeros_like(x)
        for i in range(bs):
            for c in range(channel):
                results[i, c, :, :] = x[i, c] * alpha[i, c] + beta[i, c]
        return results

How should I solve this problem?


title: How to fix 'str' object has no attribute 'seek' error in pytorch?

body:
I am trying to implement this code https://github.com/eriklindernoren/PyTorch-YOLOv3 ,and train it with my own dataset but i am running into the below error always while trying to validate it

---- Evaluating Model ----
Detecting objects:   0%|                                                                                | 0/1 [00:00<?, ?it/s]Traceback (most recent call last):
  File "train.py", line 160, in <module>
    batch_size=8,
  File "/home/harshvardhan/PyTorch-YOLOv3/test.py", line 36, in evaluate
    for batch_size, (imgs, targets) in enumerate(tqdm.tqdm(dataloader, desc="Detecting objects")):
  File "/home/harshvardhan/.conda/envs/pytorch/lib/python3.6/site-packages/tqdm/std.py", line 1081, in __iter__
    for obj in iterable:
  File "/home/harshvardhan/.conda/envs/pytorch/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 819, in __next__
    return self._process_data(data)
  File "/home/harshvardhan/.conda/envs/pytorch/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 846, in _process_data
    data.reraise()
  File "/home/harshvardhan/.conda/envs/pytorch/lib/python3.6/site-packages/torch/_utils.py", line 369, in reraise
    raise self.exc_type(msg)
AttributeError: Caught AttributeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/home/harshvardhan/.conda/envs/pytorch/lib/python3.6/site-packages/PIL/Image.py", line 2770, in open
    fp.seek(0)
AttributeError: 'str' object has no attribute 'seek'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/harshvardhan/.conda/envs/pytorch/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/harshvardhan/.conda/envs/pytorch/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/harshvardhan/.conda/envs/pytorch/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/harshvardhan/PyTorch-YOLOv3/utils/datasets.py", line 86, in __getitem__
    img = transforms.ToTensor()(Image.open(img_path).convert('RGB'))
  File "/home/harshvardhan/.conda/envs/pytorch/lib/python3.6/site-packages/PIL/Image.py", line 2772, in open
    fp = io.BytesIO(fp.read())
AttributeError: 'str' object has no attribute 'read'


title: pip install torch killed at 99% -- Excessive memory usage

body:
This is while I was installing torch on my laptop. It was getting killed continuously so I thought I will check the memory usage. It hanged my laptop, I had to take a picture with my phone.

If you can't see the image below, it shows pip using 5.8 GiB memory out of 7.8 GiB available. That was a sudden spike at 99%.

System Monitor pip memory usage


title: Pytorch 'CUDA out of memory' with only one training batch

body:
I am currently trying to implement a network similar to U-Net and am having some weird issues with loading my data to the GPU.
Specifically, when I load my model to the GPU with

model = CNN(NUM_LAYERS)
model.to(DEVICE)

the memory usage on the GPU rises to 1063 MiB. This tells me that my model is roughly one GB in memory. So far so good. Then, with

inst = (iter(dataloader)).__next__()
input_tensor = inst[0].to(DEVICE, dtype=torch.float)

I load one batch of training data to the GPU. The total GPU memory usage rises to 1093 MiB, which seems about right and tells me that the batch of 8 training images is about 30MB in size. Now I execute

output = model(input_tensor)

and the memory usage shoots up to 21123 MiB! And if I change the batch size to something bigger, say 16, I get
CUDA out of memory (GPU 1; 31.72 GiB total capacity; 29.91 GiB already allocated; 359.75 MiB free; 402.84 MiB cached).

I've tried to figure out what exactly happens when I feed a tensor to the model, but I can't seem to work out why the GPU memory would suddenly increase like that. Am I correct in saying that at this point I should have exactly three things on the GPU: the model, the input tensor and the output tensor?

The dataloader , by the way looks like this:

class AsyncCNNData(Dataset):
    def __init__(self, list_paths, resize_dims=None, transform=None, device='cpu'):
        'Initialization'
        super().__init__()

        self.list_paths = list_paths
        self.transform = transform
        self.to_tensor = transforms.ToTensor()
        self.device = device
        self.resize_dims=resize_dims

    def __len__(self):
        'Denotes the total number of samples'
        return len(self.list_paths)

    def __getitem__(self, index):
        'Generates one sample of data'
        # Select sample
        sample_path = self.list_paths[index]

        # Load data and get label
        fs = cv.FileStorage(sample_path, cv.FILE_STORAGE_READ)

        cti_list = []
        for i in range(0, NUM_LAYERS, 1):
            img_name = "layer_{0:03d}".format(i)
            img = (fs.getNode(img_name)).mat()
            diff = np.subtract(self.resize_dims, img.shape)/2
            img = cv.copyMakeBorder(img, int(diff[0]), int(diff[0]), int(diff[1]), int(diff[1]), cv.BORDER_CONSTANT, None, 1)
            cti_list.append(img)

        input_tensor = np.stack(cti_list)
        input_tensor = input_tensor[np.newaxis, :, :, :]

        img = (fs.getNode("frame")).mat()
        diff = np.subtract(self.resize_dims, img.shape)/2
        img = cv.copyMakeBorder(img, int(diff[0]), int(diff[0]), int(diff[1]), int(diff[1]), cv.BORDER_CONSTANT, None, 1)
        img = img[:, :, np.newaxis]
        output_tensor = self.to_tensor(img)

        if self.transform != None:
            input_tensor = self.transform(input_tensor)
        else:
            input_tensor = torch.from_numpy(input_tensor)

        return input_tensor, output_tensor, sample_path

and

yaml_files = pd.read_csv(DATA_PATH).values.flatten().tolist()
data = AsyncCNNData(yaml_files, device=DEVICE, resize_dims=RESIZE)

train_size = int(len(data)*TRAIN_RATIO)
data_train, data_test = random_split(data, [train_size, len(data)-train_size])
dataloader = DataLoader(data_train, batch_size=BATCH_SIZE, shuffle=True)

Many many thanks!


title: How to train the original U-Net model with PyTorch?

body:
I’m trying to implement and train the original U-Net model, but I’m stuck in when I’m trying to train the model using the ISBI Challenge Dataset.

According with the original U-Net model, the network outputs an image with 2 channels and size of 388 x 388. So, my data loader for training generates a tensor with size of [batch, channels=1, width=572, height=572] for the input images and [batch, channels=2, width=388, width=388] for target/output images.

My problem actually is that when I’m trying to use the nn.CrossEntropyLoss() the following error is raised:

  RuntimeError: invalid argument 3: only batches of spatial targets supported (3D tensors) but got targets of dimension: 4 at /opt/conda/conda-bld/pytorch_1556653099582/work/aten/src/THNN/generic/SpatialClassNLLCriterion.c:59

I’m just starting with PyTorch (newbie here)… so, I’ll really appreciate if someone could help me to overcome this problem.

The sourcecode is available on GitHub:

https://github.com/dalifreire/cnn_unet_pytorch
https://github.com/dalifreire/cnn_unet_pytorch/blob/master/unet_pytorch.ipynb

Best regards!

UPDATE

I just remove the channel dimension from my masks and everything works well… now I’m generating masks with the shape 1 [width=388, height=388].

After that, I’m working with input images (X), target masks (y) and predicted output masks (y_hat) as follow:

X     --> torch.Size([10, 1, 572, 572])
y     --> torch.Size([10, 388, 388])
y_hat --> torch.Size([10, 2, 388, 388])

But, I don’t understand why target masks (y) and predicted masks (y_hat) must have different shapes? It’s so weird for me…


title: Error when upgrading PyTorch to 0.3.0

body:
I tried to update PyTorch to the recently released 0.3.0,

conda install pytorch=0.3.0 torchvision -c pytorch

But I'm getting the following error:

NoPackagesFoundError: Dependency missing in current linux-64 channels:
 - pytorch 0.3.0* -> mkl >=2018


title: Understanding PyTorch prediction

body:
For my trained model this code :

model(x[0].reshape(1,784).cuda())

returns :

tensor([[-1.9903, -4.0458, -4.1143, -4.0074, -3.5510,  7.1074]], device='cuda:0')

My network model is defined as :

# Hyper-parameters
input_size = 784
hidden_size = 50
num_classes = 6
num_epochs = 5000
batch_size = 1
learning_rate = 0.0001

    criterion = nn.CrossEntropyLoss()
    optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

        class NeuralNet(nn.Module):
            def __init__(self, input_size, hidden_size, num_classes):
                super(NeuralNet, self).__init__()
                self.fc1 = nn.Linear(input_size, hidden_size)
                self.relu = nn.ReLU()
                self.fc2 = nn.Linear(hidden_size, num_classes)

            def forward(self, x):
                out = self.fc1(x)
                out = self.relu(out)
                out = self.fc2(out)
                return out

I'm attempting to understand the returned value :

tensor([[-1.9903, -4.0458, -4.1143, -4.0074, -3.5510,  7.1074]], device='cuda:0')

The value 7.1074 is the most probable as it maximum value in tensor array ? As 7.1074 is at position 5, is the significance here that the associated output value being predicted for input x[0] is 5 ? If so what is the intuition behind this ?


title: How to reverse the operation of torch.nn.functional.grid_sample?

body:
If we get two images, target image tgt_img and source image src_img. And, we also know the grid grid from source image to target image. Thus, we can use the F.grid_sample to generate target image from source image.

tgt_img = F.grid_sample(src_img, grid)

Is there any good idea to get the inv_grid that are from target image to source image. So that:

inv_grid = ?
src_img =  F.grid_sample(tgt_img, inv_grid)

I have tried to read the data in grid and generate a inv_grid, but I want to know if there any better and faster method

Here is an example, a grid is given as is shown below:
a grid that represent the coordinates correspondence from src_img to tgt_img

Then I want to use this grid to generate the inv_grid, which represent the coordinates correspondence from tgt_img to src_img. The result are shown here:
tgt_img to src_img


title: pytorch move model to gpu causes runtime error ('self' as cpu device type)

body:
Solved by adding .to(device) for each weights and bias like t.Tensor(w_txt[0]).to(device)) and it works perfectly on gpu!

My network works well on cpu, and I try to move my network to gpu by adding some commented lines as follows.
import torch as t
import torch.nn as nn
from torch.autograd import Variable
import pandas as pd
import numpy as np
import torch.nn.functional as F

device = t.device("cuda:0" if t.cuda.is_available() else "cpu")

def mapminmax(data,dim):
    min = np.min(data,axis=dim)
    max = np.max(data,axis=dim)
    return (data-min)/(max-min)

w_h1 = np.loadtxt('hidden1weight.txt')
w_h2 = np.loadtxt('hidden2weight.txt')
w_h3 = np.loadtxt('hidden3weight.txt')
w_h4 = np.loadtxt('predictweight.txt')
w_txt = np.array([w_h1,w_h2,w_h3,w_h4])
b_h1 = np.loadtxt('hidden1bias.txt')
b_h2 = np.loadtxt('hidden2bias.txt')
b_h3 = np.loadtxt('hidden3bias.txt')
b_h4 = np.loadtxt('predictbias.txt')
b_txt = np.array([b_h1,b_h2,b_h3,b_h4])

def parameter_init_self():
    model.hidden1.weight = t.nn.Parameter(t.Tensor(w_txt[0]))
    model.hidden2.weight = t.nn.Parameter(t.Tensor(w_txt[1]))
    model.hidden3.weight = t.nn.Parameter(t.Tensor(w_txt[2]))
    model.predict.weight = t.nn.Parameter(t.Tensor(w_txt[3]))
    model.hidden1.bias = t.nn.Parameter(t.Tensor(b_txt[0]))
    model.hidden2.bias = t.nn.Parameter(t.Tensor(b_txt[1]))
    model.hidden3.bias = t.nn.Parameter(t.Tensor(b_txt[2]))
    model.predict.bias = t.nn.Parameter(t.Tensor(b_txt[3]))

class DNNF(nn.Module):
    def __init__(self, input_size, hidden_size1, hidden_size2, hidden_size3, output_size):
        super(DNNF, self).__init__()
        self.hidden1 = t.nn.Linear(input_size, hidden1_size)
        self.hidden2 = t.nn.Linear(hidden1_size, hidden2_size)
        self.hidden3 = t.nn.Linear(hidden2_size, hidden3_size)
        self.predict = t.nn.Linear(hidden3_size, output_size)
        # self = self.cuda()
    def forward(self, x):
        x = t.tanh(self.hidden1(x)) # line 88
        x = t.tanh(self.hidden2(x))
        x = t.tanh(self.hidden3(x))
        y_pred = self.predict(x)
        return y_pred

x_df = pd.read_csv('InputData_Train.csv', header=None)
y_df = pd.read_csv('OutputData_Train.csv', header=None)
x = t.tensor(x_df.values).float()
y = t.tensor(y_df.values).float()
M = x.size()[0]
input_size = x.size()[1]
output_size = y.size()[1]
hidden1_size, hidden2_size, hidden3_size = 50, 50, 50
x_np = x.numpy()
y_np = y.numpy()
x_norm_np = mapminmax(x_np,0)
y_norm_np = mapminmax(y_np,0)
x = t.from_numpy(x_norm_np)
y = t.from_numpy(y_norm_np)
x = Variable(x, requires_grad=False)
y = Variable(y, requires_grad=False)

# x, y = x.to(device), y.to(device)  # move to gpu

loss_fn = nn.MSELoss(reduction='mean')

lr = 1e-2
mm = 0.9
EPOCH = 300

model = DNNF(input_size, hidden1_size, hidden2_size, hidden3_size, output_size)
# model = model.to(device)  # move to gpu

parameter_init_self()

param = {}
for name, para in model.named_parameters():
    param[name] = para.detach().numpy()

optimizer = t.optim.Rprop(model.parameters(), lr=lr)  # lr=1e-2

DISPINTERVAL = 10

for epoch in range(EPOCH):
    y_pred = model(x) # line 172
    loss = loss_fn(y_pred, y)
    if (epoch + 1) % DISPINTERVAL == 0:
        print('Epoch[{}/{}], loss:{:.10f}'.format(epoch+1, EPOCH, loss.item()))
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

y_pred = model(x)
E = y - y_pred
SE = E*E
MSE = t.mean(SE)
loss = loss_fn(y_pred, y)

print('MSE: ', MSE.item())
print('loss: ', loss.item())

However I got this error
Expected object of device type cuda but got device type cpu for argument #1 'self' in call to _th_addmm
  File "E:\MachineLearning\PyTorch\a2MLP_V6_GPU.py", line 88, in forward
    x = t.tanh(self.hidden1(x))
  File "E:\MachineLearning\PyTorch\a2MLP_V6_GPU.py", line 172, in <module>
    y_pred = model(x)

I tried to search on google but got no answer, the most mentioned thing is that there may be parameters defined within the model that are not moved to gpu. But as far as I can see, there is no such parameter in my model.
The error indicates that the problem is around self parameter and I don't know about it since it is not a parameter defined by me. Anyone know what's wrong?