title: Pytorch RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

body:
This code is built up as follows: My robot takes a picture, some tf computer vision model calculates where in the picture the target object starts. This information (x1 and x2 coordinate) is passed to a pytorch model. It should learn to predict the correct motor activations, in order to get closer to the target. After the movement is executed, the robot takes a picture again and the tf cv model should calculate whether the motor activation brought the robot closer to the desired state (x1 at 10, x2 coordinate at at31)

However every time i run the code pytorch is not able to calculate the gradients.

I'm wondering if this is some data-type problem or if it is a more general one: Is it impossible to calculate the gradients if the loss is not calculated directly from the pytorch network's output?

Any help and suggestions will be greatly appreciated.

#define policy model (model to learn a policy for my robot)
import torch
import torch.nn as nn
import torch.nn.functional as F
class policy_gradient_model(nn.Module):
    def __init__(self):
        super(policy_gradient_model, self).__init__()
        self.fc0 = nn.Linear(2, 2)
        self.fc1 = nn.Linear(2, 32)
        self.fc2 = nn.Linear(32, 64)
        self.fc3 = nn.Linear(64,32)
        self.fc4 = nn.Linear(32,32)
        self.fc5 = nn.Linear(32, 2)
    def forward(self,x):
        x = self.fc0(x)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.relu(self.fc3(x))
        x = F.relu(self.fc4(x))
        x = F.relu(self.fc5(x))
        return x

policy_model = policy_gradient_model().double()
print(policy_model)
optimizer = torch.optim.AdamW(policy_model.parameters(), lr=0.005, betas=(0.9,0.999), eps=1e-08, weight_decay=0.01, amsgrad=False)

#make robot move as predicted by pytorch network (not all code included)
def move(motor_controls):
#define curvature
 #   motor_controls[0] = sigmoid(motor_controls[0])
    activation_left = 1+(motor_controls[0])*99
    activation_right = 1+(1- motor_controls[0])*99

    print("activation left:", activation_left, ". activation right:",activation_right, ". time:", motor_controls[1]*100)

#start movement

#main
import cv2
import numpy as np
import time
from torch.autograd import Variable
print("start training")
losses=[]
losses_end_of_epoch=[]
number_of_steps_each_epoch=[]
loss_function = nn.MSELoss(reduction='mean')

#each epoch
for epoch in range(2):
    count=0
    target_reached=False
    while target_reached==False:
        print("epoch: ", epoch, ". step:", count)
###process and take picture
        indices = process_picture()
###binary_network(sliced)=indices as input for policy model
        optimizer.zero_grad()
###output: 1 for curvature, 1 for duration of movement
        motor_controls = policy_model(Variable(torch.from_numpy(indices))).detach().numpy()
        print("NO TANH output for motor: 1)activation left, 2)time ", motor_controls)
        motor_controls[0] = np.tanh(motor_controls[0])
        motor_controls[1] = np.tanh(motor_controls[1])
        print("TANH output for motor: 1)activation left, 2)time ", motor_controls)
###execute suggested action
        move(motor_controls)
###take and process picture2 (after movement)
        indices = (process_picture())
###loss=(binary_network(picture2) - desired
        print("calculate loss")
        print("idx", indices, type(torch.tensor(indices)))
     #   loss = 0
      #  loss = (indices[0]-10)**2+(indices[1]-31)**2
       # loss = loss/2
        print("shape of indices", indices.shape)
        array=np.zeros((1,2))
        array[0]=indices
        print(array.shape, type(array))
        array2 = torch.ones([1,2])
        loss = loss_function(torch.tensor(array).double(), torch.tensor([[10.0,31.0]]).double()).float()
        print("loss: ", loss, type(loss), loss.shape)
       # array2[0] = loss_function(torch.tensor(array).double(),
        torch.tensor([[10.0,31.0]]).double()).float()
        losses.append(loss)
#start line causing the error-message (still part of main)
###calculate gradients
        loss.backward()
#end line causing the error-message (still part of main)

###apply gradients
        optimizer.step()

#Output (so far as intented) (not all included)

#calculate loss
idx [14. 15.] <class 'torch.Tensor'>
shape of indices (2,)
(1, 2) <class 'numpy.ndarray'>
loss:  tensor(136.) <class 'torch.Tensor'> torch.Size([])

#Error Message:
Traceback (most recent call last):
  File "/home/pi/Desktop/GradientPolicyLearning/PolicyModel.py", line 259, in <module>
    array2.backward()
  File "/home/pi/.local/lib/python3.7/site-packages/torch/tensor.py", line 134, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/pi/.local/lib/python3.7/site-packages/torch/autograd/__init__.py", line 99, in
 backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn


title: Backward function in PyTorch

body:
I have some question about pytorch's backward function I don't think I'm getting the right output :
import numpy as np
import torch
from torch.autograd import Variable
a = Variable(torch.FloatTensor([[1,2,3],[4,5,6]]), requires_grad=True)
out = a * a
out.backward(a)
print(a.grad)

the output is
tensor([[ 2.,  8., 18.],
        [32., 50., 72.]])

maybe it's 2*a*a
but i think the output suppose to be
tensor([[ 2.,  4., 6.],
        [8., 10., 12.]])

2*a. cause d(x^2)/dx=2x


title: How to do scatter and gather operations in numpy?

body:
I want to implement scatter and gather operations of Tensorflow or PyTorch in Numpy. I have been scratching my head for a while. Any pointers are greatly appreciated!

PyTorch Scatter
PyTorch Gather


title: can't find the inplace operation: one of the variables needed for gradient computation has been modified by an inplace operation

body:
I am trying to compute a loss on the jacobian of the network (i.e. to perform double backprop), and I get the following error:
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation

I can't find the inplace operation in my code, so I don't know which line to fix.

*The error occurs in the last line:
            loss3.backward()

            inputs_reg = Variable(data, requires_grad=True)
            output_reg = self.model.forward(inputs_reg)

            num_classes = output.size()[1]
            jacobian_list = []
            grad_output = torch.zeros(*output_reg.size())

            if inputs_reg.is_cuda:
                grad_output = grad_output.cuda()
                jacobian_list = jacobian.cuda()

            for i in range(10):

                zero_gradients(inputs_reg)
                grad_output.zero_()
                grad_output[:, i] = 1
                jacobian_list.append(torch.autograd.grad(outputs=output_reg,
                                                  inputs=inputs_reg,
                                                  grad_outputs=grad_output,
                                                  only_inputs=True,
                                                  retain_graph=True,
                                                  create_graph=True)[0])

            jacobian = torch.stack(jacobian_list, dim=0)
            loss3 = jacobian.norm()
            loss3.backward()


title: Porting PyTorch code from CPU to GPU

body:
Following the tutorial from https://github.com/spro/practical-pytorch/blob/master/seq2seq-translation/seq2seq-translation.ipynb

There is a USE_CUDA flag that is used to control the variable and tensor types between CPU (when False) to GPU (when True) types.

Using the data from en-fr.tsv and converting the sentences to variables:

import unicodedata
import string
import re
import random
import time
import math

from gensim.corpora.dictionary import Dictionary

import torch
import torch.nn as nn
from torch.autograd import Variable
from torch import LongTensor, FloatTensor
from torch import optim
import torch.nn.functional as F

import numpy as np

MAX_LENGTH = 10
USE_CUDA = False

# Turn a Unicode string to plain ASCII, thanks to http://stackoverflow.com/a/518232/2809427
def unicode_to_ascii(s):
    return ''.join(
        c for c in unicodedata.normalize('NFD', s)
        if unicodedata.category(c) != 'Mn'
    )

# Lowercase, trim, and remove non-letter characters
def normalize_string(s):
    s = unicode_to_ascii(s.lower().strip())
    s = re.sub(r"([.!?])", r" \1", s)
    s = re.sub(r"[^a-zA-Z.!?]+", r" ", s)
    return s

SOS_IDX, SOS_TOKEN = 0, '<s>'
EOS_IDX, EOS_TOKEN = 1, '</s>'
UNK_IDX, UNK_TOKEN = 2, '<unk>'
PAD_IDX, PAD_TOKEN = 3, '<blank>'

lines = open('en-fr.tsv').read().strip().split('\n')
pairs = [[normalize_string(s).split() for s in l.split('\t')] for l in lines]
src_sents, trg_sents = zip(*pairs)

src_dict = Dictionary([[SOS_TOKEN, EOS_TOKEN, UNK_TOKEN, PAD_TOKEN]])
src_dict.add_documents(src_sents)

trg_dict = Dictionary([[SOS_TOKEN, EOS_TOKEN, UNK_TOKEN, PAD_TOKEN]])
trg_dict.add_documents(trg_sents)

def variablize_sentences(sentence, dictionary):
    indices = [dictionary.token2id[tok] for tok in sentence] + [dictionary.token2id[EOS_TOKEN]]
    var = Variable(LongTensor(indices).view(-1, 1))
    return var.cuda() if USE_CUDA else var

input_variables = [variablize_sentences(sent, src_dict) for sent in src_sents]
output_variables = [variablize_sentences(sent, trg_dict) for sent in trg_sents]

And using a Encoder-Attn-Decoder network:

class EncoderRNN(nn.Module):
    def __init__(self, input_size, hidden_size, n_layers=1):
        super(EncoderRNN, self).__init__()

        self.input_size = input_size
        self.hidden_size = hidden_size
        self.n_layers = n_layers

        self.embedding = nn.Embedding(input_size, hidden_size)
        self.gru = nn.GRU(hidden_size, hidden_size, n_layers)

        self.embedding = self.embedding.cuda() if USE_CUDA else self.embedding
        self.gru = self.gru.cuda() if USE_CUDA else self.gru

    def forward(self, word_inputs, hidden):
        seq_len = len(word_inputs)

        embedded = self.embedding(word_inputs).view(seq_len, 1, -1)
        embedded = embedded.cuda() if USE_CUDA else embedded

        output, hidden = self.gru(embedded, hidden)
        output = output.cuda() if USE_CUDA else output
        hiddne = hidden.cuda() if USE_CUDA else hidden

        return output, hidden

    def init_hidden(self):
        hidden = Variable(torch.zeros(self.n_layers, 1, self.hidden_size))
        return hidden.cuda() if USE_CUDA else hidden

class Attn(nn.Module):
    def __init__(self, method, hidden_size, max_length=MAX_LENGTH):
        super(Attn, self).__init__()

        self.method = method
        self.hidden_size = hidden_size

        if self.method == 'general':
            self.attn = nn.Linear(self.hidden_size, hidden_size)

        elif self.method == 'concat':
            self.attn = nn.Linear(self.hidden_size * 2, hidden_size)
            self.other = nn.Parameter(FloatTensor(1, hidden_size))

    def forward(self, hidden, encoder_outputs):
        seq_len = len(encoder_outputs)

        # Create variable to store attention energies
        attn_energies = Variable(torch.zeros(seq_len)) # B x 1 x S
        attn_energies = attn_energies.cuda() if USE_CUDA else attn_energies
        # Calculate energies for each encoder output
        for i in range(seq_len):
            attn_energies[i] = self.score(hidden, encoder_outputs[i])

        # Normalize energies to weights in range 0 to 1, resize to 1 x 1 x seq_len
        return F.softmax(attn_energies).unsqueeze(0).unsqueeze(0)

    def score(self, hidden, encoder_output):
        if self.method == 'dot':
            energy =torch.dot(hidden.view(-1), encoder_output.view(-1))
        elif self.method == 'general':
            energy = self.attn(encoder_output)
            energy = torch.dot(hidden.view(-1), energy.view(-1))
        elif self.method == 'concat':
            energy = self.attn(torch.cat((hidden, encoder_output), 1))
            energy = torch.dot(self.v.view(-1), energy.view(-1))
        return energy

class AttnDecoderRNN(nn.Module):
    def __init__(self, attn_model, hidden_size, output_size, n_layers=1, dropout_p=0.1):
        super(AttnDecoderRNN, self).__init__()

        # Keep parameters for reference
        self.attn_model = attn_model
        self.hidden_size = hidden_size
        self.output_size = output_size
        self.n_layers = n_layers
        self.dropout_p = dropout_p

        # Define layers
        self.embedding = nn.Embedding(output_size, hidden_size)
        self.gru = nn.GRU(hidden_size * 2, hidden_size, n_layers, dropout=dropout_p)
        self.out = nn.Linear(hidden_size * 2, output_size)

        self.embedding = self.embedding.cuda() if USE_CUDA else self.embedding
        self.gru = self.gru.cuda() if USE_CUDA else self.gru
        self.out = self.out.cuda() if USE_CUDA else self.out

        # Choose attention model
        if attn_model != 'none':
            self.attn = Attn(attn_model, hidden_size)
            self.attn = self.attn.cuda() if USE_CUDA else self.attn

    def forward(self, word_input, last_context, last_hidden, encoder_outputs):
        # Note: we run this one step at a time

        # Get the embedding of the current input word (last output word)
        word_embedded = self.embedding(word_input).view(1, 1, -1) # S=1 x B x N

        # Combine embedded input word and last context, run through RNN
        rnn_input = torch.cat((word_embedded, last_context.unsqueeze(0)), 2)
        rnn_output, hidden = self.gru(rnn_input, last_hidden)

        # Calculate attention from current RNN state and all encoder outputs; apply to encoder outputs
        attn_weights = self.attn(rnn_output.squeeze(0), encoder_outputs)
        context = attn_weights.bmm(encoder_outputs.transpose(0, 1)) # B x 1 x N

        # Final output layer (next word prediction) using the RNN hidden state and context vector
        rnn_output = rnn_output.squeeze(0) # S=1 x B x N -> B x N
        context = context.squeeze(1)       # B x S=1 x N -> B x N
        output = F.log_softmax(self.out(torch.cat((rnn_output, context), 1)))

        if USE_CUDA:
            return output.cuda(), context.cuda(), hidden.cuda(), attn_weights.cuda()
        else:
            return output, context, hidden, attn_weights

And testing the network:

encoder_test = EncoderRNN(10, 10, 2) # I, H , L
decoder_test = AttnDecoderRNN('general', 10, 10, 2) # A, H, O, L

encoder_hidden = encoder_test.init_hidden()
if USE_CUDA:
    word_inputs = Variable(torch.LongTensor([1, 2, 3]).cuda())
else:
    word_inputs = Variable(torch.LongTensor([1, 2, 3]))
encoder_outputs, encoder_hidden = encoder_test(word_inputs, encoder_hidden)
decoder_attns = torch.zeros(1, 3, 3)
decoder_hidden = encoder_hidden
decoder_context = Variable(torch.zeros(1, decoder_test.hidden_size))

decoder_output, decoder_context, decoder_hidden, decoder_attn = decoder_test(word_inputs[0], decoder_context, decoder_hidden, encoder_outputs)
print(decoder_output)
print(decoder_hidden)
print(decoder_attn)

The code works fine on CPU,

[out]:

EncoderRNN (
  (embedding): Embedding(10, 10)
  (gru): GRU(10, 10, num_layers=2)
)
AttnDecoderRNN (
  (embedding): Embedding(10, 10)
  (gru): GRU(20, 10, num_layers=2, dropout=0.1)
  (out): Linear (20 -> 10)
  (attn): Attn (
    (attn): Linear (10 -> 10)
  )
)
Variable containing:
-2.4378 -2.3556 -2.3391 -2.5070 -2.3439 -2.3415 -2.3976 -2.1832 -1.9976 -2.2213
[torch.FloatTensor of size 1x10]

Variable containing:
(0 ,.,.) =

Columns 0 to 8
  -0.2325  0.0775  0.5415  0.4876 -0.5771 -0.0687  0.1832 -0.5285  0.2508

Columns 9 to 9
  -0.1837

(1 ,.,.) =

Columns 0 to 8
  -0.1389 -0.2605 -0.0518  0.3405  0.0774  0.1815  0.0297 -0.1304 -0.1015

Columns 9 to 9
   0.2602
[torch.FloatTensor of size 2x1x10]

Variable containing:
(0 ,.,.) =
  0.3334  0.3291  0.3374
[torch.FloatTensor of size 1x1x3]

but when changing the flag to USE_GPU=True, it throws the error when initializing the decoder_test object, it throws a TypeError:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-76-b3c660013934> in <module>()
     12 decoder_context = Variable(torch.zeros(1, decoder_test.hidden_size))
     13
---> 14 decoder_output, decoder_context, decoder_hidden, decoder_attn = decoder_test(word_inputs[0], decoder_context, decoder_hidden, encoder_outputs)
     15 print(decoder_output)
     16 print(decoder_hidden)

~/.local/lib/python3.5/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    222         for hook in self._forward_pre_hooks.values():
    223             hook(self, input)
--> 224         result = self.forward(*input, **kwargs)
    225         for hook in self._forward_hooks.values():
    226             hook_result = hook(self, input, result)

<ipython-input-75-34ecfe9b3112> in forward(self, word_input, last_context, last_hidden, encoder_outputs)
     32
     33         # Combine embedded input word and last context, run through RNN
---> 34         rnn_input = torch.cat((word_embedded, last_context.unsqueeze(0)), 2)
     35         rnn_output, hidden = self.gru(rnn_input, last_hidden)
     36

~/.local/lib/python3.5/site-packages/torch/autograd/variable.py in cat(iterable, dim)
    895         @staticmethod
    896         def cat(iterable, dim=0):
--> 897             return Concat.apply(dim, *iterable)
    898
    899         @staticmethod

~/.local/lib/python3.5/site-packages/torch/autograd/_functions/tensor.py in forward(ctx, dim, *inputs)
    315         ctx.dim = dim
    316         ctx.input_sizes = [i.size(dim) for i in inputs]
--> 317         return torch.cat(inputs, dim)
    318
    319     @staticmethod

TypeError: cat received an invalid combination of arguments - got (tuple, int), but expected one of:
 * (sequence[torch.cuda.FloatTensor] seq)
 * (sequence[torch.cuda.FloatTensor] seq, int dim)
      didn't match because some of the arguments have invalid types: (tuple, int)

The question is why are that types not matching in CUDA but it works on CPU and how to resolve this?

Does PyTorch have a global flag to just change all types to CUDA types and not mess around with CPU/GPU types?


title: pytorch RuntimeError: CUDA error: device-side assert triggered

body:
I've a notebook on google colab that fails with following error

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
/usr/local/lib/python3.6/dist-packages/fastai/basic_train.py in fit(epochs, model, loss_func, opt, data, callbacks, metrics)
     93         exception = e
---> 94         raise e
     95     finally: cb_handler.on_train_end(exception)

/usr/local/lib/python3.6/dist-packages/fastai/basic_train.py in fit(epochs, model, loss_func, opt, data, callbacks, metrics)
     83                 xb, yb = cb_handler.on_batch_begin(xb, yb)
---> 84                 loss = loss_batch(model, xb, yb, loss_func, opt, cb_handler)
     85                 if cb_handler.on_batch_end(loss): break

/usr/local/lib/python3.6/dist-packages/fastai/basic_train.py in loss_batch(model, xb, yb, loss_func, opt, cb_handler)
     24     if opt is not None:
---> 25         loss = cb_handler.on_backward_begin(loss)
     26         loss.backward()

/usr/local/lib/python3.6/dist-packages/fastai/callback.py in on_backward_begin(self, loss)
    223         for cb in self.callbacks:
--> 224             a = cb.on_backward_begin(**self.state_dict)
    225             if a is not None: self.state_dict['last_loss'] = a

/usr/local/lib/python3.6/dist-packages/fastai/basic_train.py in on_backward_begin(self, smooth_loss, **kwargs)
    266         if self.pbar is not None and hasattr(self.pbar,'child'):
--> 267             self.pbar.child.comment = f'{smooth_loss:.4f}'
    268

/usr/local/lib/python3.6/dist-packages/torch/tensor.py in __format__(self, format_spec)
    377         if self.dim() == 0:
--> 378             return self.item().__format__(format_spec)
    379         return object.__format__(self, format_spec)

RuntimeError: CUDA error: device-side assert triggered

During handling of the above exception, another exception occurred:

RuntimeError                              Traceback (most recent call last)
<ipython-input-33-dd390b1c8108> in <module>()
----> 1 lr_find(learn)
      2 learn.recorder.plot()

/usr/local/lib/python3.6/dist-packages/fastai/train.py in lr_find(learn, start_lr, end_lr, num_it, stop_div, **kwargs)
     26     cb = LRFinder(learn, start_lr, end_lr, num_it, stop_div)
     27     a = int(np.ceil(num_it/len(learn.data.train_dl)))
---> 28     learn.fit(a, start_lr, callbacks=[cb], **kwargs)
     29
     30 def to_fp16(learn:Learner, loss_scale:float=512., flat_master:bool=False)->Learner:

/usr/local/lib/python3.6/dist-packages/fastai/basic_train.py in fit(self, epochs, lr, wd, callbacks)
    160         callbacks = [cb(self) for cb in self.callback_fns] + listify(callbacks)
    161         fit(epochs, self.model, self.loss_func, opt=self.opt, data=self.data, metrics=self.metrics,
--> 162             callbacks=self.callbacks+callbacks)
    163
    164     def create_opt(self, lr:Floats, wd:Floats=0.)->None:

/usr/local/lib/python3.6/dist-packages/fastai/basic_train.py in fit(epochs, model, loss_func, opt, data, callbacks, metrics)
     93         exception = e
     94         raise e
---> 95     finally: cb_handler.on_train_end(exception)
     96
     97 loss_func_name2activ = {'cross_entropy_loss': partial(F.softmax, dim=1), 'nll_loss': torch.exp, 'poisson_nll_loss': torch.exp,

/usr/local/lib/python3.6/dist-packages/fastai/callback.py in on_train_end(self, exception)
    254     def on_train_end(self, exception:Union[bool,Exception])->None:
    255         "Handle end of training, `exception` is an `Exception` or False if no exceptions during training."
--> 256         self('train_end', exception=exception)
    257
    258 class AverageMetric(Callback):

/usr/local/lib/python3.6/dist-packages/fastai/callback.py in __call__(self, cb_name, call_mets, **kwargs)
    185         "Call through to all of the `CallbakHandler` functions."
    186         if call_mets: [getattr(met, f'on_{cb_name}')(**self.state_dict, **kwargs) for met in self.metrics]
--> 187         return [getattr(cb, f'on_{cb_name}')(**self.state_dict, **kwargs) for cb in self.callbacks]
    188
    189     def on_train_begin(self, epochs:int, pbar:PBar, metrics:MetricFuncList)->None:

/usr/local/lib/python3.6/dist-packages/fastai/callback.py in <listcomp>(.0)
    185         "Call through to all of the `CallbakHandler` functions."
    186         if call_mets: [getattr(met, f'on_{cb_name}')(**self.state_dict, **kwargs) for met in self.metrics]
--> 187         return [getattr(cb, f'on_{cb_name}')(**self.state_dict, **kwargs) for cb in self.callbacks]
    188
    189     def on_train_begin(self, epochs:int, pbar:PBar, metrics:MetricFuncList)->None:

/usr/local/lib/python3.6/dist-packages/fastai/callbacks/lr_finder.py in on_train_end(self, **kwargs)
     45         # restore the valid_dl we turned of on `__init__`
     46         self.data.valid_dl = self.valid_dl
---> 47         self.learn.load('tmp')
     48         if hasattr(self.learn.model, 'reset'): self.learn.model.reset()
     49         print('LR Finder complete, type {learner_name}.recorder.plot() to see the graph.')

/usr/local/lib/python3.6/dist-packages/fastai/basic_train.py in load(self, name, device)
    202         "Load model `name` from `self.model_dir` using `device`, defaulting to `self.data.device`."
    203         if device is None: device = self.data.device
--> 204         self.model.load_state_dict(torch.load(self.path/self.model_dir/f'{name}.pth', map_location=device))
    205         return self
    206

/usr/local/lib/python3.6/dist-packages/torch/serialization.py in load(f, map_location, pickle_module)
    356         f = open(f, 'rb')
    357     try:
--> 358         return _load(f, map_location, pickle_module)
    359     finally:
    360         if new_fd:

/usr/local/lib/python3.6/dist-packages/torch/serialization.py in _load(f, map_location, pickle_module)
    527     unpickler = pickle_module.Unpickler(f)
    528     unpickler.persistent_load = persistent_load
--> 529     result = unpickler.load()
    530
    531     deserialized_storage_keys = pickle_module.load(f)

/usr/local/lib/python3.6/dist-packages/torch/serialization.py in persistent_load(saved_id)
    493             if root_key not in deserialized_objects:
    494                 deserialized_objects[root_key] = restore_location(
--> 495                     data_type(size), location)
    496             storage = deserialized_objects[root_key]
    497             if view_metadata is not None:

/usr/local/lib/python3.6/dist-packages/torch/serialization.py in restore_location(storage, location)
    376     elif isinstance(map_location, torch.device):
    377         def restore_location(storage, location):
--> 378             return default_restore_location(storage, str(map_location))
    379     else:
    380         def restore_location(storage, location):

/usr/local/lib/python3.6/dist-packages/torch/serialization.py in default_restore_location(storage, location)
    102 def default_restore_location(storage, location):
    103     for _, _, fn in _package_registry:
--> 104         result = fn(storage, location)
    105         if result is not None:
    106             return result

/usr/local/lib/python3.6/dist-packages/torch/serialization.py in _cuda_deserialize(obj, location)
     84                                'to an existing device.'.format(
     85                                    device, torch.cuda.device_count()))
---> 86         return obj.cuda(device)
     87
     88

/usr/local/lib/python3.6/dist-packages/torch/_utils.py in _cuda(self, device, non_blocking, **kwargs)
     74         else:
     75             new_type = getattr(torch.cuda, self.__class__.__name__)
---> 76             return new_type(self.size()).copy_(self, non_blocking)
     77
     78

RuntimeError: cuda runtime error (59) : device-side assert triggered at /pytorch/aten/src/THC/generic/THCTensorCopy.cpp:20

There is no information about the real cause, I tried to get the stack trace by forcing cuda to run on one gpu (as suggested here) using a cell like this

!export CUDA_LAUNCH_BLOCKING=1

But this does not seem to work, still having the same error with.

Is there another way that works with Google Colab?


title: What is the difference between .pt, .pth and .pwf extentions in PyTorch?

body:
I have seen in some code examples, that people use .pwf as model file saving format. But in PyTorch documentation .pt and .pth are recommended. I used .pwf and worked fine for small 1->16->16 convolutional network.

My question is what is the difference between these formats? Why is .pwf extension not even recommended in PyTorch documentation and why do people still use it?


title: Cannot load torchvision despite it being installed

body:
I have installed pytorch and torchvision using:

conda install pytorch-cpu -c pytorch
pip install torchvision

when I try to run the following in spyder:

import torch
import torchvision
import torchvision.transforms as transforms

I get:

Traceback (most recent call last):

  File "<ipython-input-2-0bf25e9dac67>", line 2, in <module>
    import torchvision

  File "C:\Users\lkoefoed\AppData\Local\Continuum\anaconda3\lib\site-package\torchvision\__init__.py", line 2, in <module>
from torchvision import datasets

  File "C:\Users\lkoefoed\AppData\Local\Continuum\anaconda3\lib\site-packages\torchvision\datasets\__init__.py", line 1, in <module>
from .lsun import LSUN, LSUNClass

  File "C:\Users\lkoefoed\AppData\Local\Continuum\anaconda3\lib\site-packages\torchvision\datasets\lsun.py", line 2, in <module>
from PIL import Image

  File "C:\Users\lkoefoed\AppData\Local\Continuum\anaconda3\lib\site-packages\PIL\Image.py", line 56, in <module>
from . import _imaging as core

ImportError: DLL load failed: The specified module could not be found.


title: error while import pytorch module. (The specified module could not be found.)

body:
I just newly install python 3.8 via anaconda installer and install pytorch using command
conda install pytorch torchvision cpuonly -c pytorch

when i try to import torch, I got this error message.
OSError: [WinError 126] The specified module could not be found. Error loading "C:\Users\chunc\anaconda3\lib\site-packages\torch\lib\asmjit.dll" or one of its dependencies.

I can see dll files are still in the directory.
I ran Dependency Walker and it gave me this result.
I am with this problem for a day.
What should i do if i want to use PyTorch module?


title: What does "RuntimeError: CUDA error: device-side assert triggered" in PyTorch mean?

body:
I have seen a lot of specific posts to particular case-specific problems, but no fundamental motivating explanation. What does this error:

RuntimeError: CUDA error: device-side assert triggered

mean? Specifically, what is the assert that is being triggered, why is the assert there, and how do we work backwards to debug the problem?

As-is, this error message is near useless in diagnosing any problem because of the generality that it seems to say "some code somewhere that touches the GPU" has a problem. The documentation of Cuda also does not seem helpful in this regard, though I could be wrong.
https://docs.nvidia.com/cuda/cuda-gdb/index.html


title: How does Pytorch Dataloader handle variable size data?

body:
I have a dataset that looks like below. That is the first item is the user id followed by the set of items which is clicked by the user.

0   24104   27359   6684
0   24104   27359
1   16742   31529   31485
1   16742   31529
2   6579    19316   13091   7181    6579    19316   13091
2   6579    19316   13091   7181    6579    19316
2   6579    19316   13091   7181    6579    19316   13091   6579
2   6579    19316   13091   7181    6579
4   19577   21608
4   19577   21608
4   19577   21608   18373
5   3541    9529
5   3541    9529
6   6832    19218   14144
6   6832    19218
7   9751    23424   25067   12606   26245   23083   12606

I define a custom dataset to handle my click log data.

import torch.utils.data as data
class ClickLogDataset(data.Dataset):
    def __init__(self, data_path):
        self.data_path = data_path
        self.uids = []
        self.streams = []

        with open(self.data_path, 'r') as fdata:
            for row in fdata:
                row = row.strip('\n').split('\t')
                self.uids.append(int(row[0]))
                self.streams.append(list(map(int, row[1:])))

    def __len__(self):
        return len(self.uids)

    def __getitem__(self, idx):
        uid, stream = self.uids[idx], self.streams[idx]
        return uid, stream

Then I use a DataLoader to retrieve mini batches from the data for training.

from torch.utils.data.dataloader import DataLoader
clicklog_dataset = ClickLogDataset(data_path)
clicklog_data_loader = DataLoader(dataset=clicklog_dataset, batch_size=16)

for uid_batch, stream_batch in stream_data_loader:
    print(uid_batch)
    print(stream_batch)

The code above returns differently from what I expected, I want stream_batch to be a 2D tensor of type integer of length 16. However, what I get is a list of 1D tensor of length 16, and the list has only one element, like below. Why is that ?

#stream_batch
[tensor([24104, 24104, 16742, 16742,  6579,  6579,  6579,  6579, 19577, 19577,
        19577,  3541,  3541,  6832,  6832,  9751])]


title: Pytorch to Keras code equivalence

body:
Given a below code in PyTorch what would be the Keras equivalent?

class Network(nn.Module):

    def __init__(self, state_size, action_size):
        super(Network, self).__init__()

        # Inputs = 5, Outputs = 3, Hidden = 30
        self.fc1 = nn.Linear(5, 30)
        self.fc2 = nn.Linear(30, 3)

    def forward(self, state):
        x = F.relu(self.fc1(state))
        outputs = self.fc2(x)
        return outputs

Is it this?

model = Sequential()
model.add(Dense(units=30, input_dim=5, activation='relu'))
model.add(Dense(units=30, activation='relu'))
model.add(Dense(units=3, activation='linear'))

or is it this?

model = Sequential()
model.add(Dense(units=30, input_dim=5, activation='linear'))
model.add(Dense(units=30, activation='relu'))
model.add(Dense(units=3, activation='linear'))

or is it?

model = Sequential()
model.add(Dense(units=30, input_dim=5, activation='relu'))
model.add(Dense(units=30, activation='linear'))
model.add(Dense(units=3, activation='linear'))

Thanks


title: AttributeError: 'tuple' object has no attribute 'dim', when feeding input to Pytorch LSTM network

body:
I am trying to run the following code:

import matplotlib.pylab as plt
import numpy as np
import torch
import torch.nn as nn

class LSTM(nn.Module):
    def __init__(self, input_shape, n_actions):
        super(LSTM, self).__init__()

        self.lstm = nn.LSTM(input_shape, 12)
        self.hidden2tag = nn.Linear(12, n_actions)

    def forward(self, x):
        out = self.lstm(x)
        out = self.hidden2tag(out)
        return out

state = [(1,2,3,4,5),(2,3,4,5,6),(3,4,5,6,7),(4,5,6,7,8),(5,6,7,8,9),(6,7,8,9,0)]

device = torch.device("cuda")
net = LSTM(5, 3).to(device)

state_v = torch.FloatTensor(state).to(device)

q_vals_v = net(state_v.view(1, state_v.shape[0], state_v.shape[1]))
_, action = int(torch.max(q_vals_v, dim=1).item())

And that returns this error:

Traceback (most recent call last):
  File "/home/dikkerj/Documents/PycharmProjects/LSTMReactor/QuestionStackoverflow.py", line 26, in <module>
    q_vals_v = net(state_v.view(1, state_v.shape[0], state_v.shape[1]))
  File "/home/dikkerj/.local/lib/python3.5/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/dikkerj/Documents/PycharmProjects/LSTMReactor/QuestionStackoverflow.py", line 15, in forward
    out = self.hidden2tag(out)
  File "/home/dikkerj/.local/lib/python3.5/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/dikkerj/.local/lib/python3.5/site-packages/torch/nn/modules/linear.py", line 55, in forward
    return F.linear(input, self.weight, self.bias)
  File "/home/dikkerj/.local/lib/python3.5/site-packages/torch/nn/functional.py", line 1022, in linear
    if input.dim() == 2 and bias is not None:
AttributeError: 'tuple' object has no attribute 'dim'

Anyone knows how to fix this? (to get rid of the tensor being a tuple so that it can be fed into the LSTM network)


title: Installing PyTorch via Conda

body:
Objective: Create a conda environment with pytorch and torchvision. Anaconda Navigator 1.8.3, python 3.6, MacOS 10.13.4.

What I've tried:

In Navigator, created a new environment. Tried to install pytorch and torchvision but could not because the UI search for packages does not find any packages available matching pytorch, torch, torchvision, or similar strings.
conda install pytorch torchvision -c pytorch
conda update --all

pytorch 0.3.1, torch 0.3.1, and torchvision 0.2.0 now appear as installed in the root environment. However, the root environment is no longer cloneable; the clone button is gray/disabled (it used be enabled/cloneable). I could use the root environment as a fallback but the main point of conda is to be able to create separate and disposable environments. What am I missing?

UPDATE -----------------

Running conda install -c pytorch pytorch yields:
# All requested packages already installed. But if I activate the pytorch environment and list the packages therein, there is no package containing the word "torch". If I then do conda search pytorch I get PackagesNotFoundError: The following packages are not available from current channels: - pytorch. If I activate the base environment and then do conda list then pytorch is in the package list for base. So how does one create a separate environment containing pytorch?


title: BERT tokenizer & model download

body:
I`m beginner.. I'm working with Bert. However, due to the security of the company network, the following code does not receive the bert model directly.

tokenizer = BertTokenizer.from_pretrained('bert-base-multilingual-cased', do_lower_case=False)
model = BertForSequenceClassification.from_pretrained("bert-base-multilingual-cased", num_labels=2)

So I think I have to download these files and enter the location manually.
But I'm new to this, and I'm wondering if it's simple to download a format like .py from github and put it in a location.

I'm currently using the bert model implemented by hugging face's pytorch, and the address of the source file I found is:

https://github.com/huggingface/transformers

Please let me know if the method I thought is correct, and if so, what file to get.

Thanks in advance for the comment.


title: How does pytorch broadcasting work?

body:
torch.add(torch.ones(4,1), torch.randn(4))

produces a Tensor with size: torch.Size([4,4]).

Can someone provide a logic behind this?


title: PyTorch - multiplying tensor with scalar results in zero vector

body:
I have no idea why the result is all 0 with tensor. Anything wrong here?

>>> import torch
>>> import numpy as np
>>> import math

>>> torch.__version__
'0.4.1'
>>> np.__version__
'1.15.4'

>>> torch.arange(0, 10, 2) *-(math.log(10000.0) / 10)
tensor([0, 0, 0, 0, 0])
>>> np.arange(0, 10, 2) *-(math.log(10000.0) / 10)
array([-0.        , -1.84206807, -3.68413615, -5.52620422, -7.3682723 ])

>>> torch.arange(0, 10, 2)
tensor([0, 2, 4, 6, 8])
>>> np.arange(0, 10, 2)
array([0, 2, 4, 6, 8])


title: What's the purpose of torch.autograd.Variable?

body:
I load features and labels from my training dataset. Both of them are originally numpy arrays, but I change them to the torch tensor using torch.from _numpy(features.copy()) and torch.tensor(labels.astype(np.bool)).

And I notice that torch.autograd.Variable is something like placeholder in tensorflow.

When I train my network, first I tried

features = features.cuda()
labels = labels.cuda()

outputs = Config.MODEL(features)
loss = Config.LOSS(outputs, labels)

Then I tried

features = features.cuda()
labels = labels.cuda()

input_var = Variable(features)
target_var = Variable(labels)
outputs = Config.MODEL(input_var)
loss = Config.LOSS(outputs, target_var)

Both blocks succeed in activating training, but I worried that there might be trivial difference.


title: pytorch skip connection in a sequential model

body:
I am trying to wrap my head around skip connections in a sequential model. With the functional API I would be doing something as easy as (quick example, maybe not be 100% syntactically correct but should get the idea):

x1 = self.conv1(inp)
x = self.conv2(x)
x = self.conv3(x)
x = self.conv4(x)

x = self.deconv4(x)
x = self.deconv3(x)
x = self.deconv2(x)
x = torch.cat((x, x1), 1))
x = self.deconv1(x)

I am now using a sequential model and trying to do something similar, create a skip connection that brings the activations of the first conv layer all the way to the last convTranspose. I have taken a look at the U-net architecture implemented here and it's a bit confusing, it does something like this:

upconv = nn.ConvTranspose2d(inner_nc * 2, outer_nc,
                                    kernel_size=4, stride=2,
                                    padding=1, bias=use_bias)
down = [downrelu, downconv, downnorm]
up = [uprelu, upconv, upnorm]

if use_dropout:
    model = down + [submodule] + up + [nn.Dropout(0.5)]
else:
    model = down + [submodule] + up

Isn't this just adding layers to the sequential model well, sequentially? There is the down conv which is followed by submodule (which recursively adds inner layers) and then concatenated to up which is the upconv layer. I am probably missing something important on how the Sequential API works, but how does the code snipped from U-NET actually implements the skip?


title: ValueError: Floating point image RGB values must be in the 0..1 range. while using matplotlib

body:
I want to visualize weights of the layer of a neural network. I'm using pytorch.

import torch
import torchvision.models as models
from matplotlib import pyplot as plt

def plot_kernels(tensor, num_cols=6):
    if not tensor.ndim==4:
        raise Exception("assumes a 4D tensor")
    if not tensor.shape[-1]==3:
        raise Exception("last dim needs to be 3 to plot")
    num_kernels = tensor.shape[0]
    num_rows = 1+ num_kernels // num_cols
    fig = plt.figure(figsize=(num_cols,num_rows))
    for i in range(tensor.shape[0]):
        ax1 = fig.add_subplot(num_rows,num_cols,i+1)
        ax1.imshow(tensor[i])
        ax1.axis('off')
        ax1.set_xticklabels([])
        ax1.set_yticklabels([])

    plt.subplots_adjust(wspace=0.1, hspace=0.1)
    plt.show()
vgg = models.vgg16(pretrained=True)
mm = vgg.double()
filters = mm.modules
body_model = [i for i in mm.children()][0]
layer1 = body_model[0]
tensor = layer1.weight.data.numpy()
plot_kernels(tensor)

The above gives this error ValueError: Floating point image RGB values must be in the 0..1 range.

My question is should I normalize and take absolute value of the weights to overcome this error or is there anyother way ?
If I normalize and use absolute value I think the meaning of the graphs change.

[[[[ 0.02240197 -1.22057354 -0.55051649]
   [-0.50310904  0.00891289  0.15427093]
   [ 0.42360783 -0.23392732 -0.56789106]]

  [[ 1.12248898  0.99013627  1.6526649 ]
   [ 1.09936976  2.39608836  1.83921957]
   [ 1.64557672  1.4093554   0.76332706]]

  [[ 0.26969245 -1.2997849  -0.64577204]
   [-1.88377869 -2.0100112  -1.43068039]
   [-0.44531786 -1.67845118 -1.33723605]]]

 [[[ 0.71286005  1.45265901  0.64986968]
   [ 0.75984162  1.8061738   1.06934202]
   [-0.08650422  0.83452386 -0.04468433]]

  [[-1.36591709 -2.01630116 -1.54488969]
   [-1.46221244 -2.5365622  -1.91758668]
   [-0.88827479 -1.59151018 -1.47308767]]

  [[ 0.93600738  0.98174071  1.12213969]
   [ 1.03908169  0.83749604  1.09565806]
   [ 0.71188802  0.85773659  0.86840987]]]

 [[[-0.48592842  0.2971966   1.3365227 ]
   [ 0.47920835 -0.18186836  0.59673625]
   [-0.81358945  1.23862112  0.13635623]]

  [[-0.75361633 -1.074965    0.70477796]
   [ 1.24439156 -1.53563368 -1.03012812]
   [ 0.97597247  0.83084011 -1.81764793]]

  [[-0.80762428 -0.62829626  1.37428832]
   [ 1.01448071 -0.81775147 -0.41943246]
   [ 1.02848887  1.39178836 -1.36779451]]]

 ...,
 [[[ 1.28134537 -0.00482408  0.71610934]
   [ 0.95264435 -0.09291686 -0.28001019]
   [ 1.34494913  0.64477581  0.96984017]]

  [[-0.34442815 -1.40002513  1.66856039]
   [-2.21281362 -3.24513769 -1.17751861]
   [-0.93520379 -1.99811196  0.72937071]]

  [[ 0.63388056 -0.17022935  2.06905985]
   [-0.7285465  -1.24722099  0.30488953]
   [ 0.24900314 -0.19559766  1.45432627]]]

 [[[-0.80684513  2.1764245  -0.73765725]
   [-1.35886598  1.71875226 -1.73327696]
   [-0.75233924  2.14700699 -0.71064663]]

  [[-0.79627383  2.21598244 -0.57396138]
   [-1.81044972  1.88310981 -1.63758397]
   [-0.6589964   2.013237   -0.48532376]]

  [[-0.3710472   1.4949851  -0.30245575]
   [-1.25448656  1.20453358 -1.29454732]
   [-0.56755757  1.30994892 -0.39370224]]]

 [[[-0.67361742 -3.69201088 -1.23768616]
   [ 3.12674141  1.70414758 -1.76272404]
   [-0.22565465  1.66484773  1.38172317]]

  [[ 0.28095332 -2.03035069  0.69989491]
   [ 1.97936332  1.76992691 -1.09842575]
   [-2.22433758  0.52577412  0.18292744]]

  [[ 0.48471382 -1.1984663   1.57565165]
   [ 1.09911084  1.31910467 -0.51982772]
   [-2.76202297 -0.47073677  0.03936549]]]]


title: Why does PyTorch not find my NVDIA drivers for CUDA support?

body:
I've added an GeForce GTX 1080 Ti into my machine (Running Ubuntu 18.04 and Anaconda with Python 3.7) to utilize the GPU when using PyTorch. Both cards a correctly identified:

$ lspci | grep VGA
03:00.0 VGA compatible controller: NVIDIA Corporation GF119 [NVS 310] (reva1)
04:00.0 VGA compatible controller: NVIDIA Corporation GP102 [GeForce GTX 1080 Ti] (rev a1)

The NVS 310 handles my 2-monitor setup, I only want to utilize the 1080 for PyTorch. I also installed the latest NVIDIA drivers that are currently in the repository and that seems to be fine:

$ nvidia-smi
Sat Jan 19 12:42:18 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 390.87                 Driver Version: 390.87                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  NVS 310             Off  | 00000000:03:00.0 N/A |                  N/A |
| 30%   60C    P0    N/A /  N/A |    461MiB /   963MiB |     N/A      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX 108...  Off  | 00000000:04:00.0 Off |                  N/A |
|  0%   41C    P8    10W / 250W |      2MiB / 11178MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0                    Not Supported                                       |
+-----------------------------------------------------------------------------+

Driver version 390.xx allows to run CUDA 9.1 (9.1.85) according the the NVIDIA docs. Since this is also the version in the Ubuntu repositories, I simple installed the CUDA Toolkit with:

$ sudo apt-get-installed nvidia-cuda-toolkit

And again, this seems be alright:

$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2017 NVIDIA Corporation
Built on Fri_Nov__3_21:07:56_CDT_2017
Cuda compilation tools, release 9.1, V9.1.85

and

$ apt-cache policy nvidia-cuda-toolkit
nvidia-cuda-toolkit:
  Installed: 9.1.85-3ubuntu1
  Candidate: 9.1.85-3ubuntu1
  Version table:
 *** 9.1.85-3ubuntu1 500
        500 http://sg.archive.ubuntu.com/ubuntu bionic/multiverse amd64 Packages
        100 /var/lib/dpkg/status

Lastly, I've installed PyTorch from scratch with conda

conda install pytorch torchvision -c pytorch

Also error as far as I can tell:

$ conda list
...
pytorch                   1.0.0           py3.7_cuda9.0.176_cudnn7.4.1_1    pytorch
...

However, PyTorch doesn't seem to find CUDA:

$ python -c 'import torch; print(torch.cuda.is_available())'
False

In more detail, if I force PyTorch to convert a tensor x to CUDA with x.cuda() I get the error:

Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from 82 http://...

What am I'm missing here? I'm new to this, but I think I've checked the Web already quite a bit to find any caveats like NVIDIA driver and CUDA toolkit versions?

EDIT: Some more outputs from PyTorch:

print(torch.cuda.device_count())   # --> 0
print(torch.cuda.is_available())   # --> False
print(torch.version.cuda)          # --> 9.0.176


title: Concatenating two tensors with different dimensions in Pytorch

body:
Is it possible to concatenate two tensors with different dimensions without using for loop.

e.g. Tensor 1 has dimensions (15, 200, 2048) and Tensor 2 has dimensions (1, 200, 2048). Is it possible to concatenate 2nd tensor with 1st tensor along all the 15 indices of 1st dimension in 1st Tensor (Broadcast 2nd tensor along 1st dimension of Tensor 1 while concatenating along 3rd dimension of 1st tensor)? The resulting tensor should have dimensions (15, 200, 4096).

Is it possible to accomplish this without for loop ?


title: Evaluating pytorch models: `with torch.no_grad` vs `model.eval()`

body: When I want to evaluate the performance of my model on the validation set, is it preferred to use with torch.no_grad: or model.eval()?


title: Pytorch model accuracy test

body:
I'm using Pytorch to classify a series of images.
The NN is defined as follows:

model = models.vgg16(pretrained=True)
model.cuda()
for param in model.parameters(): param.requires_grad = False

classifier = nn.Sequential(OrderedDict([
                           ('fc1', nn.Linear(25088, 4096)),
                           ('relu', nn.ReLU()),
                           ('fc2', nn.Linear(4096, 102)),
                           ('output', nn.LogSoftmax(dim=1))
                           ]))

model.classifier = classifier

The criterions and optimizers are as follows:

criterion = nn.NLLLoss()
optimizer = optim.Adam(model.classifier.parameters(), lr=0.001)

My validation function is as follows:

def validation(model, testloader, criterion):
    test_loss = 0
    accuracy = 0
    for images, labels in testloader:

        images.resize_(images.shape[0], 784)

        output = model.forward(images)
        test_loss += criterion(output, labels).item()

        ps = torch.exp(output)
        equality = (labels.data == ps.max(dim=1)[1])
        accuracy += equality.type(torch.FloatTensor).mean()

    return test_loss, accuracy

This is the piece of code that is throwing the following error:

  RuntimeError: input has less dimensions than expected

epochs = 3
print_every = 40
steps = 0
running_loss = 0
testloader = dataloaders['test']

# change to cuda
model.to('cuda')

for e in range(epochs):
    running_loss = 0
    for ii, (inputs, labels) in enumerate(dataloaders['train']):
        steps += 1

        inputs, labels = inputs.to('cuda'), labels.to('cuda')

        optimizer.zero_grad()

        # Forward and backward passes
        outputs = model.forward(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        running_loss += loss.item()

        if steps % print_every == 0:
            model.eval()
            with torch.no_grad():
                test_loss, accuracy = validation(model, testloader, criterion)

            print("Epoch: {}/{}.. ".format(e+1, epochs),
                  "Training Loss: {:.3f}.. ".format(running_loss/print_every),
                  "Test Loss: {:.3f}.. ".format(test_loss/len(testloader)),
                  "Test Accuracy: {:.3f}".format(accuracy/len(testloader)))

            running_loss = 0

Any help?


I have downloaded the ImageNet dataset via this pytorch api by setting download=True. But I cannot iterate through the dataloader.

The error says "'DataLoader' object does not support indexing"

trainset = torch.utils.data.DataLoader(
    datasets.ImageNet('/media/farshid/DataStore/temp/Imagenet/', split='train',
                      download=False))
trainloader = torch.utils.data.DataLoader(trainset, batch_size=1, shuffle=False, num_workers=1)

I tried a simple approach I just tried to run the following,

trainloader[0]

In the root directory, the pattern is

root/
    train/
          n01440764/
          n01443537/
                   n01443537_2.jpg

The docs in the official website doesnt say anything else. https://pytorch.org/docs/stable/torchvision/datasets.html#imagenet

What am I doing wrong ?


title: PyTorch Error The size of tensor a (128) must match the size of tensor b (9) at non-singleton dimension 0

body:
I am running an CNN model of HAR using PyTorch 1.01 in Anaconda with GPU
While doing the iteration it's giving me the error
The size of tensor a (128) must match the size of tensor b (9) at non-singleton dimension 0.
I believe it's the datamodel while enumerate the train_model giving error. Anyone faced similar issues in PyTorch ? Need little support as new to PyTorch.

I have tried all the datamodel tricks found in google.

'''

    def train(model, optimizer, train_loader, test_loader):
    n_batch = len(train_loader.dataset) // BATCH_SIZE
    criterion = nn.CrossEntropyLoss()

     for e in range(N_EPOCH):
       model.train()
       correct, total_loss = 0, 0
        total = 0
         for index, (sample, target) in enumerate(train_loader):
        sample, target = sample.to(DEVICE).float(), target.to(DEVICE).long()
        sample = sample.view(-1, 9, 1, 128)
        output = model(sample)
        loss = criterion(output, target)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        total_loss += loss.item()
        _, predicted = torch.max(output.data, 1)
        total += target.size(0)
        correct += (predicted == target).sum()

        if index % 20 == 0:
            tqdm.tqdm.write('Epoch: [{}/{}], Batch: [{}/{}], loss:{:.4f}'.format(e + 1, N_EPOCH, index + 1, n_batch,
                                                                                loss.item()))
    acc_train = float(correct) * 100.0 / (BATCH_SIZE * n_batch)
    tqdm.tqdm.write(
        'Epoch: [{}/{}], loss: {:.4f}, train acc: {:.2f}%'.format(e + 1, N_EPOCH, total_loss * 1.0 / n_batch,
                                                                  acc_train))

    # Testing
    model.train(False)
    with torch.no_grad():
        correct, total = 0, 0
        for sample, target in test_loader:
            sample, target = sample.to(DEVICE).float(), target.to(DEVICE).long()
            sample = sample.view(-1, 9, 1, 128)
            output = model(sample)
            _, predicted = torch.max(output.data, 1)
            total += target.size(0)
            correct += (predicted == target).sum()
    acc_test = float(correct) * 100 / total
    tqdm.tqdm.write('Epoch: [{}/{}], test acc: {:.2f}%'.format(e + 1, N_EPOCH, float(correct) * 100 / total))
    result.append([acc_train, acc_test])
    result_np = np.array(result, dtype=float)
    np.savetxt('result.csv', result_np, fmt='%.2f', delimiter=',')

 Error ----------------------------
 (7352, 1152)
 (7352, 128, 9)
 (2947, 1152)
  (2947, 128, 9)
   -----------------------------------------------------------------
    ----------
 RuntimeError                              Traceback (most recent
call last)
 <ipython-input-1-64c1adae4ee0> in <module>
 86     model = net.Network().to(DEVICE)
 87     optimizer = optim.SGD(params=model.parameters(),
 lr=LEARNING_RATE, momentum=0.9)
---> 88     train(model, optimizer, train_loader, test_loader)
 89     result = np.array(result, dtype=float)
 90     np.savetxt('result.csv', result, fmt='%.2f', delimiter=',')

 <ipython-input-1-64c1adae4ee0> in train(model, optimizer,
 train_loader, test_loader)
 29         correct, total_loss = 0, 0
 30         total = 0
 ---> 31         for index, (sample, target) in
 enumerate(train_loader):
 32             sample, target = sample.to(DEVICE).float(),
 target.to(DEVICE).long()
 33             print('Sample',sample)

 ~/anaconda3/envs/rnn_lstm_har_pytorch/lib/python3.6/site-
 packages/torch/utils/data/dataloader.py in __next__(self)
613         if self.num_workers == 0:  # same-process loading
614             indices = next(self.sample_iter)  # may raise
StopIteration
--> 615             batch = self.collate_fn([self.dataset[i] for i
in indices])
616             if self.pin_memory:
617                 batch = pin_memory_batch(batch)

  ~/anaconda3/envs/rnn_lstm_har_pytorch/lib/python3.6/site-
 packages/torch/utils/data/dataloader.py in <listcomp>(.0)
  613         if self.num_workers == 0:  # same-process loading
  614             indices = next(self.sample_iter)  # may raise
 StopIteration
 --> 615             batch = self.collate_fn([self.dataset[i] for i
 in
  indices])
  616             if self.pin_memory:
  617                 batch = pin_memory_batch(batch)

 ~/anaconda3/envs/rnn_lstm_har_pytorch/data_preprocess.py in
 __getitem__(self, index)
  97     def __getitem__(self, index):
  98         sample, target = self.samples[index],
  self.labels[index]
  ---> 99         return self.T(sample), target
  100
  101     def __len__(self):

 ~/anaconda3/envs/rnn_lstm_har_pytorch/lib/python3.6/site-
   packages/torchvision/transforms/transforms.py in __call__(self,
   img)
   58     def __call__(self, img):
   59         for t in self.transforms:
   ---> 60             img = t(img)
   61         return img
   62

   ~/anaconda3/envs/rnn_lstm_har_pytorch/lib/python3.6/site-
   packages/torchvision/transforms/transforms.py in __call__(self,
  tensor)
   161             Tensor: Normalized Tensor image.
   162         """
  --> 163         return F.normalize(tensor, self.mean, self.std,
  self.inplace)
   164
   165     def __repr__(self):

  ~/anaconda3/envs/rnn_lstm_har_pytorch/lib/python3.6/site-
  packages/torchvision/transforms/functional.py in normalize(tensor,
  mean, std, inplace)
  206     mean = torch.tensor(mean, dtype=torch.float32)
  207     std = torch.tensor(std, dtype=torch.float32)
 --> 208     tensor.sub_(mean[:, None, None]).div_(std[:, None,
  None])
  209     return tensor
  210

RuntimeError: The size of tensor a (128) must match the size of
tensor b (9) at non-singleton dimension 0

   # This is for parsing the X data, you can ignore it if you do not
   need preprocessing
   def format_data_x(datafile):
    x_data = None
    for item in datafile:
    item_data = np.loadtxt(item, dtype=np.float)
    if x_data is None:
        x_data = np.zeros((len(item_data), 1))
    x_data = np.hstack((x_data, item_data))
    x_data = x_data[:, 1:]
    print(x_data.shape)
    X = None
    for i in range(len(x_data)):
    row = np.asarray(x_data[i, :])
    row = row.reshape(9, 128).T
    if X is None:
        X = np.zeros((len(x_data), 128, 9))
    X[i] = row
    print(X.shape)
    return X

    # This is for parsing the Y data, you can ignore it if you do not
    need preprocessing
    def format_data_y(datafile):
    data = np.loadtxt(datafile, dtype=np.int) - 1
    YY = np.eye(6)[data]
    return YY

    # Load data function, if there exists parsed data file, then use
    it
    # If not, parse the original dataset from scratch
   def load_data():
   import os

    # This for processing the dataset from scratch
    # After downloading the dataset, program put it in the DATA_PATH
   folder

    #str_folder = 'data/' + 'UCI HAR Dataset/'
    DATA_PATH = 'data/'
    DATASET_PATH = DATA_PATH + 'UCI HAR Dataset/'
    TRAIN = 'train/'
    TEST = 'test/'

    INPUT_SIGNAL_TYPES = [
        "body_acc_x_",
        "body_acc_y_",
        "body_acc_z_",
        "body_gyro_x_",
        "body_gyro_y_",
        "body_gyro_z_",
        "total_acc_x_",
        "total_acc_y_",
        "total_acc_z_"
    ]

    str_train_files = [DATASET_PATH + TRAIN + 'Inertial Signals/' +
    item + 'train.txt' for item in
                       INPUT_SIGNAL_TYPES]
    str_test_files = [DATASET_PATH + TEST + 'Inertial Signals/' + item
    + 'test.txt' for item in INPUT_SIGNAL_TYPES]
    str_train_y = DATASET_PATH + TRAIN + 'y_train.txt'
    str_test_y = DATASET_PATH + TEST + 'y_test.txt'

    X_train = format_data_x(str_train_files)
    X_test = format_data_x(str_test_files)
    Y_train = format_data_y(str_train_y)
    Y_test = format_data_y(str_test_y)

   return X_train, onehot_to_label(Y_train), X_test,
   onehot_to_label(Y_test)

  def onehot_to_label(y_onehot):
  a = np.argwhere(y_onehot == 1)
  return a[:, -1]

    class data_loader(Dataset):
    def __init__(self, samples, labels, t):
    self.samples = samples
    self.labels = labels
    self.T = t

    def __getitem__(self, index):
    sample, target = self.samples[index], self.labels[index]
    return self.T(sample), target

    def __len__(self):
    return len(self.samples)

   def load(batch_size=64):
   x_train, y_train, x_test, y_test = load_data()
   x_train, x_test = x_train.reshape((-1, 9, 1, 128)),
   x_test.reshape((-1, 9, 1, 128))
   transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize(mean=(0,0,0,0,0,0,0,0,0), std=
    (1,1,1,1,1,1,1,1,1))
    ])
  train_set = data_loader(x_train, y_train, transform)
  test_set = data_loader(x_test, y_test, transform)
  train_loader = DataLoader(train_set, batch_size=batch_size,
  shuffle=True, drop_last=True)
  test_loader = DataLoader(test_set, batch_size=batch_size,
  shuffle=False)
  return train_loader, test_loader

'''


title: PyTorch set_grad_enabled(False) vs with no_grad():

body:
Assuming autograd is on (as it is by default), is there any difference (besides  indent) between doing:

with torch.no_grad():

and

torch.set_grad_enabled(False)

torch.set_grad_enabled(True)


title: How do you convert a .onnx to tflite?

body:
I've exported my model to ONNX via:

# Export the model
torch_out = torch.onnx._export(learn.model,             # model being run
                           x,                       # model input (or a tuple for multiple inputs)
                          EXPORT_PATH + "mnist.onnx", # where to save the model (can be a file or file-like object)
                           export_params=True)      # store the trained parameter weights inside the model file

And now I am trying to convert the model to a Tensorflow Lite file so that I can do inference on Android. Unfortunately, PyTorch/Caffe2 support is fairly lacking or too complex for Android but Tensorflow appears much simpler.

The documentation for ONNX to Tflite is pretty light on this.

I've tried exporting to a Tensorflow GraphDef proto via:

tf_rep.export_graph(EXPORT_PATH + 'mnist-test/mnist-tf-export.pb')

And then running toco:

toco \
--graph_def_file=mnist-tf-export.pb \
--input_format=TENSORFLOW_GRAPHDEF \
--output_format=TFLITE \
--inference_type=FLOAT \
--input_type=FLOAT \
--input_arrays=0 \
--output_arrays=add_10 \
--input_shapes=1,3,28,28 \
--output_file=mnist.tflite`

When I do though I get the following error:

File "anaconda3/lib/python3.6/site-packages/tensorflow/lite/python/convert.py", line 172, in toco_convert_protos
    "TOCO failed. See console for info.\n%s\n%s\n" % (stdout, stderr))
tensorflow.lite.python.convert.ConverterError: TOCO failed. See console for info.
2018-11-06 16:28:33.864889: I tensorflow/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: PyFunc
2018-11-06 16:28:33.874130: F tensorflow/lite/toco/import_tensorflow.cc:114] Check failed: attr.value_case() == AttrValue::kType (1 vs. 6)

Further, even when I run the command I don't know what to specify for the input_arrays or output_arrays since the model was originally built in PyTorch.

Has anyone successfully converted their ONNX model to TFlite?

Here's the ONNX file I'm trying to convert: https://drive.google.com/file/d/1sM4RpeBVqPNw1WeCROpKLdzbSJPWSK79/view?usp=sharing

Extra info

Python 3.6.6 :: Anaconda custom (64-bit)
onnx.version = '1.3.0'
tf.version = '1.13.0-dev20181106'
torch.version = '1.0.0.dev20181029'


title: Include .whl installation in requirements.txt

body:
How do I include this in the requirements.txt file?
For Linux:

pip install http://download.pytorch.org/whl/cu75/torch-0.1.12.post2-cp27-none-linux_x86_64.whl
pip install torchvision

FOR MacOS:

pip install http://download.pytorch.org/whl/torch-0.1.12.post2-cp27-none-macosx_10_7_x86_64.whl
pip install torchvision


title: Running out of memory during evaluation in Pytorch

body:
I'm training a model in pytorch. Every 10 epochs, I'm evaluating the train and test error on the entire train and test dataset. For some reason the evaluation function is causing out-of-memory on my GPU. This is strange because I have the same batchsize for training and evaluation. I believe it's due to the net.forward() method being called repeated and having all the hidden values stored in memory but I'm not sure how to get around this?

def evaluate(self, data):
    correct = 0
    total = 0
    loader = self.train_loader if data == "train" else self.test_loader
    for step, (story, question, answer) in enumerate(loader):
        story = Variable(story)
        question = Variable(question)
        answer = Variable(answer)
        _, answer = torch.max(answer, 1)

        if self.config.cuda:
            story = story.cuda()
            question = question.cuda()
            answer = answer.cuda()

        pred_prob = self.mem_n2n(story, question)[0]
        _, output_max_index = torch.max(pred_prob, 1)
        toadd = (answer == output_max_index).float().sum().data[0]
        correct = correct + toadd
        total = total + captions.size(0)

    acc = correct / total
    return acc


title: Efficient metrics evaluation in PyTorch

body:
I am new to PyTorch and want to efficiently evaluate among others F1 during my Training and my Validation Loop.

So far, my approach was to calculate the predictions on GPU, then push them to CPU and append them to a vector for both Training and Validation. After Training and Validation, I would evaluate both for each epoch using sklearn. However, profiling my code it showed, that pushing to cpu is quite a bottleneck.

for epoch in range(n_epochs):
    model.train()
    avg_loss = 0
    avg_val_loss = 0
    train_pred = np.array([])
    val_pred = np.array([])
    # Training loop (transpose X_batch to fit pretrained (features, samples) style)
    for X_batch, y_batch in train_loader:
        scores = model(X_batch)
        y_pred = F.softmax(scores, dim=1)
        train_pred = np.append(train_pred, self.get_vector(y_pred.detach().cpu().numpy()))

        loss = loss_fn(scores, self.get_vector(y_batch))
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        avg_loss += loss.item() / len(train_loader)

    model.eval()
    # Validation loop
    for X_batch, y_batch in val_loader:
        with torch.no_grad():
            scores = model(X_batch)
            y_pred = F.softmax(scores, dim=1)
            val_pred = np.append(val_pred, self.get_vector(y_pred.detach().cpu().numpy()))
            loss = loss_fn(scores, self.get_vector(y_batch))
            avg_val_loss += loss.item() / len(val_loader)

    # Model Checkpoint for best validation f1
    val_f1 = self.calculate_metrics(train_targets[val_index], val_pred, f1_only=True)
    if val_f1 > best_val_f1:
        prev_best_val_f1 = best_val_f1
        best_val_f1 = val_f1
        torch.save(model.state_dict(), self.PATHS['xlm'])
        evaluated_epoch = epoch

    # Calc the metrics
    self.save_metrics(train_targets[train_index], train_pred, avg_loss, 'train')
    self.save_metrics(train_targets[val_index], val_pred, avg_val_loss, 'val')

I am certain there is a more efficient way to
a) store the predictions without having to push them to cpu each batch. b) calculate the metrics on GPU directly?

As I am new to PyTorch, I am very grateful for any hints and feedback :)


title: Multivariate input LSTM in pytorch

body:
I would like to implement LSTM for multivariate input in Pytorch.

Following this article https://machinelearningmastery.com/how-to-develop-lstm-models-for-time-series-forecasting/ which uses keras, the input data are in shape of (number of samples, number of timesteps, number of parallel features)

in_seq1 = array([10, 20, 30, 40, 50, 60, 70, 80, 90])
in_seq2 = array([15, 25, 35, 45, 55, 65, 75, 85, 95])
out_seq = array([in_seq1[i]+in_seq2[i] for i in range(len(in_seq1))])
. . .
Input     Output
[[10 15]
 [20 25]
 [30 35]] 65
[[20 25]
 [30 35]
 [40 45]] 85
[[30 35]
 [40 45]
 [50 55]] 105
[[40 45]
 [50 55]
 [60 65]] 125
[[50 55]
 [60 65]
 [70 75]] 145
[[60 65]
 [70 75]
 [80 85]] 165
[[70 75]
 [80 85]
 [90 95]] 185

n_timesteps = 3
n_features = 2

In keras it seems to be easy:

model.add(LSTM(50, activation='relu', input_shape=(n_timesteps, n_features)))

Can it be done in other way, than creating n_features of LSTMs as first layer and feed each separately (imagine as multiple streams of sequences) and then flatten their output to linear layer?

I'm not 100% sure but by nature of LSTM the input cannot be flattened and passed as 1D array, because each sequence "plays by different rules" which the LSTM is supposed to learn.

So how does such implementation with keras equal to PyTorch
input of shape (seq_len, batch, input_size)(source https://pytorch.org/docs/stable/nn.html#lstm)

Edit:

  Can it be done in other way, than creating n_features of LSTMs as first layer and feed each separately (imagine as multiple streams of sequences) and then flatten their output to linear layer?

According to PyTorch docs the input_size parameter actually means number of features (if it means number of parallel sequences)


title: Torch.cuda.is_available() keeps switching to False

body:
I have tried several solutions which hinted at what to do when the CUDA GPU is available and CUDA is installed but the Torch.cuda.is_available() returns False. They did help but only temporarily, meaning torch.cuda-is_available() reported True but after some time, it switched back to False. I use CUDA 9.0.176 and GTX 1080. What should I do to get the permanent effect?
I tried the following methods:
https://forums.fast.ai/t/torch-cuda-is-available-returns-false/16721/5
https://github.com/pytorch/pytorch/issues/15612
Note: When torch.cuda.is_available() works fine but then at some point switches to False, then I have to restart the computer and then it works again (for some time).


title: RuntimeError: module must have its parameters and buffers on device cuda:1 (device_ids[0]) but found one of them on device: cuda:2

body:
I have 4 GPUs (0,1,2,3) and I want to run one Jupyter notebook on GPU 2 and another one on GPU 0. Thus, after executing,

 export CUDA_VISIBLE_DEVICES=0,1,2,3

for the GPU 2 notebook I do,

device = torch.device( f'cuda:{2}' if torch.cuda.is_available() else 'cpu')
device, torch.cuda.device_count(), torch.cuda.is_available(), torch.cuda.current_device(), torch.cuda.get_device_properties(1)

and after creating a new model or loading one,

model = nn.DataParallel( model, device_ids = [ 0, 1, 2, 3])
model = model.to( device)

Then, when I start training the model, I get,

RuntimeError                              Traceback (most recent call last)
<ipython-input-18-849ffcb53e16> in <module>
 46             with torch.set_grad_enabled( phase == 'train'):
 47                 # [N, Nclass, H, W]
 ---> 48                 prediction = model(X)
 49                 # print( prediction.shape, y.shape)
 50                 loss_matrix = criterion( prediction, y)

~/.local/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
491             result = self._slow_forward(*input, **kwargs)
492         else:
--> 493             result = self.forward(*input, **kwargs)
494         for hook in self._forward_hooks.values():
495             hook_result = hook(self, input, result)

~/.local/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py in forward(self, *inputs, **kwargs)
144                 raise RuntimeError("module must have its parameters and buffers "
145                                    "on device {} (device_ids[0]) but found one of "
--> 146                                    "them on device: {}".format(self.src_device_obj, t.device))
147
148         inputs, kwargs = self.scatter(inputs, kwargs, self.device_ids)

RuntimeError: module must have its parameters and buffers on device cuda:0 (device_ids[0]) but found one of them on device: cuda:2


title: Installing PyTorch

body:
I'd like to use PyTorch in a Python program. The instructions for installing it require conda. After installing Conda I ran

>conda install -c pytorch pytorch (as instructed on the PyTorch [page][1])

It looked promising -- until the end.

Solving environment: done

## Package Plan ##

  environment location: C:\ProgramData\Miniconda3

  added / updated specs:
    - pytorch

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    icc_rt-2017.0.4            |       h97af966_0         8.0 MB
    vs2015_runtime-15.5.2      |                3         2.2 MB
    pytorch-0.4.0              |py36_cuda80_cudnn7he774522_1       529.2 MB      pytorch
    mkl-2018.0.3               |                1       178.1 MB
    numpy-1.14.5               |   py36h9fa60d3_4          35 KB
    intel-openmp-2018.0.3      |                0         1.7 MB
    numpy-base-1.14.5          |   py36h5c71026_4         3.8 MB
    vc-14.1                    |       h0510ff6_3           5 KB
    blas-1.0                   |              mkl           6 KB
    conda-4.5.8                |           py36_0         1.0 MB
    mkl_fft-1.0.2              |   py36hb217b18_0         113 KB
    mkl_random-1.0.1           |   py36h77b88f5_1         268 KB
    ------------------------------------------------------------
                                           Total:       724.4 MB

The following NEW packages will be INSTALLED:

    blas:           1.0-mkl
    icc_rt:         2017.0.4-h97af966_0
    intel-openmp:   2018.0.3-0
    mkl:            2018.0.3-1
    mkl_fft:        1.0.2-py36hb217b18_0
    mkl_random:     1.0.1-py36h77b88f5_1
    numpy:          1.14.5-py36h9fa60d3_4
    numpy-base:     1.14.5-py36h5c71026_4
    pytorch:        0.4.0-py36_cuda80_cudnn7he774522_1 pytorch

The following packages will be UPDATED:

    conda:          4.5.4-py36_0                               --> 4.5.8-py36_0
    vc:             14-h0510ff6_3                              --> 14.1-h0510ff6_3
    vs2015_runtime: 14.0.25123-3                               --> 15.5.2-3

Proceed ([y]/n)? y

Downloading and Extracting Packages
icc_rt-2017.0.4      |  8.0 MB | ############################################################################## | 100%
vs2015_runtime-15.5. |  2.2 MB | ############################################################################## | 100%
pytorch-0.4.0        | 529.2 MB | ############################################################################# | 100%
mkl-2018.0.3         | 178.1 MB | ############################################################################# | 100%
numpy-1.14.5         |   35 KB | ############################################################################## | 100%
intel-openmp-2018.0. |  1.7 MB | ############################################################################## | 100%
numpy-base-1.14.5    |  3.8 MB | ############################################################################## | 100%
vc-14.1              |    5 KB | ############################################################################## | 100%
blas-1.0             |    6 KB | ############################################################################## | 100%
conda-4.5.8          |  1.0 MB | ############################################################################## | 100%
mkl_fft-1.0.2        |  113 KB | ############################################################################## | 100%
mkl_random-1.0.1     |  268 KB | ############################################################################## | 100%
Preparing transaction: done
Verifying transaction: done

But then this.

Executing transaction: failed
ERROR conda.core.link:_execute(502): An error occurred while uninstalling package 'defaults::conda-4.5.4-py36_0'.
PermissionError(13, 'Access is denied')
Attempting to roll back.

Rolling back transaction: done

PermissionError(13, 'Access is denied')

Apparently it was at least partly installed because PyCharm was able to see it. But when I asked PyCharm to install it in an environment, I got this error.

RuntimeError: PyTorch does not currently provide packages for PyPI (see status at https://github.com/pytorch/pytorch/issues/566).

Please follow the instructions at http://pytorch.org/ to install with miniconda instead.

It suggests an alternative way to install PyTorch.  So I tried that.

>conda install pytorch torchvision -c pytorch
Solving environment: failed

PackagesNotFoundError: The following packages are not available from current channels:

  - torchvision

Current channels:

  - https://conda.anaconda.org/pytorch/win-64
  - https://conda.anaconda.org/pytorch/noarch
  - https://repo.anaconda.com/pkgs/main/win-64
  - https://repo.anaconda.com/pkgs/main/noarch
  - https://repo.anaconda.com/pkgs/free/win-64
  - https://repo.anaconda.com/pkgs/free/noarch
  - https://repo.anaconda.com/pkgs/r/win-64
  - https://repo.anaconda.com/pkgs/r/noarch
  - https://repo.anaconda.com/pkgs/pro/win-64
  - https://repo.anaconda.com/pkgs/pro/noarch
  - https://repo.anaconda.com/pkgs/msys2/win-64
  - https://repo.anaconda.com/pkgs/msys2/noarch

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.

But when I do that and search for PyTorch, I eventually find myself back at the original instructions.

When I search for Torchvision, no Windows versions are listed.

Any help would be appreciated.

Thanks.


title: pytorch freeze weights and update param_groups

body:
Freezing weights in pytorch for param_groups setting.

So if one wants to freeze weights during training:

for param in child.parameters():
    param.requires_grad = False

the optimizer also has to be updated to not include the non gradient weights:

optimizer = torch.optim.Adam(filter(lambda p: p.requires_grad, model.parameters()), lr=opt.lr, amsgrad=True)

If one wants to use different weight_decay / learning rates for bias and weights/this also allows for differing learning rates:

param_groups = [{'params': model.module.bias_parameters(), 'weight_decay': args.bias_decay},
                {'params': model.module.weight_parameters(), 'weight_decay': args.weight_decay}]

param_groups a list of dics is defined and passed into SGD as follows:

optimizer = torch.optim.Adam(param_groups, args.lr,
                                 betas=(args.momentum, args.beta))

How can this be achieved with freezing individual weights? Running filter over a list of dics or is there a way of adding tensors to the optimizer separately?


title: Pytorch sum over a list of tensors along an axis

body:
I have a list of tensors of the same shape.
I would like to sum the entire list of tensors along an axis.
Does torch.cumsum perform this op along a dim?
If so it requires the list to be converted to a single tensor and summed over?


title: Get CUDA_HOME environment path PYTORCH

body:
I have cuda installed via anaconda on my system which has 2 GPUs which is getting recognized by my python.

import torch
torch.cuda.is_available()
true

However when I try to run a model via its C API, I m getting following error:

~/anaconda3/lib/python3.6/site-packages/torch/utils/cpp_extension.py in _join_cuda_home(*paths)
722     '''
723     if CUDA_HOME is None:
--> 724         raise EnvironmentError('CUDA_HOME environment variable is not set. '
725                                'Please set it to your CUDA install root.')
726     return os.path.join(CUDA_HOME, *paths)

OSError: CUDA_HOME environment variable is not set. Please set it to your CUDA install root.

https://lfd.readthedocs.io/en/latest/install_gpu.html page gives instruction to set up CUDA_HOME path if cuda is installed via their method. Since I have installed cuda via anaconda I don't know which path to set.

I tried find method but it is returning me too many paths for cuda. Can somebody help me with the path for CUDA. Thanks in advance.


title: Size mismatch for fc.bias and fc.weight in PyTorch

body:
I used the transfer learning approach to train a model and saved the best-detected weights. In another script, I tried to use the saved weights for prediction. But I am getting errors as follows. I have used ResNet for finetuning the network and have 4 classes.

RuntimeError: Error(s) in loading state_dict for ResNet:
size mismatch for fc.bias: copying a param of torch.Size([1000]) from
checkpoint, where the shape is torch.Size([4]) in current model.
size mismatch for fc.weight: copying a param of torch.Size([1000,
512]) from checkpoint, where the shape is torch.Size([4, 512]) in
current model.

I am using the following code for prediction of output:

checkpoint = torch.load("./models/custom_model13.model")
model = resnet18(pretrained=True)

model.load_state_dict(checkpoint)
model.eval()

def predict_image(image_path):
    transformation = transforms.Compose([
        transforms.RandomResizedCrop(224),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
        ])
    image_tensor = transformation(image).float()
    image_tensor = image_tensor.unsqueeze_(0)

    if torch.cuda.is_available():
        image_tensor.cuda()

    input = Variable(image_tensor)
    output = model(input)

    index = output.data.numpy().argmax()
    return index

if __name__ == "main":
    imagefile = "image.png"
    imagepath = os.path.join(os.getcwd(),imagefile)
    prediction = predict_image(imagepath)
    print("Predicted Class: ",prediction)

And the following code to train and save the model:

Data_dir = 'Dataset'
image_datasets = {x: datasets.ImageFolder(os.path.join(data_dir, x),
                                          data_transforms[x])
                  for x in ['train', 'val']}
dataloaders = {x: torch.utils.data.DataLoader(image_datasets[x], batch_size=4,
                                             shuffle=True, num_workers=4)
              for x in ['train', 'val']}
dataset_sizes = {x: len(image_datasets[x]) for x in ['train', 'val']}
class_names = image_datasets['train'].classes

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
print (device)

def save_models(epochs, model):
    torch.save(model.state_dict(), "custom_model{}.model".format(epochs))
    print("Checkpoint Saved")

def train_model(model, criterion, optimizer, scheduler, num_epochs=25):
    since = time.time()

    best_model_wts = copy.deepcopy(model.state_dict())
    best_acc = 0.0

    for epoch in range(num_epochs):
        print('Epoch {}/{}'.format(epoch, num_epochs - 1))
        print('-' * 10)

        # Each epoch has a training and validation phase
        for phase in ['train', 'val']:
            if phase == 'train':
                scheduler.step()
                model.train()  # Set model to training mode
            else:
                model.eval()   # Set model to evaluate mode

            running_loss = 0.0
            running_corrects = 0

            # Iterate over data.
            for inputs, labels in dataloaders[phase]:
                inputs = inputs.to(device)
                labels = labels.to(device)

                # zero the parameter gradients
                optimizer.zero_grad()

                # forward
                # track history if only in train
                with torch.set_grad_enabled(phase == 'train'):
                    outputs = model(inputs)
                    _, preds = torch.max(outputs, 1)
                    loss = criterion(outputs, labels)

                    # backward + optimize only if in training phase
                    if phase == 'train':
                        loss.backward()
                        optimizer.step()

                # statistics
                running_loss += loss.item() * inputs.size(0)
                running_corrects += torch.sum(preds == labels.data)

            epoch_loss = running_loss / dataset_sizes[phase]
            epoch_acc = running_corrects.double() / dataset_sizes[phase]

            print('{} Loss: {:.4f} Acc: {:.4f}'.format(
                phase, epoch_loss, epoch_acc))

            # deep copy the model
            if phase == 'train' and epoch_acc > best_acc:
                save_models(epoch,model)
                best_acc = epoch_acc
                best_model_wts = copy.deepcopy(model.state_dict())

        print()

    time_elapsed = time.time() - since
    print('Training complete in {:.0f}m {:.0f}s'.format(
        time_elapsed // 60, time_elapsed % 60))
    print('Best val Acc: {:4f}'.format(best_acc))

    # load best model weights
    model.load_state_dict(best_model_wts)
    return model

def visualize_model(model, num_images=6):
    was_training = model.training
    model.eval()
    images_so_far = 0
    fig = plt.figure()

    with torch.no_grad():
        for i, (inputs, labels) in enumerate(dataloaders['val']):
            inputs = inputs.to(device)
            labels = labels.to(device)

            outputs = model(inputs)
            _, preds = torch.max(outputs, 1)

            for j in range(inputs.size()[0]):
                images_so_far += 1
                ax = plt.subplot(num_images//2, 2, images_so_far)
                ax.axis('off')
                ax.set_title('predicted: {}'.format(class_names[preds[j]]))
                imshow(inputs.cpu().data[j])

                if images_so_far == num_images:
                    model.train(mode=was_training)
                    return
        model.train(mode=was_training)

model_ft = models.resnet18(pretrained=True)
num_ftrs = model_ft.fc.in_features
model_ft.fc = nn.Linear(num_ftrs, 4)

model_ft = model_ft.to(device)

criterion = nn.CrossEntropyLoss()

optimizer_ft = optim.SGD(model_ft.parameters(), lr=0.001, momentum=0.9)

exp_lr_scheduler = lr_scheduler.StepLR(optimizer_ft, step_size=7, gamma=0.1)

model_ft = train_model(model_ft, criterion, optimizer_ft, exp_lr_scheduler,
                       num_epochs=25)


title: Pytorch RuntimeError: "host_softmax" not implemented for 'torch.cuda.LongTensor'

body:
I am using pytorch for training models. But I got an runtime error when it was computing the cross-entropy loss.

Traceback (most recent call last):
  File "deparser.py", line 402, in <module>
    d.train()
  File "deparser.py", line 331, in train
    total, correct, avgloss = self.train_util()
  File "deparser.py", line 362, in train_util
    loss = self.step(X_train, Y_train, correct, total)
  File "deparser.py", line 214, in step
    loss = nn.CrossEntropyLoss()(out.long(), y)
  File "/home/summer2018/TF/lib/python3.5/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/summer2018/TF/lib/python3.5/site-packages/torch/nn/modules/loss.py", line 862, in forward
    ignore_index=self.ignore_index, reduction=self.reduction)
  File "/home/summer2018/TF/lib/python3.5/site-packages/torch/nn/functional.py", line 1550, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
  File "/home/summer2018/TF/lib/python3.5/site-packages/torch/nn/functional.py", line 975, in log_softmax
    return input.log_softmax(dim)
RuntimeError: "host_softmax" not implemented for 'torch.cuda.LongTensor'

I think this is because the .cuda() function or conversion between torch.Float and torch.Long. But I have tried many ways to change the variable by .cpu()/.cuda() and .long()/.float(), but it still not work. This error message can't be found when searching it on google. Can anyone helps me? Thanks!!!

This is the code cause error:

def step(self, x, y, correct, total):
    self.optimizer.zero_grad()
    out = self.forward(*x)
    loss = nn.CrossEntropyLoss()(out.long(), y)
    loss.backward()
    self.optimizer.step()
    _, predicted = torch.max(out.data, 1)
    total += y.size(0)
    correct += int((predicted == y).sum().data)
    return loss.data

And this function step() is called by:

def train_util(self):
    total = 0
    correct = 0
    avgloss = 0
    for i in range(self.step_num_per_epoch):
        X_train, Y_train = self.trainloader()
        self.optimizer.zero_grad()
        if torch.cuda.is_available():
            self.cuda()
            for i in range(len(X_train)):
                X_train[i] = Variable(torch.from_numpy(X_train[i]))
                X_train[i].requires_grad = False
                X_train[i] = X_train[i].cuda()
            Y_train = torch.from_numpy(Y_train)
            Y_train.requires_grad = False
            Y_train = Y_train.cuda()
        loss = self.step(X_train, Y_train, correct, total)
        avgloss+=float(loss)*Y_train.size(0)
        self.optimizer.step()
        if i%100==99:
            print('STEP %d, Loss: %.4f, Acc: %.4f'%(i+1,loss,correct/total))

    return total, correct, avgloss/self.data_len

The input data X_train, Y_train = self.trainloader() are numpy arrays at begining.

This is a data sample:

>>> X_train, Y_train = d.trainloader()
>>> X_train[0].dtype
dtype('int64')
>>> X_train[1].dtype
dtype('int64')
>>> X_train[2].dtype
dtype('int64')
>>> Y_train.dtype
dtype('float32')
>>> X_train[0]
array([[   0,    6,    0, ...,    0,    0,    0],
       [   0, 1944, 8168, ...,    0,    0,    0],
       [   0,  815,  317, ...,    0,    0,    0],
       ...,
       [   0,    0,    0, ...,    0,    0,    0],
       [   0,   23,    6, ...,    0,    0,    0],
       [   0,    0,  297, ...,    0,    0,    0]])
>>> X_train[1]
array([ 6,  7,  8, 21,  2, 34,  3,  4, 19, 14, 15,  2, 13,  3, 11, 22,  4,
   13, 34, 10, 13,  3, 48, 18, 16, 19, 16, 17, 48,  3,  3, 13])
>>> X_train[2]
array([ 4,  5,  8, 36,  2, 33,  5,  3, 17, 16, 11,  0,  9,  3, 10, 20,  1,
   14, 33, 25, 19,  1, 46, 17, 14, 24, 15, 15, 51,  2,  1, 14])
>>> Y_train
array([[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
       [0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       ...,
       [0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]],
      dtype=float32)

Try all possible combinations:

case 1:
loss = nn.CrossEntropyLoss()(out, y)
I get:
RuntimeError: Expected object of type torch.cuda.LongTensor but found type torch.cuda.FloatTensor for argument #2 'target'

case 2:
loss = nn.CrossEntropyLoss()(out.long(), y)
as description above

case 3:
loss = nn.CrossEntropyLoss()(out.float(), y)
I get:
RuntimeError: Expected object of type torch.cuda.LongTensor but found type torch.cuda.FloatTensor for argument #2 'target'

case 4:
loss = nn.CrossEntropyLoss()(out, y.long())
I get:
RuntimeError: multi-target not supported at /pytorch/aten/src/THCUNN/generic/ClassNLLCriterion.cu:15

case 5:
loss = nn.CrossEntropyLoss()(out.long(), y.long())
I get:
RuntimeError: "host_softmax" not implemented for 'torch.cuda.LongTensor'

case 6:
loss = nn.CrossEntropyLoss()(out.float(), y.long())
I get:
RuntimeError: multi-target not supported at /pytorch/aten/src/THCUNN/generic/ClassNLLCriterion.cu:15

case 7:
loss = nn.CrossEntropyLoss()(out, y.float())
I get:
RuntimeError: Expected object of type torch.cuda.LongTensor but found type torch.cuda.FloatTensor for argument #2 'target'

case 8:
loss = nn.CrossEntropyLoss()(out.long(), y.float())
I get:
RuntimeError: "host_softmax" not implemented for 'torch.cuda.LongTensor'

case 9:
loss = nn.CrossEntropyLoss()(out.float(), y.float())
I get:
RuntimeError: Expected object of type torch.cuda.LongTensor but found type torch.cuda.FloatTensor for argument #2 'target'


title: converting tensor to one hot encoded tensor of indices

body:
I have my label tensor of shape (1,1,128,128,128) in which the values might range from 0,24. I want to convert this to one hot encoded tensor, using the nn.fucntional.one_hot function

n = 24
one_hot = torch.nn.functional.one_hot(indices, n)

but this expects a tensor of indices, honestly, I am not sure how to get those. The only tensor I have is the label tensor of the shape described above and it contains values ranging from 1-24, not the indices

How can I get a tensor of indices from my tensor? Thanks in advance.


title: Custom loss function in PyTorch

body:
I have three simple questions.

What will happen if my custom loss function is not differentiable? Will pytorch through error or do something else?
If I declare a loss variable in my custom function which will represent the final loss of the model, should I put requires_grad = True for that variable? or it doesn't matter? If it doesn't matter, then why?
I have seen people sometimes write a separate layer and compute the loss in the forward function. Which approach is preferable, writing a function or a layer? Why?

I need a clear and nice explanation to these questions to resolve my confusions. Please help.


title: Load a single image in a pretrained pytorch net

body:
Total newbie here, I'm using this pytorch SegNet implementation with a '.pth' file containing weights from a 50 epochs training.
How can I load a single test image and see the net prediction?
I know this may sound like a stupid question but I'm stuck.
What I've got is:

from segnet import SegNet
import torch

model = SegNet(2)
model.load_state_dict(torch.load('./model_segnet_epoch50.pth'))

How do I "use" the net on a single test picture?


title: How to correctly implement a batch-input LSTM network in PyTorch?

body:
This release of PyTorch seems provide the PackedSequence for variable lengths of input for recurrent neural network. However, I found it's a bit hard to use it correctly.

Using pad_packed_sequence to recover an output of a RNN layer which were fed by pack_padded_sequence, we got a T x B x N tensor outputs where T is the max time steps, B is the batch size and N is the hidden size. I found that for short sequences in the batch, the subsequent output will be all zeros.

Here are my questions.

For a single output task where the one would need the last output of all the sequences, simple outputs[-1] will give a wrong result since this tensor contains lots of zeros for short sequences. One will need to construct indices by sequence lengths to fetch the individual last output for all the sequences. Is there more simple way to do that?
For a multiple output task (e.g. seq2seq), usually one will add a linear layer N x O and reshape the batch outputs T x B x O into TB x O and compute the cross entropy loss with the true targets TB (usually integers in language model). In this situation, do these zeros in batch output matters?


title: Documentation for PyTorch .to('cpu') or .to('cuda')

body:
I've searched through the PyTorch documenation, but can't find anything for .to() which moves a tensor to CPU or CUDA memory.

I remember seeing somewhere that calling to() on a nn.Module is an in-place operation, but not so on a tensor.

Is there a in-place version for Tensors?

Where do I find the doco for to() for both nn.Module and Tensor (and possibly elsewhere)?


title: What is Adaptive average pooling and How does it work?

body:
I recently came across a method in Pytorch when I try to implement AlexNet.
I don't understand how it works. Please explain the idea behind it with some examples. And how it is different from Maxpooling or Average poling in terms of Neural Network functionality

  nn.AdaptiveAvgPool2d((6, 6))


title: How to apply layer-wise learning rate in Pytorch?

body:
I know that it is possible to freeze single layers in a network for example to train only the last layers of a pre-trained model. What I’m looking for is a way to apply certain learning rates to different layers.

So for example a very low learning rate of 0.000001 for the first layer and then increasing the learning rate gradually for each of the following layers. So that the last layer then ends up with a learning rate of 0.01 or so.

Is this possible in pytorch? Any idea how I can archive this?