title: How does PyTorch handle labels when loading image/mask files for image segmentation?

body:
I am starting an image segmentation project using PyTorch. I have a reduced dataset in a folder and 2 subfolders - "image" to store the images and "mask" for the masked images. Images and masks are .png files with 3 channels and 256x256 pixels. Because it is image segmentation, the labelling has to be performed a pixel by pixel. I am working only with 2 classes at the moment for simplicity. So far, I achieved the following:

I was able to load my files into classes "images" or "masks" by

root_dir="./images_masks"
train_ds_untransf = torchvision.datasets.ImageFolder(root=root_dir)
train_ds_untransf.classes
Out[621]:
['images', 'masks']

and transform the data into tensors

from torchvision import transforms
train_trans = transforms.Compose([transforms.ToTensor()])
train_dataset = torchvision.datasets.ImageFolder(root=root_dir,transform=train_trans)

Each tensor in this "train_dataset" has the following shape:

train_dataset[1][0].shape
torch.Size([3, 256, 256])

Now I need to feed the loaded data into the CNN model, and have explored the PyTorch DataLoader for this

train_dataloaded = DataLoader(train_dataset, batch_size=2, shuffle=False, num_workers=4)

I use the following code to check the resulting tensor's shape

for x, y in train_dl:
    print (x.shape)
    print (y.shape)
    print(y)

and get

torch.Size([2, 3, 256, 256])
torch.Size([2])
tensor([0, 0])
torch.Size([2, 3, 256, 256])
torch.Size([2])
tensor([0, 1])
.
.
.

Shapes seem correct. However, the first problem is that I got tensors of the same folder, indicated by some "y" tensors with the same value [0, 0]. I would expect that they all are [1, 0]: 1 representing image, 0 representing masks.

The second problem is that, although the documentation is clear when labels are entire images, it is not clear as to how to apply it for labeling at the pixel level, and I am certain the labels are not correct.

What would be an alternative to correctly label this dataset?

thank you


title: Simple way to load specific sample using Pytorch dataloader

body:
I am currently training a 3D CNN for binary classification with relatively sparse labels (~ 1% of voxels in label data correspond to target class).

In order to perform basic sanity checks during the training (e.g. does the network learn at all?) it would be handy to present the network with a small, handpicked subset of training examples having an above-average fraction of target class labels.

As suggested by the Pytorch documentation, I implemented my own dataset class (inheriting from torch.utils.data.Dataset) which provides training examples via it's __get_item__ method to the torch.utils.data.DataLoader.

In the pytorch tutorials I found, the DataLoader is used as an iterator to generate the training loop like so:

for i, data in enumerate(self.dataloader):

    # Get training data
    inputs, labels = data

    # Train the network
    # [...]

What I am wondering now is whether there exist a simple way to load a single or a couple of specific training examples (using a the linear index understood by Dataset's __get_item__ method). However, DataLoader does not have a __get_item__ method and repeatedly calling __next__ until I reach the desired index does not seem elegant.

Apparently one possible way to solve this would be to define a custom sampler or batch_sampler inheriting from the abstract torch.utils.data.Sampler. But this seems over the top to retrieve a few specific samples.

I suppose I am overlooking something very simple and obvious here. Any advice appreciated!


title: Install specific version of PyTorch to conda environment

body:
Using Anaconda Navigator I created a new environment for running someone's VAE code off GitHub that uses Python 3.6 and PyTorch 0.4.0. Unfortunately, Anaconda Navigator doesn't give me the option to install an older version of PyTorch on this environment, just the PyTorch version I have currently installed. How do I install PyTorch 0.4.0 only to this new Conda environment I created? If it's possible via Anaconda Navigator, great! But I assume it's going to be done via a Conda command. I definitely don't want to mess up my other environments.

Thanks!


title: how can I insert a Tensor into another Tensor in pytorch

body:
I have pytorch Tensor with shape (batch_size, step, vec_size), for example, a Tensor(32, 64, 128), let's call it A.

I have another Tensor(batch_size, vec_size), e.g. Tensor(32, 128), let's call it B.

I want to insert B into a certain position at axis 1 of A. The insert positions are given in a Tensor(batch_size), named P.

I understand there is no Empty tensor(like an empty list) in pytorch, so, I initialize A as zeros, and add B at a certain position at axis 1 of A.

A = Variable(torch.zeros(batch_size, step, vec_size))

What I'm doing is like:

for i in range(batch_size):
    pos = P[i]
    A[i][pos] = A[i][pos] + B[i]

But I get an Error:

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation

Then, I make a clone of A each inside the loop:

for i in range(batch_size):
    A_clone =  A.clone()
    pos = P[i]
    A_clone[i][pos] = A_clone[i][pos] + B[i]

This is very slow for autograd, I wonder if there any better solutions? Thank you.


title: inputing numpy array images into pytorch neural net

body:
I have a numpy array representation of an image and I want to turn it into a tensor so I can feed it through my pytorch neural network.

I understand that the neural networks take in transformed tensors which are not arranged in [100,100,3] but [3,100,100] and the pixels are rescaled and the images must be in batches.

So I did the following:

import cv2
my_img = cv2.imread('testset/img0.png')
my_img.shape #reuturns [100,100,3] a 3 channel image with 100x100 resolution
my_img = np.transpose(my_img,(2,0,1))
my_img.shape #returns [3,100,100]
#convert the numpy array to tensor
my_img_tensor = torch.from_numpy(my_img)
#rescale to be [0,1] like the data it was trained on by default
my_img_tensor *= (1/255)
#turn the tensor into a batch of size 1
my_img_tensor = my_img_tensor.unsqueeze(0)
#send image to gpu
my_img_tensor.to(device)
#put forward through my neural network.
net(my_img_tensor)

However this returns the error:

RuntimeError: _thnn_conv2d_forward is not implemented for type torch.ByteTensor


title: torch.cuda.device_count() return 0

body:
I have installed pytorch under virtualenv via:

pip3 install https://download.pytorch.org/whl/cu100/torch-1.0.1.post2-cp35-cp35m-linux_x86_64.whl
pip3 install torchvision

But for some reason:

import torch
torch.cuda.device_count()
0
torch.cuda.is_available()
False

Looks like pytorch detected cuda 9 for some reason:

torch._C._cuda_getDriverVersion()
9010

My system:

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.5 LTS
Release:    16.04
Codename:   xenial

nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Fri_Feb__8_19:08:17_PST_2019
Cuda compilation tools, release 10.1, V10.1.105

Python 3.5.2 (default, Nov 23 2017, 16:37:01)
[GCC 5.4.0 20160609] on linux

ls -l /usr/local/ | grep cuda
lrwxrwxrwx  1 root   root      21 Mar 24 02:05 cuda -> /usr/local/cuda-10.0/
drwxr-xr-x 17 root   root    4096 Mar 24 02:06 cuda-10.0
drwxr-xr-x 15 root   root    4096 Nov 29 23:03 cuda-9.0
drwxr-xr-x 15 root   root    4096 Jan 10  2018 cuda-9.1

nvidia-smi -q | grep "Product Name"
    Product Name                    : GeForce GTX TITAN X
    Product Name                    : GeForce GTX 1080 Ti


Is there a difference between "torch.nn.CTCLoss" supported by PYTORCH and "CTCLoss" supported by torch_baidu_ctc?

i think, I didn't notice any difference when I compared the tutorial code.

Does anyone know the true?

Tutorial code is located below.

import torch
from torch_baidu_ctc import ctc_loss, CTCLoss

# Activations. Shape T x N x D.
# T -> max number of frames/timesteps
# N -> minibatch size
# D -> number of output labels (including the CTC blank)
x = torch.rand(10, 3, 6)
# Target labels
y = torch.tensor([
# 1st sample
1, 1, 2, 5, 2,
# 2nd
1, 5, 2,
# 3rd
4, 4, 2, 3,
],
dtype=torch.int,
)
# Activations lengths
xs = torch.tensor([10, 6, 9], dtype=torch.int)
# Target lengths
ys = torch.tensor([5, 3, 4], dtype=torch.int)

# By default, the costs (negative log-likelihood) of all samples are
summed.
# This is equivalent to:
#   ctc_loss(x, y, xs, ys, average_frames=False, reduction="sum")
loss1 = ctc_loss(x, y, xs, ys)

# You can also average the cost of each sample among the number of
frames.
# The averaged costs are then summed.
loss2 = ctc_loss(x, y, xs, ys, average_frames=True)

# Instead of summing the costs of each sample, you can perform
# other `reductions`: "none", "sum", or "mean"
#
# Return an array with the loss of each individual sample
losses = ctc_loss(x, y, xs, ys, reduction="none")
#
# Compute the mean of the individual losses
loss3 = ctc_loss(x, y, xs, ys, reduction="mean")
#
# First, normalize loss by number of frames, later average losses
loss4 = ctc_loss(x, y, xs, ys, average_frames=True, reduction="mean")

# Finally, there's also a nn.Module to use this loss.
ctc = CTCLoss(average_frames=True, reduction="mean", blank=0)
loss4_2 = ctc(x, y, xs, ys)

# Note: the `blank` option is also available for `ctc_loss`.
# By default it is 0.

torch.nn.CTCLoss

T = 50      # Input sequence length
C = 20      # Number of classes (excluding blank)
N = 16      # Batch size
S = 30      # Target sequence length of longest target in batch
S_min = 10  # Minimum target length, for demonstration purposes

# Initialize random batch of input vectors, for *size = (T,N,C)
input = torch.randn(T, N, C).log_softmax(2).detach().requires_grad_()

# Initialize random batch of targets (0 = blank, 1:C+1 = classes)
target = torch.randint(low=1, high=C+1, size=(N, S), dtype=torch.long)

input_lengths = torch.full(size=(N,), fill_value=T, dtype=torch.long)
target_lengths = torch.randint(low=S_min, high=S, size=(N,),
dtype=torch.long)
ctc_loss = nn.CTCLoss()
loss = ctc_loss(input, target, input_lengths, target_lengths)
loss.backward()

I am Korean. English is not my first language. So I'm not good at English. If there's anything that hasn't been delivered well, please leave a comment. I'll change the sentence as soon as possible.


title: How does pytorch's nn.Module register submodule?

body:
When I read the source code(python) of torch.nn.Module , I found the
  attribute self._modules has been used in many functions like
  self.modules(), self.children(), etc. However, I didn't find any functions
  updating it. So, where will the self._modules be updated?
  Furthermore, how does pytorch's nn.Module register submodule?

class Module(object):
    def __init__(self):
        self._backend = thnn_backend
        self._parameters = OrderedDict()
        self._buffers = OrderedDict()
        self._backward_hooks = OrderedDict()
        self._forward_hooks = OrderedDict()
        self._forward_pre_hooks = OrderedDict()
        self._modules = OrderedDict()
        self.training = True

    def named_modules(self, memo=None, prefix=''):
        if memo is None:
            memo = set()
        if self not in memo:
            memo.add(self)
            yield prefix, self
            for name, module in self._modules.items():
                if module is None:
                    continue
                submodule_prefix = prefix + ('.' if prefix else '') + name
                for m in module.named_modules(memo, submodule_prefix):
                    yield m


title: Pyinstaller unable to find dlls for the project dependencies while creating exe

body:
Pyinstaller failed to find certain dlls that are required for binding in dependencies into one exe.

Please find the error logs below.

We have tried installing these libraries:

pip3 install intel-openmp mkl

Tried adding --paths to the command, but as there are no dlls in the system, pyinstaller is unable to find them:

pyinstaller --onefile --paths <Paths-where-dll-could-be> -c main.py

These libs are missing and showed in logs as WARNINGS.

364427 WARNING: lib not found: impi.dll dependency of c:\users\1311654\appdata\local\programs\python\python36\Library\bin\mkl_blacs_intelmpi_ilp64.dll
365396 WARNING: lib not found: mpich2mpi.dll dependency of c:\users\1311654\appdata\local\programs\python\python36\Library\bin\mkl_blacs_mpich2_lp64.dll
366241 WARNING: lib not found: msmpi.dll dependency of c:\users\1311654\appdata\local\programs\python\python36\Library\bin\mkl_blacs_msmpi_lp64.dll
368089 WARNING: lib not found: msmpi.dll dependency of c:\users\1311654\appdata\local\programs\python\python36\Library\bin\mkl_blacs_msmpi_ilp64.dll
369270 WARNING: lib not found: pgf90.dll dependency of c:\users\1311654\appdata\local\programs\python\python36\Library\bin\mkl_pgi_thread.dll
369997 WARNING: lib not found: pgc14.dll dependency of c:\users\1311654\appdata\local\programs\python\python36\Library\bin\mkl_pgi_thread.dll
370791 WARNING: lib not found: pgf90rtl.dll dependency of c:\users\1311654\appdata\local\programs\python\python36\Library\bin\mkl_pgi_thread.dll
373039 WARNING: lib not found: mpich2mpi.dll dependency of c:\users\1311654\appdata\local\programs\python\python36\Library\bin\mkl_blacs_mpich2_ilp64.dll
374289 WARNING: lib not found: impi.dll dependency of c:\users\1311654\appdata\local\programs\python\python36\Library\bin\mkl_blacs_intelmpi_lp64.dll
377030 WARNING: lib not found: torch_python.dll dependency of c:\users\1311654\appdata\local\programs\python\python36\lib\site-packages\torch\_C.cp36-win_amd64.pyd
378792 WARNING: lib not found: c10_cuda.dll dependency of c:\users\1311654\appdata\local\programs\python\python36\lib\site-packages\torchvision\_C.cp36-win_amd64.pyd
379568 WARNING: lib not found: torch.dll dependency of c:\users\1311654\appdata\local\programs\python\python36\lib\site-packages\torchvision\_C.cp36-win_amd64.pyd
380290 WARNING: lib not found: caffe2.dll dependency of c:\users\1311654\appdata\local\programs\python\python36\lib\site-packages\torchvision\_C.cp36-win_amd64.pyd
381126 WARNING: lib not found: c10.dll dependency of c:\users\1311654\appdata\local\programs\python\python36\lib\site-packages\torchvision\_C.cp36-win_amd64.pyd
382053 WARNING: lib not found: torch_python.dll dependency of c:\users\1311654\appdata\local\programs\python\python36\lib\site-packages\torchvision\_C.cp36-win_amd64.pyd

As the missing lib dlls are not in the system, kindly assist the efficient way to build the exe.


title: In distributed computing, what are world size and rank?

body:
I've been reading through some documentation and example code with the end goal of writing scripts for distributed computing (running PyTorch), but the concepts confuse me.

Let's assume that we have a single node with 4 GPUs, and we want to run our script on those 4 GPUs (i.e. one process per GPU). In such a scenario, what are the rank world size and rank? I often find the explanation for world size: Total number of processes involved in the job, so I assume that that is four in our example, but what about rank?

To explain it further, another example with multiple nodes and multiple GPUs could be useful, too.


title: Pytorch: How does SGD with momentum works when optimizer has to call zero_grad() to help accumulation of gradients?

body:
In pytorch, the backward() function accumulates gradients and we have to reset it every mini-batch by calling optimizer.zero_grad(). In this case, how does the SGD with momentum works when actually momentum SGD updates the weights using exponential average of some past mini-batches.

For a beginner in Pytorch, I am confused. Doesn't it require to have past gradients to perform updates.


title: Using pre-trained sentence embeddings in PyTorch

body:
[Original post, see below for edited]

I am new to PyTorch, and trying to perform a sentence classification task in it.

I have averaged out the word embeddings in each sentence (glove embeddings) to form the sentence embedding. Hence, the dimension of each sentence embedding is the same. As per my understanding, since I already have the embeddings I don't need the embedding layer before using the LSTM.
My model is as follows:

import torch
import torch.autograd as autograd
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

class LSTM1(nn.Module):

def __init__(self,args):
    super(LSTM1,self).__init__()
    self.args = args
    self.hidden_dim = args.hidden_dim
    self.lstm = nn.LSTM(args.embed_dim, args.hidden_dim)
    self.hidden2tag = nn.Linear(args.hidden_dim, 2)
    self.hidden = self.init_hidden()

def init_hidden(self):
    return (autograd.Variable(torch.zeros(1,1, self.hidden_dim).cuda()),
        autograd.Variable(torch.zeros(1,1, self.hidden_dim).cuda()))

def forward(self,embeds):
    embeds = autograd.Variable(torch.from_numpy(embeds[0]).float().cuda())
    lstm_output, self.hidden = self.lstm(embeds.view(1, 1, -1), self.hidden)
    tag_space = self.hidden2tag(lstm_output.view(1, -1))
    scores = F.log_softmax(tag_space)
    return scores

And I pass the sentences in the form of embeddings as follows:

model = model.LSTM1(args).cuda()
criterion = nn.NLLLoss()
optimizer = optim.SGD(model.parameters(), lr=1e-5, momentum=0.9, weight_decay=1e-5)

optimizer.zero_grad()

for epoch in range(20):
    for i in range(len(sentences)):
        optimizer.zero_grad()
        model.hidden = model.init_hidden()
        target = prepare_targets(tag_phrase[i],tag_to_ix,1) #Gets a Variable(long tensor) for the target, single value (either 0 or 1)
        score = model(sentences[i]) #sentences[i] is the embedding of sentence i

        loss = criterion(score,target)
        loss.backward()
        optimizer.step()

My doubts:

So the embedding goes into the model, where in the forward function it gets converted to a Variablle(float tensor), and hence an appropriate input for the LSTM. This is my understanding of things. Is this correct?
I am backpropagating after every sentence, so does that make each sentence a separate batch? How do I split the sentences into batches and what changes do I need to make in the model then?
What is the most appropriate way to use pre-trained sentence embeddings in PyTorch?
If this code seems correct, then the problem I'm facing is that everything is getting classified into a single class out of the two. Any suggestions to resolve this error?

Thank you.

[EDIT] Improved Code

embeddings = torch.from_numpy(embeddings).float().cuda()
args.embed_num=embeddings.size(0)
args.embed_dim=embeddings.size(1)

seq = [i for i in range(13000)]
seq_tensor = torch.LongTensor(seq).cuda() #Index tensor corresponding to the embedding.

target = prepare_targets(tag_phrase[:13000],tag_to_ix,1)

train_data = torch.utils.data.TensorDataset(seq_tensor.cuda(),target.data.cuda())
trainloader = torch.utils.data.DataLoader(train_data, batch_size=100, shuffle=True)

model = model2.LSTM1(args,embeddings, 3).cuda()
criterion = nn.NLLLoss()
optimizer = optim.SGD(model.parameters(), lr=1e-4, momentum=0.9, weight_decay=1e-5)

for epoch in range(1000):
    avg_loss=0.
    for i, data in enumerate(trainloader,0):
        seq, target = data
        seq_var, target = autograd.Variable(seq.cuda()), autograd.Variable(target.cuda())
        correct=0
        optimizer.zero_grad()
        model.hidden=model.init_hidden()
        score=model(seq_var)
        loss = criterion(score,target)
        loss.backward()
        optimizer.step()
        epoch_lis.append(epoch)
        losses.append(loss.data[0])
        _,predicted = torch.max(score.data,1)
        correct += (predicted == target.data).sum()
        print i, correct

model:

class LSTM1(nn.Module):

    def __init__(self,args,embeddings, layers):
        super(LSTM1,self).__init__()
        self.num_layers=layers
        self.args = args
        self.hidden_dim = args.hidden_dim
        self.embed = nn.Embedding(args.embed_num, args.embed_dim)
        self.embed.weight = nn.Parameter(embeddings)
        self.lstm = nn.LSTM(args.embed_dim, args.hidden_dim, num_layers=self.num_layers)
        self.hidden2tag = nn.Linear(args.hidden_dim, 2)
        self.hidden = self.init_hidden()

    def init_hidden(self):
        return (autograd.Variable(torch.zeros(self.num_layers,1, self.hidden_dim).cuda()),
            autograd.Variable(torch.zeros(self.num_layers,1, self.hidden_dim).cuda()))

    def forward(self,sentence):
        embeds = self.embed(sentence)
        lstm_output, self.hidden = self.lstm(embeds.view(len(sentence), 1, -1), self.hidden)
        tag_space = self.hidden2tag(lstm_output.view(len(sentence), -1))
        scores = F.log_softmax(tag_space)
        return scores


title: How convert a list with None in it to torch.tensor()?

body:
For example I have a variable z = [1, 3, None, 5, 6]

I would like to do: torch.tensor(z)

and get something like the following:

torch.tensor([1,3, None, 5,6], dtype=torch.float)

Howhever, such attempt raises an error

  TypeError: must be real number, not NoneType

Is there a way to convert such list to a torch.tensor?

I don't want to impute this None value with something else. Numpy arrays are capable of converting such lists np.array([1, 3, None, 5, 6]), but I'd prefer not to convert back and forth variable either.


title: Is there a function in google.colab module to close the runtime

body:
Sometimes when I give run in google.colab I cant stay infront of the computer to manually disconnect from the server when the run is complete and the connection stays on even when my run is complete occupying the node for no reason.

Is there a function in google.colab so that say I can insert the function to close the connection after some epochs? I am looking for something like colab.disconnect() etc...


title: How does PyTorch module do the back prop

body: While following the instructions on extending PyTorch - adding a module, I noticed while extending Module, we don't really have to implement the backward function. The only thing we need is to apply the Function instance in the forward function and PyTorch can automatically call the backward one in the Function instance when doing the back prop. This seems like magic to me as we didn't even register the Function instance we used. I looked into the source code but didn't find anything related. Could anyone kindly point me out a place that all those actually happened?


title: AttributeError: 'Tensor' has no attribute: 'backwards'

body:
I'm posting this question for those who made the same error I did. I got this error when trying to compute my gradients:

criterion = torch.nn.CrossEntropyLoss()
loss = criterion(y_hat, y_truth)
loss.backwards()


title: pytorch DataLoader: `Tensors must have same number of dimensions`

body:
I am trying to fit an LSTM model in Pytorch. My data is too big to be read into memory and so I want to create mini-batches of data using the DataLoader function from Pytorch.

I have two features as input (X1, X2). I have one output feature (y). I am using 365 timesteps of X1 & X2 as features used to predict y.

The dimensions of my training array is:

(n_observations, n_timesteps, n_features) == (9498, 365, 2)

I don't understand why the code below isn't working because I have seen other examples where the X, y pairs have different numbers of dimensions (LSTM for runoff modelling, Pytorch's own docs )

Minimum Reproducible Example

import numpy as np
import torch
from torch.utils.data import DataLoader

train_x = torch.Tensor(np.random.random((9498, 365, 2)))
train_y = torch.Tensor(np.random.random((9498, 1)))
val_x = torch.Tensor(np.random.random((1097, 365, 2)))
val_y = torch.Tensor(np.random.random((1097, 1)))
test_x = torch.Tensor(np.random.random((639, 365, 2)))
test_y = torch.Tensor(np.random.random((639, 1)))

train_dataset = (train_x, train_y)
test_dataset = (test_x, test_y)
val_dataset = (val_x, val_y)

train_dataloader = DataLoader(train_dataset, batch_size=256)

iterator = train_dataloader.__iter__()
iterator.next()

Output:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-47-2a0b28b53c8f> in <module>
     13
     14 iterator = train_dataloader.__iter__()
---> 15 iterator.next()

/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py in __next__(self)
    344     def __next__(self):
    345         index = self._next_index()  # may raise StopIteration
--> 346         data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
    347         if self._pin_memory:
    348             data = _utils.pin_memory.pin_memory(data)

/opt/conda/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py in fetch(self, possibly_batched_index)
     45         else:
     46             data = self.dataset[possibly_batched_index]
---> 47         return self.collate_fn(data)

/opt/conda/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py in default_collate(batch)
     53             storage = elem.storage()._new_shared(numel)
     54             out = elem.new(storage)
---> 55         return torch.stack(batch, 0, out=out)
     56     elif elem_type.__module__ == 'numpy' and elem_type.__name__ != 'str_' \
     57             and elem_type.__name__ != 'string_':

RuntimeError: invalid argument 0: Tensors must have same number of dimensions: got 4 and 3 at /tmp/pip-req-build-4baxydiv/aten/src/TH/generic/THTensor.cpp:680


title: Using autograd.grad() as a parameter for a loss function (pytorch)

body:
I want to compute the gradient between two tensors in a net. The input X tensor is sent through a set of convolutional layers which give me back and output Y tensor.

I’m creating a new loss and I would like to know the MSE between gradient  of norm(Y) w.r.t. each element of X. Here the code:

# Staring tensors
X = torch.rand(40, requires_grad=True)
Y = torch.rand(40, requires_grad=True)

# Define loss
loss_fn = nn.MSELoss()

#Make some calculations
V = Y*X+2

# Compute the norm
V_norm = V.norm()

# Computing gradient to calculate the loss
for i in range(len(V)):
    if i == 0:
        grad_tensor = torch.autograd.grad(outputs=V_norm, inputs=X[i])
    else:
        grad_tensor_ = torch.autograd.grad(outputs=V_norm, inputs=X[i])
        grad_tensor = torch.cat((grad_tensor, grad_tensor_), dim=0)

# Grund truth
gt = grad_tensor * 0 + 1

#Loss
loss_g = loss_fn(grad_tensor, gt)
print(loss_g)

Unfortunately, I’ve been making tests with torch.autograd.grad(), but I could not figure out how to do it. I get teh following error: RuntimeError: One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior.

Setting allow_unused=True gives me back None which is not an option. Not sure how to compute the loss between the gradients and the norm. Any idea about how to code this loss?


title: Parallel way of applying function element-wise to a Pytorch CUDA Tensor

body: Suppose I have a torch CUDA tensor and I want to apply some function like sin() but I have explicitly defined the function F. How can I use parallel computation to apply F in Pytorch.


title: Difference between Keras' BatchNormalization and PyTorch's BatchNorm2d?

body:
I've a sample tiny CNN implemented in both Keras and PyTorch. When I print summary of both the networks, the total number of trainable parameters are same but total number of parameters and number of parameters for Batch Normalization don't match.

Here is the CNN implementation in Keras:

inputs = Input(shape = (64, 64, 1)). # Channel Last: (NHWC)

model = Conv2D(filters=32, kernel_size=(3, 3), padding='SAME', activation='relu', input_shape=(IMG_SIZE, IMG_SIZE, 1))(inputs)
model = BatchNormalization(momentum=0.15, axis=-1)(model)
model = Flatten()(model)

dense = Dense(100, activation = "relu")(model)
head_root = Dense(10, activation = 'softmax')(dense)

And the summary printed for above model is:

Model: "model_8"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
input_9 (InputLayer)         (None, 64, 64, 1)         0
_________________________________________________________________
conv2d_10 (Conv2D)           (None, 64, 64, 32)        320
_________________________________________________________________
batch_normalization_2 (Batch (None, 64, 64, 32)        128
_________________________________________________________________
flatten_3 (Flatten)          (None, 131072)            0
_________________________________________________________________
dense_11 (Dense)             (None, 100)               13107300
_________________________________________________________________
dense_12 (Dense)             (None, 10)                1010
=================================================================
Total params: 13,108,758
Trainable params: 13,108,694
Non-trainable params: 64
_________________________________________________________________

Here's the implementation of the same model architecture in PyTorch:

# Image format: Channel first (NCHW) in PyTorch
class CustomModel(nn.Module):
def __init__(self):
    super(CustomModel, self).__init__()
    self.layer1 = nn.Sequential(
        nn.Conv2d(in_channels=1, out_channels=32, kernel_size=(3, 3), padding=1),
        nn.ReLU(True),
        nn.BatchNorm2d(num_features=32),
    )
    self.flatten = nn.Flatten()
    self.fc1 = nn.Linear(in_features=131072, out_features=100)
    self.fc2 = nn.Linear(in_features=100, out_features=10)

def forward(self, x):
    output = self.layer1(x)
    output = self.flatten(output)
    output = self.fc1(output)
    output = self.fc2(output)
    return output

And following is the output of summary of the above model:

----------------------------------------------------------------
        Layer (type)               Output Shape         Param #
================================================================
            Conv2d-1           [-1, 32, 64, 64]             320
              ReLU-2           [-1, 32, 64, 64]               0
       BatchNorm2d-3           [-1, 32, 64, 64]              64
           Flatten-4               [-1, 131072]               0
            Linear-5                  [-1, 100]      13,107,300
            Linear-6                   [-1, 10]           1,010
================================================================
Total params: 13,108,694
Trainable params: 13,108,694
Non-trainable params: 0
----------------------------------------------------------------
Input size (MB): 0.02
Forward/backward pass size (MB): 4.00
Params size (MB): 50.01
Estimated Total Size (MB): 54.02
----------------------------------------------------------------

As you can see in above results, Batch Normalization in Keras has more number of parameters than PyTorch (2x to be exact). So what's the difference in above CNN architectures? If they are equivalent, then what am I missing here?


title: How to fix ' ImportError: cannot import name 'numpy_type_map' ' in Python?

body:
I've followed the instructions in Detectron and I've configured it several times: the code compiles as it should. When it comes to run the code, I get this error:

    Traceback (most recent call last):
    File "tools/train_net_step.py", line 21, in <module>
    import nn as mynn
    File "/home/federico/PycharmProjects/Detectron.pytorch/lib/nn/__init__.py", line 2, in <module>
    from .parallel import DataParallel
    File "/home/federico/PycharmProjects/Detectron.pytorch/lib/nn/parallel/__init__.py", line 3, in <module>
    from .data_parallel import DataParallel, data_parallel
    File "/home/federico/PycharmProjects/Detectron.pytorch/lib/nn/parallel/data_parallel.py", line 4, in <module>
    from .scatter_gather import scatter_kwargs, gather
    File "/home/federico/PycharmProjects/Detectron.pytorch/lib/nn/parallel/scatter_gather.py", line 8, in <module>
    from torch.utils.data.dataloader import numpy_type_map
    ImportError: cannot import name 'numpy_type_map'

I've also tried to google it many times, but I can't find a way to solve it. What can I do? I'm using PyTorch 0.4.1 and pytorch nightly 1.0.0-dev.

EDIT: Thanks to sancelot, I managed to solve that error (PyTorch 0.4.0 did the thing). Anyway, now I've got another error:

    Traceback (most recent call last):
    File "tools/train_net_step.py", line 27, in <module>
    from modeling.model_builder import Generalized_RCNN
    File "/home/federico/PycharmProjects/Detectron.pytorch/lib/modeling/model_builder.py", line 11, in <module>
    from model.roi_pooling.functions.roi_pool import RoIPoolFunction
    File "/home/federico/PycharmProjects/Detectron.pytorch/lib/model/roi_pooling/functions/roi_pool.py", line 3, in <module>
    from .._ext import roi_pooling
    File "/home/federico/PycharmProjects/Detectron.pytorch/lib/model/roi_pooling/_ext/roi_pooling/__init__.py", line 3, in <module>
    from ._roi_pooling import lib as _lib, ffi as _ffi
    ImportError: /home/federico/PycharmProjects/Detectron.pytorch/lib/model/roi_pooling/_ext/roi_pooling/_roi_pooling.so: undefined symbol: PyInt_FromLong

What I can't get this time is: is this an error given by an external library? I'm using an anaconda environment previously made by my professor, who has used it for Detectron... so I can't guess why I get this.


title: Get the value of '[UNK]' in BERT

body:
I have designed a model based on BERT to solve NER task. I am using transformers library with the "dccuchile/bert-base-spanish-wwm-cased" pre-trained model. The problem comes when my model detect an entity but the token is '[UNK]'. How could I know which is the string behind that token?

I know that an unknown token can't be reverted to the original one, but I would like to at least capture that values before passing the inputs to the model.

The code is really simple:

    sentenceIds = tokenizer.encode(sentence,add_special_tokens = True)

    inputs = pad_sequences([sentenceIds], maxlen=256, dtype="long",
                              value=0, truncating="post", padding="post")

    att_mask = torch.tensor([[int(token_id > 0) for token_id in inputs[0]]]).to(device)
    inputs = torch.tensor(inputs).to(device)

    with torch.no_grad():
        outputs = model(inputs,
                          token_type_ids=None,
                          attention_mask=att_mask)

As you see is really simple, just tokenize, padding or truncating, creating attentionMask and calling to the model.

I have tried using regex, trying to find the two tokens that are around it and things like that, but I can't solve it properly.


title: Compare Number of Equal Elements in Tensors

body: I have two tensors of dimension 1000 * 1. I want to check how many of the 1000 elements are equal in the two tensors. I think I should be able to do this in one line like Numpy but couldn't find a similar function.


title: Error importing PyTorch - Python

body:
I have tried to install PyTorch using the following command in terminal:

pip install http://download.pytorch.org/whl/torch-0.2.0.post1-cp27-none-macosx_10_7_x86_64.whl

I then run the following code in python:

import torch
torch.__file__

and get the following error:

 File "/Users/brian/anaconda/lib/python2.7/site-packages/torch/__init__.py", line 48, in <module>
import torch._dl as _dl_flags

AttributeError: 'module' object has no attribute '_dl'

I have attempted to research the issue but cannot find a clear answer. any help would be appreciated.

EDIT:
These are the results from trying to re-install PyTorch using the following commands:

pip install http://download.pytorch.org/whl/torch-0.2.0.post1-cp27-none-macosx_10_7_x86_64.whl
pip install torchvision

Results:

Requirement already satisfied: pyyaml in ./anaconda/lib/python2.7/site-packages (from torch==0.2.0.post1)
Requirement already satisfied: numpy in ./anaconda/lib/python2.7/site-packages (from torch==0.2.0.post1)

Requirement already satisfied: torchvision in ./anaconda/lib/python2.7/site-packages
Requirement already satisfied: pillow in ./anaconda/lib/python2.7/site-packages (from torchvision)
Requirement already satisfied: torch in ./anaconda/lib/python2.7/site-packages (from torchvision)
Requirement already satisfied: numpy in ./anaconda/lib/python2.7/site-packages (from torchvision)
Requirement already satisfied: six in ./anaconda/lib/python2.7/site-packages (from torchvision)
Requirement already satisfied: olefile in ./anaconda/lib/python2.7/site-packages (from pillow->torchvision)
Requirement already satisfied: pyyaml in ./anaconda/lib/python2.7/site-packages (from torch->torchvision)

EDIT II

The following command:

conda install pytorch torchvision cuda80 -c soumith

yields this result:

Fetching package metadata ...........

PackageNotFoundError: Packages missing in current channels:

   - cuda80

We have searched for the packages in the following channels:

  - https://conda.anaconda.org/soumith/osx-64
  - https://conda.anaconda.org/soumith/noarch
  - https://repo.continuum.io/pkgs/free/osx-64
  - https://repo.continuum.io/pkgs/free/noarch
  - https://repo.continuum.io/pkgs/r/osx-64
  - https://repo.continuum.io/pkgs/r/noarch
  - https://repo.continuum.io/pkgs/pro/osx-64
  - https://repo.continuum.io/pkgs/pro/noarch


title: Pytorch schedule learning rate

body:
I am trying to re-implement one paper, which suggests to adjust the learning rate as below:

The learning rate is decreased by a factor of the regression value with patience epochs 10 on the change value of 0.0001.

Should I use the torch.optim.lr_scheduler.ReduceLROnPlateau()?
I am not sure what value should I pass to each parameter.

Is the change value in the statement denotes to the parameter threshold?

Is the factor in the statement denotes to the parameter factor?


title: PyTorch: Image dimension issue

body:
I am working on an image classifier dataset. There are 31 classes in my dataset and there is a folder for each class. For training, I am loading data in the following manner:

from torchvision import datasets, transforms
import torch

def load_training(root_path, dir, batch_size, kwargs):
    transform = transforms.Compose(
        [transforms.Resize([256, 256]),
         transforms.RandomCrop(224),
         transforms.RandomHorizontalFlip(),
         transforms.ToTensor()])
    data = datasets.ImageFolder(root=root_path + dir, transform=transform)
    train_loader = torch.utils.data.DataLoader(data, batch_size=batch_size, shuffle=False, drop_last=True, **kwargs)
    return train_loader

Now for a batch size 32, each batch dim are : [32,3,224,224]. I know Pytorch uses PIL for loading images. But when I'm trying to load images from a list named images of size 32 using PIL using the following code snippet:

for i in images:
    im = Image.open(root_dir + dir + i)
    im = im.convert('RGB')
    im.load()
    im = im.resize((224, 224))
    im = np.asarray( im, dtype="int32" )
    print(im.shape)
    temp_im_2 = torch.from_numpy(np.array(temp_im)).float()

It says im.shape is (32,224,224,3). What am I doing wrong? What should I do to get the batches to be of the same shape in both cases say [32,3,224,224]?


title: TransformerEncoder with a padding mask

body:
I'm trying to implement torch.nn.TransformerEncoder with a src_key_padding_mask not equal to none. Imagine the input is of the shape src = [20, 95] and the binary padding mask has the shape src_mask = [20, 95], 1 in the position of padded tokens and 0 for other positions. I make a transformer encoder with 8 layers, each of which contain an attention with 8 heads and hidden dimension 256:

layer=torch.nn.TransformerEncoderLayer(256, 8, 256, 0.1)
encoder=torch.nn.TransformerEncoder(layer, 6)
embed=torch.nn.Embedding(80000, 256)
src=torch.randint(0, 1000, (20, 95))
src = emb(src)
src_mask = torch.randint(0,2,(20, 95))
output =  encoder(src, src_mask)

But I get the following error:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-107-31bf7ab8384b> in <module>
----> 1 output =  encoder(src, src_mask)

~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    545             result = self._slow_forward(*input, **kwargs)
    546         else:
--> 547             result = self.forward(*input, **kwargs)
    548         for hook in self._forward_hooks.values():
    549             hook_result = hook(self, input, result)

~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/transformer.py in forward(self, src, mask, src_key_padding_mask)
    165         for i in range(self.num_layers):
    166             output = self.layers[i](output, src_mask=mask,
--> 167                                     src_key_padding_mask=src_key_padding_mask)
    168
    169         if self.norm:

~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    545             result = self._slow_forward(*input, **kwargs)
    546         else:
--> 547             result = self.forward(*input, **kwargs)
    548         for hook in self._forward_hooks.values():
    549             hook_result = hook(self, input, result)

~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/transformer.py in forward(self, src, src_mask, src_key_padding_mask)
    264         """
    265         src2 = self.self_attn(src, src, src, attn_mask=src_mask,
--> 266                               key_padding_mask=src_key_padding_mask)[0]
    267         src = src + self.dropout1(src2)
    268         src = self.norm1(src)

~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    545             result = self._slow_forward(*input, **kwargs)
    546         else:
--> 547             result = self.forward(*input, **kwargs)
    548         for hook in self._forward_hooks.values():
    549             hook_result = hook(self, input, result)

~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/activation.py in forward(self, query, key, value, key_padding_mask, need_weights, attn_mask)
    781                 training=self.training,
    782                 key_padding_mask=key_padding_mask, need_weights=need_weights,
--> 783                 attn_mask=attn_mask)
    784
    785

~/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py in multi_head_attention_forward(query, key, value, embed_dim_to_check, num_heads, in_proj_weight, in_proj_bias, bias_k, bias_v, add_zero_attn, dropout_p, out_proj_weight, out_proj_bias, training, key_padding_mask, need_weights, attn_mask, use_separate_proj_weight, q_proj_weight, k_proj_weight, v_proj_weight, static_k, static_v)
   3250     if attn_mask is not None:
   3251         attn_mask = attn_mask.unsqueeze(0)
-> 3252         attn_output_weights += attn_mask
   3253
   3254     if key_padding_mask is not None:

RuntimeError: The size of tensor a (20) must match the size of tensor b (95) at non-singleton dimension 2

I was wondering if somebody could help me figure out this problem.

Thanks


I am trying to develop a gesture recognition system using Pytorch deep learning frame work. I get this error.

data = torch.cat(imgs,dim=1)
RuntimeError: expected a non-empty list of Tensors


title: DataLoader Class Errors Pytorch

body:
I am beginner pytorch user, and I am trying to use dataloader.

Actually, I am trying to implement this into my network but it takes a very long time to load. And so, I debugged my network to see if the network itself has the problem, but it turns out it has something to with my dataloader class. Here is the code:

 from torch.utils.data import Dataset, DataLoader
 import numpy as np
 import pandas as pd

class DiabetesDataset(Dataset):

  def __init__(self, csv):
      self.xy = pd.read_csv(csv)

  def __len__(self):
      return len(self.xy)

  def __getitem__(self, index):
       self.x_data = torch.Tensor(xy.iloc[:, 0:-1].values)
       self.y_data = torch.Tensor(xy.iloc[:, [-1]].values)
       return self.x_data[index], self.y_data[index]

 dataset = DiabetesDataset("trial.csv")
 train_loader = DataLoader(dataset=dataset,
                      batch_size=1,
                      shuffle=True,
                      num_workers=2)`

 for a in train_loader:
    print(a)

To verify that the dataloader causes all the delay, I created a dummy csv file with 2 columns of 1s and 2s, for a total of 10 samples for each columns. Then, I looped over the train_loader object, it has been more than 1 hr and it is still running, considering that the sample size is small and batch size is set to 1.

I am not sure as to what the error to my code is and it is causing this issue.

Any comments/inputs are greatly appreciated!


title: AssersionError: Torch not compiled with CUDA enabled

body:
I want to run this repo. I installed everything that is needed for this project.
I have Windows 8.1 operating system, seems that I don't have NVIDIA GPU (from Device Manager: Display adapters - AMD Radeon HD 7660G + 7670M Dual Graphics and AMD Radeon HD 7670M).
I installed torch with command that is presented on Pytorch web-site
pip install torch==1.6.0+cpu torchvision==0.7.0+cpu -f https://download.pytorch.org/whl/torch_stable.html

But when I run the project I receive error - AssersionError: Tourch not compiled with CUDA enabled.
Then I tried to install torch with CUDA enabled.
pip install torch===1.6.0 torchvision===0.7.0 -f https://download.pytorch.org/whl/torch_stable.html

But when I run the project I receive error - AssersionError: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from https://www.nvidia.com/Download/index.aspx.
Please, help me to solve my issue and run the project without errors.


title: How do you use PyTorch PackedSequence in code?

body:
Can someone give a full working code (not a snippet, but something that runs on a variable-length recurrent neural network) on how would you use the PackedSequence method in PyTorch?

There do not seem to be any examples of this in the documentation, github, or the internet.

https://github.com/pytorch/pytorch/releases/tag/v0.1.10


I am trying to train my neuronal network. Train in the model is correct, but I can't calculate loss. The output and the target have the same dimension.

I had tried to use torch.stack, but I can't because the size of the each input is (252, x) where x is the same in the 252 elements, but is different for the others inputs.

I use a custom Dataset:

class MusicDataSet(Dataset):
def __init__(self, transform=None):
    self.ms, self.target, self.tam = sd.cargarDatos()
    self.mean, self.std = self.NormalizationValues()
def __len__(self):
    return self.tam

def __getitem__(self, idx):
    #Normalize
    inp = (self.ms[idx]-self.mean)/self.std
    inp = torch.from_numpy(inp).float()
    inp = inp.t()
    inp = inp.to('cuda')

    target= torch.from_numpy(self.target[idx])
    target = target.long()
    target = target.t()
    target = target.to('cuda')

    return inp, target

I must say that list can't be cast with something like: target = torch.Tensor() or torch.stack() because this (252, x), as I have already said.

def music_collate_fn(batch):
    data = [item[0] for item in batch]
    data = pad_sequence(data, batch_first=True)
    target = [item[0] for item in batch]
    target = pad_sequence(target, batch_first=True)
    return data, target

musicSet = mds.MusicDataSet()
train_loader = torch.utils.data.DataLoader(musicSet,batch_size=50, collate_fn = music_collate_fn, shuffle=False)

input_dim = 252
hidden_dim = (512,1024,512)
output_dim = 88
mlp = rn.MLP(input_dim, hidden_dim, output_dim).to(device)

optimizer = torch.optim.RMSprop(mlp.parameters(), lr = learning_rate)
criterion = nn.CrossEntropyLoss()
for batch_idx, (x,y) in enumerate(train_loader):
    outputs = mlp(x.to(device))
    loss = criterion(outputs, y)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

The size of output and target is the same,

output: torch.Size([50, 288, 88])
target:  torch.Size([50, 288, 88])

But the next error apears when I try to calculate loss:

  File "<ipython-input-205-3c47d7aa11a4>", line 32, in <module>
    loss = criterion(outputs, y)

  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)

  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\loss.py", line 904, in forward
    ignore_index=self.ignore_index, reduction=self.reduction)

  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\functional.py", line 1970, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)

  File "C:\ProgramData\Anaconda3\lib\site-packages\torch
\nn\functional.py", line 1800, in nll_loss
    out_size, target.size()))

ValueError: Expected target size (50, 88), got torch.Size([50, 288, 88])


title: How do they know mean and std, the input value of transforms.Normalize

body:
The question is about the data loading tutorial from the PyTorch website. I don't know how they write the value of mean_pix and std_pix of the in transforms.Normalize without calculation

I'm unable to find any explanation relevant to this question on StackOverflow.

import torch
from torchvision import transforms, datasets

data_transform = transforms.Compose([
        transforms.RandomSizedCrop(224),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize(mean=[0.485, 0.456, 0.406],
                             std=[0.229, 0.224, 0.225])
    ])
hymenoptera_dataset = datasets.ImageFolder(root='hymenoptera_data/train',
                                           transform=data_transform)
dataset_loader = torch.utils.data.DataLoader(hymenoptera_dataset,
                                             batch_size=4, shuffle=True,
                                             num_workers=4)

The value mean=[0.485,0.456, 0.406] and std=[0.229, 0.224, 0.225] is not obvious to me. How do they get them? And why are they equal to these?


title: torch.utils.data.random_split() is not splitting the data

body:
I’m not getting split when I use torch.utils.data.random_split.

I get correct numbers for train_size and val_size, but when I do random_split, both train_data and val_data get full_data. There is no split happening.

Please help me with this issue.

class DeviceLoader(Dataset):

def __init__(self, root_dir, train=True, transform=None):
    self.file_path = root_dir
    self.train = train
    self.transform = transform
    self.file_names = ['%s/%s'%(root,file) for root,_,files in os.walk(root_dir) for file in files]
    self.len = len(self.file_names)
    self.labels = {'BP_Raw_Images':0, 'DT_Raw_Images':1, 'GL_Raw_Images':2, 'PO_Raw_Images':3, 'WS_Raw_Images':4}

def __len__(self):
    return(len(self.file_names))

def __getitem__(self, idx):
    file_name = self.file_names[idx]
    device = file_name.split('/')[5]
    img = self.pil_loader(file_name)
    if(self.transform):
        img = self.transform(img)
    cat = self.labels[device]
    if(self.train):
        return(img, cat)
    else:
        return(img, file_name)
full_data = DeviceLoader(root_dir=’/kaggle/input/devices/dataset/’, transform=transforms, train=True)
train_size = int(0.7*len(full_data))
val_size = len(full_data) - train_size
train_data, val_data = torch.utils.data.random_split(full_data,[train_size,val_size])

Expected result is to split the full_data into train_data(2000) and val_data(500). But instead, I get full_data(2500) in both train and val.


title: Pytorch Autograd: what does runtime error "grad can be implicitly created only for scalar outputs" mean

body:
I am trying to understand Pytorch autograd in depth; I would like to observe the gradient of a simple tensor after going through a sigmoid function as below:

import torch
from torch import autograd

D = torch.arange(-8, 8, 0.1, requires_grad=True)

with autograd.set_grad_enabled(True):
    S = D.sigmoid()
S.backward()

My goal is to get D.grad() but even before calling it I get the runtime error:

RuntimeError: grad can be implicitly created only for scalar outputs

I see another post with similar question but the answer over there is not applied to my question.  Thanks


title: Pytorch vs. Keras: Pytorch model overfits heavily

body:
For several days now, I'm trying to replicate my keras training results with pytorch. Whatever I do, the pytorch model will overfit far earlier and stronger to the validation set then in keras. For pytorch I use the same XCeption Code from https://github.com/Cadene/pretrained-models.pytorch.

The dataloading, the augmentation, the validation, the training schedule etc. are equivalent. Am I missing something obvious? There must be a general problem somewhere. I tried thousands of different module constellations, but nothing seems to come even close to the keras training. Can somebody help?

Keras model: val accuracy > 90%

# base model
base_model = applications.Xception(weights='imagenet', include_top=False, input_shape=(img_width, img_height, 3))

# top model
x = base_model.output
x = GlobalMaxPooling2D()(x)
x = Dense(512, activation='relu')(x)
x = Dropout(0.5)(x)
predictions = Dense(4, activation='softmax')(x)

# this is the model we will train
model = Model(inputs=base_model.input, outputs=predictions)

# Compile model
from keras import optimizers
adam = optimizers.Adam(lr=0.0001)
model.compile(loss='categorical_crossentropy',
optimizer=adam, metrics=['accuracy'])

# LROnPlateau etc. with equivalent settings as pytorch

Pytorch model: val accuracy ~81%

from xception import xception
import torch.nn.functional as F

# modified from https://github.com/Cadene/pretrained-models.pytorch
class XCeption(nn.Module):
    def __init__(self, num_classes):
        super(XCeption, self).__init__()

        original_model = xception(pretrained="imagenet")

        self.features=nn.Sequential(*list(original_model.children())[:-1])
        self.last_linear = nn.Sequential(
             nn.Linear(original_model.last_linear.in_features, 512),
             nn.ReLU(),
             nn.Dropout(p=0.5),
             nn.Linear(512, num_classes)
        )

    def logits(self, features):
        x = F.relu(features)
        x = F.adaptive_max_pool2d(x, (1, 1))
        x = x.view(x.size(0), -1)
        x = self.last_linear(x)
        return x

    def forward(self, input):
        x = self.features(input)
        x = self.logits(x)
        return x

device = torch.device("cuda")
model=XCeption(len(class_names))
if torch.cuda.device_count() > 1:
    print("Let's use", torch.cuda.device_count(), "GPUs!")
    # dim = 0 [30, xxx] -> [10, ...], [10, ...], [10, ...] on 3 GPUs
    model = nn.DataParallel(model)
model.to(device)

criterion = nn.CrossEntropyLoss(size_average=False)
optimizer = optim.Adam(model.parameters(), lr=0.0001)
scheduler = lr_scheduler.ReduceLROnPlateau(optimizer, 'min', factor=0.2, patience=5, cooldown=5)

Thank you very much!

Update:
Settings:

criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=lr)
scheduler = lr_scheduler.ReduceLROnPlateau(optimizer, 'min', factor=0.2, patience=5, cooldown=5)

model = train_model(model, train_loader, val_loader,
                        criterion, optimizer, scheduler,
                        batch_size, trainmult=8, valmult=10,
                        num_epochs=200, epochs_top=0)

Cleaned training function:

def train_model(model, train_loader, val_loader, criterion, optimizer, scheduler, batch_size, trainmult=1, valmult=1, num_epochs=None, epochs_top=0):
  for epoch in range(num_epochs):
    for phase in ['train', 'val']:
        running_loss = 0.0
        running_acc = 0
        total = 0
        # Iterate over data.
        if phase=="train":
            model.train(True)  # Set model to training mode
            for i in range(trainmult):
                for data in train_loader:
                    # get the inputs
                    inputs, labels = data
                    inputs, labels = inputs.to(torch.device("cuda")), labels.to(torch.device("cuda"))
                    # zero the parameter gradients
                    optimizer.zero_grad()
                    # forward
                    outputs = model(inputs) # notinception
                    _, preds = torch.max(outputs, 1)
                    loss = criterion(outputs, labels)
                    # backward + optimize only if in training phase
                    loss.backward()
                    optimizer.step()
                    # statistics
                    total += labels.size(0)
                    running_loss += loss.item()*labels.size(0)
                    running_acc += torch.sum(preds == labels)
                    train_loss=(running_loss/total)
                    train_acc=(running_acc.double()/total)
        else:
            model.train(False)  # Set model to evaluate mode
            with torch.no_grad():
                for i in range(valmult):
                    for data in val_loader:
                        # get the inputs
                        inputs, labels = data
                        inputs, labels = inputs.to(torch.device("cuda")), labels.to(torch.device("cuda"))
                        # zero the parameter gradients
                        optimizer.zero_grad()
                        # forward
                        outputs = model(inputs)
                        _, preds = torch.max(outputs, 1)
                        loss = criterion(outputs, labels.data)
                        # statistics
                        total += labels.size(0)
                        running_loss += loss.item()*labels.size(0)
                        running_acc += torch.sum(preds == labels)
                        val_loss=(running_loss/total)
                        val_acc=(running_acc.double()/total)
            scheduler.step(val_loss)
    return model


title: pytorch: get number of classes given an ImageFolder dataset

body:
If I have a dataset like:

image_datasets['train'] = datasets.ImageFolder(train_dir, transform=train_transforms)

How do I determine programatically the number of classes or unique labels in the dataset?


title: Order of CUDA devices

body:
I saw this solution, but it doesn't quite answer my question; it's also quite old so I'm not sure how relevant it is.

I keep getting conflicting outputs for the order of GPU units. There are two of them: Tesla K40 and NVS315 (legacy device that is never used). When I run deviceQuery, I get

Device 0: "Tesla K40m"
...
Device PCI Domain ID / Bus ID / location ID:   0 / 4 / 0

Device 1: "NVS 315"
...
Device PCI Domain ID / Bus ID / location ID:   0 / 3 / 0

On the other hand, nvidia-smi produces a different order:

 0  NVS 315

 1  Tesla K40m

Which I find very confusing. The solution I found for Tensorflow (and a similar one for Pytorch) is to use

import os
os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID"
os.environ["CUDA_VISIBLE_DEVICES"]="0"

PCI Bus ID is 4 for Tesla and 3 for NVS, so it should set it to 3 (NVS), is that right?

In pytorch I set

os.environ['CUDA_VISIBLE_DEVICES']='0'
...
device = torch.cuda.device(0)
print torch.cuda.get_device_name(0)

to get Tesla K40m

when I set instead

os.environ['CUDA_VISIBLE_DEVICES']='1'
device = torch.cuda.device(1)
print torch.cuda.get_device_name(0)

to get

UserWarning:
    Found GPU0 NVS 315 which is of cuda capability 2.1.
    PyTorch no longer supports this GPU because it is too old.

  warnings.warn(old_gpu_warn % (d, name, major, capability[1]))
NVS 315

So I'm quite confused: what's the true order of GPU devices that tf and pytorch use?


title: Getting error while installing Flair (NLP Library)

body:
I am trying to install NLP library "Flair" using pip and getting the error message:

ERROR: Could not find a version that satisfies the requirement torch>=1.0.0 (from flair) (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)
ERROR: No matching distribution found for torch>=1.0.0 (from flair)"

What should I do

I am using python3.6 and tried to install flair using cmd and pip in virtualenv. But there is still error messages.

(env) C:\Users\HP>pip install flair

I want the installation without any issue.enter image description here


title: Pytorch Categorical Cross Entropy loss function behaviour

body:
I have question regarding the computation made by the Categorical Cross Entropy Loss from Pytorch.
I have made this easy code snippet and because I use the argmax of the output tensor as the targets, I cannot understand why the loss is still high.

import torch
import torch.nn as nn
ce_loss = nn.CrossEntropyLoss()
output = torch.randn(3, 5, requires_grad=True)
targets = torch.argmax(output, dim=1)
loss = ce_loss(outputs, targets)
print(loss)

Thanks for the help understanding it.
Best regards
Jerome


title: grad_outputs in torch.autograd.grad (CrossEntropyLoss)

body:
I’m trying to get d(loss)/d(input). I know I have 2 options.

First option:

    loss.backward()
    dlossdx = x.grad.data

Second option:

    # criterion = nn.CrossEntropyLoss(reduce=False)
    # loss = criterion(y_hat, labels)
    # No need to call backward.
    dlossdx = torch.autograd.grad(outputs = loss,
                                  inputs = x,
                                  grad_outputs = ? )

My question is: if I use cross-entropy loss, what should I pass as grad_outputs in the second option?

Do I put d(CE)/d(y_hat)? Since pytorch crossentropy contains softmax, this will require me to pre-calculate softmax derivative using Kronecker delta.

Or do I put d(CE)/d(CE) which is torch.ones_like?

A conceptual answer is fine.


I'm new to pytorch. I'm trying to  implement a custom loss function by computing the absolute and relative distance and concatenating them.

def distance(p1, p2,labels):
        """
        Returns the distance between the point sets p1 and p2
        p1 = m by d matrix containing a set of points
        p2 = m by d matrix containing a different set of points

        returns: an m-length vector containing the  distance from each point in
            p1 to the corresponding point in p2
        """
        if not np.all(p1.shape == p2.shape):
            raise ValueError("p1 and p2 must be the same shape.")
        d = p1.shape[1]

        features = np.zeros(dtype=np.float32, shape=(p1.shape[0], d * 2))
        features[:, :d] = np.abs(p1 - p2)
        features[:, d:] = (p1 + p2) / 2

        return  features

The problem appears when I try to run it and  I get the following error:

File "... line 34, in distance

features[:, :d] = np.abs(p1 - p2)

File "...\Anaconda2\envs\tensorflow_gpuenv\lib\site-packages\tensorflow\python\util\dispatch.py", line 180, in wrapper
    return target(*args, **kwargs)

File "...\lib\site-packages\tensorflow\python\ops\math_ops.py", line 266, in abs
    x = ops.convert_to_tensor(x, name="x")

File "...\lib\site-packages\tensorflow\python\framework\ops.py", line 1087, in convert_to_tensor
    return convert_to_tensor_v2(value, dtype, preferred_dtype, name)

RuntimeError: Can't call numpy() on Variable that requires grad. Use var.detach().numpy() instead.


title: How to fix 'cannot initialize type TensorProto DataType' error while importing torch?

body:
I installed pytorch using pip3 command for my windows pc without GPU support.
But when I tried to import torch it is giving an error.

At first, there was a different error saying numpy version not matching and I updated the numpy to the latest version.

import torch
RuntimeError                              Traceback (most recent call last)
<ipython-input-10-c031d3dd82fc> in <module>()
----> 1 import torch

C:\Users\iamuraptha\Anaconda3\lib\site-packages\torch\__init__.py in <module>()
     82     pass
     83
---> 84 from torch._C import *
     85
     86 __all__ += [name for name in dir(_C)

RuntimeError: generic_type: cannot initialize type "TensorProtoDataType": an object with that name is already defined


title: OSError caffe2_detectron_ops_gpu PyTorch

body:
I keep getting this error when i try to run my code with PyTorch

OSError: [WinError 126] The specified module could not be found. Error loading “C:\Users\Username\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\lib\caffe2_detectron_ops_gpu.dll” or one of its dependencies.

I tried updating form cuda 10.1 to 10.2 but it didnt fix the issue. i also have cudnn installed . I uninstalled and reinstalled torch multiple times with the commands found on the get started page of PyTorch. Can someone help me resolve this i’ve being stuck for 2 days now


title: PyTorch: Variable data has to be a tensor, but got int

body:
I'm new to PyTorch and I'm doing some NLP works. I tried to wrap a Variable for LongTensor, but it shows that:

Variable data has to be a tensor, but got int

My code looks like this:

def trainIters(encoder, decoder, n_iters, abstract, article, print_every=1000, plot_every=100, learning_rate=0.01):
start = time.time()
plot_losses = []
print_loss_total = 0  # Reset every print_every
plot_loss_total = 0  # Reset every plot_every

encoder_optimizer = optim.SGD(encoder.parameters(), lr=learning_rate)
decoder_optimizer = optim.SGD(decoder.parameters(), lr=learning_rate)

print(abstract)

abstract = Variable(abstract)
article = Variable(article)
criterion = nn.NLLLoss()

for iter in range(1, n_iters + 1):
    input_variable = article[iter - 1]
    target_variable = abstract[iter - 1]

    loss = train(input_variable, target_variable, encoder, decoder, encoder_optimizer, decoder_optimizer, criterion)
    print_loss_total += loss
    plot_loss_total += loss

    if iter % print_every == 0:
        print_loss_avg = print_loss_total / print_every
        print_loss_total = 0
        print('%s (%d %d%%) %.4f' % (timeSince(start, iter / n_iters),
                                     iter, iter / n_iters * 100, print_loss_avg))

    if iter % plot_every == 0:
        plot_loss_avg = plot_loss_total / plot_every
        plot_losses.append(plot_loss_avg)
        plot_loss_total = 0

showPlot(plot_losses)

if __name__ == '__main__':
    hidden_size = 1024
    data = Data()
    vocab = data.load_dict('/users/cheng/NLP/Data/cnn/abstracts/train_set', '/users/cheng/NLP/Data/cnn/articles/train_set')
    abs_train, art_train = data.load_matrix('/users/cheng/NLP/Data/cnn/abstracts/train_set',
                                     '/users/cheng/NLP/Data/cnn/articles/train_set')
    encoder1 = model.EncoderRNN(len(vocab.idx2word), hidden_size)
    decoder1 = model.DecoderRNN(hidden_size, len(vocab.idx2word))

    if torch.cuda.is_available():
        encoder1 = encoder1.cuda()
        decoder1 = decoder1.cuda()

    trainIters(encoder1, decoder1, abs_train, art_train, 1000, print_every=100)

The error happens at "abstract = Variable(abstract)". I printed it and it looks like this:

2.0000e+00  2.6000e+01  4.0000e+00  ...   0.0000e+00  0.0000e+00  0.0000e+00
2.0000e+00  5.1000e+01  4.9000e+01  ...   0.0000e+00  0.0000e+00  0.0000e+00
2.0000e+00  5.1000e+01  4.9000e+01  ...   0.0000e+00  0.0000e+00  0.0000e+00
            ...                   ⋱                   ...
2.0000e+00  4.5500e+02  5.1000e+01  ...   0.0000e+00  0.0000e+00  0.0000e+00
2.0000e+00  1.3560e+03  5.1000e+01  ...   0.0000e+00  0.0000e+00  0.0000e+00
2.0000e+00  2.0100e+02  1.4000e+01  ...   0.0000e+00  0.0000e+00  0.0000e+00
[torch.LongTensor of size 55547x2552]

I tried to wrap a LongTensor in the interactive programming mode and it works fine, but I don't know why this is wrong.


title: sparse tensors do not have strides

body:
Using code from https://discuss.pytorch.org/t/how-to-convert-a-dense-matrix-to-a-sparse-one/7809 :

def to_sparse(x):
    """ converts dense tensor x to sparse format """
    x_typename = torch.typename(x).split('.')[-1]
    sparse_tensortype = getattr(torch.sparse, x_typename)

    indices = torch.nonzero(x)
    if len(indices.shape) == 0:  # if all elements are zeros
        return sparse_tensortype(*x.shape)
    indices = indices.t()
    values = x[tuple(indices[i] for i in range(indices.shape[0]))]
    return sparse_tensortype(indices, values, x.size())

I'm attempting to train a simple model using sparse tensors. This is a simple example where I aim to test if training using sparse tensors will increase the training time.

Here I convert the training data x to a sparse tensor :

x = to_sparse(torch.tensor([x1,x2,x3,x4]).float())

Upon training this model I receive error :

~/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/utils/data/dataset.py in <genexpr>(.0)
     38
     39     def __getitem__(self, index):
---> 40         return tuple(tensor[index] for tensor in self.tensors)
     41
     42     def __len__(self):

RuntimeError: sparse tensors do not have strides

Is this the correct method to convert a tensor to its sparse tensor equivalent ? Do I need to add strides to the sparse tensor data structure in order for it to be valid ?

Complete code below :

%reset -f

import torch
import torch.nn as nn
import torchvision
import torchvision.transforms as transforms
import numpy as np
import matplotlib.pyplot as plt
import torch.utils.data as data_utils
import torch.nn as nn
import torch.nn.functional as F

num_epochs = 20

x1 = np.array([0,0])
x2 = np.array([0,1])
x3 = np.array([1,0])
x4 = np.array([1,1])

num_epochs = 500

def to_sparse(x):
    """ converts dense tensor x to sparse format """
    x_typename = torch.typename(x).split('.')[-1]
    sparse_tensortype = getattr(torch.sparse, x_typename)

    indices = torch.nonzero(x)
    if len(indices.shape) == 0:  # if all elements are zeros
        return sparse_tensortype(*x.shape)
    indices = indices.t()
    values = x[tuple(indices[i] for i in range(indices.shape[0]))]
    return sparse_tensortype(indices, values, x.size())

x = to_sparse(torch.tensor([x1,x2,x3,x4]).float())

y = torch.tensor([0,1,1,0]).long()

train = data_utils.TensorDataset(x,y)
train_loader = data_utils.DataLoader(train , batch_size=2 , shuffle=True)

device = 'cpu'

input_size = 2
hidden_size = 100
num_classes = 2

learning_rate = .0001

torch.manual_seed(24)

def weights_init(m):
    m.weight.data.normal_(0.0, 1)

class NeuralNet(nn.Module) :
    def __init__(self, input_size, hidden_size, num_classes) :
        super(NeuralNet, self).__init__()
        self.fc1 = nn.Linear(input_size , hidden_size)
        self.relu = nn.ReLU()
        self.fc2 = nn.Linear(hidden_size , num_classes)

    def forward(self, x) :
        out = self.fc1(x)
        out = self.relu(out)
        out = self.fc2(out)
        return out

model = NeuralNet(input_size, hidden_size, num_classes).to(device)
# model.apply(weights_init)

criterionCE = nn.CrossEntropyLoss()
criterionNLL = nn.NLLLoss()

optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

for i in range(0 , 1) :

        total_step = len(train_loader)
        for epoch in range(num_epochs) :
            for i,(images , labels) in enumerate(train_loader) :
                images = images.to(device)
                labels = labels.to(device)

                outputs = model(images)
                loss = criterionCE(outputs , labels)

                optimizer.zero_grad()
                loss.backward()
                optimizer.step()

        outputs = model(x)

        print(outputs.data.max(1)[1])


title: How can i solve backward() got an unexpected keyword argument 'retain_variables'?

body:
I write the code following below but I got this error:

TypeError: backward() got an unexpected keyword argument 'retain_variables'

My code is:

def learn(self, batch_state, batch_next_state, batch_reward, batch_action):
    outputs = self.model(batch_state).gather(1, batch_action.unsqueeze(1)).squeeze(1)
    next_outputs = self.model(batch_next_state).detach().max(1)[0]
    target = self.gamma*next_outputs + batch_reward
    td_loss = F.smooth_l1_loss(outputs, target)
    self.optimizer.zero_grad()
    td_loss.backward(retain_variables = True)
    self.optimizer.step()


title: Is GEMM or BLAS used in Tensorflow, Theano, Pytorch

body:
I know that Caffe uses GEneral Matrix to Matrix Multiplication (GEMM) which is part of Basic Linear Algebra Subprograms (BLAS) library for performing convolution operations. Where a convolution is converted to matrix multiplication operation. I have referred below article. https://petewarden.com/2015/04/20/why-gemm-is-at-the-heart-of-deep-learning/

I want to understand how other deep learning frameworks like Theano, Tensorflow, Pytorch perform convolution operations. Do they use similar libraries in the backend. There might be some articles present on this topic. If someone can point me to those or can explain with an answer.

PS: I posted the same question on datascience.stackexchange.com. As I didn't get a reply there, I am posting it here as well. If there is a better forum to post this question please let me know.


title: Can't install torch on linux box using pip

body:
As the title states I am trying to install torch on linux using pip.
I run the command pip install torch==0.3.1

And I get the following output:

Collecting torch==0.3.1
  Could not find a version that satisfies the requirement torch==0.3.1 (from versions: 0.1.2, 0.1.2.post1)
No matching distribution found for torch==0.3.1

Any ideas what the issue might be?


title: How can I reduce a tensor's last dimension in PyTorch?

body:
I have tensor of shape (1, 3, 256, 256, 3). I need to reduce one of the dimensions to obtain the shape (1, 3, 256, 256). How can I do it?

Thanks!


title: TypeError: view() takes at most 2 arguments (3 given)

body:
I try to use view() in pytorch but i can't input 3 arguments.I don't know why it keep giving this error? Can anyone help me with this?

    def forward(self, input):
        lstm_out, self.hidden = self.lstm(input.view(len(input), self.batch_size, -1))


title: How to disable progress bar in Pytorch Lightning

body:
I have a lot of issues withe the  tqdm progress bar in Pytorch Lightning:

when I run trainings in a terminal, the progress bars overwrite themselves. At the end of an training epoch, a validation progress bar is printed under the training bar, but when that ends, the progress bar from the next training epoch is printed over the one from the previous epoch. hence it's not possible to see the losses from previous epochs.

INFO:root:  Name    Type Params
0   l1  Linear    7 K
Epoch 2:  56%|████████████▊          | 2093/3750 [00:05<00:03, 525.47batch/s, batch_nb=1874, loss=0.714, training_loss=0.4, v_nb=51]

the progress bars wobbles from left to right, caused by the changing in number of digits behind the decimal point of some losses.
when running in Pycharm, the validation progress bar is not printed, but in stead,

INFO:root:  Name    Type Params
0   l1  Linear    7 K
Epoch 1:  50%|█████     | 1875/3750 [00:05<00:05, 322.34batch/s, batch_nb=1874, loss=1.534, training_loss=1.72, v_nb=49]
Epoch 1:  50%|█████     | 1879/3750 [00:05<00:05, 319.41batch/s, batch_nb=1874, loss=1.534, training_loss=1.72, v_nb=49]
Epoch 1:  52%|█████▏    | 1942/3750 [00:05<00:04, 374.05batch/s, batch_nb=1874, loss=1.534, training_loss=1.72, v_nb=49]
Epoch 1:  53%|█████▎    | 2005/3750 [00:05<00:04, 425.01batch/s, batch_nb=1874, loss=1.534, training_loss=1.72, v_nb=49]
Epoch 1:  55%|█████▌    | 2068/3750 [00:05<00:03, 470.56batch/s, batch_nb=1874, loss=1.534, training_loss=1.72, v_nb=49]
Epoch 1:  57%|█████▋    | 2131/3750 [00:05<00:03, 507.69batch/s, batch_nb=1874, loss=1.534, training_loss=1.72, v_nb=49]
Epoch 1:  59%|█████▊    | 2194/3750 [00:06<00:02, 538.19batch/s, batch_nb=1874, loss=1.534, training_loss=1.72, v_nb=49]
Epoch 1:  60%|██████    | 2257/3750 [00:06<00:02, 561.20batch/s, batch_nb=1874, loss=1.534, training_loss=1.72, v_nb=49]
Epoch 1:  62%|██████▏   | 2320/3750 [00:06<00:02, 579.22batch/s, batch_nb=1874, loss=1.534, training_loss=1.72, v_nb=49]
Epoch 1:  64%|██████▎   | 2383/3750 [00:06<00:02, 591.58batch/s, batch_nb=1874, loss=1.534, training_loss=1.72, v_nb=49]
Epoch 1:  65%|██████▌   | 2445/3750 [00:06<00:02, 599.77batch/s, batch_nb=1874, loss=1.534, training_loss=1.72, v_nb=49]
Epoch 1:  67%|██████▋   | 2507/3750 [00:06<00:02, 605.00batch/s, batch_nb=1874, loss=1.534, training_loss=1.72, v_nb=49]
Epoch 1:  69%|██████▊   | 2569/3750 [00:06<00:01, 607.04batch/s, batch_nb=1874, loss=1.534, training_loss=1.72, v_nb=49]
Epoch 1:  70%|███████   | 2633/3750 [00:06<00:01, 613.98batch/s, batch_nb=1874, loss=1.534, training_loss=1.72, v_nb=49]

I would like to know if these issues can be solved or else how can I disable the progress bar and instead, just print some log details on the screen.


title: Where is the source code of pytorch conv2d?

body:
Where do I find the source code of the pytorch function conv2d?

It should be in torch.nn.functional but I only find _add_docstr lines,
if i search for conv2d. I looked here:

https://github.com/pytorch/pytorch/blob/master/torch/nn/functional.py

Update:
It's is not my typing, I do mean the function.
Conv2d class uses conv2d function from the nn.functional

Here:

https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/conv.py

On line 338:

return F.conv2d(F.pad(input, expanded_padding, mode='circular')

F is how they import functional

So then I went there, but I don't find the code.


title: Running LSTM with multiple GPUs gets "Input and hidden tensors are not at the same device"

body:
I am trying to train a LSTM layer in pytorch. I am using 4 GPUs. When initializing, I added the .cuda() function move the hidden layer to GPU. But when I run the code with multiple GPUs I am getting this runtime error :

RuntimeError: Input and hidden tensors are not at the same device

I have tried to solve the problem by using .cuda() function in the forward function like below :

self.hidden = (self.hidden[0].type(torch.FloatTensor).cuda(), self.hidden[1].type(torch.FloatTensor).cuda())

This line seems to solve the problem, but it raises my concern that if the updated hidden layer is seen in different GPUs. Should I move the vector back to cpu at the end of the forward function for a batch or is there any other way to solve the problem.


title: Indexing one PyTorch tensor by another using index_select

body:
I have a 3 x 3 PyTorch LongTensor that looks something like this:

A =
    [0, 0, 0]
    [1, 2, 2]
    [1, 2, 3]

I want to us it to index a 4 x 2 FloatTensor like this one:

B =
    [0.4, 0.5]
    [1.2, 1.4]
    [0.8, 1.9]
    [2.4, 2.9]

My intended output is the 2 x 3 x 3 FloatTensor below:

C[0,:,:] =
    [0.4, 0.4, 0.4]
    [1.2, 0.8, 0.8]
    [1.2, 0.8, 2.4]

C[1,:,:] =
    [0.5, 0.5, 0.5]
    [1.4, 1.9, 1.9]
    [1.4, 1.9, 2.9]

In other words, matrix A is indexing and broadcasting matrix B. A is the matrix of indices of B, so this operation is essentially an indexing operation.

How can this be done using the torch.index_select() function? If the solution involves adding or permuting dimensions, that's fine.


title: Caffe2 doesn't seem to see cuDNN?

body:
I am trying to install SpConv (spatially sparse convolution library), however when running python3 setup.py bdist_wheel, I get an error which seems to be related to Caffe2 not being able to see cuDNN, as I infer from this message:

CMake Error at /home/ivan/.local/lib/python3.6/site-packages/torch/share/cmake/Caffe2/Caffe2Config.cmake:96 (message):
  Your installed Caffe2 version uses cuDNN but I cannot find the cuDNN
  libraries.  Please set the proper cuDNN prefixes and / or install cuDNN.

I have tried reinstalling cuDNN but no luck. Can I manually point Caffe2 to cuDNN? This is because usually cuda is located in /usr/local/cuda/, however I have it in /usr/lib/cuda.

Full output below:

running bdist_wheel
running build
running build_py
running build_ext
Release
|||||CMAKE ARGS||||| ['-DCMAKE_PREFIX_PATH=/home/ivan/.local/lib/python3.6/site-packages/torch', '-DPYBIND11_PYTHON_VERSION=3.6', '-DSPCONV_BuildTests=OFF', '-DCMAKE_CUDA_FLAGS="--expt-relaxed-constexpr" -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/mnt/home/ivan/second.pytorch/second/spconv/build/lib.linux-x86_64-3.6/spconv', '-DCMAKE_BUILD_TYPE=Release']
-- Caffe2: CUDA detected: 9.1
-- Caffe2: CUDA nvcc is: /usr/bin/nvcc
-- Caffe2: CUDA toolkit directory: /usr
-- Caffe2: Header version is: 9.1
-- Could NOT find CUDNN (missing: CUDNN_INCLUDE_DIR)
CMake Warning at /home/ivan/.local/lib/python3.6/site-packages/torch/share/cmake/Caffe2/public/cuda.cmake:131 (message):
  Caffe2: Cannot find cuDNN library.  Turning the option off
Call Stack (most recent call first):
  /home/ivan/.local/lib/python3.6/site-packages/torch/share/cmake/Caffe2/Caffe2Config.cmake:88 (include)
  /home/ivan/.local/lib/python3.6/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:40 (find_package)
  CMakeLists.txt:20 (find_package)

-- Autodetected CUDA architecture(s):  6.1
-- Added CUDA NVCC flags for: -gencode;arch=compute_61,code=sm_61
CMake Error at /home/ivan/.local/lib/python3.6/site-packages/torch/share/cmake/Caffe2/Caffe2Config.cmake:96 (message):
  Your installed Caffe2 version uses cuDNN but I cannot find the cuDNN
  libraries.  Please set the proper cuDNN prefixes and / or install cuDNN.
Call Stack (most recent call first):
  /home/ivan/.local/lib/python3.6/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:40 (find_package)
  CMakeLists.txt:20 (find_package)

-- Configuring incomplete, errors occurred!
See also "/mnt/home/ivan/second.pytorch/second/spconv/build/temp.linux-x86_64-3.6/CMakeFiles/CMakeOutput.log".
See also "/mnt/home/ivan/second.pytorch/second/spconv/build/temp.linux-x86_64-3.6/CMakeFiles/CMakeError.log".
Traceback (most recent call last):
  File "setup.py", line 99, in <module>
    zip_safe=False,
  File "/home/ivan/.local/lib/python3.6/site-packages/setuptools/__init__.py", line 145, in setup
    return distutils.core.setup(**attrs)
  File "/usr/lib/python3.6/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/lib/python3.6/distutils/dist.py", line 955, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/usr/lib/python3/dist-packages/wheel/bdist_wheel.py", line 204, in run
    self.run_command('build')
  File "/usr/lib/python3.6/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/usr/lib/python3.6/distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
  File "/usr/lib/python3.6/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "setup.py", line 40, in run
    self.build_extension(ext)
  File "setup.py", line 82, in build_extension
    subprocess.check_call(['cmake', ext.sourcedir] + cmake_args, cwd=self.build_temp, env=env)
  File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '/mnt/home/ivan/second.pytorch/second/spconv', '-DCMAKE_PREFIX_PATH=/home/ivan/.local/lib/python3.6/site-packages/torch', '-DPYBIND11_PYTHON_VERSION=3.6', '-DSPCONV_BuildTests=OFF', '-DCMAKE_CUDA_FLAGS="--expt-relaxed-constexpr" -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/mnt/home/ivan/second.pytorch/second/spconv/build/lib.linux-x86_64-3.6/spconv', '-DCMAKE_BUILD_TYPE=Release']' returned non-zero exit status 1.

System: Ubuntu 18.04, CUDA 9.1, cuDNN 7.6.3


title: How to parallelize RNN function in Pytorch with DataParallel

body:
Here's an RNN model to run character based language generation:

class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size, n_layers):
        super(RNN, self).__init__()
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.output_size = output_size
        self.n_layers = n_layers

        self.encoder = nn.Embedding(input_size, hidden_size)
        self.GRU = nn.GRU(hidden_size, hidden_size, n_layers, batch_first=True)
        self.decoder = nn.Linear(hidden_size, output_size)

    def forward(self, input, batch_size):
        self.init_hidden(batch_size)
        input = self.encoder(input)
        output, self.hidden = self.GRU(input, self.hidden)
        output = self.decoder(output.view(batch_size, self.hidden_size))
        return output

    def init_hidden(self, batch_size):
        self.hidden = Variable(torch.randn(self.n_layers, batch_size, self.hidden_size).cuda())

I instantiate the model using DataParallel, to split the batch of inputs across my 4 GPUs:

net = torch.nn.DataParallel(RNN(n_chars, hidden_size, n_chars, n_layers)).cuda()

Here's the full code.

Unfortunately, DataParallel requires the inputs to have batch_size as the first dimension, but GRU function expects hidden tensor to have batch_size as second dimension:

output, self.hidden = self.GRU(input, self.hidden)

The code as is throws the following error (note the printouts showing that encoder is correctly executed on 4 GPUs):

...
forward function: encoding input of shape: (16L, 1L)
forward function: encoding input of shape: (16L, 1L)
forward function: encoding input of shape: (16L,
forward function: encoding input of shape:

forward function: GRU processing input of shape:
1L)
 ( (16L, 16L1L, 1L), 100L)
forward function: GRU processing input of shape:
 (16L, 1L,
forward function: GRU processing input of shape:100L)
 (16L
forward function: GRU processing input of shape:, 1L, 100L) (
16L, 1L, 100L)

Traceback (most recent call last):
  File "gru2.py", line 166, in <module>
    output = net(c, batch_size)
  File "/root/miniconda2/lib/python2.7/site-packages/torch/nn/modules/module.py", line 206, in __call__
    result = self.forward(*input, **kwargs)
  File "/root/miniconda2/lib/python2.7/site-packages/torch/nn/parallel/data_parallel.py", line 61, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/root/miniconda2/lib/python2.7/site-packages/torch/nn/parallel/data_parallel.py", line 71, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs)
  File "/root/miniconda2/lib/python2.7/site-packages/torch/nn/parallel/parallel_apply.py", line 45, in parallel_apply
    raise output
RuntimeError: Expected hidden size (2, 16L, 100), got (2L, 64L, 100L)

Here the model has 2 layers, batch_size=64, and hidden_size = 100.

How do I parallelize the GRU operation in the forward function?


This is literally all the code that I am trying to run:
from transformers import AutoModelWithLMHead, AutoTokenizer
import torch

tokenizer = AutoTokenizer.from_pretrained("microsoft/DialoGPT-small")
model = AutoModelWithLMHead.from_pretrained("microsoft/DialoGPT-small")

I am getting this error:
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-14-aad2e7a08a74> in <module>
----> 1 from transformers import AutoModelWithLMHead, AutoTokenizer
      2 import torch
      3
      4 tokenizer = AutoTokenizer.from_pretrained("microsoft/DialoGPT-small")
      5 model = AutoModelWithLMHead.from_pretrained("microsoft/DialoGPT-small")

ImportError: cannot import name 'AutoModelWithLMHead' from 'transformers' (c:\python38\lib\site-packages\transformers\__init__.py)

What do I do about it?


title: How to use Visdom (Tensorboard for PyTorch) with Google Colaboratory?

body:
I would like to use visdom when training my networks using Google Colaboratory.

This topic explains how to use Tensorboard thanks to ngrok :
https://stackoverflow.com/a/48468512/9686730

Is there something like this for visdom?

Thank you !


title: Unable to load model weights while predicting (using pytorch)

body:
I have trained a Mask RCNN network using PyTorch and am trying to use the obtained weights to predict the location of apples in an image..

I am using the dataset from this paper, and here is the github link to code being used

I am simply following the instructions as provided in the ReadMe file..

Here is the command i wrote in prompt (removed personal info)

  python predict_rcnn.py --data_path "my_directory\datasets\apples-minneapple\detection" --output_file "my_directory\samples\apples\predictions" --weight_file "my_directory\samples\apples\weights\model_19.pth" --mrcnn

model_19.pth is the file with all the weights generated after the 19th epoch

Error is as follows:

Loading model
Traceback (most recent call last):
  File "predict_rcnn.py", line 122, in <module>
    main(args)
  File "predict_rcnn.py", line 77, in main
    model.load_state_dict(checkpoint['model'], strict=False)
KeyError: 'model'

I will paste predict_rcnn.py for convenience:

import os
import torch
import torch.utils.data
import torchvision
import numpy as np

from data.apple_dataset import AppleDataset
from torchvision.models.detection.faster_rcnn import FastRCNNPredictor
from torchvision.models.detection.mask_rcnn import MaskRCNNPredictor

import utility.utils as utils
import utility.transforms as T

######################################################
# Predict with either a Faster-RCNN or Mask-RCNN predictor
# using the MinneApple dataset
######################################################
def get_transform(train):
    transforms = []
    transforms.append(T.ToTensor())
    if train:
        transforms.append(T.RandomHorizontalFlip(0.5))
    return T.Compose(transforms)

def get_maskrcnn_model_instance(num_classes):
    # load an instance segmentation model pre-trained pre-trained on COCO
    model = torchvision.models.detection.maskrcnn_resnet50_fpn(pretrained=False)

    # get number of input features for the classifier
    in_features = model.roi_heads.box_predictor.cls_score.in_features
    # replace the pre-trained head with a new one
    model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)

    # now get the number of input features for the mask classifier
    in_features_mask = model.roi_heads.mask_predictor.conv5_mask.in_channels
    hidden_layer = 256
    # and replace the mask predictor with a new one
    model.roi_heads.mask_predictor = MaskRCNNPredictor(in_features_mask, hidden_layer, num_classes)
    return model

def get_frcnn_model_instance(num_classes):
    # load an instance segmentation model pre-trained pre-trained on COCO
    model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=False)

    # get number of input features for the classifier
    in_features = model.roi_heads.box_predictor.cls_score.in_features
    # replace the pre-trained head with a new one
    model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)
    return model

def main(args):
    num_classes = 2
    device = args.device

    # Load the model from
    print("Loading model")
    # Create the correct model type
    if args.mrcnn:
        model = get_maskrcnn_model_instance(num_classes)
    else:
        model = get_frcnn_model_instance(num_classes)

    # Load model parameters and keep on CPU

    checkpoint = torch.load(args.weight_file, map_location=device)
    #checkpoint = torch.load(args.weight_file, map_location=lambda storage, loc: storage)
    model.load_state_dict(checkpoint['model'], strict=False)
    model.eval()

    print("Creating data loaders")
    dataset_test = AppleDataset(args.data_path, get_transform(train=False))
    data_loader_test = torch.utils.data.DataLoader(dataset_test, batch_size=1,
                                                   shuffle=False, num_workers=1,
                                                   collate_fn=utils.collate_fn)

    # Create output directory
    base_path = os.path.dirname(args.output_file)
    if not os.path.exists(base_path):
        os.makedirs(base_path)

    # Predict on bboxes on each image
    f = open(args.output_file, 'a')
    for image, targets in data_loader_test:
        image = list(img.to(device) for img in image)
        outputs = model(image)
        for ii, output in enumerate(outputs):
            img_id = targets[ii]['image_id']
            img_name = data_loader_test.dataset.get_img_name(img_id)
            print("Predicting on image: {}".format(img_name))
            boxes = output['boxes'].detach().numpy()
            scores = output['scores'].detach().numpy()

            im_names = np.repeat(img_name, len(boxes), axis=0)
            stacked = np.hstack((im_names.reshape(len(scores), 1), boxes.astype(int), scores.reshape(len(scores), 1)))

            # File to write predictions to
            np.savetxt(f, stacked, fmt='%s', delimiter=',', newline='\n')

if __name__ == "__main__":
    import argparse
    parser = argparse.ArgumentParser(description='PyTorch Detection')
    parser.add_argument('--data_path', required=True, help='path to the data to predict on')
    parser.add_argument('--output_file', required=True, help='path where to write the prediction outputs')
    parser.add_argument('--weight_file', required=True, help='path to the weight file')
    parser.add_argument('--device', default='cuda', help='device to use. Either cpu or cuda')
    model = parser.add_mutually_exclusive_group(required=True)
    model.add_argument('--frcnn', action='store_true', help='use a Faster-RCNN model')
    model.add_argument('--mrcnn', action='store_true', help='use a Mask-RCNN model')

    args = parser.parse_args()
    main(args)


title: Conda install of pytorch fails

body:
I created an environment with conda and I want to install pytorch in it, but it doesn't work. After I get inside my environment with source activate env_name I tried this: conda install pytorch torchvision -c pytorch (I also tried it like this: conda install -c pytorch pytorch torchvision) but I am getting this error:

Using Anaconda Cloud api site https://api.anaconda.org
Fetching package metadata: ......
Solving package specifications: ......
Error: Could not find some dependencies for pytorch: mkl >=2018, cudatoolkit >=9.0,<9.1, blas * mkl, cudatoolkit >=10.0,<10.1, cudatoolkit >=9.2,<9.3, blas * openblas, cudnn 7.0.*, cudatoolkit 9.*

Did you mean one of these?

    pytorch, pytorch-gpu, pytorch-cpu

Did you mean one of these?

    cudatoolkit

You can search for this package on anaconda.org with

    anaconda search -t conda cudatoolkit 9.*

 (and similarly for the other packages)

Here are my installed packages:

backports                 1.0                      py34_0
backports.shutil-get-terminal-size 1.0.0                     <pip>
decorator                 4.0.11                   py34_0
get_terminal_size         1.0.0                    py34_0
ipython                   4.2.0                    py34_0
ipython-genutils          0.1.0                     <pip>
ipython_genutils          0.1.0                    py34_0
libgfortran               1.0                           0
numpy                     1.9.2                    py34_0
openssl                   1.0.2l                        0
path.py                   10.0                     py34_0
pexpect                   4.2.1                    py34_0
pickleshare               0.7.4                    py34_0
pip                       9.0.1                    py34_1
ptyprocess                0.5.1                    py34_0
python                    3.4.5                         0
readline                  6.2                           2
scipy                     0.16.0               np19py34_0
setuptools                27.2.0                   py34_0
simplegeneric             0.8.1                    py34_1
six                       1.10.0                   py34_0
sqlite                    3.13.0                        0
tk                        8.5.18                        0
traitlets                 4.3.1                    py34_0
wheel                     0.29.0                   py34_0
xz                        5.2.3                         0
zlib                      1.2.11                        0

What should I do? Thank you!


title: How to make cuda unavailable in pytorch

body:
i'm running some code with cudas, and I need to test the same code on CPU to compare running time. To decide between regular pytorch tensor and cuda float tensor, the library I use calls torch.cuda.is_available(). Is there an easy method to make this function return false? I tried changing the Cuda visible devices with

os.environ["CUDA_VISIBLE_DEVICES"]=""

but torch.cuda.is_available() still return True. I went into pytorch source code, and in my case, torch.cuda.is_avaible returns

torch._C._cuda_getDeviceCount() > 0

I assume I should be able to "hide" my GPU at the start of my notebook, so the device count is equal to zero, but i didn't get any success so far. Any help is appreciated :)


title: PyTorch not downloading

body:
I go to the PyTorch website and select the following options

PyTorch Build: Stable (1.2)

Your OS: Windows

Package: pip

Language: Python 3.7

CUDA: None

(All of these are correct)

Than it displays a command to run

pip3 install torch==1.2.0+cpu torchvision==0.4.0+cpu -f https://download.pytorch.org/whl/torch_stable.html

I have already tried to mix around the the different options but none of them has worked.

ERROR: ERROR: Could not find a version that satisfies the requirement torch==1.2.0+cpu (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)
ERROR: No matching distribution found for torch==1.2.0+cpu

I tried to do pip install pytorch but pytorch doesn't support pypi