title: Optimizer got an empty parameter list in pytorch

body:
I try to use convolutional network structures in a VAE experiment, the code is something like:

Class CNN(nn.Module):...

hparams1=[...]
hparams2=[...]

Model1=CNN(hyparams1)
Model2=CNN(hyparams2)

def VAE(x):
   encoder=Model1(x)
   z=reparameterize(encoder)  #reparameterize is defined
   decoder=Model2(z)
   return decoder

if args.cuda:
    Model1.cuda()
    Model2.cuda()

parameters = chain(Model1.parameters(), Model2.parameters())
optimizer = optim.Adam(parameters, lr=1e-3)

Error:Optimizer got an empty parameter list.

The same problem occurs when I defind VAE as a class:

class VAE(nn.Module):
  def encoder(x,model):
     return model(x)

  def reparameterize():

  def decoder(z,model):
     return model(z)

  def forward(x,Model1,Model2)
     output=encoder(x,Model1)
     z=reparameterize(output)
     return decoder(z,Model2)

model=VAE()
optimizer = optim.Adam(model.parameters(), lr=1e-3)

Error:Optimizer got an empty parameter list.

I guess maybe the problem is that I fail to initialize the parameters in a class VAE, but I don't know how to deal with it. Thank you very much!


title: pytorch masked_fill: why can't I mask all zeros?

body:
I want to mask the all the zeros in the score matrix with -np.inf, but I can only get part of zeros masked, looked like

you see in the upper right corner there are still zeros that didn't get masked with -np.inf

Here's my codes:

q = torch.Tensor([np.random.random(10),np.random.random(10),np.random.random(10), np.random.random(10), np.zeros((10,1)), np.zeros((10,1))])
k = torch.Tensor([np.random.random(10),np.random.random(10),np.random.random(10), np.random.random(10), np.zeros((10,1)), np.zeros((10,1))])
scores = torch.matmul(q, k.transpose(0,1)) / math.sqrt(10)
mask = torch.Tensor([1,1,1,1,0,0])
mask = mask.unsqueeze(1)
scores = scores.masked_fill(mask==0, -np.inf)

Maybe the mask is wrong?


title: RuntimeError: Expected object of scalar type Long but got scalar type Float for argument #2 'mat2' how to fix it?

body:
import torch.nn as nn
import torch
import torch.optim as optim
import itertools

class net1(nn.Module):
    def __init__(self):
        super(net1,self).__init__()

        self.pipe = nn.Sequential(
            nn.Linear(10,10),
            nn.ReLU()
        )

    def forward(self,x):
        return self.pipe(x.long())

class net2(nn.Module):
    def __init__(self):
        super(net2,self).__init__()

        self.pipe = nn.Sequential(
            nn.Linear(10,20),
            nn.ReLU(),
            nn.Linear(20,10)
        )

    def forward(self,x):
        return self.pipe(x.long())

netFIRST = net1()
netSECOND = net2()

learning_rate = 0.001

opt = optim.Adam(itertools.chain(netFIRST.parameters(),netSECOND.parameters()), lr=learning_rate)

epochs = 1000

x = torch.tensor([1,2,3,4,5,6,7,8,9,10],dtype=torch.long)
y = torch.tensor([10,9,8,7,6,5,4,3,2,1],dtype=torch.long)

for epoch in range(epochs):
    opt.zero_grad()

    prediction = netSECOND(netFIRST(x))
    loss = (y.long() - prediction)**2
    loss.backward()

    print(loss)
    print(prediction)
    opt.step()

error:

  line 49, in  prediction = netSECOND(netFIRST(x))

  line 1371, in linear; output = input.matmul(weight.t())

  RuntimeError: Expected object of scalar type Long but got scalar type
  Float for argument #2 'mat2'

I don't really see what I'm doing wrong. I have tried to turn everything in a Long in every possible way. I don't really get the way typing works for pytorch. Last time I tried something with just one layer and it forced me to use type int.
Could someone explain how the typing is established in pytorch and how to prevent and fix errors like this??
A lot I mean an awful lot of thanks in advance, this problem really bothers me and I can't seem to fix it no matter what I try.


title: PyTorch LSTM - using word embeddings instead of nn.Embedding()

body:
Is the nn.Embedding() essential for learning for an LSTM?

I am using an LSTM in PyTorch to predict NER - example of a similar task is here - https://pytorch.org/tutorials/beginner/nlp/sequence_models_tutorial.html

Code wise, I am using code almost identical to the code in the tutorial above.

The only detail is -
I am using word2Vec instead of nn.Embedding().

So I remove the nn.Embedding() layer and provide the forward function the features from the word2Vec directly. The RNN does not learn.

Hence, Is the nn.Embedding() essential for learning for an LSTM?


title: PyTorch: Expected input batch_size (12) to match target batch_size (64)

body:
I tried out PyTorch and wanted to write a program for MNIST. But, I got the error message:

  Expected input batch_size (12) to match target batch_size (64)

I searched for a solution but I don't understand what's wrong with my code.

#kwargs is empty because I don't use cuda
kwargs = {}
train_data = torch.utils.data.DataLoader(
    datasets.MNIST('data', train=True, download=True,
                    transform=transforms.Compose([transforms.ToTensor(),
                    transforms.Normalize((0.1307,),(0.3081,))])),
    batch_size=64, shuffle=True, **kwargs)

test_data = torch.utils.data.DataLoader(
    datasets.MNIST('data', train=False,
                    transform=transforms.Compose([transforms.ToTensor(),
                    transforms.Normalize((0.1307,),(0.3081,))])),
    batch_size=64, shuffle=True, **kwargs)

class Netz(nn.Module):
    def __init__(self):
        super(Netz, self).__init__()
        self.conv1 = nn.Conv2d(1,10, kernel_size=5)
        self.conv2 = nn.Conv2d(10, 20, kernel_size=5)
        self.conv_dropout = nn.Dropout2d()
        self.fc1 = nn.Linear(320, 60)
        self.fc2 = nn.Linear(60, 10)

    def forward(self, x):
        x = self.conv1(x)
        x = F.max_pool2d(x, 2)
        x = F.relu(x)
        x = self.conv2(x)
        x = self.conv_dropout(x)
        x = F.max_pool2d(x, 2)
        x = F.relu(x)
        print(x.shape)
        x = x.view(-1, 320)
        x = self.fc1(x)
        x = x.view(-1, 320)
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        return F.log_softmax(x, dim=0)

model = Netz()

optimizer = optim.SGD(model.parameters(), lr=0.1, momentum=0.8)
def train(epoch):
    model.train()

    for batch_id, (data, target) in enumerate(train_data):
        data = Variable(data)
        target = Variable(target)
        optimizer.zero_grad()
        out = model(data)
        print(out.shape)
        criterion = nn.CrossEntropyLoss()
        loss = criterion(out, target)
        loss.backward()
        optimizer.step()
        print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'. format(
            epoch, batch_id * len(data), len(train_data.dataset),
            100. * batch_id / len(train_data), loss.data[0]))

The output should show the epoch and some other information. Actually, I print out the shape of my tensor but I don't know what's wrong. Here is the error message:

/home/michael/Programmierung/Python/PyTorch/venv/bin/python /home/michael/Programmierung/Python/PyTorch/mnist.py
torch.Size([64, 20, 4, 4])
torch.Size([12, 10])
Traceback (most recent call last):
  File "/home/michael/Programmierung/Python/PyTorch/mnist.py", line 69, in <module>
    train(epoch)
  File "/home/michael/Programmierung/Python/PyTorch/mnist.py", line 60, in train
    loss = criterion(out, target)
  File "/home/michael/Programmierung/Python/PyTorch/venv/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/michael/Programmierung/Python/PyTorch/venv/lib/python3.6/site-packages/torch/nn/modules/loss.py", line 942, in forward
    ignore_index=self.ignore_index, reduction=self.reduction)
  File "/home/michael/Programmierung/Python/PyTorch/venv/lib/python3.6/site-packages/torch/nn/functional.py", line 2056, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
  File "/home/michael/Programmierung/Python/PyTorch/venv/lib/python3.6/site-packages/torch/nn/functional.py", line 1869, in nll_loss
    .format(input.size(0), target.size(0)))
ValueError: Expected input batch_size (12) to match target batch_size (64).

Process finished with exit code 1


title: How to use pack_padded_sequence with multiple variable-length input with the same label in pytorch

body:
I have a model which takes three variable-length inputs with the same label. Is there a way I could use pack_padded_sequence somehow? If so, how should I sort my sequences?

For example,

a = (([0,1,2], [3,4], [5,6,7,8]), 1) # training data is in length 3,2,4; label is 1
b = (([0,1], [2], [6,7,8,9,10]), 1)

Both a and b will be fed into three separated LSTMs and the result will be merged to predict the target.


title: What does flatten_parameters() do?

body:
I saw many Pytorch examples using flatten_parameters in the forward function of the RNN

self.rnn.flatten_parameters()

I saw this RNNBase and it is written that it

  Resets parameter data pointer so that they can use faster code paths

What does that mean?


title: Most efficient way to use a large data set for PyTorch?

body:
Perhaps this question has been asked before, but I'm having trouble finding relevant info for my situation.

I'm using PyTorch to create a CNN for regression with image data. I don't have a formal, academic programming background, so many of my approaches are ad-hoc and just terribly inefficient. May times I can go back through my code and clean things up later because the inefficiency is not so drastic that performance is significantly affected. However, in this case, my method for using the image data takes a long time, uses a lot of memory, and it is done every time I want to test a change in the model.

What I've done is essentially loaded the image data into numpy arrays, saved those arrays in an .npy file, and then when I want to use said data for the model I import all of the data in that file. I don't think the data set is really THAT large, as it is comprised of 5000, 3 color channel images of size 64x64. Yet my memory usage shoots up to 70%-80% (out of 16gb) when it is being loaded, and it takes 20-30 seconds to load in every time.

My guess is that I'm being dumb about the way I'm loading it in, but frankly I'm not sure what the standard is. Should I, in some way, put the image data somewhere before I need it, or should the data be loaded directly from the image files? And in either case, what is the best, most efficient way to do that, independent of file structure?

I would really appreciate any help on this.


title: How to install nvidia apex on Google Colab

body:
what I did is follow the instruction on the official github site

!git clone https://github.com/NVIDIA/apex
!cd apex
!pip install -v --no-cache-dir ./

it gives me the error:

ERROR: Directory './' is not installable. Neither 'setup.py' nor 'pyproject.toml' found.
Exception information:
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/pip/_internal/cli/base_command.py", line 178, in main
    status = self.run(options, args)
  File "/usr/local/lib/python3.6/dist-packages/pip/_internal/commands/install.py", line 326, in run
    self.name, wheel_cache
  File "/usr/local/lib/python3.6/dist-packages/pip/_internal/cli/base_command.py", line 268, in populate_requirement_set
    wheel_cache=wheel_cache
  File "/usr/local/lib/python3.6/dist-packages/pip/_internal/req/constructors.py", line 248, in install_req_from_line
    "nor 'pyproject.toml' found." % name
pip._internal.exceptions.InstallationError: Directory './' is not installable. Neither 'setup.py' nor 'pyproject.toml' found.


title: No matching distribution found for torch==1.5.0+cpu on Heroku

body:
I am trying to deploy my Django app which uses a machine learning model. And the machine learning model requires pytorch to execute.
When i am trying to deploy it is giving me this error

ERROR: Could not find a version that satisfies the requirement torch==1.5.0+cpu (from -r /tmp/build_4518392d43f43bc52f067241a9661c92/requirements.txt (line 23)) (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2, 0.4.1, 0.4.1.post2, 1.0.0, 1.0.1, 1.0.1.post2, 1.1.0, 1.2.0, 1.3.0, 1.3.1, 1.4.0, 1.5.0)
       ERROR: No matching distribution found for torch==1.5.0+cpu (from -r /tmp/build_4518392d43f43bc52f067241a9661c92/requirements.txt (line 23))
 !     Push rejected, failed to compile Python app.
 !     Push failed

My requirements.txt is

asgiref==3.2.7
certifi==2020.4.5.1
chardet==3.0.4
cycler==0.10.0
dj-database-url==0.5.0
Django==3.0.6
django-heroku==0.3.1
future==0.18.2
gunicorn==20.0.4
idna==2.9
imageio==2.8.0
kiwisolver==1.2.0
matplotlib==3.2.1
numpy==1.18.4
Pillow==7.1.2
psycopg2==2.8.5
pyparsing==2.4.7
python-dateutil==2.8.1
pytz==2020.1
requests==2.23.0
six==1.14.0
sqlparse==0.3.1
torch==1.5.0+cpu
torchvision==0.6.0+cpu
urllib3==1.25.9
whitenoise==5.0.1

And runtime.txt is python-3.7.5
However installing it on my computer is not giving any type of error when i use command pip install torch==1.5.0+cpu I am using python 3.7.5 and pip 20.0.2.
Complete code is here.
How to solve this issue i really need to deploy my app. Thanks


title: Change Image format from NHWC to NCHW for Pytorch

body:
In pytorch we need images in NCHW format but my images are NHWC.

What is the procedure to feed this image to CNN? (I have found this solution which suggests to use "permute" but where and how should i use it?)


title: In pytorch how do you use add_param_group () with a optimizer?

body:
The documentation is pretty vague and there aren't example codes to show you how to use it. The documentation for it is

  Add a param group to the Optimizer s param_groups.

  This can be useful when fine tuning a pre-trained network as frozen
  layers can be made trainable and added to the Optimizer as training
  progresses.

  Parameters:    param_group (dict) – Specifies what Tensors should be
  optimized along with group optimization options. (specific) –

I am assuming I can get a param_group parameter by feeding the values I get from a model's state_dict()? E.g. all the actual weight values? I am asking this because I want to make a progressive network, which means I need to constantly feed Adam parameters from newly created convolutions and activations modules.


title: Converting a scipy coo_matrix to pytorch sparse tensor

body:
I have a coo_matrix:

from scipy.sparse import coo_matrix
coo = coo_matrix((3, 4), dtype = "int8")

That I want converted to a pytorch sparse tensor. According to the documentation https://pytorch.org/docs/master/sparse.html it should follow the coo format, but I cannot find a simple way to do the conversion. Any help would be greatly appreciated!


title: PyTorch - create padded tensor from sequences of variable length

body:
I am looking for a good (efficient and preferably simple) way to create padded tensor from sequences of variable length / shape. The best way I can imagine so far is a naive approach like this:

import torch
seq = [1,2,3]      # seq of variable length
max_len = 5        # maximum length of seq
t = torch.zeros(5) # padding value
for i, e in enumerate(seq):
    t[i] = e
print(t)

Output:

tensor([ 1.,  2.,  3.,  0.,  0.])

Is there a better way to do so?

I haven't found something yet, but I guess there must be something better.

I'm thinking of some function to extend the sequence tensor to the desired shape with the desired padding. Or something to create the padded tensor directly from the sequence. But of course other approaches are welcome too.


title: Problem with missing and unexpected keys while loading my model in Pytorch

body:
I'm trying to load the model using this tutorial: https://pytorch.org/tutorials/beginner/saving_loading_models.html#saving-loading-model-for-inference . Unfortunately I'm very beginner and I face some problems.

I have created checkpoint:

checkpoint = {'epoch': epochs, 'model_state_dict': model.state_dict(), 'optimizer_state_dict': optimizer.state_dict(),'loss': loss}
torch.save(checkpoint, 'checkpoint.pth')

Then I wrote class for my network and I wanted to load the file:

class Network(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(9216, 4096)
        self.fc2 = nn.Linear(4096, 1000)
        self.fc3 = nn.Linear(1000, 102)

    def forward(self, x):
        x = self.fc1(x)
        x = F.relu(x)
        x = self.fc2(x)
        x = F.relu(x)
        x = self.fc3(x)
        x = log(F.softmax(x, dim=1))
        return x

Like that:

def load_checkpoint(filepath):
    checkpoint = torch.load(filepath)
    model = Network()
    model.load_state_dict(checkpoint['model_state_dict'])
    optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
    epoch = checkpoint['epoch']
    loss = checkpoint['loss']

model = load_checkpoint('checkpoint.pth')

I got this error (edited to show whole communicate):

RuntimeError: Error(s) in loading state_dict for Network:
    Missing key(s) in state_dict: "fc1.weight", "fc1.bias", "fc2.weight", "fc2.bias", "fc3.weight", "fc3.bias".
    Unexpected key(s) in state_dict: "features.0.weight", "features.0.bias", "features.3.weight", "features.3.bias", "features.6.weight", "features.6.bias", "features.8.weight", "features.8.bias", "features.10.weight", "features.10.bias", "classifier.fc1.weight", "classifier.fc1.bias", "classifier.fc2.weight", "classifier.fc2.bias", "classifier.fc3.weight", "classifier.fc3.bias".

This is my model.state_dict().keys():

odict_keys(['features.0.weight', 'features.0.bias', 'features.3.weight',
'features.3.bias', 'features.6.weight', 'features.6.bias',
'features.8.weight', 'features.8.bias', 'features.10.weight',
'features.10.bias', 'classifier.fc1.weight', 'classifier.fc1.bias',
'classifier.fc2.weight', 'classifier.fc2.bias', 'classifier.fc3.weight',
'classifier.fc3.bias'])

This is my model:

AlexNet(
  (features): Sequential(
    (0): Conv2d(3, 64, kernel_size=(11, 11), stride=(4, 4), padding=(2, 2))
    (1): ReLU(inplace)
    (2): MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=False)
    (3): Conv2d(64, 192, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2))
    (4): ReLU(inplace)
    (5): MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=False)
    (6): Conv2d(192, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (7): ReLU(inplace)
    (8): Conv2d(384, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (9): ReLU(inplace)
    (10): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (11): ReLU(inplace)
    (12): MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=False)

((classifier): Sequential(
(fc1): Linear(in_features=9216, out_features=4096, bias=True)
(relu1): ReLU()
(fc2): Linear(in_features=4096, out_features=1000, bias=True)
(relu2): ReLU()
(fc3): Linear(in_features=1000, out_features=102, bias=True)
(output): LogSoftmax()
)
)

It's my first network ever and I'm blundering along. Thanks for steering me into right direction!


title: What is the difference between register_parameter and register_buffer in PyTorch?

body:
Module's parameters get changed during training, that is, they are what is learnt during training of a neural network, but what is a buffer?
and is it learnt during neural network training?


title: Why doesn't my simple pytorch network work on GPU device?

body:
I built a simple network from a tutorial and I got this error:

  RuntimeError: Expected object of type torch.cuda.FloatTensor but found
  type torch.FloatTensor for argument #4 'mat1'

Any help? Thank you!

import torch
import torchvision

device = torch.device("cuda:0")
root = '.data/'

dataset = torchvision.datasets.MNIST(root, transform=torchvision.transforms.ToTensor(), download=True)

dataloader = torch.utils.data.DataLoader(dataset, batch_size=4)

class Net(torch.nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.out = torch.nn.Linear(28*28, 10)

    def forward(self, x):
        x = x.view(x.size(0), -1)
        x = self.out(x)
        return x

net = Net()
net.to(device)

for i, (inputs, labels) in enumerate(dataloader):
    inputs.to(device)
    out = net(inputs)


title: Getting CUDA out of memory

body:
Im trying to train a network but i get,
I set my batch-size as 300 and i get this error,but even if i reduce this to 100 i still get this error,and more frustratingly for running 10 epoch on ~1200 images it takes about 40 minutes,any suggestions what is going wrong and how may i speed the process!
Any tips will be extremely helpful,Thanks in advance.

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-31-3b43ff4eea72> in <module>()
      5         labels = Variable(labels).cuda()
      6
----> 7         optimizer.zero_grad()
      8         outputs = cnn(images)
      9         loss = criterion(outputs, labels)

/usr/local/lib/python3.5/dist-packages/torch/optim/optimizer.py in     zero_grad(self)
    114                 if p.grad is not None:
    115                     if p.grad.volatile:
--> 116                         p.grad.data.zero_()
    117                     else:
    118                         data = p.grad.data

RuntimeError: cuda runtime error (2) : out of memory at /pytorch  /torch/lib/THC/generic/THCTensorMath.cu:35`

Even though my GPU's are free

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 384.111                Driver Version: 384.111                       |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 108...  Off  | 00000000:05:00.0 Off |                   N/A |
| 23%   18C    P8    15W / 250W |  10864MiB / 11172MiB |      0%      Default |
    +-------------------------------+----------------------+----------------------+

|   1  GeForce GTX 108...  Off  | 00000000:08:00.0 Off |                  N/A |
 | 23%   20C    P8    15W / 250W |     10MiB / 11172MiB |          0%      Default
+-------------------------------+----------------------+---------------


title: Pytorch DataParallel doesn't work when the model contain tensor operation

body:
If my model contains only nn.Module layers such as nn.Linear, nn.DataParallel works fine.

x = torch.randn(100,10)

class normal_model(torch.nn.Module):
    def __init__(self):
        super(normal_model, self).__init__()
        self.layer = torch.nn.Linear(10,1)

    def forward(self, x):
        return self.layer(x)

model = normal_model()
model = nn.DataParallel(model.to('cuda:0'))
model(x)

However, when my model contains a tensor operation such as the following

class custom_model(torch.nn.Module):
    def __init__(self):
        super(custom_model, self).__init__()
        self.layer = torch.nn.Linear(10,5)
        self.weight = torch.ones(5,1, device='cuda:0')
    def forward(self, x):
        return self.layer(x) @ self.weight

model = custom_model()
model = torch.nn.DataParallel(model.to('cuda:0'))
model(x)

It gives me the following error

  RuntimeError: Caught RuntimeError in replica 1 on device 1. Original
  Traceback (most recent call last):   File
  "/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py",
  line 60, in _worker
      output = module(*input, **kwargs)   File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py",
  line 541, in call
      result = self.forward(*input, **kwargs)   File "", line 7, in forward
      return self.layer(x) @ self.weight RuntimeError: arguments are located on different GPUs at
  /pytorch/aten/src/THC/generic/THCTensorMathBlas.cu:277

How to avoid this error when we have some tensor operations in our model?


title: Iterate over two Pytorch tensors at once?

body:
I have two Pytorch tensors (really, just 1-D lists), t1 and t2. Is it possible to iterate over them in parallel, i.e. do something like

for a,b in zip(t1,t2)
?

Thanks.


title: save predictions from pytorch model

body:
Im following the pytorch transfer learning tutorial and applying it to the kaggle seed classification task,Im just not sure how to save the predictions in a csv file so that i can make the submission,
Any suggestion would be helpful,This is what i have ,

use_gpu = torch.cuda.is_available()
 model = models.resnet50(pretrained=True)
for param in model.parameters():
    param.requires_grad = False

num_ftrs = model.fc.in_features
model.fc = torch.nn.Linear(num_ftrs, len(classes))
if use_gpu:
    model = model.cuda()

criterion = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.fc.parameters(), lr=0.001, momentum=0.9)
exp_lr_scheduler = lr_scheduler.StepLR(optimizer, step_size=7, gamma=0.1)

loaders = {'train':train_loader, 'valid':valid_loader, 'test': test_loader}

model = train_model(loaders, model, criterion, optimizer, exp_lr_scheduler, num_epochs=50)


title: Torch C++: Getting the value of a int tensor by using *.data()

body:
In the C++ version of Libtorch, I found that I can get the value of a float tensor by *tensor_name[0].data<float>(), in which instead of 0 I can use any other valid index. But, when I have defined an int tensor by adding option at::kInt into the tensor creation, I cannot use this structure to get the value of the tensor, i.e. something like *tensor_name[0].data<at::kInt>() or *tensor_name[0].data<int>() does not work and the debugger keeps saying that Couldn't find method at::Tensor::data<at::kInt> or Couldn't find method at::Tensor::data<int>.
I can get the values by auto value_array = tensor_name=accessor<int,1>(), but it was easier to use *tensor_name[0].data<int>(). Can you please let me know how I can use data<>() to get the value of an int tensor?
I also have a same problem with bool type.


title: Check if PyTorch tensors are equal within epsilon

body:
How do I check if two PyTorch tensors are semantically equal?

Given floating point errors, I want to know if the the elements differ only by a small epsilon value.


title: Flatten layer of PyTorch build by sequential container

body:
I am trying to build a cnn by sequential container of PyTorch, my problem is I cannot figure out how to flatten the layer.

main = nn.Sequential()
self._conv_block(main, 'conv_0', 3, 6, 5)
main.add_module('max_pool_0_2_2', nn.MaxPool2d(2,2))
self._conv_block(main, 'conv_1', 6, 16, 3)
main.add_module('max_pool_1_2_2', nn.MaxPool2d(2,2))
main.add_module('flatten', make_it_flatten)

What should I put in the "make_it_flatten"?
I tried to flatten the main but it do not work, main do not exist something call view

main = main.view(-1, 16*3*3)


title: Tensorflow: Hierarchical Softmax Implementation

body:
I'm currently having text inputs represented by vector, and I want to classify their categories. Because they are multi-level categories, I meant to use Hierarchical Softmax.

Example:

 - Computer Science
     - Machine Learning
     - NLP
 - Economics
 - Maths
     - Algebra
     - Geometry

I don't know how to implement it in Tensorflow. All examples I've met is using other frameworks.

Thanks


title: How can I make a neural network that has multiple outputs using pytorch?

body: Is my question even right? I looked everywhere but couldn't find a single thing. I'm pretty sure this was addressed when I learned keras, but how do I implement it in pytorch?


title: pip - Installing specific package version does not work

body:
I was trying to install a library (allennlp) via pip3. But it complained about the PyTorch version. While allennlp requires torch=0.4.0 I have torch=0.4.1:

...
Collecting torch==0.4.0 (from allennlp)
  Could not find a version that satisfies the requirement torch==0.4.0 (from allennlp) (from versions: 0.1.2, 0.1.2.post1, 0.4.1)
No matching distribution found for torch==0.4.0 (from allennlp)

Also manually install:

pip3 install torch==0.4.0

Doesn't work either:

  Could not find a version that satisfies the requirement torch==0.4.0 (from versions: 0.1.2, 0.1.2.post1, 0.4.1)
No matching distribution found for torch==0.4.0

Same for other versions.

Python is version Python 3.7.0 installed via brew on Mac OS.

I remember that some time ago I was able to switch between version 0.4.0 and 0.3.1 by using pip3 install torch==0.X.X.

How do I solve this?


title: Convolutional NN for text input in PyTorch

body:
I am trying to implement a text classification model using a CNN. As far as I know, for text data, we should use 1d Convolutions. I saw an example in pytorch using Conv2d but I want to know how can I apply Conv1d for text? Or, it is actually not possible?
Here is my model scenario:
Number of in-channels: 1, Number of out-channels: 128
Kernel size : 3 (only want to consider trigrams)
Batch size : 16

So, I will provide tensors of shape, <16, 1, 28, 300> where 28 is the length of a sentence. I want to use Conv1d which will give me 128 feature maps of length 26 (as I am considering trigrams).
I am not sure, how to define nn.Conv1d() for this setting. I can use Conv2d but want to know is it possible to achieve the same using Conv1d?


title: "Runtimeerror: bool value of tensor with more than one value is ambiguous" fastai

body:
I am doing semantic segmentation using cityscapes dataset in fastai. I want to ignore some classes while calculating accuracy. this is how I defined accuracy according to fastai deep learning course:

name2id = {v:k for k,v in enumerate(classes)}
unlabeled = name2id['unlabeled']
ego_v = name2id['ego vehicle']
rectification = name2id['rectification border']
roi = name2id['out of roi']
static = name2id['static']
dynamic = name2id['dynamic']
ground = name2id['ground']

def acc_cityscapes(input, target):
    target = target.squeeze(1)

    mask=(target!=unlabeled and target!= ego_v and target!= rectification
    and target!=roi and target !=static and target!=dynamic and
    target!=ground)

return (input.argmax(dim=1)[mask]==target[mask]).float().mean()

This code works if I ignore only one of the classes:

mask=target!=unlabeled

But when I am trying to ignore multiple classes like this:

mask=(target!=unlabeled and target!= ego_v and target!= rectification
    and target!=roi and target !=static and target!=dynamic and
    target!=ground)

I am getting this error:

runtimeError: bool value of tensor with more than one value is ambiguous

Any Idea how do I solve this?


title: PyTorch Geometric CUDA installation issues on Google Colab

body:
I was working on a PyTorch Geometric project using Google Colab for CUDA support. Since it's library isn't present by default, I run:

!pip install --upgrade torch-scatter
!pip install --upgrade torch-sparse
!pip install --upgrade torch-cluster
!pip install --upgrade torch-spline-conv
!pip install torch-geometric

Recently, while importing torch_geometric, owing to version upgrades, there's a CUDA version mismatch saying:

  RuntimeError: Detected that PyTorch and torch_sparse were compiled with different CUDA versions. PyTorch has CUDA version 10.1 and torch_sparse has CUDA version 10.0. Please reinstall the torch_sparse that matches your PyTorch install.

To solve this, I tried using conda for specific CUDA version as:

!conda install pytorch==1.4.0 cudatoolkit=10.0 -c pytorch

Yet, on running print(torch.version.cuda), I get 10.1 as the output and not 10.0 as I wanted.

This is a recent error since it wasn't throwing up this issue in past week. Any best practice to solve this issue?


title: Can I obtain the expression of derivation by using Pytorch or Tensorflow?

body:
My question is whether I could obtain the expression after I compute the derivation by using PyTorch or TensorFlow. For better understanding my question I provide an example below:

I have a function (x1, x2, and x3 are all the decision variables)

y = x1 + x1*x2 + x1*x3

The derivation of y in terms of x1 should be:

1 + x2 + x3

My question is, if I use PyTorch or TensorFlow do the computation, could I print out the expression of "1 + x2 + x3"? Very much appreciate if the codes are provided no matter by using PyTorch or TensorFlow.


title: LSTM with Attention

body:
I am trying to add attention mechanism to stacked LSTMs implementation https://github.com/salesforce/awd-lstm-lm

All examples online use encoder-decoder architecture, which I do not want to use (do I have to for the attention mechanism?).

Basically, I have used https://webcache.googleusercontent.com/search?q=cache:81Q7u36DRPIJ:https://github.com/zhedongzheng/finch/blob/master/nlp-models/pytorch/rnn_attn_text_clf.py+&cd=2&hl=en&ct=clnk&gl=uk

def __init__(self, rnn_type, ntoken, ninp, nhid, nlayers, dropout=0.5, dropouth=0.5, dropouti=0.5, dropoute=0.1, wdrop=0, tie_weights=False):
    super(RNNModel, self).__init__()
    self.encoder = nn.Embedding(ntoken, ninp)
    self.rnns = [torch.nn.LSTM(ninp if l == 0 else nhid, nhid if l != nlayers - 1 else (ninp if tie_weights else nhid), 1, dropout=0) for l in range(nlayers)]
    for rnn in self.rnns:
        rnn.linear = WeightDrop(rnn.linear, ['weight'], dropout=wdrop)
    self.rnns = torch.nn.ModuleList(self.rnns)
    self.attn_fc = torch.nn.Linear(ninp, 1)
    self.decoder = nn.Linear(nhid, ntoken)

    self.init_weights()

def attention(self, rnn_out, state):
    state = torch.transpose(state, 1,2)
    weights = torch.bmm(rnn_out, state)# torch.bmm(rnn_out, state)
    weights = torch.nn.functional.softmax(weights)#.squeeze(2)).unsqueeze(2)
    rnn_out_t = torch.transpose(rnn_out, 1, 2)
    bmmed = torch.bmm(rnn_out_t, weights)
    bmmed = bmmed.squeeze(2)
    return bmmed

def forward(self, input, hidden, return_h=False, decoder=False, encoder_outputs=None):
    emb = embedded_dropout(self.encoder, input, dropout=self.dropoute if self.training else 0)
    emb = self.lockdrop(emb, self.dropouti)

    new_hidden = []
    raw_outputs = []
    outputs = []
    for l, rnn in enumerate(self.rnns):
        temp = []
        for item in emb:
            item = item.unsqueeze(0)
            raw_output, new_h = rnn(item, hidden[l])

            raw_output = self.attention(raw_output, new_h[0])

            temp.append(raw_output)
        raw_output = torch.stack(temp)
        raw_output = raw_output.squeeze(1)

        new_hidden.append(new_h)
        raw_outputs.append(raw_output)
        if l != self.nlayers - 1:
            raw_output = self.lockdrop(raw_output, self.dropouth)
            outputs.append(raw_output)
    hidden = new_hidden

    output = self.lockdrop(raw_output, self.dropout)
    outputs.append(output)

    outputs = torch.stack(outputs).squeeze(0)
    outputs = torch.transpose(outputs, 2,1)
    output = output.transpose(2,1)
    output = output.contiguous()
    decoded = self.decoder(output.view(output.size(0)*output.size(1), output.size(2)))
    result = decoded.view(output.size(0), output.size(1), decoded.size(1))
    if return_h:
        return result, hidden, raw_outputs, outputs
    return result, hidden

This model is training, but my loss is quite high as compared to the model without the attention model.


title: Pytorch trying to make a NN received an invalid combination of arguments

body:
I am trying to build my first NN with pytroch and got an issue.

  TypeError: new() received an invalid combination of arguments - got (float, int, int, int), but expected one of:
   * (torch.device device)
   * (torch.Storage storage)
   * (Tensor other)
   * (tuple of ints size, torch.device device)
   * (object data, torch.device device)

Now I know what this is saying in that I am not passing the right type to the method  or init. But I dont know what I should pass as it looks right to me.

def main():
#Get the time and data
now = datetime.datetime.now()
hourGlassToStack = 2 #Hourglasses to stack
numModules= 2        #Residual Modules for each hourglass
numFeats = 256      #Number of features in each hourglass
numRegModules = 2   #Depth regression modules

print("Creating Model")
model = HourglassNet3D(hourGlassToStack, numModules, numFeats,numRegModules).cuda()
print("Model Created")

this is the main method that created the model.
It then calls this methods.

class HourglassNet3D(nn.Module):

  def __init__(self, nStack, nModules, nFeats, nRegModules):
    super(HourglassNet3D, self).__init__()
    self.nStack = nStack
    self.nModules = nModules
    self.nFeats = nFeats
    self.nRegModules = nRegModules
    self.conv1_ = nn.Conv2d(3, 64, bias = True, kernel_size = 7, stride = 2, padding = 3)
   self.bn1 = nn.BatchNorm2d(64)
    self.relu = nn.ReLU(inplace = True)
    self.r1 = Residual(64, 128)
    self.maxpool = nn.MaxPool2d(kernel_size = 2, stride = 2)
    self.r4 = Residual(128, 128)
    self.r5 = Residual(128, self.nFeats)

    _hourglass, _Residual, _lin_, _tmpOut, _ll_, _tmpOut_, _reg_ = [], [], [], [], [], [], []
    for i in range(self.nStack):
      _hourglass.append(Hourglass(4, self.nModules, self.nFeats))
      for j in range(self.nModules):
        _Residual.append(Residual(self.nFeats, self.nFeats))
      lin = nn.Sequential(nn.Conv2d(self.nFeats, self.nFeats, bias = True, kernel_size = 1, stride = 1),
                      nn.BatchNorm2d(self.nFeats), self.relu)
      _lin_.append(lin)
      _tmpOut.append(nn.Conv2d(self.nFeats, 16, bias = True, kernel_size = 1, stride = 1))
  _ll_.append(nn.Conv2d(self.nFeats, self.nFeats, bias = True, kernel_size = 1, stride = 1))
  _tmpOut_.append(nn.Conv2d(16, self.nFeats, bias = True, kernel_size = 1, stride = 1))

for i in range(4):
  for j in range(self.nRegModules):
    _reg_.append(Residual(self.nFeats, self.nFeats))

self.hourglass = nn.ModuleList(_hourglass)
self.Residual = nn.ModuleList(_Residual)
self.lin_ = nn.ModuleList(_lin_)
self.tmpOut = nn.ModuleList(_tmpOut)
self.ll_ = nn.ModuleList(_ll_)
self.tmpOut_ = nn.ModuleList(_tmpOut_)
self.reg_ = nn.ModuleList(_reg_)

self.reg = nn.Linear(4 * 4 * self.nFeats,16 )

And this then call this

class Residual(nn.Module):
#set the number ofinput and output for each layer
def __init__(self, numIn, numOut):
   super(Residual, self).__init__()
   self.numIn = numIn
   self.numOut = numOut
   self.bn = nn.BatchNorm2d(self.numIn)
   self.relu = nn.ReLU(inplace = True)
   self.conv1 = nn.Conv2d(self.numIn, self.numOut / 2, bias = True, kernel_size = 1)
   self.bn1 = nn.BatchNorm2d(self.numOut / 2)
   self.conv2 = nn.Conv2d(self.numOut / 2, self.numOut / 2, bias = True, kernel_size = 3, stride = 1, padding = 1)
   self.bn2 = nn.BatchNorm2d(self.numOut / 2)
   self.conv3 = nn.Conv2d(self.numOut / 2, self.numOut, bias = True, kernel_size = 1)

   if self.numIn != self.numOut:
       self.conv4 = nn.Conv2d(self.numIn, self.numOut, bias = True, kernel_size = 1)

all of this looks fine to me, but I dont know how I am suppose to pass this then if I am doing it wrong.
Thank you for any help


title: Pytorch: no CUDA-capable device is detected on Linux

body:
When trying to run some Pytorch code I get this error:

THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=74 error=38 : no CUDA-capable device is detected
Traceback (most recent call last):
File "demo.py", line 173, in test
pca = torch.FloatTensor( np.load('../basics/U_lrw1.npy')[:,:6]).cuda()
RuntimeError: cuda runtime error (38) : no CUDA-capable device is detected at /pytorch/aten/src/THC/THCGeneral.cpp:74

I a running a cloud virtual machine using the 'Google Deep Learning VM'
Version: tf-gpu.1-13.m25
Based on: Debian GNU/Linux 9.9 (stretch) (GNU/Linux 4.9.0-9-amd64 x86_64\n)
Linux tf-gpu-interruptible 4.9.0-9-amd64 #1 SMP Debian 4.9.168-1 (2019-04-12) x86_64

Environment info:

$ nvidia-smi
Sun May 26 05:32:33 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.72       Driver Version: 410.72       CUDA Version: 10.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K80           Off  | 00000000:00:04.0 Off |                    0 |
| N/A   42C    P0    74W / 149W |      0MiB / 11441MiB |    100%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

 $ echo $CUDA_PATH

 $ echo $LD_LIBRARY_PATH
/usr/local/cuda/lib64:/usr/local/nccl2/lib:/usr/local/cuda/extras/CUPTI/lib64

$ env | grep CUDA
CUDA_VISIBLE_DEVICES=0

$ pip freeze
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.
7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
audioread==2.1.7
backports.functools-lru-cache==1.5
certifi==2019.3.9
chardet==3.0.4
cloudpickle==1.1.1
cycler==0.10.0
dask==1.2.2
decorator==4.4.0
dlib==19.17.0
enum34==1.1.6
filelock==3.0.12
funcsigs==1.0.2
future==0.17.1
gdown==3.8.1
idna==2.8
joblib==0.13.2
kiwisolver==1.1.0
librosa==0.6.3
llvmlite==0.28.0


title: Differences between `torch.Tensor` and `torch.cuda.Tensor`

body:
We can allocate a tensor on GPU using torch.Tensor([1., 2.], device='cuda'). Are there any differences using that way rather than torch.cuda.Tensor([1., 2.]), except we can pass in a specific CUDA device to the former one?

Or in other words, in which scenario is torch.cuda.Tensor() necessary?


title: Expected object of type torch.FloatTensor but found type torch.cuda.FloatTensor for argument #2 'weight'

body:
Firstly I have used like 'model.cuda()' to convert model and data to cuda. But it still has such a problem. I debug every layer of the model, and weights for every module has iscuda=True. So does anyone know why there is such a problem?

I have two models, one is resnet50 and another one which contains the first one as backbone.

class FC_Resnet(nn.Module):
    def __init__(self, model, num_classes):
        super(FC_Resnet, self).__init__()

        # feature encoding
        self.features = nn.Sequential(
            model.conv1,
            model.bn1,
            model.relu,
            model.maxpool,
            model.layer1,
            model.layer2,
            model.layer3,
            model.layer4)

        # classifier
        num_features = model.layer4[1].conv1.in_channels
        self.classifier = nn.Sequential(
            nn.Conv2d(num_features, num_classes, kernel_size=1, bias=True))

    def forward(self, x):
        # children=self.features.children()
        # for child in children:
        #     if child.weight is not None:
        #         print(child.weight.device)
        x = self.features(x)
        x = self.classifier(x)
        return x

def fc_resnet50(num_classes=20, pre_trained=True):
    model = FC_Resnet(models.resnet50(pre_trained), num_classes)

    return model

And another one:

class PeakResponseMapping(nn.Sequential):
    def __init__(self, *args, **kargs):
        super(PeakResponseMapping, self).__init__(*args)
        ...

    def forward(self, input, class_threshold=0, peak_threshold=30, retrieval_cfg=None):
        assert input.dim() == 4
        if self.inferencing:
            input.requires_grad_()

        class_response_maps = super(PeakResponseMapping, self).forward(input)

        return class_response_maps

And the main is very simple:

def main():
    dataset = VOC(img_transform=image_transform())
    dataloader = DataLoader(dataset, batch_size=BATCH_SIZE, shuffle=True)

    model = peak_response_mapping(fc_resnet50(), win_size=3, sub_pixel_locating_factor=8, enable_peak_stimulation=True)
    model=model.cuda()

    for step, (b_x, b_y) in enumerate(dataloader):
        b_x.cuda()
        b_y.cuda()

        result = model.forward(b_x)


title: tensorflow stop_gradient equivalent in pytorch

body: What is the tf.stop_gradient() equivalent (provides a way to not compute gradient with respect to some variables during back-propagation) in pytorch?


title: Is there any pytorch function can combine the specific continuous dimensions of tensor into one?

body:
Let's call the function I'm looking for "magic_combine", which can combine the continuous dimensions of tensor I give to it. For more specific, I want it to do the following thing:

a = torch.zeros(1, 2, 3, 4, 5, 6)
b = a.magic_combine(2, 5) # combine dimension 2, 3, 4
print(b.size()) # should be (1, 2, 60, 6)

I know that torch.view() can do the similar thing. But I'm just wondering if there is any more elegant way to achieve the goal?


title: How to train pytorch model with numpy data and batch size?

body:
I am learning the basics of pytorch and thought to create a simple 4 layer nerual network with dropout to train IRIS dataset for classification. After refering to many tutorials I wrote this code.

import pandas as pd
from sklearn.datasets import load_iris
import torch
from torch.autograd import Variable

epochs=300
batch_size=20
lr=0.01

#loading data as numpy array
data = load_iris()
X=data.data
y=pd.get_dummies(data.target).values

#convert to tensor
X= Variable(torch.from_numpy(X), requires_grad=False)
y=Variable(torch.from_numpy(y), requires_grad=False)
print(X.size(),y.size())

#neural net model
model = torch.nn.Sequential(
    torch.nn.Linear(4, 10),
    torch.nn.ReLU(),
    torch.nn.Dropout(),
    torch.nn.Linear(10, 5),
    torch.nn.ReLU(),
    torch.nn.Dropout(),
    torch.nn.Linear(5, 3),
    torch.nn.Softmax()
)

print(model)

# Loss and Optimizer
optimizer = torch.optim.Adam(model.parameters(), lr=lr)
loss_func = torch.nn.CrossEntropyLoss()

for i in range(epochs):
    # Forward pass
    y_pred = model(X)

    # Compute and print loss.
    loss = loss_func(y_pred, y)
    print(i, loss.data[0])

    # Before the backward pass, use the optimizer object to zero all of the
    # gradients for the variables it will update (which are the learnable weights
    # of the model)
    optimizer.zero_grad()

    # Backward pass
    loss.backward()

    # Calling the step function on an Optimizer makes an update to its parameters
    optimizer.step()

There are currently two problems I am facing.

I want to set a batch size of 20. How should I do this?
At this step y_pred = model(X) its showing this error

Error

 TypeError: addmm_ received an invalid combination of arguments - got (int, int, torch.DoubleTensor, torch.FloatTensor), but expected one of:
 * (torch.DoubleTensor mat1, torch.DoubleTensor mat2)
 * (torch.SparseDoubleTensor mat1, torch.DoubleTensor mat2)
 * (float beta, torch.DoubleTensor mat1, torch.DoubleTensor mat2)
 * (float alpha, torch.DoubleTensor mat1, torch.DoubleTensor mat2)
 * (float beta, torch.SparseDoubleTensor mat1, torch.DoubleTensor mat2)
 * (float alpha, torch.SparseDoubleTensor mat1, torch.DoubleTensor mat2)
 * (float beta, float alpha, torch.DoubleTensor mat1, torch.DoubleTensor mat2)
      didn't match because some of the arguments have invalid types: (int, int, torch.DoubleTensor, !torch.FloatTensor!)
 * (float beta, float alpha, torch.SparseDoubleTensor mat1, torch.DoubleTensor mat2)
      didn't match because some of the arguments have invalid types: (int, int, !torch.DoubleTensor!, !torch.FloatTensor!)


title: How to profiling layer-by-layer in Pytroch?

body:
I have tried to profile layer-by-layer of DenseNet in Pytorch as caffe-time tool.

First trial : using autograd.profiler like below

...
model = models.__dict__['densenet121'](pretrained=True)
model.to(device)

with torch.autograd.profiler.profile(use_cuda=True) as prof:
    model.eval()
print(prof)
...

But the any results are shown except for this message :

<unfinished torch.autograd.profile>

Ultimately, I want to profile network archtiectures(i.g.DenseNet) to check where bottlenecks happen.

Could anyone do this?


title: How to fix 'ImportError: /home/... .../lib/libtorch.so.1: undefined symbol: nvrtcGetProgramLogSize' in DGL?

body:
I get an error in the importation of pytorch inside dgl (Deep Graph Library by DeepMind), concretely:

ImportError: /home/user/anaconda3/envs/my_env/lib/python3.7/site-packages/torch/lib/libtorch.so.1: undefined symbol: nvrtcGetProgramLogSize

I tried to reinstall pytorch (uninstall reinstall with conda un/install). I also search in google and I found this https://github.com/pytorch/pytorch/issues/14973. There, they solve it linking in libnvrtc.so and libcuda.so, but I have no idea what that means. Does anyone know it?

This is the basic code:

import dgl
from parseador import train_df

g = dgl.DGLGraph()
g.add_nodes(5)
g.add_edges([0, 0, 0, 0], [1, 2, 3, 4])
g.ndata['h'] = th.randn(5, 3)
g.edata['h'] = th.randn(4, 4)

And this is the error:

Traceback (most recent call last):
  File "/home/user/Documentos/Repo/grafos.py", line 1, in <module>
    import dgl
  File "/home/user/anaconda3/envs/my_env/lib/python3.7/site-packages/dgl/__init__.py", line 2, in <module>
    from . import function
  File "/home/user/anaconda3/envs/my_env/lib/python3.7/site-packages/dgl/function/__init__.py", line 5, in <module>
    from .message import *
  File "/home/user/anaconda3/envs/my_env/lib/python3.7/site-packages/dgl/function/message.py", line 7, in <module>
    from .. import backend as F
  File "/home/user/anaconda3/envs/my_env/lib/python3.7/site-packages/dgl/backend/__init__.py", line 46, in <module>
    load_backend(os.environ.get('DGLBACKEND', 'pytorch').lower())
  File "/home/user/anaconda3/envs/my_env/lib/python3.7/site-packages/dgl/backend/__init__.py", line 18, in load_backend
    mod = importlib.import_module('.%s' % mod_name, __name__)
  File "/home/user/anaconda3/envs/my_env/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/home/user/anaconda3/envs/my_env/lib/python3.7/site-packages/dgl/backend/pytorch/__init__.py", line 1, in <module>
    from .tensor import *
  File "/home/user/anaconda3/envs/my_env/lib/python3.7/site-packages/dgl/backend/pytorch/tensor.py", line 5, in <module>
    import torch as th
  File "/home/user/anaconda3/envs/my_env/lib/python3.7/site-packages/torch/__init__.py", line 102, in <module>
    from torch._C import *
ImportError: /home/user/anaconda3/envs/my_env/lib/python3.7/site-packages/torch/lib/libtorch.so.1: undefined symbol: nvrtcGetProgramLogSize

How can I fix this error? some time ago I run correctly this code in Windows 10, now I'm running Ubuntu 18.04.


title: PyTorch custom dataset dataloader returns strings (of keys) not tensors

body:
I am trying to load my own dataset and I use a custom Dataloader that reads in images and labels and converts them to PyTorch Tensors. However when the Dataloader is instantiated it returns strings x "image" and y "labels" but not the real values or tensors when read (iter)

 print(self.train_loader)  # shows a Tensor object
tic = time.time()
with tqdm(total=self.num_train) as pbar:
    for i, (x, y) in enumerate(self.train_loader):  # x and y are returned as string (where it fails)

        if self.use_gpu:
            x, y = x.cuda(), y.cuda()
        x, y = Variable(x), Variable(y)

This is how dataloader.py looks like:

 from __future__ import print_function, division #ds
import numpy as np
from utils import plot_images

import os #ds
import pandas as pd #ds
from skimage import io, transform #ds
import torch
from torchvision import datasets
from torch.utils.data import Dataset, DataLoader #ds
from torchvision import transforms
from torchvision import utils #ds
from torch.utils.data.sampler import SubsetRandomSampler

class CDataset(Dataset):

    def __init__(self, csv_file, root_dir, transform=None):
        """
        Args:
            csv_file (string): Path to the csv file with annotations.
            root_dir (string): Directory with all the images.
            transform (callable, optional): Optional transform to be applied
                on a sample.
        """
        self.frame = pd.read_csv(csv_file)
        self.root_dir = root_dir
        self.transform = transform

    def __len__(self):
        return len(self.frame)

    def __getitem__(self, idx):
        img_name = os.path.join(self.root_dir,
                                self.frame.iloc[idx, 0]+'.jpg')
        image = io.imread(img_name)
#       image = image.transpose((2, 0, 1))
        labels = np.array(self.frame.iloc[idx, 1])#.as_matrix() #ds
        #landmarks = landmarks.astype('float').reshape(-1, 2)
        #print(image.shape)
        #print(img_name,labels)
        sample = {'image': image, 'labels': labels}

        if self.transform:
            sample = self.transform(sample)

        return sample

class ToTensor(object):
    """Convert ndarrays in sample to Tensors."""

    def __call__(self, sample):
        image, labels = sample['image'], sample['labels']
        #print(image)
        #print(labels)
        # swap color axis because
        # numpy image: H x W x C
        # torch image: C X H X W
        image = image.transpose((2, 0, 1))
        #print(image.shape)
        #print((torch.from_numpy(image)))
        #print((torch.from_numpy(labels)))
        return {'image': torch.from_numpy(image),
                'labels': torch.from_numpy(labels)}

def get_train_valid_loader(data_dir,
                           batch_size,
                           random_seed,
                           #valid_size=0.1, #ds
                           #shuffle=True,
                           show_sample=False,
                           num_workers=4,
                           pin_memory=False):
    """
    Utility function for loading and returning train and valid
    multi-process iterators over the MNIST dataset. A sample
    9x9 grid of the images can be optionally displayed.

    If using CUDA, num_workers should be set to 1 and pin_memory to True.

    Args
    ----
    - data_dir: path directory to the dataset.
    - batch_size: how many samples per batch to load.
    - random_seed: fix seed for reproducibility.
    - #ds valid_size: percentage split of the training set used for
      the validation set. Should be a float in the range [0, 1].
      In the paper, this number is set to 0.1.
    - shuffle: whether to shuffle the train/validation indices.
    - show_sample: plot 9x9 sample grid of the dataset.
    - num_workers: number of subprocesses to use when loading the dataset.
    - pin_memory: whether to copy tensors into CUDA pinned memory. Set it to
      True if using GPU.

    Returns
    -------
    - train_loader: training set iterator.
    - valid_loader: validation set iterator.
    """
    #ds
    #error_msg = "[!] valid_size should be in the range [0, 1]."
    #assert ((valid_size >= 0) and (valid_size <= 1)), error_msg
    #ds

    # define transforms
    #normalize = transforms.Normalize((0.1307,), (0.3081,))
    trans = transforms.Compose([
        ToTensor(), #normalize,
    ])

    # load train dataset
    #train_dataset = datasets.MNIST(
    #    data_dir, train=True, download=True, transform=trans
    #)

    train_dataset = CDataset(csv_file='/home/Desktop/6June17/util/train.csv',
                                    root_dir='/home/caffe/data/images/',transform=trans)

    # load validation dataset
    #valid_dataset = datasets.MNIST( #ds
    #    data_dir, train=True, download=True, transform=trans #ds
    #)

    valid_dataset = CDataset(csv_file='/home/Desktop/6June17/util/eval.csv',
                                    root_dir='/home/caffe/data/images/',transform=trans)

    num_train = len(train_dataset)
    train_indices = list(range(num_train))
    #ds split = int(np.floor(valid_size * num_train))

    num_valid = len(valid_dataset) #ds
    valid_indices = list(range(num_valid)) #ds

    #if shuffle:
    #    np.random.seed(random_seed)
    #    np.random.shuffle(indices)

    #ds train_idx, valid_idx = indices[split:], indices[:split]
    train_idx = train_indices #ds
    valid_idx = valid_indices #ds

    train_sampler = SubsetRandomSampler(train_idx)
    valid_sampler = SubsetRandomSampler(valid_idx)

    train_loader = torch.utils.data.DataLoader(
        train_dataset, batch_size=batch_size, sampler=train_sampler,
        num_workers=num_workers, pin_memory=pin_memory,
    )

    print(train_loader)

    valid_loader = torch.utils.data.DataLoader(
        valid_dataset, batch_size=batch_size, sampler=valid_sampler,
        num_workers=num_workers, pin_memory=pin_memory,
    )

    # visualize some images
    if show_sample:
        sample_loader = torch.utils.data.DataLoader(
            dataset, batch_size=9, #shuffle=shuffle,
            num_workers=num_workers, pin_memory=pin_memory
        )
        data_iter = iter(sample_loader)
        images, labels = data_iter.next()
        X = images.numpy()
        X = np.transpose(X, [0, 2, 3, 1])
        plot_images(X, labels)

    return (train_loader, valid_loader)

def get_test_loader(data_dir,
                    batch_size,
                    num_workers=4,
                    pin_memory=False):
    """
    Utility function for loading and returning a multi-process
    test iterator over the MNIST dataset.

    If using CUDA, num_workers should be set to 1 and pin_memory to True.

    Args
    ----
    - data_dir: path directory to the dataset.
    - batch_size: how many samples per batch to load.
    - num_workers: number of subprocesses to use when loading the dataset.
    - pin_memory: whether to copy tensors into CUDA pinned memory. Set it to
      True if using GPU.

    Returns
    -------
    - data_loader: test set iterator.
    """
    # define transforms
    #normalize = transforms.Normalize((0.1307,), (0.3081,))
    trans = transforms.Compose([
        ToTensor(), #normalize,
    ])

    # load dataset
    #dataset = datasets.MNIST(
    #    data_dir, train=False, download=True, transform=trans
    #)

    test_dataset = CDataset(csv_file='/home/Desktop/6June17/util/test.csv',
                                    root_dir='/home/caffe/data/images/',transform=trans)

    test_loader = torch.utils.data.DataLoader(
        test_dataset, batch_size=batch_size, shuffle=False,
        num_workers=num_workers, pin_memory=pin_memory,
    )

    return test_loader

#for i_batch, sample_batched in enumerate(dataloader):
#    print(i_batch, sample_batched['image'].size(),
#          sample_batched['landmarks'].size())

#    # observe 4th batch and stop.
#    if i_batch == 3:
#        plt.figure()
#        show_landmarks_batch(sample_batched)
#        plt.axis('off')
#        plt.ioff()
#        plt.show()
#        break

A minimal working sample will be difficult to post here but basically I am trying to modify this project http://torch.ch/blog/2015/09/21/rmva.html which works smoothly with MNIST. I am just trying to run it with my own dataset with the custom dataloader.py I use above.

It instantiates a Dataloader like this:

in trainer.py:

 if config.is_train:
    self.train_loader = data_loader[0]
    self.valid_loader = data_loader[1]
    self.num_train = len(self.train_loader.sampler.indices)
    self.num_valid = len(self.valid_loader.sampler.indices)

-> run from main.py:

 if config.is_train:
    data_loader = get_train_valid_loader(
        config.data_dir, config.batch_size,
        config.random_seed, #config.valid_size,
        #config.shuffle,
        config.show_sample, **kwargs
    )


title: PyTorch : How to properly create a list of nn.Linear()

body:
I have created a class that has nn.Module as subclass.

In my class, I have to create N number of linear transformation, where N is given as class parameters.

I therefore proceed as follow :

    self.list_1 = []

    for i in range(N):
        self.list_1.append(nn.Linear(self.x, 1, bias=mlp_bias))

In the forward method, i call these matrices (with list_1[i]) and concat the results.

Two things :

1)

Even though I use model.cuda(), these Linear transform are used on cpu and i get the following error :

  RuntimeError: Expected object of type Variable[torch.cuda.FloatTensor] but found type Variable[torch.FloatTensor] for argument #1 'mat2'

I have to do

self.list_1.append(nn.Linear(self.x, 1, bias=mlp_bias).cuda())

This is not required if instead, i do :

self.nn = nn.Linear(self.x, 1, bias=mlp_bias)

and then use self.nn directly.

2)

For more obvious reason, when I print(model) in my main, the Linear matrices in my list arent printed.

Is there any other way. maybe using bmm ? I find it less easy, and i actually want to have my N results separately.

Thank you in advance,

M


title: Calculate covariance matrix for complex data in two channels (no complex data type)

body:
I have complex-valued data given in 2 channels of a matrix (one is the real, one the imaginary part, so the matrix dimensions are (height, width, 2), since Pytorch does not have native complex data types. I now want to calculate the covariance matrix. The stripped-down numpy calculation adapted for Pytorch is this:

def cov(m, y=None):

    if m.ndimension() > 2:
        raise ValueError("m has more than 2 dimensions")

    if y.ndimension() > 2:
        raise ValueError('y has more than 2 dimensions')

    X = m
    if X.shape[0] == 0:
        return torch.tensor([]).reshape(0, 0)
    if y is not None:
        X = torch.cat((X, y), dim=0)

    ddof = 1

    avg = torch.mean(X, dim=1)

    fact = X.shape[1] - ddof

    if fact <= 0:
        import warnings
        warnings.warn("Degrees of freedom <= 0 for slice",
                      RuntimeWarning, stacklevel=2)
        fact = 0.0

    X -= avg[:, None]
    X_T = X.t()
    c = dot(X, X_T)
    c *= 1. / fact
    return c.squeeze()

Now in numpy, this would transparently work with complex numbers, but I cannot simply feed a 3-d array with the last dimension being (real, imag) and hope it will work.

How can I adapt the calculation to obtain the complex covariance matrix with real and imaginary channels?


title: Google-Colaboratory - How to Refresh google-drive?

body:
I am using google-colaboratory GPU to train NN models.
My python/pytorch code is stored in google-drive.
I am able to mount my drive in colaboratory  and train models.
But any python code changes in the "my drive"  is not updated to google-colaboratory even after rebooting my PC and start all again.

To clear the google-colaboratory cache I tried :

!google-drive-ocamlfuse -cc

But it does not work:

/bin/bash: google-drive-ocamlfuse: command not found

How to clean this cache and avoid waiting hours before my code being taken into account by google-colaboratory ? Thanks in advance

PS : the method I used to mount:

from google.colab import drive
drive.mount('/content/drive/')


title: PyTorch: copy_if failed to synchronize: device-side assert triggered

body:
Getting the following error:

  RuntimeError: copy_if failed to synchronize: device-side assert triggered

when trying to execute:

if torch.nonzero(gt_class_ids > n_classes).size()[0] > 0:

where gt_class_ids is Torch.cuda.LongTensor of size [128] and n_classes = 81.
Running on cuda 9.x


title: I don't understand pytorch input sizes of conv1d, conv2d

body:
I have a data of 2 temporal series of 18 points each one. So I organized in a matrix of 18 rows and 2 columns (with 180 samples to classify in 2 classes - activated and non-activated).

So, I want to do a CNN with this data, my kernel walks in one direction, along the lines (temporal). Examples of the figure attached.

My data 18x2

In my code, I don't know how channels I have, in comparison to RGB with 3 channels.  And don't know the input sizes of the layers, and how to calculate to know the fully connected layer.

I need to use conv1d ? conv2d? conv3d ?
Based on Understand conv 1D 2D 3D, I have 2D inputs and I want to do 1D convolution (because I move my kernel in one direction), is it correct ?

How I pass the kernel size (3,2) for example?

My data is in this form, after using DataLoader with batch_size= 4:

print(data.shape, label.shape)

torch.Size([4, 2, 18]) torch.Size([4, 1])

My Convolutional Model is:

OBS: I just put any number of input/output size.

# Creating our CNN Model -> 1D convolutional with 2D input (HbO, HbR)

class ConvModel(nn.Module):
    def __init__(self):
        super(ConvModel, self).__init__()
        self.conv1 = nn.Conv1d(in_channels=1,  out_channels= 18, kernel_size=3, stride = 1)
# I dont know the in/out channels of the first conv
        self.maxpool = nn.MaxPool1d(kernel_size=3, stride=3)
        self.conv2 = nn.Conv1d(18, 32, kernel_size=3)
        self.fc1 = nn.Linear(200, 100)  #What I put in/out here ?
        self.fc2 = nn.Linear(100, 50)
        self.fc3 = nn.Linear(50, 2)

    def forward(self, x):
        x = F.relu(self.mp(self.conv1(x)))
        x = self.maxpool(x)

        x = F.relu(self.mp(self.conv2(x)))
        x = self.maxpool(x)

        x = x.view(-1, ??)  # flatten the tensor, which number here ?

        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)

        return x


title: Filter data in pytorch tensor

body:
I have a tensor X like [0.1, 0.5, -1.0, 0, 1.2, 0], and I want to implement a function called filter_positive(), it can filter the positive data into a new tensor and return the index of the original tensor. For example:

new_tensor, index = filter_positive(X)

new_tensor = [0.1, 0.5, 1.2]
index = [0, 1, 4]

How can I implement this function most efficiently in pytorch?


title: conv2d function in pytorch

body:
I'm trying to use the function torch.conv2d from Pytorch but can't get a result I understand...

Here is a simple example where the kernel (filt) is the same size as the input (im) to explain what I'm looking for.

import pytorch

filt = torch.rand(3, 3)
im = torch.rand(3, 3)

I want to compute a simple convolution with no padding, so the result should be a scalar (i.e. a 1x1 tensor).

I tried this with conv2d:

# I have to convert image and kernel to 4 dimensions tensors to use conv2d
im_torch = im.reshape((im_height, filt_height, 1, 1))
filt_torch = filt.reshape((filt_height, im_height, 1, 1))
out = torch.nn.functional.conv2d(im_torch, filt_torch, stride=1, padding=0)
print(out)

But the result is not what I expected:

tensor([[[[0.6067]], [[0.3564]], [[0.5397]]],
    [[[0.2557]], [[0.0493]], [[0.2562]]],
    [[[0.6067]], [[0.3564]], [[0.5397]]]])

To give an idea of what I'd like, I want to reproduce scipy convolve2d behavior:

import scipy.signal
out_scipy = scipy.signal.convolve2d(im.detach().numpy(), filt.detach().numpy(), 'valid')
print(out_scipy)

which prints:

array([[1.195723]], dtype=float32)


This is the error i get when I try to train my network.

The class we used to store Images from the Caltech 101 dataset was provided us by our teachers.

from torchvision.datasets import VisionDataset

from PIL import Image

import os
import os.path
import sys

def pil_loader(path):
    # open path as file to avoid ResourceWarning (https://github.com/python-pillow/Pillow/issues/835)
    with open(path, 'rb') as f:
        img = Image.open(f)
        return img.convert('RGB')

class Caltech(VisionDataset):
    def __init__(self, root, split='train', transform=None, target_transform=None):
        super(Caltech, self).__init__(root, transform=transform, target_transform=target_transform)

        self.split = split # This defines the split you are going to use
                           # (split files are called 'train.txt' and 'test.txt')

        '''
        - Here you should implement the logic for reading the splits files and accessing elements
        - If the RAM size allows it, it is faster to store all data in memory
        - PyTorch Dataset classes use indexes to read elements
        - You should provide a way for the __getitem__ method to access the image-label pair
          through the index
        - Labels should start from 0, so for Caltech you will have lables 0...100 (excluding the background class)
        '''
        # Open file in read only mode and read all lines
        file = open(self.split, "r")
        lines = file.readlines()

        # Filter out the lines which start with 'BACKGROUND_Google' as asked in the homework
        self.elements = [i for i in lines if not i.startswith('BACKGROUND_Google')]

        # Delete BACKGROUND_Google class from dataset labels
        self.classes = sorted(os.listdir(os.path.join(self.root, "")))
        self.classes.remove("BACKGROUND_Google")

    def __getitem__(self, index):
        '''
        __getitem__ should access an element through its index
        Args:
            index (int): Index
        Returns:
            tuple: (sample, target) where target is class_index of the target class.
        '''

        img = Image.open(os.path.join(self.root, self.elements[index].rstrip()))

        target = self.classes.index(self.elements[index].rstrip().split('/')[0])

        image, label = img, target # Provide a way to access image and label via index
                           # Image should be a PIL Image
                           # label can be int

        # Applies preprocessing when accessing the image
        if self.transform is not None:
            image = self.transform(image)

        return image, label

    def __len__(self):
        '''
        The __len__ method returns the length of the dataset
        It is mandatory, as this is used by several other components
        '''
        # Provides a way to get the length (number of elements) of the dataset
        length =  len(self.elements)
        return length

Whereas the preprocessing phase is done by this code:

# Define transforms for training phase
train_transform = transforms.Compose([transforms.Resize(256),      # Resizes short size of the PIL image to 256
                                      transforms.CenterCrop(224),  # Crops a central square patch of the image
                                                                   # 224 because torchvision's AlexNet needs a 224x224 input!
                                                                   # Remember this when applying different transformations, otherwise you get an error
                                      transforms.ToTensor(), # Turn PIL Image to torch.Tensor
                                      transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)) # Normalizes tensor with mean and standard deviation
])
# Define transforms for the evaluation phase
eval_transform = transforms.Compose([transforms.Resize(256),
                                      transforms.CenterCrop(224),
                                      transforms.ToTensor(),
                                      transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))

In the end this is the preparation of datasets and dataloader:

# Clone github repository with data
if not os.path.isdir('./Homework2-Caltech101'):
  !git clone https://github.com/MachineLearning2020/Homework2-Caltech101.git

# Commands to execute when there is an error saying no file or directory related to ./Homework2-Caltech101/
# !rm -r ./Homework2-Caltech101/
# !git clone https://github.com/MachineLearning2020/Homework2-Caltech101.git

DATA_DIR = 'Homework2-Caltech101/101_ObjectCategories'
SPLIT_TRAIN = 'Homework2-Caltech101/train.txt'
SPLIT_TEST = 'Homework2-Caltech101/test.txt'

# 1 - Data preparation
myTrainDS = Caltech(DATA_DIR, split = SPLIT_TRAIN, transform=train_transform)
myTestDS = Caltech(DATA_DIR, split = SPLIT_TEST, transform=eval_transform)

print('My Train DS: {}'.format(len(myTrainDS)))
print('My Test DS: {}'.format(len(myTestDS)))

# 1 - Data preparation
myTrain_dataloader = DataLoader(myTrainDS, batch_size=BATCH_SIZE, shuffle=True, num_workers=4, drop_last=True)
myTest_dataloader = DataLoader(myTestDS, batch_size=BATCH_SIZE, shuffle=False, num_workers=4)

Okay now the two .txt files contain the lists of images we want to have in the train and test splits, so we have to get them from there, but that should have been done correctly. The thing is that when I approach my training phase (see code later) I am presented the error in the title. I already tried to add the following line in the transform function:

[...]
transforms.Lambda(lambda x: x.repeat(3, 1, 1)),

after the centercrop, but it says that Image has no attribute repeat, so I'm kinda stuck.

The training code line which gives me the error is the following:

# Iterate over the dataset
  for images, labels in myTrain_dataloader:

If needed, full error is:

RuntimeError                              Traceback (most recent call last)

<ipython-input-197-0e4710a9855d> in <module>()
     47
     48   # Iterate over the dataset
---> 49   for images, labels in myTrain_dataloader:
     50
     51     # Bring data over the device of choice

2 frames

/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py in __next__(self)
    817             else:
    818                 del self._task_info[idx]
--> 819                 return self._process_data(data)
    820
    821     next = __next__  # Python 2 compatibility

/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py in _process_data(self, data)
    844         self._try_put_index()
    845         if isinstance(data, ExceptionWrapper):
--> 846             data.reraise()
    847         return data
    848

/usr/local/lib/python3.6/dist-packages/torch/_utils.py in reraise(self)
    383             # (https://bugs.python.org/issue2651), so we work around it.
    384             msg = KeyErrorMessage(msg)
--> 385         raise self.exc_type(msg)

RuntimeError: Caught RuntimeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop
    data = fetcher.fetch(index)
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "<ipython-input-180-0b00b175e18c>", line 72, in __getitem__
    image = self.transform(image)
  File "/usr/local/lib/python3.6/dist-packages/torchvision/transforms/transforms.py", line 70, in __call__
    img = t(img)
  File "/usr/local/lib/python3.6/dist-packages/torchvision/transforms/transforms.py", line 175, in __call__
    return F.normalize(tensor, self.mean, self.std, self.inplace)
  File "/usr/local/lib/python3.6/dist-packages/torchvision/transforms/functional.py", line 217, in normalize
    tensor.sub_(mean[:, None, None]).div_(std[:, None, None])
RuntimeError: output with shape [1, 224, 224] doesn't match the broadcast shape [3, 224, 224]

I'm using Alexnet and the code I was provided is the following:

    net = alexnet() # Loading AlexNet model

# AlexNet has 1000 output neurons, corresponding to the 1000 ImageNet's classes
# We need 101 outputs for Caltech-101
net.classifier[6] = nn.Linear(4096, NUM_CLASSES) # nn.Linear in pytorch is a fully connected layer
                                                 # The convolutional layer is nn.Conv2d

# We just changed the last layer of AlexNet with a new fully connected layer with 101 outputs
# It is mandatory to study torchvision.models.alexnet source code


title: How to create a torchtext.data.TabularDataset directly from a list or dict

body:
torchtext.data.TabularDataset can be created from a TSV/JSON/CSV file and then it can be used for building the vocabulary from Glove, FastText or any other embeddings. But my requirement is to create a torchtext.data.TabularDataset directly, either from a list or a dict.

Current implementation of the code by reading TSV files

self.RAW = data.RawField()
self.TEXT = data.Field(batch_first=True)
self.LABEL = data.Field(sequential=False, unk_token=None)

self.train, self.dev, self.test = data.TabularDataset.splits(
    path='.data/quora',
    train='train.tsv',
    validation='dev.tsv',
    test='test.tsv',
    format='tsv',
    fields=[('label', self.LABEL),
            ('q1', self.TEXT),
            ('q2', self.TEXT),
            ('id', self.RAW)])

self.TEXT.build_vocab(self.train, self.dev, self.test, vectors=GloVe(name='840B', dim=300))
self.LABEL.build_vocab(self.train)

sort_key = lambda x: data.interleave_keys(len(x.q1), len(x.q2))

self.train_iter, self.dev_iter, self.test_iter = \
    data.BucketIterator.splits((self.train, self.dev, self.test),
                               batch_sizes=[args.batch_size] * 3,
                               device=args.gpu,
                               sort_key=sort_key)

This is the current working code for reading data from a file. So in order to create the dataset directly from a List/Dict I tried inbuilt functions like Examples.fromDict or Examples.fromList but then while coming to the last for loop, it throws an error that AttributeError: 'BucketIterator' object has no attribute 'q1'


title: Importing PyTorch in PyCharm using Anaconda

body: I just installed PyCharm and Anaconda. I installed PyTorch to Anaconda and i can even use "import torch" in Anaconda. I've created a new Project in PyCharm with the Anaconda Interpreter but i still can't use PyTorch in PyCharm.


title: Does a clean and extendable LSTM implementation exists in PyTorch?

body:
I would like to create an LSTM class by myself, however, I don't want to rewrite the classic LSTM functions from scratch again.

Digging in the code of PyTorch, I only find a dirty implementation involving at least 3-4 classes with inheritance:

https://github.com/pytorch/pytorch/blob/98c24fae6b6400a7d1e13610b20aa05f86f77070/torch/nn/modules/rnn.py#L323
https://github.com/pytorch/pytorch/blob/98c24fae6b6400a7d1e13610b20aa05f86f77070/torch/nn/modules/rnn.py#L12
https://github.com/pytorch/pytorch/blob/98c24fae6b6400a7d1e13610b20aa05f86f77070/torch/nn/_functions/rnn.py#L297

Does a clean PyTorch implementation of an LSTM exist somewhere? Any links would help.

For example, I know that clean implementations of a LSTM exists in TensorFlow, but I would need to derive a PyTorch one.

For a clear example, what I'm searching for is an implementation as clean as this, but in PyTorch:


title: when importing pytorch microsoft visual C++ Redistributable is not installed

body:
I work in a windows machine with GPU.
I have installed pytorch in a conda environment with
conda install pytorch torchvision cudatoolkit=10.1 -c pytorch

then I run python and inside of python I do  import torch and I get this error
           Python 3.6.10 |Anaconda, Inc.| (default, May  7 2020, 19:46:08) [MSC v.1916 64 bit (AMD64)] on win32
                    Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
Microsoft Visual C++ Redistributable is not installed, this may lead to the DLL load failure.
It can be downloaded at https://aka.ms/vs/16/release/vc_redist.x64.exe
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
File "C:\Users\aliag\anaconda3\envs\pytorchPractice\lib\site-packages\torch\__init__.py", line 127, in <module>
raise err
OSError: [WinError 126] 指定されたモジュールが見つかりません。
Error loading "C:\Users\aliag\anaconda3\envs\pytorchPractice\lib\site-packages\torch\lib\asmjit.dll"
or one of its dependencies.
>>>

How can I correct this error?


title: HTTP Error 503: Service Unavailable when trying to download MNIST data

body:
I'm trying to run code that I wrote a week ago on Google Colab (and it worked), but I'm getting this error now for some reason.
#libraries
import torch
import torchvision
from torchvision import datasets, transforms
import torch.nn as nn

train_loader = torch.utils.data.DataLoader(
  torchvision.datasets.MNIST(root='./data', train=True, download=True,
                             transform=torchvision.transforms.Compose([
                               torchvision.transforms.ToTensor(),
                               torchvision.transforms.Normalize(
                                 (0.1307,), (0.3081,))
                             ])),
  batch_size=batch_size_train, shuffle=True)

test_loader = torch.utils.data.DataLoader(
  torchvision.datasets.MNIST(root='./data', train=False, download=True,
                             transform=torchvision.transforms.Compose([
                               torchvision.transforms.ToTensor(),
                               torchvision.transforms.Normalize(
                                 (0.1307,), (0.3081,))
                             ])),
  batch_size=batch_size_test, shuffle=True)

Output:
Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz to ./data/MNIST/raw/train-images-idx3-ubyte.gz
---------------------------------------------------------------------------
HTTPError                                 Traceback (most recent call last)
<ipython-input-3-03d990013546> in <module>()
      6                                torchvision.transforms.ToTensor(),
      7                                torchvision.transforms.Normalize(
----> 8                                  (0.1307,), (0.3081,))
      9                              ])),
     10   batch_size=batch_size_train, shuffle=True)

11 frames
/usr/lib/python3.7/urllib/request.py in http_error_default(self, req, fp, code, msg, hdrs)
    647 class HTTPDefaultErrorHandler(BaseHandler):
    648     def http_error_default(self, req, fp, code, msg, hdrs):
--> 649         raise HTTPError(req.full_url, code, msg, hdrs, fp)
    650
    651 class HTTPRedirectHandler(BaseHandler):

HTTPError: HTTP Error 503: Service Unavailable

I'm not entirely sure what's wrong. Could they have moved the dataset?
Update:
Looks like you can't access the mnist dataset here anymore


title: NLP Transformers: Best way to get a fixed sentence embedding-vector shape?

body:
I'm loading a language model from torch hub (CamemBERT a French RoBERTa-based model) and using it do embed some french sentences:

import torch
camembert = torch.hub.load('pytorch/fairseq', 'camembert.v0')
camembert.eval()  # disable dropout (or leave in train mode to finetune)

def embed(sentence):
   tokens = camembert.encode(sentence)
   # Extract all layer's features (layer 0 is the embedding layer)
   all_layers = camembert.extract_features(tokens, return_all_hiddens=True)
   embeddings = all_layers[0]
   return embeddings

# Here we see that the shape of the embedding vector depends on the number of tokens in the sentence

u = embed(sentence="Bonjour, ça va ?")
u.shape # torch.Size([1, 7, 768])
v = embed(sentence="Salut, comment vas-tu ?")
v.shape # torch.Size([1, 9, 768])

Imagine now in order to do some semantic search, I want to calculate the cosine distance between the vectors (tensors in our case) u and v :

cos = torch.nn.CosineSimilarity(dim=1)
cos(u, v) # will throw an error since the shape of `u` is different from the shape of `v`

I'm asking what is the best method to use in order to always get the same embedding shape for a sentence regardless the count of its tokens?

=> The first solution I'm thinking of is calculating the mean on axis=1 (embedding of a sentence is the mean embedding its tokens) since axis=0 and axis=2 have always the same size:

cos = torch.nn.CosineSimilarity(dim=1)
cos(u.mean(axis=1), v.mean(axis=1)) # works now and gives 0.7269

But, I'm afraid that I'm hurting the embedding of the sentence when calculating the mean since it gives the same weight for each token (maybe multiplying by TF-IDF?).

=> The second solution is to pad shorter sentences out. That means:

giving a list of sentences to embed at a time (instead of embedding sentence by sentence)
look up for the sentence with the longest tokens and embed it, get its shape S
for the rest of sentences embed then pad zero to get the same shape S (the sentence has 0 in the rest of dimensions)

What are your thoughts?
What other techniques would you use and why?

Thanks in advance!


title: Python Neural Network: 'numpy.ndarray' object has no attribute 'dim'

body:
I am using this database for modeling
http://archive.ics.uci.edu/ml/datasets/Car+Evaluation

after preprocessing

    X_train = df.drop('class', axis=1).to_numpy()
    y_train = df['class'].to_numpy()

    X_train, X_test, y_train, y_test = train_test_split(X_train, y_train, test_size=0.2)

class

class network(nn.Module):

def __init__(self, input_size, hidden1_size, hidden2_size, num_classes):
    super(network, self).__init__()
    self.fc1 = nn.Linear(input_size, hidden1_size)
    self.relu1 = nn.ReLU()
    self.fc2 = nn.Linear(hidden1_size, hidden2_size)
    self.relu2 = nn.ReLU()
    self.fc3 = nn.Linear(hidden2_size, num_classes)

  def forward(self, x):
    out = self.fc1(x)
    out = self.relu1(out)
    out = self.fc2(out)
    out = self.relu2(out)
    out = self.fc3(out)
    return out

net = network(input_size=6, hidden1_size=5, hidden2_size=4, num_classes=4)
optimizer = torch.optim.SGD(net.parameters(), lr=0.2)
loss_func = torch.nn.MSELoss()

Error is in this block

plt.ion()

for t in range(200):
    prediction = net(X_train)     # input x and predict based on x

    loss = loss_func(prediction, y_train)     # must be (1. nn output, 2. target)

    optimizer.zero_grad()   # clear gradients for next train
    loss.backward()         # backpropagation, compute gradients
    optimizer.step()        # apply gradients

    if t % 5 == 0:
        # plot and show learning process
        plt.cla()
        plt.scatter(x.data.numpy(), y.data.numpy())
        plt.plot(x.data.numpy(), prediction.data.numpy(), 'r-', lw=5)
        plt.text(0.5, 0, 'Loss=%.4f' % loss.data.numpy(), fontdict={'size': 20, 'color':  'red'})
        plt.pause(0.1)

plt.ioff()
plt.show()

Error message

AttributeError                            Traceback (most recent call last)
 in ()
      2
      3 for t in range(200):
----> 4     prediction = net(X_train)     # input x and predict based on x
      5
      6     loss = loss_func(prediction, y_train)     # must be (1. nn output, 2. target)

> 4 frames /usr/local/lib/python3.6/dist-packages/torch/nn/functional.py
> in linear(input, weight, bias)    1606         if any([type(t) is not
> Tensor for t in tens_ops]) and has_torch_function(tens_ops):    1607
> return handle_torch_function(linear, tens_ops, input, weight,
> bias=bias)
> -> 1608     if input.dim() == 2 and bias is not None:    1609         # fused op is marginally faster    1610         ret = torch.addmm(bias, input, weight.t())
>
> AttributeError: 'numpy.ndarray' object has no attribute 'dim'


title: How are the pytorch dimensions for linear layers calculated?

body:
In the PyTorch tutorial, the constructed network is

Net(
  (conv1): Conv2d(1, 6, kernel_size=(5, 5), stride=(1, 1))
  (conv2): Conv2d(6, 16, kernel_size=(5, 5), stride=(1, 1))
  (fc1): Linear(in_features=400, out_features=120, bias=True)
  (fc2): Linear(in_features=120, out_features=84, bias=True)
  (fc3): Linear(in_features=84, out_features=10, bias=True)
)

And used to process images with dimensions 1x32x32. They mention, that the network cannot be used with images with a different size.

The two convolutional layers seem to allow for an arbitrary number of features, so the linear layers seem to be related to getting the 32x32 into into 10 final features.

I do not really understand, how the numbers 120 and 84 are chosen there and why the result matches with the input dimensions.

And when I try to construct a similar network, I actually get the problem with the dimension of the data.

When I for example use a simpler network:

Net(
  (conv1): Conv2d(3, 8, kernel_size=(5, 5), stride=(1, 1))
  (conv2): Conv2d(8, 16, kernel_size=(5, 5), stride=(1, 1))
  (fc1): Linear(in_features=400, out_features=3, bias=True)
)

for an input of the size 3x1200x800, I get the error message:

RuntimeError: size mismatch, m1: [1 x 936144], m2: [400 x 3] at /pytorch/aten/src/TH/generic/THTensorMath.cpp:940

Where does the number 936144 come from and how do I need to design the network, such that the dimensions are matching?


title: AttributeError: 'torch.return_types.max' object has no attribute 'dim' - Maxpooling Channel

body:
I'm trying to do maxpooling over channel dimension:

class ChannelPool(nn.Module):
    def forward(self, input):
        return torch.max(input, dim=1)

but I get the error

AttributeError: 'torch.return_types.max' object has no attribute 'dim'


title: Pytorch maxpooling over channels dimension

body: I was trying to build a cnn to with Pytorch, and had difficulty in maxpooling. I have taken the cs231n held by Stanford. As I recalled, maxpooling can be used as a dimensional deduction step, for example, I have this (1, 20, height, width) input ot max_pool2d (assuming my batch_size is 1). And if I use (1, 1) kernel, I want to get output like this: (1, 1, height, width), which means the kernel should be slide over the channel dimension. However, after checking the pytorch docs, it says the kernel slides over height and width. And thanks to @ImgPrcSng on Pytorch forum who told me to use max_pool3d, and it turned out worked well. But there is still a reshape operation between the output of the conv2d layer and the input of the max_pool3d layer. So it is hard to be aggregated into a nn.Sequential, so I wonder is there another way to do this?


title: (pytorch) I want to normalize [0 255] integer tensor to [0 1] float tensor

body:
I want to normalize [0 255] integer tensor to [0 1] float tensor.

I used cifar10 dataset and wanted to deal with integer image tensor.
so I made them integer tensor
when I loaded dataset, I used "transforms.ToTensor()" so the values were set to [0 1] float

tensor([[[0.4588, 0.4588, 0.4588,  ..., 0.4980, 0.4980, 0.5020],
         [0.4706, 0.4706, 0.4706,  ..., 0.5098, 0.5098, 0.5137],
         [0.4824, 0.4824, 0.4824,  ..., 0.5216, 0.5216, 0.5294],
         ...,
         [0.3098, 0.3020, 0.2863,  ..., 0.4549, 0.3608, 0.3137],
         [0.2902, 0.2902, 0.2902,  ..., 0.4431, 0.3333, 0.3020],
         [0.2706, 0.2941, 0.2941,  ..., 0.4157, 0.3529, 0.3059]],

        [[0.7725, 0.7725, 0.7725,  ..., 0.7569, 0.7569, 0.7608],
         [0.7765, 0.7765, 0.7765,  ..., 0.7608, 0.7608, 0.7686],
         [0.7765, 0.7765, 0.7765,  ..., 0.7608, 0.7608, 0.7725],
         ...,
         [0.6510, 0.6314, 0.6078,  ..., 0.6941, 0.6510, 0.6392],
         [0.6314, 0.6235, 0.6118,  ..., 0.6784, 0.6196, 0.6275],
         [0.6157, 0.6235, 0.6157,  ..., 0.6549, 0.6431, 0.6314]],

To make them [0 255] integer tensor.

temp = np.floor(temp_images*256)
temp_int = torch.tensor(temp, dtype=torch.int32)
temp_images = torch.clamp(temp, 0, 255)

and the result was

torch.IntTensor
tensor([[[[ 94., 100., 100.,  ...,  98., 100., 102.],
          [ 86., 100., 101.,  ...,  83.,  91., 103.],
          [ 90., 100.,  99.,  ...,  80.,  66.,  86.],
          ...,
          [ 92.,  92.,  90.,  ...,  77., 107., 119.],
          [ 76.,  91., 100.,  ...,  95., 158., 170.],
          [ 86.,  83.,  87.,  ...,  97., 176., 205.]],

         [[105., 111., 111.,  ..., 109., 112., 113.],
          [ 97., 111., 112.,  ...,  94., 102., 114.],
          [101., 111., 110.,  ...,  90.,  77.,  97.],
          ...,
          [111., 110., 108.,  ...,  88., 120., 131.],
          [ 95., 108., 114.,  ..., 105., 165., 172.],
          [106., 100., 101.,  ..., 108., 183., 206.]],

         [[ 62.,  68.,  68.,  ...,  66.,  68.,  70.],
          [ 55.,  69.,  70.,  ...,  51.,  59.,  71.],
          [ 59.,  69.,  68.,  ...,  48.,  34.,  54.],
          ...,
          [ 59.,  59.,  56.,  ...,  54.,  95., 107.],
          [ 49.,  61.,  66.,  ...,  76., 152., 166.],
          [ 61.,  55.,  54.,  ...,  73., 170., 206.]]],

before forwarding them to the network,
I want to make them [0 1] float tensor again.

So I tried

transforms.Normalize((0.5,0.5,0.5),(0.5,0.5,0.5))

But, the result is not normalized to [0 1] and rather it becomes bigger...!

tensor([[[117., 117., 117.,  ..., 127., 127., 128.],
         [120., 120., 120.,  ..., 130., 130., 131.],
         [123., 123., 123.,  ..., 133., 133., 135.],
         ...,
         [ 79.,  77.,  73.,  ..., 116.,  92.,  80.],
         [ 74.,  74.,  74.,  ..., 113.,  85.,  77.],
         [ 69.,  75.,  75.,  ..., 106.,  90.,  78.]],

        [[197., 197., 197.,  ..., 193., 193., 194.],
         [198., 198., 198.,  ..., 194., 194., 196.],
         [198., 198., 198.,  ..., 194., 194., 197.],

to

tensor([[[233., 233., 233.,  ..., 253., 253., 255.],
         [239., 239., 239.,  ..., 259., 259., 261.],
         [245., 245., 245.,  ..., 265., 265., 269.],
         ...,
         [157., 153., 145.,  ..., 231., 183., 159.],
         [147., 147., 147.,  ..., 225., 169., 153.],
         [137., 149., 149.,  ..., 211., 179., 155.]],

        [[393., 393., 393.,  ..., 385., 385., 387.],
         [395., 395., 395.,  ..., 387., 387., 391.],
         [395., 395., 395.,  ..., 387., 387., 393.],
         ...,
         [331., 321., 309.,  ..., 353., 331., 325.],
         [321., 317., 311.,  ..., 345., 315., 319.],
         [313., 317., 313.,  ..., 333., 327., 321.]],

How I can normalize [0 255] integer tensor to [0 1] float tensor?


title: Pytorch: AdamW and Adam with weight decay

body: Is there any difference between torch.optim.Adam(weight_decay=0.01) and torch.optim.AdamW()? Link to the docs: torch.optim


title: Create a torch::Tensor from C/C++ array without using "from_blob(...)..."

body:
Using the C++ libtorch frontend for Pytorch
I want to create a  torch::Tensor from a C++ double[] array.  Comming from a legacy C/C++ API.
I could not find a simple documentation about the subject not in docs nor in the forums.
Something like:
double array[5] = {1, 2, 3, 4, 5};
auto tharray = torch::Tensor(array, 5, torch::Device(torch::kCUDA));

The only thing I found is to use torch::from_blob but then I would have to clone() and use to(device) if I wanted to use it with CUDA.
double array[] = { 1, 2, 3, 4, 5};
auto options = torch::TensorOptions().dtype(torch::kFloat64);
torch::Tensor tharray = torch::from_blob(array, {5}, options);

Is there any cleaner way of doing so?


title: Why does autograd not produce gradient for intermediate variables?

body:
trying to wrap my head around how gradients are represented and how autograd works:

import torch
from torch.autograd import Variable

x = Variable(torch.Tensor([2]), requires_grad=True)
y = x * x
z = y * y

z.backward()

print(x.grad)
#Variable containing:
#32
#[torch.FloatTensor of size 1]

print(y.grad)
#None

Why does it not produce a gradient for y? If y.grad = dz/dy, then shouldn't it at least produce a variable like y.grad = 2*y?


title: Pytorch: RuntimeError: expected dtype Float but got dtype Long

body:
I encounter this weird error when building a simple NN in Pytorch. I dont understand this error and why this consern Long and Float datatype in backward function. Anyone encounter this before? Thanks for any help.
Traceback (most recent call last):
  File "test.py", line 30, in <module>
    loss.backward()
  File "/home/liuyun/anaconda3/envs/torch/lib/python3.7/site-packages/torch/tensor.py", line 198, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/liuyun/anaconda3/envs/torch/lib/python3.7/site-packages/torch/autograd/__init__.py", line 100, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: expected dtype Float but got dtype Long (validate_dtype at /opt/conda/conda-bld/pytorch_1587428398394/work/aten/src/ATen/native/TensorIterator.cpp:143)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x4e (0x7f5856661b5e in /home/liuyun/anaconda3/envs/torch/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #1: at::TensorIterator::compute_types() + 0xce3 (0x7f587e3dc793 in /home/liuyun/anaconda3/envs/torch/lib/python3.7/site
-packages/torch/lib/libtorch_cpu.so)
frame #2: at::TensorIterator::build() + 0x44 (0x7f587e3df174 in /home/liuyun/anaconda3/envs/torch/lib/python3.7/site-packages
/torch/lib/libtorch_cpu.so)
frame #3: at::native::smooth_l1_loss_backward_out(at::Tensor&, at::Tensor const&, at::Tensor const&, at::Tensor const&, long)
 + 0x193 (0x7f587e22cf73 in /home/liuyun/anaconda3/envs/torch/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #4: <unknown function> + 0xe080b7 (0x7f58576960b7 in /home/liuyun/anaconda3/envs/torch/lib/python3.7/site-packages/torc
h/lib/libtorch_cuda.so)
frame #5: at::native::smooth_l1_loss_backward(at::Tensor const&, at::Tensor const&, at::Tensor const&, long) + 0x16e (0x7f587
e23569e in /home/liuyun/anaconda3/envs/torch/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #6: <unknown function> + 0xed98af (0x7f587e71c8af in /home/liuyun/anaconda3/envs/torch/lib/python3.7/site-packages/torc
h/lib/libtorch_cpu.so)
frame #7: <unknown function> + 0xe22286 (0x7f587e665286 in /home/liuyun/anaconda3/envs/torch/lib/python3.7/site-packages/torc
h/lib/libtorch_cpu.so)

Here is the source code:
import torch
import torch.nn as nn
import numpy as np
import torchvision
from torchvision import models
from UTKLoss import MultiLoss
from ipdb import set_trace

# out features [13, 2, 5]
model_ft = models.resnet18(pretrained=True)
num_ftrs = model_ft.fc.in_features
model_ft.fc = nn.Linear(num_ftrs, 20)
model_ft.cuda()

criterion = MultiLoss()
optimizer = torch.optim.Adam(model_ft.parameters(), lr = 1e-3)

image = torch.randn((1, 3, 128, 128)).cuda()
age = torch.randint(110, (1,)).cuda()
gender = torch.randint(2, (1,)).cuda()
race = torch.randint(5, (1,)).cuda()
optimizer.zero_grad()
output = model_ft(image)
age_loss, gender_loss, race_loss = criterion(output, age, gender, race)
loss = age_loss + gender_loss + race_loss
loss.backward()
optimizer.step()

Here is what I define my loss function
import torch
import torch.nn as nn
import torch.nn.functional as F

class MultiLoss(nn.Module):
    def __init__(self):
        super().__init__()

    def forward(self, output, age, gender, race):
        age_pred = output[:, :13]
        age_pred = torch.sum(age_pred, 1)
        gender_pred = output[:, 13: 15]
        race_pred = output[:, 15:]
        age_loss = F.smooth_l1_loss(age_pred.view(-1, 1), age.cuda())
        gender_loss = F.cross_entropy(gender_pred, torch.flatten(gender).cuda(), reduction='sum')
        race_loss = F.cross_entropy(race_pred, torch.flatten(race).cuda(), reduction='sum')
        return age_loss, gender_loss, race_loss