title: How to fit mathematical formula with neural network?

body:
I would like to fit a pytorch feed forward network on a crafted dataset with dependency between labels y and two features from the dataset.

Dataset is generated using np.random.random_sample for a distribution between 0 and 1 and label is computed using the two functions below:

sum_bin_label
sum_mod_label

The first function I can see that both training and validation loss of the neural network is decreasing and eventually it is able to approximate the function with close to 100%, what is expected, but for the second function that is using sum and modulo(num_classes) it is unable to make any progress. I have tried multiple learning rates and network architectures but did not manage to fit it.

I am interested to see how that function can be fitted.

Bellow is a simple example that can be pasted directly to a jupyter notebook or any kind of python repl for that matter.

Thanks in advance!

Imports

import torch
import numpy as np
from sklearn.model_selection import train_test_split
import torch.utils.data as utils
DATASHAPE = (2000, 2)
NUM_CLASSES = 3

Functions and classes used

def sum_mod_label(x):
    return np.array([x for x in map(
        lambda x: x % NUM_CLASSES, map(int, (x[:, 0] + x[:, 1]) * 100))])

def sum_bin_label(x):
    def binit(x):
        if x < 0.807:
            return 0
        if x < 1.169:
            return 1
        return 2

    return np.array(
        [x for x in map(lambda x: binit(x), x[:, 0] + x[:, 1])])

class RandomModuloDataset(utils.Dataset):
    def __init__(self, shape, label_fn):
        self.data = np.random.random_sample(shape)
        self.label = label_fn(self.data)

    def __len__(self):
        return len(self.data)

    def __getitem__(self, idx):
        return self.data[idx, :], self.label[idx]

class FeedForward(torch.nn.Module):
    def __init__(self, input_size, num_classes):
        super().__init__()
        self.input_size = input_size
        self.num_classes = num_classes

        self.relu = torch.nn.ReLU()
        self.softmax = torch.nn.Softmax(dim=-1)

        self.fc1 = torch.nn.Linear(
            self.input_size, self.input_size)

        self.fc2 = torch.nn.Linear(
            self.input_size, self.num_classes)

    def forward(self, x, **kwargs):
        output = self.fc2(self.relu(self.fc1(x.float())))
        return self.softmax(output)

def fitit(trainloader, epochs=10):
    neurons = DATASHAPE[1]

    net = FeedForward(neurons, NUM_CLASSES)
    criterion = torch.nn.CrossEntropyLoss()
    optimizer = torch.optim.SGD(net.parameters(), lr=0.001, momentum=0.9)

    for epoch in range(epochs):
        for i, data in enumerate(trainloader, 0):
            inputs, labels = data

            optimizer.zero_grad()

            outputs = net(inputs)
            loss = criterion(outputs, labels)
            loss.backward()
            optimizer.step()

        print('[%d] loss: %.3f' %
              (epoch + 1, loss.item()))

Iteration with first function (eventually converges)

sum_bin_tloader = utils.DataLoader(
    RandomModuloDataset(DATASHAPE, sum_bin_label))
fitit(sum_bin_tloader, epochs=50)

[1] loss: 1.111
[2] loss: 1.133
[3] loss: 1.212
[4] loss: 1.264
[5] loss: 1.261
[6] loss: 1.199
[7] loss: 1.094
[8] loss: 1.011
[9] loss: 0.958
[10] loss: 0.922
[11] loss: 0.896
[12] loss: 0.876
[13] loss: 0.858
[14] loss: 0.844
[15] loss: 0.831
[16] loss: 0.820
[17] loss: 0.811
[18] loss: 0.803
[19] loss: 0.795
[20] loss: 0.788
[21] loss: 0.782
[22] loss: 0.776
[23] loss: 0.771
[24] loss: 0.766
[25] loss: 0.761
[26] loss: 0.757
[27] loss: 0.753
[28] loss: 0.749
[29] loss: 0.745
[30] loss: 0.741
[31] loss: 0.738
[32] loss: 0.734
[33] loss: 0.731
[34] loss: 0.728
[35] loss: 0.725
[36] loss: 0.722
[37] loss: 0.719
[38] loss: 0.717
[39] loss: 0.714
[40] loss: 0.712
[41] loss: 0.709
[42] loss: 0.707
[43] loss: 0.705
[44] loss: 0.703
[45] loss: 0.701
[46] loss: 0.699
[47] loss: 0.697
[48] loss: 0.695
[49] loss: 0.693
[50] loss: 0.691

Iteration with second function (does not converge)

sum_mod_tloader = utils.DataLoader(
    RandomModuloDataset(DATASHAPE, sum_mod_label))
fitit(sum_mod_tloader, epochs=50)

[1] loss: 1.059
[2] loss: 1.065
[3] loss: 1.079
[4] loss: 1.087
[5] loss: 1.091
[6] loss: 1.092
[7] loss: 1.092
[8] loss: 1.092
[9] loss: 1.092
[10] loss: 1.091
[11] loss: 1.091
[12] loss: 1.091
[13] loss: 1.091
[14] loss: 1.091
[15] loss: 1.090
[16] loss: 1.090
[17] loss: 1.090
[18] loss: 1.090
[19] loss: 1.090
[20] loss: 1.090
[21] loss: 1.090
[22] loss: 1.089
[23] loss: 1.089
[24] loss: 1.089
[25] loss: 1.089
[26] loss: 1.089
[27] loss: 1.089
[28] loss: 1.089
[29] loss: 1.089
[30] loss: 1.089
[31] loss: 1.089
[32] loss: 1.089
[33] loss: 1.089
[34] loss: 1.089
[35] loss: 1.089
[36] loss: 1.089
[37] loss: 1.089
[38] loss: 1.089
[39] loss: 1.089
[40] loss: 1.089
[41] loss: 1.089
[42] loss: 1.089
[43] loss: 1.089
[44] loss: 1.089
[45] loss: 1.089
[46] loss: 1.089
[47] loss: 1.089
[48] loss: 1.089
[49] loss: 1.089
[50] loss: 1.089

I expect to be able to fit both functions, since NN should be able to find any function y=f(x) describing the dependend variable, but the training is not progressing for sum_mod_label.

Using catboost I was able to get reasonable accuracy (~75% on the sum_mod_label)


title: How to know if python process is garbage collecting too often?

body:
I am tuning the performance of a python job (that uses Pytorch and Nvidia Cuda).  My python process runs on a shared cluster where the max RAM needed must be explicitly specified. Since the lower is better, we tend to set lower RAM that doesn't cause Out of Memory.

In particular, I noticed that the RAM could make quite a difference on performance. For instance, if I set RAM=6GB, my job takes 27Hours. If I set max RAM=10GB (keeping all other variables same), the same job takes about ~16Hrs. Most of the work is done on GPU using GPU RAM, so the CPU and  CPU RAM is only for housekeeping and moving tensors.

My suspect is garbage collector runs too often when I set less RAM. I had observed this kind of behavior when I was dealing with JavaVM, and I had tools to inspect how much time my JVM process spends in the garbage collector.

However, with python, I am clueless.

Are there any ways to inspect the memory management, in particular, Garbage Collector time (fraction of runtime)?


title: NeuroX, ValueError: badly formed hexadecimal UUID string

body:
Now this is probably a long shot, but I have a question about a certain flask app called NeuroX, https://github.com/fdalvi/NeuroX. It is a flask app for analyzing neural networks. The details are not that important, but you run the app and enter paths to model and text files. When trying to run the app, I get the following error.

builtins.ValueError

ValueError: badly formed hexadecimal UUID string
Traceback (most recent call last)

    File "/home/anaconda3/envs/neurox-env/lib/python3.6/site-packages/flask/app.py", line 2309, in __call__

    return self.wsgi_app(environ, start_response)

    File "/home/anaconda3/envs/neurox-env/lib/python3.6/site-packages/flask/app.py", line 2295, in wsgi_app

    response = self.handle_exception(e)

    File "/home/anaconda3/envs/neurox-env/lib/python3.6/site-packages/flask/app.py", line 1741, in handle_exception

    reraise(exc_type, exc_value, tb)

    File "/home/anaconda3/envs/neurox-env/lib/python3.6/site-packages/flask/_compat.py", line 35, in reraise

    raise value

    File "/home/anaconda3/envs/neurox-env/lib/python3.6/site-packages/flask/app.py", line 2292, in wsgi_app

    response = self.full_dispatch_request()

    File "/home/anaconda3/envs/neurox-env/lib/python3.6/site-packages/flask/app.py", line 1815, in full_dispatch_request

    rv = self.handle_user_exception(e)

    File "/home/anaconda3/envs/neurox-env/lib/python3.6/site-packages/flask/app.py", line 1718, in handle_user_exception

    reraise(exc_type, exc_value, tb)

    File "/home/anaconda3/envs/neurox-env/lib/python3.6/site-packages/flask/_compat.py", line 35, in reraise

    raise value

    File "/home/anaconda3/envs/neurox-env/lib/python3.6/site-packages/flask/app.py", line 1813, in full_dispatch_request

    rv = self.dispatch_request()

    File "/home/anaconda3/envs/neurox-env/lib/python3.6/site-packages/flask/app.py", line 1799, in dispatch_request

    return self.view_functions[rule.endpoint](**req.view_args)

    File "/home/Desktop/neurox/NeuroX/src/app.py", line 235, in analyze

    project = database.Project.get(database.Project.id == uuid.UUID(project_id))

    File "/home/anaconda3/envs/neurox-env/lib/python3.6/uuid.py", line 140, in __init__

    raise ValueError('badly formed hexadecimal UUID string')

    ValueError: badly formed hexadecimal UUID string

Since I am not expecting anyone to go through this, my question is simply, does anyone have an idea what causes this? Is it because I am running Ubuntu?


title: How to go to definition of a pytorch function in vscode

body:
When I try the vscode remote SSH, I find that it can not go to definition of some pytorch function such as torch.device, but some function is normal such as torch.tensor.

Is it a error or it is normal for vscode? Thanks!


title: How to load(pytorch) bigger dataset in to GPU and do mini batch training?

body:
as we know, batch size affects training accuracy. (smaller batch size with higher acc.). but using smaller batch size also means longer training time.
Assume there are larger memory in GPU (ex. Nvidia RTX 2080 with 8GB memory on GPU). Is there a way to load bigger data size into GPU and train with smaller dataset, in order to minimize the data access between CPU and GPU also get the better accuracy with mini training batch?


title: how to load a .pb vgg16 trained model

body:
I am trying to load a trained model for image beautification saved at local device through the code below:

# load network
network_pkl = misc.locate_network_pkl(args.results_dir)
print('Loading network from "%s"...' % network_pkl)
G, D, Gs = misc.load_network_pkl(args.results_dir, None)

def load_network_pkl(run_id_or_result_subdir_or_network_pkl, snapshot=None):
    return load_pkl(locate_network_pkl(run_id_or_result_subdir_or_network_pkl, snapshot))

def load_pkl(filename):
    with open(filename, 'rb') as file:
         return legacy.LegacyUnpickler(file, encoding='latin').load()

def save_pkl(obj, filename):
    with open(filename, 'wb') as file:
        pickle.dump(obj, file, protocol=pickle.HIGHEST_PROTOCOL)

and I got this error as below:

  Initializing TensorFlow... Loading network from
  "C:\Users\wissa\Desktop\Out-source-projects/vgg_tf.pb"... Traceback
  (most recent call last):   File "beautify_image.py", line 46, in

      G, D, Gs = misc.load_network_pkl(args.results_dir, None)   File "C:\Users\wissa\Desktop\Out-source-projects\Beholder-GAN-master\misc.py",
  line 674, in load_network_pkl
      return load_pkl(locate_network_pkl(run_id_or_result_subdir_or_network_pkl,
  snapshot))   File
  "C:\Users\wissa\Desktop\Out-source-projects\Beholder-GAN-master\misc.py",
  line 436, in load_pkl
      return pickle.load(file)
  _pickle.UnpicklingError: invalid load key, ', ' '.

could you please help me to fix it?


title: How to transform a 2D and index tensors for torch.nn.utils.rnn.pack_sequence

body:
I have a sequences collection in the following form:

sequences = torch.tensor([[2,1],[5,6],[3,0])
indexes = torch.tensor([1,0,1])

that is, the sequence 0 is made of just [5,6], and the sequence 1 is made of [2,1] , [3,0]. Mathematically sequence[i] = { sequences[j] such that i = indexes[j] }

I need to feed these sequences into an LSTM. Since these are variable-length sequences, pytorch documentation states to use something like torch.nn.utils.rnn.pack_sequence.

Sadly, this method and its like want, as input, a list of tensors where each of them is a L x *, with L being the length of the single sequence.

How can build something that can be fed into a pytorch LSTM?

P.s. throughout the code I work with these tensors using scatter and gather functionalities but I can't find a way to use them to achieve this goal.


title: multinomial() missing 1 required positional arguments: "num_samples"

body:
I have the following issue running the script map.py
map.py call ai.py for a Self Driving Car test visualization:

File "ai.py", line 64, in select_action
  action = probs.multinomial()

I use:

Spyder 4.1.3 form Anaconda 1.9.12 with python 3.6.10
Pytorch 1.5.0

Code is in the following Git repository: https://github.com/renedayala/ai


title: PyTorch VERY different results on different machines using docker and CPU

body:
I am trying to create some CI tests for a deep learning model and therefore created a Dockerfile to install PyTorch and all my requirements and finally run the tests.
FROM pytorch/pytorch
ADD . / project/
RUN (cd project/; pip install -r requirements.txt)
CMD ( cd project/; pytest -v --cov=my_project)

The tests are basically computing an image from 0 - 1  and comparing it to a reference image (saved as npy). The test is checking
if the avg L2 norm of the pixels is below a threshold of 1e-7.
diff_image = np.linalg.norm(target_image_np - reference_image_np, axis=2)
avg_error = np.mean(diff_image)
assert avg_error < 1e-7

The tests pass 12/15 test cases. However, 3 cases are failing quite badly.
=========================== short test summary info ============================
FAILED test_nst.py::test_nst_gatys - assert 0.0021541715 < 1e-07
FAILED test_nst.py::test_nst_gatys_style - assert 0.12900369 < 1e-07
FAILED test_nst.py::test_nst_wct - assert 0.027357593 < 1e-07
=================== 3 failed, 12 passed in 670.27s (0:11:10) ===================

The weird thing is that this ONLY happens on the CI server. On my local machine, all tests pass. Does anybody have an idea why this is happening? As far as I know, using the CPU as well as fixed seeds should return at least results which differ only numerically.
Thanks for any feedback!


title: Pytorch Simple Linear Sigmoid Network not learning

body:
I'm learning pytorch and tried to train a network as an XOR gate. Everything runs smoothly, but it just does not learn. It does changes it weights, yet it converges in a result for every input that is way out of the expected results.
I have tried with many learning rates and weights initialization.
So the inputs are A and B gates and it should return 1 if both are equals or 0 otherwise, like this :

    [0,0] => 1
    [0,1] => 0
    [1,0] => 0
    [1,1] => 1

This is my attempt of modeling and training the model:

    import torch as torch
    import torch.nn as nn

    class Network(nn.Module):

        def __init__(self):
            super(Network, self).__init__()
            self.x1 = nn.Linear(2,4)
            self.s1 = nn.Sigmoid()
            self.x2 = nn.Linear(4,1)
            self.s2 = nn.Sigmoid()

        def init(self):
            nn.init.uniform_(self.x1.weight)
            nn.init.uniform_(self.x2.weight)

        def forward(self, feats):
            f1 = torch.tensor(feats).float()
            xr1= self.x1(f1)
            xs1= self.s1(xr1)
            xr2= self.x2(xs1)
            out= self.s2(xr2)
            return out

        def train(self,val_expected,feats_next):
            val_expected_tensor = torch.tensor(val_expected)
            criterion = nn.MSELoss()
            optimizer = torch.optim.SGD(self.parameters(), lr=0.01)
            def closure():
                optimizer.zero_grad()
                resp = self.forward(feats_next)
                error = criterion(resp,val_expected_tensor)
                error.backward()
                return error
            optimizer.step(closure)

    net = Network()
    net.init()

    for input in ([0.,0.],[0.,1.],[1.,0.],[1.,1.]):
        response=net.forward(input)
        print(response)

    print ("--TRAIN START-")
    for i in range(1000):
        net.train([1.],[0.,0.])
        net.train([0.],[1.,0.])
        net.train([0.],[0.,1.])
        net.train([1.],[1.,1.])
    print ("---TRAIN END---")

    for input in ([0.,0.],[0.,1.],[1.,0.],[1.,1.]):
        response=net.forward(input)
        print(response)

This is a run with 100000 iterations at 0.001 learning rate:

    tensor([0.7726], grad_fn=)
    tensor([0.7954], grad_fn=)
    tensor([0.8229], grad_fn=)
    tensor([0.8410], grad_fn=)
    --TRAIN START-
    *.........*.........*.........*.........*.........*.........*.........*.........*.........*.........
    ---TRAIN END---
    tensor([0.6311], grad_fn=)
    tensor([0.6459], grad_fn=)
    tensor([0.6770], grad_fn=)
    tensor([0.6906], grad_fn=)

I'm really lost here. Shound't this work?


title: How to define the loss function using the output of intermediate layers?

body:
class Model(nn.Module):
def __init__(self):
    super(Model, self).__init__()
    self.encoder = nn.Linear(300, 100)
    self.dense1 = nn.Sequential(nn.Linear(100, 10),nn.ReLU())
    self.dense2 = nn.Sequential(nn.Linear(10, 5),nn.ReLU())
    self.dense3 = nn.Sequential(nn.Linear(5, 1))
def forward(self, x):
    x = self.encoder(x)
    x = self.dense1(x)
    x = self.dense2(x)
    x = self.dense3(x)
    return x

I am working on a regression problem, and I need to use the output of the dense2 layer to calculate the loss.
output of dense2 layer is 5 dimensional (5x1).
I am using PyTorch.
Dataset: Suppose i am using 300 features and i need to predict some score(a floating value).
Input: 300 Features
Output: Some Floating Value


title: Finding means and stds of a bunch of torch.Tensors (that are converted from ndarray images)

body:
to_tensor = transforms.ToTensor()
img = to_tensor(train_dataset[0]['image'])
img

Converts my images values between 0 and 1 which is expected.  It also converts img which is an ndarray to a torch.Tensor.
Previously, without using to_tensor (which I need it now), the following code snippet worked (not sure if this is the best way to find means and stds of the train set, however now doesn't work. How can I make it work?
image_arr = []

for i in range(len(train_dataset)):
    image_arr.append(to_tensor(train_dataset[i]['image']))

print(np.mean(image_arr, axis=(0, 1, 2)))
print(np.std(image_arr, axis=(0, 1, 2)))

The error is:
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-147-0e007c030629> in <module>
      4     image_arr.append(to_tensor(train_dataset[i]['image']))
      5
----> 6 print(np.mean(image_arr, axis=(0, 1, 2)))
      7 print(np.std(image_arr, axis=(0, 1, 2)))

<__array_function__ internals> in mean(*args, **kwargs)

~/anaconda3/lib/python3.7/site-packages/numpy/core/fromnumeric.py in mean(a, axis, dtype, out, keepdims)
   3333
   3334     return _methods._mean(a, axis=axis, dtype=dtype,
-> 3335                           out=out, **kwargs)
   3336
   3337

~/anaconda3/lib/python3.7/site-packages/numpy/core/_methods.py in _mean(a, axis, dtype, out, keepdims)
    133
    134 def _mean(a, axis=None, dtype=None, out=None, keepdims=False):
--> 135     arr = asanyarray(a)
    136
    137     is_float16_result = False

~/anaconda3/lib/python3.7/site-packages/numpy/core/_asarray.py in asanyarray(a, dtype, order)
    136
    137     """
--> 138     return array(a, dtype, copy=False, order=order, subok=True)
    139
    140

ValueError: only one element tensors can be converted to Python scalars


title: pytorch can't shuffle the dataset

body:
i am trying to make an ai with the mnist dataset from torchvision and makeing it with pytorch, but when i type some of the code that shuffle the data, and run it, it say:
    trainset = torch.utils.data.Dataloader(train, batch_size=10, shuffle=True)
AttributeError: module 'torch.utils.data' has no attribute 'Dataloader'

i tried a difrent method but it still do not work, and it says:
    trainset = torch.autograd.Variable.DataLoader(train, batch_size=10, shuffle=True)
AttributeError: type object 'Variable' has no attribute 'DataLoader'

the code i use is:
import torch
import numpy as np
import torchvision
from torchvision import transforms, datasets

train = datasets.MNIST("", train=True, download=True,
                       transform = transforms.Compose([transforms.ToTensor()]))

test = datasets.MNIST("", train=False, download=True,
                       transform = transforms.Compose([transforms.ToTensor()]))

trainset = torch.utils.data.Dataloader(train, batch_size=10, shuffle=True)
testset = torch.utils.data.Dataloader(test, batch_size=10, shuffle=True)

for data in trainset:
    print(data)
    break

the eror from this code:
    trainset = torch.utils.data.Dataloader(train, batch_size=10, shuffle=True)
AttributeError: module 'torch.utils.data' has no attribute 'Dataloader'

i tried a new version but it still not working:
import torch
import numpy as np
import torchvision
from torchvision import transforms, datasets

train = datasets.MNIST("", train=True, download=True,
                       transform = transforms.Compose([transforms.ToTensor()]))

test = datasets.MNIST("", train=False, download=True,
                       transform = transforms.Compose([transforms.ToTensor()])

trainset = torch.autograd.Variable.DataLoader(train, batch_size=10, shuffle=True)
testset = torch.autograd.Variable.DataLoader(test, batch_size=10, shuffle=True)

for data in trainset:
    print(data)
    break

the eror from this code:
    trainset = torch.autograd.Variable.DataLoader(train, batch_size=10, shuffle=True)
AttributeError: type object 'Variable' has no attribute 'DataLoader'

i am stil confused why it is not working,i was folowing a tutorial but it did not work


title: Calculating gradient from network output in PyTorch gives error

body:
I am trying to use a manually calculate a gradient using the output of my network, I will then use this in a loss function. I have managed to get an example working in keras, but converting it to PyTorch has proven more difficult
I have a model like:
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(1, 50)
        self.fc2 = nn.Linear(50, 10)
        self.fc3 = nn.Linear(10, 1)

    def forward(self, x):
        x = F.sigmoid(self.fc1(x))
        x = F.sigmoid(self.fc2(x))
        x = self.fc3(x)
        return x

and some data:
x = torch.unsqueeze(torch.linspace(-1, 1, 101), dim=1)
x = Variable(x)

I can then try find a gradient like:
output = net(x)
grad = torch.autograd.grad(outputs=output, inputs=x, retain_graph=True)[0]

I want to be able to find the gradient of each point, then do something like:
err_sqr = (grad - x)**2
loss = torch.mean(err_sqr)**2

However, at the moment if I try to do this I get the error:
grad can be implicitly created only for scalar outputs

I have tried changing the shape of my network output to fix this, but if I change it to much it says its not part of the graph. I can get rid of that error by allowing that, but then it says my gradient is None. I've managed to get this working in keras, so I'm confident that its possible here too, I just need a hand!
My questions are:

Is there a way to "fix" what I have to allow me to calculate the gradient


title: Tokenizing & encoding dataset uses too much RAM

body:
Trying to tokenize and encode data to feed to a neural network.
I only have 25GB RAM and everytime I try to run the code below my google colab crashes. Any idea how to prevent his from happening? “Your session crashed after using all available RAM”
I thought tokenize/encoding chunks of 50000 sentences would work but unfortunately not.
The code works on a dataset with length 1.3 million. The current dataset has a length of  5 million.
max_q_len = 128
max_a_len = 64
trainq_list = train_q.tolist()
batch_size = 50000

def batch_encode(text, max_seq_len):
      for i in range(0, len(trainq_list), batch_size):
        encoded_sent = tokenizer.batch_encode_plus(
            text,
            max_length = max_seq_len,
            pad_to_max_length=True,
            truncation=True,
            return_token_type_ids=False
        )
      return encoded_sent

    # tokenize and encode sequences in the training set
    tokensq_train = batch_encode(trainq_list, max_q_len)

The tokenizer comes from HuggingFace:
tokenizer = BertTokenizerFast.from_pretrained('bert-base-multilingual-uncased')


title: Pytorch: a similar process to reverse pooling and replicate padding?

body:
I have a tensor A that has shape (batch_size, width, height). Assume that it has these values:
A = torch.tensor([[[0, 1],
                   [1, 0]]])

I am also given a number K that is a positive integer. Let K=2 in this case. I want to do a process that is similar to reverse pooling and replicate padding. This is the expected output:
B = torch.tensor([[[0, 0, 1, 1],
                   [0, 0, 1, 1],
                   [1, 1, 0, 0],
                   [1, 1, 0, 0]]])

Explanation: for each element in A, we expand it to the matrix of shape (K, K), and put it in the result tensor. We continue to do this with other elements, and let the stride between them equals to the kernel size (that is, K).
How can I do this in PyTorch? Currently, A is a binary mask, but it could be better if I can expand it to non-binary case.


title: nn.Linear(feature_size, 1)*n vs nn.Linear(feature_size, n) in PyTorch

body:
I write two types of liner-classifier in PyTorch:

torch.manual_seed(0)
fc = []
for i in range(n):
  fc.append(nn.Linear(feature_size, 1))

The other:

torch.manual_seed(0)
fc = nn.Linear(feature_size, n)

And different results were obtained using these two types of fc in a
multi-label classification model.

Actually, those fc initialized differently and leads to different results. Which one is correct and what should I do if I want the similar results using two types of fc.

Additional Information:

I find out the reason lead to bad result:
The first type FC not updates in training!
But I don't know why there is no updating, my code as follow:

x = self.features(input)
res = []
for i in range(self.num_classes):
    res.append(self.fc[i](x.cpu()))
res = torch.cat(res, 1)
return res.cuda()

Any idea about this?


title: Memory keeps rising when training FFNN pyTorch

body:
I have a Feed forward Neural Network which classifies the MNIST data set.
For some reason the memory keeps approaching 99% no matter how big the batch size is.
I don't have anything which is increasing in size - every dynamic variable is getting overridden after the first epoch but even after epoch number 70 the memory keeps rising.

Im running it on 8GB memory, 2.8 ghZ intel i-5 (7th gen) quad core (Ubuntu 18.04)

batch_size =50 #Number of x's we pass through the net at each iteration
num_epochs =100 #Number of times the entire training data is through the    net

n_train = len(xtrain)
n_batch_train = n_train // batch_size
n_val =len(xval)
n_batch_val = n_val // batch_size

# loss/acc
train_acc,train_loss =[],[]
val_acc,val_loss =[],[]

test_acc,test_loss =[],[]

#Get parameters from the net

par=[]

for i in range(len(layers)-1):
    par=par+list(net.L[i].parameters())

#Optimizer

optimizer = optim.Adam(par,lr=0.001)

#interval of x

get_slice = lambda i,size: range(i*size,(i+1)*size)

for e in range(num_epochs):
  curr_loss =0
  net.train()
  for i in range(n_batch_train):

    x_interval = get_slice(i,batch_size)
    slze = get_slice(i,batch_size)
    #Batchnorm
    bn = nn.BatchNorm1d(num_features = num_features)
    x_batch = bn(Variable(torch.from_numpy(xtrain[slze])))

    out = (net(x_batch)).double()
    target_batch = Variable(torch.from_numpy(ytrain[slze]).double())
    L=criterion(out,target_batch)

    #Update gradients
    optimizer.zero_grad()
    L.backward()
    optimizer.step()

    #Store training accuracy and loss
    train_acc.append(accuracy(target_batch, out).data)
    train_loss.append(L.data.numpy())

#### Validate ####
  net.eval()
  for j in range(n_batch_val):
    slze = get_slice(j,batch_size)
    val_batch = Variable(torch.from_numpy(xval[slze]))
    val_out = (net(bn(val_batch))).double()
    target_batch = Variable(torch.from_numpy(yval[slze]).double())

    #Store val acc and loss
    val_acc.append(accuracy(target_batch,val_out).data)


title: Selection/Filter with indices using pytorch

body:
I have a 3-dimensional numpy array, for example:

 x = [[[0.3, 0.2, 0.5],
       [0.1, 0.2, 0.7],
       [0.2, 0.2, 0.6]]]

The indices array is also 3-dimensional, like:

indices = [[[0],
            [1],
            [2]]]

I expect the output is:

 output= [[[0.3],
           [0.2],
           [0.6]]]

I tried the torch.index_select and torch.gather function, but I couldn't find a right way to deal with the dimension. Thanks for any help!


title: pytorch autograd obstructs script from terminating

body:
Whenever I call autograds backward my script never terminates. backwardis not per se blocking, all lines after it are still executed, the script just does not terminate. It appears that there is some sort of working thread in the background which hangs, but I was not able to find any information about it.

I originally encountered the problem while training neural networks, however I eventually found a very short example with the same behavior:

import torch

x = torch.randn(3, requires_grad=True)
y = x * 2
print(y)

gradients = torch.tensor([0.1, 1.0, 0.0001], dtype=torch.float)
y.backward(gradients)
print(x.grad)

print("all done")

When I remove the backward line, the script finishes as expected. Otherwise I see a process call python in the task manager, if I terminate it by hand, the script execution also terminates.

I installed pytorch on Windows 7 using conda (conda create --name grad_test pytorch -c pytorch) in the most recent, stable version (python 3.7, pytorch 1.2.0).


title: Building a dataset with dataloader pytorch getting error cannot import name 'read_data_sets'

body:
Loading data into dataset using pytorch dataloader.
Getting error cannot import name 'read_data_sets'

Tried searaching for results from similar issues.
If there is confusion about file instead of module and it can't find read_data_sets in your file  How do i change to fix?

class MRDataset(data.Dataset):
    def __init__(self, root_dir, task, plane, train=True, transform=None, weights=None):
        super().__init__()
        self.task = task
        self.plane = plane
        self.root_dir = root_dir
        self.train = train
        if self.train:
            self.folder_path = self.root_dir + 'train/{0}/'.format(plane)
            self.records = pd.read_csv(
                self.root_dir + 'train-{0}.csv'.format(task), header=None, names=['id', 'label'])
        else:
            transform = None
            self.folder_path = self.root_dir + 'valid/{0}/'.format(plane)
            self.records = pd.read_csv(
                self.root_dir + 'valid-{0}.csv'.format(task), header=None, names=['id', 'label'])

        self.records['id'] = self.records['id'].map(
            lambda i: '0' * (4 - len(str(i))) + str(i))
        self.paths = [self.folder_path + filename +
                      '.npy' for filename in self.records['id'].tolist()]
        self.labels = self.records['label'].tolist()

        self.transform = transform
        if weights is None:
            pos = np.sum(self.labels)
            neg = len(self.labels) - pos
            self.weights = torch.FloatTensor([1, neg / pos])
        else:
            self.weights = torch.FloatTensor(weights)

    def __len__(self):
        return len(self.paths)

    def __getitem__(self, index):
        array = np.load(self.paths[index])
        label = self.labels[index]
        if label == 1:
            label = torch.FloatTensor([[0, 1]])
        elif label == 0:
            label = torch.FloatTensor([[1, 0]])

        if self.transform:
            array = self.transform(array)
        else:
            array = np.stack((array,)*3, axis=1)
            array = torch.FloatTensor(array)

        # if label.item() == 1:
        #     weight = np.array([self.weights[1]])
        #     weight = torch.FloatTensor(weight)
        # else:
        #     weight = np.array([self.weights[0]])
        #     weight = torch.FloatTensor(weight)

        return array, label, self.weights

There is a model and train class to run this.  Arguments specified in train.
Running the train should load data and run through model


title: How to load all 5 batches of CIFAR10 in a single data structure as on MNIST in PyTorch?

body:
With Mnist I have a single file with the labels and a single file for the train, so I simply do:

self.data = datasets.MNIST(root='./data', train=True, download=True)

Basically I create a set of labels (from 0-9) and save the i-th position of the image in the data structure, to create my custom tasks:

   def make_tasks (self):
            self.task_to_examples = {} #task 0-9
            self.all_tasks = set (self.data.train_labels.numpy ())
            for i, digit in enumerate (self.data.train_labels.numpy ()):
                if str(digit) not in self.task_to_examples:
                    self.task_to_examples[str(digit)] = []
                self.task_to_examples[str(digits)].append(i)

I don't understand how to do the same thing using CIFAR10 because it is divided into 5 batches, I would like all the data in a single structure.


title: Why is pyTorch "destroying" my numpy arrays?

body:
I have am working with numpy tensors of shape (N,2,128,128).

When trying to visualize these as images (I reconstruct via ifft2), numpy and pyTorch seems to mix things up in an crazy manner ...

I have checked with small dummy arrays and when I pass a numpy ndarray to a torch.FloatTensor the values are exactly the same at the same positions (same shape!), but when I try to do an ifft2 on the torch tensor ones, the result is different than on the non-torch tensor! Can someone help me make sense of this ?

A small reproducible example is:

x=np.random.rand(3,2,2,2)
xTorch=torch.FloatTensor(x)

#visualize then in the interpreter, they are the same!
#
#now show the magnitude of an inverse fourier transform
plt.imshow(np.abs(np.fft.ifft2(xTorch[0,0,:,:]+1j*xTorch[0,1,:,:])))
plt.show()

plt.imshow(np.abs(np.fft.ifft2(x[0,0,:,:]+1j*x[0,1,:,:])))
plt.show()

#they are not the same ! What is the problem!?

I found out that if I use: torch.Tensor.cpu(xTorch).detach().numpy() I can get the same result, but what does that mean?!

P.S.
ALso, note that I know the correct visualization is with the x and not the xTensor, so it seems that torch is changing something when I do the ifft2 .. or when I reconstruct the 2 channels...or maybe there is a problem/bug with complex numbers ...
If you look inside : np.abs(np.fft.ifft2(x[0,0,:,:]+1j*x[0,1,:,:])) and the xTorch one, the values are so different, that it is not just a problem of floating point error, it is something serious, but I can't figure it out and it's driving me crazy.


title: Training NeuralCoref for new language

body:
I'm currently training the Neuralcoref model for Dutch using the SoNar corpus by following this instruction.

The first step that I did was to transform the MMAX format of the SoNars corpus to the CONLL format using this script.
Here is the CONLL format obtained after running the script

After that, I changed the load_file function accordingly into something like this:

def load_file(full_name, debug=False):
"""
load a *._conll file
Input: full_name: path to the file
Output: list of tuples for each conll doc in the file, where the tuple contains:
    (utts_text ([str]): list of the utterances in the document
     utts_tokens ([[str]]): list of the tokens (conll words) in the document
     utts_corefs: list of coref objects (dicts) with the following properties:
        coref['label']: id of the coreference cluster,
        coref['start']: start index (index of first token in the utterance),
        coref['end': end index (index of last token in the utterance).
     utts_speakers ([str]): list of the speaker associated to each utterances in the document
     name (str): name of the document
     part (str): part of the document
    )
"""
docs = []
with io.open(full_name, "rt", encoding="utf-8", errors="strict") as f:
    lines = list(f)
    utts_text = []
    utts_tokens = []
    utts_corefs = []
    utts_speakers = []
    tokens = []
    corefs = []
    speaker = "UNKNOWN"
    name = ""
    part = ""
    index = 0
    for line_index, line in enumerate(lines):
        cols = line.split()
        if '#begin document' in line:
            if debug:
                print("New Document")
            name = cols[2].replace("(",'').replace(");","")
            part = cols[-1]

        #End of utterance
        if len(cols) == 0:
            if tokens:
                if debug:
                    print("End of utterance ",corefs)
                utts_text.append("".join(t + " " for t in tokens))
                utts_tokens.append(tokens)
                utts_speakers.append(speaker)
                utts_corefs.append(corefs)
                tokens = []
                corefs = []
                index = 0
                speaker = "UNKNOWN"
                continue

        # End of doc
        elif len(cols) == 2:
            if debug:
                print("End of doc")
            if cols[0] == "#end":
                if debug:
                    print("Saving doc")
                docs.append(
                    (utts_text, utts_tokens, utts_corefs, utts_speakers, name, part)
                )
                utts_text = []
                utts_tokens = []
                utts_corefs = []
                utts_speakers = []

        # New document
        elif len(cols) == 5:
            if debug:
                print("New doc")
            if cols[0] == "#begin":
                name = re.match(r"\((.*)\);", cols[2]).group(1)

                try:
                    part = cols[4]
                except ValueError:
                    print("Error parsing document part " + line)
                if debug:
                    print("New doc: ", name, part)
                tokens = []
                corefs = []
                index = 0
            else:
                raise ValueError("Error on begin line " + line)

        # Inside utterance
        elif len(cols) == 4:
            if debug:
                print("Inside utterance")

            assert int(cols[1]) == index, "Index error on " + line
            if speaker:
                assert speaker == 'UNKNOWN', 'Things are good'
            else:
                speaker = 'UNKNOWN'
                if debug:
                    print("speaker", speaker)

            if cols[-1] != "-":
                coref_expr = cols[-1].split("|")
                if debug:
                    print("coref_expr", coref_expr)

                if not coref_expr:
                    raise ValueError("Coref expression empty " + line)

                for tok in coref_expr:
                    if debug:
                        print("coref tok", tok)
                    try:
                        match = re.match(r"^(\(?)(\d+)(\)?)$", tok)
                    except:
                        print("error getting coreferences for line " + line)
                    assert match is not None, (
                        "Error parsing coref " + tok + " in " + line
                    )
                    num = match.group(2)
                    assert num is not "", (
                        "Error parsing coref " + tok + " in " + line
                    )
                    if match.group(1) == "(":
                        if debug:
                            print("New coref", num)
                        corefs.append({"label": num, "start": index, "end": None})
                    if match.group(3) == ")":
                        j = None
                        for i in range(len(corefs) - 1, -1, -1):
                            if debug:
                                print("i", i)
                            if (
                                corefs[i]["label"] == num
                                and corefs[i]["end"] is None
                            ):
                                j = i
                                break
                        assert j is not None, "coref closing error " + line
                        if debug:
                            print("End coref", num)
                        corefs[j]["end"] = index
            tokens.append(clean_token(cols[2]))
            index += 1
        else:
            raise ValueError("Line not standard " + line)
    return docs

The embedding that I used was trained on the same corpus (SoNars), it is a word2vec model with 300 dimensions (quite big compare to the original one with only 50 dimensions). After that, I generated the static_word_vocab.txt file consists of corresponding words.

Here are my questions:

I'm not sure why they used the tuned_word_embeddings files. Digging deeper to this matter, I found out that this file is much shorter than the static_word_embedding files. It contains the words that are in and not in the static_word_embedding files. For this reason, I used the static_word_embedding files as the tuned_word_embedding and it seemed to work. I still doubting this is wrong.
I don't understand how they define missing and *UNK* tokens in document.py
Cannot run the PERL evaluation code. Whenever the code run into the evaluator.get_scores() function. It shown this exception which seem like the PERL scorer was not loaded properly. No matter what I did, it does not get imported to the scorer_wrapper.pl:
Here is the error when load Scorer code
I temporarily removed all the evaluator.get_scores() in the train code. Then run it, however, this time I got an exception of RuntimeError: size mismatch, m1: [4 x 4184], m2: [674 x 1000] at C:\w\b\windows\pytorch\aten\src\TH/generic/THTensorMath.cpp:41. I guess I did something wrong while processing the data.

Would be nice if someone here with experience can give me (and other people with the same issue) some basic guidance on this or even some comments are appreciated.


title: Cnn model using pytorch

body:
I have images and labels. I divided them into test and train sets.
(xtrain,
ytrain,
xtest,
ytest).
x refers to images and y refers to label.
How to use these sets in the following train model
 **# Train the model
  total_step = len(train_loader)
 for epoch in range(num_epochs):
   for i, (images, labels) in enumerate(train_loader):
    images = images.to(device)
    labels = labels.to(device)

    # Forward pass
    outputs = model(images)
    loss = criterion(outputs, labels)

    # Backward and optimize
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    if (i+1) % 100 == 0:
        print ('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}'
               .format(epoch+1, num_epochs, i+1, total_step, loss.item()))
 # Test the model
 model.eval()  # eval mode (batchnorm uses moving mean/variance instead of mini-batch
                       mean/variance)**


title: GPU Memoryerror while using pytorch with empty GPU

body:
I am using HW setup with 3 GPUs, and want to train my model with following code :
with torch.cuda.device(2):
    train_load, val_load = SRD.load_sr_st_dataset(route_img,route_dpt)
    #print(train_load.dataset.__sizeof__())
    sr_stereo = SRD.sr_stereo(max_d=200)
    sr_stereo.cuda()
    optimizer = torch.optim.Adam(sr_stereo.parameters(),lr = 0.001)
    criterion = nn.MSELoss(reduction='sum')
    scheduler = torch.optim.lr_scheduler.MultiStepLR(optimizer,
            milestones=range(5,100,5), gamma=0.5, last_epoch=-1)
    sr_stereo.train()

    for e in range(epoch_num):
        scheduler.step()
        for sample,valid in zip(train_load,val_load):
            l10,l20,r10,r20,depth = SRD.parsedata(sample)
            l10v, l20v, r10v, r20v, depthv = SRD.parsedata(valid)
            out_train = sr_stereo(l10,l20,r10,r20)
            out_val = sr_stereo(l10v, l20v, r10v, r20v)
            optimizer.zero_grad()
            loss_train = criterion(out_train,depth)
            loss.backward()
            optimizer.step()
            scheduler.step()

Since I already have 2 models being trained in GPU0 and GPU1 , I want this model to be run at GPU2.
And I got error :
RuntimeError: CUDA out of memory. Tried to allocate 282.00 MiB (GPU 2; 31.88 GiB total capacity; 29.99 GiB already allocated; 78.81 MiB free; 30.04 GiB reserved in total by PyTorch)

Is there anything that I am missing ? I cannot understand why the error occurs since GPU 2 has plenty of memory and does not run any training.


title: Saving model in pytorch and keras

body: I have trained model with keras and saved in with the help of pytorch. Will it cause any problems in the future. As far as I know the only difference between them is Keras saves its model's weights as doubles while PyTorch saves its weights as floats.


title: Explanation of the tuple grad_fn.next_functions in PyTorch

body:
The torch computation graph is composed of grad_fn. For the terminating node, the grad_fn object has an attribute called next_functions which is a tuple. I understand that using the first element (0th index) of the tuple, I can reconstruct the computation graph for gradients. But I was wondering what does the second element (1st index) of the tuple mean?
In one of the answers in PyTorch forums, it is said that:

The number is the input number to the next backward function, so can only be non-zero when a function has multiple differentiable outputs (there aren’t that many, but e.g. the RNN functions typically do).

But I am don't understand this statement. Can someone explain this with an example perhaps?


title: Using both vCPU's with google cloud computing. Python code. PyTorch

body:
I am new to cloud computing. I made a virtual machine in google cloud computing, machine type:
e2-highcpu-2 (2 vCPU's, 2 GB geheugen)
I run a script with running the command
python3 simulation1.py
When I look at the output control screen, I note that only 50% of the CPU power is used. So I just use one of my 2 CPU's. Is there a way to make full use of the computing power ?


title: retain_graph problem with GRU in Pytorch 1.6

body:
I am aware that, while employing loss.backward() we need to specify retain_graph=True if there are multiple networks and multiple loss functions to optimize each network separately. But even with (or without) specifying this parameter I am getting errors. Following is an MWE to reproduce the issue (on PyTorch 1.6).
import torch
from torch import nn
from torch import optim
torch.autograd.set_detect_anomaly(True)

class GRU1(nn.Module):
    def __init__(self):
        super(GRU1, self).__init__()
        self.brnn = nn.GRU(input_size=2, bidirectional=True, num_layers=1, hidden_size=100)

    def forward(self, x):
        return self.brnn(x)

class GRU2(nn.Module):
    def __init__(self):
        super(GRU2, self).__init__()
        self.brnn = nn.GRU(input_size=200, bidirectional=True, num_layers=1, hidden_size=1)

    def forward(self, x):
        return self.brnn(x)

gru1 = GRU1()
gru2 = GRU2()
gru1_opt = optim.Adam(gru1.parameters())
gru2_opt = optim.Adam(gru2.parameters())
criterion = nn.MSELoss()

for i in range(100):
    gru1_opt.zero_grad()
    gru2_opt.zero_grad()
    vector = torch.randn((15, 100, 2))
    gru1_output, _ = gru1(vector)  # (15, 100, 200)
    loss_gru1 = criterion(gru1_output, torch.randn((15, 100, 200)))
    loss_gru1.backward(retain_graph=True)
    gru1_opt.step()
    gru1_output, _ = gru1(vector)  # (15, 100, 200)
    gru2_output, _ = gru2(gru1_output)  # (15, 100, 2)
    loss_gru2 = criterion(gru2_output, torch.randn((15, 100, 2)))
    loss_gru2.backward(retain_graph=True)
    gru2_opt.step()
    print(f"GRU1 loss: {loss_gru1.item()}, GRU2 loss: {loss_gru2.item()}")

With retain_graph set to True I get the error
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [100, 300]], which is output 0 of TBackward, is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

The error without the parameter is
RuntimeError: Trying to backward through the graph a second time, but the saved intermediate results have already been freed. Specify retain_graph=True when calling backward the first time.

which is expected.
Please point at what needs to be changed in the above code for it to begin training. Any help is appreciated.


title: What is the difference in RobertaTokenizer() and from_pretrained() way of initialising RobertaTokenizer?

body:
I am a newbie to huggingface transformers and facing the below issue in training a RobertaForMaskedLM LM from scratch:
First, I have trained and saved a ByteLevelBPETokenizer as follows:
tokenizer = ByteLevelBPETokenizer()
print('Saving tokenizer at:', training_file)
tokenizer.train(files=training_file, vocab_size=VOCAB_SIZE, min_frequency=2,
special_tokens=["<s>","<pad>","</s>","<unk>","<mask>"])
tokenizer.save_model(tokenizer_mdl_dir)

Then, trained RobertaForMaskedLM using this tokenizer by creating a RobertaTokenizer as follows:
roberta_tokenizer = RobertaTokenizer(tokenizer_mdl + "/vocab.json", tokenizer_mdl + "/merges.txt")

But now, when I try to test the trained LM using a fill-mask pipeline,
fill_mask_pipeline = pipeline("fill-mask", model=roberta_model, tokenizer=roberta_tokenizer)

I got the below error:

PipelineException: No mask_token () found on the input

So, I realized, the tokenizer that I have loaded, is tokenizing the <mask> token as well. But I couldn't understand why it is doing so. Please help me understand this.
After trying several things, I loaded the tokenizer differently,
roberta_tokenizer = RobertaTokenizer.from_pretrained(tokenizer_mdl)

And, now the fill_mask_pipeline runs without errors. So, what is the difference between loading a tokenizer using RobertaTokenizer() and using the .from_pretrained() method?


title: Extracting feature vector for grey images via ResNet18: output with shape [1, 224, 224] doesn't match the broadcast shape [3, 224, 224]

body:
I have 600x800 images that have only 1 channel. I am trying to use pre-trained ResNet18 to extract their features however the code expects 3 channel:
import torch
import torchvision
import torchvision.models as models
from PIL import Image

img = Image.open("labeled-data/train_moth/moth/frame163.png")

# Load the pretrained model
model = models.resnet18(pretrained=True)

# Use the model object to select the desired layer
layer = model._modules.get('avgpool')

# Set model to evaluation mode
model.eval()

transforms = torchvision.transforms.Compose([
    torchvision.transforms.Resize(256),
    torchvision.transforms.CenterCrop(224),
    torchvision.transforms.ToTensor(),
    torchvision.transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])

def get_vector(image):
    # Create a PyTorch tensor with the transformed image
    t_img = transforms(image)
    t_img = torch.cat((t_img, t_img, t_img), 0)
    # Create a vector of zeros that will hold our feature vector
    # The 'avgpool' layer has an output size of 512
    my_embedding = torch.zeros(512)

    # Define a function that will copy the output of a layer
    def copy_data(m, i, o):
        my_embedding.copy_(o.flatten())                 # <-- flatten

    # Attach that function to our selected layer
    h = layer.register_forward_hook(copy_data)
    # Run the model on our transformed image
    with torch.no_grad():                               # <-- no_grad context
        model(t_img.unsqueeze(0))                       # <-- unsqueeze
    # Detach our copy function from the layer
    h.remove()
    # Return the feature vector
    return my_embedding

Here's the error I am getting:
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-5-59ab45f8c1e6> in <module>
     42
     43
---> 44 pic_vector = get_vector(img)

<ipython-input-5-59ab45f8c1e6> in get_vector(image)
     21 def get_vector(image):
     22     # Create a PyTorch tensor with the transformed image
---> 23     t_img = transforms(image)
     24     t_img = torch.cat((t_img, t_img, t_img), 0)
     25     # Create a vector of zeros that will hold our feature vector

~/anaconda3/lib/python3.7/site-packages/torchvision/transforms/transforms.py in __call__(self, img)
     59     def __call__(self, img):
     60         for t in self.transforms:
---> 61             img = t(img)
     62         return img
     63

~/anaconda3/lib/python3.7/site-packages/torchvision/transforms/transforms.py in __call__(self, tensor)
    210             Tensor: Normalized Tensor image.
    211         """
--> 212         return F.normalize(tensor, self.mean, self.std, self.inplace)
    213
    214     def __repr__(self):

~/anaconda3/lib/python3.7/site-packages/torchvision/transforms/functional.py in normalize(tensor, mean, std, inplace)
    296     if std.ndim == 1:
    297         std = std[:, None, None]
--> 298     tensor.sub_(mean).div_(std)
    299     return tensor
    300

RuntimeError: output with shape [1, 224, 224] doesn't match the broadcast shape [3, 224, 224]

    pic_vector = get_vector(img)

Code is from: https://stackoverflow.com/a/63552285/2414957

I thought using
t_img = torch.cat((t_img, t_img, t_img), 0)

would be helpful but I was wrong.
Here's a bit about image:
$ identify frame163.png
frame163.png PNG 800x600 800x600+0+0 8-bit Gray 256c 175297B 0.000u 0:00.000


title: Convert a simple cnn from keras to pytorch

body:
Can anyone please help me to convert this model to PyTorch? I already tried to convert from Keras to PyTorch like this How can I convert this keras cnn model to pytorch version but training results were different. Thank you.
input_3d = (1, 64, 96, 96)
pool_3d = (2, 2, 2)
model = Sequential()
model.add(Convolution3D(8, 3, 3, 3, name='conv1', input_shape=input_3d,
                          data_format='channels_first'))
model.add(MaxPooling3D(pool_size=pool_3d, name='pool1'))
model.add(Convolution3D(8, 3, 3, 3, name='conv2',data_format='channels_first'))
model.add(MaxPooling3D(pool_size=pool_3d, name='pool2'))
model.add(Convolution3D(8, 3, 3, 3, name='conv3',data_format='channels_first'))
model.add(MaxPooling3D(pool_size=pool_3d, name='pool3'))
model.add(Flatten())
model.add(Dense(2000, activation='relu', name='dense1'))
model.add(Dropout(0.5, name='dropout1'))
model.add(Dense(500, activation='relu', name='dense2'))
model.add(Dropout(0.5, name='dropout2'))
model.add(Dense(3, activation='softmax', name='softmax'))

_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
conv1 (Conv3D)               (None, 8, 60, 94, 94)     224
_________________________________________________________________
pool1 (MaxPooling3D)         (None, 8, 30, 47, 47)     0
_________________________________________________________________
conv2 (Conv3D)               (None, 8, 28, 45, 45)     1736
_________________________________________________________________
pool2 (MaxPooling3D)         (None, 8, 14, 22, 22)     0
_________________________________________________________________
conv3 (Conv3D)               (None, 8, 12, 20, 20)     1736
_________________________________________________________________
pool3 (MaxPooling3D)         (None, 8, 6, 10, 10)      0
_________________________________________________________________
flatten_1 (Flatten)          (None, 4800)              0
_________________________________________________________________
dense1 (Dense)               (None, 2000)              9602000
_________________________________________________________________
dropout1 (Dropout)           (None, 2000)              0
_________________________________________________________________
dense2 (Dense)               (None, 500)               1000500
_________________________________________________________________
dropout2 (Dropout)           (None, 500)               0
_________________________________________________________________
softmax (Dense)              (None, 3)                 1503
=================================================================


title: Pytorch batch row-wise application of function

body:
I would like to figure out a way to apply a function which calculates pairwise distances, let's call it dists(A, B), row-wise for every input element in a batch, meaning:
(100, 16, 3) -- input, 100 is the batch size so 100 instances, 16 is let's say image size, and 3 filters (asking for Conv2D)
(5, 3) -- tensor for which I want to calculate the row-wise distance (assume it's A in dists(A, B) and is fixed)
Now for every instance I am supposed to get back a matrix of shape (5, 16), easily. So, I could use a for to span the batch and get my final (100,5,16) result. However, I would love to know if there is an easier way to apply my function row-wise, in parallel, using GPU.
Thank you very much for your time.


title: PyTorch - Creating Federated CIFAR-10 Dataset

body:
I'm training a neural network (doesn't matter which one) on CIFAR-10 dataset. I'm using Federated Learning:

I have 10 models, each model having access to its own part of the dataset. At every time step, each model makes a step using its own data, and then the global model is an average of the model (this version is based on this, but I tried a lot of options):

def server_aggregate(server_model, client_models):
    global_dict = server_model.state_dict()
    for k in global_dict.keys():
        global_dict[k] = torch.stack([client_models[i].state_dict()[k].float() for i in range(len(client_models))], 0).mean(0)
    server_model.load_state_dict(global_dict)
    for model in client_models:
        model.load_state_dict(server_model.state_dict())

To be specific, each machine only has access to a data corresponding to a single class. I.e. machine 0 has only samples corresponding to class 0, etc. I'm doing it the following way:

def split_into_classes(full_ds, batch_size, num_classes=10):
  class2indices = [[] for _ in range(num_classes)]
  for i, y in enumerate(full_ds.targets):
    class2indices[y].append(i)

  datasets = [torch.utils.data.Subset(full_ds, indices) for indices in class2indices]
  return [DataLoader(ds, batch_size=batch_size, shuffle=True) for ds in datasets]

Problem. During training, I can see that my federated training loss decreases. However, I never see my test loss/accuracy improve (acc is always around 10%).
Moreover, when I check accuracy on train/test datasets:

For the federated dataset, the accuracy improves.
For the testing dataset, the accuracy doesn't improve.
(Most surprising) for the training dataset, the accuracy doesn't improve. Note that this dataset is essentially the same as federated dataset, but not split into classes. The checking code is the following:

def epoch_summary(model, fed_loaders, true_train_loader, test_loader, frac):
  with torch.no_grad():
    train_len = 0
    train_loss, train_acc = 0, 0
    for train_loader in fed_loaders:
      cur_loss, cur_acc, cur_len = true_results(model, train_loader, frac)
      train_loss += cur_len * cur_loss
      train_acc += cur_len * cur_acc
      train_len += cur_len

    train_loss /= train_len
    train_acc /= train_len

    true_train_loss, true_train_acc, true_train_len = true_results(model, true_train_loader, frac)
    test_loss, test_acc, test_len = true_results(model, test_loader, frac)

  print("TrainLoss: {:.4f} TrainAcc: {:.2f} TrueLoss: {:.4f} TrueAcc: {:.2f} TestLoss: {:.4f} TestAcc: {:.2f}".format(
        train_loss, train_acc, true_train_loss, true_train_acc, test_loss, test_acc
        ), flush=True)

The full code can be found here. Things which don't seem to matter:

Model. I got the same problem for Resnet models and for some other models.
How I aggregate the models. I tried using state_dict or directly manipulate model.parameters(), no effect.
How I learn the models. I tried using optim.SGD or directly update param.data -= learning_rate * param.grad, no effect.
Computational graph. I've tried adding .detach().clone() and with torch.no_grad() into all possible places, no effect.

So I'm suspecting that the problem is somehow with the federated data itself (especially given strange accuracy results). What can be a problem?


title: ValueError: Target size (torch.Size([10, 1])) must be the same as input size (torch.Size([10, 2]))

body:
A binary classification problem with Batch Size = 10. Trying to use torch.nn.BCEWithLogitsLoss().
~\Anaconda3\envs\notebook\lib\site-packages\torch\nn\functional.py in binary_cross_entropy_with_logits(input, target, weight, size_average, reduce, reduction, pos_weight)
   2578
   2579     if not (target.size() == input.size()):
-> 2580         raise ValueError("Target size ({}) must be the same as input size ({})".format(target.size(), input.size()))
   2581
   2582     return torch.binary_cross_entropy_with_logits(input, target, weight, pos_weight, reduction_enum)

ValueError: Target size (torch.Size([1, 10])) must be the same as input size (torch.Size([10, 2]))

Here is my training code:
def train(epochs):
    print('Starting training..')
    for e in range(0, epochs):
        exp_lr_scheduler.step()
        print('='*20)
        print(f'Starting epoch {e + 1}/{epochs}')
        print('='*20)
        train_loss = 0.
        val_loss = 0.
        resnet18.train() # set model to training phase
        for train_step, (images, labels) in enumerate(dl_train):
            optimizer.zero_grad()
            outputs = resnet18(images)
            outputs = outputs.float()
            loss = loss_fn(outputs, labels.unsqueeze(0))
            loss.backward()
            optimizer.step()
            train_loss += loss.item()
            if train_step % 20 == 0:
                print('Evaluating at step', train_step)
                accuracy = 0
                resnet18.eval() # set model to eval phase
                for val_step, (images, labels) in enumerate(dl_val):
                    outputs = resnet18(images)
                    outputs = outputs.float()
                    loss = loss_fn(outputs, labels.unsqueeze(0))
                    val_loss += loss.item()
                    _, preds = torch.max(outputs, 1)
                    accuracy += sum((preds == labels).numpy())
                val_loss /= (val_step + 1)
                accuracy = accuracy/len(val_dataset)
                print(f'Validation Loss: {val_loss:.4f}, Accuracy: {accuracy:.4f}')
                show_preds()
                resnet18.train() #set model to training phase
                if accuracy >= 0.95:
                    print('Performance condition satisfied, stopping..')
                    return
        train_loss /= (train_step + 1)
        print(f'Training Loss: {train_loss:.4f}')
    print('Training complete..')**

train(epochs=30)


title: sqrt_vml_cpu not implemented for 'Long'

body:
a = torch.full([2, 2], 9)

b = a.sqrt()
print(b)

b = a.rsqrt()
print(b)

RuntimeError: sqrt_vml_cpu not implemented for 'Long'

a is torch.LongTensor, but sqrt and rsqrt do not suppor Long, what should I do?


title: GPU underutilized in Actor Critic (A2C) Stable Baselines3 implementation

body:
I am trying to use A2C of StablesBaselines3 for training an agent on my custom environment. My problem is that my GPU Utilization is very less (around 10 % only) while my CPU utilization has hit the ceiling. Because of this the training is very very slow. I have tried the following things as per this discussion thread addressed by @araffin
[https://github.com/hill-a/stable-baselines/issues/308]

Expanding the Actor and Critic architecture to a three layer neural
network having 256, 256 and 128 neurons respectively. The GPU
utilization did increase after that but it was only marginal
(increased from 10 % to 15 %) as in this suggestion
changed device argument of A2C method to 'cuda' from the default which is 'auto' -
No improvement

It is not entirely clear what worked for the user in the above mentioned discussion.
My A2C is intialized as follows:
import gym
from stable_baselines3 import A2C
policy_kwargs = dict(activation_fn=th.nn.ReLU,
                     net_arch=[dict(pi=[256, 256, 128], vf=[256, 256, 128])])
model = A2C("MlpPolicy", env, verbose=1,  learning_rate=linear_schedule(0.001),\
 n_steps=50, gamma=0.8, gae_lambda=1.0, ent_coef=0.2, vf_coef=0.2, \
tensorboard_log=logdir,  policy_kwargs=policy_kwargs, seed=50, device='cuda' )
model.learn(total_timesteps=50000)

GPU configurations are:

Cuda version - 9.0
GPU - NVIDIA RTX 2080i
GPU mem - 11GB (out of that only about 1600MB is being used)

CPU configurations

Threads per Core - 2
Core per socket - 8
Intel i9 core 9900 K @ 3.6 Ghz
CPU Mem - 32GB
SW versions
Torch - 1.6.0
Tensorflow GPU - 2.4.1 (although I know SB3 implementation uses Torch)
Python 3.7 on anaconda environment

Any help will be appreciated!


title: Why are PyTorch "convolutions" implemented as cross-correlations?

body:
PyTorch convolutions are actually implemented as cross-correlations. This shouldn't produce issues in training a convolution layer, since one is just a flipped version of the other (and hence the learned function will be equally powerful), but it does prove an issue when:

trying to implement an actual convolution with the functional library
trying to copy the weights of an actual convolution from another deep learning library

The authors say the following in Deep Learning with PyTorch:

Convolution, or more precisely, discrete convolution1...
1. There is a subtle difference between PyTorch's convolution and mathematics' convolution: one argument's sign is flipped. If we were in a pedantic mood, we could call PyTorch's convolutions discrete cross-correlations.

But they don't explain why it was implemented like this. Is there a reason?
Maybe something similar to how the PyTorch implementation of CrossEntropyLoss isn't actually cross entropy but an analogous function taking "logits" as inputs instead of raw probabilities (to avoid numerical instability)?


title: Stuck at this error "RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu"

body:
I am running the following code. If I try to run it on CPU only it runs fine but it takes too much time to train. So I thought to change the runtime to GPU and made appropriate changes. Now it is stuck.
import torch
from models.bert_attention_model import AttentionModel
from models.bert_cnn_model import BERTCNNSentiment

import sys

if sys.argv[1].lower() =="hinglish":
  data_path = "../data/hinglish/"
elif sys.argv[1].lower() == "spanglish":
  data_path = "../data/spanglish/"
else:
  print("Format: %s %s" %(argv[0], argv[1]))

train_name = "train.txt"
test_name = "test.txt"
model_save_names = ["../checkpoint/cnn_model.txt", "../checkpoint/attention_model.txt"]

import random
import numpy as np

SEED = 1234

random.seed(SEED)
np.random.seed(SEED)
torch.manual_seed(SEED)
torch.backends.cudnn.deterministic = True

from transformers import BertTokenizer, AutoTokenizer, XLMRobertaTokenizer

tokenizer = XLMRobertaTokenizer.from_pretrained("xlm-roberta-base")
print('XLM Roberta Tokenizer Loaded...')

init_token_idx = tokenizer.cls_token_id
eos_token_idx = tokenizer.sep_token_id
pad_token_idx = tokenizer.pad_token_id
unk_token_idx = tokenizer.unk_token_id

max_input_length = 150
print("Max input length: %d" %(max_input_length))

def tokenize_and_cut(sentence):
    tokens = tokenizer.tokenize(sentence)
    tokens = tokens[:max_input_length-2]
    return tokens

from torchtext import data

UID = data.Field(sequential=False, use_vocab=False, pad_token=None)
TEXT = data.Field(batch_first = True,
                  use_vocab = False,
                  tokenize = tokenize_and_cut,
                  preprocessing = tokenizer.convert_tokens_to_ids,
                  init_token = init_token_idx,
                  eos_token = eos_token_idx,
                  pad_token = pad_token_idx,
                  unk_token = unk_token_idx)

LABEL = data.LabelField()

from torchtext import datasets

fields = [('uid',UID),('text', TEXT),('label', LABEL)]
train_data, test_data = data.TabularDataset.splits(
                                        path = data_path,
                                        train = train_name,
                                        test = test_name,
                                        format = 'tsv',
                                        fields = fields,
                                        skip_header = True)
train_data, valid_data = train_data.split(random_state = random.seed(SEED))
print('Data loading complete')
print(f"Number of training examples: {len(train_data)}")
print(f"Number of validation examples: {len(valid_data)}")
print(f"Number of test examples: {len(test_data)}")

tokens = tokenizer.convert_ids_to_tokens(vars(train_data.examples[0])['text'])

LABEL.build_vocab(train_data, valid_data)

print(LABEL.vocab.stoi)

BATCH_SIZE = 128

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print("Device in use:",device)
train_iterator, valid_iterator, test_iterator = data.BucketIterator.splits(
    (train_data, valid_data, test_data),
    sort_key=lambda x: len(x.text),
    batch_size = BATCH_SIZE,
    device = device)

print('Iterators created')

print('Downloading XLM Roberta model...')

from transformers import XLMRobertaModel
bert = XLMRobertaModel.from_pretrained('xlm-roberta-base')

import torch.nn as nn
import torch.nn.functional as F
from torch.autograd import Variable
print('XLM Roberta model downloaded')

OUTPUT_DIM = 3
DROPOUT = 0.3
N_FILTERS = 100
FILTER_SIZES = [2,3,4]
HIDDEN_DIM = 100

model_names = ["CNN_Model", "Attention_Model"]
models = [  BERTCNNSentiment(bert, OUTPUT_DIM, DROPOUT, N_FILTERS, FILTER_SIZES),
            AttentionModel(bert, BATCH_SIZE, OUTPUT_DIM, HIDDEN_DIM, 50000, 768)  ]

def count_parameters(model):
    return sum(p.numel() for p in model.parameters() if p.requires_grad)

for i in range(2):
    print(f'The {models[i]} has {count_parameters(models[i]):,} trainable parameters')

for i in range(2):
    print("Parameters for " + f'{model_names[i]}')
    for name, param in models[i].named_parameters():
        if param.requires_grad:
            print(name)

import torch.optim as optim
from sklearn.metrics import confusion_matrix
def clip_gradient(model, clip_value):
    params = list(filter(lambda p: p.grad is not None, model.parameters()))
    for p in params:
        p.grad.data.clamp_(-clip_value, clip_value)

optimizers = [optim.Adam(models[0].parameters()), optim.Adam(models[1].parameters())]

criterion = nn.CrossEntropyLoss()
nll_loss = nn.NLLLoss()
log_softmax = nn.LogSoftmax()

for i in range(2):
    models[i] = models[i].to(device)
criterion = criterion.to(device)
nll_loss = nll_loss.to(device)
log_softmax = log_softmax.to(device)

from sklearn.metrics import f1_score

def categorical_accuracy(preds, y):
    count0,count1,count2 = torch.zeros(1),torch.zeros(1),torch.zeros(1)
    count0 = torch.zeros(1).to(device)
    count1 = torch.zeros(1).to(device)
    count2 = torch.zeros(1).to(device)
    total0,total1,total2 = torch.FloatTensor(1),torch.FloatTensor(1),torch.FloatTensor(1)
    max_preds = preds.argmax(dim = 1, keepdim = True) # get the index of the max probability
    correct = max_preds.squeeze(1).eq(y)
    predictions = max_preds.squeeze(1)
    true_correct = [0,0,0]
    for j,i in enumerate(y.cpu().numpy()):
      true_correct[y.cpu().numpy()[j]]+=1
      if i==0:
        count0+=correct[j]
        total0+=1
      elif i==1:
        count1+=correct[j]
        total1+=1
      elif i==2:
        count2+=correct[j]
      else:
        total2+=1
    metric=torch.FloatTensor([count0/true_correct[0],count1/true_correct[1],count2/true_correct[2],f1_score(y.cpu().numpy(),predictions.cpu().numpy(),average='macro')])
    return correct.sum() / torch.FloatTensor([y.shape[0]]),metric,confusion_matrix(y.cpu().numpy(),max_preds.cpu().numpy())

def train(model, iterator, optimizer, criterion, i):

    epoch_loss = 0
    epoch_acc = 0

    model.train()

    for batch in iterator:
        optimizer.zero_grad()

        if (i == 0):
          predictions =  model(batch.text).squeeze(1)
        else:
          predictions =  model(batch.text, batch_size = len(batch)).squeeze(1)

        loss = criterion(predictions, batch.label)

        acc,_,_ = categorical_accuracy(predictions, batch.label)

        loss.backward()
        clip_gradient(model, 1e-1)
        optimizer.step()

        epoch_loss += loss.item()
        epoch_acc += acc.item()

    return epoch_loss / len(iterator), epoch_acc / len(iterator)

def evaluate(model, iterator, criterion, i):

    epoch_loss = 0
    epoch_acc = 0
    epoch_all_acc = torch.FloatTensor([0,0,0,0])
    confusion_mat = torch.zeros((3,3))
    confusion_mat_temp = torch.zeros((3,3))

    model.eval()

    with torch.no_grad():

        for batch in iterator:
            if (i == 0):
              predictions = model(batch.text).squeeze(1)
            else:
              predictions = model(batch.text,batch_size=len(batch)).squeeze(1)

            loss = criterion(predictions, batch.label)

            acc,all_acc,confusion_mat_temp = categorical_accuracy(predictions, batch.label)
            epoch_loss += loss.item()
            epoch_acc += acc.item()
            epoch_all_acc += all_acc
            confusion_mat+=confusion_mat_temp
    return epoch_loss / len(iterator), epoch_acc / len(iterator),epoch_all_acc/len(iterator),confusion_mat

import time

def epoch_time(start_time, end_time):
    elapsed_time = end_time - start_time
    elapsed_mins = int(elapsed_time / 60)
    elapsed_secs = int(elapsed_time - (elapsed_mins * 60))
    return elapsed_mins, elapsed_secs

N_EPOCHS = 40

best_f1 = [-1, -1]
for epoch in range(N_EPOCHS):
    for i in range(2):
        start_time = time.time()

        train_loss, train_acc = train(models[i], train_iterator, optimizers[i], criterion, i)
        valid_loss, valid_acc,tot,conf = evaluate(models[i], valid_iterator, criterion, i)
        f1 = tot[3]
        end_time = time.time()

        epoch_mins, epoch_secs = epoch_time(start_time, end_time)

        if f1 > best_f1[i]:
            best_f1[i] = f1

            path = model_save_names[i]
            print(path)
            torch.save(models[i].state_dict(), path)

        print(f'Epoch: {epoch+1:02} | Epoch Time: {epoch_mins}m {epoch_secs}s')
        print(f'\tTrain Loss: {train_loss:.3f} | Train Acc: {train_acc*100:.2f}%')
        print(f'\t Val. Loss: {valid_loss:.3f} |  Val. Acc: {valid_acc*100:.2f}%')
        print(tot)
        print(conf)

for i in range(2):
    path = model_save_names[i]
    models[i].load_state_dict(torch.load(path))

def ensemble_evaluate(models, iterator, criterion):

    epoch_loss = 0
    epoch_acc = 0
    epoch_all_acc = torch.FloatTensor([0,0,0,0])
    models[0].eval()
    models[1].eval()
    confusion_mat = torch.zeros((3,3))
    confusion_mat_temp = torch.zeros((3,3))

    with torch.no_grad():

        for batch in iterator:

          predictions0 = models[0](batch.text).squeeze(1)
          predictions1 = models[1](batch.text, batch_size=len(batch)).squeeze(1)

          predictions = F.softmax(predictions0, dim=1) * F.softmax(predictions1, dim=1)
          loss = criterion(predictions, batch.label)

          acc,all_acc,confusion_mat_temp = categorical_accuracy(predictions, batch.label)
          epoch_loss += loss.item()
          epoch_acc += acc.item()
          epoch_all_acc += all_acc
          confusion_mat += confusion_mat_temp
    print(confusion_mat)
    return epoch_loss / len(iterator), epoch_acc / len(iterator),epoch_all_acc/len(iterator)

def ensemble_write_to_file(models, test_iterator):
    label_dict = {'0':'negative', '1':'neutral', '2':'positive'}
    file = open("answer.txt", "w")
    file.write('Uid,Sentiment\n')
    count = 0
    for batch in test_iterator:
      predictions0 = models[0](batch.text).squeeze(1)
      predictions1 = models[1](batch.text, batch_size=len(batch)).squeeze(1)
      predictions = F.softmax(predictions0, dim=1) * F.softmax(predictions1, dim=1)
      max_preds = predictions.argmax(dim = 1, keepdim = True).detach().cpu().numpy()
      for i,row in enumerate(batch.uid.cpu().numpy()):
        count += 1
        label_number = max_preds[i][0]
        label_number_str = list(LABEL.vocab.stoi.keys())[list(LABEL.vocab.stoi.values()).index(label_number)]
        predicted_label_name = label_dict[label_number_str]
        if count != len(test_data):
          file.write('%s,%s\n'%(row,predicted_label_name))
        else:
          file.write('%s,%s'%(row,predicted_label_name))
    file.close()

valid_loss, valid_acc, tot = ensemble_evaluate(models, test_iterator, criterion)
print(f'\t Val. Loss: {valid_loss:.3f} |  Val. Acc: {valid_acc*100:.2f}%')
print(tot)

Here is the output which is I am getting. I have shown only the requird output necessary for debugging
Traceback (most recent call last):
  File "main.py", line 268, in <module>
    train_loss, train_acc = train(models[i], train_iterator, optimizers[i], criterion, i)
  File "main.py", line 212, in train
    acc,_,_ = categorical_accuracy(predictions, batch.label)
  File "main.py", line 184, in categorical_accuracy
    count1+=correct[j]
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

Can someone tell me where to make the appropriate changes so that the code may run on the GPU in Google Colab.
Edit_1:
Thank you so much for the help you provided it actually solved my problem, but now a similar new problem arised. Note that I have updated the code to include the changes you mentioned
Traceback (most recent call last):
  File "main.py", line 271, in <module>
    train_loss, train_acc = train(models[i], train_iterator, optimizers[i], criterion, i)
  File "main.py", line 215, in train
    acc,_,_ = categorical_accuracy(predictions, batch.label)
  File "main.py", line 194, in categorical_accuracy
    return correct.sum() / torch.FloatTensor([y.shape[0]]),metric,confusion_matrix(y.cpu().numpy(),max_preds.cpu().numpy())
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!


title: Understanding pip installation behavior

body:
I was installing pytorch in an environment 'ptrch' i had created using conda. For my combination of requirements (windows, python 3.6, no cuda, conda) The official site asks for torchvision to be installed as well. it asked me to do pip3 install torchvision --no-deps. I did it from inside ptrch but torchvision got installed to global install location (Anaconda3\Lib\site-packages) instead of inside ptrch.

my understanding has been that pip installs to the env it is installed into

The reason i thought I found was that pip3 was available in Anaconda3/Scripts but there was no pip3 in ptrch. question - Is it the general behavior. it searches for inside environment and if the script isnt found there goes to global packages location? I think i know the answer - Anaconda3/Scripts is in my PATH. just needed to confirm if i know right
I copy pasted pip3 from Anaconda3/Scripts to envs/ptrch/Scripts and then ran pip3 install torchvision --no-deps from inside ptrch again. but it said all the requirements are satisfied. There is no torchvision in ptrch. question - Why did this happen?
I thought that maybe it is a side effect of caching or something. I decided to remove pip3 from its original location and now it is only in ptrch and now when i run pip3 install torchvision --no-deps it again says all the requirements are satisfied. question - why did this happen?

So now i know one thing that i am confused about how pip3 installation procedure works. Can somebody break it down to me as an answer to the scenario above. Also, how do i install torchvision inside ptrch? Please mention anything else you might think is important but is not directly related to the scenario above


title: What transformation do I need to do in order to run dataset through neural network?

body:
I'm new to deep learning and Pytorch, but I hope someone can help me out with this. My dataset contains images from different sizes. I'm trying to create a simple neural network that can classify images. However, I'm getting mismatch errors.

Neural network

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 32, 3)
        self.conv2 = nn.Conv2d(32, 32, 3)
        self.fc1 = nn.Linear(32 * 3 * 3, 200)
        self.fc2 = nn.Linear(200, 120)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        x = F.relu(self.conv2(x))
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        return x
net = Net()

My first convolution layer has 1 input channel, because I transform the images to grayscale images. 32 output channels was an arbitrary decision. The final fully-connected layer has 120 output channels, because there are 120 different classes.

Determine transformations and assign training set and validation set

transform = transforms.Compose(
    [transforms.Grayscale(1),
     transforms.RandomCrop((32,32)),
     transforms.ToTensor(),
     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

data_dir = 'dataset'
full_dataset = datasets.ImageFolder(os.path.join(data_dir, 'train'), transform = transform)

train_size = int(0.8 * len(full_dataset))
val_size = len(full_dataset) - train_size
trainset, valset = torch.utils.data.random_split(full_dataset, [train_size, val_size])

trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,
                                           shuffle=True, num_workers=2)
valloader = torch.utils.data.DataLoader(valset, batch_size=4,
                                           shuffle=False, num_workers=2)
classes = full_dataset.classes

I transform the images to grayscale, because they are gray anyway. I crop the images to 32, because the images have different sizes and I figured that they must all be the same size when putting it through the neural network. Everything is working fine so far.

Train neural network

for epoch in range(2):  # loop over the dataset multiple times

    running_loss = 0.0
    for i, data in enumerate(trainloader, 0):
        # get the inputs
        inputs, labels = data

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = net(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()
        if i % 2000 == 1999:    # print every 2000 mini-batches
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 2000))
            running_loss = 0.0

print('Finished Training')

When running this last piece of code, I get the following error: size mismatch, m1: [3584 x 28], m2: [288 x 200] at /Users/soumith/miniconda2/conda-bld/pytorch_1532623076075/work/aten/src/TH/generic/THTensorMath.cpp:2070 when the following line is being executed: outputs = net(inputs)

My code is a variation of the code provided in this Pytorch tutorial. Can someone tell me what I'm doing wrong?

UPDATE

I updated the neural network class to this:

class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()
        # 1 input image channel, 6 output channels, 5x5 square convolution
        # kernel
        self.conv1 = nn.Conv2d(1, 6, 5)
        self.conv2 = nn.Conv2d(6, 16, 5)
        # an affine operation: y = Wx + b
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        # Max pooling over a (2, 2) window
        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
        # If the size is a square you can only specify a single number
        x = F.max_pool2d(F.relu(self.conv2(x)), 2)
        x = x.view(-1, self.num_flat_features(x))
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

    def num_flat_features(self, x):
        size = x.size()[1:]  # all dimensions except the batch dimension
        num_features = 1
        for s in size:
            num_features *= s
        return num_features

net = Net()

But now I get an error at loss = criterion(outputs, labels):
Assertion cur_target >= 0 && cur_target < n_classes' failed.  at /Users/soumith/miniconda2/conda-bld/pytorch_1532623076075/work/aten/src/THNN/generic/ClassNLLCriterion.c:93


title: How to load a saved argparser and overwrite the current one?

body:
During training my model, I save some training arguments in argparser, something like:

torch.save({'iteration': iter,
            'model': model.state_dict(),
            'voc': voc,
            'args': args
            }, 'path')))

I was wondering how could I load the saved args and (more importantly) overwrite the current args? For example，

from conf import args
train_args = torch.load(args.path)['args']
args = train_args

However, the above codes are ineffective. How could I overwrite the variables in  args with the corresponding ones in train_args?


title: `conda install pytorch` results in 'Illegal instruction' with `import torch`

body:
With a fresh install of anaconda, followed by conda install pytorch, I get an Illegal instruction (crashing the python interpreter) whenever I import torch:

(base) ~$ python -c 'import torch'
Illegal instruction
(base) 132 ~$


title: What does group.setdefault() do in this context?

body:
I looked at the documentation [https://docs.python.org/2/library/stdtypes.html#dict.setdefault]

  setdefault(key[, default]):

  If key is in the dictionary, return its value.

  If not, insert key with a value of default and return default. default defaults to None

However, seeing as the output of group.setdefault() isn't being saved in a variable. I find this odd.
To be more specific, this code is included in the definition of a class of a scheduler. I am using pytorch and as I'm new, I'm having difficulties understanding the code. So any python/pytorch ninja that can figure this out?

Here's a simplified version of the class:

     class BaseLRScheduler(_LRScheduler):
         def __init__(self, optimizer, last_epoch=-1):

             self.optimizer = optimizer
             if last_epoch == -1:
                for group in optimizer.param_groups:
                    group.setdefault('initial_lr', group['lr'])

             self.base_lrs = list(
                  map(lambda group: group['initial_lr'], optimizer.param_groups))
             self.step(last_epoch + 1)
             self.last_epoch = last_epoch

This is the problematic part of code for me:

     if last_epoch == -1:
         for group in optimizer.param_groups:
              group.setdefault('initial_lr', group['lr'])


title: Batch Matrix Multiplication with Dynamic Batch Size

body:
I am writing a procedure using Batch Matrix Multiplication, maybe not in under a common setting. I am considering the following inputs:

# Let's say I have a list of points in R^3, from 3 distinct objects
# (so my data batch has 3 data entry)
# X: (B1+B2+B3) * 3
X = torch.tensor([[1,1,1],[1,1,1],
                  [2,2,2],[2,2,2],[2,2,2],
                  [3,3,3],])
# To indicate which object the points are corresponding to,
# I have a list of indices (say, starting from 0):
# idx: (B1+B2+B3)
idx = torch.tensor([0,0,1,1,1,2])
# For each point from the same object, I want to multiply it to a 3x3 matrix, A_i.
# As I have 3 objects here, I have A_0, A_1, A_2.
# A: 3 x 3 x 3
A = torch.tensor([[[1,1,1],[1,1,1],[1,1,1]],
                  [[2,2,2],[2,2,2],[2,2,2]],
                  [[3,3,3],[3,3,3],[3,3,3]]])

And the desired output is:

out = X.unsqueeze(1).bmm(A[idx])
out = out.squeeze(1)                # just to remove excessive dimension
# out = torch.tensor([[[1,1,1]],[[1,1,1]],            # obj0 mult with A_0
                      [[2,2,2]],[[2,2,2]],[[2,2,2]],  # obj1 mult with A_1
                      [[3,3,3]],])                    # obj2 mult with A_2

It is actually pretty handy in pytorch, just one line!

Here, I want to improve this procedure. Note that I am using A[idx] to replicate one matrix A_i for each point, so I can use torch.bmm() function here (1 point <-> 1 matrix). Afaik, it will need to allocate memory for intermediate representation of A[idx]. In general, if I have BN objects in my data batch, size of A[idx] = (B1+...+BN)*3*3, which can be very large.

Therefore, I am wondering if replication of matrix A_i can be avoided.

I have found most previously asked questions about Batch Mat. Mult. only assume fixed batch size. Here the same question as mine was asked, and a solution in tensorflow was provided. However, the solution was implemented using tf.tile(), which is also replicating matrices.

To summarize, my question is about batch matrix multiplication, while achieving:

- dynamic batch size
    - input shape: (B1+...+BN) x 3
    - index shape: (B1+...+BN)
- memory efficiency
    - probably w/out massive replication of matrix

I am using pytorch here, but I also accept other implementations. I also accept representing inputs (e.g. matrices to multiply, A) in other structures if it results in a better memory efficiency.


title: data_block api :show_batch cause CUDA unknown error

body:
I am a newbie and then trying to learn fastai data block api

Here is the mistake, the code is exactly the same as the tutorial：

coco = untar_data(URLs.COCO_TINY)
path=coco/'train.json'
images, lbl_bbox = get_annotations(coco/'train.json')
img2bbox = dict(zip(images, lbl_bbox))
get_y_func = lambda o:img2bbox[o.name]

data = (ObjectItemList.from_folder(coco)
    .split_by_rand_pct()
    .label_from_func(get_y_func)
    .transform(get_transforms(), tfm_y=True)
    .databunch(bs=1, num_workers=0,collate_fn=bb_pad_collate))

data.show_batch(rows=2, ds_type=DatasetType.Valid, figsize=(6,6))

Then the error is：

  File "D:\Anaconda3\envs\pytorch-gpu\lib\site-packages\IPython\core\interactiveshell.py", line
 3326, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-7-25e60680c0ba>", line 15, in <module>
   data.show_batch(rows=2, ds_type=DatasetType.Valid, figsize=(6,6))
File "D:\Anaconda3\envs\pytorch-gpu\lib\site-packages\fastai\basic_data.py", line 185, in show_batch
   x,y = self.one_batch(ds_type, True, True)
File "D:\Anaconda3\envs\pytorch-gpu\lib\site-packages\fastai\basic_data.py", line 168, in one_batch
  try:     x,y = next(iter(dl))
File "D:\Anaconda3\envs\pytorch-gpu\lib\site-packages\fastai\basic_data.py", line 75, in __iter__
  for b in self.dl: yield self.proc_batch(b)
File "D:\Anaconda3\envs\pytorch-gpu\lib\site-packages\torch\utils\data\dataloader.py", line 348,__next__
  data = _utils.pin_memory.pin_memory(data)
File "D:\Anaconda3\envs\pytorch-gpu\lib\site-packages\torch\utils\data\_utils\pin_memory.py", line
  55, in pin_memory
  return [pin_memory(sample) for sample in data]
File "D:\Anaconda3\envs\pytorch-gpu\lib\site-packages\torch\utils\data\_utils\pin_memory.py", line
  55, in <listcomp>
  return [pin_memory(sample) for sample in data]
File "D:\Anaconda3\envs\pytorch-gpu\lib\site-packages\torch\utils\data\_utils\pin_memory.py", line
  47, in pin_memory
return data.pin_memory()
RuntimeError: CUDA error: unknown error

Regarding this error, the forums on the Internet are all setting the parameters of the DataLoader, but it does not seem to be used here

How would I go about this?


title: Cuda version for cat in Pytorch?

body:
I am trying to build a CNN architecture for Word Embedding. I am trying to concatenate two tensors using torch.cat but its throwing this error:

     22         print(z1.size())
---> 23         zcat = torch.cat(lz, dim = 2)
     24         print("zcat",zcat.size())
     25         zcat2=zcat.reshape([batch_size, 1, 100, 3000])

RuntimeError: Expected object of backend CUDA but got backend CPU for sequence element 1 in sequence argument at position #1 'tensors'

Attaching the architecture for reference as well:

    def __init__(self,vocab_size,embedding_dm,pad_idx):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size,embedding_dim,padding_idx = pad_idx)
        self.convs = nn.ModuleList([nn.Conv2d(in_channels = 1,out_channels = 50,kernel_size = (1,fs)) for fs in (3,4,5)])
        self.conv2 = nn.Conv2d(in_channels = 50,out_channels = 100,kernel_size = (1,2))
        self.fc1 = nn.Linear(100000,150) #Change this
        self.fc2 = nn.Linear(150,1)
        self.dropout = nn.Dropout(0.5)
    def forward(self,text):
        print("text",text.size())
        embedded = self.embedding(text.T)
        embedded = embedded.permute(0, 2,1)
        print("embedded",embedded.size())
        x=embedded.size(2)
        y=3000-x
        print(y,"hello")
        batch_size=embedded.size(0)
        z=np.zeros((batch_size,100,y))
        z1=torch.from_numpy(z).float()
        lz=[embedded,z1]
        print(z1.size())
        zcat = torch.cat(lz, dim = 2)
        print("zcat",zcat.size())
        zcat2=zcat.reshape([batch_size, 1, 100, 3000])
        print("zcat2",zcat2.size())
#         embedded = embedded.reshape([embedded.shape[0],1,])
        print(embedded.size(),"embedding")
        conved = [F.relu(conv(embedded)) for conv in self.convs]
        pooled = [F.max_pool2d(conv,(1,2)) for conv in conved]
        print("Pool")
        for pl in pooled:
            print(pl.size())
        cat = torch.cat(pooled,dim = 3)
        print("cat",cat.size())
        conved2 = F.relu(self.conv2(cat))
        print("conved2",conved2.size())
        pooled2 = F.max_pool2d(conved2,(1,2))
        print(pooled2.size(),"pooled2")
        return 0
#         return pooled2

Am I doing something wrong? Help appreciated. Thanks!


title: How can I resize Olivetti Dataset images 64x64 to 32x32 ?? I am getting error

body:
batch_boyut = 2

train_loader = torch.utils.data.DataLoader(
    X_egitim, batch_size=batch_boyut)
val_loader = torch.utils.data.DataLoader(
    X_val, batch_size=batch_boyut)

class CNNModule(nn.Module):
    def __init__(self):
        super(CNNModule, self).__init__()
        self.conv1 = nn.Conv2d(1, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 40)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 5 * 5)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

model = CNNModule()
print(model)
model.forward(X_egitim)

I am getting this error.How can I solve? How can resize olivetti dataset images from 64x64 to 32x32 ?
I think I should change x.view but I don't know how I can

x = x.view(-1, 16 * 5 * 5)

RuntimeError: shape '[-1, 400]' is invalid for input of size 865280


title: How to build wheel files with CUDA support?

body:
How do I create .whl wheel file with CUDA support? I have a fork of facebook's detectron2 project and I have made some changes. Now I would like to build the project. When I run this command: python setup.py bdist_wheel, it creates a wheel file with the name: projectname-0.1.1-cp37-cp37m-linux_x86_64.whl. However, when I check the official wheels website of facebook's detectron2, the wheel files have a unique naming convention i.e. detectron2-0.1.3+cu102-cp38-cp38-linux_x86_64.whl. Here you can see that the project was built with CUDA10.2 capabilities. I assume my project's wheel files are built to be run only on the CPU. How can I achieve the same? Or am I wrong in my assumption?

Thanks for the help!