title: Can pytorch's autograd handle torch.cat?

body:
I'm trying to implement a simple neural network that is supposed to learn an grayscale image. The input consist of the 2d indices of a pixel, the output should be the value of that pixel.

The net is constructed as follows: Each neuron is connected to the input (i.e. the indices of the pixel) as well as to the output of each of the previous neurons. The output is just the output of the last neuron in this sequence.

This kind of network has been very successfull in learning images, as demonstrated e.g. here.

The Problem:
In my implementation the loss function stays between 0.2 and 0.4 depending on the number of neurons, the learning rate and the number of iterations used, which is pretty bad. Also if you compare the output to what what we've trained it for there it just looks like noise. But this is the first time I used torch.cat within the network, so I'm not sure whether this is the culprit. Can anyone see what I'm doing wrong?

from typing import List
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.nn import Linear

class My_Net(nn.Module):
    lin: List[Linear]

    def __init__(self):
        super(My_Net, self).__init__()
        self.num_neurons = 10
        self.lin = nn.ModuleList([nn.Linear(k+2, 1) for k in range(self.num_neurons)])

    def forward(self, x):
        v = x
        recent = torch.Tensor(0)
        for k in range(self.num_neurons):
            recent = F.relu(self.lin[k](v))
            v = torch.cat([v, recent], dim=1)
        return recent

    def num_flat_features(self, x):
        size = x.size()[1:]
        num = 1
        for i in size():
            num *= i
        return num

my_net = My_Net()
print(my_net)

#define a small 3x3 image that the net is supposed to learn
my_image = [[1.0, 1.0, 1.0], [0.0, 1.0, 0.0], [0.0, 1.0, 0.0]] #represents a T-shape
my_image_flat = []    #output of the net is the value of a pixel
my_image_indices = [] #input to the net is are the 2d indices of a pixel
for i in range(len(my_image)):
    for j in range(len(my_image[i])):
        my_image_flat.append(my_image[i][j])
        my_image_indices.append([i, j])

#optimization loop
for i in range(100):
    inp = torch.Tensor(my_image_indices)

    out = my_net(inp)

    target = torch.Tensor(my_image_flat)
    criterion = nn.MSELoss()
    loss = criterion(out.view(-1), target)
    print(loss)

    my_net.zero_grad()
    loss.backward()
    optimizer = optim.SGD(my_net.parameters(), lr=0.001)
    optimizer.step()

print("output of current image")
print([[my_net(torch.Tensor([[i,j]])).item() for i in range(3)] for j in range(3)])
print("output of original image")
print(my_image)


title: Torch - Interpolate missing values

body:
I have a stock of tensor images of a form NumOfImagesxHxW that includes zeros. I am looking for a way to interpolate the missing values (zeros) using the information in the same image only (no connection between the images). Is there a way to do it using pytorch?

F.interpolate seems to work only for reshaping. I need to fill the zeros, while keeping the dimensions and the gradients of the tensor.

Thanks.


title: How to train my neural network faster by running CPU and GPU in parallel

body:
I'm trying to train a (pretty big) neural network using a GPU. The network is written in pytorch. I use python 3.6.3 running on ubuntu 16.04. Currently, the code is running, but it's taking about twice as long as it should to run because my data-grabbing process using the CPU is run in series to the training process using the GPU. Essentially, I grab a mini-batch from file using a mini-batch generator, send that mini-batch to the GPU and then train the network on that minibatch. I've timed the two processes (grabbing a mini batch and training on that mini batch), and they are similar in how long they take (both take around 200ms). I'd like to do something similar to keras' fit_generator method which runs the data-grabbing in parallel to the training (it creates a que of minibatches that can be sent to the GPU when the GPU wants to train on that mini batch). What is the best way to do that?  For concreteness, my data generator code and training code run something like this (pseudocode):

    #This generator opens a file, grabs and yields a mini batch
    def data_gen(PATH,batch_size=32):
        with h5py.File(PATH,'r') as f:
            for mini-batch in mini-batches:
                X = f['X'][mini-batch]
                Y = f['Y'][mini-batch]
                yield (X,Y)

    for epoch in range(epochs):
        for data in data_gen(PATH):
                mini_X,mini_Y = data
                mini_X = autograd.Variable(torch.Tensor(mini_X))
                mini_Y = autograd.Variable(torch.Tensor(mini_Y))
                out = net(mini_X)
                loss = F.binary_cross_entropy(out,mini_Y)
                loss.backward()
                optimizer.step()

Something like that. As you can see, I use the data_gen as an actual generator for the for-loop, so it's being run sequentially with the training. I would like to run it in parallel and have it generate a que of minibatches which I can then feed to my network. Currently, it takes more than 5 hours to run one epoch, I think with a parallelized version of this, I could get that down to 3 hours or less. I looked into multiprocessing on python, but the explanation on the official documentation was a bit dense for me since I have only limited prior experience in parallel computing. If there's some resources I could take a look at, pointing me towards those resources would be very helpful too! Thanks.


title: In Torch C++ API, How to write to the internal data of a tensor fastly?

body:
I am using torch C++ frontend and want to have a tensor with specified value in it. To achieve this one may allocate memory and set value by hand, then use torch::from_blob to build a tensor on the memory block, but it seems not clean enough for me.

In the very bottom of this document I found out that I can use subscript to directly access and modify the data. However, this approach has a big running time overhead, likely because the subscript access will treat the element of tensor as a 0-d tensor. The following code will cost more than 2 seconds on my machine (-O3 optimization level), which is unreasonably long for modern CPU.

    torch::Tensor tensor = torch::empty({1000, 1000});
    for(int i=0; i < 1000; i++)
    {
        for(int j=0 ; j < 1000; j++)
        {
            tensor[i][j] = calc_tensor_data(i,j);
        }
    }

Is there a clean and fast way to achieve this goal?


title: pytorch: how to directly find gradient w.r.t. loss

body:
In theano, it was very easy to get the gradient of some variable w.r.t. a given loss:

loss = f(x, w)
dl_dw = tt.grad(loss, wrt=w)

I get that pytorch goes by a different paradigm, where you'd do something like:

loss = f(x, w)
loss.backwards()
dl_dw = w.grad

The thing is I might not want to do a full backwards propagation through the graph - just along the path needed to get to w.

I know you can define Variables with requires_grad=False if you don't want to backpropagate through them.  But then you have to decide that at the time of variable-creation (and the requires_grad=False property is attached to the variable, rather than the call which gets the gradient, which seems odd).

My Question is is there some way to backpropagate on demand (i.e. only backpropagate along the path needed to compute dl_dw, as you would in theano)?


title: Compute trace of matrix product using numpy/pytorch broadcasting

body:
Let A be an (nxm)-matrix and M an (mxm)-matrix. Writing tr() for the trace of a matrix, I need to compute tr(AM(A^T)). However, the final trace operation throws away most of the computation. Can I use numpy's or pytorch's broadcasting rules to compute only the necessary diagonal of AM(A^T)?

Update:
Here is my solution to compute the diagonal in PyTorch:

torch.sum(torch.sum(A.t()[:,None,:]*M[:,:,None],0)*A.t(),0)


title: PyTorch - How to Save Custom Dataset to disk to use with torchvision.datasets?

body:
I'd like to create a custom PyTorch dataset of ZCA-whitened CIFAR-10 that I can subsequently load using torchvision's function torchvision.datasets.CIFAR10(). So far, I can successfully whiten the data (see code below), but I don't know how to save the data to disk in a manner that allows it to be loaded using torchvision.datasets.CIFAR10(). How do I do this?

Code to ZCA-whiten CIFAR 10:

    trainset = torchvision.datasets.CIFAR10(
        root='./datasets',
        train=True,
        download=False)
    train_data = trainset.data.reshape(-1, 32*32*3)
    zca_matrix = zca_whitening_matrix(train_data.T)
    whitened_training_data = np.matmul(zca_matrix, train_data.T).T
    whitened_training_data = whitened_training_data.reshape((-1, 32, 32, 3))

    # whiten CIFAR-10 testing data
    testset = torchvision.datasets.CIFAR10(
        root='./datasets',
        train=False,
        download=False)
    testdata = testset.data.reshape(-1, 32*32*3)
    whitened_test_data = np.matmul(zca_matrix, testdata.T).T
    whitened_test_data = whitened_test_data.reshape((-1, 32, 32, 3))

Is the best way to really just save the numpy arrays, as shown here?

PyTorch: How to use DataLoaders for custom Datasets


title: Multi-Step time series LSTM Network in pytorch

body:
I am having issues with the LSTM function in pytorch. I am using an LSTM neural network to forecast a certain value. The input is multidimensional (multiple features) and the output should be one dimensional (only one feature that needs to be forecasted). I want to forecast something 1-6 timesteps in advance. I want to use multi timestep input as well. Now I have to different ways of achieving this but neither of them seem to work.

The first one is making 6 different many-to-one networks each forecasting another timestep 1-6h in advance, but still using the recurrent structure (see picture) when forecasting multiple time steps ahead (so not just shifting the target data by 1-6 hours). Another option would be a many to many neural network, this would work fine as well I think. (also see picture)

My input now looks like: where first the different parameters at the same timestep are grouped and then further each time all the timesteps we want to use in the forecast are grouped (look back)

tensor([[[-0.2800, -0.6381, -0.1033, -0.4941,  0.0016],
         [-0.3159,  0.1378, -0.1010, -0.4529,  0.0016],
         [-0.2800,  0.1378, -0.0963, -0.4706,  0.1150],
         ...,
         [-0.5673, -0.2149, -0.0598, -0.4000,  0.2850],
         [-0.3518, -0.4265, -0.0669, -0.3646,  0.3417],
         [-0.2440, -0.0738, -0.0657, -0.3823,  0.2283]],

        [[-0.3159,  0.1378, -0.1010, -0.4529,  0.0016],
         [-0.2800,  0.1378, -0.0963, -0.4706,  0.1150],
         [-0.7469,  0.1731, -0.0845, -0.4176,  0.3417],
         ...,
         [-0.3518, -0.4265, -0.0669, -0.3646,  0.3417],
         [-0.2440, -0.0738, -0.0657, -0.3823,  0.2283],
         [-0.1722, -0.5323, -0.0610, -0.4117,  0.2283]],

        [[-0.2800,  0.1378, -0.0963, -0.4706,  0.1150],
         [-0.7469,  0.1731, -0.0845, -0.4176,  0.3417],
         [-0.7829, -0.4265, -0.0692, -0.4176,  0.4550],
         ...,
         [-0.2440, -0.0738, -0.0657, -0.3823,  0.2283],
         [-0.1722, -0.5323, -0.0610, -0.4117,  0.2283],
         [-0.1363, -0.8850, -0.0669, -0.4294,  0.1150]],

        ...,

        [[-0.3518,  0.2083, -0.1386,  0.8479, -0.1684],
         [-0.3518,  0.4552, -0.1398,  0.9480,  0.0016],
         [-0.2800, -0.4265, -0.1398,  0.9126,  0.0583],
         ...,
         [-1.0343, -0.1443, -0.1433,  0.8479,  0.0016],
         [-0.8906,  0.3847, -0.1445,  1.0304, -0.2251],
         [-0.7829, -0.0385, -0.1433,  1.0127, -0.1117]],

        [[-0.3518,  0.4552, -0.1398,  0.9480,  0.0016],
         [-0.2800, -0.4265, -0.1398,  0.9126,  0.0583],
         [-0.4596, -0.9202, -0.1410,  0.8479,  0.1150],
         ...,
         [-0.8906,  0.3847, -0.1445,  1.0304, -0.2251],
         [-0.7829, -0.0385, -0.1433,  1.0127, -0.1117],
         [-0.8547,  0.2436, -0.1422,  0.9715, -0.0550]],

        [[-0.2800, -0.4265, -0.1398,  0.9126,  0.0583],
         [-0.4596, -0.9202, -0.1410,  0.8479,  0.1150],
         [-0.6392, -0.5323, -0.1422,  0.8655,  0.0016],
         ...,
         [-0.7829, -0.0385, -0.1433,  1.0127, -0.1117],
         [-0.8547,  0.2436, -0.1422,  0.9715, -0.0550],
         [-0.9984, -0.0033, -0.1422,  0.8597,  0.0583]]])

And the output looks like, where the different timesteps 1-6h in advance are grouped.  I can change this easily.

tensor([[[ -7.],
         [ -9.],
         [-11.],
         [-13.],
         [-13.]],

        [[ -9.],
         [-11.],
         [-13.],
         [-13.],
         [-10.]],

        [[-11.],
         [-13.],
         [-13.],
         [-10.],
         [ -9.]],

        ...,

        [[-12.],
         [-10.],
         [-10.],
         [-10.],
         [ -9.]],

        [[-10.],
         [-10.],
         [-10.],
         [ -9.],
         [ -8.]],

        [[-10.],
         [-10.],
         [ -9.],
         [ -8.],
         [-10.]]])

Now I have no idea how to use the LSTM structure to do multi timesteps forecasting. The output layer should be linear.

The batch size does not really matter to me, I think it can be one for now.


title: Understanding the backward mechanism of LSTMCell in Pytorch

body:
I want to hook into the backward pass of a LSTMCell function in pytorch so in the initialization pass I do the following (num_layers=4, hidden_size=1000, input_size=1000):

self.layers = nn.ModuleList([
        LSTMCell(
            input_size=input_size,
            hidden_size=hidden_size,
        )
        for layer in range(num_layers)
    ])

for l in self.layers:
    l.register_backward_hook(backward_hook)

In the forward pass I simply iterate the LSTMCell over sequence length and the num_layers as follow:

for j in range(seqlen):
    input = #some tensor of size (batch_size, input_size)
    for i, rnn in enumerate(self.layers):
        # recurrent cell
        hidden, cell = rnn(input, (prev_hiddens[i], prev_cells[i]))

Where input is of size (batch_size, input_size), prev_hiddens[i] is size of (batch_size, hidden_size), prev_cells[i] is of size (batch_size, hidden_size).

In the backward_hook I print the size of the tensors that are input to this function:

def backward_hook(module, grad_input, grad_output):
    for grad in grad_output:
        print ("grad_output {}".format(grad))

    for grad in grad_input:
         print ("grad_input.size () {}".format(grad.size()))

As the results, for the first time backward_hook is called for example:

[A] For grad_output I get 2 tensors among which the second tensor is None. This is understandable because in the backward phase we have a gradient of internal states (c) and gradient of output (h). The last iteration in time dimension has no future hidden so its gradient is None.

[B] For grad_input I get 5 tensors (batch_size=9):

grad_input.size () torch.Size([9, 4000])
grad_input.size () torch.Size([9, 4000])
grad_input.size () torch.Size([9, 1000])
grad_input.size () torch.Size([4000])
grad_input.size () torch.Size([4000])

My questions are:

(1) Is my understanding from [A] correct?

(2) How do I interpret the 5 tensors from the grad_input tuple? I thought there should have only 3 since there are only 3 inputs to the LSTMCell forward()?

Thanks


title: How to impove the speed of tf.data.experimental.CsvDataset in tensorflow 1.13.1?

body:
I am coding a toy example for csv data in Tensorflow.
I have implemented three types of data loader in tensorflow and pytorch to compare their speeds. Here is the code:

First, with tensorflow api tf.data.experimental.CsvDataset:

def parse_data(x, n_classes):
    x = tf.convert_to_tensor(x)
    return x[:-1], tf.one_hot(indices=tf.cast(x[-1], tf.int32), depth=n_classes)

if __name__=='__main__':
    dataset_train = tf.data.experimental.CsvDataset('/home/david/Dataset/timit/test.csv', [tf.float32] * 430,
                                                    header=False,
                                                    field_delim=' ')
    dataset_train = dataset_train.map(lambda *x_: parse_data(x_, 1928))
    dataset_train = dataset_train.batch(128)
    dataset_train = dataset_train.prefetch(1)
    iterator = dataset_train.make_initializable_iterator()

    x_in, y = iterator.get_next()

    x = tf.layers.Dense(units=1024, activation=tf.nn.relu)(x_in)
    x = tf.layers.Dense(units=1024, activation=tf.nn.relu)(x)
    x = tf.layers.Dense(units=1024, activation=tf.nn.relu)(x)
    x = tf.layers.Dense(units=1024, activation=tf.nn.relu)(x)
    logits = tf.layers.Dense(units=1928, activation=None)(x)

    loss = tf.losses.softmax_cross_entropy(y, logits)
    optimizer = tf.train.AdamOptimizer()
    optimizer.minimize(loss)

    sess = tf.Session()
    sess.run(tf.global_variables_initializer())
    sess.run(iterator.initializer)
    running_loss = 0.0
    time_last = time.time()
    epoch = 0
    i = 0
    while True:
        try:
            running_loss += sess.run(loss)  # , feed_dict={x: data, y: labels})
            if (i + 1) % 5 == 0:
                print('\r[epoch: %2d, batch: %5d, time: %5f] loss: %.3f' % (
                    epoch + 1, i + 1, time.time() - time_last, running_loss / i), end=' ')
                time_last = time.time()
            i += 1
        except tf.errors.OutOfRangeError:
            pass

Second, using pandas and tf.placeholder:

if __name__ == '__main__':
    x_in = tf.placeholder(shape=[128, 429], dtype=tf.float32)
    y_in = tf.placeholder(shape=[128], dtype=tf.int32)
    y = tf.one_hot(y_in, depth=1928)

    x = tf.layers.Dense(units=1024, activation=tf.nn.relu)(x_in)
    x = tf.layers.Dense(units=1024, activation=tf.nn.relu)(x)
    x = tf.layers.Dense(units=1024, activation=tf.nn.relu)(x)
    x = tf.layers.Dense(units=1024, activation=tf.nn.relu)(x)
    logits = tf.layers.Dense(units=1928, activation=None)(x)

    loss = tf.losses.softmax_cross_entropy(y, logits)
    optimizer = tf.train.AdamOptimizer()
    optimizer.minimize(loss)

    sess = tf.Session()
    sess.run(tf.global_variables_initializer())

    w = pd.read_csv('/home/david/Dataset/timit/test.csv', header=None, delim_whitespace=True).values

    for epoch in range(23):
        running_loss = 0.0
        time_last = time.time()
        i = 0
        indexes = np.random.permutation(w.shape[0])
        w_ = w[indexes, :]
        while True:
            if i * 128 + 128 > w.shape[0]:
                break
            running_loss += sess.run(loss,
                                     feed_dict={x_in: w_[i * 128:i * 128 + 128, :-1],
                                                y_in: w_[i * 128:i * 128 + 128, -1]})
            if (i + 1) % 5 == 0:
                print('\r[epoch: %2d, batch: %5d, time: %5f] loss: %.3f' % (
                    epoch + 1, i + 1, time.time() - time_last, running_loss / i), end=' ')
                time_last = time.time()
            i += 1

Third, with pytorch and pandas:

class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()

        self.fc1 = nn.Linear(429, 1024)
        self.fc2 = nn.Linear(1024, 1024)
        self.fc3 = nn.Linear(1024, 1024)
        self.fc4 = nn.Linear(1024, 1024)
        self.fc5 = nn.Linear(1024, 1928)

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.relu(self.fc3(x))
        x = F.relu(self.fc4(x))
        x = self.fc5(x)
        return x

class CsvDataset(data.Dataset):
    """Face Landmarks dataset."""

    def __init__(self, csv_file):
        """
        Args:
            csv_file (string): Path to the csv file with annotations.
            root_dir (string): Directory with all the images.
            transform (callable, optional): Optional transform to be applied
                on a sample.
        """
        self.landmarks_frame = pd.read_csv(csv_file, header=None, delim_whitespace=True)

    def __len__(self):
        return len(self.landmarks_frame)

    def __getitem__(self, idx):
        landmarks = self.landmarks_frame.values[idx, :]
        return landmarks[:-1], landmarks[-1]

if __name__ == '__main__':
    net = Net()
    device = torch.device('cuda:0')
    print(device)
    net.to(device)

    optimizer = optim.Adam(net.parameters(), lr=0.001)
    criterion = nn.CrossEntropyLoss()

    dataset = CsvDataset('/home/david/Dataset/timit/train.csv')
    trainloader = torch.utils.data.DataLoader(dataset, batch_size=128, shuffle=True)

    for epoch in range(23):

        running_loss = 0.0
        time_last = time.time()
        for i, data in enumerate(trainloader):
            inputs, labels = data
            inputs = inputs.float().to(device)
            labels = labels.long().to(device)

            optimizer.zero_grad()

            output = net(inputs)
            loss = criterion(output, labels)
            loss.backward()
            optimizer.step()

            running_loss += loss.item()
            if (i + 1) % 5 == 0:
                print('\r[epoch: %2d, batch: %5d, time: %5f] loss: %.3f' % (
                    epoch + 1, i + 1, time.time() - time_last, running_loss / i), end=' ')
                time_last = time.time()
        print('')
    print('Finished Training')

I record the time cost for training in five batches:

First, CsvDataset: 1.382647s
Second, tf.placeholder: 0.013263s
Third, pytorch: 0.042086s

I guess it is partly because that tf.data.experimental.CsvDataset do io operations before each batch to extract data from csv file (Is this ture, or there are other reasons?).

However, it is too slow comparing to the other two. Is there any chance for improvement? How can I set the tf.data.experimental.CsvDataset api to load all csv data at the very beginning?

Or can I say that tf.data.experimental.CsvDataset is implemented only for the csv dataset that is too big to store in the memory? Because the time cost seems like intolerable.


title: Difference of torch.matmul and python built-in @ operator to do matrix multiplication

body:
Can I always replace torch.matmul with python's built-in @ operator to do the matrix multiplication? Please assume that I know the difference between torch.matmul, torch.mm and many others. I just want to make sure how many of them can be safely replaced by @ operator without sacrificing speed or some native support from torch.

If it does no harm, I would like to extensively use them in the future.


As described in the issue I opened, I get the following error when running the Pytorch inverse-cooking model on CPU:

RuntimeError: expected device cpu and dtype Byte but got device cpu and dtype Bool

I have tried running the demo.ipynb file in both my laptop's Intel i7-4700HQ 8 threads and my desktop Ryzen 3700x. I was using Arch Linux on my laptop and Manjaro on my desktop.

The model works fine when I run it on Google Collabs GPU.

According to the demo.ipynb file the model should be able to run on CPU as well. Does anyone know if I have to tweak any parameters in order to make it work?


title: Convert string to byte for pytorch loader

body:
The method of downloading a pytorch model path is not in my control and I am trying to figure out a way to convert downloaded string data to byte data.  The code below downloads my saved model from Dropbox and uses bytes with utf-8 encoding to encode the string. The problem is when I use torch.load with BytesIO I get a UnpicklingError with invalid load key, '<'.

    data = bytes(self.Download("https://www.dropbox.com/s/exampleurl/checkpoint.pth?dl=1"), 'utf-8')

    self.agent.local.load_state_dict(torch.load(BytesIO(data ), map_location=lambda storage, loc: storage))

The code below worked perfectly until requests was disabled and I am now trying to use the method above.

    dropbox_url = "https://www.dropbox.com/s/exampleurl/checkpoint.pth?dl=1"

    data = requests.get(dropbox_url )

    self.agent.local.load_state_dict(torch.load(BytesIO(data.content), map_location=lambda storage, loc: storage))

I just need to figure out a way to convert the string to bytes data the correct way.


title: Training TFBertForSequenceClassification with custom X and Y data

body:
I am working on a TextClassification problem, for which I am trying to traing my model on TFBertForSequenceClassification given in huggingface-transformers library.

I followed the example given on their github page, I am able to run the sample code with given sample data using tensorflow_datasets.load('glue/mrpc').
However, I am unable to find an example on how to load my own custom data and pass it in
model.fit(train_dataset, epochs=2, steps_per_epoch=115, validation_data=valid_dataset, validation_steps=7).

How can I define my own X, do tokenization of my X and prepare train_dataset with my X and Y. Where X represents my input text and Y represents classification category of given X.

Sample Training dataframe :

    text    category_index
0   Assorted Print Joggers - Pack of 2 ,/ Gray Pri...   0
1   "Buckle" ( Matt ) for 35 mm Width Belt  0
2   (Gagam 07) Barcelona Football Jersey Home 17 1...   2
3   (Pack of 3 Pair) Flocklined Reusable Rubber Ha...   1
4   (Summer special Offer)Firststep new born baby ...   0


title: How can one clone a Pytoch dataset into another variable?

body:
I want to create several subsets of the MNIST dataset provided in Pytorch. Each subset should have different classes. What I tried was the following:

def split_MNIST(mnist_set, digits):
    dset = mnist_set
    classes = []
    indices = dset.targets == digits[0]
    classes.append(dset.classes[digits[0]])
    if len(digits) > 1:
        for digit in digits[1:]:
            idx = dset.targets == digit
            indices = indices + idx
            classes.append(dset.classes[digit])
    dset.targets = dset.targets[indices]
    dset.data = dset.data[indices]
    dset.classes = classes
    return dset

train = datasets.MNIST("../data", train=True, download=True,
                        transform=transforms.Compose([transforms.ToTensor()]))

test =datasets.MNIST("../data", train=False, download=True,
                      transform=transforms.Compose([transforms.ToTensor()]))

tr = split_MNIST(train, [1,2,3])

trainset = torch.utils.data.DataLoader(tr, batch_size=16, shuffle=True)

This works, but instead of creating a new dataset, it actually changes the original train variable. Is there a way to create a clone of the dataset instead to preserve the original one?


title: How is log_softmax() implemented to compute its value (and gradient) with better speed and numerical stability?

body:
Both MXNet and PyTorch provide special implementation for computing log(softmax()), which is faster and numerically more stable. However, I cannot find the actual Python implementation for this function, log_softmax(), in either package.

Can anyone explain how this is implemented, or better, point me to the relevant source code?


title: Autoencoder MaxUnpool2d missing 'Indices' argument

body:
The following model returns the error:
TypeError: forward() missing 1 required positional argument: 'indices'

I've exhausted many online examples and they all look similar to my code. My maxpool layer returns both the input and the indices for the unpool layer. Any ideas on what's wrong?

class autoencoder(nn.Module):
def __init__(self):
    super(autoencoder, self).__init__()
    self.encoder = nn.Sequential(
        ...
        nn.MaxPool2d(2, stride=1, return_indices=True)
    )
    self.decoder = nn.Sequential(
        nn.MaxUnpool2d(2, stride=1),
        ...
    )

def forward(self, x):
    x = self.encoder(x)
    x = self.decoder(x)
    return x


title: Can autograd in pytorch handle a repeated use of a layer within the same module?

body:
Suppose I have a layer layer in a torch module and use it twice or more times during a single forward step, in a way that the result output by this layer is later again inputed to the same layer. Can pytorch's autograd compute the grad of the weights of this layer correctly?

Here is an mwe of what I am talking about:

import torch
import torch.nn as nn
import torch.nn.functional as F

class net(nn.Module):
    def __init__(self,in_dim,out_dim):
        super(net,self).__init__()
        self.layer = nn.Linear(in_dim,out_dim,bias=False)

    def forward(self,x):
        x = self.layer(x)
        x = self.layer(x)
        return x

input_x = torch.tensor([10.])
label = torch.tensor([5.])
n = net(1,1)
loss_fn = nn.MSELoss()

out = n(input_x)
loss = loss_fn(out,label)
n.zero_grad()
loss.backward()

for param in n.parameters():
    w = param.item()
    g = param.grad

print('Input = %.4f; label = %.4f'%(input_x,label))
print('Weight = %.4f; output = %.4f'%(w,out))
print('Gradient w.r.t. the weight is %.4f'%(g))
print('And it should be %.4f'%(4*(w**2*input_x-label)*w*input_x))

And the output is (may be different on your computer if the initial value of the weight is different):

Input = 10.0000; label = 5.0000
Weight = 0.9472; output = 8.9717
Gradient w.r.t. the weight is 150.4767
And it should be 150.4766

In this example, I have defined a module with only one linear layer (in_dim=out_dim=1 and no bias). w is the weight of this layer; input_x is the input value; label is the desired value. Since the loss is chosen as MSE, the formula for the loss is

((w^2)*input_x-label)^2

Computing by hand, we have

dw/dx = 2*((w^2)*input_x-label)*(2*w*input_x)

The output of my example above shows that autograd gives the same result as computed by hand, giving me a reason to believe that it can work in this case. But in a real application, the layer may have inputs and outputs of higher dimensions, a nonlinear activation function after it, and the neural network could have multiple layers.

What I want to ask is: can I trust autograd to handle such situation, but a lot more complicated than that in my example? How does it work when a layer is called iteratively?


title: What is hp_metric in TensorBoard and how to get rid of it?

body:
I am new to Tensorboard.
I am using fairly simple code running an experiment, and this is the output:

I don't remember asking for a hp_metric graph, yet here it is.
What is it and how do I get rid of it?

Full code to reproduce, using Pytorch Lightning (not that I think anyone should have to reproduce this to answer):
Please notice the ONLY line dereferencing TensorBoard is
self.logger.experiment.add_scalars("losses", {"train_loss": loss}, global_step=self.current_epoch)

import torch
from torch import nn
import torch.nn.functional as F
from typing import List, Optional
from pytorch_lightning.core.lightning import LightningModule
from Testing.Research.toy_datasets.ClustersDataset import ClustersDataset
from torch.utils.data import DataLoader
from Testing.Research.config.ConfigProvider import ConfigProvider
from pytorch_lightning import Trainer, seed_everything
from torch import optim
import os
from pytorch_lightning.loggers import TensorBoardLogger

class VAEFC(LightningModule):
    # see https://towardsdatascience.com/understanding-variational-autoencoders-vaes-f70510919f73
    # for possible upgrades, see https://arxiv.org/pdf/1602.02282.pdf
    # https://stats.stackexchange.com/questions/332179/how-to-weight-kld-loss-vs-reconstruction-loss-in-variational-auto-encoder
    def __init__(self, encoder_layer_sizes: List, decoder_layer_sizes: List, config):
        super(VAEFC, self).__init__()
        self._config = config
        self.logger: Optional[TensorBoardLogger] = None

        assert len(encoder_layer_sizes) >= 3, "must have at least 3 layers (2 hidden)"
        # encoder layers
        self._encoder_layers = nn.ModuleList()
        for i in range(1, len(encoder_layer_sizes) - 1):
            enc_layer = nn.Linear(encoder_layer_sizes[i - 1], encoder_layer_sizes[i])
            self._encoder_layers.append(enc_layer)

        # predict mean and covariance vectors
        self._mean_layer = nn.Linear(encoder_layer_sizes[
                                         len(encoder_layer_sizes) - 2],
                                     encoder_layer_sizes[len(encoder_layer_sizes) - 1])
        self._logvar_layer = nn.Linear(encoder_layer_sizes[
                                           len(encoder_layer_sizes) - 2],
                                       encoder_layer_sizes[len(encoder_layer_sizes) - 1])

        # decoder layers
        self._decoder_layers = nn.ModuleList()
        for i in range(1, len(decoder_layer_sizes)):
            dec_layer = nn.Linear(decoder_layer_sizes[i - 1], decoder_layer_sizes[i])
            self._decoder_layers.append(dec_layer)

        self._recon_function = nn.MSELoss(reduction='mean')

    def _encode(self, x):
        for i in range(len(self._encoder_layers)):
            layer = self._encoder_layers[i]
            x = F.relu(layer(x))

        mean_output = self._mean_layer(x)
        logvar_output = self._logvar_layer(x)
        return mean_output, logvar_output

    def _reparametrize(self, mu, logvar):
        if not self.training:
            return mu
        std = logvar.mul(0.5).exp_()
        if std.is_cuda:
            eps = torch.cuda.FloatTensor(std.size()).normal_()
        else:
            eps = torch.FloatTensor(std.size()).normal_()
        reparameterized = eps.mul(std).add_(mu)
        return reparameterized

    def _decode(self, z):
        for i in range(len(self._decoder_layers) - 1):
            layer = self._decoder_layers[i]
            z = F.relu((layer(z)))

        decoded = self._decoder_layers[len(self._decoder_layers) - 1](z)
        # decoded = F.sigmoid(self._decoder_layers[len(self._decoder_layers)-1](z))
        return decoded

    def _loss_function(self, recon_x, x, mu, logvar, reconstruction_function):
        """
        recon_x: generating images
        x: origin images
        mu: latent mean
        logvar: latent log variance
        """
        binary_cross_entropy = reconstruction_function(recon_x, x)  # mse loss TODO see if mse or cross entropy
        # loss = 0.5 * sum(1 + log(sigma^2) - mu^2 - sigma^2)
        kld_element = mu.pow(2).add_(logvar.exp()).mul_(-1).add_(1).add_(logvar)
        kld = torch.sum(kld_element).mul_(-0.5)
        # KL divergence Kullback–Leibler divergence, regularization term for VAE
        # It is a measure of how different two probability distributions are different from each other.
        # We are trying to force the distributions closer while keeping the reconstruction loss low.
        # see https://towardsdatascience.com/understanding-variational-autoencoders-vaes-f70510919f73

        # read on weighting the regularization term here:
        # https://stats.stackexchange.com/questions/332179/how-to-weight-kld-loss-vs-reconstruction-loss-in-variational
        # -auto-encoder
        return binary_cross_entropy + kld * self._config.regularization_factor

    def training_step(self, batch, batch_index):
        orig_batch, noisy_batch, _ = batch
        noisy_batch = noisy_batch.view(noisy_batch.size(0), -1)

        recon_batch, mu, logvar = self.forward(noisy_batch)

        loss = self._loss_function(
            recon_batch,
            orig_batch, mu, logvar,
            reconstruction_function=self._recon_function
        )
        # self.logger.experiment.add_scalars("losses", {"train_loss": loss})
        self.logger.experiment.add_scalars("losses", {"train_loss": loss}, global_step=self.current_epoch)
        # self.logger.experiment.add_scalar("train_loss", loss, self.current_epoch)
        self.logger.experiment.flush()
        return loss

    def train_dataloader(self):
        default_dataset, train_dataset, test_dataset = ClustersDataset.clusters_dataset_by_config()
        train_dataloader = DataLoader(train_dataset, batch_size=self._config.batch_size, shuffle=True)
        return train_dataloader

    def test_dataloader(self):
        default_dataset, train_dataset, test_dataset = ClustersDataset.clusters_dataset_by_config()
        test_dataloader = DataLoader(test_dataset, batch_size=self._config.batch_size, shuffle=True)
        return test_dataloader

    def configure_optimizers(self):
        optimizer = optim.Adam(model.parameters(), lr=self._config.learning_rate)
        return optimizer

    def forward(self, x):
        mu, logvar = self._encode(x)
        z = self._reparametrize(mu, logvar)
        decoded = self._decode(z)
        return decoded, mu, logvar

if __name__ == "__main__":
    config = ConfigProvider.get_config()
    seed_everything(config.random_seed)
    latent_dim = config.latent_dim
    enc_layer_sizes = config.enc_layer_sizes + [latent_dim]
    dec_layer_sizes = [latent_dim] + config.dec_layer_sizes
    model = VAEFC(config=config, encoder_layer_sizes=enc_layer_sizes, decoder_layer_sizes=dec_layer_sizes)

    logger = TensorBoardLogger(save_dir='tb_logs', name='VAEFC')
    logger.hparams = config  # TODO only put here relevant stuff
    # trainer = Trainer(gpus=1)
    trainer = Trainer(deterministic=config.is_deterministic,
                      #auto_lr_find=config.auto_lr_find,
                      #log_gpu_memory='all',
                      # min_epochs=99999,
                      max_epochs=config.num_epochs,
                      default_root_dir=os.getcwd(),
                      logger=logger
                      )
    # trainer.tune(model)
    trainer.fit(model)
    print("done training vae with lightning")

ClustersDataset.py
from torch.utils.data import Dataset
import matplotlib.pyplot as plt
import torch
import numpy as np
from Testing.Research.config.ConfigProvider import ConfigProvider

class ClustersDataset(Dataset):
    __default_dataset = None
    __default_dataset_train = None
    __default_dataset_test = None

    def __init__(self, cluster_size: int, noise_factor: float = 0, transform=None, n_clusters=2, centers_radius=4.0):
        super(ClustersDataset, self).__init__()
        self._cluster_size = cluster_size
        self._noise_factor = noise_factor
        self._n_clusters = n_clusters
        self._centers_radius = centers_radius
        # self._transform = transform
        self._size = self._cluster_size * self._n_clusters

        self._create_data_clusters()
        self._combine_clusters_to_array()
        self._normalize_data()
        self._add_noise()

        # self._plot()
        pass

    @staticmethod
    def clusters_dataset_by_config():
        if ClustersDataset.__default_dataset is not None:
            return \
                ClustersDataset.__default_dataset, \
                ClustersDataset.__default_dataset_train, \
                ClustersDataset.__default_dataset_test
        config = ConfigProvider.get_config()
        default_dataset = ClustersDataset(
            cluster_size=config.cluster_size,
            noise_factor=config.noise_factor,
            transform=None,
            n_clusters=config.n_clusters,
            centers_radius=config.centers_radius
        )

        train_size = int(config.train_size * len(default_dataset))
        test_size = len(default_dataset) - train_size
        train_dataset, test_dataset = torch.utils.data.random_split(default_dataset, [train_size, test_size])

        ClustersDataset.__default_dataset = default_dataset
        ClustersDataset.__default_dataset_train = train_dataset
        ClustersDataset.__default_dataset_test = test_dataset

        return default_dataset, train_dataset, test_dataset

    def _create_data_clusters(self):
        self._clusters = [torch.zeros((self._cluster_size, 2)) for _ in range(self._n_clusters)]
        centers_radius = self._centers_radius
        for i, c in enumerate(self._clusters):
            r, x, y = 3.0, centers_radius * np.cos(i * np.pi * 2 / self._n_clusters), centers_radius * np.sin(
                i * np.pi * 2 / self._n_clusters)
            cluster_length = 1.1
            cluster_start = i * 2 * np.pi / self._n_clusters
            cluster_end = cluster_length * (i + 1) * 2 * np.pi / self._n_clusters
            cluster_inds = torch.linspace(start=cluster_start, end=cluster_end, steps=self._cluster_size,
                                          dtype=torch.float)
            c[:, 0] = r * torch.sin(cluster_inds) + y
            c[:, 1] = r * torch.cos(cluster_inds) + x

    def _plot(self):
        plt.figure()
        plt.scatter(self._noisy_values[:, 0], self._noisy_values[:, 1], s=1, color='b', label="noisy_values")
        plt.scatter(self._values[:, 0], self._values[:, 1], s=1, color='r', label="values")
        plt.legend(loc="upper left")
        plt.show()

    def _combine_clusters_to_array(self):
        size = self._size
        self._values = torch.zeros(size, 2)
        self._labels = torch.zeros(size, dtype=torch.long)
        for i, c in enumerate(self._clusters):
            self._values[i * self._cluster_size: (i + 1) * self._cluster_size, :] = self._clusters[i]
            self._labels[i * self._cluster_size: (i + 1) * self._cluster_size] = i

    def _add_noise(self):
        size = self._size

        mean = torch.zeros(size, 2)
        std = torch.ones(size, 2)
        noise = torch.normal(mean, std)
        self._noisy_values = torch.zeros(size, 2)
        self._noisy_values[:] = self._values
        self._noisy_values = self._noisy_values + noise * self._noise_factor

    def _normalize_data(self):
        values_min, values_max = torch.min(self._values), torch.max(self._values)
        self._values = (self._values - values_min) / (values_max - values_min)
        self._values = self._values * 2 - 1

    def __len__(self):
        return self._size  # number of samples in the dataset

    def __getitem__(self, index):
        item = self._values[index, :]
        noisy_item = self._noisy_values[index, :]
        # if self._transform is not None:
        #     noisy_item = self._transform(item)
        return item, noisy_item, self._labels[index]

    @property
    def values(self):
        return self._values

    @property
    def noisy_values(self):
        return self._noisy_values

Config values (ConfigProvider just returns those as an object)
num_epochs: 15
batch_size: 128
learning_rate: 0.0001
auto_lr_find: False

noise_factor: 0.1
regularization_factor: 0.0

cluster_size: 5000
n_clusters: 5
centers_radius: 4.0
train_size: 0.8

latent_dim: 8

enc_layer_sizes: [2, 200, 200, 200]
dec_layer_sizes: [200, 200, 200, 2]

retrain_vae: False
random_seed: 11
is_deterministic: True


title: How to free allocated GiB memory in EC2 CUDA GPU instance

body:
I have a p3.2xlarge instance, I ran a couple of experiments on the instance, and now that I want to run a new experiment (deep learning) I get the following error:

RuntimeError: CUDA out of memory. Tried to allocate 64.00 MiB (GPU 0; 15.78 GiB total capacity; 14.70 GiB already allocated; 34.44 MiB free; 14.76 GiB reserved in total by PyTorch)

I wonder if there's any way that I can free the allocated memories so that I can run my experiment? Is freeing the memories even the solution for such an error?


I am attempting to use AlexNet as a feature extractor for my 3 channel image inputs derived from .wav file data. I have the input to the feature extractor of shape (593, 3, 227, 227). However, when using the AlexNet model, I am getting the error
Traceback (most recent call last):
  File "MainUI.py", line 1625, in <module>
    main(False)
  File "MainUI.py", line 1604, in main
    accuracy_measurement(oversample)
  File "MainUI.py", line 1463, in accuracy_measurement
    features = model.extract_features(features.double())
  File "/Users/sruthikurada/opt/anaconda3/lib/python3.7/site-packages/alexnet_pytorch/model.py", line 77, in extract_features
    x = self.features(inputs)
  File "/Users/sruthikurada/opt/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/Users/sruthikurada/opt/anaconda3/lib/python3.7/site-packages/torch/nn/modules/container.py", line 100, in forward
    input = module(input)
  File "/Users/sruthikurada/opt/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/Users/sruthikurada/opt/anaconda3/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 353, in forward
    return self._conv_forward(input, self.weight)
  File "/Users/sruthikurada/opt/anaconda3/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 350, in _conv_forward
    self.padding, self.dilation, self.groups)
RuntimeError: Expected object of scalar type Double but got scalar type Float for argument #3 'mat1' in call to _th_addmm_

My code that results in this error:
features, labels = extract_features(train_files)

print(features.shape) # (593, 3, 227, 227)

import torch
from alexnet_pytorch import AlexNet
model = AlexNet.from_pretrained('alexnet')

features = torch.from_numpy(features).type('torch.DoubleTensor')

features = model.extract_features(features.double()) # <-- This is where the error occurs
print(features.shape)

As you can see, I used the double() command, but that didn't help. Could you kindly provide some assistance?


title: RNN with many-to-one setup - which output to use

body:
I going through a series of machine learning examples that use RNNs for document classification (many-to-one). In most tutorials, the RNN output of the last time step is used, i.e., fed into one or more dense layers to map it to the number of classes (e.g., [1], [2]).

However, I also came across some examples where, instead of the last output, the average of the outputs over all time steps is used (mean pooling?, e.g., [3]). The dimensions of this averaged output are of course the same as for the last output. So computationally, both approaches just work the same.

My questions is now, what is the intuition between the two different approaches. Due to recursive nature, the last output also reflects the output of the previous time steps. So why the idea of averaging the RNN outputs over all time steps. When to use what?


title: How can I randomly shuffle the labels of a Pytorch Dataset?

body:
I am new to Pytorch, and I am having troubles with some technicalities. I have downloaded the MNIST dataset, using the following command:

train_dataset = dsets.MNIST(root='./data',
                            train=True,
                            transform=transforms.ToTensor(),
                            download=True)

I now need to run some experiments on this dataset, but using random labels. How can I shuffle/reassign them randomly?

I am trying to do it manually, but it tells me that " 'tuple' object does not support item assignment". How can I do it then?

Second question: How can I remove a training point from the dataset? It gives me the same error, when I try to do it.

Thank you!!


title: How to manually implement padding for pytorch convolutions

body:
I'm trying to port some pytorch code to tensorflow 2.0 and am having difficulty figuring out how to translate the convolution functions between the two. The way both libraries deal with padding is the sticking point. Basically, I'd like to understand how I can manually produce the padding that pytorch does under the hood so that I can translate that to tensorflow.
The code below works if I don't do any padding, but I can't figure out how to make the two implementations match once any padding is added.
output_padding = SOME NUMBER
padding = SOME OTHER NUMBER
strides = 128

tensor = np.random.rand(2, 258, 249)
filters = np.random.rand(258, 1, 256)

out_torch = F.conv_transpose1d(
    torch.from_numpy(tensor).float(),
    torch.from_numpy(filters).float(),
    stride=strides,
    padding=padding,
    output_padding=output_padding)

def pytorch_transpose_conv1d(inputs, filters, strides, padding, output_padding):
    N, L_in = inputs.shape[0], inputs.shape[2]
    out_channels, kernel_size = filters.shape[1], filters.shape[2]
    time_out = (L_in - 1) * strides - 2 * padding + (kernel_size - 1) + output_padding + 1
    padW = (kernel_size - 1) - padding

    # HOW DO I PAD HERE TO GET THE SAME OUTPUT AS IN PYTORCH
    inputs = tf.pad(inputs, [(?, ?), (?, ?), (?, ?)])

    return tf.nn.conv1d_transpose(
        inputs,
        tf.transpose(filters, perm=(2, 1, 0)),
        output_shape=(N, out_channels, time_out),
        strides=strides,
        padding="VALID",
        data_format="NCW")

out_tf = pytorch_transpose_conv1d(tensor, filters, strides, padding, output_padding)
assert np.allclose(out_tf.numpy(), out_torch.numpy())


title: Multi-variable & Multi-target Regression Problems with deep learning

body:
I would like to design a neural network model that has multiple input variables (4) and multiple output(3). Not sure where to improve.
Possible problems:

Data Loader and Data Normalization :

I found my input data have 4 3~4 digit values, therefore did normalization. I am not sure if my method was correct.
and I also did this normalization on the input values, although output values also have a wide range of numbers but want to correctly predicts what the output value is.

Model Itself:

I am trying to build multiple input and multiple output model
so i am putting 4 input variables and get 3 output variables.
How  should I improve my model?
(Appreciate with your help but, Just Use CNN, RNN, LSTM ... does not really help..)

Loss Function and Optimizer

Currently Using nn.MSELoss() loss function and torch.optim.SGD(model.parameters(), lr=learning_rate) optimizer.
However, that makes NaN, NaN, NaN output (probably some number goes too big?)
and tried nn.L1Loss()
Loss.item value significantly decreased but not sure if this the values that i want.
and I don't see much change (learning process) in the result

Evaluation Metrics

It seems I can't plot the prediction and y because it is 3d vector
and also can't do simple equality comparison because numbers usually don't match in many decimal points.
How should I do evaluation in such circumstances?

Any additional tips or help will be very helpful.
Thank you in advance

#This is a dataloader that I have and did normalization with f.normalize
class Loader(Dataset):

    def __init__(self, data):
      self.data = data
      // without normalization process
      //self.features = torch.tensor(data.iloc[:, 1:4].values)
      // with normalization process
      self.features = f.normalize(torch.tensor(data.iloc[:, 1:5].values), p=2, dim=[-2,1])
      self.targets = torch.tensor(data.iloc[:, 5:8].values)

    def __getitem__(self, index):
      feature = self.features[index]
      target = self.targets[index]
      return feature, target

    def __len__(self):
      return self.data.shape[0]

# model
class MultivariateLinearRegressionModel(nn.Module):
  def __init__(self):
    super().__init__()
    self.linear = nn.Linear(4, 3)

  def forward(self, x):
    return self.linear(x)

# main codes to run training
trainset = Loader(data=training_data)
train_loader = DataLoader(dataset = trainset,
                         batch_size=batch_size,
                         shuffle=False)

model = MultivariateLinearRegressionModel().to(device)
criterion = torch.nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)
total_step = len(test_loader)
loss_list = []
for epoch in range(100):
    for i, (x, y) in enumerate(test_loader):
        x = x.to(device, dtype=torch.float)
        y = y.to(device, dtype=torch.float)
        # Forward pass

        prediction = model(x)
        loss = criterion(prediction, y)

        # Zero gradients, perform a backward pass, and update the weights.
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if (i+1) % 1000 == 0:
            print ('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}'.format(epoch+1, num_epochs, i+1, total_step, loss.item()))
            loss_list.append(loss.item())

Data Format: Before Normalization

tensor([[3671.4275, 3920.7729, 3562.8547, 3409.9354],
        [2613.0593, 4144.3052, 3963.4261, 2520.0331],
        [3217.1497, 4907.6748, 3618.3077, 1625.5708],
        [3495.5350, 3740.1072, 3372.8023, 3222.0030],
        [2668.3124, 4059.1723, 3856.5733, 2555.0729],
        [3148.7100, 3581.6413, 3238.2163, 2892.8446],
        [3259.0295, 4951.7812, 3691.3874, 1729.6103],
        [2383.1404, 4416.9705, 4282.0778, 2368.6338],
        [3233.4030, 3509.1483, 3131.9635, 2950.0310],
        [2796.2451, 4666.8626, 3963.1061, 2038.5555]], device='cuda:0',
       dtype=torch.float64)
tensor([[  86.8043, -125.5781,  514.8793],
        [ 599.4942,  -25.5526,  869.6853],
        [ 767.3758, -575.2572,  932.4783],
        [  85.4733, -129.7880,  653.4727],
        [ 539.4341,  -38.1205,  870.6907],
        [ 150.6393, -117.4500,  877.1568],
        [ 766.7727, -558.0292,  871.3760],
        [ 806.4800,   11.9661,  873.6658],
        [  98.3526, -130.0950,  883.7505],
        [ 780.5635, -253.1192,  876.8691]], device='cuda:0',
       dtype=torch.float64)

Data Format: After Normalization

tensor([[0.0063, 0.0069, 0.0061, 0.0058],
        [0.0067, 0.0072, 0.0065, 0.0062],
        [0.0064, 0.0070, 0.0062, 0.0059],
        [0.0056, 0.0092, 0.0077, 0.0039],
        [0.0051, 0.0081, 0.0078, 0.0049],
        [0.0058, 0.0074, 0.0068, 0.0054],
        [0.0054, 0.0078, 0.0073, 0.0051],
        [0.0063, 0.0097, 0.0073, 0.0034],
        [0.0047, 0.0087, 0.0084, 0.0046],
        [0.0058, 0.0093, 0.0076, 0.0038]], device='cuda:0',
       dtype=torch.float64)
tensor([[  98.2855, -130.0667,  883.9658],
        [  86.6183, -130.0066,  716.1242],
        [  93.5488, -130.8305,  837.4871],
        [ 778.3415, -295.5290,  876.0615],
        [ 598.6805,  -25.7674,  869.6824],
        [ 322.8932,  -82.1197,  873.7214],
        [ 473.1678,  -51.6608,  871.5651],
        [ 768.4887, -535.2723,  871.2855],
        [ 806.6830,    6.9960,  877.1038],
        [ 775.0177, -365.2128,  875.5223]], device='cuda:0',
       dtype=torch.float64)

# Result
Epoch [1/100], Step [1000/5500], Loss: 320.4197
Epoch [1/100], Step [2000/5500], Loss: 262.3183
Epoch [1/100], Step [3000/5500], Loss: 220.1559
Epoch [1/100], Step [4000/5500], Loss: 236.1293
Epoch [1/100], Step [5000/5500], Loss: 34.1634
Epoch [2/100], Step [1000/5500], Loss: 440.7753
Epoch [2/100], Step [2000/5500], Loss: 494.4890
Epoch [2/100], Step [3000/5500], Loss: 224.8119
Epoch [2/100], Step [4000/5500], Loss: 145.8345
Epoch [2/100], Step [5000/5500], Loss: 97.7785


title: Cross Entropy Calculation in PyTorch tutorial

body:
I'm reading the Pytorch tutorial of a multi-class classification problem. And I find the behavior of Loss calculation in Pytorch confuses me a lot. Can you help me with this?

The model used for classification goes like this:

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 5 * 5)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

The training process goes as follows:

optimizer.zero_grad()
outputs = net(inputs)
loss = nn.CrossEntropyLoss(outputs, labels)
loss.backward()
optimizer.step()

My question is: What's the exact behavior of Loss calculation in Pytorch here? During each iteration, the input of nn.CrossEntropyLoss() has two parts:

The output of the model, which is a 10 by 1 tensor, with different values in it. This is a tensor without normalized into probability.
The label as a scalar, like 1 or 2 or 3.

As far as I know, the calculation of cross-entropy usually used between two tensors like:

Target as [0,0,0,1], where 1 is the right class
Output tensor as [0.1,0.2,0.3,0.4], where the sum as 1.

So based on this assumption, nn.CrossEntropyLoss() here needs to achieve:

Firstly normalize the output tensor into possibility one.
Encode the label into one-hot ones, like 2 in 5 class as [0,1,0,0,0]. The length must be the same as output tensor.
Then calculate the loss.

May I ask is this what nn.CrossEntropyLoss() does? Or do we need to one-hot encoding the true label before we input into the model?

Thank you a lot for your time in advance!


title: Failed to compute dot product of torch.cuda.FloatTensor

body:
I used a GPU to compute the dot product of the output of neural networks and a torch.cuda.FloatTensor (both of them are stored in GPU) but got an error saying:

TypeError: dot received an invalid combination of arguments - got (torch.cuda.FloatTensor) but expected (torch.FloatTensor tensor).

the codes are like

p = torch.exp(vector.dot(ht))

here vector is a torch FloatTensor and ht is the output of neural networks.

I've struggled with these things for days but still got no idea. Thanks in advance for any possible solution!


title: Pytorch/CUDA - Cannot find GPU

body:
I'm trying to use my GPU to train a network. I installed PyTorch and CUDA with anaconda. I have an NVIDA GTX 860M on my computer I would like to use.

When using the command :

$ lspci | grep NVIDIA

I get the output :

01:00.0 3D controller: NVIDIA Corporation GM107M [GeForce GTX 860M] (rev a2)

So the graphic card is there. But in a Jupyter notebook, the following lines :

print(torch.cuda.device_count())
print(torch.cuda.is_available())
print(torch.version.cuda)

Return :

0
False
9.0.176

I have tried updating the latest Cuda driver using conda it did not solve the issue. My PyTorch version is 0.4.1.post2. I couldn't find an answer browsing through the different online forums.
I am on Ubuntu 18.04.1 LTS.
Thanks' in advance !


title: Input dimension error on pytorch's forward check

body:
I am creating an RNN with pytorch, it looks like this:

class MyRNN(nn.Module):
    def __init__(self, batch_size, n_inputs, n_neurons, n_outputs):
        super(MyRNN, self).__init__()

        self.n_neurons = n_neurons
        self.batch_size = batch_size
        self.n_inputs = n_inputs
        self.n_outputs = n_outputs

        self.basic_rnn = nn.RNN(self.n_inputs, self.n_neurons)

        self.FC = nn.Linear(self.n_neurons, self.n_outputs)

    def init_hidden(self, ):
        # (num_layers, batch_size, n_neurons)
        return torch.zeros(1, self.batch_size, self.n_neurons)

    def forward(self, X):
        self.batch_size = X.size(0)
        self.hidden = self.init_hidden()

        lstm_out, self.hidden = self.basic_rnn(X, self.hidden)
        out = self.FC(self.hidden)

        return out.view(-1, self.n_outputs)

My input x looks like this:

tensor([[-1.0173e-04, -1.5003e-04, -1.0218e-04, -7.4541e-05, -2.2869e-05,
         -7.7171e-02, -4.4630e-03, -5.0750e-05, -1.7911e-04, -2.8082e-04,
         -9.2992e-06, -1.5608e-05, -3.5471e-05, -4.9127e-05, -3.2883e-01],
        [-1.1193e-04, -1.6928e-04, -1.0218e-04, -7.4541e-05, -2.2869e-05,
         -7.7171e-02, -4.4630e-03, -5.0750e-05, -1.7911e-04, -2.8082e-04,
         -9.2992e-06, -1.5608e-05, -3.5471e-05, -4.9127e-05, -3.2883e-01],
        ...

        [-6.9490e-05, -8.9197e-05, -1.0218e-04, -7.4541e-05, -2.2869e-05,
         -7.7171e-02, -4.4630e-03, -5.0750e-05, -1.7911e-04, -2.8082e-04,
         -9.2992e-06, -1.5608e-05, -3.5471e-05, -4.9127e-05, -3.2883e-01]],
       dtype=torch.float64)

and is a batch of 64 vectors with size 15.

When trying to test this model by doing:

BATCH_SIZE = 64
N_INPUTS = 15
N_NEURONS = 150
N_OUTPUTS = 1
model = MyRNN(BATCH_SIZE, N_INPUTS, N_NEURONS, N_OUTPUTS)
model(x)

I get the following error:

File "/home/tt/anaconda3/envs/venv/lib/python3.6/site-packages/torch/nn/modules/rnn.py", line 126, in check_forward_args
    expected_input_dim, input.dim()))
RuntimeError: input must have 3 dimensions, got 2

How can I fix it?


title: Indexing on axis by list in PyTorch

body:
I have Variables lengths_X of size (10L,) and A of size (10L, 16L, 5L).

I want to use lengths_X to index along the second axis of A. In other words, I want to get a new tensor predicted_Y of size (10L, 5L) that indexes axis 1 at i for all entries with index i in axis 0.

What is the best way to do this in PyTorch?


title: How to combine multiple h5 files?

body: Limited by the device, I could only produce several h5 files (the format of each file are same with shape of [idx, 1, 224, 224]) for huge dataset (>100GB) and now I'm confused about the solution to combine these files into a single one for further training on PyTorch. enter image description here


I am trying to use 3d conv on cifar10 data set (just for fun). I see the docs that we usually have the input be 5d tensors (N,C,D,H,W). Am I really forced to pass 5 dimensional data necessarily?

The reason I am skeptical is because 3D convolutions simply mean my conv moves across 3 dimensions/directions. So technically I could have 3d 4d 5d or even 100d tensors and then should all work as long as its at least a 3d tensor. Is that not right?

I tried it real quick and it did give an error:

import torch

def conv3d_example():
    N,C,H,W = 1,3,7,7
    img = torch.randn(N,C,H,W)
    ##
    in_channels, out_channels = 1, 4
    kernel_size = (2,3,3)
    conv = torch.nn.Conv3d(in_channels, out_channels, kernel_size)
    ##
    out = conv(img)
    print(out)
    print(out.size())

##
conv3d_example()
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-3-29c73923cc64> in <module>
     15
     16 ##
---> 17 conv3d_example()

<ipython-input-3-29c73923cc64> in conv3d_example()
     10     conv = torch.nn.Conv3d(in_channels, out_channels, kernel_size)
     11     ##
---> 12     out = conv(img)
     13     print(out)
     14     print(out.size())

~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    491             result = self._slow_forward(*input, **kwargs)
    492         else:
--> 493             result = self.forward(*input, **kwargs)
    494         for hook in self._forward_hooks.values():
    495             hook_result = hook(self, input, result)

~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/conv.py in forward(self, input)
    474                             self.dilation, self.groups)
    475         return F.conv3d(input, self.weight, self.bias, self.stride,
--> 476                         self.padding, self.dilation, self.groups)
    477
    478

RuntimeError: Expected 5-dimensional input for 5-dimensional weight 4 1 2 3, but got 4-dimensional input of size [1, 3, 7, 7] instead

cross posted:

https://discuss.pytorch.org/t/how-does-one-use-3d-convolutions-on-standard-3-channel-images/53330
How does one use 3D convolutions on standard 3 channel images?


title: Pytorch sending inputs/targets to device

body:
Currently using the pycocotools 2.0 library.

My train_loader is:

train_loader, test_loader = get_train_test_loader('dataset', batch_size=16, num_workers=4)

However the training lines of code:

        for i, data in enumerate(train_loader, 0):
            images, targets = data
            images = images.to(device)
            targets = targets.to(device)

Results in error. Variables data, images, and targets are all of class tuple

Traceback (most recent call last):
  File "train.py", line 40, in <module>
    images = images.to(device)
AttributeError: 'tuple' object has no attribute 'to'

How can I properly send these to cuda device?'

Edit:

I can send images[0].to(device) no problem. How can I send the rest?


title: Issue with running a single prediction with PyTorch

body:
I have a trained model using PyTorch now I want to simpy run it on one example

>>> model
nn.Sequential {
  [input -> (0) -> (1) -> (2) -> (3) -> (4) -> (5) -> (6) -> (7) -> (8) -> (9) -> (10) -> output]
  (0): nn.SpatialConvolutionMap
  (1): nn.Tanh
  (2): nn.SpatialMaxPooling(2x2, 2, 2)
  (3): nn.SpatialConvolutionMap
  (4): nn.Tanh
  (5): nn.SpatialMaxPooling(2x2, 2, 2)
  (6): nn.Reshape(6400)
  (7): nn.Linear(6400 -> 128)
  (8): nn.Tanh
  (9): nn.Linear(128 -> 5)
  (10): nn.LogSoftMax
}

Then I load an image from my test set:

image = cv2.imread('image.png',cv2.IMREAD_GRAYSCALE)
transformation = transforms.Compose([transforms.ToTensor()])
image_tensor = transformation(image).float()
inp = Variable(image_tensor)

and finally try to run the network

output = model(inp)

But I get error TypeError: 'Sequential' object is not callable


title: PyTorch 3 reshaping error

body:
When training a CNN using PyTorch in Python, I get the following error:

RuntimeError: invalid argument 2: size '[-3 x 3136]' is invalid for input with 160000 elements at /opt/conda/conda-bld/pytorch-cpu_1515613813020/work/torch/lib/TH/THStorage.c:41

This is related to the x.view line in the model below:

class Net(nn.Module):

    def __init__(self):
        super(Net,self).__init__()
        self.conv1  = nn.Conv2d(3,32,5,padding=2) # 1 input, 32 out, filter size = 5x5, 2 block outer padding
        self.conv2  = nn.Conv2d(32,64,5,padding=2) # 32 input, 64 out,  filter size = 5x5, 2 block padding
        self.fc1    = nn.Linear(64*7*7,1024) # Fully connected layer
        self.fc2    = nn.Linear(1024,2) #Fully connected layer 2 out.

    def forward(self,x):
        x = F.max_pool2d(F.relu(self.conv1(x)), 2) # Max pool over convolution with 2x2 pooling
        x = F.max_pool2d(F.relu(self.conv2(x)), 2) # Max pool over convolution with 2x2 pooling
        x = x.view(-1,64*7*7) # tensor.view() reshapes the tensor
        x = F.relu(self.fc1(x)) # Activation function after passing through fully connected layer
        x = F.dropout(x, training=True) #Dropout regularisation
        x = self.fc2(x) # Pass through final fully connected layer
        return F.log_softmax(x) # Give results using softmax

model = Net()
print(model)

I'm not sure if this is a result of the images having 3 channels or something else entirely. I understand that this command should reshape the images into single dimensional arrays ready for the fully connected layer so I'm not sure how to fix this issue when the error is claiming an input of 160000 elements.


title: ImportError:numpy.core.multiarray failed to import when use pytorch

body:
When I import torch in cmd window I get the error :

ImportError:numpy.core.multiarray failed to import' after using pyinstaller.

I tried numpy 1.18.4, 1.18.3 and 1.16.3. but it still doesn't work.

By the way, I use Windows 10, anaconda, Python 3.6 and torch 1.5.0.

Thank you for your advice.

File ''site-packages\torch\__init__.py'',line 136 in <module>
ImportError:numpy.core.multiarray failed to import


title: 1x1 convolution as classification layer in Pytorch

body:
I am trying to classify image patches into 10 different categories using a neural network. My idea (borrowed from this article is to use the first 5 layers of a pretrained VGG network and apply a 1x1 convolution to this encoder. So, given the first 5 layers of a VGG network:

Sequential(
    (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (1): ReLU(inplace=True)
    (2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (3): ReLU(inplace=True)
    (4): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (5): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (6): ReLU(inplace=True)
    (7): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (8): ReLU(inplace=True)
    (9): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (10): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (11): ReLU(inplace=True)
    (12): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (13): ReLU(inplace=True)
    (14): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (15): ReLU(inplace=True)
    (16): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (17): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (18): ReLU(inplace=True)
    (19): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (20): ReLU(inplace=True)
    (21): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (22): ReLU(inplace=True)
    (23): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (24): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (25): ReLU(inplace=True)
    (26): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (27): ReLU(inplace=True)
    (28): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (29): ReLU(inplace=True)
    (30): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
)

How can I add a layer to classify each activation into n_classes? I am trying with a 1x1 convolution from 512 (number of filters in last layer) to n_classes, something like this:

nn.Sequential(
    nn.Conv2d(in_channels=512, out_channels=n_classes, kernel_size=1),
    nn.Softmax(dim=1)
    )

This network gives (batch_size, n_classes H // 32, W // 32) outputs. However, all outputs for each class are equal for a given image, I mean that, for a given batch, B and class C, all values are the same.

I have tried training with CrossEntropyLoss and even converting to logits and applying BCELoss, but without success. I also have tried other networks in my train loop and this do not happens.

Any clue?

Thanks


title: Huggingface pre trained bert model is not working

body:
I have pre-trained a bert model with custom corpus then got vocab file, checkpoints, model.bin, tfrecords, etc.
Then I loaded the model as below :
# Load pre-trained model (weights)
model = BertModel.from_pretrained('/content/drive/My Drive/Anirban_test_pytorch')

But when I am trying to use the model for any task (like q and a, prediction of mask word etc.) then getti9ng below error
from transformers import pipeline
nlp = pipeline("fill-mask", model="model")
nlp(f"This is the best thing I've {nlp.tokenizer.mask_token} in my life.")

ERROR:
OSError: Can't load config for 'model'. Make sure that:

'model' is a correct model identifier listed on 'https://huggingface.co/models'

or 'model' is the correct path to a directory containing a config.json file

Can you please help me?


title: Pytorch:1.2.0 - AttributeError: 'Conv2d' object has no attribute 'weight'

body:
I am trying to initialise the following weights the following way:

def _initialize_weights(self):
    for m in self.modules():
        if isinstance(m, nn.Conv2d):
            nn.init.kaiming_normal_(m.weight)
            if m.bias is not None:
                nn.init.constant_(m.bias, 0)
        elif isinstance(m, nn.BatchNorm2d):
            nn.init.constant_(m.weight, 1)
            nn.init.constant_(m.bias, 0)

This gives me the Attribute error. nn.Conv2d definitely has an attribute named 'weight', Not sure why I am getting this error.


title: Why tensorflow GPU memory usage decreasing when I increasing the batch size?

body:
Recently I implemented a VGG-16 network using both Tensorflow and PyTorch, data set is CIFAR-10. Each picture is 32 * 32 RGB.

I use a 64 batch size in beginning, while I found PyTorch using much less GPU memory than tensorflow. Then I did some experiments and got a figure, which is posted below.

After some researching, I known the tensorflow using BFC algorithm to manage memory. So it's can explain why tensorflow's memory using decreasing or increasing by 2048, 1024, ... MB and sometimes the memory use not increasing when batch size is bigger.

But I am still confused, why the memory use is lower when batch size is 512 than batch size is 384, 448 etc. which has a smaller batch size. The same as when batch size is from 1024 to 1408, and batch size is 2048 to 2688.

Here is my source code:

PyTorch:https://github.com/liupeng3425/tesorflow-vgg/blob/master/vgg-16-pytorch.py

Tensorflow:https://github.com/liupeng3425/tesorflow-vgg/blob/master/vgg-16.py

edit:
I have two Titan XP on my computer, OS: Linux Mint 18.2 64-bit.

I determine GPU memory usage with command nvidia-smi.

My code runs on GPU1, which is defined in my code:

os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
os.environ["CUDA_VISIBLE_DEVICES"] = "1"

And I am sure there only one application using GPU1.

GPU memory usage can be determined by the application list below.
For example, like the posted screen shot below, process name is /usr/bin/python3 and its GPU memory usage is 1563 MiB.


title: Getting Cuda Out of Memory while running Longformer Model in Google Colab. Similar code using Bert is working fine

body:
I am working on text classification using Longformer Model. I took even just first 100 rows of dataframe. I am getting memory error. I am using google colab.
This is my model :
model = LongformerForMultiSequenceClassification.from_pretrained('allenai/longformer-base-4096',
                                        config=config)
# Accessing the model configuration
configuration = model.config

Training Loop

for epoch in tqdm(range(1, epochs+1)):

    model.train()

    loss_train_total = 0

    progress_bar = tqdm(dataloader_train, desc='Epoch {:1d}'.format(epoch), leave=False, disable=False)
    for batch in progress_bar:

        #this will empty the gradients from the previous iterations
        model.zero_grad()

        #take out inputs
        batch = tuple(b.to(device) for b in batch)

        inputs = {'input_ids':      batch[0],
                  'attention_mask': batch[1],
                  'labels':         batch[2],
                 }
        #insert the input into the model and get the result
        outputs = model(**inputs)

        #calculate loss
        loss = outputs[0]
        loss_train_total += loss.item()

        #this will calculate the gradients
        loss.backward()
        # for preventening gradient explosion
        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
        #this will update the weights
        optimizer.step()
        #optimizing learning rate
        scheduler.step()

        progress_bar.set_postfix({'training_loss': '{:.3f}'.format(loss.item()/len(batch))})

    torch.save(model.state_dict(), f'/content/Gdrive/My Drive/finetuned_longformer_epoch_{epoch}.model')
    #torch.save(model.state_dict(), f'checkpoint{epoch}.pth')

    tqdm.write(f'\nEpoch {epoch}')

    loss_train_avg = loss_train_total/len(dataloader_train)
    tqdm.write(f'Training loss: {loss_train_avg}')

    val_loss, predictions, true_vals = evaluate(dataloader_validation)
    val_f1 = f1_score_func(predictions, true_vals)
    tqdm.write(f'Validation loss: {val_loss}')
    tqdm.write(f'F1 Score (Weighted): {val_f1}')

Error :
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-32-7e534d564c0a> in <module>()
     20               }
     21      #insert the input into the model and get the result
---> 22      outputs = model(**inputs)
     23
     24      #calculate loss

12 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in dropout(input, p, training, inplace)
    971     return (_VF.dropout_(input, p, training)
    972             if inplace
--> 973             else _VF.dropout(input, p, training))
    974
    975

RuntimeError: CUDA out of memory. Tried to allocate 182.00 MiB (GPU 0; 11.17 GiB total capacity; 10.23 GiB already allocated; 59.81 MiB free; 10.69 GiB reserved in total by PyTorch)

You can check my config file and model structure and custom class for Global Attention or My complete code is on colab is here :
https://colab.research.google.com/drive/19JkCht_4u6UrwcUcWNnSD2YtnsJYer0H?usp=sharing
I ran a similar code using BERT and it works without any problem.
 I am new to datascience. Please help.


title: Unable to install PyTorch on Windows 10 (x86_64) with Cuda 11.0 using pip

body:
I tried following the instructions on pytorch.org and ran the command provided by them for my configuration, but I get the following error
ERROR: Could not find a version that satisfies the requirement torch===1.7.0+cu110 (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)
ERROR: No matching distribution found for torch===1.7.0+cu110
A  similar error is being thrown for any other installation methods that I try, e.g. earlier cuda versions, CPU only etc. My Python version is 3.9 and was installed using the .exe installer provided at python.org, if that's relevant.


title: What is the desired behavior of average pooling with padding?

body:
Recently I've trained a neural network using pytorch and there is an average pooling layer with padding in it. And I'm confused about the behavior of it as well as the definition of average pooling with padding.

For example, if we have a input tensor:

[[1, 2, 3],
 [4, 5, 6],
 [7, 8, 9]]

When padding is one and kernel size 3, the input to the first kernel should be:

 0, 0, 0
 0, 1, 2
 0, 4, 5

The output from the pytorch is 12/4 = 3 (ignoring padded 0)， but I think it should be 12/9 = 1.333

Can anyone explain this to me?

Much appreciated.


title: How to encode categorical data that have variable length so could be fetched to nn.Embedding in PyTorch

body:
Let's say i have a data field named movie_genre for each sample movie, it is selected from the following genres:

Action
Adventure
Animation
Comedy
...

And for each movie, it might contain multiple genres:

mid    genres
1      Action | Adventure
2      Animation
3      Comedy | Adventure | Action

which means, the movie's genres is a variable list.

If i use one hot vector to encode the genre, Action can be encoded as (1, 0, 0, 0), Adventure can be encoded as(0, 1, 0, 0), and so on.

So movie with mid1 can be encoded as (1, 1, 0, 0), mid2's genre can be encoded as (0, 0, 1, 0), and so on.

However, the pytorch embedding layer nn.Embedding takes tensor containing the indices as input, but not one-hot vector. So how should i encode the data so that it can be fetched into the embedding layer?


title: RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! when resuming training

body:
I saved a checkpoint while trainig on gpu.
after reloading the checkpoint and continue training i get the following error.
Traceback (most recent call last):
  File "main.py", line 140, in <module>
    train(model,optimizer,train_loader,val_loader,criteria=args.criterion,epoch=epoch,batch=batch)
  File "main.py", line 71, in train
    optimizer.step()
  File "/opt/conda/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 26, in decorate_context
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/optim/sgd.py", line 106, in step
    buf.mul_(momentum).add_(d_p, alpha=1 - dampening)
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

my training code is:
def train(model,optimizer,train_loader,val_loader,criteria,epoch=0,batch=0):
    batch_count = batch
    if criteria == 'l1':
        criterion = L1_imp_Loss()
    elif criteria == 'l2':
        criterion = L2_imp_Loss()
    if args.gpu and torch.cuda.is_available():
        model.cuda()
        criterion = criterion.cuda()

    print(f'{datetime.datetime.now().time().replace(microsecond=0)} Starting to train..')

    while epoch <= args.epochs-1:
        print(f'********{datetime.datetime.now().time().replace(microsecond=0)} Epoch#: {epoch+1} / {args.epochs}')
        model.train()
        interval_loss, total_loss= 0,0
        for i , (input,target) in enumerate(train_loader):
            batch_count += 1
            if args.gpu and torch.cuda.is_available():
                input, target = input.cuda(), target.cuda()
            input, target = input.float(), target.float()
            pred = model(input)
            loss = criterion(pred,target)
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
            ....

the saving proccess happend after finishing each epoch.
torch.save({'epoch': epoch,'batch':batch_count,'model_state_dict': model.state_dict(),'optimizer_state_dict':
                    optimizer.state_dict(),'loss': total_loss/len(train_loader),'train_set':args.train_set,'val_set':args.val_set,'args':args}, f'{args.weights_dir}/FastDepth_Final.pth')

I cant figure why i get this error.
args.gpu == True , and Im passing the model, all data, and loss function to cuda, somehow there is still a tensor on cpu, could anyone figure out whats wrong?
Thanks.


title: pytorch: dyld: Symbol not found

body:
I've installed pytorch and fastai via conda:

conda list
...
fastai                    1.0.28                     py_1    fastai
pytorch                   1.0.0                   py3.6_1    pytorch
torchtext                 0.3.1                     <pip>
torchvision               0.2.1                      py_2    pytorch

I'm using one of the fastai models.
The code to load the model is this ( the very last line is the one that fails):

import numpy as np
import torch
from fastai import untar_data, URLs
import pickle
from fastai.text import get_language_model
from torchtext import data

# puzzling the pieces together
# get weights and itos
model_path = untar_data(URLs.WT103, data=False)
fnames = [list(model_path.glob(f'*.{ext}'))[0] for ext in ['pth', 'pkl']]
wgts_fname, itos_fname = fnames
itos = pickle.load(open(itos_fname, 'rb'))
wgts = torch.load(wgts_fname, map_location=lambda storage, loc: storage)

It produces the error:

dyld: Symbol not found: _PySlice_Unpack
  Referenced from: /anaconda3/envs/t1/lib/python3.6/site-packages/torch/lib/libtorch_python.dylib
  Expected in: flat namespace

After browsing SO, I found this related question:
dyld: Symbol not found: error how to resolve this issue

Apparently an error about dyld symbols is related to missing dependencies/broken paths to binaries.

In this case it looks like a pytorch internal problem to me.
How can something like this happen when using a package manager like conda?
My operating system is MacOS 10.14.2


title: Convert np array of arrays to torch tensor when inner arrays are of different sizes

body:
I have several videos, which I have loaded frame by frame into a numpy array of arrays. For example if I have 8 videos, they are converted into an 8 dimensional numpy array of arrays where each inner array has a different dimension depending on the number of frames of the individual video. When I print

array.shape

my output is (8,)

Now I would like to create a dataloader for this data, and for that I would like to convert this numpy array into a torch tensor. However when I try to convert it using the torch.from_numpy or even simply the torch.tensor functions I get the error

TypeError: can't convert np.ndarray of type numpy.object_. The only supported types are: float64, float32, float16, int64, int32, int16, int8, uint8, and bool.

which I assume is because my inner arrays are of different sizes. One possible solution is to artificially add a dimension to my videos to make them be of the same size and then use np.stack but that may lead to possible problems later on. Is there any better solution?

Edit: Actually adding a dimension won't work because np.stack requires all dimensions to be the same.

Edit: Sample Array would be something like:

[ [1,2,3], [1,2], [1,2,3,4] ]

This is stored as a (3,) shaped np array. The real arrays are actually 4-dimensional( Frames x Height x Width x Channels), so this is just an example.


title: Reverse Image search (for image duplicates) on local computer

body:
I have a bunch of poor quality photos that I extracted from a pdf. Somebody I know has the good quality photo's somewhere on her computer(Mac), but it's my understanding that it will be difficult to find them.
I would like to

loop through each poor quality photo
perform a reverse image search using each poor quality photo as the query image and using this persons computer as the database to search for the higher quality images
and create a copy of each high quality image in one destination folder.

Example pseudocode
for each image in poorQualityImages:
    search ./macComputer for a higherQualityImage of image
    copy higherQualityImage to ./higherQualityImages

I need to perform this action once.
I am looking for a tool, github repo or library which can perform this functionality more so than a deep understanding of content based image retrieval.

There's a post on reddit where someone was trying to do something similar
imgdupes is a program which seems like it almost achieves this, but I do not want to delete the duplicates, I want to copy the highest quality duplicate to a destination folder

Update
Emailed my previous image processing prof and he sent me this

Off the top of my head, nothing out of the box.
No guaranteed solution here, but you can narrow the search space.
You’d need a little program that outputs the MSE or SSIM similarity
index between two images, and then write another program or shell
script that scans the hard drive and computes the MSE between each
image on the hard drive and each query image, then check the images
with the top X percent similarity score.
Something like that.  Still not maybe guaranteed to find everything
you want.  And if the low quality images are of different pixel
dimensions than the high quality images, you’d have to do some image
scaling to get the similarity index.   If the poor quality images have
different aspect ratios, that’s even worse.
So I think it’s not hard but not trivial either.  The degree of
difficulty is partly dependent on the nature of the corruption in the
low quality images.

UPDATE
Github project I wrote which achieves what I want


title: Understanding PyTorch training batches

body:
Reading https://stanford.edu/~shervine/blog/pytorch-how-to-generate-data-parallel & https://discuss.pytorch.org/t/how-does-enumerate-trainloader-0-work/14410 I'm trying to understand how training epochs behave in PyTorch.

Take this outer and inner loop :

for epoch in range(num_epochs):
    for i1,i2 in enumerate(training_loader):

Is this a correct interpretation : ?

For each invocation of the outer loop/epoch the entire training set, in above example training_loader is iterated per batch. This means the model does not process one instance per training cycle. Per training cycle ( for epoch in range(num_epochs): ) the entire training set is processed in chunks/batches where the batch size is determined when creating training_loader


title: Trying to understand what "save_for_backward" is in Pytorch

body:
I have some knowledge in Pytorch,but i don't really understand the mechanisms of classes within Pytorch.
For example in the link: https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html
you can find the following code:
import torch

class MyReLU(torch.autograd.Function):

    @staticmethod
    def forward(ctx, input):
        ctx.save_for_backward(input)
        return input.clamp(min=0)

    @staticmethod
    def backward(ctx, grad_output):
        input, = ctx.saved_tensors
        grad_input = grad_output.clone()
        grad_input[input < 0] = 0
        return grad_input

i am only focusing on the forward method of this class,and i am wondering what
ctx.save_for_backward(input)

does.Whether the previous line of code is present or not is irrelevent when i try the forward method on a concrete example:
a=torch.eye(3)
rel=MyReLU()
print(rel.forward(rel,a))

as i get the same result in both cases.Could someone explain me what is happening and why it is useful to add the save_for_backward?
Thank you in advance.


title: Creating ".t7" torch file in python

body:
By using pytorch, I can manage to save file like the following:

torch.save(object, "object.t7")

However, I cannot open it through Lua.

When I use the following command in Lua,

local info1 = torch.load("object.t7")

I receive the following error:

  /home/hasan/build/torch/install/bin/luajit:
  ...an/build/torch/install/share/lua/5.1/threads/threads.lua:315:
  ...an/build/torch/install/share/lua/5.1/threads/threads.lua:183:
  [thread 4 callback]
  ...e/hasan/build/torch/install/share/lua/5.1/torch/File.lua:375:
  unknown object stack traceback:   [C]: in function 'error'
    ...e/hasan/build/torch/install/share/lua/5.1/torch/File.lua:375: in
  function 'readObject'
    ...e/hasan/build/torch/install/share/lua/5.1/torch/File.lua:409: in
  function 'load'
    /home/hasan/GAN/icml2016/data/donkey_folder_txt_coco.lua:112: in
  function
    [C]: in function 'xpcall'
    ...an/build/torch/install/share/lua/5.1/threads/threads.lua:234: in
  function 'callback'
    ...asan/build/torch/install/share/lua/5.1/threads/queue.lua:65: in
  function
  <...asan/build/torch/install/share/lua/5.1/threads/queue.lua:41>  [C]:
  in function 'pcall'
    ...asan/build/torch/install/share/lua/5.1/threads/queue.lua:40: in
  function 'dojob'  [string "  local Queue = require
  'threads.queue'..."]:15: in main chunk

I need to run a torch training in Lua, with preprocessed data through Pytorch.

Is there any way to save ".t7" file via pytorch, and open it through Lua?

Thanks,


title: How to get the imagenet dataset on which pytorch models are trained on

body:
Can anyone please tell me how to download the complete imagenet dataset on which the pytorch torchvision models are trained on and their Top-1 error is reported on?

I have downloaded Tiny-Imagenet from Imagenet website and used pretrained resnet-101 model which provides only 18% Top-1 accuracy.


title: How to convert yolov4 weights.wt to pytorch weights .pt?

body:
I was trying out the yolov4 from https://github.com/theAIGuysCode/YOLOv4-Cloud-Tutorial and I wanted to convert the weights from .wt files to .pt files for pytorch
Is there a way I can do that?


title: Why does torch.from_numpy require a different byte ordering while matplotlib doesn't?

body:
Here is a piece of code (running on Linux CentOS 7.7.1908, x86_64)

import torch    #v1.3.0
import numpy as np  #v1.14.3
import matplotlib.pyplot as plt
from astropy.io.fits import getdata   #v3.0.2
data, hdr = getdata("afile.fits", 0, header=True) #gives dtype=float32 2d array
plt.imshow(data)
plt.show()

This gives a nice 512x512 image

Now, I would like to convert "data" into a PyTorch tensor:

a = torch.from_numpy(data)

Although, PyTorch raises:

  ValueError: given numpy array has byte order different from the native
  byte order. Conversion between byte orders is currently not supported.

Well, I have tried different manipulations with no success: ie. byteswap(), copy()

An idea?

PS: the same error occurs when I transfer my data to Mac OSX (Mojave) while still matplotlib is ok.


title: Problems with PyTorch MLP when training the MNIST dataset retrieved from Keras

body:
I have finished a PyTorch MLP model for the MNIST dataset, but got two different results: 0.90+ accuracy when using MNIST dataset from PyTorch, but ~0.10 accuracy when using MNIST dataset from Keras.
Below is my code with dependency: PyTorch 0.3.0.post4, keras 2.1.3, tensorflow backend 1.4.1 gpu version.

from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import numpy as np
import torch as pt
import torchvision as ptv
from keras.datasets import mnist
from torch.nn import functional as F
from torch.utils.data import Dataset, DataLoader

# training data from PyTorch
train_set = ptv.datasets.MNIST("./data/mnist/train", train=True, transform=ptv.transforms.ToTensor(), download=True)
test_set = ptv.datasets.MNIST("./data/mnist/test", train=False, transform=ptv.transforms.ToTensor(), download=True)
train_dataset = DataLoader(train_set, batch_size=100, shuffle=True)
test_dataset = DataLoader(test_set, batch_size=10000, shuffle=True)

class MLP(pt.nn.Module):
    """The Multi-layer perceptron"""
    def __init__(self):
        super(MLP, self).__init__()
        self.fc1 = pt.nn.Linear(784, 512)
        self.fc2 = pt.nn.Linear(512, 128)
        self.fc3 = pt.nn.Linear(128, 10)
        self.use_gpu = True

    def forward(self, din):
        din = din.view(-1, 28 * 28)
        dout = F.relu(self.fc1(din))
        dout = F.relu(self.fc2(dout))
        # return F.softmax(self.fc3(dout))
        return self.fc3(dout)

model = MLP().cuda()
print(model)

# loss func and optim
optimizer = pt.optim.SGD(model.parameters(), lr=1)
criterion = pt.nn.CrossEntropyLoss().cuda()

def evaluate_acc(pred, label):
    pred = pred.cpu().data.numpy()
    label = label.cpu().data.numpy()
    test_np = (np.argmax(pred, 1) == label)
    test_np = np.float32(test_np)
    return np.mean(test_np)

def evaluate_loader(loader):
    print("evaluating ...")
    accurarcy_list = []
    for i, (inputs, labels) in enumerate(loader):
        inputs = pt.autograd.Variable(inputs).cuda()
        labels = pt.autograd.Variable(labels).cuda()
        outputs = model(inputs)
        accurarcy_list.append(evaluate_acc(outputs, labels))
    print(sum(accurarcy_list) / len(accurarcy_list))

def training(d, epochs):
    for x in range(epochs):
        for i, data in enumerate(d):

            optimizer.zero_grad()

            (inputs, labels) = data
            inputs = pt.autograd.Variable(inputs).cuda()
            labels = pt.autograd.Variable(labels).cuda()

            outputs = model(inputs)

            loss = criterion(outputs, labels)
            loss.backward()

            optimizer.step()

            if i % 200 == 0:
                print(i, ":", evaluate_acc(outputs, labels))

# Training MLP for 4 epochs with MNIST dataset from PyTorch
training(train_dataset, 4)

# The accuracy is ~0.96.
evaluate_loader(test_dataset)

print("###########################################################")

def load_mnist():
    (x, y), (x_test, y_test) = mnist.load_data()
    x = x.reshape((-1, 1, 28, 28)).astype(np.float32)
    x_test = x_test.reshape((-1, 1, 28, 28)).astype(np.float32)
    y = y.astype(np.int64)
    y_test = y_test.astype(np.int64)
    print("x.shape", x.shape, "y.shape", y.shape,
          "\nx_test.shape", x_test.shape, "y_test.shape", y_test.shape,
          )

    return x, y, x_test, y_test

class TMPDataset(Dataset):
    """Dateset for loading Keras MNIST dataset."""
    def __init__(self, a, b):
        self.x = a
        self.y = b

    def __getitem__(self, item):
        return self.x[item], self.y[item]

    def __len__(self):
        return len(self.y)

x_train, y_train, x_test, y_test = load_mnist()
# Create dataloader for MNIST dataset from Keras.
test_loader = DataLoader(TMPDataset(x_test, y_test), num_workers=1, batch_size=10000)
train_loader = DataLoader(TMPDataset(x_train, y_train), shuffle=True, batch_size=100)

# Evaluate the performance of MLP trained on PyTorch dataset and the accurach is ~0.96.
evaluate_loader(test_loader)
evaluate_loader(train_loader)

model = MLP().cuda()
print(model)

optimizer = pt.optim.SGD(model.parameters(), lr=1)
criterion = pt.nn.CrossEntropyLoss().cuda()

# Train now on MNIST dataset from Keras.
training(train_loader, 4)

# Evaluate the trianed model on MNIST dataset from Keras and result in performance ~0.10...
evaluate_loader(test_loader)
evaluate_loader(train_loader)

I had checked some samples from Keras MNIST dataset and found no error.
I am wondering what is wrong with the datasets?
The code can run without error, run it to see the results.


title: Cannot import opencv normally in Pycharm with conda env

body:
I want to use Pytorch and OpenCV in Pycharm. So I created a virtual conda env and installed some packages to use.

When I execute Python in the Windows terminal and import torch and cv2, it worked well.

(Pytorch) C:\Users\choib>python
Python 3.7.1 | packaged by conda-forge | (default, Nov 13 2018, 19:01:41) [MSC v.1900 64 bit (AMD64)] :: Anaconda, Inc. on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
>>> import torch
>>>

It does not work normally with Pycharm. If I import cv2 before torch or torchvision or without them, it raises an error.

Example

import cv2 as cv
import torch

Result

D:\program\anaconda3\envs\pytorch\python.exe C:/Users/choib/Desktop/YOLO_01/YOLO_image.py
Traceback (most recent call last):
  File "C:/Users/choib/Desktop/YOLO_01/YOLO_image.py", line 1, in <module>
    import cv2 as cv
ImportError: DLL load failed: The specified module could not be found.

But if I import cv2 after torch or torchvision, it has no problem.

As a result, I don't have any problem to execute my code. But I wonder why I need to import torch or torchvision unnecessarily to avoid this ImportError.


title: PyTorch Module with attrs cannot get parameter list

body:
The attr's package somehow ruins pytorch's parameter() method for a module. I am wondering if anyone has any work-arounds or solutions, so that the two packages can seamlessly integrate?

If not, any advice on which github to post the issue to? My instinct would be to post this onto attr's github, but the stack trace is almost entirely relevant to pytorch's codebase.

Python 3.7.3
attrs== 19.1.0
torch==1.1.0.post2
torchvision==0.3.0

import attr
import torch

class RegularModule(torch.nn.Module):
    pass

@attr.s
class AttrsModule(torch.nn.Module):
    pass

module = RegularModule()
print(list(module.parameters()))

module = AttrsModule()
print(list(module.parameters()))

The actual output is:

$python attrs_pytorch.py
[]
Traceback (most recent call last):
  File "attrs_pytorch.py", line 18, in <module>
    print(list(module.parameters()))
  File "/usr/local/anaconda3/envs/bgg/lib/python3.7/site-packages/torch/nn/modules/module.py", line 814, in parameters
    for name, param in self.named_parameters(recurse=recurse):
  File "/usr/local/anaconda3/envs/bgg/lib/python3.7/site-packages/torch/nn/modules/module.py", line 840, in named_parameters
    for elem in gen:
  File "/usr/local/anaconda3/envs/bgg/lib/python3.7/site-packages/torch/nn/modules/module.py", line 784, in _named_members
    for module_prefix, module in modules:
  File "/usr/local/anaconda3/envs/bgg/lib/python3.7/site-packages/torch/nn/modules/module.py", line 975, in named_modules
    if self not in memo:
TypeError: unhashable type: 'AttrsModule'

The expected output is:

$python attrs_pytorch.py
[]
[]