title: Training on variable length data - EEG data classification

body:
I'm a student working on a project involving using EEG data to perform lie detection. I will be working with raw EEG data from 2 channels and will record the EEG data during the duration that the subject is replying to the question. Thus, the data will be a 2-by-variable length array stored in a csv file, which holds the sensor readings from each of the two sensors. For example, it would look something like this:

Time (ms) | Sensor 1 | Sensor 2|
--------------------------------
10        | 100.2    | -324.5  |
20        | 123.5    | -125.8  |
30        | 265.6    | -274.9  |
40        | 121.6    | -234.3  |
             ....
2750      | 100.2    | -746.2  |

I want to predict, based on this data, whether the subject is lying or telling the truth (thus, binary classification.) I was planning on simply treating this as structured data and training based on that. However, on second thought, that wouldn't work at all because of a few reasons:

The order in which the data is organized matters as it is continuous time data.
The length of the data is variable as, again, it's time data and the time it takes for the subject to lie/tell the truth is inconsistent.
I don't know how to deal with there being multiple channels of data.

How would I go about setting up a training model for this type of data? I think this is a "time series classification" problem, but I'm not sure. Any kind of help would be greatly appreciated. Thank you in advance!


title: tensorflow matrix_band_part function equivalent in pytorch

body:
I need to create upper triangular masking tensor in pytorch. in tensorflow it's easy using matrix_band_part.  is it any pytorch equivalent for this function?
it's like numpy triu_indices but I need for tensor, not just matrix.


title: ML Model inputs and outputs analogy

body:
As I learn more and more about ML (I am a mobile DEV) I'm starting to form an analogy in my head. I would like the communities opinion / validation.

As a front end DEV you have a backend and an API that you can make requests to. The standard format for the inputs and outputs to the API is JSON.

I'm running into a problem with ML Models that I am trying to use where I don't know how to read the expected input (API) and I don't know how to decode the expected output.

So far I my experience has been fragmented because some models say "Give me an image of [1,2,120,120]" or something like that.

To analogize, is there a unified way to define inputs and outputs for a ML model like JSON unifies the inputs and outputs for an backend API?

If so, what are some rules one must follow to encode and decode data into this format?


title: Inplace arithmetic operation versus normal arithmetic operation in PyTorch Tensors

body:
I am trying to build Linear regression using Pytorch framework and while implementing Gradient Descent, I observed two different outputs based on how I use arithmetic operation in Python code. Below is the code:

 #X and Y are input and target labels respectively
 X = torch.randn(100,1)*10
 Y = X + 3*torch.randn(100,1) +2
 plt.scatter(X.numpy(),Y.numpy())

 #Initialiation of weight and bias
 w = torch.tensor(1.0,requires_grad=True)
 b = torch.tensor(1.0,requires_grad=True)

 #forward pass
 def forward_feed(x):
   y = w*x +b
   return y

 #Parameters Learning
 epochs = 100
 lr = 0.00008
 loss_list = []
 for epoch in range(epochs):
   print('epoch',epoch)
   Y_pred = forward_feed(X)
   loss = torch.sum((Y - Y_pred)**2)
   loss_list.append(loss)
   loss.backward()
   with torch.no_grad():
     w -= lr*w.grad
     b -= lr*b.grad
     w.grad.zero_()
     b.grad.zero_()

If I use this code, I get the expected results i.e my code is able to estimate weight and bias. However, If I change the gradient descent code line like below:

  w =w- lr*w.grad
  b =b- lr*b.grad

I get the below error:

 AttributeError                            Traceback (most recent call
 last)
 <ipython-input-199-84b86804d4d5> in <module>()
 ---> 41         w.grad.zero_()
      42         b.grad.zero_()

 AttributeError: 'NoneType' object has no attribute 'zero_'

Can anyone please help me with this?

I did try checking answers on google and found a related link: https://github.com/pytorch/pytorch/issues/7731. But this is exactly opposite to what I am facing. As per this link, they say that inplace assignment is causing a problem because tensors share the same storage. However, For my code, Inplace operation works not the normal operation.


title: mask first k elements in a 3D tensor in PyTorch (different k for each row)

body:
I have a tensor M of dimensions [NxQxD] and a 1d tensor of indices idx (of size N). I want to efficiently create a tensor mask of dimensions [NxQxD] such that mask[i,j,k] = 1 iff j <= idx[i], i.e. I want to keep only the idx[i] first dimensions out of Q in the second dimension (dim=1) of M, for every row i.

Thanks!


title: Is this a correct way to use nn.Transformer?

body:
I use nn.Transformer to predict 2 length sequences as mentioned in dataset below

import torch, torch.nn as nn

class A(nn.Module):
  def __init__(self):
    super().__init__()
    self.embed_src = nn.Embedding(2, 10)
    self.embed_target = nn.Embedding(2, 10)
    self.transformer = nn.Transformer(10, 2)
    self.lin = nn.Linear(10, 2)
    self.softmax = nn.Softmax(dim=-1)
  def forward(self, inp, tgt):
    embed_src = self.embed_src(inp)
    embed_target = self.embed_target(tgt)
    # print(embedding.shape)
    output = self.transformer(embed_src.view(len(inp), 1, -1), embed_target.view((len(tgt)), 1, -1))
    output = self.lin(output)
    print('output.shape', output.shape, 'output', output)
    print('embed_target', embed_target.view(2, 1, -1))
    print('softmax probabilities', self.softmax(output))
    return output.permute(0, 2, 1)

model = A()
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)

dataset = [
                [([0, 1, 0, 1, 0, 1]), ([[0], [1]])],
                [([1, 0, 1, 0, 1, 0]), ([[1], [0]])],
                [([0, 0, 0, 0, 0, 0]), ([[0], [0]])],
                [([1, 1, 1, 1, 1, 1]), ([[1], [1]])],
                [([1, 1, 0, 0, 1, 1]), ([[0], [0]])],
                [([0, 0, 1, 1, 0, 0]), ([[1], [1]])]
               ]
tensor_dataset = []
for i in range(6):
  tensor_dataset.append([torch.tensor(dataset[i][0]), torch.tensor(dataset[i][1])])

criterion = nn.CrossEntropyLoss()

for i in range(1000):
  optimizer.zero_grad()
  print(tensor_dataset[i%6][0], tensor_dataset[i%6][1])
  loss = criterion(model(tensor_dataset[i%6][0], tensor_dataset[i%6][1]), tensor_dataset[i%6][1])
  print(loss)
  loss.backward()
  optimizer.step()

do I need to set self.embed_target requires_grad as False?
is output.permute(0, 2, 1) correct way to use with nn.CrossEntropyLoss?


title: Algorithim of how Conv2d is implemented in PyTorch

body:
I am working on an inference model of a pytorch onnx model which is why this question is being asked.

Assume, I have a image with dimensions 32 x 32 x 3 (CIFAR-10 dataset). I pass it through a Conv2d with dimensions : 3 x 192 x 5 x 5. The command I used is: Conv2d(3, 192, kernel_size=5, stride=1, padding=2)

Using the formula (stated here for reference pg12 https://arxiv.org/pdf/1603.07285.pdf) I should be getting an output image with dimensions 28 x 28 x 192 (input - kernel + 1 = 32 - 5 + 1).

Question is how has PyTorch implemented this 4d tensor 3 x 192 x 5 x 5 to get me an output of 28 x 28 x 192 ? The layer is a 4d tensor and the input image is a 2d one.

How is the kernel (5x5) spread in the image matrix 32 x 32 x 3 ? What does the kernel convolve with first -> 3 x 192  or 32 x 32?

Note : I have understood the 2d aspects of things. I am asking the above questions in 3 or more.


title: Predicting the surface of the car using its 2d bbox and plate bbox

body:
I'm trying to solve an interesting problem w/o using GPU intensive model in inference time. (No Deep Learning)

Input: 2D Image which contains car(s) in it, with accurate bboxes, and also a bbox of the plate's car. (We also know that the cameras are located just a bit above the cars)

Output: Surface of the car prediction (the bottom side of a cuboid in 3d bbox)

Approach 1: I'm trying to leverage the fact that I have some prior knowledge except the 2d bbox of the car, but also the 2d bbox of the plate, which can give me the orientation of the car, I thought about taking an angle between the center bbox of the car and the center bbox of the 2d plate to understand what is the direction the car is facing at.

After I know the direction the car is facing to, I also can roughly know where should be one of the edges of the surface because of the fact that the 3d bbox is bounded by the 2d bbox (thus the surface is also bounded), and the fact that the 2d bbox of the plate is a few pixels far from the surface, so one of the edges of the surface can be estimated.

But, the problem here is determining the lateral edges, how 'long' should they be. I'm not quite sure how to estimate the lateral sides of the bottom surface, but I think it can be somehow inferred by the size of the 2d bbox of the car (which again, should bound that surface). Maybe I'll be able to solve it after finding the edge of the surface, and then exploring ways to infer the lateral edges of that surface.

Approach 2: Annotating the data with 3d bboxes with a pre-trained model, and trying to predict the 3d bbox from a 2d bbox (and probably some more priors like 2d bbox of the plate), but I'm not using a deep model to do it, but a simple NN with a few layers to predict the 3d bbox. (trained in a supervised manner)


title: Pytorch loss does't change in vgg 19 model

body:
In pytorch I made a model vgg19 for classification tiny imagenet:

model = nn.Sequential(
    nn.BatchNorm2d(3),
    nn.Conv2d(in_channels=3, out_channels=64, kernel_size=(3,3), padding=1),
    nn.ReLU(),
    nn.Conv2d(in_channels=64, out_channels=64, kernel_size=(3,3), padding=1),
    nn.ReLU(),
    nn.MaxPool2d((2,2)),
    nn.Conv2d(in_channels=64, out_channels=128, kernel_size=(3,3), padding=1),
    nn.ReLU(),
    nn.Conv2d(in_channels=128, out_channels=128, kernel_size=(3,3), padding=1),
    nn.ReLU(),
    nn.MaxPool2d((2,2)),
    nn.Conv2d(in_channels=128, out_channels=256, kernel_size=(3,3), padding=1),
    nn.ReLU(),
    nn.Conv2d(in_channels=256, out_channels=256, kernel_size=(3,3), padding=1),
    nn.ReLU(),
    nn.Conv2d(in_channels=256, out_channels=256, kernel_size=(3,3), padding=1),
    nn.ReLU(),
    nn.Conv2d(in_channels=256, out_channels=256, kernel_size=(3,3), padding=1),
    nn.ReLU(),
    nn.MaxPool2d((2,2)),
    nn.Conv2d(in_channels=256, out_channels=512, kernel_size=(3,3), padding=1),
    nn.ReLU(),
    nn.Conv2d(in_channels=512, out_channels=512, kernel_size=(3,3), padding=1),
    nn.ReLU(),
    nn.Conv2d(in_channels=512, out_channels=512, kernel_size=(3,3), padding=1),
    nn.ReLU(),
    nn.Conv2d(in_channels=512, out_channels=512, kernel_size=(3,3), padding=1),
    nn.ReLU(),
    nn.MaxPool2d((2,2)),
    nn.Conv2d(in_channels=512, out_channels=512, kernel_size=(3,3), padding=1),
    nn.ReLU(),
    nn.Conv2d(in_channels=512, out_channels=512, kernel_size=(3,3), padding=1),
    nn.ReLU(),
    nn.Conv2d(in_channels=512, out_channels=512, kernel_size=(3,3), padding=1),
    nn.ReLU(),
    nn.Conv2d(in_channels=512, out_channels=512, kernel_size=(3,3), padding=1),
    nn.ReLU(),
    nn.MaxPool2d((2,2)),
    nn.Flatten(),
    nn.Linear(25088, 4096),
    nn.Linear(4096, 1000),
    nn.Linear(1000, 200),
    nn.Softmax(),
    nn.Dropout2d(),
)

In the learning process, the loss remains at approximately the same value (5.3 +- 0.01).

###############
iter = 4000 / 80000
loss = 5.295811176300049
###############
iter = 4800 / 80000
loss = 5.298299789428711
###############
iter = 5600 / 80000
loss = 5.309792995452881
###############
iter = 6400 / 80000
loss = 5.3179707527160645
###############
iter = 7200 / 80000
loss = 5.3179707527160645

I have already tried to increase and decrease lr, batch_size. But still no idea how to fix it.

Training code
(epochs = 10, loss = cross_entropy, optim = Adam(lr = 0.01), X_batch.shape = (batch_size, 3, 224, 224):

for epoch in range(num_epochs):
    i = 0
    for (X_batch, y_batch) in train_batch_gen:
        X_batch = Variable(torch.FloatTensor(X_batch)).cuda()
        y_batch = Variable(torch.LongTensor(y_batch)).cuda()
        logits = model.cuda().forward(X_batch)
        opt.zero_grad()
        loss = lossFunc(logits, y_batch)
        loss.backward()
        opt.step()

        train_loss.append(loss.data.cpu().numpy())
        if i % (batch_size*100) == 0:
            print("###############")
            print(f"iter = {i} / {80000}")
            print(f"loss = {np.mean(train_loss[-len(train_dataset) // batch_size :])}")
        i += batch_size


title: Convert Tensor to CUDA in Pytorch C++, But the content changed

body:
everyone, I'm facing a problem that I need to copy a tensor to CUDA. The code are as follows:

torch::Tensor r_data1 = torch::from_blob(raw_data.data(), {batch_size, arr_size}).to(torch::kInt64);
auto tmp_data = static_cast<size_t*>(r_data1.storage().data());
size_t tmp_size = r_data1.numel();
cout << "r data1 tensor ";
for (size_t i = 0; i < tmp_size; i++) {
    cout << tmp_data[i] << "\t";
}
cout << endl;

torch::Tensor r_data2 = r_data1.cuda();
tmp_data = static_cast<size_t*>(r_data2.cpu().storage().data());
tmp_size = r_data2.numel();
cout << "r data tensor ";
for (size_t i = 0; i < tmp_size; i++) {
    cout << tmp_data[i] << "\t";
}
cout << endl;

But the results are:
r data1 tensor 101  4886    2456    4689    3787    2336    2356    1298    2128    2356    4886    2456    4689 1298   2128    2356    736 6963    7252    5401    3215    3333    1921    3441    8110    1384
r data tensor 139682015587936   139682014923568 2456    4689    3787    2336    2356    1298    2128    2356    4886 2456   4689    1298    2128    2356    736 6963    7252    5401    3215    3333    1921    3441    8110    1384
The ids have been changed. Why? Have you come into this problem?


title: torch cannot allocate small size tensor (< 1GB) on GPU but it can for CPU on the same node with 400+ GB memory on databricks

body:
My question is relevant to my previous one pytorch allocate memory for small size tensor on cpu and gpu but got error on a node with more than 400 GB.
But, it is different so I created a new thread.
In this question, I have changed size of the input tensor size.
 import torch
 from torch import nn
 import numpy as np
 num_embedding, num_dim = 14000, 300
 embedding = nn.Embedding(num_embedding, num_dim)

 row, col = 8000, 302
 t = [[x for x in range(col)] for _ in range(row)]

 t1 = torch.tensor(t)
 print(t1.shape)  # torch.Size([8000, 302])

 type(t1), t1.device, (t1.nelement() * t1.element_size())/(1024**3) # (torch.Tensor, device(type='cpu'), 0.01800060272216797)

 tt = embedding(t1)
 embedding.forward(t1)

 t2 = t1.cuda()
 t2.device, t2.shape, t2.grad, t2.nelement(), t2.element_size(), (t2.nelement() * t2.element_size())/(1024**3) # (device(type='cuda', index=0), torch.Size([8000, 302]), None, 2416000, 8, 0.01800060272216797)

 embedding_cuda = embedding.cuda()
 torch.cuda.empty_cache()
 embedding_cuda(t2) # RuntimeError: CUDA out of memory. Tried to allocate 2.70 GiB (GPU 0; 11.17 GiB total capacity; 7.19 GiB already allocated; 2.01 GiB free; 8.88 GiB reserved in total by PyTorch)

Why the small size tensor (0.018 GB) can be allocated to cpu but cannot be allocated to gpu on the same node (p2.8xlarge) ? why it requires 2.7 GB, which 100 times larger than its original size at least ?
I have checked most posts at https://stackoverflow.com/search?q=RuntimeError%3A+CUDA+out+of+memory.+Tried+to+allocate+GiB
but, none of them can help me about this.


title: How to pad a sequence in Pytorch?

body:
I am trying to learn about pack_padded_sequence more and want to test it in this small dataset.
I managed to merge two tensors of different sequence length but when I try to pad the sequence it gives me an error. I am following a given example in the comments but with a different dataset. why do we "pack" the sequences in pytorch?
Code:
!wget https://raw.githubusercontent.com/jbrownlee/Datasets/master/airline-passengers.csv

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import torch
import torch.nn as nn
from torch.autograd import Variable
from sklearn.preprocessing import MinMaxScaler

training_set = pd.read_csv('airline-passengers.csv')

def sliding_windows(data, seq_length):
    x = []
    y = []

    for i in range(len(data)-seq_length-1):
        _x = data[i:(i+seq_length)]
        _y = data[i+seq_length]
        x.append(_x)
        y.append(_y)

    return x,np.array(y)

sc = MinMaxScaler()
training_data = sc.fit_transform(training_set)

seq_length = 8
x, y = sliding_windows(training_data, seq_length)

train_size = int(len(y) * 0.67)
test_size = len(y) - train_size

trainX = Variable(torch.Tensor(np.array(x[0:train_size])))
trainY = Variable(torch.Tensor(np.array(y[0:train_size])))

seq_length = 4
x1, y1 = sliding_windows(training_data, seq_length)

train_size = int(len(y1) * 0.67)
test_size = len(y1) - train_size

trainX1 = Variable(torch.Tensor(np.array(x1[0:train_size])))
trainY1 = Variable(torch.Tensor(np.array(y1[0:train_size])))

seq_batch = [trainX,
             trainX1]

seq_lens = [8, 4]

added_seq_batch = torch.nn.utils.rnn.pad_sequence(seq_batch, batch_first=True)

Error:
RuntimeError                              Traceback (most recent call last)
<ipython-input-26-599f28dada0a> in <module>()
----> 1 added_seq_batch = torch.nn.utils.rnn.pad_sequence(seq_batch, batch_first=True)

/usr/local/lib/python3.6/dist-packages/torch/nn/utils/rnn.py in pad_sequence(sequences, batch_first, padding_value)
    369         # use index notation to prevent duplicate references to the tensor
    370         if batch_first:
--> 371             out_tensor[i, :length, ...] = tensor
    372         else:
    373             out_tensor[:length, i, ...] = tensor

RuntimeError: The expanded size of the tensor (8) must match the existing size (4) at non-singleton dimension 1.  Target sizes: [93, 8, 1].  Tensor sizes: [93, 4, 1]


title: Pytorch lightning Datamodule override warning: Signature of method '.setup()' does not match signature of base method in class 'LightningDataModule'

body:
The following is a working Pytorch Lightning DataModule.
import os
from pytorch_lightning import LightningDataModule
import torchvision.datasets as datasets
from torchvision.transforms import transforms
import torch
from torch.utils.data import DataLoader
from Testing.Research.config.paths import mnist_data_download_folder

class PressureDataModule(LightningDataModule):
    def __init__(self, config):
        super().__init__()
        self._config = config

    def prepare_data(self):
        pass

    def setup(self, stage):
        # transform
        transform = transforms.Compose([transforms.ToTensor()])
        mnist_train_full = datasets.MNIST(mnist_data_download_folder, train=True, download=False, transform=self._transforms)
        mnist_test = datasets.MNIST(mnist_data_download_folder, train=False, download=False, transform=self._transforms)

        # train/val split
        train_size = int(self._config.train_size /
                         (self._config.train_size + self._config.val_size) * len(mnist_train_full))
        val_size = len(mnist_train_full) - train_size
        mnist_train, mnist_val = torch.utils.data.random_split(mnist_train_full, [train_size, val_size])

        # assign to use in dataloaders
        self._train_dataset = mnist_train
        self._val_dataset = mnist_val
        self._test_dataset = mnist_test

    def train_dataloader(self):
        return DataLoader(self._train_dataset, batch_size=self._config.batch_size, num_workers=self._config.num_workers)

    def val_dataloader(self):
        return DataLoader(self._val_dataset, batch_size=self._config.batch_size, num_workers=self._config.num_workers)

    def test_dataloader(self):
        return DataLoader(self._test_dataset, batch_size=self._config.batch_size, num_workers=self._config.num_workers)

Pycharm doesn't like setup with

Signature of method 'PressureDataModule.setup()' does not match
signature of base method in class 'LightningDataModule'

If there is no match, why is Pycharm crying?
Is it because of different parameters? What is the correct number of parameters?

What's the correct way to fix this?


title: How to create a submodel from a pretrained model in pytorch without having to rewrite the whole architecture?

body:
So, I have been working on neural style transfer in Pytorch, but I'm stuck at the point where we have to run the input image through limited number of layers and minimize the style loss. Long story short, I want to find a way in Pytorch to evaluate the input at different layers of the architecture(I'm using vgg16). I have seen this problem solved very simply in keras, but I wanted to see if there is a similar way in pytorch as well or not.
from keras.applications.vgg16 import VGG16

model = VGG16()
model = Model(inputs=model.inputs, outputs=model.layers[1].output)


title: How to log the graph from fairseq to tensorboard

body:
I got the impression from reading tutorials that this would be easy: slap in a call to add_graph().
First tricky part was working out where to put it. I've gone with fairseq_cli/train.py at the very top of the enumerate() loop in train():
for i, samples in enumerate(progress):

    if i == 0:
        # Output graph for tensorboard
        writer = progress._writer("")  #The "" is tag
        writer.add_graph(trainer._model, samples)
        writer.flush()

I'm passing --tensorboard-logdir mydir/ into the call to fairseq-train. That causes a TensorboardProgressBarWrapper wrapper around SimpleProgressBar (or whatever logging format you use), and so I'm trying to re-use the writer instance. (Maybe that is my mistake?)
I can get the model out of the trainer object, and the other thing we need for add_graph is some data, which is why I've put it in the above loop, so I can use samples.  I've also tried samples[0] with exactly the same error message.
Which is:
Tracer cannot infer type of ({'id': tensor([743216, 642485,  92182, 793806, 494734, 275334,  53282, 449572,   1758,
  ...
  20734, 469070, 678489, 473213]), 'nsentences': 832, 'ntokens': 29775, 'net_input': {'src_tokens': tensor([[  225,    30,   874,  ...,  1330,    84,     2],
  [  442,   734,  7473,  ...,    38,     5,     2],
  ...,
  [ 6238, 11411,   428,  ..., 10387,     5,     2]]), 'src_lengths': tensor([21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21,
  ...
  21, 21, 21, 21]), 'prev_output_tokens': tensor([[   2,   22,  133,  ..., 4445,   46,    1],
  ...,
  [   2,   22,   59,  ...,  112,    6,    1]])}, 'target': tensor([[ 22, 133, 203,  ...,  46,   2,   1],
  [132, 429,  40,  ...,  46,   2,   1],
  ...,
  [ 22,  59, 177,  ...,   6,   2,   1]])},)
:Dictionary inputs to traced functions must have consistent type. Found Tensor and int
Error occurs, No graph saved

I'm hoping someone might understand that error message better than me? Tips on better places to put the call to add_graph(), or better ways to approach this whole thing (getting a visual representation of my model), are also welcome. (PyTorchViz hasn't been updated for over 2 years, and apparently won't work with latest pytorch.)
Tensorboard logging of the training is working, by the way.
ADDITIONAL
I poked around some more, and found it is going to call forward() of my model, which (ignoring optional args), looks like:
def forward(self, src_tokens, src_lengths, prev_output_tokens):

so I tried manually creating empty data:
dummy_data = {'src_tokens':torch.zeros((1,256)),'src_lengths':torch.zeros((1,256)),'prev_output_tokens':torch.zeros((1,256))}
writer.add_graph(trainer._model,dummy_data,verbose=True)

That fails with:
TypeError: forward() missing 2 required positional arguments: 'src_lengths' and 'prev_output_tokens'

And if I use kwargs-style:
dummy_data = {'src_tokens':torch.zeros((1,256)),'src_lengths':torch.zeros((1,256)),'prev_output_tokens':torch.zeros((1,256))}
writer.add_graph(trainer._model,**dummy_data,verbose=True)

it fails with:
TypeError: add_graph() got an unexpected keyword argument 'src_tokens'

I'm starting to think this may be a limitation in add_graph() and it will only work with a simple def forward(self, x): ?


title: RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED on google colab

body:
I am trying to train a custom dataset on SlowFast repository using google colab, but I can't get past this error. Looking around the internet said this has to do with batch size but that did not help. Any help is welcome. Thanks.
Traceback (most recent call last):
  File "tools/run_net.py", line 44, in <module>
    main()
  File "tools/run_net.py", line 25, in main
    launch_job(cfg=cfg, init_method=args.init_method, func=train)
  File "/content/gdrive/My Drive/SlowFast/slowfast/utils/misc.py", line 297, in launch_job
    func(cfg=cfg)
  File "/content/gdrive/My Drive/SlowFast/tools/train_net.py", line 383, in train
    misc.log_model_info(model, cfg, use_train_input=True)
  File "/content/gdrive/My Drive/SlowFast/slowfast/utils/misc.py", line 174, in log_model_info
    get_model_stats(model, cfg, "flop", use_train_input)
  File "/content/gdrive/My Drive/SlowFast/slowfast/utils/misc.py", line 152, in get_model_stats
    count_dict, *_ = model_stats_fun(model, inputs)
  File "/usr/local/lib/python3.7/dist-packages/fvcore/nn/flop_count.py", line 130, in flop_count
    for op, flop in flop_counter.by_operator().items():
  File "/usr/local/lib/python3.7/dist-packages/fvcore/nn/jit_analysis.py", line 239, in by_operator
    stats = self._analyze()
  File "/usr/local/lib/python3.7/dist-packages/fvcore/nn/jit_analysis.py", line 494, in _analyze
    graph = _get_scoped_trace_graph(self._model, self._inputs, self._aliases)
  File "/usr/local/lib/python3.7/dist-packages/fvcore/nn/jit_analysis.py", line 161, in _get_scoped_trace_graph
    graph, _ = _get_trace_graph(module, inputs)
  File "/usr/local/lib/python3.7/dist-packages/torch/jit/_trace.py", line 1139, in _get_trace_graph
    outs = ONNXTracedModule(f, strict, _force_outplace, return_inputs, _return_inputs_states)(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/torch/jit/_trace.py", line 130, in forward
    self._force_outplace,
  File "/usr/local/lib/python3.7/dist-packages/torch/jit/_trace.py", line 116, in wrapper
    outs.append(self.inner(*trace_inputs))
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 887, in _call_impl
    result = self._slow_forward(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 860, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/content/gdrive/My Drive/SlowFast/slowfast/models/video_model_builder.py", line 588, in forward
    x = self.s1(x)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 887, in _call_impl
    result = self._slow_forward(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 860, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/content/gdrive/My Drive/SlowFast/slowfast/models/stem_helper.py", line 114, in forward
    x[pathway] = m(x[pathway])
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 887, in _call_impl
    result = self._slow_forward(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 860, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/content/gdrive/My Drive/SlowFast/slowfast/models/stem_helper.py", line 190, in forward
    x = self.conv(x)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 887, in _call_impl
    result = self._slow_forward(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 860, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/conv.py", line 521, in forward
    self.padding, self.dilation, self.groups)
RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED

My cuda version,
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2020 NVIDIA Corporation
Built on Wed_Jul_22_19:09:09_PDT_2020
Cuda compilation tools, release 11.0, V11.0.221
Build cuda_11.0_bu.TC445_37.28845127_0


title: PyTorch | getting "RuntimeError: Found dtype Long but expected Float"

body:
I'm trying to train a CNN on a custom dataset. Code:
Dataset.py
class MyDataset(Dataset):
    def __init__(self, csv_file, root_dir):
        self.annotations = pd.read_csv(csv_file)
        self.root_dir = root_dir
        self.transform = transform

    def __len__(self):
        return len(self.annotations)

    def __getitem__(self, index):
        img_path = os.path.join(self.root_dir, self.annotations.iloc[index, 0])
        y_label = torch.tensor(int(self.annotations.iloc[index, 1]))
        img = cv2.imread(img_path)

        # resize
        res = cv2.resize(img, dsize=(50, 50), interpolation=cv2.INTER_CUBIC)

        # convert image to tensor
        res = torch.from_numpy(res)

        return (res, y_label)

Model.py
class ConvNet(torch.nn.Module):
    def __init__(self):
        super(ConvNet, self).__init__()

        f2 = 4
        self.layer2 = nn.Sequential(
            nn.Conv2d(50, f2, kernel_size=5, padding=2),
            nn.ReLU(),
            nn.BatchNorm2d(f2),
            nn.MaxPool2d(kernel_size=2, stride=2))
        self.fc1 = nn.Linear(100, 200)
        self.fc2 = nn.Linear(200, 20)
        self.fc3 = nn.Linear(20, 1)

    def forward(self, x):
        x = self.layer2(x.float())
        x = x.reshape(x.size(0), -1)
        x = self.fc1(x)
        x = self.fc2(x)
        x = self.fc3(x)
        return x

and here is my training code:
dataset = MyDataset(
    csv_file='dataset.csv',
    root_dir='tmp')

train_set, test_set = torch.utils.data.random_split(dataset, lengths=[500, 70])

train_loader = DataLoader(dataset=train_set, batch_size=16, shuffle=True)
test_loader = DataLoader(dataset=test_set, batch_size=16, shuffle=True)

model = ConvNet()
criterion = nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.0001)

for epoch in range(20):
    losses = []

    for batch_idx, (data, targets) in enumerate(train_loader):

        data = data.to(device=device)
        targets = targets.to(device=device)

        # forward
        scores = model(data)
        loss = criterion(scores, targets)
        losses.append(loss.item())

        # backward
        optimizer.zero_grad()
        loss.backward()

        optimizer.step()

    print('Cost: {0} = {1}'.format(epoch, sum(losses)/len(losses)))

But I get RuntimeError: Found dtype Long but expected Float. This probably comes from the fact that I do x = self.layer2(x.float()) to avoid overflow.
I would like to know how to fix that error. It's difficult to pin-point where the exact problem comes from.
How can I solve this?


title: RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same - PyTorch

body:
I'm trying to push both my mode and data, images and labels, to run on the GPU by doing:
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

Followed by:
count = 0
loss_list = []
iteration_list = []
accuracy_list = []
epochs = 30

for epoch in range(epochs):
    for i, (images, labels) in enumerate(trainloader):
            net = net.to(device)
            images.to(device)
            labels.to(device)

            optimizer.zero_grad()
            outputs = net(images)
            loss = criterion(outputs, labels)
            loss.backward()
            optimizer.step()

            count += 1

            if count % 50 == 0:
                correct = 0
                total = 0

                for i, (images, labels) in enumerate(testloader):
                    images.to(device)
                    labels.to(device)

                    outputs = net(images)
                    predicted = torch.max(outputs.data, 1)[1]
                    total += len(labels)
                    correct += (predicted == labels).sum()
                accuracy = 100 * correct / float(total)

                loss_list.append(loss.data)
                iteration_list.append(count)
                accuracy_list.append(accuracy)

                if count % 500 == 0:
                    print("Iteration: {} Loss: {} Accuracy: {} %".format(count, loss.data, accuracy))

I'm explicitly pushing my model and data to device however I am met by the error:
RuntimeError                              Traceback (most recent call last)
<ipython-input-341-361b906da73d> in <module>()
     12
     13             optimizer.zero_grad()
---> 14             outputs = net(images)
     15             loss = criterion(outputs, labels)
     16             loss.backward()

4 frames
/usr/local/lib/python3.7/dist-packages/torch/nn/modules/conv.py in _conv_forward(self, input, weight, bias)
    394                             _pair(0), self.dilation, self.groups)
    395         return F.conv2d(input, weight, bias, self.stride,
--> 396                         self.padding, self.dilation, self.groups)
    397
    398     def forward(self, input: Tensor) -> Tensor:

RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same

I feel like I'm doing the right thing by pushing both model and data to GPU but I can't figure out why it's not working. Does somebody know what's going wrong? Thank you in advance.


title: Making a prediction from a trained convolution network

body:
Here is my convolution net that creates training data , then trains on this data using a single convolution with relu activation :

train_dataset = []
mu, sigma = 0, 0.1 # mean and standard deviation
num_instances = 10

for i in range(num_instances) :
    image = []
    image_x = np.random.normal(mu, sigma, 1000).reshape((1 , 100, 10))
    train_dataset.append(image_x)

mu, sigma = 100, 0.80 # mean and standard deviation
for i in range(num_instances) :
    image = []
    image_x = np.random.normal(mu, sigma, 1000).reshape((1 , 100, 10))
    train_dataset.append(image_x)

labels_1 = [1 for i in range(num_instances)]
labels_0 = [0 for i in range(num_instances)]

labels = labels_1 + labels_0

print(labels)

x2 = torch.tensor(train_dataset).float()
y2 = torch.tensor(labels).long()

my_train2 = data_utils.TensorDataset(x2, y2)
train_loader2 = data_utils.DataLoader(my_train2, batch_size=batch_size_value, shuffle=False)

import torch
import torch.nn as nn
import torchvision
import torchvision.transforms as transforms

# Device configuration
device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
# device = 'cpu'

# Hyper parameters
num_epochs = 50
num_classes = 2
batch_size = 5
learning_rate = 0.001

# Convolutional neural network (two convolutional layers)
class ConvNet(nn.Module):
    def __init__(self, num_classes=1):
        super(ConvNet, self).__init__()
        self.layer1 = nn.Sequential(
            nn.Conv2d(1, 16, kernel_size=5, stride=1, padding=2),
            nn.BatchNorm2d(16),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2))
        self.layer2 = nn.Sequential(
            nn.Conv2d(16, 32, kernel_size=5, stride=1, padding=2),
            nn.BatchNorm2d(32),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2))
        self.fc = nn.Linear(32*25*2, num_classes)

    def forward(self, x):
        out = self.layer1(x)
        out = self.layer2(out)
        out = out.reshape(out.size(0), -1)
        out = self.fc(out)
        return out

model = ConvNet(num_classes).to(device)

# Loss and optimizer
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

# Train the model
total_step = len(train_loader2)
for epoch in range(num_epochs):
    for i, (images, labels) in enumerate(train_loader2):
        images = images.to(device)
        labels = labels.to(device)

        # Forward pass
        outputs = model(images)
        loss = criterion(outputs, labels)

        # Backward and optimize
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if (i % 10) == 0:
            print ('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}'
                   .format(epoch+1, num_epochs, i+1, total_step, loss.item()))

To make a single prediction I use :

model(x2[10].unsqueeze_(0).cuda())

Which outputs :

tensor([[ 4.4880, -4.3128]], device='cuda:0')

Should this not return an image tensor of shape (100,10) of the prediction ?

Update : In order to perform a prediction I use :

torch.argmax(model(x2[2].unsqueeze_(0).cuda()), dim=1)

src : https://discuss.pytorch.org/t/argmax-with-pytorch/1528/11

torch.argmax in this context returns the position of the value that maximises the prediction.


title: how to fix capsule training problem for a single class of MNIST dataset?

body:
I am training a Capsule Network with both encoder and decoder part. It works perfectly fine with all the classes (10 classes) of the MNIST data set. But when I am extracting a single class say (class 0 or class 5) and then training the capsule network, the reconstruction of the image is very poor.

Where do I need to change the network setting, or do I have an error in my data preparation?

I tried:

I changed the total class from 10 (for ten digits to 1 for 1 digit and even for 2 for 2 digits).
When I am using the default MNIST dataset, I am getting no error or tensor size, but when I am extracting a particular class and then passing it into the network, I am facing issues like a) Dimensional Issues b) Float tensor warning.

I fixed these things but manually adding a dimension and converting the data to data.float().cuda() tensor. I did this for both the case i.e when I am using the 10 Digit Capsules and when I am using the 1 Digit Capsules for training a single class digit.

But after this, the network is running fine, but I am getting really blurred and poor reconstructions. While when I am training the whole MNIST dataset without extracting any class and passing it to the network, it doesn't throw any error and the reconstruction works really fine.

I would love to share the more detail and other parts of the code -

import numpy as np

import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.autograd import Variable
from torch.optim import Adam
from torchvision import datasets, transforms

USE_CUDA = True

### **Here we prepare the data for the complete 10 class digit training**###
class Mnist:
    def __init__(self, batch_size):
        dataset_transform = transforms.Compose([
                       transforms.ToTensor(),
                       transforms.Normalize((0.1307,), (0.3081,))
                   ])

        train_dataset = datasets.MNIST('../data', train=True, download=True, transform=dataset_transform)
        test_dataset = datasets.MNIST('../data', train=False, download=True, transform=dataset_transform)

        self.train_loader  = torch.utils.data.DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
        self.test_loader = torch.utils.data.DataLoader(test_dataset, batch_size=batch_size, shuffle=True)

## **Here is my code for extracting a single class digit extraction**##
class Mnist:
    def __init__(self,batch_size):

        dataset_transform = transforms.Compose([
                            transforms.ToTensor(),
                            transforms.Normalize((0.1307,), (0.3081,))
        ])

        train_mnist = datasets.MNIST("../data", train=True)
        test_mnist = datasets.MNIST("../data", train= False)
        train_image, train_label = train_mnist.train_data, train_mnist.train_labels
        test_image, test_label = test_mnist.test_data, test_mnist.test_labels

        train_0, test_0 = [train_image[key] for (key, label) in enumerate(train_label) if int(label) == 5],[test_image[key] for (key, label) in enumerate(test_label) if int(label) == 5]
        train_label_0, test_label_0 = zero__train = [train_label[key] for (key, label) in enumerate(train_label) if int(label) == 5],[test_label[key] for (key, label) in enumerate(test_label) if int(label) == 5]

        train_dataset = tuple(zip(train_0, train_label_0))
        test_dataset = tuple(zip(test_0, test_label_0))

        self.train_loader  = torch.utils.data.DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
        self.test_loader = torch.utils.data.DataLoader(test_dataset, batch_size=batch_size, shuffle=True)

# Here is the main code for the capsule training.

''' The below code is used for training the 1 class but using the 10 Digit capsules
'''
class ConvLayer(nn.Module):
    def __init__(self, in_channels=1, out_channels=256, kernel_size=9):
        super(ConvLayer, self).__init__()

        self.conv = nn.Conv2d(in_channels=in_channels,
                               out_channels=out_channels,
                               kernel_size=kernel_size,
                               stride=1
                             )

    def forward(self, x):
        return F.relu(self.conv(x))
class PrimaryCaps(nn.Module):
    def __init__(self, num_capsules=8, in_channels=256, out_channels=32, kernel_size=9):
        super(PrimaryCaps, self).__init__()

        self.capsules = nn.ModuleList([
            nn.Conv2d(in_channels=in_channels, out_channels=out_channels, kernel_size=kernel_size, stride=2, padding=0)
                          for _ in range(num_capsules)])

    def forward(self, x):
        u = [capsule(x) for capsule in self.capsules]
        u = torch.stack(u, dim=1)
        u = u.view(x.size(0), 32 * 6 * 6, -1)
        return self.squash(u)

    def squash(self, input_tensor):
        squared_norm = (input_tensor ** 2).sum(-1, keepdim=True)
        output_tensor = squared_norm *  input_tensor / ((1. + squared_norm) * torch.sqrt(squared_norm))
        return output_tensor
class DigitCaps(nn.Module):
    def __init__(self, num_capsules=10, num_routes=32 * 6 * 6, in_channels=8, out_channels=16):
        super(DigitCaps, self).__init__()

        self.in_channels = in_channels
        self.num_routes = num_routes
        self.num_capsules = num_capsules

        self.W = nn.Parameter(torch.randn(1, num_routes, num_capsules, out_channels, in_channels))

    def forward(self, x):
        batch_size = x.size(0)
        x = torch.stack([x] * self.num_capsules, dim=2).unsqueeze(4)
#         print(f"x at epoch {epoch} is equal to : {x}")

        W = torch.cat([self.W] * batch_size, dim=0)
#         print(f"W at epoch {epoch} is equal to : {W}")
        u_hat = torch.matmul(W, x)
#         print(f"u_hatat epoch {epoch} is equal to : {u_hat}")

        b_ij = Variable(torch.zeros(1, self.num_routes, self.num_capsules, 1))
        if USE_CUDA:
            b_ij = b_ij.cuda()
#             print(f"b_ij at epoch {epoch} is equal to : {b_ij}")

        num_iterations = 3
        for iteration in range(num_iterations):
            c_ij = F.softmax(b_ij, dim =1)
            c_ij = torch.cat([c_ij] * batch_size, dim=0).unsqueeze(4)

            s_j = (c_ij * u_hat).sum(dim=1, keepdim=True)
            v_j = self.squash(s_j)
#             print(f"b_ij at iteration {iteration} is equal to : {b_ij}")
            if iteration < num_iterations - 1:
                a_ij = torch.matmul(u_hat.transpose(3, 4), torch.cat([v_j] * self.num_routes, dim=1))
                b_ij = b_ij + a_ij.squeeze(4).mean(dim=0, keepdim=True)

        return v_j.squeeze(1)

    def squash(self, input_tensor):
        squared_norm = (input_tensor ** 2).sum(-1, keepdim=True)
        output_tensor = squared_norm *  input_tensor / ((1. + squared_norm) * torch.sqrt(squared_norm))
        return output_tensor
class Decoder(nn.Module):
    def __init__(self):
        super(Decoder, self).__init__()

        self.reconstraction_layers = nn.Sequential(
            nn.Linear(16 * 10, 512),
            nn.ReLU(inplace=True),
            nn.Linear(512, 1024),
            nn.ReLU(inplace=True),
            nn.Linear(1024, 784),
            nn.Sigmoid()
        )

    def forward(self, x, data):
        classes = torch.sqrt((x ** 2).sum(2))
        classes = F.softmax(classes, dim =1)

        _, max_length_indices = classes.max(dim=1)
        masked = Variable(torch.sparse.torch.eye(10))
        if USE_CUDA:
            masked = masked.cuda()
        masked = masked.index_select(dim=0, index=max_length_indices.squeeze(1).data)

        reconstructions = self.reconstraction_layers((x * masked[:, :, None, None]).view(x.size(0), -1))
        reconstructions = reconstructions.view(-1, 1, 28, 28)

        return reconstructions, masked
class CapsNet(nn.Module):
    def __init__(self):
        super(CapsNet, self).__init__()
        self.conv_layer = ConvLayer()
        self.primary_capsules = PrimaryCaps()
        self.digit_capsules = DigitCaps()
        self.decoder = Decoder()

        self.mse_loss = nn.MSELoss()

    def forward(self, data):
        output = self.digit_capsules(self.primary_capsules(self.conv_layer(data)))
        reconstructions, masked = self.decoder(output, data)
        return output, reconstructions, masked

    def loss(self, data, x, target, reconstructions):
        return self.margin_loss(x, target) + self.reconstruction_loss(data, reconstructions)
#         return self.reconstruction_loss(data, reconstructions)

    def margin_loss(self, x, labels, size_average=True):
        batch_size = x.size(0)

        v_c = torch.sqrt((x**2).sum(dim=2, keepdim=True))

        left = F.relu(0.9 - v_c).view(batch_size, -1)
        right = F.relu(v_c - 0.1).view(batch_size, -1)
#         print(f"shape of labels, left and right respectively - {labels.size(), left.size(), right.size()}")

        loss = labels * left + 0.5 * (1.0 - labels) * right
        loss = loss.sum(dim=1).mean()

        return loss

    def reconstruction_loss(self, data, reconstructions):
        loss = self.mse_loss(reconstructions.view(reconstructions.size(0), -1), data.view(reconstructions.size(0), -1))
        return loss*0.0005

capsule_net = CapsNet()
if USE_CUDA:
    capsule_net = capsule_net.cuda()
optimizer = Adam(capsule_net.parameters())
capsule_net

##### Here is the problem while training####
batch_size = 100
mnist = Mnist(batch_size)

n_epochs = 5

for epoch in range(n_epochs):
    capsule_net.train()
    train_loss = 0
    for batch_id, (data, target) in enumerate(mnist.train_loader):

        target = torch.eye(10).index_select(dim=0, index=target)
        data, target = Variable(data), Variable(target)

        if USE_CUDA:
            data, target = data.cuda(), target.cuda()
            data, target = data.float().cuda(), target.float().cuda() # Here I changed the data to float and it's required only when I am using my extracted dataset for a single class
            data = data[:,:,:] # Use this when 1st MNist data is used
#             data = data[:,None,:,:] # Use this when I am using my extracted single class digits

        optimizer.zero_grad()
        output, reconstructions, masked = capsule_net(data)
        loss = capsule_net.loss(data, output, target, reconstructions)
        loss.backward()
        optimizer.step()

        train_loss += loss.item()

#         if batch_id % 100 == 0:
#             print ("train accuracy:", sum(np.argmax(masked.data.cpu().numpy(), 1) ==
#                                    np.argmax(target.data.cpu().numpy(), 1)) / float(batch_size))

    print (train_loss / len(mnist.train_loader))

I used this to see the main data as image and the reconstructed image

import matplotlib
import matplotlib.pyplot as plt

def plot_images_separately(images):
    "Plot the six MNIST images separately."
    fig = plt.figure()
    for j in range(1, 10):
        ax = fig.add_subplot(1, 10, j)
        ax.matshow(images[j-1], cmap = matplotlib.cm.binary)
        plt.xticks(np.array([]))
        plt.yticks(np.array([]))
    plt.show()
plot_images_separately(data[:10,0].data.cpu().numpy())
plot_images_separately(reconstructions[:10,0].data.cpu().numpy())


title: Sudden Drop of Volatile GPU-Util Pytorch

body:
I am running https://github.com/jadore801120/attention-is-all-you-need-pytorch this code that has implemented the paper Attention-is-all-you-need with PyTorch with Gigaword dataset, so the task is changed into text summarization from the original paper's intent, machine translation (dataset name is WST2016).

The dataset is much bigger in Gigaword comparing to WST2016, total size of training dataset is 3,800,000 and 29,056 respectively.

The problem is, when I train Gigaword, exactly when the graph trains the 31% of the entire data, it gets dramatically slows down, up to 30 times become slower linearly with GPU util become 0-15% from 99-100%.

I can't figure out why the training gets slow down exactly at 31% of a batch.

What would be the possible start point to debug this problem?


title: Getting the last layer from a pretrained pytorch for transfer learning?

body:
This is what I did:

list(tmp.state_dict().keys())[-1].split('.')[0]

What is the proper way?
My goal is to replace the last layer for the purpose of transfer learning.


title: Constructing Key:Value pair from list comprehension in Python

body:
I am trying to extend some code.

What works:

pretrained_dict = {k: v for k, v pretrained_dict.items if k in model_dict}

However, if I extend it to:

pretrained_dict = {k: v if k in model_dict else k1:v1 for k, v, k1, v1 in zip(pretrained_dict.items(), model_dict.items()) }

The code fails, If I put the else at the end it still fails:

 pretrained_dict = {k: v if k in model_dict for k, v, k1, v1 in zip(pretrained_dict.items(), model_dict.items()) else k1:v1}
                                                 ^
SyntaxError: invalid syntax

How can I construct the key value pair using an if else condition over two lists?


title: how to clear a variable to free ram memory during inference?

body:
I'm trying to train a unet on small scale images and perform inferences on large ones.
my idea was to train on my gpu and do the inference stage on cpu.
the problem is that I don't have enough ram memory.
during the forward pass, I have tried deleting a variable that I no longer need, but the memory is not being allocated again.
what am I doing wrong?
here is my forward pass :

def forward(self, x):
    x1 = self.in_conv(x)
    del x
    x2 = self.down1(x1)
    x3 = self.down2(x2)
    x4 = self.down3(x3)
    x5 = self.down4(x4)
    x_n = self.up1(x5, x4)
    x_n = self.up2(x_n, x3)
    x_n = self.up3(x_n, x2)
    x_n = self.up4(x_n, x1)
    return self.out_conv(x_n)

the first line

x1 = self.in_conv(x)

is the one that takes most of the ram. however, I don't need x after that, and "del x" doesn't do the trick.


title: Does reshaping a tensor retain the characteristics of original tensor?

body:
I have a tensor T of the shape (8, 5, 300), where 8 is the batch size, 5 is the number of documents in each batch, and 300 is the encoding of each of the document. If I reshape the Tensor as follows, does the properties of my Tensor remain the same?

T = T.reshape(5, 300, 8)
T.shape
>> Size[5, 300, 8]

So, does this new Tensor indicate the same properties as the original one? By the properties, I mean, can I say that this is also a Tensor of batch size 8, with 5 documents for each batch, and a 300 dimensional encoding for each document?

Does this affect the training of the model? If reshaping of Tensor messes up the datapoints, then there is no point in training. For example, If reshaping like above gives output as a batch of 5 samples, with 300 documents of size 8 each. If it happens so, then it's useless, since I do not have 300 documents, neither do I have batch of 5 samples.

I need to reshape it like this because my model in between produces output of the shape [8, 5, 300], and the next layer accepts input as [5, 300, 8].


title: How to manage the hidden state dims when using pad_sequence?

body:
Using Pytorch LSTM architecture trying to build a text generation model. For every batch, I'm using pad_sequence to have min padding for every sequence, therefore I have a variable dims batch (batch_size * seq_len). I'm applying also pack_padded_seq to only give the non-zero (non-padding) tokens to the LSTM. But, the variable dims batch throws an error while feeding it to the LSTM as following; Expected hidden[0] size (1, 8, 16), got (1, 16, 16). In this error, I have provided batch size 16 with 8 tokens for every sequence, but the hidden state is 16 * 16.

I have tried to create the hidden state in the forward function, but that did not work well. How can I create the hidden state such that it can accept variable dims batch and it will not be lost for the whole epoche?

class RNNModule(nn.Module):
    def __init__(self, n_vocab, seq_size, embedding_size, lstm_size):
        super(RNNModule, self).__init__()
        self.seq_size = seq_size
        self.lstm_size = lstm_size
        self.embedding, num_embeddings, embedding_dim = create_emb_layer(weight_matrix, False)
        self.lstm = nn.LSTM(embedding_size,
                        lstm_size,
                        num_layers=flags.n_layers,
                        batch_first=True
                        )
        self.dense = nn.Linear(lstm_size, n_vocab)

    def forward(self, x,length,prev_state):

        embed = self.embedding(x)
        packed_input = db.pack_src(embed,length)
        packed_output, state = self.lstm(packed_input,prev_state)
        padded,_ = db.pad_pack(packed_output)
        logits = self.dense(padded)
        return logits, state

    def zero_state(self, batch_size = flags.batch_size):
        return (torch.zeros(flags.n_layers, batch_size, self.lstm_size),
            torch.zeros(flags.n_layers, batch_size, self.lstm_size))

input: tensor([[  19,    9,    4,    3,   68,    8,    6,    2],
    [  19,    9,    4,    3,    7,    8,    6,    2],
    [   3,   12,   17,   10,    6,   40,    2,    0],
    [   4,    3,  109,    7,    6,    2,    0,    0],
    [ 188,    6,    7,   18,    3,    2,    0,    0],
    [   4,    3,   12,    6,    7,    2,    0,    0],
    [   6,    7,    3,   13,    2,    0,    0,    0],
    [   3,   28,   17,   69,    2,    0,    0,    0],
    [   6,    3,   12,   11,    2,    0,    0,    0],
    [   3,   13,    6,    7,    2,    0,    0,    0],
    [   3,    6,    7,   13,    2,    0,    0,    0],
    [   6,    3,   23,    7,    2,    0,    0,    0],
    [   3,   28,   10,    2,    0,    0,    0,    0],
    [   6,    3,   23,    2,    0,    0,    0,    0],
    [   3,    6,   37,    2,    0,    0,    0,    0],
    [1218,    2,    0,    0,    0,    0,    0,    0]])

Zero tokens are padding.
Embedding size: 64
LSTM size: 16
batch size: 16


title: Pytorch: LSTM does not learn

body:
I'm having a hard time training my LSTM model, it does not seem to learn at all. The training loss is hardly decreasing and accuracy changes for very simple models (1 layer, few lstm units) but eventually gets stuck at 45%, just like the more complex models right from the start. Most of the times, it only predicts one class as output.
I've tried varying all the hyper parameters, but it doesn't really seem to change anything so I'm afraid I'm missing something obvous.

Here is my model:

Embedding_Model(nn.Module):
    def __init__(self, dim, vocab_size, classes, lstm_units=100, num_layers=3, bidirectional=True):
        super(Embedding_Model, self).__init__()

        self.word_embeddings = nn.Embedding(vocab_size, dim)
        self.lstm =  nn.LSTM(dim, lstm_units, num_layers=num_layers, bidirectional=bidirectional, batch_first=True)
        self.fc = nn.Sequential(
            nn.Linear(lstm_units*(1+int(bidirectional)), classes),
            nn.Softmax(dim=1)
        )

    def forward(self, x):
        x = self.word_embeddings(x)
        x, (h,c)= self.lstm(x)
        x = x.transpose(0,1)
        x = self.fc(x[-1])
        return x

My training set contains roughly 5000 input sequences. The length of the input sequences is 1400 (padded with zeros). There are 150,000 different tokens, I've tried embedding dimensions between 10 and 200.
I use 3 output classes (quite balanced in the training set) and cross entropy loss.

Did I mess up anything obvious? I understand that the training set is rather small, but at least I would expect some overfitting. But the model does not seem to learn anything at all.


title: Need help understanding the gradient function in pytorch

body:
The following code

w = np.array([[2., 2.],[2., 2.]])
x = np.array([[3., 3.],[3., 3.]])
b = np.array([[4., 4.],[4., 4.]])
w = torch.tensor(w, requires_grad=True)
x = torch.tensor(x, requires_grad=True)
b = torch.tensor(b, requires_grad=True)

y = w*x + b
print(y)
# tensor([[10., 10.],
#         [10., 10.]], dtype=torch.float64, grad_fn=<AddBackward0>)

y.backward(torch.FloatTensor([[1, 1],[ 1, 1]]))

print(w.grad)
# tensor([[3., 3.],
#         [3., 3.]], dtype=torch.float64)

print(x.grad)
# tensor([[2., 2.],
#         [2., 2.]], dtype=torch.float64)

print(b.grad)
# tensor([[1., 1.],
#         [1., 1.]], dtype=torch.float64)

As the tensor argument inside gradient function is an all ones tensor in the shape of the input tensor, my understanding says that

w.grad means derivative of y w.r.t w, and produces b,
x.grad means derivative of y w.r.t x, and produces b and
b.grad means derivative of y w.r.t b, and produces all ones.

Out of these, only point 3 answer is matching my expected result. Can someone help me in understanding the first two answers. I think I understand the accumulation part, but don't think that is happening here.


title: Tensorboard only showing dots, line not showing

body:
I am using tensorboard through pytorch and have created a scaler which shows accuracy, loss, and validation accuracy.  When I open the graph it only shows dot and dosen't show a line connecting these dots.
Here is a snippet of my code
tb = SummaryWriter(comment=f'-{epoch}')

tb.add_scalar("Accuracy", float(correct/len(train_tensor)*100), epoch)
tb.add_scalar("Loss", loss, epoch)
tb.add_scalar("ValiAccuracy", float(correct/len(train_tensor)*100), epoch)
tb.close()

What is leading tensorboard to not show these lines?


title: Function with torch.mm showing error while using torch.optim

body:
I am kind of a newbie with PyTorch. Please forgive me if the question is childish. I am trying to minimize a function using PyTorch's optim. The function includes matrix multiplication. The details are given below.
First I have a tensor:
Xv.requires_grad_()
XT.requires_grad_()

My Objective Function:
def errorFun(x):
    ax = x[0]
    ay = x[1]
    x0 = x[2]
    y0 = x[3]

    A = torch.tensor([[ax, 0., x0], [0., ay, y0], [0., 0., 1.]], dtype=torch.float64)
    B = torch.tensor([[b11, b12, b13], [b21, b22, b23], [b31, b32, b33]], dtype=torch.float64)
    H = torch.mm(A, B)
    Ps = torch.mm(H, X)
    px = Ps[0,:]
    py = Ps[1,:]
    PX = torch.stack([px, py], dim=0)
    PX.requires_grad_()
    return mseloss(PX, XT)

I am minimizing it:
for ii in range(n_optim_steps):
    optimizer.zero_grad()
    loss = errorFun(params)

    #print('Step # {}, loss: {}'.format(ii, loss.item()))
    loss.backward()
    # Access gradient if necessary
    grad = params.grad.data
    optimizer.step()

But I am getting this error:
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-54-84b874448a25> in <module>()
     77     loss.backward()
     78     # Access gradient if necessary
---> 79     grad = params.grad.data
     80     optimizer.step()
     81

AttributeError: 'NoneType' object has no attribute 'data'

Thanks in advance.


title: Cupy works well with TITAN V, but not with TITAN RTX

body:
I am using cupy to run a cuda code with pytorch.
My env is
ubuntu 20, anaconda-python 3.7.6, nvidia-driver 440, cuda 10.2, cupy-cuda102, torch 1.4.0
First, I wrote a simple main code
import data_load_test
from tqdm import tqdm
import torch
from torch.utils.data import DataLoader

def main():

    dataset = data_load_test.DataLoadTest()
    training_loader = DataLoader(dataset, batch_size=1)
    with torch.cuda.device(0):
        pbar = tqdm(training_loader)
        for epoch in range(3):
            for i, img in enumerate(pbar):
                print("see the message")

if __name__ == "__main__":
    main()

and data loader like this.
from torch.utils.data import Dataset
import cv2
import cupy as cp

def read_cuda_file(cuda_path):
    f = open(cuda_path, 'r')
    source_line = ""
    while True:
        line = f.readline()
        if not line: break
        source_line = source_line + line
    f.close()
    return source_line

class DataLoadTest(Dataset):
    def __init__(self):
        source = read_cuda_file("cuda/cuda_code.cu")
        cuda_source = '''{}'''.format(source)
        module = cp.RawModule(code=cuda_source)
        self.myfunc = module.get_function('myfunc')

        self.input = cp.asarray(cv2.imread("hi.png",-1), cp.uint8)
        h, w, c = self.input.shape
        self.h = h
        self.w = w
        self.output = cp.zeros((w, h, 3), dtype=cp.uint8)

        self.block_size = (32, 32)
        self.grid_size = (h // self.block_size[1], w // self.block_size[0])

    def __len__(self):
        return 1

    def __getitem__(self, idx):
        self.myfunc(self.grid_size, self.block_size, (self.input, self.output, self.h, self.w))
        return cp.asnumpy(self.output)

And my cuda code is,
#define PI 3.14159265358979323846f
extern "C"{
__global__ void myfunc(const unsigned char* refImg, unsigned char* warpImg, const long long cols, const long long rows)
{

    long long x = blockDim.x * blockIdx.x + threadIdx.x;
    long long y = blockDim.y * blockIdx.y + threadIdx.y;

    long long indexImg = y * cols + x;

    warpImg[indexImg * 3] = 0;
    warpImg[indexImg * 3 + 1] = 1;
    warpImg[indexImg * 3 + 2] = 2;
}
}

I have two GPUs TITAN V (device 0) and TITAN RTX (device 1)
When I run this code with TITAN V,(main function 3rd line)
with torch.cuda.device(0):

it works fine, but
with TITAN RTX,
with torch.cuda.device(1):

It gives an error message like this.
  File "cupy/core/raw.pyx", line 66, in cupy.core.raw.RawKernel.__call__
  File "cupy/cuda/function.pyx", line 162, in cupy.cuda.function.Function.__call__
  File "cupy/cuda/function.pyx", line 144, in cupy.cuda.function._launch
  File "cupy/cuda/driver.pyx", line 293, in cupy.cuda.driver.launchKernel
  File "cupy/cuda/driver.pyx", line 118, in cupy.cuda.driver.check_status
cupy.cuda.driver.CUDADriverError: CUDA_ERROR_CONTEXT_IS_DESTROYED: context is destroyed

Please help.


title: How to replace every row that are filled with zero with a certain value from a Pytorch tensor?

body:
I have a Pytorch tensor of the size bsize x 50 x 50 where some of the rows are completely filled with zeros:
         [[0, 2, 0,  ..., 0, 0, 0],
         [2, 0, 2,  ..., 0, 0, 0],
         [0, 2, 0,  ..., 0, 0, 0],
         ...,
         [0, 0, 0,  ..., 0, 0, 0],
         [0, 0, 0,  ..., 0, 0, 0],
         [0, 0, 0,  ..., 0, 0, 0]],

        [[0, 2, 0,  ..., 0, 0, 0],
         [2, 0, 2,  ..., 0, 0, 0],
         [0, 2, 0,  ..., 0, 0, 0],

I want to  replace the rows filled with zeros with a negative value -100 throughout in the tensor.
Expected tensor -
         [[0, 2, 0,  ..., 0, 0, 0],
         [2, 0, 2,  ..., 0, 0, 0],
         [0, 2, 0,  ..., 0, 0, 0],
         ...,
         [-100, -100, -100,  ..., -100,-100, -100],
         [-100, -100, -100,  ..., -100,-100, -100],
         [-100, -100, -100,  ..., -100,-100, -100]],

        [[0, 2, 0,  ..., 0, 0, 0],
         [2, 0, 2,  ..., 0, 0, 0],
         [0, 2, 0,  ..., 0, 0, 0],

Whats the best way to do this avoiding a loop over the row shape ?


title: Simple PytorchBenchmark from transformers lib of Huggingface script gives CUDA initialisation error

body:
I am trying to run a simple benchmark script from the transformers lib from Huggingface, but it fails due to a CUDA error, which leads to another error:
1 / 1
Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
Traceback (most recent call last):
  File "/home/cbarkhof/code-thesis/Experimentation/Benchmarking/benchmarking-models-claartje.py", line 12, in <module>
    benchmark.run()
  File "/home/cbarkhof/.local/lib/python3.6/site-packages/transformers/benchmark/benchmark_utils.py", line 674, in run
    memory, inference_summary = self.inference_memory(model_name, batch_size, sequence_length)
ValueError: too many values to unpack (expected 2)

The script simply follows the example like shown on this page:
from transformers import PyTorchBenchmark, PyTorchBenchmarkArguments

benchmark_args = PyTorchBenchmarkArguments(models=["bert-base-uncased"],
                                           batch_sizes=[8],
                                           sequence_lengths=[8, 32, 128, 512],
                                           save_to_csv=True,
                                           log_filename='log',
                                           env_info_csv_file='env_info')

benchmark = PyTorchBenchmark(benchmark_args)
benchmark.run()

If anyone can point me to why this might be happening. Please let me know :). Cheers!


title: How to change Pytorch model to work with 3d input instead 2d input?

body:
I am trying to train an agent to play Connect4 game. I found an example of how it can be trained. Representation of board is 1x6x7 array:
[[[0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0]
  [0 0 0 0 0 0 2]
  [0 0 0 0 0 0 1]]]

This neural network architecture is used:
class Net(BaseFeaturesExtractor):
    def __init__(self, observation_space: gym.spaces.Box, features_dim: int = 256):
        super(Net, self).__init__(observation_space, features_dim)
        # We assume CxHxW images (channels first)
        # Re-ordering will be done by pre-preprocessing or wrapper
        n_input_channels = observation_space.shape[0]
        self.cnn = nn.Sequential(
            nn.Conv2d(n_input_channels, 32, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.Conv2d(32, 64, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.Conv2d(64, 128, kernel_size=3, stride=1, padding=0),
            nn.ReLU(),
            nn.Flatten(),
        )

        # Compute shape by doing one forward pass
        with th.no_grad():
            n_flatten = self.cnn(th.as_tensor(observation_space.sample()[None]).float()).shape[1]

        self.linear = nn.Sequential(nn.Linear(n_flatten, features_dim), nn.ReLU())

    def forward(self, observations: th.Tensor) -> th.Tensor:
        return self.linear(self.cnn(observations))

And it scored not so bad on a game with an agent 2 which moves randomly:
Agent 1 Win Percentage: 0.59
Agent 2 Win Percentage: 0.38
Number of Invalid Plays by Agent 1: 3
Number of Invalid Plays by Agent 2: 0
Number of Draws (in 100 game rounds): 0

Here 3 layers representation was suggested as one of the ways how an agent can be improved:

I have tried to implement it and this is the example of the new 3 layer representation of board:
[[[0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0]
  [0 0 0 0 0 0 1]]

 [[0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0]
  [0 0 0 0 0 0 1]
  [0 0 0 0 0 0 0]]

 [[0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0]
  [0 0 0 0 0 0 1]
  [0 0 0 0 0 0 0]
  [1 1 1 1 1 1 0]]]

When I run this with the current neural network architecture, the agent is not able to train appropriately:
Agent 1 Win Percentage: 0.0
Agent 2 Win Percentage: 0.0
Number of Invalid Plays by Agent 1: 100
Number of Invalid Plays by Agent 2: 0
Number of Draws (in 100 game rounds): 0

Here you can see my code.
As you can see now I have 3 layers instead of one. That's why I have tried to use Conv3d:
class Net(BaseFeaturesExtractor):
    def __init__(self, observation_space: gym.spaces.Box, features_dim: int = 256):
        super(Net, self).__init__(observation_space, features_dim)
        # We assume CxHxW images (channels first)
        # Re-ordering will be done by pre-preprocessing or wrapper
        n_input_channels = observation_space.shape[0]
        self.cnn = nn.Sequential(
            nn.Conv3d(n_input_channels, 32, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.Conv3d(32, 64, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.Conv3d(64, 128, kernel_size=3, stride=1, padding=0),
            nn.ReLU(),
            nn.Flatten(),
        )

        # Compute shape by doing one forward pass
        with th.no_grad():
            n_flatten = self.cnn(th.as_tensor(observation_space.sample()[None]).float()).shape[1]

        self.linear = nn.Sequential(nn.Linear(n_flatten, features_dim), nn.ReLU())

When I try to run this code, it is showing this error:

RuntimeError: Expected 5-dimensional input for 5-dimensional weight
[32, 1, 3, 3, 3], but got 4-dimensional input of size [1, 3, 6, 7]
instead

My question: how can I use Conv3D layer with 3x6x7 shaped input?


title: Calculate Batch Pairwise Sinkhorn Distance in PyTorch

body:
I have two tensors and both are of same shape. I want to calculate pairwise sinkhorn distance using GeomLoss.
What i have tried:
import torch
import geomloss  # pip install git+https://github.com/jeanfeydy/geomloss

a = torch.rand((8,4))
b = torch.rand((8,4))

geomloss.SamplesLoss('sinkhorn')(a,b)
# ^ input shape [batch, feature_dim]
# will return a scalar value

geomloss.SamplesLoss('sinkhorn')(a.unsqueeze(1),b.unsqueeze(1))
# ^ input shape [batch, n_points, feature_dim]
# will return a tensor of size [batch] of distances between a[i] and b[i] for each i

However I would like to compute pairwise distance where the resultant tensor should be of size [batch, batch]. To achieve this, I tried the following to use broadcasting:
geomloss.SamplesLoss('sinkhorn')(a.unsqueeze(0), b.unsqueeze(1))

But I got this error message:

ValueError: Samples x and y should have the same batchsize.


title: How does the nn.Embedding module relate intuitively to the idea of an embedding in general?

body:
So, I'm having a hard time understanding nn.Embedding. Specifically, I can't connect the dots between what I understand about embeddings as a concept and what this specific implementation is doing.
My understanding of an embedding is that it is a smaller dimension representation of some larger dimension data point. So it maps data in N-d to a M-d latent/embedding space such that M < N.
As I understand it, this mapping is achieved through the learning process, as in an auto-encoder. The encoder learns the optimal embedding so that the decoder can reconstruct the original input.
So my question is, how does this relate to nn.Embedding module:

A simple lookup table that stores embeddings of a fixed dictionary and size.
This module is often used to store word embeddings and retrieve them using indices. The input to the module is a list of indices, and the output is the corresponding word embeddings.

does this layer "learn" a lower dimensional representation of a larger input space? Or is it something else entirely?
What I'm looking for is to take the very abstract language of the documentation to something real:
Let's say I have some input x. This input might be a vectorized image or maybe some sequence daily temperature data. In any case, this input x has 100 elements (100 days of temperature, or a 10x10 image).
How can you explain the use of nn.Embedding() in this case?
What does each argument mean in a real world context?


title: How can I fold a Tensor that I unfolded with PyTorch that has overlap?

body:
I have a Tensor of size: torch.Size([1, 63840]) which I then unrolled:
inp_unfolded = inp_seq.unfold(1, 160, 80)

that gives me a shape of: torch.Size([1, 797, 160])
How can I re-fold that to get a Tensor of torch.Size([1, 63840])?


title: Different Matrix multiplication behaviour between Keras and Pytorch

body:
I was trying to understand how matrix multiplication works over 2 dimensions in DL frameworks and I stumbled upon an article here.
He used Keras to explain the same and it works for him.
But when I try to reproduce the same code in Pytorch, it fails with the error as in the output of the following code
Pytorch Code:
a = torch.ones((2,3,4))
b = torch.ones((7,4,5))
c = torch.matmul(a,b)
print(c.shape)

Output: RuntimeError: The size of tensor a (2) must match the size of tensor b (7) at non-singleton dimension 0

Keras Code:
a = K.ones((2,3,4))
b = K.ones((7,4,5))
c = K.dot(a,b)
print(c.shape)

Output:(2, 3, 7, 5)

Can somebody explain what is it that I'm doing wrong?


title: Python calculate phash from an image located at a url

body:
I want to calculate the phash from about 10.000.000 pictures, from which i only have the url where they are located at.
I know how to download a picture and then calculate the phash after that, but i always have to safe the picture first.
Is it possible to download the picture and calculate the phash without saving it or is it even possible to not download the picture at all and just calculate the phash just using the url?
This is my code to download the first ten pictures and calculate the phash:
folder, pic_savefolder = 'data', 'data/pictures'
file = 'external-asset-url-clean.csv'
path = os.path.join(folder,file)
df = pd.read_csv(path, header=None, names=["URL"])
counter = 0
hashes = set()
headers = {"User-Agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36"}

for image_url in df['URL']:
    filename = image_url.split('/')[-1]
    try:
        r = requests.get(image_url, allow_redirects=False,verify=False, headers=headers)
        pathlong = os.path.join(pic_savefolder, filename)
        with open(pathlong,"wb") as f:
            f.write(r.content)
            hash = imagehash.phash(Image.open(pathlong))
            hashes.add((hash))
        counter += 1
        if counter > 10:
            break
except Exception as e:
    print(e)
    print("\n")


title: Modifying the Learning Rate in the middle of the Model Training in Deep Learning

body:
Below is the code to configure TrainingArguments consumed from the HuggingFace transformers library to finetune the GPT2 language model.
training_args = TrainingArguments(
        output_dir="./gpt2-language-model", #The output directory
        num_train_epochs=100, # number of training epochs
        per_device_train_batch_size=8, # batch size for training #32, 10
        per_device_eval_batch_size=8,  # batch size for evaluation #64, 10
        save_steps=100, # after # steps model is saved
        warmup_steps=500,# number of warmup steps for learning rate scheduler
        prediction_loss_only=True,
        metric_for_best_model = "eval_loss",
        load_best_model_at_end = True,
        evaluation_strategy="epoch",
        learning_rate=0.00004, # learning rate
    )

early_stop_callback = EarlyStoppingCallback(early_stopping_patience  = 3)

trainer = Trainer(
        model=gpt2_model,
        args=training_args,
        data_collator=data_collator,
        train_dataset=train_dataset,
        eval_dataset=test_dataset,
        callbacks = [early_stop_callback],
 )

The number of epochs as 100 and learning_rate as 0.00004 and also the early_stopping is configured with the patience value as 3.
The model ran for 5/100 epochs and noticed that the difference in loss_value is negligible. The latest checkpoint is saved as checkpoint-latest.
Now Can I modify the learning_rate may be to 0.01 from 0.00004 and resume the training from the latest saved checkpoint - checkpoint-latest? Doing that will be efficient?
Or to train with the new learning_rate value should I start the training from the beginning?


title: Why the input type changed while call _call_impl in pytorch

body:
I'm working at converting pytorch model to openvino IR, the nn of my pytorch is vgg16, and the method is faster rcnn, there is a weird problem when I debug the code.
when I run the following code
fc7 = self.classifier(pool, abc="abc123")
the pool object is OpenVinoTensor, abc="abc123" is the test parameters for debuging.
And the classifier is defined as following:

Sequential(
(0): Linear(in_features=25088, out_features=4096, bias=True)
(1): ReLU(inplace=True)
(2): Linear(in_features=4096, out_features=4096, bias=True)
(3): ReLU(inplace=True)

after self.classifier(pool, abc="abc123") is called, it calls torch.nn.Module._call_impl
def _call_impl(self, *input, **kwargs)
then I found the "input" here is '(ReLU_57,)', which is a tuple, I think it should be OpenVinoTensor, as well as "kwargs" is abc="abc123".
What happened after I call classifier?


title: Delete duplicated rows in torch.tensor

body:
I have a torch.tensor of shape (n,m) and I want to remove the duplicated rows (or at least find them). For example:
t1 = torch.tensor([[1, 2, 3], [4, 5, 6], [1, 2, 3], [4, 5, 6]])
t2 = remove_duplicates(t1)

t2 should be now equal to tensor([[1, 2, 3], [4, 5, 6]]), that is rows 1 and 3 are removed. Do you know a way to perform this operation?
I was thinking to do something with torch.unique but I cannot figure out what to do.


title: CUDA out of memory.Tried to allocate 14.00 MiB (GPU 0;4.00 GiB total capacity;2 GiB already allocated;6.20 MiB free;2GiB reserved intotal by PyTorch)

body:
I am trying to run this code from fastai
from fastai.vision.all import *
path = untar_data(URLs.PETS)/'images'

def is_cat(x): return x[0].isupper()
dls = ImageDataLoaders.from_name_func(
    path, get_image_files(path), valid_pct=0.2, seed=42,
    label_func=is_cat, item_tfms=Resize(224), num_workers = 0)

learn = cnn_learner(dls, resnet34, metrics=error_rate)
learn.fine_tune(1)

I get the following error

RuntimeError: CUDA out of memory. Tried to allocate 14.00 MiB (GPU 0;
4.00 GiB total capacity; 2.20 GiB already allocated; 6.20 MiB free; 2.23 GiB reserved in total by PyTorch)

I also tried running
import torch
torch.cuda.empty_cache()

and restarting the kernel which was of no use
Any help would be appreciated


title: Pytorch expand not trainable?

body:
I have a sequence model and I want to concatenate a same piece of small embedding to the input. Specifically, that looks like

if the original embedding is ABCDEFG, the expected one would be

A  B  C  D  E  F  G

X  X  X  X  X  X  X

From a coding aspect, that's something like:

embed_0_out = embed_0(input_0) # sequence input
embed_1_out = embed_1(input_1) # single input

embed_combined_out = torch.cat([embed_0_out, embed_1_out.unsqueeze(1).expand(batch_size, input_0_len, 10)], 2))

However, it seems like the weight for embed_1 is not changing after each iteration, suggesting that it is not trainable. Thus, is there any way to make it trainable?


title: Torch-Model deployed in AI-Platform generates "Prediction failed: unknown error."

body:
I modified this code to address my own image-based object detection problem using a yolo_v2 built in Pytorch (weight file is 192 mb).

Everything runs as it should locally, and I can deploy the model without errors using a similar command like in the previous Notebook: I use Python Version 3.5, and I remove --framework='SCIKIT_LEARN', which otherwise would cause an error when creating the version. It deploys without making trouble also when using debug-logging.

Yet, when I try to predict, I receive this error:

{
  "error": "Prediction failed: unknown error."
}

My preprocessor expects three inputs, one of which a GCS address to a stored image, the other two are indices I pass as strings, which get converted to integers internally. A request which locally does exactly what it should looks somewhat like this

    requests = {"filename" : "gs://bucket/imagepath.jpg",
    "coordinateX": "100", "coordinateY": "100"}

I can import my custom model, run it, and receive the expected output.

Yet, following the structure of the Notebook above, after deploying the model and using JSON format to predict, it leads to the error as mentioned.

Also, using the console-based interface inserting this

{
  "instances": [
    {
      "filename": "gs://bucket/imagepath.jpg",
      "coordinateX": "100",
      "coordinateY": "100"}]}

leads to the same error.

I've seen people report similar errors, and for them trial & error regarding the exact formatting of the request helped. On the other hand, I'm out of ideas of the form I should bring my data into to make it work.


title: Is there a handy way to dump the running_stats for a pytorch model?

body:
I'm writing a C version of the pytorch model to run it on my special hardware.
Everything looks ok so far, except the running_mean and running_var in every batchnorm layer.

We have a python code to dump all named_parameters, but nothing to do for the running_stats, although we need to use it in the forwarding computation.

So is there a way to dump it with sort of built-in features?
I searched pytorch doc, no help on my task.
Otherwise I might need to write a regexp code to recognize and dump them.

Thanks a lot.
/Patrick

for name, param in model.named_parameters():
    # here can dump weight and bias, but not running_stats
    names.append(name)
    shapes.append(list(param.data.numpy().shape))
    values.append(param.data.numpy().flatten().tolist())


title: Can I customize the dictionary of a pre-trained transformer neural machine translation model?

body: There are many pre-trained machine translations models available, but it seems like they all need to be run with the dictionary they are trained with. The dictionaries sometimes can have less coverage for my data set (even BPE based ones), and sometimes miss important words as unknowns. What are the best ways to customize the pretrained models to a dictionary learned from my own data set? For example, some way to transfer learn, like unfreezing the encoder layers and retraining?


title: Error in loading Glove vectors using TorchText on Kaggle Kernel

body:
I am trying to load the Glove embedding vectors in kaggle kernel using TorchText lib.

from torchtext import vocab
vec = vocab.Vectors('glove.6B.100d.txt', '../input/glove6b100dtxt/')

I am getting the following error:

OSError: [Errno 30] Read-only file system: '../input/glove6b100dtxt/glove.6B.100d.txt.pt'

Adding screenshot for more clarification:


title: PyTorch TensorBoard write frequency

body:
I'm trying to write my training and validation losses to tensorboard using torch (torch.utils.tensorboard) and it looks like it only writes up to 1000 data points, no matter what the actual number of iterations are. For example, running the following code,

writer1 = SummaryWriter('runs/1')
writer2 = SummaryWriter('runs/2')

for i in range(2000):
    writer1.add_scalar('tag', 1, i)

for i in range(20000):
    writer2.add_scalar('tag', 1, i)

both yield 1000 points exactly when examining and downloaded csv, and even on the tensorboard dashboard, the first points start at steps 5 and 18 and increment such that the total number of steps are 1000, rather than 2,000 and 20,000.

I don't know if this is tensorboard's default behaviour or if its PyTorch's decision, but either way, is there a way to write every single step?


title: How to fix documentation format in VS Code for python kite?

body: I am using pytorch in VSCode but the format of the documentation is just terrible (see image). Do you have any pointers on how we can set the documentation format in VSCode? I believe that documentation contains LaTeX formulas.... Check the image in the link


title: How to conditionally construct a tensor from two other tensors in PyTorch on the GPU?

body:
An example:
import torch

pred = torch.tensor([1,2,1,0,0], device='cuda:0')
correct = torch.tensor([1,0,1,1,0], device='cuda:0')
assigned = torch.tensor([1,2,2,1,0], device='cuda:0')

I want result = tensor([1,2,1,1,0], device='cuda:0').

Basically, when pred is same as correct then correct else assigned.

Moreover, I want to exclude this computation from the gradient computation.

Is there a way to do this without iterating over the tensors?


title: Pytorch Multi-GPU Issue

body: I want to train my model with 2 GPU(id 5, 6), so I run my code with CUDA_VISIBLE_DEVICES=5,6 train.py. However, when I printed torch.cuda.current_device I still got the id 0 rather than 5,6. But torch.cuda.device_count is 2, which semms right. How can I use GPU5,6 correctly?


title: How to prevent torch2trt AttributeError on import?

body:
Whenever importing torch2trt on a JetRacer with a Jetson Nano, it throws an AttributeError. Is there a way to fix this?
I've installed the JetRacer Image from Waveshare: jetcard_v0p0p0 and manually installed torch2trt from here:
git clone https://github.com/NVIDIA-AI-IOT/torch2trt
cd torch2trt
sudo python setup.py install

Running
from torch2trt import torch2trt

gives the following error:
    ---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-4-3f7e7e224fb7> in <module>
----> 1 from torch2trt import torch2trt

/usr/lib/python3.6/importlib/_bootstrap.py in _find_and_load(name, import_)

/usr/lib/python3.6/importlib/_bootstrap.py in _find_and_load_unlocked(name, import_)

/usr/lib/python3.6/importlib/_bootstrap.py in _load_unlocked(spec)

/usr/lib/python3.6/importlib/_bootstrap.py in _load_backward_compatible(spec)

/usr/local/lib/python3.6/dist-packages/torch2trt-0.1.0-py3.6.egg/torch2trt/__init__.py in <module>
      1 from .torch2trt import *
----> 2 from .converters import *
      3 import tensorrt as trt
      4
      5

/usr/lib/python3.6/importlib/_bootstrap.py in _find_and_load(name, import_)

/usr/lib/python3.6/importlib/_bootstrap.py in _find_and_load_unlocked(name, import_)

/usr/lib/python3.6/importlib/_bootstrap.py in _load_unlocked(spec)

/usr/lib/python3.6/importlib/_bootstrap.py in _load_backward_compatible(spec)

/usr/local/lib/python3.6/dist-packages/torch2trt-0.1.0-py3.6.egg/torch2trt/converters/__init__.py in <module>
     15 from .Linear import *
     16 from .LogSoftmax import *
---> 17 from .activation import *
     18 from .adaptive_avg_pool2d import *
     19 from .adaptive_max_pool2d import *

/usr/lib/python3.6/importlib/_bootstrap.py in _find_and_load(name, import_)

/usr/lib/python3.6/importlib/_bootstrap.py in _find_and_load_unlocked(name, import_)

/usr/lib/python3.6/importlib/_bootstrap.py in _load_unlocked(spec)

/usr/lib/python3.6/importlib/_bootstrap.py in _load_backward_compatible(spec)

/usr/local/lib/python3.6/dist-packages/torch2trt-0.1.0-py3.6.egg/torch2trt/converters/activation.py in <module>
      1 from torch2trt.torch2trt import *
      2 from torch2trt.module_test import add_module_test
----> 3 from .unary import UnaryModule
      4
      5

/usr/lib/python3.6/importlib/_bootstrap.py in _find_and_load(name, import_)

/usr/lib/python3.6/importlib/_bootstrap.py in _find_and_load_unlocked(name, import_)

/usr/lib/python3.6/importlib/_bootstrap.py in _load_unlocked(spec)

/usr/lib/python3.6/importlib/_bootstrap.py in _load_backward_compatible(spec)

/usr/local/lib/python3.6/dist-packages/torch2trt-0.1.0-py3.6.egg/torch2trt/converters/unary.py in <module>
     71
     72 @tensorrt_converter('torch.reciprocal')
---> 73 @tensorrt_converter('torch.reciprocal_')
     74 @tensorrt_converter('torch.Tensor.reciprocal')
     75 @tensorrt_converter('torch.Tensor.reciprocal_')

/usr/local/lib/python3.6/dist-packages/torch2trt-0.1.0-py3.6.egg/torch2trt/torch2trt.py in tensorrt_converter(method, is_real, enabled, imports)
    587         module, module_name, qual_name = importlib.import_module(method.__module__), method.__module__, method.__qualname__
    588
--> 589     method_impl = eval('copy.deepcopy(module.%s)' % qual_name)
    590
    591     def register_converter(converter):

/usr/local/lib/python3.6/dist-packages/torch2trt-0.1.0-py3.6.egg/torch2trt/torch2trt.py in <module>

AttributeError: module 'torch' has no attribute 'reciprocal_'

I have already reinstalled torch2trt to no avail.
Kind regards


title: Target and output shape/type for binary classification using PyTorch

body:
so I have some annotated images that I want to use to train a binary image classifier but I have been having issues creating the dataset and actually getting a test model to train. Each image is either of a certain class or not so I want to set up a binary classification dataset/model using PyTorch. I had some questions:

should labels be float or long?
what shape should my labels be?
I am using a resnet18 class from torchvision model, should my final softmax layer have one or two outputs?
what shapes should my target be, during training, if my batch size is 200?
what shape should my outputs be?

Thanks in advance
Quote
Delete


title: Pytorch -> [Onnx -> tensorflow] -> tflite generates lots of redundant conv2d operators

body:
I am converting efficientnet from onnx to tensorflow for further conversion to tflite. The conversion from onnx to tensorflow yields strange results
Onnx has 1 conv2d operator

And after converting to tensorflow and later tflite model I get this stack of convs(All of them couldn't fit in the picture, there are 32 of the in total, as the number of kernels in onnx)

I suspect the issue is with onnx-tf conversion. It seems like the conv2d output are split into 32 separate outputs, then they are processed individually on separate conv2d operator and then concatenated together.
There are 2 connected issues on their github.
First has a fix for depthwise convs but it can't be applied in my case
https://github.com/onnx/onnx-tensorflow/issues/473
https://github.com/onnx/onnx-tensorflow/issues/754
Looking at the source of https://github.com/onnx/onnx-tensorflow/blob/master/onnx_tf/handlers/backend/conv_mixin.py
They do indeed split the output produce separate convolutions and later concatenate them.
Can this split to multiple convolutions be avoided?


title: Pytorch-tutorial: Strange input argument in class definition

body:
I'm reading through some pytorch tutorials. Below is the definition of a residual block. However in the forward method each function handle only takes one argument out while in the __init__ function these functions have different number of input arguments:

# Residual Block
class ResidualBlock(nn.Module):
    def __init__(self, in_channels, out_channels, stride=1, downsample=None):
        super(ResidualBlock, self).__init__()
        self.conv1 = conv3x3(in_channels, out_channels, stride)
        self.bn1 = nn.BatchNorm2d(out_channels)
        self.relu = nn.ReLU(inplace=True)
        self.conv2 = conv3x3(out_channels, out_channels)
        self.bn2 = nn.BatchNorm2d(out_channels)
        self.downsample = downsample

    def forward(self, x):
        residual = x
        out = self.conv1(x)
        out = self.bn1(out)
        out = self.relu(out)
        out = self.conv2(out)
        out = self.bn2(out)
        if self.downsample:
            residual = self.downsample(x)
        out += residual
        out = self.relu(out)
        return out

Does anyone know how this works?
Is it a standard python class inheritance feature or is this specific to pytorch?


title: Strange behaviour in PyTorch

body:
I started learning pyTorch recently.I moved from Torch community as I like Python.

I encountered this strange behaviour in pyTorch.Any insights as why this happened would be appreciated.

x=torch.Tensor(10,1).fill_(1)
y=torch.Tensor(10).fill_(2)
print(x.size())     #output is torch.Size([10, 1])
print(y.size())     #output is  torch.Size([10])
z=x+y
z.size()
z.size()            #output is torch.Size([10, 10])

Output of z is 10 x 10 with value 3 which means it is clearly adding the Tensors(I also checked with other numbers) and then constructing  a 10 x 10 Tensor with the values after addition.

Can someone explain me why is this happening . I tried it in Torch(Lua) it did a cumulative addition and returned the Tensor I expected.


title: Testing Accuracy is less than half of Trainings Accuracy, and manual testing result is way off

body:
Firstly I have to admit that I am total beginner of PyTorch and CNN image classification.

I am making an app to classify cat breeds.

The image sets I gathered have around 300-500 per breed, with a total of 62 breeds, plus one other set which represents non-cats which contains 600 samples. I have split the samples into training and testing respectively at 4:1 ratio.

The training results are quite disappointing. The training accuracy can reach as much as 90% but the testing results are only 39%.

Here are the hyperparameters:

LR is 0.1, momentum is 0.1, and batch_size is 128, wideresnet is using 40 layers of widen factor of 10.

Please see the source code at:

https://github.com/silver-xu/wideresnet-trial

I have searched all over the internet and pretty much 90% of the articles are about pre-compiled datasets like cifar or MNIST. As a result lots of the codes which I found only is optimised for one type of dataset only.

Thanks for all the help! Critics are also Welcome!

Here is training output for Epoch 5:

Epoch: [5][0/170]       Time 0.237 (0.237)      Loss 3.3054 (3.3054)    Prec@1 13.281 (13.281)
Epoch: [5][10/170]      Time 0.229 (0.228)      Loss 3.2665 (3.3118)    Prec@1 14.844 (13.920)
Epoch: [5][20/170]      Time 0.227 (0.227)      Loss 3.0962 (3.2856)    Prec@1 17.969 (14.695)
Epoch: [5][30/170]      Time 0.228 (0.227)      Loss 3.3670 (3.2853)    Prec@1 10.938 (14.844)
Epoch: [5][40/170]      Time 0.229 (0.227)      Loss 3.3259 (3.2917)    Prec@1 15.625 (15.282)
Epoch: [5][50/170]      Time 0.228 (0.227)      Loss 3.2016 (3.2931)    Prec@1 14.844 (14.859)
Epoch: [5][60/170]      Time 0.227 (0.227)      Loss 3.3739 (3.3071)    Prec@1 11.719 (14.677)
Epoch: [5][70/170]      Time 0.227 (0.227)      Loss 3.4417 (3.3042)    Prec@1 15.625 (14.833)
Epoch: [5][80/170]      Time 0.226 (0.227)      Loss 3.2507 (3.2996)    Prec@1 10.938 (14.911)
Epoch: [5][90/170]      Time 0.224 (0.227)      Loss 3.2627 (3.2978)    Prec@1 14.844 (15.093)
Epoch: [5][100/170]     Time 0.226 (0.227)      Loss 3.3668 (3.2946)    Prec@1 14.062 (15.060)
Epoch: [5][110/170]     Time 0.225 (0.227)      Loss 3.2839 (3.2915)    Prec@1 10.156 (14.921)
Epoch: [5][120/170]     Time 0.227 (0.227)      Loss 3.3308 (3.2906)    Prec@1 11.719 (14.837)
Epoch: [5][130/170]     Time 0.224 (0.227)      Loss 3.1656 (3.2885)    Prec@1 21.875 (14.909)
Epoch: [5][140/170]     Time 0.226 (0.227)      Loss 3.2521 (3.2851)    Prec@1 20.312 (14.966)
Epoch: [5][150/170]     Time 0.227 (0.227)      Loss 3.1261 (3.2825)    Prec@1 14.844 (14.989)
Epoch: [5][160/170]     Time 0.227 (0.227)      Loss 3.4400 (3.2802)    Prec@1 10.938 (15.018)
Test: [0/43]    Time 0.262 (0.262)      Loss 3.6978 (3.6978)    Prec@1 8.594 (8.594)
Test: [10/43]   Time 0.074 (0.091)      Loss 3.3584 (3.3736)    Prec@1 17.188 (13.139)
Test: [20/43]   Time 0.074 (0.083)      Loss 3.3834 (3.4058)    Prec@1 12.500 (12.537)
Test: [30/43]   Time 0.074 (0.080)      Loss 3.4457 (3.3994)    Prec@1 14.844 (12.802)
Test: [40/43]   Time 0.074 (0.079)      Loss 3.2851 (3.3946)    Prec@1 16.406 (13.281)
 * Prec@1 13.130