title: Moving member tensors with module.to() in PyTorch

body:
I am building a Variational Autoencoder (VAE) in PyTorch and have a problem writing device agnostic code. The Autoencoder is a child of nn.Module with an encoder and decoder network, which are too. All weights of the network can be moved from one device to another by calling net.to(device).

The problem I have is with the reparametrization trick:

encoding = mu + noise * sigma

The noise is a tensor of the same size as mu and sigma and saved as a member variable of the autoencoder module. It is initialized in the constructor and resampled in-place each training step. I do it that way to avoid constructing a new noise tensor each step and pushing it to the desired device. Additionally, I want to fix the noise in the evaluation. Here is the code:

class VariationalGenerator(nn.Module):
    def __init__(self, input_nc, output_nc):
        super(VariationalGenerator, self).__init__()

        self.input_nc = input_nc
        self.output_nc = output_nc
        embedding_size = 128

        self._train_noise = torch.randn(batch_size, embedding_size)
        self._eval_noise = torch.randn(1, embedding_size)
        self.noise = self._train_noise

        # Create encoder
        self.encoder = Encoder(input_nc, embedding_size)
        # Create decoder
        self.decoder = Decoder(output_nc, embedding_size)

    def train(self, mode=True):
        super(VariationalGenerator, self).train(mode)
        self.noise = self._train_noise

    def eval(self):
        super(VariationalGenerator, self).eval()
        self.noise = self._eval_noise

    def forward(self, inputs):
        # Calculate parameters of embedding space
        mu, log_sigma = self.encoder.forward(inputs)
        # Resample noise if training
        if self.training:
            self.noise.normal_()
        # Reparametrize noise to embedding space
        inputs = mu + self.noise * torch.exp(0.5 * log_sigma)
        # Decode to image
        inputs = self.decoder(inputs)

        return inputs, mu, log_sigma

When I now move the autoencoder to the GPU with net.to('cuda:0') I get an error in forwarding because the noise tensor is not moved.

I don't want to add a device parameter to the constructor, because then it is still not possible to move it to another device later. I also tried to wrap the noise into nn.Parameter so that it is affected by net.to(), but that gives an error from the optimizer, as the noise is flagged as requires_grad=False.

Anyone has a solution to move all of the modules with net.to()?


I am trying to apply reiforcement learning mechanism to classification tasks.
I know it is useless thing to do because deep learning can overperform rl in the tasks. Anyway in research purposes I am doing.

I reward agent if he's correct positive 1 or not negative -1
and computate loss FUNC with predicted_action(predicted_class) and reward.

But I get an error:

  element 0 of tensors does not require grad and does not have a grad_fn

 # creating model
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()

        self.pipe = nn.Sequential(nn.Linear(9, 120),
                                 nn.ReLU(),
                                 nn.Linear(120, 64),
                                 nn.ReLU(),
                                 nn.Linear(64,2),
                                 nn.Softmax()
                                 )

    def forward(self, x):
        return self.pipe(x)

def env_step(action, label, size):
    total_reward = []

    for i in range(size):
        reward = 0

        if action[i] == label[i]:
            total_reward.append(reward+1)
            continue
        else:
            total_reward.append(reward-1)
            continue

    return total_reward

if __name__=='__main__':
    epoch_size = 100
    net = Net()
    criterion = nn.MSELoss()
    optimizer = optim.Adam(params=net.parameters(), lr=0.01)

    total_loss = deque(maxlen = 50)

    for epoch in range(epoch_size):
        batch_index = 0
        for i in range(13):
            # batch sample
            batch_xs = torch.FloatTensor(train_state[batch_index: batch_index+50])   # make tensor
            batch_ys = torch.from_numpy(train_label[batch_index: batch_index+50]).type('torch.LongTensor')  # make tensor

            # action_prob; e.g classification prob
            actions_prob = net(batch_xs)
            #print(actions_prob)
            action = torch.argmax(actions_prob, dim=1).unsqueeze(1)
            #print(action)
            reward = np.array(env_step(action, batch_ys, 50))
            #print(reward)

            reward = torch.from_numpy(reward).unsqueeze(1).type('torch.FloatTensor')
            #print(reward)
            action = action.type('torch.FloatTensor')

            optimizer.zero_grad()
            loss = criterion(action, reward)
            loss.backward()
            optimizer.step()

            batch_index += 50


title: Pytorch, Unable to get repr for

body:
I'm implementing some RL in PyTorch and had to write my own mse_loss function (which I found on Stackoverflow ;) ).
The loss function is:

def mse_loss(input_, target_):
    return torch.sum(
        (input_ - target_) * (input_ - target_)) / input_.data.nelement()

Now, in my training loop, the first input is something like:

tensor([-1.7610e+10]), tensor([-6.5097e+10])

With this input I'll get the error:

Unable to get repr for <class 'torch.Tensor'>

Computing a = (input_ - target_) works fine, while b = a * a respectively b = torch.pow(a, 2) will fail with the error metioned above.

Does anyone know a fix for this?

Thanks a lot!

Update:
I just tried using torch.nn.functional.mse_loss which will result in the same error..


I am trying to run a pytorch code with jupyter notebook and I got this error

RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 2.00 GiB total capacity; 1.23 GiB already allocated; 18.83 MiB free; 1.25 GiB reserved in total by PyTorch)

I had already find answer. and most of all say just reduce the batch size.
I have tried reduce the batch size from 20 to 10 to 2 and 1. Right now still can't run the code

My GPU is Quadro p620


title: Adam optimizer with warmup on PyTorch

body:
In the paper Attention is all you need, under section 5.3, the authors suggested to increase the learning rate linearly and then decrease proportionally to the inverse square root of steps.

How do we implement this in PyTorch with Adam optimizer? Preferably without additional packages.


title: Implementing word dropout in pytorch

body:
I want to add word dropout to my network so that I can have sufficient training examples for training the embedding of the "unk" token. As far as I'm aware, this is standard practice. Let's assume the index of the unk token is 0, and the index for padding is 1 (we can switch them if that's more convenient).

This is a simple CNN network which implements word dropout the way I would have expected it to work:

class Classifier(nn.Module):
    def __init__(self, params):
        super(Classifier, self).__init__()
        self.params = params
        self.word_dropout = nn.Dropout(params["word_dropout"])
        self.pad = torch.nn.ConstantPad1d(max(params["window_sizes"])-1, 1)
        self.embedding = nn.Embedding(params["vocab_size"], params["word_dim"], padding_idx=1)
        self.convs = nn.ModuleList([nn.Conv1d(1, params["feature_num"], params["word_dim"] * window_size, stride=params["word_dim"], bias=False) for window_size in params["window_sizes"]])
        self.dropout = nn.Dropout(params["dropout"])
        self.fc = nn.Linear(params["feature_num"] * len(params["window_sizes"]), params["num_classes"])

    def forward(self, x, l):
        x = self.word_dropout(x)
        x = self.pad(x)
        embedded_x = self.embedding(x)
        embedded_x = embedded_x.view(-1, 1, x.size()[1] * self.params["word_dim"]) # [batch_size, 1, seq_len * word_dim]
        features = [F.relu(conv(embedded_x)) for conv in self.convs]
        pooled = [F.max_pool1d(feat, feat.size()[2]).view(-1, params["feature_num"]) for feat in features]
        pooled = torch.cat(pooled, 1)
        pooled = self.dropout(pooled)
        logit = self.fc(pooled)
        return logit

Don't mind the padding - pytorch doesn't have an easy way of using non zero padding in CNNs, much less trainable non-zero padding, so I'm doing it manually. Dropout also doesn't allow me to use non zero dropout, and I want to separate the padding token from the unk token. I'm keeping it in my example because it's the reason for this question's existence.

This doesn't work because dropout wants Float Tensors so that it can scale them properly, while my input is Long Tensors that don't need to be scaled.

Is there an easy way of doing this in pytorch? I essentially want to use LongTensor-friendly dropout (bonus: better if it will let me specify a dropout constant that isn't 0, so that I could use zero padding).


title: Different behavior with torch.mean(data, 0) and torch.mean(data) for vector

body:
I was surprised to find different behavior with torch.mean(data, 0) and torch.mean(data), where "data" — is 1D-tensor (vector, not a matrix or something else):

from torchvision import datasets
import torch

path = './MNIST_data'

data = datasets.MNIST(path, train=True, download=True).train_data.view(-1).float()

print(torch.mean(data))
print(torch.mean(data, 0))

Results after execution:

tensor(33.3184)
tensor(33.4961)

Could anyone suppose what is going on? I assumed that the results should be the same.


title: pytorch Rnn.py RuntimeError: CUDNN_STATUS_INTERNAL_ERROR

body:
I'm getting an CUDNN_STATUS_INTERNAL_ERROR error like below.

python train_v2.py

Traceback (most recent call last):
  File "train_v2.py", line 113, in <module>
    main()
  File "train_v2.py", line 74, in main
    model.cuda()
  File "/home/ahkim/Desktop/squad_vteam/src/model.py", line 234, in cuda
    self.network.cuda()
  File "/home/ahkim/anaconda3/envs/san/lib/python3.6/site-packages/torch/nn/modules/module.py", line 249, in cuda
    return self._apply(lambda t: t.cuda(device))
  File "/home/ahkim/anaconda3/envs/san/lib/python3.6/site-packages/torch/nn/modules/module.py", line 176, in _apply
    module._apply(fn)
  File "/home/ahkim/anaconda3/envs/san/lib/python3.6/site-packages/torch/nn/modules/module.py", line 176, in _apply
    module._apply(fn)
  File "/home/ahkim/anaconda3/envs/san/lib/python3.6/site-packages/torch/nn/modules/module.py", line 176, in _apply
    module._apply(fn)
  File "/home/ahkim/anaconda3/envs/san/lib/python3.6/site-packages/torch/nn/modules/rnn.py", line 112, in _apply
    self.flatten_parameters()
  File "/home/ahkim/anaconda3/envs/san/lib/python3.6/site-packages/torch/nn/modules/rnn.py", line 105, in flatten_parameters
    self.batch_first, bool(self.bidirectional))
RuntimeError: CUDNN_STATUS_INTERNAL_ERROR

What should I try to resolve this issue?
I tried deleting .nv but no success.

nvidia-smi

Wed Aug  8 10:56:29 2018
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 390.67                 Driver Version: 390.67                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX TIT...  Off  | 00000000:04:00.0 Off |                  N/A |
| 22%   21C    P8    15W / 250W |    125MiB / 12212MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX TIT...  Off  | 00000000:05:00.0 Off |                  N/A |
| 22%   24C    P8    14W / 250W |     11MiB / 12212MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  GeForce GTX TIT...  Off  | 00000000:08:00.0 Off |                  N/A |
| 22%   23C    P8    14W / 250W |     11MiB / 12212MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  GeForce GTX TIT...  Off  | 00000000:09:00.0 Off |                  N/A |
| 22%   23C    P8    15W / 250W |     11MiB / 12212MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   4  GeForce GTX TIT...  Off  | 00000000:85:00.0 Off |                  N/A |
| 22%   24C    P8    14W / 250W |     11MiB / 12212MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   5  GeForce GTX TIT...  Off  | 00000000:86:00.0 Off |                  N/A |
| 22%   23C    P8    15W / 250W |     11MiB / 12212MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   6  GeForce GTX TIT...  Off  | 00000000:89:00.0 Off |                  N/A |
| 22%   21C    P8    15W / 250W |     11MiB / 12212MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   7  GeForce GTX TIT...  Off  | 00000000:8A:00.0 Off |                  N/A |
| 22%   23C    P8    15W / 250W |     11MiB / 12212MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      1603      C   /usr/bin/python                              114MiB |
+-----------------------------------------------------------------------------+

Update:

The same code runs without error using Nvidia
Driver Version: 396.26 (cuda V9.1.85. torch.backends.cudnn.version():
7102). I'm getting an error using Driver Version: 390.67 (cuda V9.1.85. torch.backends.cudnn.version(): 7102)


title: Freezing layers in pre-trained bert model

body: How to freeze the last two layers in the above pre-trained model (dropout and classifier layers)? So that when the model is run, I will get a dense layer as output.


title: How to fix "“TypeError: img should be PIL Image. Got ”?

body:
I am a beginner and I am learning to code an image classifier. My goal is to create a predict function.

Any suggestion to fix it?

In this project, I want to use the predict function to recognize different flower species. So I could check their labels later.

Attempt to fix: I have already used the unsqueeze_(0) method and changing from numpy to torch method . I usually get the same error message shown below:

  TypeError: img should be PIL

Code:

    # Imports here
    import pandas as pd
    import numpy as np

    import torch
    from torch import nn
    from torchvision import datasets, transforms, models
    import torchvision.models as models
    import torch.nn.functional as F
    import torchvision.transforms.functional as F
    from torch import optim
    import json

    from collections import OrderedDict
    import numpy as np
    import matplotlib.pyplot as plt
    import seaborn as sns
    %matplotlib inline
    from PIL import Image

    def process_image(image):
     #Scales, crops, and normalizes a PIL image for a PyTorch model,
            #returns an Numpy array
        # Process a PIL image for use in a PyTorch model
        process = transforms.Compose([
            transforms.Resize(256),
            transforms.CenterCrop(224),
            transforms.ToTensor(),
            transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                 std=[0.229, 0.224, 0.225])
        ])
        image = process(image)
        return image

    # Predict
    #Predict the class (or classes) of an image using a trained deep learning model.
    def predict(image, model, topk=5):

        img = process_image(image)
        img = img.unsqueeze(0)

        output = model.forward(img)
        probs, labels = torch.topk(output, topk)
        probs = probs.exp()

        # Reverse the dict
        idx_to_class = {val: key for key, val in model.class_to_idx.items()}
        # Get the correct indices
        top_classes = [idx_to_class[each] for each in classes]

        return labels, probs

    #Passing
    probs, classes = predict(image, model)
    print(probs)
    print(classes)

Error:

TypeError                                 Traceback (most recent call last)
<ipython-input-92-b49fdcab5791> in <module>()
----> 1 probs, classes = predict(image, model)
     2 print(probs)
     3 print(classes)

<ipython-input-91-05809355bfe0> in predict(image, model, topk)
     2     ‘’' Predict the class (or classes) of an image using a trained deep learning model.
     3     ‘’'
----> 4     img = process_image(image)
     5     img = img.unsqueeze(0)
     6

<ipython-input-20-02663a696e34> in process_image(image)
    11                              std=[0.229, 0.224, 0.225])
    12     ])
---> 13     image = process(image)
    14     return image

/opt/conda/lib/python3.6/site-packages/torchvision-0.2.1-py3.6.egg/torchvision/transforms/transforms.py in __call__(self, img)
    47     def __call__(self, img):
    48         for t in self.transforms:
---> 49             img = t(img)
    50         return img
    51

/opt/conda/lib/python3.6/site-packages/torchvision-0.2.1-py3.6.egg/torchvision/transforms/transforms.py in __call__(self, img)
   173             PIL Image: Rescaled image.
   174         “”"
--> 175         return F.resize(img, self.size, self.interpolation)
   176
   177     def __repr__(self):

/opt/conda/lib/python3.6/site-packages/torchvision-0.2.1-py3.6.egg/torchvision/transforms/functional.py in resize(img, size, interpolation)
   187     “”"
   188     if not _is_pil_image(img):
--> 189         raise TypeError(‘img should be PIL Image. Got {}’.format(type(img)))
   190     if not (isinstance(size, int) or (isinstance(size, collections.Iterable) and len(size) == 2)):
   191         raise TypeError(‘Got inappropriate size arg: {}’.format(size))

TypeError: img should be PIL Image. Got <class ‘str’>

All I want is to get these similar result. Thank you!

    predict(image,model)
    print(probs)
    print(classes)

    tensor([[ 0.5607,  0.3446,  0.0552,  0.0227,  0.0054]], device='cuda:0')
    tensor([[  8,   1,  31,  24,   7]], device='cuda:0')


title: Is column selection in pytorch differentiable?

body:
Is column selection in Pytorch differentiable? for eg if I want to select a single column from each row to make a new row X 1 array and then backdrop using this new array, will the backdrop work properly?

qvalues = qvalues[range(5),[0,1,0,1,0]]

if element selection is done as shown above from a 5*2 tensor?


title: No module named 'cupy' on Google Colab

body:
I'm working on the FastPhotoStyle project:
https://github.com/NVIDIA/FastPhotoStyle

and I follow the steps of its tutorial:
https://github.com/NVIDIA/FastPhotoStyle/blob/master/TUTORIAL.md

I'm running Example 1 on Google Colab where default environment is

CUDA 10.0
Python 3.6
Chainer 5.4.0
CuPy 5.4.0

This is how I tried on Colab Notebook:

https://colab.research.google.com/drive/1oBgdJgXCLlUQhpwPoG1Uom3OKTzHR4BF

After running

!python3 demo.py --output_image_path results/example1.png

Here's the error message I got:

  Traceback (most recent call last):     File "demo.py", line 9, in

  import process_stylization

  File "/content/drive/FastPhotoStyle/process_stylization.py", line 14,
  in

  from smooth_filter import smooth_filter

  File "/content/drive/FastPhotoStyle/smooth_filter.py", line 327, in

  from cupy.cuda import function

  ModuleNotFoundError: No module named 'cupy'

Could someone help me with it please?


title: How can I load a partial pretrained pytorch model?

body:
I'm trying to get a pytorch model running on a sentence classification task. As I am working with medical notes I am using ClinicalBert (https://github.com/kexinhuang12345/clinicalBERT) and would like to use its pre-trained weights. Unfortunately the ClinicalBert model only classifies text into 1 binary label while I have 281 binary labels. I am therefore trying to implement this code https://github.com/kaushaltrivedi/bert-toxic-comments-multilabel/blob/master/toxic-bert-multilabel-classification.ipynb where the end classifier after bert is 281 long.

How can I load the pre-trained Bert weights from the ClinicalBert model without loading the classification weights?

Naively trying to load the weights from the pretrained ClinicalBert weights I get the following error:

size mismatch for classifier.weight: copying a param with shape torch.Size([2, 768]) from checkpoint, the shape in current model is torch.Size([281, 768]).
size mismatch for classifier.bias: copying a param with shape torch.Size([2]) from checkpoint, the shape in current model is torch.Size([281]).

I currently tried to replace the from_pretrained function from the pytorch_pretrained_bert package and pop the classifier weights and biases like this:

def from_pretrained(cls, pretrained_model_name, state_dict=None, cache_dir=None, *inputs, **kwargs):
    ...
    if state_dict is None:
        weights_path = os.path.join(serialization_dir, WEIGHTS_NAME)
        state_dict = torch.load(weights_path, map_location='cpu')
    state_dict.pop('classifier.weight')
    state_dict.pop('classifier.bias')
    old_keys = []
    new_keys = []
    ...

And I get the following error message:
INFO - modeling_diagnosis -   Weights of BertForMultiLabelSequenceClassification not initialized from pretrained model: ['classifier.weight', 'classifier.bias']

In the end I would like to load the bert embeddings from the clinicalBert pretrained weights and have the top classifier weights initialized randomly.


title: PyTorch nn.Linear layer output nan on well formed input and weights

body:
(repo for bug: https://github.com/zihualiu/pytorch_linear_bug)

I recently ran into a weird bug in Pytorch and I hope you can help me. In one of my networks I have a fully connected layer denoted as net.fc_h1. However during training, I realized this layer is outputing NaNs before activation. So I put it in pdb, hoping that it will yield me something. below is the logs:

# in network declaration:
def forward(self, obs):
    z1 = self.fc_h1(obs)
    if np.isnan(np.sum(z1.data.numpy())):
        pdb.set_trace()
    h1 = F.tanh(z1)
    ...

The NaN is indeed captured, but I realized in pdb if you ran the operation again, the result would be something salient:

(Pdb) z1.sum()
Variable containing:
nan
[torch.FloatTensor of size 1]

(Pdb) self.fc_h1(obs).sum()
Variable containing:
771.5120
[torch.FloatTensor of size 1]

When I checked to see if either my input or weights contains NaN, I get the following:
    (Pdb) self.fc_h1.weight.max()
    Variable containing:
     0.2482
    [torch.FloatTensor of size 1]

(Pdb) self.fc_h1.weight.mean()
Variable containing:
1.00000e-03 *
  1.7761
[torch.FloatTensor of size 1]

(Pdb) self.fc_h1.weight.min()
Variable containing:
-0.2504
[torch.FloatTensor of size 1]

(Pdb) obs.max()
Variable containing:
 6.9884
[torch.FloatTensor of size 1]

(Pdb) obs.min()
Variable containing:
-6.7855
[torch.FloatTensor of size 1]

(Pdb) obs.mean()
Variable containing:
1.00000e-02 *
 -1.5033
[torch.FloatTensor of size 1]
(Pdb) self.fc_h1.bias.max()
Variable containing:
 0.2482
[torch.FloatTensor of size 1]

(Pdb) self.fc_h1.bias.mean()
Variable containing:
1.00000e-03 *
  3.9104
[torch.FloatTensor of size 1]

(Pdb) self.fc_h1.bias.min()
Variable containing:
-0.2466
[torch.FloatTensor of size 1]

It seems both the input, weight and bias are all in good shape. Any insight on how a linear layer can yield NaN if everything is well formed?

EDIT: More weirdness
So I tried to run the forward pass again, interestingly, multiple forward passes gives me different results:

(Pdb) self.fc_h1(obs)
Variable containing:
 2.2321e-01 -6.2586e-01 -1.9004e-01  ...  -4.2521e-01  8.6175e-01  8.6866e-01
-7.2699e-02  7.8234e-01 -5.8862e-01  ...   2.4041e-01 -1.7577e-01  6.9928e-01
-7.2699e-02  7.8234e-01 -5.8862e-01  ...   2.4041e-01 -1.7577e-01  6.9928e-01
                ...                   ⋱                   ...
-6.4686e-02 -1.5819e+00  5.7410e-01  ...  -6.4127e-01  5.2837e-01 -1.3166e+00
 3.9214e-01  2.8727e-01 -5.5699e-01  ...  -8.3164e-01 -5.1795e-01 -3.7637e-01
-9.6061e-01  1.4780e-01  5.3614e-02  ...  -1.5042e+00  6.0759e-02 -3.6862e-01
[torch.FloatTensor of size 4096x170]

(Pdb) self.fc_h1(obs)
Variable containing:
 2.2321e-01 -6.2586e-01 -1.9004e-01  ...  -4.2521e-01  8.6175e-01  8.6866e-01
-7.2699e-02  7.8234e-01 -5.8862e-01  ...   2.4041e-01 -1.7577e-01  6.9928e-01
-7.2699e-02  7.8234e-01 -5.8862e-01  ...   2.4041e-01 -1.7577e-01  6.9928e-01
                ...                   ⋱                   ...
        nan         nan         nan  ...          nan  5.2837e-01 -1.3166e+00
        nan         nan         nan  ...          nan -5.1795e-01 -3.7637e-01
        nan         nan         nan  ...          nan  6.0759e-02 -3.6862e-01
[torch.FloatTensor of size 4096x170]

I'm not using GPU either, just CPU.


title: Pytorch hidden state LSTM

body:
Why do we need to initialize the hidden state h0 in LSTM in pytorch. As h0 will anyways be calculated and get overwritten ? Isn't it like

int a
a = 0

a = 4

Even if we do not do a=0, it should be fine..


title: Training with BatchNorm in pytorch

body:
I'm wondering if I need to do anything special when training with BatchNorm in pytorch. From my understanding the gamma and beta parameters are updated with gradients as would normally be done by an optimizer. However, the mean and variance of the batches are updated slowly using momentum.

So do we need to specify to the optimizer when the mean and variance parameters are updated, or does pytorch automatically take care of this?
Is there a way to access the mean and variance of the BN layer so that I can make sure it was changing while I trained the model.

If needed here is my model and training procedure:

def bn_drop_lin(n_in:int, n_out:int, bn:bool=True, p:float=0.):
    "Sequence of batchnorm (if `bn`), dropout (with `p`) and linear (`n_in`,`n_out`) layers followed by `actn`."
    layers = [nn.BatchNorm1d(n_in)] if bn else []
    if p != 0: layers.append(nn.Dropout(p))
    layers.append(nn.Linear(n_in, n_out))

    return nn.Sequential(*layers)

class Model(nn.Module):
    def __init__(self, i, o, h=()):
        super().__init__()

        nodes = (i,) + h + (o,)
        self.layers = nn.ModuleList([bn_drop_lin(i,o, p=0.5)
                                     for i, o in zip(nodes[:-1], nodes[1:])])

    def forward(self, x):
        x = x.view(x.shape[0], -1)
        for layer in self.layers[:-1]:
            x = F.relu(layer(x))

        return self.layers[-1](x)

Training:

for i, data in enumerate(trainloader):
    # get the inputs; data is a list of [inputs, labels]
    inputs, labels = data

    # zero the parameter gradients
    optimizer.zero_grad()

    # forward + backward + optimize
    outputs = net(inputs)
    loss = criterion(outputs, labels)
    loss.backward()
    optimizer.step()


title: computing gradients for every individual sample in a batch in PyTorch

body:
I'm trying to implement a version of differentially private stochastic gradient descent (e.g., this), which goes as follows:

Compute the gradient with respect to each point in the batch of size L, then clip each of the L gradients separately, then average them together, and then finally perform a (noisy) gradient descent step.

What is the best way to do this in pytorch?

Preferably, there would be a way to simulataneously compute the gradients for each point in the batch:

x # inputs with batch size L
y #true labels
y_output = model(x)
loss = loss_func(y_output,y) #vector of length L
loss.backward() #stores L distinct gradients in each param.grad, magically

But failing that, compute each gradient separately and then clip the norm before accumulating, but

x # inputs with batch size L
y #true labels
y_output = model(x)
loss = loss_func(y_output,y) #vector of length L
for i in range(loss.size()[0]):
    loss[i].backward(retain_graph=True)
    torch.nn.utils.clip_grad_norm(model.parameters(), clip_size)

accumulates the ith gradient, and then clips, rather than clipping before accumulating it into the gradient.  What's the best way to get around this issue?


I want to import the torch,but then the interpreter return this result,i have no idea how to deal with it

Traceback (most recent call last):
  File "D:/Programing/tool/Python/learn_ml_the_hard_way/ML/scipy1.py", line 1, in <module>
    import  torch
  File "D:\Programing\python\Anaconda3.5\lib\site-packages\torch\__init__.py", line 84, in <module>
    from torch._C import *
ModuleNotFoundError: No module named 'torch._C'


title: pytorch instance tensor not moved to gpu even with explicit cuda() call

body:
I'm working on a project where the model requires access to a tensor that i declare in the constructor init of the class (im sub-classing torch.nn.Module class)  and then i need to use this tensor in the forward() method via a simple matmul() , the model is sent to gpu via a cuda() call:

model = Model()
model.cuda()

However when i do forward-propagation of a simple input X through:

model(X) # or model.forward(X)

I get

  RuntimeError: Expected object of type torch.cuda.FloatTensor but found
  type torch.FloatTensor for argument #2 'mat2'

Indicating that the second argument of matmul(the instance tensor i declared) is on CPU and it was expected on GPU (as the rest of the model and data).

In matmul, the tensor is transposed via matrix.t()

I even tried overriding the cuda() method thorugh:

def cuda(self):
    super().cuda()
    self.matrix.cuda()

The data is already in the GPU ,meaning the following line of code was already executed:

X = X.cuda()

Also the error explcitly says argument 2 of matmul which for this case is the tensor(called matrix) not X.


title: How do i add ctc beam search decoder in crnn model (pytorch)

body: I am following the CRNN implementation of https://github.com/meijieru/crnn.pytorch, but seems like it is not using beam search for decoding the words. Can someone tell me how to add beam search decoding in the same model? At the same time in Tensorflow, there is an inbuilt tf.nn.ctc_beam_search_decoder.


title: The conflict is caused by: The user requested tensorboard==2.1.0 tensorflow 1.15.4 depends on tensorboard=1.15.0

body:
I am trying to install a package VIBE from a git repo and inistally I was installing its dependencies. The code is located here: https://github.com/mkocabas/VIBE how should I fix this?
Here's the error I got:
(vibe-env) mona@mona:~/research/VIBE$ pip install -r requirements.txt
Requirement already satisfied: numpy==1.17.5 in /home/mona/anaconda3/envs/vibe-env/lib/python3.7/site-packages (from -r requirements.txt (line 4)) (1.17.5)
Requirement already satisfied: torchvision==0.5.0 in /home/mona/anaconda3/envs/vibe-env/lib/python3.7/site-packages (from -r requirements.txt (line 19)) (0.5.0)
Collecting git+https://github.com/mattloper/chumpy.git (from -r requirements.txt (line 24))
  Cloning https://github.com/mattloper/chumpy.git to /tmp/pip-req-build-vdh2h3jw
Collecting git+https://github.com/mkocabas/yolov3-pytorch.git (from -r requirements.txt (line 25))
  Cloning https://github.com/mkocabas/yolov3-pytorch.git to /tmp/pip-req-build-ay_gkil2
Collecting git+https://github.com/mkocabas/multi-person-tracker.git (from -r requirements.txt (line 26))
  Cloning https://github.com/mkocabas/multi-person-tracker.git to /tmp/pip-req-build-l9jgk1qb
Requirement already satisfied: six>=1.11.0 in /home/mona/anaconda3/envs/vibe-env/lib/python3.7/site-packages (from chumpy==0.70->-r requirements.txt (line 24)) (1.15.0)
Collecting filterpy==1.4.5
  Using cached filterpy-1.4.5-py3-none-any.whl
Requirement already satisfied: numpy==1.17.5 in /home/mona/anaconda3/envs/vibe-env/lib/python3.7/site-packages (from -r requirements.txt (line 4)) (1.17.5)
Collecting gdown==3.6.4
  Downloading gdown-3.6.4.tar.gz (5.2 kB)
Requirement already satisfied: six>=1.11.0 in /home/mona/anaconda3/envs/vibe-env/lib/python3.7/site-packages (from chumpy==0.70->-r requirements.txt (line 24)) (1.15.0)
Collecting h5py==2.10.0
  Using cached h5py-2.10.0-cp37-cp37m-manylinux1_x86_64.whl (2.9 MB)
Requirement already satisfied: six>=1.11.0 in /home/mona/anaconda3/envs/vibe-env/lib/python3.7/site-packages (from chumpy==0.70->-r requirements.txt (line 24)) (1.15.0)
Requirement already satisfied: numpy==1.17.5 in /home/mona/anaconda3/envs/vibe-env/lib/python3.7/site-packages (from -r requirements.txt (line 4)) (1.17.5)
Collecting joblib==0.14.1
  Downloading joblib-0.14.1-py2.py3-none-any.whl (294 kB)
     |████████████████████████████████| 294 kB 5.6 MB/s
Collecting llvmlite==0.32.1
  Downloading llvmlite-0.32.1-cp37-cp37m-manylinux1_x86_64.whl (20.2 MB)
     |████████████████████████████████| 20.2 MB 14.1 MB/s
Collecting matplotlib==3.1.3
  Using cached matplotlib-3.1.3-cp37-cp37m-manylinux1_x86_64.whl (13.1 MB)
Requirement already satisfied: numpy==1.17.5 in /home/mona/anaconda3/envs/vibe-env/lib/python3.7/site-packages (from -r requirements.txt (line 4)) (1.17.5)
Collecting numba==0.47.0
  Downloading numba-0.47.0-cp37-cp37m-manylinux1_x86_64.whl (3.7 MB)
     |████████████████████████████████| 3.7 MB 33.0 MB/s
Requirement already satisfied: setuptools in /home/mona/anaconda3/envs/vibe-env/lib/python3.7/site-packages (from numba==0.47.0->-r requirements.txt (line 6)) (51.0.0.post20201207)
Requirement already satisfied: numpy==1.17.5 in /home/mona/anaconda3/envs/vibe-env/lib/python3.7/site-packages (from -r requirements.txt (line 4)) (1.17.5)
Collecting opencv-python==4.1.2.30
  Downloading opencv_python-4.1.2.30-cp37-cp37m-manylinux1_x86_64.whl (28.3 MB)
     |████████████████████████████████| 28.3 MB 29.4 MB/s
Requirement already satisfied: numpy==1.17.5 in /home/mona/anaconda3/envs/vibe-env/lib/python3.7/site-packages (from -r requirements.txt (line 4)) (1.17.5)
Collecting pillow==6.2.1
  Downloading Pillow-6.2.1-cp37-cp37m-manylinux1_x86_64.whl (2.1 MB)
     |████████████████████████████████| 2.1 MB 107.9 MB/s
Collecting progress==1.5
  Downloading progress-1.5.tar.gz (5.8 kB)
Collecting pyrender==0.1.36
  Downloading pyrender-0.1.36-py3-none-any.whl (1.2 MB)
     |████████████████████████████████| 1.2 MB 23.0 MB/s
Requirement already satisfied: numpy==1.17.5 in /home/mona/anaconda3/envs/vibe-env/lib/python3.7/site-packages (from -r requirements.txt (line 4)) (1.17.5)
Requirement already satisfied: six>=1.11.0 in /home/mona/anaconda3/envs/vibe-env/lib/python3.7/site-packages (from chumpy==0.70->-r requirements.txt (line 24)) (1.15.0)
Collecting PyYAML==5.3.1
  Using cached PyYAML-5.3.1-cp37-cp37m-linux_x86_64.whl
Collecting scikit-image==0.16.2
  Downloading scikit_image-0.16.2-cp37-cp37m-manylinux1_x86_64.whl (26.5 MB)
     |████████████████████████████████| 26.5 MB 25.7 MB/s
Collecting scikit-video==1.1.11
  Using cached scikit_video-1.1.11-py2.py3-none-any.whl (2.3 MB)
Requirement already satisfied: numpy==1.17.5 in /home/mona/anaconda3/envs/vibe-env/lib/python3.7/site-packages (from -r requirements.txt (line 4)) (1.17.5)
Collecting scipy==1.4.1
  Using cached scipy-1.4.1-cp37-cp37m-manylinux1_x86_64.whl (26.1 MB)
Requirement already satisfied: numpy==1.17.5 in /home/mona/anaconda3/envs/vibe-env/lib/python3.7/site-packages (from -r requirements.txt (line 4)) (1.17.5)
Collecting smplx==0.1.13
  Downloading smplx-0.1.13-py3-none-any.whl (26 kB)
Requirement already satisfied: torch>=1.0.1.post2 in /home/mona/anaconda3/envs/vibe-env/lib/python3.7/site-packages (from smplx==0.1.13->-r requirements.txt (line 7)) (1.4.0)
Requirement already satisfied: numpy==1.17.5 in /home/mona/anaconda3/envs/vibe-env/lib/python3.7/site-packages (from -r requirements.txt (line 4)) (1.17.5)
Collecting tensorboard==2.1.0
  Downloading tensorboard-2.1.0-py3-none-any.whl (3.8 MB)
     |████████████████████████████████| 3.8 MB 29.3 MB/s
Requirement already satisfied: setuptools in /home/mona/anaconda3/envs/vibe-env/lib/python3.7/site-packages (from numba==0.47.0->-r requirements.txt (line 6)) (51.0.0.post20201207)
Requirement already satisfied: numpy==1.17.5 in /home/mona/anaconda3/envs/vibe-env/lib/python3.7/site-packages (from -r requirements.txt (line 4)) (1.17.5)
Requirement already satisfied: six>=1.11.0 in /home/mona/anaconda3/envs/vibe-env/lib/python3.7/site-packages (from chumpy==0.70->-r requirements.txt (line 24)) (1.15.0)
Requirement already satisfied: wheel>=0.26 in /home/mona/anaconda3/envs/vibe-env/lib/python3.7/site-packages (from tensorboard==2.1.0->-r requirements.txt (line 18)) (0.36.1)
Collecting tensorflow==1.15.4
  Downloading tensorflow-1.15.4-cp37-cp37m-manylinux2010_x86_64.whl (110.5 MB)
     |████████████████████████████████| 110.5 MB 22 kB/s
Requirement already satisfied: numpy==1.17.5 in /home/mona/anaconda3/envs/vibe-env/lib/python3.7/site-packages (from -r requirements.txt (line 4)) (1.17.5)
Requirement already satisfied: six>=1.11.0 in /home/mona/anaconda3/envs/vibe-env/lib/python3.7/site-packages (from chumpy==0.70->-r requirements.txt (line 24)) (1.15.0)
Requirement already satisfied: wheel>=0.26 in /home/mona/anaconda3/envs/vibe-env/lib/python3.7/site-packages (from tensorboard==2.1.0->-r requirements.txt (line 18)) (0.36.1)
INFO: pip is looking at multiple versions of tensorboard to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of smplx to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of scipy to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of scikit-video to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of scikit-image to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of pyyaml to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of pyrender to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of progress to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of pillow to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of opencv-python to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of numpy to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of numba to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of multi-person-tracker to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of matplotlib to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of llvmlite to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of joblib to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of h5py to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of gdown to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of filterpy to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of chumpy to determine which version is compatible with other requirements. This could take a while.
ERROR: Cannot install -r requirements.txt (line 17) and tensorboard==2.1.0 because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested tensorboard==2.1.0
    tensorflow 1.15.4 depends on tensorboard<1.16.0 and >=1.15.0

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies

(vibe-env) mona@mona:~/research/VIBE$ python
Python 3.7.9 (default, Aug 31 2020, 12:42:55)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.__version__
'1.4.0'

Here are all the commands I ran before this:
(base) mona@mona:~/research/VIBE$ export CONDA_ENV_NAME=vibe-env

(base) mona@mona:~/research/VIBE$ conda create -n $CONDA_ENV_NAME python=3.7

(base) mona@mona:~/research/VIBE$ eval "$(conda shell.bash hook)"

(base) mona@mona:~/research/VIBE$ conda activate $CONDA_ENV_NAME

(vibe-env) mona@mona:~/research/VIBE$ pip install numpy==1.17.5 torch==1.4.0 torchvision==0.5.0

(vibe-env) mona@mona:~/research/VIBE$ pip install git+https://github.com/giacaglia/pytube.git --upgrade


title: Pytorch tensor - How to get the indexes by a specific tensor

body:
I have a tensor

t = torch.tensor([[1, 0, 0, 0], [0, 0, 1, 0], [0, 1, 0, 0], [1, 0, 0, 0]])

and a query tensor

q = torch.tensor([1, 0, 0, 0])

Is there a way to get the indexes of q like

indexes = t.index(q) # get back [0, 3]

in pytorch?


title: pytorch PIP and CONDA error?

body:
Guys I am new to python and deeplearning world

I tried to install pytorch using conda

I get this Error...

(base) C:\WINDOWS\system32>conda install pytorch

  `Solving environment: failed

PackagesNotFoundError: The following packages are not available from current channels:

pytorch

Current channels:

https://repo.continuum.io/pkgs/main/win-64
https://repo.continuum.io/pkgs/main/noarch
https://repo.continuum.io/pkgs/free/win-64

Couldnt post all the channels due to reputation issue on stackoverflow...

Trying Pip for installing Pytorch It just opens pytorch site after this error:

  (base) C:\WINDOWS\system32>pip install pytorch
  Collecting pytorch
    Using cached pytorch-0.1.2.tar.gz
  Building wheels for collected packages: pytorch
    Running setup.py bdist_wheel for pytorch ... error
    Complete output from command C:\ProgramData\Anaconda3\python.exe -u -c "import setuptools, tokenize;file='C:\Users\micha\AppData\Local\Temp\pip-build-t86penrg\pytorch\setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" bdist_wheel -d C:\Users\micha\AppData\Local\Temp\tmpqmo4j08upip-wheel- --python-tag cp36:
    Traceback (most recent call last):
      File "", line 1, in
      File "C:\Users\micha\AppData\Local\Temp\pip-build-t86penrg\pytorch\setup.py", line 17, in
        raise Exception(message)
    Exception: You should install pytorch from http://pytorch.org

Failed building wheel for pytorch
  Running setup.py clean for pytorch
Failed to build pytorch
Installing collected packages: pytorch
  Running setup.py install for pytorch ... error
    Complete output from command C:\ProgramData\Anaconda3\python.exe -u -c "import setuptools, tokenize;file='C:\Users\micha\AppData\Local\Temp\pip-build-t86penrg\pytorch\setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record C:\Users\micha\AppData\Local\Temp\pip-vms7q49e-record\install-record.txt --single-version-externally-managed --compile:
    Traceback (most recent call last):
      File "", line 1, in
      File "C:\Users\micha\AppData\Local\Temp\pip-build-t86penrg\pytorch\setup.py", line 13, in
        raise Exception(message)
    Exception: You should install pytorch from http://pytorch.org

----------------------------------------

Exception:
Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\lib\site-packages\pip\commands\install.py", line 342, in run
    prefix=options.prefix_path,
  File "C:\ProgramData\Anaconda3\lib\site-packages\pip\req\req_set.py", line 784, in install
    **kwargs
  File "C:\ProgramData\Anaconda3\lib\site-packages\pip\req\req_install.py", line 878, in install
    spinner=spinner,
  File "C:\ProgramData\Anaconda3\lib\site-packages\pip\utils__init__.py", line 707, in call_subprocess
    % (command_desc, proc.returncode, cwd))
pip.exceptions.InstallationError: Command "C:\ProgramData\Anaconda3\python.exe -u -c "import setuptools, tokenize;file='C:\Users\micha\AppData\Local\Temp\pip-build-t86penrg\pytorch\setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record C:\Users\micha\AppData\Local\Temp\pip-vms7q49e-record\install-record.txt --single-version-externally-managed --compile" failed with error code 1 in C:\Users\micha\AppData\Local\Temp\pip-build-t86penrg\pytorch\

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\lib\site-packages\pip\basecommand.py", line 215, in main
    status = self.run(options, args)
  File "C:\ProgramData\Anaconda3\lib\site-packages\pip\commands\install.py", line 385, in run
    requirement_set.cleanup_files()
  File "C:\ProgramData\Anaconda3\lib\site-packages\pip\req\req_set.py", line 729, in cleanup_files
    req.remove_temporary_source()
  File "C:\ProgramData\Anaconda3\lib\site-packages\pip\req\req_install.py", line 977, in remove_temporary_source
    rmtree(self.source_dir)
  File "C:\ProgramData\Anaconda3\lib\site-packages\pip_vendor\retrying.py", line 49, in wrapped_f
    return Retrying(*dargs, **dkw).call(f, *args, **kw)
  File "C:\ProgramData\Anaconda3\lib\site-packages\pip_vendor\retrying.py", line 212, in call
    raise attempt.get()
  File "C:\ProgramData\Anaconda3\lib\site-packages\pip_vendor\retrying.py", line 247, in get
    six.reraise(self.value[0], self.value[1], self.value[2])
  File "C:\ProgramData\Anaconda3\lib\site-packages\six.py", line 693, in reraise
    raise value
  File "C:\ProgramData\Anaconda3\lib\site-packages\pip_vendor\retrying.py", line 200, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "C:\ProgramData\Anaconda3\lib\site-packages\pip\utils__init__.py", line 102, in rmtree
    onerror=rmtree_errorhandler)
  File "C:\ProgramData\Anaconda3\lib\shutil.py", line 494, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\ProgramData\Anaconda3\lib\shutil.py", line 393, in _rmtree_unsafe
    onerror(os.rmdir, path, sys.exc_info())
  File "C:\ProgramData\Anaconda3\lib\site-packages\pip\utils__init__.py", line 114, in rmtree_errorhandler
    func(path)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\Users\micha\AppData\Local\Temp\pip-build-t86penrg\pytorch'

I also tried conda install -c peterjc123 pytorch=0.1.12 and soumith but I get the same error not found

Any Idea where I am going wrong

Tried other forum tips and post also reinstalled Anaconda but still the same issue


title: Is computing `loss.backward` for multiple losses performant in pytorch?

body:
I would like to calculate the gradient of my model for several loss functions.

I would like to find out if calculating successive backwards calls with retain_graph=True is cheap or expensive.

In theory I would expect that the first call should be slower than those following the first, because the computational graph does not have to be reevaluated, but just a few matrix multiplications need to be made.

In practice I found it hard to benchmark.

My code:

# Code in file nn/two_layer_net_nn.py
import torch

D_in = 40
model = torch.load('model.pytorch')
device = torch.device('cpu')
def loss1(y_pred,x):
    return (y_pred*(0.5-x.clamp(0,1))).sum()

def loss2(y_pred,x):
    return (y_pred*(1-x.clamp(0,1))).sum()

# Predict random input
x = torch.rand(1,D_in, device=device,requires_grad=True)
y_pred = model(x)

# Is this
%%timeit
loss = loss1(y_pred,x)
loss.backward(retain_graph=True)

202 µs ± 4.34 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

# Slower than this?
%%timeit
loss = loss2(y_pred,x)
loss.backward(retain_graph=True)

216 µs ± 27.1 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

# Are successive backwards calls cheap?
loss = lossX(y_pred,x)
loss.backward(retain_graph=True)

I think that %%timeit doesn't work because it will run several iterations and then average over it.

How can I measure whether successive calls to backward will be fast?
What does retain_graph=True actually mean for performance?


title: Run multiple models of an ensemble in parallel with PyTorch

body:
My neural network has the following architecture:

input -> 128x (separate fully connected layers) -> output averaging

I am using a ModuleList to hold the list of fully connected layers. Here's how it looks at this point:

class MultiHead(nn.Module):
    def __init__(self, dim_state, dim_action, hidden_size=32, nb_heads=1):
        super(MultiHead, self).__init__()

        self.networks = nn.ModuleList()
        for _ in range(nb_heads):
            network = nn.Sequential(
                nn.Linear(dim_state, hidden_size),
                nn.Tanh(),
                nn.Linear(hidden_size, dim_action)
            )
            self.networks.append(network)

        self.cuda()
        self.optimizer = optim.Adam(self.parameters())

Then, when I need to calculate the output, I use a for ... in construct to perform the forward and backward pass through all the layers:

q_values = torch.cat([net(observations) for net in self.networks])

# skipped code which ultimately computes the loss I need

self.optimizer.zero_grad()
loss.backward()
self.optimizer.step()

This works! But I am wondering if I couldn't do this more efficiently. I feel like by doing a for...in, I am actually going through each separate FC layer one by one, while I'd expect this operation could be done in parallel.


title: How to balance (oversampling) unbalanced data in PyTorch (with WeightedRandomSampler)?

body:
I have a 2-class problem and my data is highly unbalanced. I have 232550 samples from one class and 13498 from the second class. PyTorch docs and the internet tells me to use the class WeightedRandomSampler for my DataLoader.

I have tried using the WeightedRandomSampler but I keep getting errors.

    trainratio = np.bincount(trainset.labels)
    classcount = trainratio.tolist()
    train_weights = 1./torch.tensor(classcount, dtype=torch.float)
    train_sampleweights = train_weights[trainset.labels]
    train_sampler = WeightedRandomSampler(weights=train_sampleweights,
    num_samples = len(train_sampleweights))
    trainloader = DataLoader(trainset, sampler=train_sampler,
    shuffle=False)

I can not see why I am getting this error when initializing the WeightedRandomSampler class?

I have tried other similar workarounds but so far all attempts produce some error.
How should I implement this to balance my train, validation and test data?

Currently getting this error:

  train__sampleweights = train_weights[trainset.labels] ValueError: too
  many dimensions 'str'


title: Pytorch : W ParallelNative.cpp:206

body:
I'm trying to use a pre-trained template on my image set by following the tutorial right here :
https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html
Only I always get this "error" when I run my code and the console locks up :
[W ParallelNative.cpp:206] Warning: Cannot set number of intraop threads after parallel work has started or after set_num_threads call when using native parallel backend (function set_num_threads)
Thank you in advance for your help,


title: KeyError when enumerating over dataloader

body:
I'm trying to iterate over a pytorch dataloader initialized as follows:

trainDL = torch.utils.data.DataLoader(X_train,batch_size=BATCH_SIZE, shuffle=True, **kwargs)

where X_train is a pandas dataframe like this one:

So, I'm not being able to do the following statement, since I'm getting a KeyError in the 'enumerate':

for batch_idx, (data, _) in enumerate(trainDL):
    {stuff}

has anyone a clue of what's happening?

EDIT:

The error I get is:

KeyError                                  Traceback (most recent call last)
~/.local/share/virtualenvs/Pipenv-l_wD1rT4/lib/python3.6/site-packages/pandas/core/indexes/base.py in get_loc(self, key, method, tolerance)
   2896             try:
-> 2897                 return self._engine.get_loc(key)
   2898             except KeyError:

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()

KeyError: 40592

During handling of the above exception, another exception occurred:

KeyError                                  Traceback (most recent call last)
<ipython-input-63-95142e0748bb> in <module>
----> 1 for batch_idx, (data, _) in enumerate(trainDL):
      2     print(".")

~/.local/share/virtualenvs/Pipenv-l_wD1rT4/lib/python3.6/site-packages/torch/utils/data/dataloader.py in __next__(self)
    344     def __next__(self):
    345         index = self._next_index()  # may raise StopIteration
--> 346         data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
    347         if self._pin_memory:
    348             data = _utils.pin_memory.pin_memory(data)

~/.local/share/virtualenvs/Pipenv-l_wD1rT4/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py in fetch(self, possibly_batched_index)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---> 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

~/.local/share/virtualenvs/Pipenv-l_wD1rT4/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py in <listcomp>(.0)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---> 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

~/.local/share/virtualenvs/Pipenv-l_wD1rT4/lib/python3.6/site-packages/pandas/core/frame.py in __getitem__(self, key)
   2993             if self.columns.nlevels > 1:
   2994                 return self._getitem_multilevel(key)
-> 2995             indexer = self.columns.get_loc(key)
   2996             if is_integer(indexer):
   2997                 indexer = [indexer]

~/.local/share/virtualenvs/Pipenv-l_wD1rT4/lib/python3.6/site-packages/pandas/core/indexes/base.py in get_loc(self, key, method, tolerance)
   2897                 return self._engine.get_loc(key)
   2898             except KeyError:
-> 2899                 return self._engine.get_loc(self._maybe_cast_indexer(key))
   2900         indexer = self.get_indexer([key], method=method, tolerance=tolerance)
   2901         if indexer.ndim > 1 or indexer.size > 1:

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()

KeyError: 40592


title: Pytorch tensor indexing: How to gather rows by tensor containing indices

body:
I have the tensors:
ids: shape (7000,1) containing indices like [[1],[0],[2],...]
x: shape(7000,3,255)
ids tensor encodes the index of bold marked dimension of x which should be selected.
I want to gather the selected slices in a resulting vector:
result: shape (7000,255)
Background:
I have some scores (shape = (7000,3)) for each of the 3 elements and want only to select the one with the highest score. Therefore, I used the function
ids = torch.argmax(scores,1,True)

giving me the maximum ids. I already tried to do it with gather function:
result = x.gather(1,ids)

but that didn't work.


title: How to visualize my training history in pytorch?

body:
How do you guys visualize the training history of your pytorch model like in keras here.

I have a pytorch trained model and I want to see the graph of its training.
Can I do this using only matplotlib? If yes, can someone give me resources to follow.


title: Pairwise similarity matrix between a set of vectors in PyTorch

body:
Let's suppose that we have a 3D PyTorch tensor, where the first dimension represents the batch_size, as follows:

import torch
import torch.nn as nn
x = torch.randn(32, 100, 25)

That is, for each i, x[i] is a set of 100 25-dimensional vectors. I would like to compute the similarity (e.g., the cosine similarity -- but in general any such pairwise distance/similarity matrix) of these vectors for each batch item.

That is, for each x[i] I need to compute a [100, 100] matrix which will contain the pairwise similarities of the above vectors. More specifically, the (i,j)-th element of this matrix should contain the similarity (or the distance) between the i-th and the j-th row of (the 100x25) x[t], for all t=1, ..., batch_size.

If I use torch.nn.CosineSimilarity(), no matter what dim I'm using, the result is either [100, 25] (dim=0), or [32, 25] (dim=1) , where I need a tensor of size [32, 100, 100]. I would expect torch.nn.CosineSimilarity() to work this way (since, at least to me, it looks more intuitive), but it doesn't.

Could that be done using something like below?

torch.matmul(x, x.permute(0, 2, 1))

I guess that this could give a distance matrix, but what if I need an arbitrary pairwise operation? Should I build this operation using the above?

Or maybe should I repeat x in a way so I can use the built-in torch.nn.CosineSimilarity()?

Thank you.


title: Image size of 256x256 (not 299x299) fed into Inception v3 model (PyTorch) and works?

body:
I am testing out the pretrained inception v3 model on Pytorch. I fed it an image size 256x256 and also resized it up to 299x299. In both cases, the image was classified correctly.

Can someone explain why the PyTorch pretrained model can accept an image that's not 299x299?


title: How to create torch.tensor object and to update only some of its elements?

body:
Let's say I want to create torch.tensor object of size [2,3] filled with random elements, and I intend to use this matrix in the network and optimize it's values. However, I want to update only some of the the values in the matrix.

I know that it can be done for a tensor by setting up parameter requires_grad To True or False. However, the following code

z = torch.rand([2,3], requires_grad=True)
z[-1][-1].requires_grad=False

does not work as expected

RuntimeError: you can only change requires_grad flags of leaf variables. If you want to use a computed variable in a subgraph that doesn't require differentiation use var_no_grad = var.detach().

How to fix this RuntimeError? How to initialize torch tensor and then define which elements there would have requires_grad =True?

If I write code in a similar manner:

z = torch.rand([2,3], requires_grad=False)
z[-1][-1].requires_grad=True

There will be no error, but no change of the requires_grad as well.


title: Data Normalization using Pytorch

body:
I'm starting to work on the classification of images dataset, as many tutorials I followed; it starts by normalizing the data (train and test data)

My question is: if I want to normalize the data by shifting and scaling it with a factor of 0.5

What does this mean 'the factor of something x'?

I know that it will be used within .Normalize():

transform_train = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize(),
])

transform_test = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize(),
])

But I'm a little bit confused about the meaning shift and scale (maybe it's like resize?) by the factor of X

Thanks.


title: Loss is 'nan' all the time when training the neural network in PyTorch

body:
I assigned different  weight_decayfor the parameters, and the training loss and testing loss were all nan.

I printed the prediction_train,loss_train,running_loss_train,prediction_test,loss_test,and running_loss_test ,they were all nan.

And I have checked the data with numpy.any(numpy.isnan(dataset)), it returned False.

If I use optimizer = torch.optim.Adam(wnn.parameters()) rather than assigning different weight_decay for the parameters, there would be no problem.

Could you please tell me how to fix it? Here are the codes, I defined the activation function by myself. Thank you:)

class Morlet(nn.Module):
def __init__(self):
    super(Morlet,self).__init__()
def forward(self,x):
    x=(torch.cos(1.75*x))*(torch.exp(-0.5*x*x))
    return x

morlet=Morlet()

class WNN(nn.Module):
def __init__(self):
    super(WNN,self).__init__()
    self.a1=torch.nn.Parameter(torch.randn(64,requires_grad=True))
    self.b1=torch.nn.Parameter(torch.randn(64,requires_grad=True))
    self.layer1=nn.Linear(30,64,bias=False)
    self.out=nn.Linear(64,1)
def forward(self,x):
    x=self.layer1(x)
    x=(x-self.b1)/self.a1
    x=morlet(x)
    out=self.out(x)
    return out
wnn=WNN()

optimizer = torch.optim.Adam([{'params': wnn.layer1.weight, 'weight_decay':0.01},
                          {'params': wnn.out.weight, 'weight_decay':0.01},
                          {'params': wnn.out.bias, 'weight_decay':0},
                          {'params': wnn.a1, 'weight_decay':0.01},
                          {'params': wnn.b1, 'weight_decay':0.01}])
criterion = nn.MSELoss()

for epoch in range(10):
prediction_test_list=[]
running_loss_train=0
running_loss_test=0
for i,(x1,y1) in enumerate(trainloader):
    prediction_train=wnn(x1)
    #print(prediction_train)
    loss_train=criterion(prediction_train,y1)
    #print(loss_train)
    optimizer.zero_grad()
    loss_train.backward()
    optimizer.step()
    running_loss_train+=loss_train.item()
    #print(running_loss_train)
tr_loss=running_loss_train/train_set_y_array.shape[0]
for i,(x2,y2) in enumerate(testloader):
    prediction_test=wnn(x2)
    #print(prediction_test)
    loss_test=criterion(prediction_test,y2)
    #print(loss_test)
    running_loss_test+=loss_test.item()
    print(running_loss_test)
    prediction_test_list.append(prediction_test.detach().cpu())
ts_loss=running_loss_test/test_set_y_array.shape[0]

print('Epoch {} Train Loss:{}, Test Loss:{}'.format(epoch+1,tr_loss,ts_loss))

test_set_y_array_plot=test_set_y_array*(dataset.max()-dataset.min())+dataset.min()
prediction_test_np=torch.cat(prediction_test_list).numpy()
prediction_test_plot=prediction_test_np*(dataset.max()-dataset.min())+dataset.min()

plt.plot(test_set_y_array_plot.flatten(),'r-',linewidth=0.5,label='True data')
plt.plot(prediction_test_plot,'b-',linewidth=0.5,label='Predicted data')

plt.legend()
plt.show()

print('Finish training')

The output was:

Epoch 1 Train Loss:nan, Test Loss:nan

And there was only the true data on the plot, as the picture shows.


title: Multi-label, multi-class image classifier (ConvNet) with PyTorch

body:
I am trying to implement an image classifier (CNN/ConvNet) with PyTorch where I want to read my labels from a csv-file. I have 4 different classes and an image may belong to more than one class.

I have read through the PyTorch Tutorial and this Stanford tutorial and this one, but none of them cover my specific case. I have managed to build a custom function of the torch.utils.data.Dataset class which works fine for reading the labels from a csv-file for a binary classifier only though.

This is the code for the torch.utils.data.Dataset class I have so far (slightly modified from the third tutorial linked above):

import torch
import torchvision.transforms as transforms
import torch.utils.data as data
from PIL import Image
import numpy as np
import pandas as pd

class MyCustomDataset(data.Dataset):
# __init__ function is where the initial logic happens like reading a csv,
# assigning transforms etc.
def __init__(self, csv_path):
    # Transforms
    self.random_crop = transforms.RandomCrop(800)
    self.to_tensor = transforms.ToTensor()
    # Read the csv file
    self.data_info = pd.read_csv(csv_path, header=None)
    # First column contains the image paths
    self.image_arr = np.asarray(self.data_info.iloc[:, 0])
    # Second column is the labels
    self.label_arr = np.asarray(self.data_info.iloc[:, 1])
    # Calculate len
    self.data_len = len(self.data_info.index)

# __getitem__ function returns the data and labels. This function is
# called from dataloader like this
def __getitem__(self, index):
    # Get image name from the pandas df
    single_image_name = self.image_arr[index]
    # Open image
    img_as_img = Image.open(single_image_name)
    img_cropped = self.random_crop(img_as_img)
    img_as_tensor = self.to_tensor(img_cropped)

    # Get label(class) of the image based on the cropped pandas column
    single_image_label = self.label_arr[index]

    return (img_as_tensor, single_image_label)

def __len__(self):
    return self.data_len

Specifically, I am trying to read my labels from a file with the following structure:

And my specific problem is, that I can't figure out how to implement this into my Dataset class. I think I am missing the link between the (manual) assignment of the labels in the csv and how they are read by PyTorch, as I am rather new to the framework.
I'd appreciate any help on how to get this to work, or if there are actually examples covering this, a link would be highly appreciated as well!


title: Pytorch, efficient way extend a tensor by its first and last element

body:
I have a tensor in pytorch. I want to extend it on a specific dimension from the beginning and the end by k positions with the first and last elements of that dimension respectively.
Say I have the tensor with data [[0, 0, 0], [1, 1, 1], [2, 2, 2]]. Operation extend(dim, k) would change it in this way:

extend(0, 1):  [[0, 0, 0], [0, 0, 0], [0, 0, 0], [1, 1, 1], [2, 2, 2], [2, 2, 2], [2, 2, 2]]

extend(1, 1): [0, 0, 0, 0, 0], [1, 1, 1, 1, 1], [2, 2, 2, 2, 2]]

What is an efficient way to do this (compliant with tensor.requires_grad=true)


title: How to select parameters for nn.Linear layer while training a CNN?

body:
I am trying to train a CNN to classify images from the Fashion-MNIST data using Conv2d, Maxpool and Linear layers. I came across a code as mentioned below with in_features = 12*4*4 in nn.Linear layer.

Can I please get help on how to select an in_features parameter for nn.Linear layer?

class Network(nn.Module):
    def __init__(self):
        super(Network, self).__init__()
        self.conv1 = nn.Conv2d(in_channels=1, out_channels=6, kernel_size=5)
        self.conv2 = nn.Conv2d(in_channels=6, out_channels=12, kernel_size=5)

        self.fc1 = nn.Linear(in_features=12*4*4, out_features=120)
        self.fc2 = nn.Linear(in_features=120, out_features=60)
        self.out = nn.Linear(in_features=60, out_features=10)


title: pytorch geometric "Detected that PyTorch and torch_sparse were compiled with different CUDA versions" on google colab

body:
I'm new to pytorch geometric, tried to install it to my computer but failed, so I'm trying to run the code on Google Colab instead. According to this previous question (which didn't help me and I'mnot sure its the same issue):

PyTorch Geometric CUDA installation issues on Google Colab

I did:

!pip install --upgrade torch-scatter

!pip install --upgrade torch-sparse

!pip install --upgrade torch-cluster

!pip install --upgrade torch-spline-conv

!pip install torch-geometric

!pip install torch-cluster==latest+cu101 -f https://s3.eu-central-1.amazonaws.com/pytorch-geometric.com/whl/torch-1.4.0.html

!pip install torch-scatter==latest+cu101 torch-sparse==latest+cu101 torch-spline-conv==latest+cu101 -f https://s3.eu-central-1.amazonaws.com/pytorch-geometric.com/whl/torch-1.4.0.html

they print:

Successfully installed torch-cluster-1.5.4

Successfully installed torch-scatter-2.0.4 torch-sparse-0.6.1 torch-spline-conv-1.2.0

However, when I try to run

import torch_geometric.datasets as datasets

I get:

RuntimeError: Detected that PyTorch and torch_sparse were compiled with different CUDA versions. PyTorch has CUDA version 10.1 and torch_sparse has CUDA version 0.0. Please reinstall the torch_sparse that matches your PyTorch install.

Any help would be greatly appretiated.


title: Is Pytorch DataLoader Iteration order stable?

body:
Is the iteration order for a Pytorch Dataloader guaranteed to be the same (under mild conditions)?

For instance:

dataloader = DataLoader(my_dataset, batch_size=4,
                        shuffle=True, num_workers=4)
print("run 1")
for batch in dataloader:
  print(batch["index"])

print("run 2")
for batch in dataloader:
  print(batch["index"])

So far, I've tried testing it and it appears to not be fixed, same order for both runs. Is there a way to make the order the same? Thanks

edit: i have also tried doing

unlabeled_sampler = data.sampler.SubsetRandomSampler(unlabeled_indices)
unlabeled_dataloader = data.DataLoader(train_dataset,
                sampler=unlabeled_sampler, batch_size=args.batch_size, drop_last=False)

and then iterating through the dataloader twice, but the same non-determinism results.


title: How does one reset the dataloader in pytorch?

body: I was trying to reset the dataloader manually but was unable. I tried everything here https://discuss.pytorch.org/t/how-could-i-reset-dataloader-or-count-data-batch-with-iter-instead-of-epoch/22902/4 but no luck. Anyone know how to reset the data loader AND also have the suffle/randomness of the batches not be broken?


title: What is the function in TensorFlow that is equivalent to expand() in PyTorch?

body:
Let's say I have a 2 x 3 matrix and I want to create a 6 x 2 x 3 matrix where each element in the first dimension is the original 2 x 3 matrix.

In PyTorch, I can do this:

import torch
from torch.autograd import Variable
import numpy as np

x = np.array([[1, 2, 3], [4, 5, 6]])
x = Variable(torch.from_numpy(x))

# y is the desired result
y = x.unsqueeze(0).expand(6, 2, 3)

What is the equivalent way to do this in TensorFlow? I know unsqueeze() is equivalent to tf.expand_dims() but I don't TensorFlow has anything equivalent to expand(). I'm thinking of using tf.concat on a list of the 1 x 2 x 3 tensors but am not sure if this is the best way to do it.


title: pytorch+tensorboard error " AttributeError: 'Tensor' object has no attribute 'items' "

body:
Good afternoon.
I want to log the loss of the train using the tensorboard in pytorch.
I got an error there.

AttributeError: 'Tensor' object has no attribute 'items'

I want to solve this error and check the log using tensorboard.
Here I show my code.

l_mse = mseloss(img,decoder_out)
writer.add_scalars("MSE",l_mse,n_iter)

img is real image in GAN and decoder_out is Generator output.
then I have error blow.

Traceback (most recent call last):
  File "main.py", line 39, in <module>
    main()
  File "main.py", line 22, in main
    solover.train(dataloader)
  File "path to my file", line 239, in train
    writer.add_scalars("MSE",l_mse,n_iter)
  File "/~~/anaconda3/lib/python3.7/site-packages/torch/utils/tensorboard/writer.py", line 378, in add_scalars
    for tag, scalar_value in tag_scalar_dict.items():
AttributeError: 'Tensor' object has no attribute 'items'

I tried

writer.add_scalars("MSE",l_mse,n_iter).eval()
writer.add_scalars("MSE",l_mse.item(),n_iter)
writer.add_scalars("MSE",l_mse.detach().cpu().numpy(),n_iter)

but still not work well.


title: Convert 3D Tensor to 4D Tensor in Pytorch

body:
I had difficulty finding information on reshaping in PyTorch. Tensorflow is quite easy.

My tensor has shape torch.Size([3, 480, 480]).
I want to convert it to a 4D tensor with shape [1,3,480,480].
How do I do that?


title: What is loss_cls and loss_bbox and why are they always zero in training

body:
I'm trying to train a custom dataset on using faster_rcnn using the Pytorch implementation of Detectron here. I have made changes to the dataset and configuration according to the guidelines in the repo.

The training process is carried out successfully, but the loss_cls and loss_bbox values are 0 from the beginning and even though the training is completed, final output cannot be used to make an evaluation or an inference.
I would like to know what these two mean and how to get those values to change during the training. The exact model I'm using here is e2e_faster_rcnn_R-50-FPN_1x

Any help regarding this would be appreciated. I' using Ubuntu 16.04 with Python 3.6 on Anaconda, CUDA 9, cuDNN 7.


title: PyTorch - RuntimeError: Error(s) in loading state_dict for VGG:

body:
I've trained a model using PyTorch and saved a state dict file. I have loaded the pre-trained model using the code below. I am getting an error message regarding RuntimeError: Error(s) in loading state_dict for VGG:
RuntimeError: Error(s) in loading state_dict for VGG:
    Missing key(s) in state_dict: "features.0.weight", "features.0.bias", "features.2.weight", "features.2.bias", "features.5.weight", "features.5.bias", "features.7.weight", "features.7.bias", "features.10.weight", "features.10.bias", "features.12.weight", "features.12.bias", "features.14.weight", "features.14.bias", "features.17.weight", "features.17.bias", "features.19.weight", "features.19.bias", "features.21.weight", "features.21.bias", "features.24.weight", "features.24.bias", "features.26.weight", "features.26.bias", "features.28.weight", "features.28.bias", "classifier.0.weight", "classifier.0.bias", "classifier.3.weight", "classifier.3.bias", "classifier.6.weight", "classifier.6.bias".
    Unexpected key(s) in state_dict: "state_dict", "optimizer_state_dict", "globalStep", "train_paths", "test_paths".

I am following instruction available at this site: https://pytorch.org/tutorials/beginner/saving_loading_models.html#saving-loading-model-across-devices
Many Thanks
import argparse
import datetime
import glob
import os
import random
import shutil
import time
from os.path import join

import numpy as np
import pandas as pd
import torch
import torch.nn as nn
from torch.utils.data import DataLoader
from torch.utils.tensorboard import SummaryWriter
from torchvision.transforms import ToTensor
from tqdm import tqdm
import torch.optim as optim

from convnet3 import Convnet
from dataset2 import CellsDataset

from convnet3 import Convnet
from VGG import VGG
from dataset2 import CellsDataset
from torchvision import models
from Conv import Conv2d

parser = argparse.ArgumentParser('Predicting hits from pixels')
parser.add_argument('name',type=str,help='Name of experiment')
parser.add_argument('data_dir',type=str,help='Path to data directory containing images and gt.csv')
parser.add_argument('--weight_decay',type=float,default=0.0,help='Weight decay coefficient (something like 10^-5)')
parser.add_argument('--lr',type=float,default=0.0001,help='Learning rate')
args = parser.parse_args()

metadata = pd.read_csv(join(args.data_dir,'gt.csv'))
metadata.set_index('filename', inplace=True)

# create datasets:

dataset = CellsDataset(args.data_dir,transform=ToTensor(),return_filenames=True)
dataset = DataLoader(dataset,num_workers=4,pin_memory=True)
model_path = '/Users/nubstech/Documents/GitHub/CellCountingDirectCount/VGG_model_V1/checkpoints/checkpoint.pth'

class VGG(nn.Module):
    def __init__(self, pretrained=True):
        super(VGG, self).__init__()
        vgg = models.vgg16(pretrained=pretrained)
        # if pretrained:
        vgg.load_state_dict(torch.load(model_path))
        features = list(vgg.features.children())
        self.features4 = nn.Sequential(*features[0:23])

        self.de_pred = nn.Sequential(Conv2d(512, 128, 1, same_padding=True, NL='relu'),
                                     Conv2d(128, 1, 1, same_padding=True, NL='relu'))

    def forward(self, x):
        x = self.features4(x)
        x = self.de_pred(x)

        return x

model=VGG()
#model.load_state_dict(torch.load(model_path),strict=False)
model.eval()

#optimizer = torch.optim.Adam(model.parameters(),lr=args.lr,weight_decay=args.weight_decay)

for images, paths in tqdm(dataset):

    targets = torch.tensor([metadata['count'][os.path.split(path)[-1]] for path in paths]) # B
    targets = targets.float()

    # code to print training data to a csv file
    #filename=CellsDataset(args.data_dir,transform=ToTensor(),return_filenames=True)
    output = model(images) # B x 1 x 9 x 9 (analogous to a heatmap)
    preds = output.sum(dim=[1,2,3]) # predicted cell counts (vector of length B)
    print(preds)
    paths_test = np.array([paths])
    names_preds = np.hstack(paths)
    print(names_preds)
    df=pd.DataFrame({'Image_Name':names_preds, 'Target':targets.detach(), 'Prediction':preds.detach()})
    print(df)
    # save image name, targets, and predictions
    df.to_csv(r'model.csv', index=False, mode='a')

Code for saving the state dict
        torch.save({'state_dict':model.state_dict(),
                    'optimizer_state_dict':optimizer.state_dict(),
                    'globalStep':global_step,
                    'train_paths':dataset_train.files,
                    'test_paths':dataset_test.files},checkpoint_path)


title: PyTorch RuntimeError: Tensor for argument #1 'self' is on CPU, but expected them to be on GPU

body:
I'm using PyTorch for my Logistic Regression model but whenever I run the model summary I get an error

RuntimeError: Tensor for 'out' is on CPU, Tensor for argument #1 'self' is on CPU, but expected them to be on GPU (while checking arguments for addmm)

Code
# Convert data to tensors
X_train = torch.Tensor(X_train)
y_train = torch.LongTensor(y_train)
X_test = torch.Tensor(X_test)
y_test = torch.LongTensor(y_test)

class LogisticRegression(nn.Module):
    def __init__(self, input_features, num_classes):
        super(LogisticRegression, self).__init__()
        self.fc1 = nn.Linear(input_dim, num_classes)

    def forward(self, x_in, apply_softmax = False):
        y_pred = self.fc1(x_in)
        if apply_softmax:
            y_pred = F.softmax(y_pred, dim = 1)
        return y_pred

INPUT_DIM = X_train.shape[1]
NUM_CLASSES = len(y_train.unique())

model = LogisticRegression(input_features = INPUT_DIM, num_classes = NUM_CLASSES)
print(model.named_parameters)
summary(model, input_size=(INPUT_DIM,))

My way does not work as expected, how do I go about fixing the problem?


title: bool value of Tensor with more than one value is ambiguous Pytorch error

body:
I have a Neural network that i want to train but i keep getting an "bool value of Tensor with more than one value is ambiguous" Error.

Here is my network:

from torch.autograd import Variable
from torch.utils.data import Dataset, DataLoader
import torch.nn as nn
from torch.nn import functional as F
from DataSetLoader import ReplaysDataSet

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.l1 = torch.nn.Linear(12,24)
        self.l2 = torch.nn.Linear(24, 20)
        self.l3 = torch.nn.Linear(20, 16)
        self.l4 = torch.nn.Linear(16, 10)
        self.l5 = torch.nn.Linear(10, 6)
        self.l6 = torch.nn.Linear(6, 1)

    def forward(self, t):
        t = F.relu(self.l1(t))
        t = F.relu(self.l2(t))
        t = F.relu(self.l3(t))
        t = F.relu(self.l4(t))
        t = F.relu(self.l5(t))
        y_pred = F.relu(self.l6(t))

        return y_pred
    def get_num_correct(self,preds,labels):
        return preds.argmax(dim=1).eq(labels).sum()

class Test():
    model = Model()
    optimiser = torch.optim.SGD(model.parameters(), lr=0.01)

    dataset = ReplaysDataSet()
    trainLoader = DataLoader(dataset=dataset, batch_size=250, shuffle=True)
    batch = next(iter(trainLoader))
    criterion = nn.MSELoss
    for epoch in range(10):
        totalLoss = 0
        totalCorrect = 0
        for batch in trainLoader:

            data, label = batch
            prediction = model(data)
            prediction = prediction.reshape(250)
            print(prediction)
            print(label)
            loss = criterion(prediction, label)
            optimiser.zero_grad()

            loss.backward()
            optimiser.step()

            totalLoss += loss.item()
            print(totalLoss)
            totalCorrect += model.get_num_correct(prediction, label)
            print(totalCorrect)

Here is my Data loader

import torch
from torch.autograd import Variable
from torch.utils.data import Dataset, DataLoader
from torch.nn import functional as F

class ReplaysDataSet(Dataset):
    def __init__(self):
        self.xy = np.genfromtxt("dataset.csv", delimiter=',', dtype=np.float32)
        self.length = len(self.xy)
        self.x_data = torch.from_numpy(self.xy[0:,1:])
        self.y_data = torch.from_numpy(self.xy[0:,0])
        self.length = len(self.xy)
    def __getitem__(self, index):
        return self.x_data[index], self.y_data[index]
    def __len__(self):
        return self.length

And here is some data from the csv im training on

1,303,497,784,748,743,169,462,479,785,310,26,701
1,658,598,645,786,248,381,80,428,248,530,591,145
0,38,490,796,637,130,380,226,359,720,392,464,497
0,94,752,645,801,381,479,475,381,227,645,445,248
0,59,806,254,521,91,538,212,645,609,227,545,531
1,65,254,685,565,91,796,445,658,465,485,472,184
1,385,248,211,612,82,38,485,652,212,373,563,26
1,796,596,785,310,145,479,142,685,748,635,798,474
1,380,658,485,645,36,598,806,428,786,798,645,113
0,743,214,625,642,530,784,645,641,65,598,786,637

The error i get is

Traceback (most recent call last):
  File "C:/Users/tayya/PycharmProjects/untitled/NetworkFile.py", line 32, in <module>
    class Test():
  File "C:/Users/tayya/PycharmProjects/untitled/NetworkFile.py", line 50, in Test
    loss = criterion(prediction, label)
File "C:\Users\tayya\PycharmProjects\untitled\venv\lib\site-packages\torch\nn\modules\loss.py", line 428, in __init__
super(MSELoss, self).__init__(size_average, reduce, reduction)
  File "C:\Users\tayya\PycharmProjects\untitled\venv\lib\site-packages\torch\nn\modules\loss.py", line 12, in __init__
    self.reduction = _Reduction.legacy_get_string(size_average, reduce)
  File "C:\Users\tayya\PycharmProjects\untitled\venv\lib\site-packages\torch\nn\_reduction.py", line 36, in legacy_get_string
    if size_average and reduce:
RuntimeError: bool value of Tensor with more than one value is ambiguous

Any help would be greatly appreciated. I'm new to NN's so if i made a glaring mistake, my apologies.


title: pytorch model returns NANs after first round

body:
This is my first time writing a Pytorch-based CNN.  I've finally gotten the code to run to the point of producing output for the first data batch, but on the second batch produces nans.  I greatly simplified the model for debugging purposes, but it's still not working right.  The model shown here is just a few fully connected layers with a linear output.
I am guessing that the problem is the the back-propagation step, but it's unclear to me where and why.
Here is a very simplified version of the model that still produces the error:
Data loader:
batch_size = 36
device = 'cuda'
# note "rollaxis" to move channel from last to first dimension
# X_train is n input images x 70 width x 70 height x 3 channels
# Y_train is n doubles
torch_train = utils.TensorDataset(torch.from_numpy(np.rollaxis(X_train, 3, 1)).float(), torch.from_numpy(Y_train).float())
train_loader = utils.DataLoader(torch_train, batch_size=batch_size, shuffle=True)

Define and create the model:
def MyCNN(**kwargs):
    return MyCNN_model_simple(**kwargs)

# switched from Sequential() style to assist debugging
class MyCNN_model_simple(nn.Module):
    def __init__(self, **kwargs):
        super(MyCNN_model_simple, self).__init__()
        self.fc1 = FullyConnected( 3 * 70 * 70, 100)
        self.fc2 = FullyConnected( 100, 100)
        self.last = nn.Linear(100, 1)
#         self.net = nn.Sequential(
#             self.fc1,
#             self.fc2,
#             self.last,
#             nn.Flatten()
#         )
    def forward(self, x):
        print(f"x shape A: {x.shape}")
        x = torch.flatten(x, 1)
        print(f"x shape B: {x.shape}")
        x = self.fc1(x)
        print(f"x shape C: {x.shape}")
        x = self.fc2(x)
        print(f"x shape D: {x.shape}")
        x = self.last(x)
        print(f"x shape E: {x.shape}")
        x = torch.flatten(x)
        print(f"x shape F: {x.shape}")
        return x
#        return self.net(x)

class FullyConnected(nn.Module):
    def __init__(self, in_channels, out_channels, dropout=None):
        super(FullyConnected, self).__init__()
        layers = []
        layers.append(nn.Linear(in_channels, out_channels, bias=True))
        layers.append(nn.ReLU())
        if dropout != None:
            layers.append(nn.Dropout(p=dropout))
        self.net = nn.Sequential(*layers)

    def forward(self, x):
        return self.net(x)

model = MyCNN()
# convert to 16-bit half-precision to save memory
model.half()
model.to(torch.device('cuda'))

Run the model:
loss_fn = nn.MSELoss()
dev = torch.device('cuda')
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3, weight_decay=1e-4)
losses = []
max_batches = 2

def process_batch():
    inputs = images.half().to(dev)
    values = scores.half().to(dev)

    # clear accumulated gradients
    optimizer.zero_grad()
    # make predictions
    outputs = model(inputs)
    # calculate and save the loss
    model_out = torch.flatten(outputs)
    print(f"Outputs: {model_out}")
    loss = loss_fn(model_out.half(), torch.flatten(values))
    losses.append( loss.item() )
    # backpropogate the loss
    loss.backward()
    # adjust parameters to computed gradients
    optimizer.step()

model.train()
i = 0
for images, scores in train_loader:
    process_batch()
    i += 1
    if i > max_batches: break

Stdout:
x shape A: torch.Size([36, 3, 70, 70])
x shape B: torch.Size([36, 9800])
x shape C: torch.Size([36, 100])
x shape D: torch.Size([36, 100])
x shape E: torch.Size([36, 1])
x shape F: torch.Size([36])
Outputs: tensor([0.0406, 0.0367, 0.0446, 0.0529, 0.0406, 0.0391, 0.0397, 0.0391, 0.0415,
        0.0443, 0.0410, 0.0406, 0.0349, 0.0396, 0.0368, 0.0401, 0.0343, 0.0419,
        0.0428, 0.0385, 0.0345, 0.0431, 0.0287, 0.0328, 0.0309, 0.0416, 0.0473,
        0.0352, 0.0422, 0.0375, 0.0428, 0.0345, 0.0368, 0.0319, 0.0365, 0.0382],
       device='cuda:0', dtype=torch.float16, grad_fn=<AsStridedBackward>)

x shape A: torch.Size([36, 3, 70, 70])
x shape B: torch.Size([36, 9800])
x shape C: torch.Size([36, 100])
x shape D: torch.Size([36, 100])
x shape E: torch.Size([36, 1])
x shape F: torch.Size([36])
Outputs: tensor([nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan],
       device='cuda:0', dtype=torch.float16, grad_fn=<AsStridedBackward>)

x shape A: torch.Size([36, 3, 70, 70])
x shape B: torch.Size([36, 9800])
x shape C: torch.Size([36, 100])
x shape D: torch.Size([36, 100])
x shape E: torch.Size([36, 1])
x shape F: torch.Size([36])
Outputs: tensor([nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan],
       device='cuda:0', dtype=torch.float16, grad_fn=<AsStridedBackward>)

You can see the nans that are coming out of the model starting with the second batch.  Is there anything obviously wrong that I'm doing?  If anyone has tips on best practices for debugging pytorch module runs that I can use to track down the problem, that would be very helpful.
Thanks.


title: PyTorch: Dataloader for time series task

body:
I have a Pandas dataframe with n rows and k columns loaded into memory. I would like to get batches for a forecasting task where the first training example of a batch should have shape (q, k) with q referring to the number of rows from the original dataframe (e.g. 0:128). The next example should be (128:256, k) and so on. So, ultimately, one batch should have the shape (32, q, k) with 32 corresponding to the batch size.
Since TensorDataset from data_utils does not work here, I am wondering what the best way would be. I tried to use np.array_split() to get as first dimension the number of possible splits of q values in order to write a custom DataLoader but then reshaping is not guaranteed to work since not all arrays have the same shape.
Here is a minimal example to make it more clear. In this case, batch size is 3 and q is 2:
import pandas as pd
import numpy as np
df = pd.DataFrame(data=np.arange(0,30).reshape(10,3),columns=['A','B','C'])

The dataset:
    A   B   C
0   0   1   2
1   3   4   5
2   6   7   8
3   9   10  11
4   12  13  14
5   15  16  17
6   18  19  20
7   21  22  23
8   24  25  26
9   27  28  29

The first batch in this case should have the shape (3,2,3) and look like:
array([[[ 0.,  1.,  2.],
        [ 3.,  4.,  5.]],

       [[ 3.,  4.,  5.],
        [ 6.,  7.,  8.]],

       [[ 6.,  7.,  8.],
        [ 9., 10., 11.]]])


title: Unable to create a tensor using torch.Tensor

body:
i was trying to create a tensor as below.

import torch
t = torch.tensor(2,3)

i got the following error.

  TypeError                                 Traceback (most recent call
  last)  in ()
  ----> 1 a=torch.tensor(2,3)

  TypeError: tensor() takes 1 positional argument but 2 were given

so, i tried the following

import torch
t = torch.Tensor(2,3)
# No error while creating the tensor
# When i print i get an error
print(t)

i get the following error

  RuntimeError                              Traceback (most recent call
  last)  in ()
  ----> 1 print(a)

  D:\softwares\anaconda\lib\site-packages\torch\tensor.py in
  repr(self)
       55         # characters to replace unicode characters with.
       56         if sys.version_info > (3,):
  ---> 57             return torch._tensor_str._str(self)
       58         else:
       59             if hasattr(sys.stdout, 'encoding'):

  D:\softwares\anaconda\lib\site-packages\torch_tensor_str.py in
  _str(self)
      216             suffix = ', dtype=' + str(self.dtype) + suffix
      217
  --> 218         fmt, scale, sz = _number_format(self)
      219         if scale != 1:
      220             prefix = prefix + SCALE_FORMAT.format(scale) + ' ' * indent

  D:\softwares\anaconda\lib\site-packages\torch_tensor_str.py in
  _number_format(tensor, min_sz)
       94     # TODO: use fmod?
       95     for value in tensor:
  ---> 96         if value != math.ceil(value.item()):
       97             int_mode = False
       98             break

  RuntimeError: Overflow when unpacking long

But, according to This SO Post, he was able to create a tensor. Am i missing something here. Also, why was i able to create a tensor with Tensor(capital T) and not with tensor(small t)


title: Use torch.eq() only for some value in Pytorch

body:
Is there a way to use torch.eq() or a similar function to compute element-based equality but only for some elements?
Let's say I need to know how many 1s are equal in the two tensors but I don't care about other numbers.

Any idea how to do this?


title: pytorch - How to Save and load model from DistributedDataParallel learning

body:
I'm new to the Pytorch DstributedDataParallel(), but I found that most of the tutorials save the local rank 0 model during training. Which means if I get 3 machine with 4 GPU on each of them, at the final I'll get 3 model that save from each machine.

For example in pytorch ImageNet tutorial on line 252:

if not args.multiprocessing_distributed or (args.multiprocessing_distributed
                and args.rank % ngpus_per_node == 0):
            save_checkpoint({...})

They save the model if rank % ngpus_per_node == 0.

To the best of my knowledge, DistributedDataParallel() will automatic do all reduce to the loss on the backend, without doing any further job, every process can sync the loss automatically base on that.
All the model on each process will only have slightly different at the end of the process. That mean we only need to save one model is enough.

So why don't we just save model on rank == 0, but rank % ngpus_per_node == 0 ?

And which model should I used for if I get multiple model?

If this is the right way for saving model in distribute learning, should I merge them, used one of them, or inference the result base on all three models?

Please let me know if I'm wrong.


title: how to serve pytorch or sklearn models using tensorflow serving

body:
I have found tutorials and posts which only says to serve tensorflow models using tensor serving.
In model.conf file, there is a parameter model_platform in which tensorflow or any other platform can be mentioned. But how, do we export other platform models in tensorflow way so that it can be loaded by tensorflow serving.


title: I am getting Attribute error as 'int' object has no attribute 'to'

body:
I am writing a python code in Kaggle notebook for Image Classification. In the training part, I am getting an error

AttributeError                            Traceback (most recent call last)
<ipython-input-22-052723d8ce9d> in <module>
      5     test_loss = 0.0
      6     for images,label in enumerate(train_loader):
----> 7         images,label = images.to(cuda),label.to(cuda)
      8         optimizer.zero_grad()
      9

AttributeError: 'int' object has no attribute 'to'

This is the following code, (I am giving only 2 parts, pls tell if you need more)

train_loader = torch.utils.data.DataLoader(train_data,batch_size = 128,num_workers =0,shuffle =True)
test_loader = torch.utils.data.DataLoader(test_data,batch_size = 64,num_workers =0,shuffle =False)

epoch = 10

for e in range(epoch):
    train_loss = 0.0
    test_loss = 0.0
    for images,label in enumerate(train_loader):
        images,label = images.to(cuda),label.to(cuda)
        optimizer.zero_grad()

        output = model(images)
        _,predict = torch.max(output.data, 1)
        loss = criterion(output,labels)
        loss.backward()
        optimizer.step()

        train_loss += loss.item()
        train_size += label.size(0)
        train_success += (predict==label).sum().item()

        print("train_accuracy is {.2f}".format(100*(train_success/train_size)) )


title: Add dense layer on top of Huggingface BERT model

body:
I want to add a dense layer on top of the bare BERT Model transformer outputting raw hidden-states, and then fine tune the resulting model. Specifically, I am using this base model. This is what the model should do:

Encode the sentence (a vector with 768 elements for each token of the sentence)
Keep only the first vector (related to the first token)
Add a dense layer on top of this vector, to get the desired transformation

So far, I have successfully encoded the sentences:
from sklearn.neural_network import MLPRegressor

import torch

from transformers import AutoModel, AutoTokenizer

# List of strings
sentences = [...]
# List of numbers
labels = [...]

tokenizer = AutoTokenizer.from_pretrained("dbmdz/bert-base-italian-xxl-cased")
model = AutoModel.from_pretrained("dbmdz/bert-base-italian-xxl-cased")

# 2D array, one line per sentence containing the embedding of the first token
encoded_sentences = torch.stack([model(**tokenizer(s, return_tensors='pt'))[0][0][0]
                                 for s in sentences]).detach().numpy()

regr = MLPRegressor()
regr.fit(encoded_sentences, labels)

In this way I can train a neural network by feeding it with the encoded sentences. However, this approach clearly does not fine tune the base BERT model. Can anybody help me? How can I build a model (possibly in pytorch or using the Huggingface library) that can be entirely fine tuned?


I have a question that How to get the total number of batch iteration from pytorch dataloader?
The following is a common code for training
for i, batch in enumerate(dataloader):

Then, is there any method to get the total number of iteration for the "for loop"?
In my NLP problem, the total number of iteration is different from int(n_train_samples/batch_size)...
For example, if I truncate train data only 10,000 samples and set the batch size as 1024, then 363 iteration occurs in my NLP problem.
I wonder how to get the number of total iteration in "the for-loop".
Thank you.


title: Need to change GPU option to CPU in a python pytorch based code

body:
The code basically trains the usual MNIST image dataset but it does the training on a GPU. I need to change this option so the code trains the model using my laptop computer. I need to substitute the .cuda() at the second line for the equivalent in CPU.

I know there are many examples online on how to train neural networks using the MNIST database but what is special about this code is that it does the optimization using a PID controller (commonly used in industry) and I need the code as part of my research.

net = Net(input_size, hidden_size, num_classes)
net.cuda()
net.train()
#Loss and Optimizer
criterion = nn.CrossEntropyLoss()
optimizer = PIDOptimizer(net.parameters(), lr=learning_rate, weight_decay=0.0001, momentum=0.9, I=I, D=D)
# Train the Model
for epoch in range(num_epochs):
    train_loss_log = AverageMeter()
    train_acc_log = AverageMeter()
    val_loss_log = AverageMeter()
    val_acc_log = AverageMeter()
    for i, (images, labels) in enumerate(train_loader):
        # Convert torch tensor to Variable
        images = Variable(images.view(-1, 28*28).cuda())
        labels = Variable(labels.cuda())

Would need to be able to run the code without using the .cuda() option which is for training using a GPU. Need to run it on my PC.

Here's the source code in case needed.

https://github.com/tensorboy/PIDOptimizer

Many thanks, community!