title: Is it safe to truncate torchaudio's loaded 16-bit audios to `float16` from `float32`?

body:
I have multiple WAV files with 16 bits of depth/precision. torchaudio.info(...) recognizes this, giving me:
precision = {int} 16

Yet when I use torchaudio.load(...), I get a float32 dtype for the resulting tensor.
With a tensor called audio, I know that I can do audio.half() to truncate it to 16 bits, reducing memory usage of my dataset. But is this an operation that will preserve precision of all possible original values? I'm not lowering the dtype's precision below the original audio's precision, but there may be a good reason I'm unaware of as to why torchaudio still returns float32.


title: Unflatten a tensor back to an image

body:
I am working on GANs and I want to visualize the image formed.
For this, I was trying
def show_images(image_tensor, num_images=9, size=(1, 28, 28)):
    image_unflat = image_tensor.detach().cpu.view(-1, *size)
    image_grid = make_grid(image_unflat[:num_images], nrow=3)
    plt.imshow(image_grid.permute(1, 2, 0).squeeze())
    plt.show()

but when I am trying to show_image(some_tensor), I am getting an error as
image_unflat = image_tensor.detach().cpu.view(-1, *size)
AttributeError: 'builtin_function_or_method' object has no attribute 'view'

Here, the size of some_tensor is N x 784.


title: JIT the collate function in Pytorch

body:
I need to create a DataLoader where the collator function would require to have non trivial computation, actually a double layer loop which is significantly slowing down the training process. For example, consider this toy code where I try to use numba to JIT the collate function:
import torch
import torch.utils.data

import numba as nb

class Dataset(torch.utils.data.Dataset):
    def __init__(self):
        self.A = np.zeros((100000, 300))
        self.B = np.ones((100000, 300))

    def __getitem__(self, index):
        return self.A[index], self.B[index]

    def __len__(self):
        return self.A.shape[0]

@nb.njit(cache=True)
def _collate_fn(batch):
    batch_data = np.zeros((len(batch), 300))
    for i in range(len(batch)):
        batch_data[i] = batch[i][0] + batch[i][1]

    return batch_data

and then I create the DataLoader as follows:
train_dataset = Dataset()

train_loader = torch.utils.data.DataLoader(
    train_dataset,
    batch_size=256,
    num_workers=6,
    collate_fn=_collate_fn,
    shuffle=True)

However this just gets stuck but works fine if I remove the JITing of the _collate_fn. I am not able to understand what is happening here. I don't have to stick to numba and can use anything which will help me overcome the loop inefficiencies in Python. TIA and Happy 12,021.


title: Export PyTorch model with pack_padded_sequence to ONNX fails

body:
Thank you for your time helping me out on this problem. I am trying to export my module to ONNX, but when my modules include nn.utils.rnn.pack_padded_sequence anywhere, it fails to export to ONNX, with the error:
RuntimeError: ONNX export failed: Cannot export individual pack_padded_sequence or pad_packed_sequence; these operations must occur in pairs.
My model only needs to use one of the two.. Also, aren't these inverses of each other? So using them in pairs would just give you back your original tensor, right...?
Here is the code needed to reproduce the problem. I create a sample net that uses the same features as my model in question, with a pack_padded_sequence included. I create test inputs, and run them through the model with no problem to make a test output. I then attempt to export the model to ONNX, and it fails, due to the pack_padded_sequence.
import torch
from torch import nn

class SampleNet(nn.Module):
    def __init__(self):

        super().__init__()

        self.BatchNorm1 = nn.BatchNorm1d(5)

    def forward(self, X, X_len):
        # Mask
        X_pack = torch.nn.utils.rnn.pack_padded_sequence(X, X_len, batch_first=True, enforce_sorted=False)

        # norm
        X_ln = torch.nn.utils.rnn.PackedSequence(
            data=self.BatchNorm1(X_pack.data),
            batch_sizes=X_pack.batch_sizes,
            sorted_indices=X_pack.sorted_indices,
            unsorted_indices=X_pack.unsorted_indices,
        )

        return X_ln

example_input = torch.randn(1, 30, 5, requires_grad=True)
example_input_length = torch.abs(torch.randn(1, requires_grad=True)) + 1

print("Example input: %s" % example_input)
print("Example input lengths: %s" % list(example_input.size()))

net = SampleNet() # Model instantiation
net.eval()

example_output = net(example_input, example_input_length) # Check that forward pass works as expected:
print(f"Output from Net (works): {example_output}")

torch.onnx.export(model=net,
                  args=(example_input, example_input_length),
                  example_outputs=example_output,
                  f="test_pack_unpack.onnx",
                  verbose=True,
                  input_names=["x", "x_length"],
                  output_names=["preds"]
                 )# Export to ONNX (fails)

Which gives me the error stated in the beginning of my post. To work around this, I built my own pack_padded_sequence function like so:
def my_pack_padded_sequence(input, lengths, batch_first=False, enforce_sorted=True):

    lengths = torch.as_tensor(lengths, dtype=torch.int64)

    lengths, sorted_indices = torch.sort(lengths, descending=True)
    sorted_indices = sorted_indices.to(input.device)
    batch_dim = 0 if batch_first else 1
    input = input.index_select(batch_dim, sorted_indices)

    data, batch_sizes = torch._VF._pack_padded_sequence(input, lengths, batch_first)

    unsorted_indices = torch.empty_like(sorted_indices, memory_format=torch.legacy_contiguous_format)
    unsorted_indices.scatter_(0, sorted_indices, torch.arange(0, sorted_indices.numel(), device=sorted_indices.device))

    return data, batch_sizes, sorted_indices, unsorted_indices

But, I can't seem to find what the call torch._VF._pack_padded_sequence(input, lengths, batch_first) is actually doing, I can't find it anywhere in the source code! I'm guessing it uses C functionality? Anyway, with this, ONNX still complains about the line in question.
Is there any way to fix this issue? Perhaps some way to know the behavior of torch._VF._pack_padded_sequence and write it myself? It seems as though there are open issues on this that are year+ old, but absolutely no response from torch.
Thank you so much in advance for your time!


title: Pytorch BatchNorm3d / InstanceNorm3d not working when data size (1,C,1,1,1)

body:
I'm training a neural network in PyTorch which at some point has a BatchNorm3d(C).
Normally, I'm training it with a batch size of 1, and the input of this specific level will then be of shape (1, C, 1, 1, 1).
Unfortunately, the BatchNorm then fails with the error message:

ValueError: Expected more than 1 value per channel when training, got input size torch.Size([1, 32, 1, 1, 1])

The same happens, when I use the InstanceNorm3d. It works perfectly fine when I use a batch size greater than two (i.e. the input will then be of shape (2, C, 1, 1, 1).
Does anyone know a solution to this problem? What am I missing?
The problem can be reproduced with the following snippet:
import torch
x_working = torch.ones([2,32,1,1,1])
x_not_working = torch.ones([1,32,1,1,1])
norm = torch.nn.InstanceNorm3d(32)
out_working = norm(x_working)
out_not_working=norm(x_not_working)


title: Is there a function in PyTorch for matrix left division?

body:
MATLAB has the backslash "\" operator. SciPy has "lsqr." Does PyTorch have an equivalent operator that solves systems of linear equations?
Specifically, I need to solve the matrix equation for A*X=B for A, and I need autograd to be able to backpropagate error through the operation.


title: PyTorch conv2d doesn't propagate torch.channels_last memory format

body:
When I try to use the torch.nn.conv2d operator on tensors which have the channels_last memory format, then the output does not keep this memory format.
I don't understand why, since conv2d is listed in the PyTorch wiki listing operators with channels last support.
Am I missing something?
Code to reproduce (tested with pytorch 1.6.0 and 1.7.0, on CPU, on Ubuntu 20.04):
import torch
import torch.nn.functional as F

N, C, H, W = 10, 4, 64, 128
out_channels = 2
kernel_size = (3, 3)

memory_format = torch.channels_last

tsr = torch.randn(N, C, H, W).to(memory_format=memory_format)
kernel = torch.randn(out_channels, C, *kernel_size).to(memory_format=memory_format)

conv_out = F.conv2d(tsr, kernel)

print(conv_out.is_contiguous(memory_format=memory_format)) # False


title: PyTorch and PyTorch-operator kubeflow pipelines

body:
I am trying to integrate pytorch and pytorch-operators into kubeflow pipelines and I am not able to get a good resource for both. Is this possible in the current implementation?
I understand that TFJob and PyTorchJob all run training containers on top of a kubernetes cluster but I am trying to integrate them into a pipeline.


title: PyTorch multiprocessing error: ConnectionRefusedError: [Errno 111] Connection refused

body:
When I run the following code,
import torch.multiprocessing as mp
import torch

def f(q):
    t = torch.ones([2,3])
    q.put(t)

if __name__ == '__main__':
    mp.set_start_method('spawn')
    q = mp.Queue()
    p1 = mp.Process(target=f, args=(q,))
    p2 = mp.Process(target=f, args=(q,))
    p1.start()
    p2.start()
    p1.join()
    p2.join()
    a = q.get()
    b = q.get()
    print(a, b)

I get the error:
ConnectionRefusedError: [Errno 111] Connection refused

It seems that this error is caused by torch.ones([2,3]). If in method f I put something else into q, such as a list or a value, it works fine. I've tried several approaches but none of them works for me. I really have no idea why this happens and how to deal with it. Any help would be much appreciated.


title: Sphinx exclude torch.tensor as T

body:
I'm aware of how to exclude files from Sphinx.
However, a handful of my code uses the standard pattern of:
import torch.tensor as T

T then ends up getting the torch tensor documentation in my modules. How can I avoid this?
I am using these sphinx extensions:
extensions = [
    "sphinx.ext.autodoc",
    "sphinx.ext.intersphinx",
    "sphinx_rtd_theme",
]

Here are example warnings from make html:
/Users/joseph/dev/torchsynth/torchsynth/parameter.py:docstring of torchsynth.parameter.T:1: WARNING: Inline emphasis start-string without end-string.
/Users/joseph/dev/torchsynth/torchsynth/util.py:docstring of torchsynth.util.T:1: WARNING: Inline emphasis start-string without end-string.

For example, in torchsynth/parameter.py the imports are
from typing import Optional

import torch
import torch.nn as nn
from torch import tensor as T

This is the sphinx branch in github. We just started writing our sphinx docs, so please be gentle.
Our rst files right now are just skeletons. For example, module.rst is:
torchsynth
==========

.. toctree::
   :maxdepth: 4

   torchsynth

You can see other rsts here in the branch.
Attached is a screenshot of my py-modindex.py:

You can see the floating 't' at the top left. The plus/minus button is overly scaled, I think I need to adjust the theme, but that's not my issue.


title: Split ML model in 2 models

body:
Having a very simple general trained ML model, I would like to be able to split it into 2 other models having the following structure:
InitialModel has N layers with M neurons each. InitialModel can be treated as a linear regression or classification task, so the output will be a single value.
As a concrete example, I can treat my InitialModel as follows:
class SimpleAgeRegrNet3D(nn.Module):
    def __init__(self):
        super(SimpleAgeRegrNet3D, self).__init__()
        self.encoder = nn.Sequential(
            # 64
            nn.Conv3d(1, 6, kernel_size=3),
            nn.MaxPool3d(2),
            nn.ReLU(True),
            # 31
            nn.Conv3d(6, 16, kernel_size=3),
            nn.MaxPool3d(2),
            nn.ReLU(True),
            # 14
            nn.Conv3d(16, 120, kernel_size=3),
            nn.MaxPool3d(2),
            nn.ReLU(True),
            # 6
            nn.Conv3d(120, 84, kernel_size=3),
            nn.MaxPool3d(2),
            nn.ReLU(True),
            # 2
            nn.Conv3d(84, 1, kernel_size=2),
        )

    def forward(self, x):
        return self.encoder(x)

MyInput -> InitialModel -> Output (shape=(1,))

After transformation:

MyInput -> Model1 (output shape = (M,)) -> (input shape= (M,))Model2 -> Output (shape=(1,))

Ideally, if my trained InitialModel has N layers, I would like to create Model1 which has the first N * 3/4 layers (for example) and Model2 which has the rest of the layers of the initial model. In the end, I would like to be able to reconstruct them so if I have Model1 and Person2 has Model2, I can send the output of Model1 to Person2 and then he will do the final inference.


title: Convert Conv2Dtranspose into Upsample and conv in generator layer Pytorch

body:
I'm working with GANs and I was getting artifacts and the best way to remove them is to change the generator blocks conv2dtranspose into an upsample and a convolution as suggested by many papers.
I'm trying to generate ASL hand symbols via my personal dataset

this is the block code that implies conv2dtranspose as mentioned in the screenshot above
def _block(self, in_channels, out_channels, kernel_size, stride, padding):
        return nn.Sequential(
            nn.ConvTranspose2d(
                in_channels, out_channels, kernel_size, stride, padding, bias=False,
            ),
            nn.BatchNorm2d(out_channels),
            nn.LeakyReLU(0.2),

What I'm trying to achieve is change the block so that it up samples the image and applies convolution to it producing the same output image size as the conv2dTranspose
this is what i tried
nn.Upsample(scale_factor = 2, mode='bilinear'),
            nn.ReflectionPad2d(1),
            nn.Conv2d(out_channels, int(out_channels / 2),kernel_size=3, stride=1, padding=0),
            nn.BatchNorm2d(out_channels),
            nn.LeakyReLU(0.2),

However I was getting the following error
[![enter image description here][3]][3]
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-12-4fa61054bc2c> in <module>
     81         for _ in range(CRITIC_ITERATIONS):
     82             noise = torch.randn(cur_batch_size, Z_DIM, 1, 1).to(device)
---> 83             fake = gen(noise,labels)
     84             critic_real = critic(real,labels).reshape(-1)
     85             critic_fake = critic(fake,labels).reshape(-1)

~\Anaconda3\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

<ipython-input-10-6af8005acf1f> in forward(self, x, labels)
     77         embedding=self.embed(labels).unsqueeze(2).unsqueeze(3)
     78         x=torch.cat([x,embedding],dim=1)
---> 79         return self.net(x)
     80
     81

~\Anaconda3\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

~\Anaconda3\lib\site-packages\torch\nn\modules\container.py in forward(self, input)
    115     def forward(self, input):
    116         for module in self:
--> 117             input = module(input)
    118         return input
    119

~\Anaconda3\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

~\Anaconda3\lib\site-packages\torch\nn\modules\container.py in forward(self, input)
    115     def forward(self, input):
    116         for module in self:
--> 117             input = module(input)
    118         return input
    119

~\Anaconda3\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

~\Anaconda3\lib\site-packages\torch\nn\modules\conv.py in forward(self, input)
    421
    422     def forward(self, input: Tensor) -> Tensor:
--> 423         return self._conv_forward(input, self.weight)
    424
    425 class Conv3d(_ConvNd):

~\Anaconda3\lib\site-packages\torch\nn\modules\conv.py in _conv_forward(self, input, weight)
    418                             _pair(0), self.dilation, self.groups)
    419         return F.conv2d(input, weight, self.bias, self.stride,
--> 420                         self.padding, self.dilation, self.groups)
    421
    422     def forward(self, input: Tensor) -> Tensor:

RuntimeError: Given groups=1, weight of size [1024, 2048, 1, 1], expected input[64, 200, 2, 2] to have 2048 channels, but got 200 channels instead

Also if I change the stride to 1 it gives me embedding errors
=================================================================
Layer (type:depth-idx)                   Param #
=================================================================
├─Sequential: 1-1                        --
|    └─Sequential: 2-1                   --
|    |    └─Upsample: 3-1                --
|    |    └─Conv2d: 3-2                  411,648
|    |    └─BatchNorm2d: 3-3             4,096
|    |    └─LeakyReLU: 3-4               --
|    └─Sequential: 2-2                   --
|    |    └─Upsample: 3-5                --
|    |    └─Conv2d: 3-6                  2,098,176
|    |    └─BatchNorm2d: 3-7             2,048
|    |    └─LeakyReLU: 3-8               --
|    └─Sequential: 2-3                   --
|    |    └─Upsample: 3-9                --
|    |    └─Conv2d: 3-10                 524,800
|    |    └─BatchNorm2d: 3-11            1,024
|    |    └─LeakyReLU: 3-12              --
|    └─Sequential: 2-4                   --
|    |    └─Upsample: 3-13               --
|    |    └─Conv2d: 3-14                 131,328
|    |    └─BatchNorm2d: 3-15            512
|    |    └─LeakyReLU: 3-16              --
|    └─Sequential: 2-5                   --
|    |    └─Upsample: 3-17               --
|    |    └─Conv2d: 3-18                 32,896
|    |    └─BatchNorm2d: 3-19            256
|    |    └─LeakyReLU: 3-20              --
|    └─ConvTranspose2d: 2-6              6,147
|    └─Tanh: 2-7                         --
├─Embedding: 1-2                         2,700
=================================================================
Total params: 3,215,631
Trainable params: 3,215,631
Non-trainable params: 0
=================================================================
=================================================================
Layer (type:depth-idx)                   Param #
=================================================================
├─Sequential: 1-1                        --
|    └─Conv2d: 2-1                       2,080
|    └─LeakyReLU: 2-2                    --
|    └─Sequential: 2-3                   --
|    |    └─Conv2d: 3-1                  32,768
|    |    └─InstanceNorm2d: 3-2          128
|    |    └─LeakyReLU: 3-3               --
|    └─Sequential: 2-4                   --
|    |    └─Conv2d: 3-4                  131,072
|    |    └─InstanceNorm2d: 3-5          256
|    |    └─LeakyReLU: 3-6               --
|    └─Sequential: 2-5                   --
|    |    └─Conv2d: 3-7                  524,288
|    |    └─InstanceNorm2d: 3-8          512
|    |    └─LeakyReLU: 3-9               --
|    └─Sequential: 2-6                   --
|    |    └─Conv2d: 3-10                 2,097,152
|    |    └─InstanceNorm2d: 3-11         1,024
|    |    └─LeakyReLU: 3-12              --
|    └─Conv2d: 2-7                       8,193
├─Embedding: 1-2                         442,368
=================================================================
Total params: 3,239,841
Trainable params: 3,239,841
Non-trainable params: 0
=================================================================
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-3-4fa61054bc2c> in <module>
     83             fake = gen(noise,labels)
     84             critic_real = critic(real,labels).reshape(-1)
---> 85             critic_fake = critic(fake,labels).reshape(-1)
     86             gp = gradient_penalty(critic,labels, real, fake, device=device)
     87             loss_critic = (

~\Anaconda3\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

<ipython-input-1-4154f5f318f2> in forward(self, x, labels)
     38     def forward(self, x,labels):
     39         embedding=self.embed(labels).view(labels.shape[0],1,self.img_size,self.img_size)
---> 40         x=torch.cat([x,embedding],dim=1)
     41         return self.disc(x)
     42

RuntimeError: Sizes of tensors must match except in dimension 2. Got 128 and 64 (The offending index is 0)

Could someone explain how would I achieve this for my architecture.


title: pytorch: GRU cannot update hidden_state in-place

body:
I got a problem when using pytorch to implement a GRU network:
My codes are as below:
import torch

class GRU_model(torch.nn.Module):
    def __init__(self, device):
        super(GRU_model, self).__init__()
        self.h = torch.randn((1,1,5), device=device, dtype=torch.float)

        self.GRU_1 = torch.nn.GRU(input_size=5, hidden_size=5)

    def forward(self, a):
        output, self.h = self.GRU_1(a, self.h)
        return output

if __name__ == '__main__':
    learn_rate=1e-4
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    model = GRU_model(device).to(device=device)
    optimizer = torch.optim.Adam(model.parameters(), lr=learn_rate)

    for i in range(10):
        a = torch.randn((1, 1, 5), device=device, dtype=torch.float)
        output = model(a)
        loss = (a - output).mean()

        optimizer.zero_grad()
        loss.backward(retain_graph=True)
        optimizer.step()

and I got an error like this:
Traceback (most recent call last):
  File "C:/Users/Administrator_/Desktop/Graduation_Project/MIDI_Music_style_transfer/GRU_toy_in-place_hidden_states_change/main.py", line 40, in <module>
    loss.backward(retain_graph=True)
  File "C:\Users\Administrator_\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\tensor.py", line 245, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "C:\Users\Administrator_\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\autograd\__init__.py", line 147, in backward
    allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [15, 5]] is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

I just want to update the hidden_state in the GRU after an epoch but it just doesn't work!
I'd be really appreciated if you could me help out!


title: How to check point on validation metrics in Pytorch lightning?

body:
When using the following code fragment to log metrics:
    def training_epoch_end(self, outs):
        self.__common_epoch_end_report(mode="train")

    def validation_epoch_end(self, outs):
        self.__common_epoch_end_report(mode="validation")

and
   def __common_epoch_end_report(self, mode: str):
        """
        Args:
            mode: one of ["train", "validation", "test"]
        """
        tb = self.logger.experiment

        acc, conf, f1, mcc = self.__select_metrics_by_mode(mode)

        acc_val = acc.compute()
        f1_val = f1.compute()
        tb.add_scalars('accuracy', {mode: acc_val}, global_step=self.current_epoch)
        tb.add_scalars('F1', {mode: f1_val}, global_step=self.current_epoch)

        self.log(fr'accuracy\{mode}', acc_val, on_step=False, on_epoch=True, prog_bar=True)
        self.log(fr'F1\{mode}', f1_val, on_step=False, on_epoch=True, prog_bar=True)

    def __select_metrics_by_mode(self, mode):
        if mode == "train":
            acc = self.train_acc
            f1 = self.train_f1
            mcc = self._train_mcc
            conf = self.train_confusion
        elif mode == "validation":
            acc = self.val_acc
            f1 = self.val_f1
            mcc = self._val_mcc
            conf = self.val_confusion
        elif mode == "test":
            acc = self.test_acc
            f1 = self.test_f1
            mcc = self._test_mcc
            conf = self.test_confusion
        else:
            raise ValueError("unsupported mode")
        return acc, conf, f1, mcc

I verified that __common_epoch_end_report is indeed entered both with mode='train' and with mode='validation'.
However, only the metrics logged from train are available for checkpointing:
checkpoint_callback = ModelCheckpoint(
        ...
        save_top_k=10,
        monitor=fr'F1\validation'
)
trainer = Trainer(
      ...
        callbacks=[checkpoint_callback],
)

Getting the following error:
pytorch_lightning.utilities.exceptions.MisconfigurationException: ModelCheckpoint(monitor='F1\validation') not found in the returned metrics: ['loss\\train_step', 'loss\\train_epoch', 'loss\\train', 'accuracy\\train', 'F1\\train'].

How to allow check pointing by validation metrics in Pytorch-lightning?


I am starting to learn about DQN, and I am trying to solve the FrozenLake-v0 problem from scratch by my self and using Pytorch so I will put the whole code since it's connected.
class LinearDeepQNetwork(nn.Module):
  def __init__(self,lr,n_action,input_dim):
    super(LinearDeepQNetwork,self).__init__()
    self.f1=nn.Linear(input_dim,128)
    self.f2=nn.Linear(128,n_action)
    self.optimizer=optim.Adam(self.parameters(),lr=lr)
    self.loss=nn.MSELoss()
    self.device=T.device('cuda' if T.cuda.is_available() else 'cpu')
    self.to(self.device)

  def forward(self,state):
    layer1=F.relu(self.f1(state))
    actions=self.f2(layer1)

    return actions

the second class is the agent, and the problem is in the learning function
nd class is the agent, and the problem is in the learning function
class Agent():
  def __init__(self,input_dim,n_action,lr,gamma=0.99,
               epslion=1.0,eps_dec=1e-5,eps_min=0.01):
    self.input_dim=input_dim
    self.n_action=n_action
    self.lr=lr
    self.gamma=gamma
    self.epslion=epslion
    self.eps_dec=eps_dec
    self.eps_min=eps_min
    self.action_space=[i for i in range(self.n_action)]

    self.Q=LinearDeepQNetwork(self.lr,self.n_action,self.input_dim)

  def choose_action(self,observation):
    if np.random.random()>self.epslion:
      #conveate the state into tensor
      state=T.tensor(observation).to(self.Q.device)
      actions=self.Q.forward(state)
      action=T.argmax(actions).item()
    else:
      action=np.random.choice(self.action_space)

    return action

  def decrement_epsilon(self):
    self.epslion=self.epslion-self.eps_dec \
                  if self.epslion > self.eps_min else self.eps_min

  def OH(self,x,l):
    x = T.LongTensor([[x]])
    one_hot = T.FloatTensor(1,l)
    return one_hot.zero_().scatter_(1,x,1)

  def learn(self,state,action,reward,state_):
    self.Q.optimizer.zero_grad()
    states=Variable(self.OH(state,16)).to(self.Q.device)
    actions=T.tensor(action).to(self.Q.device)
    rewards=T.tensor(reward).to(self.Q.device)
    state_s=Variable(self.OH(state_,16)).to(self.Q.device)

    q_pred=self.Q.forward(states)[actions]

    q_next=self.Q.forward(state_s).max()

    q_target=reward+self.gamma*q_next
    loss=self.Q.loss(q_target,q_pred).to(self.Q.device)
    loss.backward()
    self.Q.optimizer.step()
    self.decrement_epsilon()

now the problem when I run the following code it gives me an error in the learning phase, and it gives me this error index 1 is out of bounds for dimension 0 with size 1.
env=gym.make('FrozenLake-v0')
n_games=5000
scores=[]
eps_history=[]

agent=Agent(env.observation_space.n,env.action_space.n,0.0001)

for i in tqdm(range(n_games)):
  score=0
  done=False
  obs=env.reset()

  while not done:
    action=agent.choose_action(obs)
    obs_,reward,done,_=env.step(action)
    score+=reward

    agent.learn(obs,action,reward,obs_)
    obs=obs_
  scores.append(score)
  eps_history.append(agent.epslion)
  if i % 100 ==0:
    avg_score=np.mean(scores[-100:])
    print(f'score={score}  avg_score={avg_score} epsilon={agent.epslion} i={i}')

I think the problem is in the shape of the values between the NN and the agent class, but I can't figure out the problem.
Error traceback:
IndexError                                Traceback (most recent call last)
<ipython-input-10-2e279f658721> in <module>()
     17     score+=reward
     18
---> 19     agent.learn(obs,action,reward,obs_)
     20     obs=obs_
     21   scores.append(score)

<ipython-input-8-5359b19ec4fa> in learn(self, state, action, reward, state_)
     39     state_s=Variable(self.OH(state_,16)).to(self.Q.device)
     40
---> 41     q_pred=self.Q.forward(states)[actions]
     42
     43     q_next=self.Q.forward(state_s).max()

IndexError: index 1 is out of bounds for dimension 0 with size 1


title: How to display a video in colab, using a PyTorch tensor of RGB image arrays?

body:
I have a tensor of shape (125, 3, 128, 128):

125 frames
3 channels (RGB)
each frame 128 x 128 size.
values in the tensor are in the range [0,1].

I want to display the video of these 125  frames, using Pytorch in Google Colab. How can I do that?


title: issue with calculating accuracy

body:
i'm using Torch Metrics to try to calculate the accuracy of my model. But i'm getting this error. I tried using .to(device="cuda:0") but I got a cuda initialization error. I also tried using .cuda() but that didn't work either. I'm using PyTorch lightning with a Titan Xp GPU. Im using a mish activation function with the Movie-lens data set.
code:

# %% [markdown]
# # Data Preprocessing
#
# Before we start building and training our model, let's do some preprocessing to get the data in the required format.

# %% [code] {"_kg_hide-input":true,"_kg_hide-output":true}
import pandas as pd
import numpy as np
from tqdm.notebook import tqdm

import torch
import torch.nn as nn
from torch.utils.data import Dataset, DataLoader
import pytorch_lightning as pl
import torch.nn.functional as F
from pytorch_lightning.callbacks import EarlyStopping
import wandb
import torchmetrics

wandb.init(project="Mocean-Recommendor",entity="maxall4")

config = wandb.config

def mish(x):
    return (x*torch.tanh(F.softplus(x)))

np.random.seed(123)

# %% [markdown]
# First, we import the ratings dataset.

# %% [code]
ratings = pd.read_csv('rating.csv',
                      parse_dates=['timestamp'])

# %% [markdown]
# In order to keep memory usage manageable within Kaggle's kernel, we will only use data from 30% of the users in this dataset. Let's randomly select 30% of the users and only use data from the selected users.

# %% [code]
rand_userIds = np.random.choice(ratings['userId'].unique(),
                                size=int(len(ratings['userId'].unique())*0.3),
                                replace=False)

ratings = ratings.loc[ratings['userId'].isin(rand_userIds)]

print('There are {} rows of data from {} users'.format(len(ratings), len(rand_userIds)))

# %% [code]
ratings.sample(5)

# %% [code]
ratings['rank_latest'] = ratings.groupby(['userId'])['timestamp'] \
                                .rank(method='first', ascending=False)

train_ratings = ratings[ratings['rank_latest'] != 1]
test_ratings = ratings[ratings['rank_latest'] == 1]

# drop columns that we no longer need
train_ratings = train_ratings[['userId', 'movieId', 'rating']]
test_ratings = test_ratings[['userId', 'movieId', 'rating']]

# %% [markdown]
# ### Converting the dataset into an implicit feedback dataset

# %% [code]
train_ratings.loc[:, 'rating'] = 1

train_ratings.sample(5)

# %% [markdown]
# The code below generates 4 negative samples for each row of data. In other words, the ratio of negative to positive samples is 4:1. This ratio is chosen arbitrarily but I found that it works rather well (feel free to find the best ratio yourself!)

# %% [code]
# Get a list of all movie IDs
all_movieIds = ratings['movieId'].unique()

# Placeholders that will hold the training data
users, items, labels = [], [], []

# This is the set of items that each user has interaction with
user_item_set = set(zip(train_ratings['userId'], train_ratings['movieId']))

# 4:1 ratio of negative to positive samples
num_negatives = 4

for (u, i) in tqdm(user_item_set):
    users.append(u)
    items.append(i)
    labels.append(1) # items that the user has interacted with are positive
    for _ in range(num_negatives):
        # randomly select an item
        negative_item = np.random.choice(all_movieIds)
        # check that the user has not interacted with this item
        while (u, negative_item) in user_item_set:
            negative_item = np.random.choice(all_movieIds)
        users.append(u)
        items.append(negative_item)
        labels.append(0) # items not interacted with are negative

# %% [code]
class MovieLensTrainDataset(Dataset):
    """MovieLens PyTorch Dataset for Training

    Args:
        ratings (pd.DataFrame): Dataframe containing the movie ratings
        all_movieIds (list): List containing all movieIds

    """

    def __init__(self, ratings, all_movieIds):
        self.users, self.items, self.labels = self.get_dataset(ratings, all_movieIds)

    def __len__(self):
        return len(self.users)

    def __getitem__(self, idx):
        return self.users[idx], self.items[idx], self.labels[idx]

    def get_dataset(self, ratings, all_movieIds):
        users, items, labels = [], [], []
        user_item_set = set(zip(ratings['userId'], ratings['movieId']))

        num_negatives = 4
        for u, i in user_item_set:
            users.append(u)
            items.append(i)
            labels.append(1)
            for _ in range(num_negatives):
                negative_item = np.random.choice(all_movieIds)
                while (u, negative_item) in user_item_set:
                    negative_item = np.random.choice(all_movieIds)
                users.append(u)
                items.append(negative_item)
                labels.append(0)

        return torch.tensor(users), torch.tensor(items), torch.tensor(labels)

# %% [code]
acc_metric = torchmetrics.Accuracy()
class NCF(pl.LightningModule):
    """ Neural Collaborative Filtering (NCF)

        Args:
            num_users (int): Number of unique users
            num_items (int): Number of unique items
            ratings (pd.DataFrame): Dataframe containing the movie ratings for training
            all_movieIds (list): List containing all movieIds (train + test)
    """

    def __init__(self, num_users, num_items, ratings, all_movieIds):
        super().__init__()
        self.user_embedding = nn.Embedding(num_embeddings=num_users, embedding_dim=8)
        self.item_embedding = nn.Embedding(num_embeddings=num_items, embedding_dim=8)
        self.fc1 = nn.Linear(in_features=16, out_features=64)
        self.fc2 = nn.Linear(in_features=64, out_features=32)
        self.output = nn.Linear(in_features=32, out_features=1)
        self.ratings = ratings
        self.all_movieIds = all_movieIds

    def on_validation_end(self,outputs):
        loss = torch.stack([x['val_loss'] for x in outputs]).mean()
        return { 'loss' : loss }
    def forward(self, user_input, item_input):

        # Pass through embedding layers
        user_embedded = self.user_embedding(user_input)
        item_embedded = self.item_embedding(item_input)

        # Concat the two embedding layers
        vector = torch.cat([user_embedded, item_embedded], dim=-1)

        # Pass through dense layer
        vector = mish(self.fc1(vector))
        vector = mish(self.fc2(vector))

        # Output layer
        pred = nn.Sigmoid()(self.output(vector))

        return pred

    def training_step(self, batch, batch_idx):
        user_input, item_input, labels = batch
        predicted_labels = self(user_input, item_input)
        loss = nn.BCELoss()(predicted_labels, labels.view(-1, 1).float())

        acc = acc_metric(predicted_labels,labels)

        wandb.log({"loss": loss,"acc":acc})
        return loss

    def configure_optimizers(self):
        return torch.optim.Adam(self.parameters())

    def train_dataloader(self):
        return DataLoader(MovieLensTrainDataset(self.ratings, self.all_movieIds),
                          batch_size=512, num_workers=4)

# %% [markdown]
# We instantiate the NCF model using the class that we have defined above.

# %% [code]
num_users = ratings['userId'].max()+1
num_items = ratings['movieId'].max()+1

all_movieIds = ratings['movieId'].unique()

model = NCF(num_users, num_items, train_ratings, all_movieIds)

# %% [code]
wandb.watch(model)
early_stopping = EarlyStopping(
        monitor='loss',
        min_delta=0.00,
        patience=3,
        verbose=False,
        mode='min',
    )

trainer = pl.Trainer(max_epochs=100, gpus=1, reload_dataloaders_every_epoch=True,
                     progress_bar_refresh_rate=50, logger=False, checkpoint_callback=True,callbacks=[early_stopping])

trainer.fit(model)

# %% [markdown]
# ### Hit Ratio @ 10

# %% [code]
# User-item pairs for testing
test_user_item_set = set(zip(test_ratings['userId'], test_ratings['movieId']))

# Dict of all items that are interacted with by each user
user_interacted_items = ratings.groupby('userId')['movieId'].apply(list).to_dict()

hits = []
for (u,i) in tqdm(test_user_item_set):
    interacted_items = user_interacted_items[u]
    not_interacted_items = set(all_movieIds) - set(interacted_items)
    selected_not_interacted = list(np.random.choice(list(not_interacted_items), 99))
    test_items = selected_not_interacted + [i]

    predicted_labels = np.squeeze(model(torch.tensor([u]*100),
                                        torch.tensor(test_items)).detach().numpy())

    top10_items = [test_items[i] for i in np.argsort(predicted_labels)[::-1][0:10].tolist()]

    if i in top10_items:
        hits.append(1)
    else:
        hits.append(0)

print("The Hit Ratio @ 10 is {:.2f}".format(np.average(hits)))
wandb.log({"hit ratio": np.average(hits)})

error:
Traceback (most recent call last):
  File "main.py", line 359, in <module>
    trainer = pl.Trainer(max_epochs=100, gpus=1, reload_dataloaders_every_epoch=True,
  File "/home/max/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 499, in fit
    self.dispatch()
  File "/home/max/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 546, in dispatch
    self.accelerator.start_training(self)
  File "/home/max/.local/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 73, in start_training
    self.training_type_plugin.start_training(trainer)
  File "/home/max/.local/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 114, in start_training
    self._results = trainer.run_train()
  File "/home/max/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 637, in run_train
    self.train_loop.run_training_epoch()
  File "/home/max/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 492, in run_training_epoch
    batch_output = self.run_training_batch(batch, batch_idx, dataloader_idx)
  File "/home/max/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 654, in run_training_batch
    self.optimizer_step(optimizer, opt_idx, batch_idx, train_step_and_backward_closure)
  File "/home/max/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 425, in optimizer_step
    model_ref.optimizer_step(
  File "/home/max/.local/lib/python3.8/site-packages/pytorch_lightning/core/lightning.py", line 1390, in optimizer_step
    optimizer.step(closure=optimizer_closure)
  File "/home/max/.local/lib/python3.8/site-packages/pytorch_lightning/core/optimizer.py", line 214, in step
    self.__optimizer_step(*args, closure=closure, profiler_name=profiler_name, **kwargs)
  File "/home/max/.local/lib/python3.8/site-packages/pytorch_lightning/core/optimizer.py", line 134, in __optimizer_step
    trainer.accelerator.optimizer_step(optimizer, self._optimizer_idx, lambda_closure=closure, **kwargs)
  File "/home/max/.local/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 277, in optimizer_step
    self.run_optimizer_step(optimizer, opt_idx, lambda_closure, **kwargs)
  File "/home/max/.local/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 282, in run_optimizer_step
    self.training_type_plugin.optimizer_step(optimizer, lambda_closure=lambda_closure, **kwargs)
  File "/home/max/.local/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 163, in optimizer_step
    optimizer.step(closure=lambda_closure, **kwargs)
  File "/home/max/.local/lib/python3.8/site-packages/torch/optim/optimizer.py", line 89, in wrapper
    return func(*args, **kwargs)
  File "/home/max/.local/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/home/max/.local/lib/python3.8/site-packages/torch/optim/adam.py", line 66, in step
    loss = closure()
  File "/home/max/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 648, in train_step_and_backward_closure
    result = self.training_step_and_backward(
  File "/home/max/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 742, in training_step_and_backward
    result = self.training_step(split_batch, batch_idx, opt_idx, hiddens)
  File "/home/max/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 293, in training_step
    training_step_output = self.trainer.accelerator.training_step(args)
  File "/home/max/.local/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 156, in training_step
    return self.training_type_plugin.training_step(*args)
  File "/home/max/.local/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 125, in training_step
    return self.lightning_module.training_step(*args, **kwargs)
  File "main.py", line 318, in training_step
    print(type(labels))
  File "/home/max/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/max/.local/lib/python3.8/site-packages/torchmetrics/metric.py", line 152, in forward
    self.update(*args, **kwargs)
  File "/home/max/.local/lib/python3.8/site-packages/torchmetrics/metric.py", line 199, in wrapped_func
    return update(*args, **kwargs)
  File "/home/max/.local/lib/python3.8/site-packages/torchmetrics/classification/accuracy.py", line 142, in update
    self.correct += correct
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!


I created this simplified version of VGG16:
class VGG16COMBO(nn.Module):

    def __init__(self, num_classes):
        super(VGG16COMBO, self).__init__()

        # calculate same padding:
        # (w - k + 2*p)/s + 1 = o
        # => p = (s(o-1) - w + k)/2

        self.block_1 = nn.Sequential(
            nn.Conv2d(in_channels=1,
                      out_channels=64,
                      kernel_size=(3, 3),
                      stride=(1, 1),
                      # (1(32-1)- 32 + 3)/2 = 1
                      padding=1),
            nn.BatchNorm2d(64),
            nn.ReLU(),
            nn.Conv2d(in_channels=64,
                      out_channels=64,
                      kernel_size=(3, 3),
                      stride=(1, 1),
                      padding=1),
            nn.BatchNorm2d(64),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=(2, 2),
                         stride=(2, 2))
        )

        self.block_2 = nn.Sequential(
            nn.Conv2d(in_channels=64,
                      out_channels=128,
                      kernel_size=(3, 3),
                      stride=(1, 1),
                      padding=1),
            nn.BatchNorm2d(128),
            nn.ReLU(),
            nn.Conv2d(in_channels=128,
                      out_channels=128,
                      kernel_size=(3, 3),
                      stride=(1, 1),
                      padding=1),
            nn.BatchNorm2d(128),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=(2, 2),
                         stride=(2, 2))
        )

        self.block_3 = nn.Sequential(
            nn.Conv2d(in_channels=128,
                      out_channels=256,
                      kernel_size=(3, 3),
                      stride=(1, 1),
                      padding=1),
            nn.BatchNorm2d(256),
            nn.ReLU(),
            nn.Conv2d(in_channels=256,
                      out_channels=256,
                      kernel_size=(3, 3),
                      stride=(1, 1),
                      padding=1),
            nn.BatchNorm2d(256),
            nn.ReLU(),
            nn.Conv2d(in_channels=256,
                      out_channels=256,
                      kernel_size=(3, 3),
                      stride=(1, 1),
                      padding=1),
            nn.BatchNorm2d(256),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=(2, 2),
                         stride=(2, 2))
        )

        self.block_4 = nn.Sequential(
            nn.Conv2d(in_channels=256,
                      out_channels=512,
                      kernel_size=(3, 3),
                      stride=(1, 1),
                      padding=1),
            nn.BatchNorm2d(512),
            nn.ReLU(),
            nn.Conv2d(in_channels=512,
                      out_channels=512,
                      kernel_size=(3, 3),
                      stride=(1, 1),
                      padding=1),
            nn.BatchNorm2d(512),
            nn.ReLU(),
            nn.Conv2d(in_channels=512,
                      out_channels=512,
                      kernel_size=(3, 3),
                      stride=(1, 1),
                      padding=1),
            nn.BatchNorm2d(512),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=(2, 2),
                         stride=(2, 2))
        )

        self.classifier = nn.Sequential(
            nn.Linear(2048, 4096),
            nn.ReLU(True),
            nn.Dropout(p=0.25),
            nn.Linear(4096, 4096),
            nn.ReLU(True),
            nn.Dropout(p=0.25),
            nn.Linear(4096, num_classes),
        )

    def forward(self, m, c):

        m = self.block_1(m)
        m = self.block_2(m)
        m = self.block_3(m)
        m = self.block_4(m)
        m = m.view(m.size(0), -1)
        m = self.classifier(m)

        c = self.block_1(c)
        c = self.block_2(c)
        c = self.block_3(c)
        c = self.block_4(c)
        c = c.view(c.size(0), -1)
        c = self.classifier(c)

        x = torch.cat((m, c), dim=1)
        return x

You can see that in forward I pass 2 elements, m and c. m is referred to MNIST, and c to CIFAR10, because i want a multi input neural network (or a network with shared weights).
Then:
modelcombo = VGG16COMBO(1).cuda()
print(modelcombo)

# Define an optimizier
import torch.optim as optim
optimizer = optim.SGD(modelcombo.parameters(), lr = 0.01)
# Define a loss
criterion = nn.BCEWithLogitsLoss()

This is my training function:
#train da modificare con entrambi i dataset
def train(net, loaders, optimizer, criterion, epochs=20, dev=dev, save_param = False, model_name="valerio"):
    try:
        net = net.to(dev)
        #print(net)
        # Initialize history
        history_loss = {"train": [], "val": [], "test": []}
        history_accuracy = {"train": [], "val": [], "test": []}
        # Store the best val accuracy
        best_val_accuracy = 0

        # Process each epoch
        for epoch in range(epochs):
            # Initialize epoch variables
            sum_loss = {"train": 0, "val": 0, "test": 0}
            sum_accuracy = {"train": 0, "val": 0, "test": 0}
            # Process each split
            for split in ["train", "val", "test"]:
                if split == "train":
                  net.train()
                else:
                  net.eval()
                # Process each batch
                for (input, labels) in loaders[split]:
                    # Move to CUDA
                    input = input.to(dev)
                    labels = labels.to(dev)
                    # Reset gradients
                    optimizer.zero_grad()
                    # Compute output
                    pred = net(input)
                    #pred = pred.squeeze(dim=1) # Output shape is [Batch size, 1], but we want [Batch size]
                    labels = labels.unsqueeze(1)
                    labels = labels.float()
                    loss = criterion(pred, labels)
                    # Update loss
                    sum_loss[split] += loss.item()
                    # Check parameter update
                    if split == "train":
                        # Compute gradients
                        loss.backward()
                        # Optimize
                        optimizer.step()
                    # Compute accuracy
                    #pred_labels = pred.argmax(1) + 1
                    pred_labels = (pred >= 0.5).long() # Binarize predictions to 0 and 1
                    batch_accuracy = (pred_labels == labels).sum().item()/input.size(0)
                    # Update accuracy
                    sum_accuracy[split] += batch_accuracy
            # Compute epoch loss/accuracy
            epoch_loss = {split: sum_loss[split]/len(loaders[split]) for split in ["train", "val", "test"]}
            epoch_accuracy = {split: sum_accuracy[split]/len(loaders[split]) for split in ["train", "val", "test"]}

            # Store params at the best validation accuracy
            if save_param and epoch_accuracy["val"] > best_val_accuracy:
              #torch.save(net.state_dict(), f"{net.__class__.__name__}_best_val.pth")
              torch.save(net.state_dict(), f"{model_name}_best_val.pth")
              best_val_accuracy = epoch_accuracy["val"]

            # Update history
            for split in ["train", "val", "test"]:
                history_loss[split].append(epoch_loss[split])
                history_accuracy[split].append(epoch_accuracy[split])
            # Print info
            print(f"Epoch {epoch+1}:",
                  f"TrL={epoch_loss['train']:.4f},",
                  f"TrA={epoch_accuracy['train']:.4f},",
                  f"VL={epoch_loss['val']:.4f},",
                  f"VA={epoch_accuracy['val']:.4f},",
                  f"TeL={epoch_loss['test']:.4f},",
                  f"TeA={epoch_accuracy['test']:.4f},")
    except KeyboardInterrupt:
        print("Interrupted")
    finally:
        # Plot loss
        plt.title("Loss")
        for split in ["train", "val", "test"]:
            plt.plot(history_loss[split], label=split)
        plt.legend()
        plt.show()
        # Plot accuracy
        plt.title("Accuracy")
        for split in ["train", "val", "test"]:
            plt.plot(history_accuracy[split], label=split)
        plt.legend()
        plt.show()

But when I do the training
# Train model
train(modelcombo, loaders, optimizer, criterion, epochs=10, dev=dev)

I obtain this error:
TypeError: forward() missing 1 required positional argument: 'c'

What I have to change, the net or the training function? I think the problem is in the training function, because I have to pass loaders, and loaders_cifar, but I don't know how. In particular, I have to cat loaders of mnist and loaders of cifar before passing them to the training function, or I have to modify the for (input, labels) in loaders[split]: in something like for (input, labels) in loaders[split] and loaders_cifar[split]:?
EDIT: I created this function:
def itr_merge(*itrs):
    for itr in itrs:
        for v in itr:
            yield v

Edited the training function in this way:
#train da modificare con entrambi i dataset
def train2(net, loaders, loaders_cifar, optimizer, criterion, epochs=20, dev=dev, save_param = False, model_name="valerio"):
    try:
        net = net.to(dev)
        #print(net)
        # Initialize history
        history_loss = {"train": [], "val": [], "test": []}
        history_accuracy = {"train": [], "val": [], "test": []}
        # Store the best val accuracy
        best_val_accuracy = 0

        # Process each epoch
        for epoch in range(epochs):
            # Initialize epoch variables
            sum_loss = {"train": 0, "val": 0, "test": 0}
            sum_accuracy = {"train": 0, "val": 0, "test": 0}
            # Process each split
            for split in ["train", "val", "test"]:
                if split == "train":
                  net.train()
                else:
                  net.eval()
                # Process each batch
                for x in itr_merge(loaders[split], loaders_cifar[split]):
                  for (input, labels) in loaders[split]:
                      # Move to CUDA
                      input = input.to(dev)
                      labels = labels.to(dev)
                      # Reset gradients
                      optimizer.zero_grad()
                      # Compute output
                      pred = net(input)
                      #pred = pred.squeeze(dim=1) # Output shape is [Batch size, 1], but we want [Batch size]
                      labels = labels.unsqueeze(1)
                      labels = labels.float()
                      loss = criterion(pred, labels)
                      # Update loss
                      sum_loss[split] += loss.item()
                      # Check parameter update
                      if split == "train":
                          # Compute gradients
                          loss.backward()
                          # Optimize
                          optimizer.step()
                      # Compute accuracy
                      #pred_labels = pred.argmax(1) + 1
                      pred_labels = (pred >= 0.5).long() # Binarize predictions to 0 and 1
                      batch_accuracy = (pred_labels == labels).sum().item()/input.size(0)
                      # Update accuracy
                      sum_accuracy[split] += batch_accuracy
            # Compute epoch loss/accuracy
            epoch_loss = {split: sum_loss[split]/len(loaders[split]) for split in ["train", "val", "test"]}
            epoch_accuracy = {split: sum_accuracy[split]/len(loaders[split]) for split in ["train", "val", "test"]}

            # Store params at the best validation accuracy
            if save_param and epoch_accuracy["val"] > best_val_accuracy:
              #torch.save(net.state_dict(), f"{net.__class__.__name__}_best_val.pth")
              torch.save(net.state_dict(), f"{model_name}_best_val.pth")
              best_val_accuracy = epoch_accuracy["val"]

            # Update history
            for split in ["train", "val", "test"]:
                history_loss[split].append(epoch_loss[split])
                history_accuracy[split].append(epoch_accuracy[split])
            # Print info
            print(f"Epoch {epoch+1}:",
                  f"TrL={epoch_loss['train']:.4f},",
                  f"TrA={epoch_accuracy['train']:.4f},",
                  f"VL={epoch_loss['val']:.4f},",
                  f"VA={epoch_accuracy['val']:.4f},",
                  f"TeL={epoch_loss['test']:.4f},",
                  f"TeA={epoch_accuracy['test']:.4f},")
    except KeyboardInterrupt:
        print("Interrupted")
    finally:
        # Plot loss
        plt.title("Loss")
        for split in ["train", "val", "test"]:
            plt.plot(history_loss[split], label=split)
        plt.legend()
        plt.show()
        # Plot accuracy
        plt.title("Accuracy")
        for split in ["train", "val", "test"]:
            plt.plot(history_accuracy[split], label=split)
        plt.legend()
        plt.show()

But I have still the same error


title: How to pretrain/select CNN for Biomedical Video Analysis

body:
Data
I am trying to train a model in the biomedical domain with a rather specialized task (flow prediction).
My input consists of video clips and I would like to predict either a single image or a video. I have pixel wise ground truth for every frame of the video but the number of videos is very limited.
I do however have a lot of unlabeled data of related scenes, which is why I considered transfer learning/pretraining of some sort.

Architecture
Architecture wise I am considering a CNN, RNN combination where the CNN provides a representation of the input frames for the RNN to learn about the temporal relationship between input frames.

Now my question is: What kind of CNN do I use and on what do I pretrain it? Since I am working with biomedical data I would assume image net as well as most other image data sets do not really help as the image content is very different. Are there any data sets/tasks/networks I could use for this purpose?


title: Fully connected layers with unflattened input

body:
(batch_size, 70, 2, 2) -> Linear(2, 2)

(batch_size, 140, 2) -> Linear(2, 2)

(batch_size, 280) -> Linear(280, 2)

Can someone explain to me how Fully Connected layers work with non-flattened input data? I don't really know how to think about > 2D matrix multiplication. Are all the above equivalent? (Linear is the fully connected module in pytorch)


title: Computation of loss is taking very much time on GPU in pytorch

body:
I've been trying to implement DCGAN in pytorch. But during training, a single iteration of training loop takes more than 7-8 minutes on GPU on google collab. I can't understand what is wrong in the code. I have been trying many techniques to overcome this issue but nothing seems to be working..

Here's My training loop and it was taking more than 7-8 minutes on a single iteration:

device = torch.device("cuda:0")

dis = Discriminator().to(device)
Gen = Generator().to(device)

GAN_loss = nn.BCELoss().to(device)

D_optimizer = optim.Adam(dis.parameters(), lr = 0.0002, betas = (0.5, 0.999))
G_optimizer = optim.Adam(Gen.parameters(), lr = 0.0002, betas = (0.5, 0.999))

path = 'gdrive/My Drive/New_data/'
path2 = 'gdrive/My Drive/New_cropped/'

train_data_list = os.listdir(path)
train_data_len = len(train_data_list)

minibatch_size = 64
epochs = 10
G_losses = []
D_losses = []

final_itr = (train_data_len + minibatch_size - 1) // minibatch_size
data_list = [train_data_list[i * minibatch_size : (i + 1) * minibatch_size] for i in range(final_itr)]

for epoch in range(epochs):
    for count, data in enumerate(data_list):

        train_img = []
        sample_img = []

        for image in data:
            img_train = cv2.imread(path + image).T/255
            img_train = img_train.reshape(1, img_train.shape[0], img_train.shape[1], img_train.shape[2])

            img_sample = cv2.imread(path2 + image,0).T/255
            img_sample = img_sample.reshape(1, 1, img_sample.shape[0], img_sample.shape[1])
            train_img.append(img_train)
            sample_img.append(img_sample)

            assert(img_sample.shape == (1, 1, 144, 144))

            train_image = Variable(torch.from_numpy(np.concatenate(train_img, axis = 0)).cuda())
            sample_image = Variable(torch.from_numpy(np.concatenate(sample_img, axis = 0)).cuda())
            label = torch.full((train_image.shape[0],), real_label, device=device)

        #Training the discriminator... minimizing -(log(D(x)) - log(1 - D(G(Z))))

        dis.zero_grad()
        Gen.zero_grad()

        G_z = Gen(sample_image.detach())

        disc_real_out = dis(train_image.detach()).view(-1)
        error_real = GAN_loss(disc_real_out, label)
        error_real.backward()

        disc_fake_out = dis(G_z.detach()).view(-1)
        label.fill_(fake_label)
        error_fake = GAN_loss(disc_fake_out, label)
        error_fake.backward()

        total_disc_error = error_real + error_fake
        D_optimizer.step()

        #Training the Generator... maximizing log(D(G(Z))))

        D_G_z = dis(G_z.detach()).view(-1)
        label.fill_(real_label)
        error_gen = GAN_loss(D_G_z, label)
        error_gen.backward()
        G_optimizer.step()

        G_losses.append(error_gen.item())
        D_losses.append(total_disc_error.item())

        print("Discriminator Loss : ", total_disc_error.item(), "\t", "Generator Loss : ", error_gen.item())


title: Does PyTorch have a RandomState-like object for random number generation?

body:
in numpy i can

import numpy as np

rs = np.random.RandomState(seed=0)

and then pass that object around, eg for dependency injection.

Does PyTorch have a similar interface? I can't find anything in the docs, but maybe i'm missing something.


title: Create self attributes in python using for loop when using pytorch

body:
Within pytorch, creating layers, can be semi automated, thus the reason for using a for loop.

One of the main issues is that these layers cannot stored within a list or dictionary or else back propagation will not work.

Thus the reason for a work around.

Within the object, assigning new self attributes

How do i replace this

self.res1 = 1
self.res2 = 2
self.res3 = 3

with this

for i in range(2):
  res_name = 'res'+str(i+1)
  self.res_name = i

Now that i have created objects this way, how can I access them in the same way. For example, if we assume self.res_name is now an object?

for i in range(2):
   res_name = 'res'+str(i+1)
   out = self.res_name(out)


title: How to optimize a simulation metric with deep learning without target values?

body:
I am trying to use an RNN model that outputs bus routes and its input is the demand matrix. The bus routes are then used in a simulation which spits out a metric of how the routes performed. The question is, since there is no target value of bus routes, how do I back propagate the simulation result?

To explain the question with simple python code:

"""
The model is an RNN that takes 400,24,24 matrix as input

dimension 0 represents time, dimension 1 represents departure bus stop and dimension 2 represents the arrival bus stop. Each value is a count of the number of passengers who departed at a bus stop with an arrival bus stop in mind in a specific time

output is 64,24 matrix which will be reshaped to 8,8,24
dimension 0 is the sequence index, dimension 1 is the index of bus (there are 8 buses), dimension 2 is the softmaxed classifier dimension of 24 different bus stops. From the output, 8 bus stops are picked per bus with a sequence

These sequences are then used for path generations of buses and they are evaluated from a simulation
"""

model.train()
optimizer.zero_grad()
out = model(demand)#out is 64,24 demand is 400,24,24
demand, performance = simulation(out)#assume performance as float
#here the out has grad_fn but the performance does not
loss = SOME_NUMBER - performance
loss = torch.FloatTensor(loss)
#here I need to back propagate and it is the confusing part

#simply doing loss.backward() does nothing because no grad_fn

#out.backward() requires 64,24 gradients computed somehow from 1 #metric, causes complete divergence within few steps

optimizer.step()


title: Creating a matrix of validation results with correct counts

body:
I have a trained 10-class machine learning model similar to an MNIST digit classifier. I want to determine how often each digit class is correct and, if a mistake is made, which class it is confused for. I want to use this to create a confusion matrix, among other things.

Anyway, the input for each pass is a batch of pictures from the validation set (shape (32,3,224,224) where 3, 224, and 224 are the picture dimensions and 32 is the batch size) and the labels (shape 32,1) of class numbers those pictures match. The model output is (shape 32,1) and lists the class numbers the model thinks they best match. I can easily find how many matches there are by comparing the labels and outputs tensors, but I am having a hard time telling how the misclassifications are missed. Here's a snippet from the main validation loop

# Main validation loop
valid_accuracy = 0.0
model.eval()
device = 'cuda'
raw_counts = torch.zeros((11,11)) # leave room for totals in the last row and column
with torch.no_grad():
    for inputs, labels in validloader:
        # Run each image through the network to get log probabilities of each class
        inputs, labels = inputs.to(device), labels.to(device)
        logps = model.forward(inputs)

        # Calculate accuracy
        ps = torch.exp(logps) # 32 X 10: probability of each label in every case
        top_p, top_class = ps.topk(1, dim=1)
        equals = top_class == labels.view(*top_class.shape)
        valid_accuracy += torch.mean(equals.type(torch.FloatTensor)).item()

        # Count confusions
        raw_counts[labels[:],top_class[:,0]] += 1  # <<<<---- This is the problem!

        # Accumulate letter-by-letter certainties
        for i in range(ll):
            sum_ps[i,] += sum(ps[labels==i])
            letter_counts[i] += len(ps[labels==i])

# Print validation accuracy
valid_accuracy = valid_accuracy/len(validloader)
print(f"Validation accuracy: {valid_accuracy:.3f}")

The problem is where I try to count the confusions. With only ten classes and a batch size of 32 I am guaranteed to have repeated labels. But saying raw_counts[labels[:],top_class[:,0]] += 1 only increments by one each row of the raw_counts matrix. For example, in the debugger just before this line:

(Pdb) top_class[:,0]
tensor([5, 9, 5, 0, 2, 3, 3, 8, 2, 9, 6, 3, 0, 3, 1, 3, 3, 4, 0, 1, 5, 2, 8, 4,
        5, 3, 6, 5, 0, 3, 2, 1], device='cuda:0')
(Pdb) labels[:]
tensor([5, 9, 5, 0, 2, 3, 3, 8, 2, 9, 6, 3, 0, 3, 1, 3, 3, 4, 0, 1, 5, 2, 8, 4,
        5, 3, 8, 5, 0, 3, 2, 1], device='cuda:0')
(Pdb) raw_counts
tensor([[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]])

Everything is as expected. But after executing the line:

(Pdb) n
> /home/model.py(219)main()
-> for i in range(10):
(Pdb) raw_counts
tensor([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
        [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 1., 0., 1., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0.],
        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]])

Even though five labels in this batch are 5's and the model got them all right, still raw_counts[4,4] == 1. Is there any pythonic way to count all five correct answers without messy for loops and the like?


title: How to calculate autograd.grad in pytorch internally?

body:
I tried to get a grad in pytorch.
So, I coded it like below:

x = torch.tensor([1., 2., 3.], requires_grad=True)
y = x * x

torch.autograd.grad(y, x, torch.tensor([2., 2., 2.])

Output:

tensor([4., 8., 12.])

How was it calculated internally?

I didn't know exactly why should I specify 'grad_outputs' as paramerter in torch.autograd.grad() and how was it computed.


title: Why is the true positive - false negative distribution always the same

body:
I have a neural network that I use it for binary classification. I change the size of training data and predict on the test set. By looking at the results, the difference between tp and fn is always the same and the difference between tn and fp is always the same. For example, in iteration #2, tp#2 - tp#1 = -91 and fn#2 - fn#1 = +91. Also, fp#2 - fp#1 = -46 and tn#2 - tn#1 = +46. As another example, tp#3 - tp#2 = -35 and fn#2 - fn#2 = +35.

Iteration #1
tn=119, fp=173, fn=110, tp=407
Iteration #2
tn=165, fp=127, fn=201, tp=316
Iteration #3
tn=176, fp=116, fn=236, tp=281
Iteration #4
tn=157, fp=135, fn=207, tp=310
Iteration #5
tn=155, fp=137, fn=214, tp=303

I have tried various architectures of neural nets, but I always get the same numbers. Do you have an idea what is wrong.

The following is a very simple network that I use:

class AllCnns(nn.Module):
    def __init__(self, vocab_size, embedding_size):
        torch.manual_seed(0)
        super(AllCnns, self).__init__()
        self.word_embeddings = nn.Embedding(vocab_size, embedding_size)
        self.conv1 = nn.Conv1d(embedding_size, 64, 3)
        self.drop1 = nn.Dropout(0.3)
        self.max_pool1 = nn.MaxPool1d(2)
        self.flat1 = nn.Flatten()
        self.fc1 = nn.Linear(64*80, 100)
        self.fc2 = nn.Linear(100, 1)

    def forward(self, sentence):
        embedding = self.word_embeddings(sentence).permute(0, 2, 1)
        conv1 = F.relu(self.conv1(embedding))
        drop1 = self.drop1(conv1)
        max_pool1 = self.max_pool1(drop1)
        flat1 = self.flat1(max_pool1)
        fc1 = F.relu(self.fc1(flat1))
        fc2 = torch.sigmoid(self.fc2(fc1))
        return fc2


title: dimension problem using VGG16 architecture

body:
I have tried to use the vgg16 architecture and afterward a convolution layer.
I know the output of vgg16 is suppose to be 512 channels of 7x7, but for some reason, it just gives the output as some sort of one-dimensional list.

therefore,I keep getting the following error :
RuntimeError: Expected 4-dimensional input for 4-dimensional weight 100 512 3 3, but got 2-dimensional input of size [64, 25088] instead

I've tried to reshape the output of the vgg16 with no success. any suggestions?
all I want is to put a convolution layer, and not fully connected layer, after the vgg16 so it won't have too many parameters

the code (only the part of vgg16 and the network I want to add )

#Choosing our pretrained network model
pre_trained_model = tv.models.vgg16(pretrained=True)
pre_trained_model = pre_trained_model

# freeze parameters
for param in pre_trained_model.parameters():
param.requires_grad = False

# add layers to train
from collections import OrderedDict
#classifier = classifier.nn.Size([1, 512, 7, 7])
classifier = nn.Sequential(OrderedDict([
                      ('conv1', nn.Conv2d(512, 100, 3)),
                      ('fc1', nn.Linear(2500, 400)),
                      ('dr1', nn.Dropout(p = 0.3)),
                      ('bn1', nn.BatchNorm1d(num_features=400)),
                      ('relu2', nn.ReLU()),
                      ('fc2', nn.Linear(400, 100)),
                      ('dr2', nn.Dropout(p = 0.3)),
                      ('bn2', nn.BatchNorm1d(num_features=100)),
                      ('relu3', nn.ReLU()),
                      ('fc3', nn.Linear(100, 2)),
                      ]))


title: With PyTorch, how is my Conv1d dimension reducing when I have padding?

body:
My conv module is:

        return torch.nn.Sequential(
            torch.nn.Conv1d(
                in_channels=in_channels,
                out_channels=in_channels,
                kernel_size=2,
                stride=1,
                dilation=1,
                padding=1
            ),
            torch.nn.ReLU(),
            torch.nn.Conv1d(
                in_channels=in_channels,
                out_channels=in_channels,
                kernel_size=2,
                stride=1,
                dilation=2,
                padding=1
            ),
            torch.nn.ReLU(),
            torch.nn.Conv1d(
                in_channels=in_channels,
                out_channels=in_channels,
                kernel_size=2,
                stride=1,
                dilation=4,
                padding=1
            ),
            torch.nn.ReLU()
        )

And in forward, I have:

down_out = self.downscale_time_conv(inputs)

inputs has a .size of torch.Size([8, 161, 24]). I'd expect down_out to have the same size, but instead it has: torch.Size([8, 161, 23])

Where did that last element go?


title: U-Net with Pre-Trained ResNet throws dimension error must match

body:
I have an RGB image of mask for Segmentation of dimensions 900x600 (width, height)
My U-Net code is the ff. I do not really want to resize the output too much it is fine if it is resized without losing much of the aspect ratio.

import torch
import torch.nn as nn
from torchvision import models

def convrelu(in_channels, out_channels, kernel, padding):
  return nn.Sequential(
      nn.Conv2d(in_channels, out_channels, kernel, padding=padding),
      nn.ReLU(inplace=True)
  )

class ResNetUNet(nn.Module):
  def __init__(self, n_class=1):
    super().__init__()

    self.base_model = models.resnet18(pretrained=True)
    self.base_layers = list(self.base_model.children())

    self.layer0 = nn.Sequential(*self.base_layers[:3]) # size=(N, 64, x.H/2, x.W/2)
    self.layer0_1x1 = convrelu(64, 64, 1, 0)
    self.layer1 = nn.Sequential(*self.base_layers[3:5]) # size=(N, 64, x.H/4, x.W/4)
    self.layer1_1x1 = convrelu(64, 64, 1, 0)
    self.layer2 = self.base_layers[5]  # size=(N, 128, x.H/8, x.W/8)
    self.layer2_1x1 = convrelu(128, 128, 1, 0)
    self.layer3 = self.base_layers[6]  # size=(N, 256, x.H/16, x.W/16)
    self.layer3_1x1 = convrelu(256, 256, 1, 0)
    self.layer4 = self.base_layers[7]  # size=(N, 512, x.H/32, x.W/32)
    self.layer4_1x1 = convrelu(512, 512, 1, 0)

    self.upsample = nn.Upsample(scale_factor=2, mode='bilinear', align_corners=True)

    self.conv_up3 = convrelu(256 + 512, 512, 3, 1)
    self.conv_up2 = convrelu(128 + 512, 256, 3, 1)
    self.conv_up1 = convrelu(64 + 256, 256, 3, 1)
    self.conv_up0 = convrelu(64 + 256, 128, 3, 1)

    self.conv_original_size0 = convrelu(3, 64, 3, 1)
    self.conv_original_size1 = convrelu(64, 64, 3, 1)
    self.conv_original_size2 = convrelu(64 + 128, 64, 3, 1)

    self.conv_last = nn.Conv2d(64, n_class, 1)

  def forward(self, input):
    x_original = self.conv_original_size0(input)
    x_original = self.conv_original_size1(x_original)

    layer0 = self.layer0(input)
    layer1 = self.layer1(layer0)
    layer2 = self.layer2(layer1)
    layer3 = self.layer3(layer2)
    layer4 = self.layer4(layer3)

    layer4 = self.layer4_1x1(layer4)
    x = self.upsample(layer4)
    layer3 = self.layer3_1x1(layer3)
    x = torch.cat([x, layer3], dim=1)
    x = self.conv_up3(x)

    x = self.upsample(x)
    layer2 = self.layer2_1x1(layer2)
    x = torch.cat([x, layer2], dim=1)
    x = self.conv_up2(x)

    x = self.upsample(x)
    layer1 = self.layer1_1x1(layer1)
    x = torch.cat([x, layer1], dim=1)
    x = self.conv_up1(x)

    x = self.upsample(x)
    layer0 = self.layer0_1x1(layer0)
    x = torch.cat([x, layer0], dim=1)
    x = self.conv_up0(x)

    x = self.upsample(x)
    x = torch.cat([x, x_original], dim=1)
    x = self.conv_original_size2(x)

    out = self.conv_last(x)

    return out

for this command

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = ResNetUNet()
model = model.to(device)

# check keras-like model summary using torchsummary
from torchsummary import summary
summary(model, input_size=(3, 600, 900))

it throws the error:

     54     x = self.upsample(layer4)
     55     layer3 = self.layer3_1x1(layer3)
---> 56     x = torch.cat([x, layer3], dim=1)
     57     x = self.conv_up3(x)
     58

RuntimeError: Sizes of tensors must match except in dimension 3. Got 57 and 58

Not sure what to do here. Could someone help me how to solve this?


title: What is a fast way to get the individual gradients in pytorch?

body:
In pytorch it is very easy to specify a loss and get the gradients for that loss across a batch using something like:
loss = criterion(outputs, labels)
loss.backward(retain_graph=True)

I am used to doing this and training models using the backwards function. The aggregated weights are easy to get:
layer.grad # this works.

However, I want to get the gradients of each individual training item. The backwards function will sum up over the training items in order to get a gradient for the whole batch.
I have found one solution, which is as follows:
for n in range(inputs.shape[0]):
    loss = criterion(outputs[[n]], labels[[n]])
    loss.backward(retain_graph=True)
    individual_gradients[n] = layer.grad

But this is very slow. I understand there is a speedup in doing the backward pass with a batch, but I don't understand why the difference is so stark: On average it's about 20-30 times slower this way. Any help would be appreciated. Specifically if you know a good way of getting those individual gradients that would be fantastic.


title: resize new image relative to the original image size pytorch

body:
old_dims = torch.FloatTensor([image.width, image.height, image.width, image.height]).unsqueeze(0)

I have old_dims with
   old_dims = torch.FloatTensor([image.width, image.height, image.width, image.height]).unsqueeze(0)

want to resize the boxes with new dimension
boxes-> tensor([[ 487., 1766.,  666., 2039.],
        [ 658., 1741.,  834., 2031.],
        [ 704., 1886.,  927., 2072.],
        [ 830., 1275., 1459., 2068.],
        [1028., 1943., 1848., 2307.]], dtype=torch.float64)
old_dim-> tensor([[5354., 3735., 5354., 3735.]])
new size ->(3,300,300)

new dimension is (3,300,300)
show should I achive this?


title: Changing to Python 64 bit version seems to have broken things

body:
I couldn't figure out why installing PyTorch was failing when using the exact procedure in the official "getting started" page. Then I found one little blurb deep in a forum that I should check that I had 64-bit python installed. Sure enough, I did not. So I uninstalled the 32-bit version and installed the 64-bit.
Now, when I type python --version, I get
bash: /c/Users/wesle/AppData/Local/Microsoft/WindowsApps/python: Permission denied

Which is strange, because that's never where Python was installed in the first place. Kinda a funny place to look for it now.
But, typing py --version, which is a solution I found here does show the correct version.
Another thing that seems broken is Jupyter Lab. Now I get bash: jupyter: command not found.
Oh, here's a big one I just found: I can't run my scripts in PyCharm now. So Jupyter and my IDE are toast. I guess I've been needing to learn the command line a bit more...
All joking aside, I'm kinda afraid to do anything else at this point since apparently I screwed up pretty bad already.
Lifeline, please...
Oh and yes, my PATH variable is up-to-date with the new version.


title: How to create a balancing cycling iterator in PyTourch?

body:
Say I have 2 classes. And for one I have only 17 samples and for another 83. I want to always have equal amount of data from each class per epoch (meaning 17 by 17 in this case). Also, I want to slide sampling a window across the class where I have more data each epoch (first 17, next 17, ...).
Currently I have a looping sampling iterator like this:
class CyclicIterator:
    def __init__(self, loader, sampler):
        self.loader = loader
        self.sampler = sampler
        self.epoch = 0
        self._next_epoch()

    def _next_epoch(self):
        self.iterator = iter(self.loader)
        self.epoch += 1

    def __len__(self):
        return len(self.loader)

    def __iter__(self):
        return self

    def __next__(self):
        try:
            return next(self.iterator)
        except StopIteration:
            self._next_epoch()
            return next(self.iterator)

I wonder how to force all samples from each class to be of equal count per epoch?


title: local binary pattern transform

body:
Here is my local binary pattern function:
def lbp(x):

imgUMat = np.float32(x)

gray = cv2.cvtColor(imgUMat, cv2.COLOR_RGB2GRAY)

radius = 2
n_points = 8 * radius

METHOD = 'uniform'

lbp = local_binary_pattern(gray, n_points, radius, METHOD)
lbp = torch.from_numpy(lbp).long()

return lbp

Here I call lbp function:
input_img = plt.imread(trn_fnames[31])
x = lbp(input_img)

When I use x.shape it is:
torch.Size([600, 600])

Sounds good!!!
But my problem is when I use transforms.Lambda(lbp) in my transform function, my output image is torch.Size([600])
tfms = transforms.Compose([

transforms.Lambda(lbp)])

train_ds = datasets.ImageFolder(trn_dir, transform = tfms)

(train_ds[0][0][0]).shape
torch.Size([600])!!! >>>> my problem

I need torch.Size([600, 600])
I also different ways such as this:
tfms = transforms.Compose([

transforms.Lambda(lbp),
transforms.ToPILImage(),
transforms.Resize((sz, sz))])

And I got this error:
TypeError: pic should be Tensor or ndarray. Got <class ‘torch.Tensor’>.

I also added
transforms.ToTensor()])

But still have the same error:
TypeError: pic should be Tensor or ndarray. Got <class ‘torch.Tensor’>.

I’ll appreciate to your comments please!
Thank you.


title: Installing Nvidia NCCL "nickel" on windows system

body:
I am trying to build a distributed data parallel training architecture using PyTorch and came across the Horovod library. Halfway through installation I discovered that NCCL 2 (a core requirement for Horovod) is not available for windows.
I would appreciate any help for a solution or a workaround for this issue.
Thanks in advance.


title: can anyone explain what "out = self(images)" do in below code

body:
I am not able to understand, if prediction is calculated in forward method, then why there is need "out = self(images)" and what it will do. I am bit confuse about this code.
class MnistModel(nn.Module):
def __init__(self):
    super().__init__()
    self.linear = nn.Linear(input_size, num_classes)

def forward(self, xb):
    xb = xb.reshape(-1, 784)
    out = self.linear(xb)
    return out

def training_step(self, batch):
    images, labels = batch
    out = self(images)                  # Generate predictions
    loss = F.cross_entropy(out, labels) # Calculate loss
    return loss

def validation_step(self, batch):
    images, labels = batch
    out = self(images)                    # Generate predictions
    loss = F.cross_entropy(out, labels)   # Calculate loss
    acc = accuracy(out, labels)           # Calculate accuracy
    return {'val_loss': loss, 'val_acc': acc}

def validation_epoch_end(self, outputs):
    batch_losses = [x['val_loss'] for x in outputs]
    epoch_loss = torch.stack(batch_losses).mean()   # Combine losses
    batch_accs = [x['val_acc'] for x in outputs]
    epoch_acc = torch.stack(batch_accs).mean()      # Combine accuracies
    return {'val_loss': epoch_loss.item(), 'val_acc': epoch_acc.item()}

def epoch_end(self, epoch, result):
    print("Epoch [{}], val_loss: {:.4f}, val_acc: {:.4f}".format(epoch, result['val_loss'], result['val_acc']))

model = MnistModel()


title: Different CUDA results when running from menu mode and terminal mode

body:
I have a simple test.py code like below in vscode:
import torch
# check if pytorch is using cuda/gpu
print("CUDA available: ", torch.cuda.is_available())

and the following settings.json
{
    "python.venvPath": "/Users/myname/Anaconda3/envs",
    "python.pythonPath": "C:\\Users\\myname\\Anaconda3\\envs\\pytorch\\python.exe"
}

I have installed cpu version of pytorch in the base conda, and a virtual environment named "pytorch" with regular/gpu version of pytorch installed.
Now when I ran the code from terminal with: python test.py, I had results:
CUDA available:  False

While if I ran the code from menu, either using start debugging (F5) or run without debugging (Ctrl+F5), results were both:
CUDA available:  True

What's the problem? and how to use CUDA in terminal mode?


title: PyTorch Jupyter Notebook image unable to find torch

body:
I have built a pytorch jupyter notebook image using the Dockerfile below. The only thing I changed from Tensorflow Jupyter Dockerfile is the base image (From Tensorflow to PyTorch).
However, when I launch the Notebook in Kubeflow, I’m unable to import torch. However, with !pip list, I can actually find the torch module. Any solutions?

ARG BASE_IMAGE=pytorch/pytorch:1.5.1-cuda10.1-cudnn7-runtime
FROM $BASE_IMAGE
ARG TF_SERVING_VERSION=0.0.0
ARG NB_USER=jovyan
# TODO: User should be refactored instead of hard coded jovyan
USER root
ENV DEBIAN_FRONTEND noninteractive
ENV NB_USER $NB_USER
ENV NB_UID 1000
ENV HOME /home/$NB_USER
ENV NB_PREFIX /
ENV PATH $HOME/.local/bin:$PATH
# Use bash instead of sh
SHELL ["/bin/bash", "-c"]
RUN apt-get update && apt-get install -yq --no-install-recommends \
  apt-transport-https \
  build-essential \
  bzip2 \
  ca-certificates \
  curl \
  g++ \
  git \
  gnupg \
  graphviz \
  locales \
  lsb-release \
  openssh-client \
  sudo \
  unzip \
  vim \
  wget \
  zip \
  emacs \
  python3-pip \
  python3-dev \
  python3-setuptools \
  && apt-get clean && \
  rm -rf /var/lib/apt/lists/*
# Install Nodejs for jupyterlab-manager
RUN curl -sL https://deb.nodesource.com/setup_12.x | sudo -E bash -
RUN apt-get update && apt-get install -yq --no-install-recommends \
  nodejs \
  && apt-get clean && \
  rm -rf /var/lib/apt/lists/*
ENV DOCKER_CREDENTIAL_GCR_VERSION=1.4.3
RUN curl -LO https://github.com/GoogleCloudPlatform/docker-credential-gcr/releases/download/v${DOCKER_CREDENTIAL_GCR_VERSION}/docker-credential-gcr_linux_amd64-${DOCKER_CREDENTIAL_GCR_VERSION}.tar.gz && \
    tar -zxvf docker-credential-gcr_linux_amd64-${DOCKER_CREDENTIAL_GCR_VERSION}.tar.gz && \
    mv docker-credential-gcr /usr/local/bin/docker-credential-gcr && \
    rm docker-credential-gcr_linux_amd64-${DOCKER_CREDENTIAL_GCR_VERSION}.tar.gz && \
    chmod +x /usr/local/bin/docker-credential-gcr
# Install AWS CLI
RUN curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "/tmp/awscli-bundle.zip" && \
    unzip /tmp/awscli-bundle.zip && ./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws && \
    rm -rf ./awscli-bundle
# Install Azure CLI
RUN curl -sL https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor | tee /etc/apt/trusted.gpg.d/microsoft.asc.gpg > /dev/null && \
    AZ_REPO=$(lsb_release -cs) && \
    echo "deb [arch=amd64] https://packages.microsoft.com/repos/azure-cli/ $AZ_REPO main" | tee /etc/apt/sources.list.d/azure-cli.list && \
    apt-get update && \
    apt-get install azure-cli
RUN echo "en_US.UTF-8 UTF-8" > /etc/locale.gen && \
    locale-gen
ENV LC_ALL en_US.UTF-8
ENV LANG en_US.UTF-8
ENV LANGUAGE en_US.UTF-8
# Create NB_USER user with UID=1000 and in the 'users' group
# but allow for non-initial launches of the notebook to have
# $HOME provided by the contents of a PV
RUN useradd -M -s /bin/bash -N -u $NB_UID $NB_USER && \
    chown -R ${NB_USER}:users /usr/local/bin && \
    mkdir -p $HOME && \
    chown -R ${NB_USER}:users ${HOME}
RUN export CLOUD_SDK_REPO="cloud-sdk-$(lsb_release -c -s)" && \
    echo "deb https://packages.cloud.google.com/apt $CLOUD_SDK_REPO main" > /etc/apt/sources.list.d/google-cloud-sdk.list && \
    curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add - && \
    apt-get update && \
    apt-get install -y google-cloud-sdk kubectl
# Install Tini - used as entrypoint for container
RUN cd /tmp && \
    wget --quiet https://github.com/krallin/tini/releases/download/v0.18.0/tini && \
    echo "12d20136605531b09a2c2dac02ccee85e1b874eb322ef6baf7561cd93f93c855 *tini" | sha256sum -c - && \
    mv tini /usr/local/bin/tini && \
    chmod +x /usr/local/bin/tini
# Install base python3 packages
RUN pip3 --no-cache-dir install \
    jupyter-console==6.0.0 \
    jupyterlab \
    kubeflow-fairing==1.0.1
RUN docker-credential-gcr configure-docker && chown ${NB_USER}:users $HOME/.docker/config.json
# Configure container startup
EXPOSE 8888
USER jovyan
ENTRYPOINT ["tini", "--"]
CMD ["sh","-c", "jupyter notebook --notebook-dir=/home/${NB_USER} --ip=0.0.0.0 --no-browser --allow-root --port=8888 --NotebookApp.token='' --NotebookApp.password='' --NotebookApp.allow_origin='*' --NotebookApp.base_url=${NB_PREFIX}"]


title: Correct way to feed data to RNN in PyTorch

body:
I have a data sequence a which is of shape [seq_len, 2], seq_len is the length of the sequence. There is time correlation among elements of a[:, 0] and a[:, 1], but a[:, 0] and a[:, 1] are independent of each other. For training I prepare data of shape [batch_size, seq_len, 2]. The initialization of BRNN that I use is
birnn_layer = nn.RNN(input_size=2, hidden_size=100, batch_first=True, bidirectional=True)

From the docs,

input_size – The number of expected features in the input x

hidden_size – The number of features in the hidden state h

What does "number of expected features" mean? Since there is correlation along the seq_len axis should my input_size be set as seq_len and the input be permuted? Thanks.


title: Teacher force training PyTorch

body:
I am trying to do a seq2seq prediction. For this, I have a LSTM layer followed by a fully connected layer. I employ Teacher training during the training phase and would like to skip this (I maybe wrong here) during testing phase. I have not found a direct way of doing this so I have taken the approach shown below.
def forward(self, inputs, future=0, teacher_force_ratio=0.2, target=None):
    outputs = []
    for idx in range(future):
        rnn_out, _ = self.rnn(inputs)
        output = self.fc1(rnn_out)
        if self.teacher_training:
            new_input = output if np.random.random() >= teacher_force_ratio else target[idx]
        else:
            new_input = output
        inputs = new_input

I use a bool variable teacher_training to check if Teacher training is needed or not. Is this correct? If yes, is there a better way to do this? Thanks.


title: Difference between loading my trained model in a pretrained model and loading it in not pretrained one?

body:
I trained an Inception_v3 for my task. I have 3 classes.
After training I try to test my trained model and I use following code to load my model:
model = models.inception_v3(pretrained=True, aux_logits=False)
model.fc = nn.Linear(model.fc.in_features, 3)
model.load_state_dict(torch.load(My Model Path.pth))

Download a pretrained Inception_v3, change output features and load my weight in this model.
I obtained very good results as I expect from validation phase.
If I use the same code but with pretrained=False the test go very bad.
model = models.inception_v3(pretrained=False, aux_logits=False)
model.fc = nn.Linear(model.fc.in_features, 3)
model.load_state_dict(torch.load(My Model Path.pth))

Since in the model I download I load my weights, there should be no difference between pretrained True or False.
Does anyone know what changes?


title: Ho to save all output images in CNN into a file

body:
I am running a CNN network and want to save output images (testing data) into a file. I made a function for visualization and saving. I am calling function in test_loader to save each of output images into a file from 1 to 100 which is total of testing data. I can just save the last output image in a test_loader loop. I am not sure how to save all 100 images.
Plot function is :
def stress_plot(y, y_hat):

    y_hat = y_hat[0,0].cpu().detach().numpy()
    y = y[0,0].cpu().detach().numpy()

    y_hat = denorm(y_hat)
    y = denorm(y)

    plt.figure()
    sns.heatmap(y, cmap='jet', square=True, xticklabels=False,
                yticklabels=False)
    plt.text(200,-20, "Real Stress", fontsize = 15, color='Black')
    plt.figure()
    sns.heatmap(y_hat, cmap='jet', square=True, xticklabels=False,
                yticklabels=False)
    plt.text(200,-20, "Predicted Stress", fontsize = 15, color='Black')
    plt.save('output/%d.png' % i)
    plt.show()

Call the function in test_loader:
model.eval()
    with torch.no_grad():
        for data in test_loader:
            x_test_batch, y_test_batch = data[0].to(device,
                dtype=torch.float), data[1].to(device, dtype=torch.float)

            y_test_pred = model(x_test_batch)
            mse_val_loss = criterion(y_test_batch, y_test_pred, x_test_batch, mse)
            stress_plot( y_test_batch, y_test_pred)


title: Convolutional Neural Networks batch size

body:
Question1.
Let us say the image have a shape of (batchsize=100,height=28,width=28,channel=1)
and if we put this image in the model CNN underneath,
class CNN(torch.nn.Module):

        def __init__(self):
            super().__init__()

            # ImgIn shape=(100, 28, 28, 1)
            #   Image shape after Conv     -> (100, 28, 28, 32)
            #    Image shape after Pool     -> (100, 14, 14, 32)
            self.layer1 = torch.nn.Sequential(
                torch.nn.Conv2d(1, 32, kernel_size=3, stride=1, padding=1),
                torch.nn.ReLU(),
                torch.nn.MaxPool2d(kernel_size=2, stride=2))

            # second layer
            # ImgIn shape=(100, 14, 14, 32)
            #   Image shape after Conv      ->(100, 14, 14, 64)
            #   Image shape after Pool      ->(100, 7, 7, 64)
            self.layer2 = torch.nn.Sequential(
                torch.nn.Conv2d(32, 64, kernel_size=3, stride=1, padding=1),
                torch.nn.ReLU(),
                torch.nn.MaxPool2d(kernel_size=2, stride=2))

            # THESE PART CONFUSE!!!
            self.fc = torch.nn.Linear(7 * 7 * 64, 10, bias=True)

            torch.nn.init.xavier_uniform_(self.fc.weight)

        def forward(self, x):
            out = self.layer1(x)
            out = self.layer2(out)
            out = out.view(out.size(0), -1)
            out = self.fc(out)
            return out

WHAT HAPPENS to the batchsize of the image after self.fc.
Is it size(batch_size=100,10)?
Qustion 2.
Also, I am confused of mini-batch. If there is 5 mini batch, and the loss is [5,-5,4,-4,0], then the average loss will be zero, then will the neural network stop training even if there is loss in each mini-batch?
Qustion 3.
Can neural network be expressed my complex matrix multiplication?


title: Why do semi-positive definite matrices have negative eigenvalues when I use the function torch.eig()?

body:
Why do semi-positive definite matrices have negative eigen values when I use the function torch.eig()? How can I resolve this problem?
tp1 = torch.tensor([[123., 43, 4], [-23, 44, -1]])
a = torch.mm(tp1.t(), tp1)
eigVals, eigVecs = torch.eig(a, eigenvectors=True)

Result for eigVals is:
tensor([[ 1.7055e+04, 0.0000e+00], [ 2.4053e+03, 0.0000e+00], [-7.5416e-06, 0.0000e+00]])


title: Why do sometimes CNN models predict just one class out of all others?

body:
I am relatively new to the deep learning landscape, so please don't be as mean as Reddit! It seems like a general question so I won't be giving my code here as it doesn't seem necessary (if it is, here's the link to colab)
A bit about the data: You can find the original data here. It is a downsized version of the original dataset of 82 GB.
Once I trained my CNN on this, it predicts 'No Diabetic Retinopathy' (No DR) every single time, leading to an accuracy of 73%. Is the reason for this is just the vast amount of No DR images or something else? I have no idea! The 5 classes I have for prediction are ["Mild", "Moderate", "No DR", "Proliferative DR", "Severe"].
It's probably just bad code, was hoping you guys could help


title: pytorch gathering from a 1d tensor

body:
I have a long list of values stored in a 1d tensor from which I like to gather values. Given a 2d tensor where the second dimension denotes a set of indices, I would like to compute a new 2d tensor where the second dimension are all the values gathered given by the indices. Basically, I have a batch of indices and I use those indices to obtain a batch of values from the 1d-tensor.
I am new to pytorch but I managed to compute exactly that using pytorch's gather function
list_values = torch.tensor([1, 2, 3, 4, 5, 6])
list_values = list_values.unsqueeze(0)
list_values = list_values.expand((2, 6))
indices = torch.tensor([[1, 2], [2, 3]])
result = torch.gather(list_values, 1, indices)

This works perfectly fince and gives the correct result. However, If I am not mistaken, the expand operation is quite expensive in terms of memory as the number of elements in "list_value" grows.
Is there a better solution ?


title: Manually changed quantized weights not reflecting in state_dict()

body:
PyTorch: 1.7.0
I modified the quantized weights a net post-quantization as follows:
# instantiate the quantized net (not shown here).

# get one of the conv layers
tmp = model_int8.state_dict()['features.0.weight']
scales = tmp.q_per_channel_scales()
zero_pts = tmp.q_per_channel_zero_points()
axis = tmp.q_per_channel_axis()

# get int repr
tmp_int8 = tmp.int_repr()

# change value (value change is dependent on the int8 value)
tmp_int8[0][0][0][0] = new_value

# how to convert tmp_int8 to torch.qint8 type?
new_tmp = torch._make_per_channel_quantized_tensor(tmp_int8, scales, zero_pts, axis)

# based on the above step:
model_int8.features[0].weight = new_tmp

However, model_int8.features[0].weight shows updated values, but model_int8.state_dict()['features.0.weight'] shows old vales.
I also tried saving the modified model and reloading, but the problem persists.
Question is which weight values are being used for inference? I do not see change in the classification results.


title: Generating a model's confidence from a (1,2) pytorch tensor

body:
So in my classification neural network, my final tensor is
a = tensor([[ 546.3831, -796.4016]], grad_fn=<AddmmBackward>)

I would get which category the program thinks it is by calling
a.max(1)

But is there a way to calculate the confidence in which the network has in this decision?
I am using this to generate a heat map type of thing, where the confidence of multiple images will come together to generate a heat map image.