title: How to compute the validation loss? (Simple linear regression)

body:
I am currently learning how to use PyTorch to build a neural network. I have learned keras before and I would like to do the same thing in PyTorch like 'model.fit' and plotting a graph containing both training loss and validation loss.
In order to know whether the model is underfitting or not, I have to plot a graph to compare the training loss and validation loss.
However, I cannot compute the right validation loss. I know that gradients should only be updated during training and it should not be updated during testing/validation. With no change in gradients, does it mean the loss will not change? Sorry, my concept is not clear enough. But I think not, loss should be computed by comparing expected output and prediction using loss function.
In my code, 80 datasets are used for training and 20 datasets are used for validation. In my code, the neural network is prediction this formula:  y =2X^3 + 7X^2 - 8*X + 120
It is easy to compute so I use this for learning how to build neural network through PyTorch.
Here is my code:
import torch
import torch.nn as nn    #neural network model
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import torch.nn.functional as F
from torch.autograd import Variable
from sklearn.preprocessing import MinMaxScaler

#Load datasets
dataset = pd.read_csv('test_100.csv')

X = dataset.iloc[:, :-1].values
Y = dataset.iloc[:, -1:].values

X_scaler = MinMaxScaler()
Y_scaler = MinMaxScaler()
print(X_scaler.fit(X))
print(Y_scaler.fit(Y))
X = X_scaler.transform(X)
Y = Y_scaler.transform(Y)

x_temp_train = X[:79]
y_temp_train = Y[:79]
x_temp_test = X[80:]
y_temp_test = Y[80:]

X_train = torch.FloatTensor(x_temp_train)
Y_train = torch.FloatTensor(y_temp_train)
X_test = torch.FloatTensor(x_temp_test)
Y_test = torch.FloatTensor(y_temp_test)

D_in = 1 # D_in is input features
H = 24 # H is hidden dimension
D_out = 1 # D_out is output features.

#Define a Artifical Neural Network model
class Net(nn.Module):
#------------------Two Layers------------------------------
    def __init__(self, D_in, H, D_out):
        super(Net, self).__init__()

        self.linear1 = nn.Linear(D_in, H)
        self.linear2 = nn.Linear(H, D_out)

    def forward(self, x):
        h_relu = self.linear1(x).clamp(min=0)
        prediction = self.linear2(h_relu)
        return prediction
model = Net(D_in, H, D_out)
print(model)

#Define a Loss function and optimizer
criterion = torch.nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.2) #2e-7

#Training
inputs = Variable(X_train)
outputs = Variable(Y_train)
inputs_val = Variable(X_test)
outputs_val = Variable(Y_test)
loss_values = []
val_values = []
epoch = []
for phase in ['train', 'val']:
    if phase == 'train':
        model.train()  # Set model to training mode
    else:
        optimizer.zero_grad() #zero the parameter gradients
        model.eval()   # Set model to evaluate mode
    for i in range(50):      #epoch=50
        if phase == 'train':
            model.train()
            prediction = model(inputs)
            loss = criterion(prediction, outputs)
            print('train loss')
            print(loss)
            loss_values.append(loss.detach())
            optimizer.zero_grad() #zero the parameter gradients
            epoch.append(i)
            loss.backward()       #compute gradients(dloss/dx)
            optimizer.step()      #updates the parameters
        elif phase == 'val':
            model.eval()
            prediction_val = model(inputs_val)
            loss_val = criterion(prediction_val, outputs_val)
            print('validation loss')
            print(loss)
            val_values.append(loss_val.detach())
            optimizer.zero_grad() #zero the parameter gradients

plt.plot(epoch,loss_values)
plt.plot(epoch, val_values)
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train','validation'], loc='upper left')
plt.show()

Here is the result:
train loss
tensor(0.9788, grad_fn=<MseLossBackward>)
tensor(2.0834, grad_fn=<MseLossBackward>)
tensor(3.2902, grad_fn=<MseLossBackward>)
tensor(0.8851, grad_fn=<MseLossBackward>)
tensor(0.0832, grad_fn=<MseLossBackward>)
tensor(0.0402, grad_fn=<MseLossBackward>)
tensor(0.0323, grad_fn=<MseLossBackward>)
tensor(0.0263, grad_fn=<MseLossBackward>)
tensor(0.0217, grad_fn=<MseLossBackward>)
tensor(0.0181, grad_fn=<MseLossBackward>)
tensor(0.0153, grad_fn=<MseLossBackward>)
tensor(0.0132, grad_fn=<MseLossBackward>)
tensor(0.0116, grad_fn=<MseLossBackward>)
tensor(0.0103, grad_fn=<MseLossBackward>)
tensor(0.0094, grad_fn=<MseLossBackward>)
tensor(0.0087, grad_fn=<MseLossBackward>)
tensor(0.0081, grad_fn=<MseLossBackward>)
tensor(0.0077, grad_fn=<MseLossBackward>)
tensor(0.0074, grad_fn=<MseLossBackward>)
tensor(0.0072, grad_fn=<MseLossBackward>)
tensor(0.0070, grad_fn=<MseLossBackward>)
tensor(0.0068, grad_fn=<MseLossBackward>)
tensor(0.0067, grad_fn=<MseLossBackward>)
tensor(0.0067, grad_fn=<MseLossBackward>)
tensor(0.0066, grad_fn=<MseLossBackward>)
tensor(0.0065, grad_fn=<MseLossBackward>)
tensor(0.0065, grad_fn=<MseLossBackward>)
tensor(0.0065, grad_fn=<MseLossBackward>)
tensor(0.0064, grad_fn=<MseLossBackward>)
tensor(0.0064, grad_fn=<MseLossBackward>)
tensor(0.0064, grad_fn=<MseLossBackward>)
tensor(0.0064, grad_fn=<MseLossBackward>)
tensor(0.0063, grad_fn=<MseLossBackward>)
tensor(0.0063, grad_fn=<MseLossBackward>)
tensor(0.0063, grad_fn=<MseLossBackward>)
tensor(0.0063, grad_fn=<MseLossBackward>)
tensor(0.0063, grad_fn=<MseLossBackward>)
tensor(0.0062, grad_fn=<MseLossBackward>)
tensor(0.0062, grad_fn=<MseLossBackward>)
tensor(0.0062, grad_fn=<MseLossBackward>)
tensor(0.0062, grad_fn=<MseLossBackward>)
tensor(0.0062, grad_fn=<MseLossBackward>)
tensor(0.0062, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)

validation loss
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)

Train Loss Vs. Validation Loss
The validation loss is a flat line. It is not what I want.


title: How can I shuffle the labels of a dataset?

body:
I have downloaded the MNIST dataset, using the following command:

train_dataset = dsets.MNIST(root='./data',
                            train=True,
                            transform=transforms.ToTensor(),
                            download=True)

I now need to run some experiments on this dataset (MNIST), but shuffling the labels of the training set. How can I shuffle/reassign them randomly? I have tried the following:

train_dataset = dsets.MNIST(root='./data',
                            train=True,
                            transform=transforms.ToTensor(),
                            target_transform=lambda y: torch.randint(0, 10, (1,)).item(),
                            download=True)

But I have noticed that what comes after the lambda function makes the labels shuffle during the training process, e.g. they change at every epoch. This way, I won't reach 100% training accuracy, which is what I am aiming for. How can I shuffle these labels in  a way that is completely random, making sure that these labels won't change during the training process?

Thank you!!


title: argparse unrecognized arguments

body:
i'am running pytorch DBPN(super resolution model) which the author of the paper  uploaded. But, i'm in trouble to training these model. Maybe, a problem is came from argparse. But i can't recognize how to fix it.. please help me, Thanks.

DBPN code link : https://github.com/alterzero/DBPN-Pytorch

`Testing started at 2:14 am..
C:\Users\CY\Anaconda3\envs\pytorch\python.exe "C:\Program Files\JetBrains\PyCharm Community Edition 2018.2.4\helpers\pycharm\_jb_pytest_runner.py" --path C:/Users/CY/Desktop/SuperResol/DBPN-Pytorch-master/main.py
Launching pytest with arguments C:/Users/CY/Desktop/SuperResol/DBPN-Pytorch-master/main.py in C:\Users\CY\Desktop\SuperResol\DBPN-Pytorch-master

============================= test session starts =============================
platform win32 -- Python 3.5.6, pytest-3.8.1, py-1.6.0, pluggy-0.7.1
rootdir: C:\Users\CY\Desktop\SuperResol\DBPN-Pytorch-master, inifile:usage: _jb_pytest_runner.py [-h] [--upscale_factor UPSCALE_FACTOR]
                            [--batchSize BATCHSIZE]
                            [--testBatchSize TESTBATCHSIZE]
                            [--nEpochs NEPOCHS] [--snapshots SNAPSHOTS]
                            [--lr LR] [--gpu_mode GPU_MODE]
                            [--threads THREADS] [--seed SEED] [--gpus GPUS]
                            [--data_dir DATA_DIR]
                            [--data_augmentation DATA_AUGMENTATION]
                            [--hr_train_dataset HR_TRAIN_DATASET]
                            [--train_dataset TRAIN_DATASET]
                            [--hr_test_dataset HR_TEST_DATASET]
                            [--test_dataset TEST_DATASET]
                            [--model_type MODEL_TYPE]
                            [--patch_size PATCH_SIZE]
                            [--pretrained_sr PRETRAINED_SR]
                            [--pretrained PRETRAINED]
                            [--save_folder SAVE_FOLDER] [--prefix PREFIX]
_jb_pytest_runner.py: error: unrecognized arguments: C:/Users/CY/Desktop/SuperResol/DBPN-Pytorch-master/main.py

main.py:None (main.py)
main.py:44: in <module>
    opt = parser.parse_args()
..\..\..\Anaconda3\envs\pytorch\lib\argparse.py:1729: in parse_args
    self.error(msg % ' '.join(argv))
..\..\..\Anaconda3\envs\pytorch\lib\argparse.py:2385: in error
    self.exit(2, _('%(prog)s: error: %(message)s\n') % args)
..\..\..\Anaconda3\envs\pytorch\lib\argparse.py:2372: in exit
    _sys.exit(status)
E   SystemExit: 2
collected 0 items / 1 errors

=================================== ERRORS ====================================
__________________________ ERROR collecting main.py ___________________________
main.py:44: in <module>
    opt = parser.parse_args()
..\..\..\Anaconda3\envs\pytorch\lib\argparse.py:1729: in parse_args
    self.error(msg % ' '.join(argv))
..\..\..\Anaconda3\envs\pytorch\lib\argparse.py:2385: in error
    self.exit(2, _('%(prog)s: error: %(message)s\n') % args)
..\..\..\Anaconda3\envs\pytorch\lib\argparse.py:2372: in exit
    _sys.exit(status)
E   SystemExit: 2
------------------------------- Captured stderr -------------------------------
usage: _jb_pytest_runner.py [-h] [--upscale_factor UPSCALE_FACTOR]
                            [--batchSize BATCHSIZE]
                            [--testBatchSize TESTBATCHSIZE]
                            [--nEpochs NEPOCHS] [--snapshots SNAPSHOTS]
                            [--lr LR] [--gpu_mode GPU_MODE]
                            [--threads THREADS] [--seed SEED] [--gpus GPUS]
                            [--data_dir DATA_DIR]
                            [--data_augmentation DATA_AUGMENTATION]
                            [--hr_train_dataset HR_TRAIN_DATASET]
                            [--train_dataset TRAIN_DATASET]
                            [--hr_test_dataset HR_TEST_DATASET]
                            [--test_dataset TEST_DATASET]
                            [--model_type MODEL_TYPE]
                            [--patch_size PATCH_SIZE]
                            [--pretrained_sr PRETRAINED_SR]
                            [--pretrained PRETRAINED]
                            [--save_folder SAVE_FOLDER] [--prefix PREFIX]
_jb_pytest_runner.py: error: unrecognized arguments: C:/Users/CY/Desktop/SuperResol/DBPN-Pytorch-master/main.py
!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!
=========================== 1 error in 0.52 seconds ===========================
Process finished with exit code 0
`


title: PyTorch - How to derive manually ResNet?

body:
I managed to find this implementation of ResNet-18:

def __init__(self):
    super(ResNet18,self).__init__()
    self.block1 = nn.Sequential(
        nn.Conv2d(1,64,kernel_size=2,stride=2,padding=3,bias=False),
        nn.BatchNorm2d(64),
        nn.ReLU(True)
    )
    self.block2 = nn.Sequential(
       nn.MaxPool2d(1,1),
       ResidualBlock(64,64),
       ResidualBlock(64,64,2)
    )
    self.block3 = nn.Sequential(
        ResidualBlock(64,128),
        ResidualBlock(128,128,2)
    )
    self.block4 = nn.Sequential(
        ResidualBlock(128,256),
        ResidualBlock(256,256,2)
    )
    self.block5 = nn.Sequential(
        ResidualBlock(256,512),
        ResidualBlock(512,512,2)
    )
    self.avgpool = nn.AvgPool2d(2)
    # vowel_diacritic
    self.fc1 = nn.Linear(512,11)
    # grapheme_root
    self.fc2 = nn.Linear(512,168)
    # consonant_diacritic
    self.fc3 = nn.Linear(512,7)

def forward(self,x):
    x = self.block1(x)
    x = self.block2(x)
    x = self.block3(x)
    x = self.block4(x)
    x = self.block5(x)
    x = self.avgpool(x)
    x = x.view(x.size(0),-1)
    x1 = self.fc1(x)
    x2 = self.fc2(x)
    x3 = self.fc3(x)
    return x1,x2,x3

class ResidualBlock(nn.Module):

    def __init__(self,in_channels,out_channels,stride=1,kernel_size=3,padding=1,bias=False):
        super(ResidualBlock,self).__init__()
        self.cnn1 = nn.Sequential(nn.Conv2d(in_channels,out_channels,kernel_size,stride,padding,bias=False), nn.BatchNorm2d(out_channels), nn.ReLU(True) )

        self.cnn2 = nn.Sequential( nn.Conv2d(out_channels,out_channels,kernel_size,1,padding,bias=False), nn.BatchNorm2d(out_channels) )

        if stride != 1 or in_channels != out_channels:
            self.shortcut = nn.Sequential( nn.Conv2d(in_channels,out_channels,kernel_size=1,stride=stride,bias=False), nn.BatchNorm2d(out_channels) )
        else:
            self.shortcut = nn.Sequential()

    def forward(self,x):
        residual = x
        x = self.cnn1(x)
        x = self.cnn2(x)
        x += self.shortcut(residual)
        x = nn.ReLU(True)(x)
        return x

I am still new to PyTorch and I am trying to obtain the architectures for ResNets 34, 50 and 101 in a similar format as above for ResNet18. Initially I thought that the only changes that I have to make are the last parameter of each block according to the source GitHub code at https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py, where for ResNet34 we have return _resnet('resnet34', BasicBlock, [3, 4, 6, 3], pretrained, progress, **kwargs).

Am I doing this wrong? I am still reading up on the documentation and trying to make sense of the methodologies, so some guidance will be deeply appreciated. :')


title: Running and building Pytorch on Google Colab

body:
I am trying to run a python package that requires pytorch-gpu. I have change the runtime type of my Colab notebook to GPU. When I run the command, I am facing the following error. Not sure if I am able to build pytorch on colab myself?

Traceback (most recent call last):
  File "inference_unet.py", line 9, in <module>
    import torchvision.transforms as transforms
  File "/usr/local/lib/python3.6/dist-packages/torchvision/__init__.py", line 10, in <module>
    from .extension import _HAS_OPS
  File "/usr/local/lib/python3.6/dist-packages/torchvision/extension.py", line 58, in <module>
    _check_cuda_version()
  File "/usr/local/lib/python3.6/dist-packages/torchvision/extension.py", line 54, in _check_cuda_version
    .format(t_major, t_minor, tv_major, tv_minor))
RuntimeError: Detected that PyTorch and torchvision were compiled with different CUDA versions. PyTorch has CUDA Version=10.2 and torchvision has CUDA Version=10.1. Please reinstall the torchvision that matches your PyTorch install.


title: Is there an equivalent function pytorch's loss fuction nn.crossEntropyLoss() in keras?

body:
I'm converting conv2d model written by pytorch to keras(Tensorflow 2.0).

I've almost done, but last question is delaying my work.

The title says it all: is there an equivalent function pytorch's loss fuction nn.crossEntropyLoss() in keras?

Thanks for your help.


title: pytorch recover from RuntimeError: CUDA error: device-side assert triggered without restarting script

body:
I suppose everybody who has worked with Pytorch knows the error
RuntimeError: CUDA error: device-side assert triggered
to some extend.
I'm generating a lot of data with GPU-Code in my script (200k+ long vectors), so it takes a while.
I'm doing this in batches via a generator as I do not have the memory to store all vectors in my GPU at once. The generator has the following structure:

    for i in range(0, len(inputs), batch_size):
        try:
            <generate the vectors>
            yield 1, <the vectors>  # Here it was successful
        except RuntimeError:
            print(f'could not generate vectors {index} to {index + batch_size}')
            yield 0, (i, i+ batch_size)  # Here the input was malformed

I know that some of the input is malformed to the point that generating vectors from it will fail with a runtime error and that's fine, it's not even 1% of my dataset. I want to get the indices and deal with it later.

Here's my problem
Once vector creation fails, the GPU is basically bricked and will respond to all requests with aforementioned error. Validating all input beforehand would be cumbersome and slow. I don't want to do it. I want to roll over all malformed inputs and deal with it later.

My question is
How can I recover the GPU from this bricked state as easy and fast as possible? All questions that I have found so far ask about fixing the underlying error, which I do not need to do. I just want to get on with generating vectors from my dataset.


title: Error : OSError: [WinError 126] Le module spécifié est introuvable

body:
I am trying to install pytorch on Windows 10 in a virtual env and I am getting this error :

(test) C:\Users\Emmanuelle>py
Python 3.7.7 (default, May  6 2020, 11:45:54) [MSC v.1916 64 bit (AMD64)] :: Anaconda, Inc. on win32

Warning:
This Python interpreter is in a conda environment, but the environment has
not been activated.  Libraries may fail to load.  To activate this environment
please see https://conda.io/activation

Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\Emmanuelle\Anaconda3\lib\site-packages\torch\__init__.py", line 81, in <module>
    ctypes.CDLL(dll)
  File "C:\Users\Emmanuelle\Anaconda3\lib\ctypes\__init__.py", line 364, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: [WinError 126] Le module spécifié est introuvable

How can I handle it ?


title: How can I implement multiple hidden layers in an RNN (PyTorch)?

body:
My Pytorch RNN for name classification does not allow me to choose multiple hidden layers. If I choose more than 1 layer I get the following error message:

  Traceback (most recent call last): File "TRAIN2.py", line 63,

  line 104, in training: y_hat = y_hat.view(batch_size, n_categories)
  RuntimeError: shape '[26718, 6]' is invalid for input of size 320616

I don't know exactly what the problem is but the input size is doubled when I choose 2 layers, tripled with 3 layers and so on.. My output size is 6 and when I choose 1 layer it works but with two layers there are twice as many inputs and I get the error and with 3 layers there are 3 times as many inputs. Shouldn't the input size stay the same regardless of the number of layers?

I am attaching the main file (where the error appears in the loss computation) and my network architecture.

for epoch in range(start_epoch, epochs):

   for batch_number, (x_batch, y_batch) in enumerate(batches):

      # Initialise the hidden layer
      hidden = model.initialise_hidden(batch_size=batch_size)

      if torch.cuda.is_available():
        x_batch = x_batch.to(device)
        y_batch = y_batch.to(device)

      # Model
      y_hat = rnn_gru(x_batch)

      # Loss
      y_hat = y_hat.view(batch_size, n_categories)
      loss = criterion(y_hat, y_batch)
      current_loss = current_loss + loss.item()

      # Restore Gradients
      optimizer.zero_grad()

      # Backward
      loss.backward()

      # Step
      optimizer.step()

And this is my model

class RNN_GRU(nn.Module):

  # input_size = Vocabulary size
  # hidden_size = Size of the hidden "units" in the GRU's
  # n_layers -> Connections between the GRU's
  # dropout -> fraction of nodes to be shut down

  def __init__(self, input_size, embedding_size, hidden_size, output_size, n_layers, dropout):
    super(RNN_GRU, self).__init__()

    # Members
    self.n_layers = n_layers
    self.hidden_size = hidden_size

    # 1) Embedding
    self.embedding = nn.Embedding(num_embeddings=input_size, embedding_dim=embedding_size)

    # 2) Packing
    #self.pack = nn.utils.rnn.pack_padded_sequence(batch_first=False)

    # 3) GRU
    self.gru = nn.GRU(input_size=embedding_size, hidden_size=hidden_size, num_layers=n_layers, bias=True, batch_first=False, dropout=dropout, bidirectional=False)

    # 4) Linear
    self.linear = nn.Linear(in_features=hidden_size, out_features=output_size, bias=True)

  def forward(self, x):

    # X -> [batch_size, n]
    #     [x11, x12, x13, ..., x1n1]
    # X:  [x21, x22, x23, ..., x2n2]
    #     [x31, x32, x33, ..., x3n3]
    # xYZ -> Name Y, character Z code
    # nY -> Length (num chars) of name Y. n1, n2, n3, ..., can be different (if no padding)

    batch_size = x.size(0)

    # 0.1) From the input name, get the embedding vector (for dimension requirements, it has to be transposed)
    input = self.embedding(x.t())

I am aware that using multiple layers does probably not increase performance but I want to use a non-zero dropout rate for which I'm being told that it is only active in all layers but the last and with only 1 layer therefore useless...

Thank you in advance for your help.


title: How to encode a RGB Image to an n_class One Hot Tensor in PyTorch

body:
So, I am working on a segmentation task and what I need to do is convert an RGB image to an n channel one hot matrix to be used as labels for a U-Net Model

What I have done is calculating the colors in the images. The number or colors are equivalent to the number of classes.

What I try do do in the PerPixelClassMatrix function is iterate through the Image and make a matrix of n dimensions of 0s and 1s since I have the color and class for each pixel.

import glob
from tqdm import tqdm
import numpy as np

class HotEncoder():
    def __init__(self, dir, extension, is_binary=True):
        self.dir = dir
        self.extension = extension
        self.is_binary = is_binary
        if is_binary:
            self.color = {(0, 0, 0): 1, (255, 255, 255): 2}
        else:
            self.color = dict()

    def gen_colors(self):
        """Iterates through the entire dataset and finds the total colours
            in the images so that they can be used to one hot the image matrix
            for the training data"""
        if self.is_binary:
            return self.color
        else:
            n_color=1
            images = glob.glob(self.dir + '/*.' + self.extension)
            for img in tqdm(images, desc="Generating Color Pallte to Hot Encode"):
                image = skimage.io.imread(img)
                shape_ = image.shape
                for x in range(shape_[0]):
                    for y in range(shape_[1]):
                        clr= tuple(image[x][y][:])
                        if clr not in self.color.keys():
                            self.color.update({n_color: clr})
                            n_color+=1
                        else:
                            pass
        return self.color

    def PerPixelClassMatrix(self, Image):
        """Takes an Image and returns a per pixel class
            identification map"""
        class_list= []
        class_mat= np.array([])
        shape_= Image.shape
        for x in range(shape_[0]):
            for y in range(shape_[1]):
                clr= tuple(Image[x][y][:])
                if clr in self.color.keys():
                    class_list.append(self.color[clr])
                else:
                    pass
        return class_list

I dont want to run an entire loop for generating a one hot image of n channel. Is there an easy way to construct such a matrix which the colors known.


title: Python coverage report covering only test file

body:
I’m pretty new to contributing to open source projects and am trying to get some coverage reports so I can find out what needs more / better testing. However, I am having trouble getting the full coverage of a test. This is for pytorch

For example, lets say I want to get the coverage report of test_indexing_py.

I run the command:

pytest test_indexing.py --cov=../ --cov-report=html

Resulting in this:

================================================= test session starts =================================================
platform win32 -- Python 3.7.4, pytest-5.2.1, py-1.8.0, pluggy-0.13.0
rootdir: C:\Projects\pytorch
plugins: hypothesis-5.4.1, arraydiff-0.3, cov-2.8.1, doctestplus-0.4.0, openfiles-0.4.0, remotedata-0.3.2
collected 62 items

test_indexing.py ............................s.................................                                  [100%]

----------- coverage: platform win32, python 3.7.4-final-0 -----------
Coverage HTML written to dir htmlcov

=========================================== 61 passed, 1 skipped in 50.43s ============================================

Ok, looks like the tests ran. Now when I check the html coverage report, I only get the coverage for the test file and not for the classes tested (the tests are ordered by coverage percentage).

As you can see, I am getting coverage for only test_indexing.py. How do I get the full coverage report including the classes tested?

Any guidance will be greatly appreciated.


title: pytorch multi-class lstm predicting all one class on testing

body:
I'm working on a project (my first AI project) and I've hit a bit of a wall. When performing testing on my trained classifier, it's predicting that everything is of class 1. Now the data set is heavily biased to class 1; however, I've implemented weights to compensate for this. Just concerned that I've coded this wrong or missed something. Please let me know if you see anything.

This is the setup and training

  batchSize = 50

trainingLoad = DataLoader(trainingData, shuffle = True, batch_size = batchSize, drop_last=True)
validationLoad = DataLoader(validationData, shuffle = True, batch_size = batchSize, drop_last=True)
testingLoad = DataLoader(testingData, shuffle = True, batch_size = batchSize, drop_last=True)

vocabularySize = len(wordToNoDict)
output = 3
embedding = 400
hiddenDimension = 524
layers = 4

classifierModel = Classifier.HateSpeechDetector(device, vocabularySize, output, embedding, hiddenDimension, layers)
classifierModel.to(device)

path = 'Program\data\state_dict2.pt'

weights = torch.tensor([1203/1203, 1203/15389, 1203/3407])
criterion = nn.CrossEntropyLoss(weight = weights)

trainClassifier(classifierModel, trainingLoad, validationLoad, device, batchSize, criterion, path)

test(classifierModel, path, testingLoad, batchSize, device, criterion)
def trainClassifier(model, trainingData, validationData, device, batchSize, criterion, path):
epochs = 5
counter = 0
testWithValiEvery = 10
clip = 5
valid_loss_min = np.Inf

lr=0.0001
optimizer = torch.optim.Adam(model.parameters(), lr=lr)

model.train()

for i in range(epochs):

    h = model.init_hidden(batchSize, device)
    for inputs, labels in trainingData:
        h = tuple([e.data for e in h])
        inputs, labels = inputs.to(device), labels.to(device)
        model.zero_grad()
        output, h = model(inputs, h)
        loss = criterion(output.squeeze(), labels.long())
        loss.backward()
        nn.utils.clip_grad_norm_(model.parameters(), clip)
        optimizer.step()
        counter += 1
        print(counter)

        if counter%testWithValiEvery == 0:
            print("validating")
            val_h = model.init_hidden(batchSize, device)
            val_losses = []
            model.eval()
            for inp, lab in validationData:
                val_h = tuple([each.data for each in val_h])
                inp, lab = inp.to(device), lab.to(device)

                out, val_h = model(inp, val_h)#

                val_loss = criterion(out.squeeze(), lab.long())
                val_losses.append(val_loss.item())

            model.train()
            print("Epoch: {}/{}...".format(i+1, epochs),
                "Step: {}...".format(counter),
                "Loss: {:.6f}...".format(loss.item()),
                "Val Loss: {:.6f}".format(np.mean(val_losses)))
            if np.mean(val_losses) <= valid_loss_min:
                torch.save(model.state_dict(), path)
                print('Validation loss decreased ({:.6f} --> {:.6f}).  Saving model ...'.format(valid_loss_min,np.mean(val_losses)))
                print('model saved')
                valid_loss_min = np.mean(val_losses)

This is the classifier - Fair amount of random commenting here where i've meddled with bits

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as op
import torchvision
from torch.utils.data import TensorDataset, DataLoader
from torchvision import transforms, datasets

class HateSpeechDetector(nn.Module):
    def __init__(self, device, vocabularySize, output, embedding, hidden, layers, dropProb=0.5):
        super(HateSpeechDetector, self).__init__()
        #Number of outputs (Classes/Categories)
        self.output = output
        #Number of layers in the LSTM
        self.numLayers = layers
        #Number of hidden neurons in each LSTM layer
        self.hiddenDimensions = hidden
        #Device being used for by model (CPU or GPU)
        self.device = device

        #Embedding layer finds correlations in words by converting word integers into vectors
        self.embedding = nn.Embedding(vocabularySize, embedding)
        #LSTM stores important data in memory, using it to help with future predictions
        self.lstm = nn.LSTM(embedding,hidden,layers,dropout=dropProb,batch_first=True)
        #Dropout is used to randomly drop nodes. This helps to prevent overfitting of the model during training
        self.dropout = nn.Dropout(dropProb)

        #Establishing 4 simple layers and a sigmoid output
        self.fc = nn.Linear(hidden, hidden)
        self.fc2 = nn.Linear(hidden, hidden)
        self.fc3 = nn.Linear(hidden, hidden)
        self.fc4 = nn.Linear(hidden, hidden)
        self.fc5 = nn.Linear(hidden, hidden)
        self.fc6 = nn.Linear(hidden, output)
        self.softmax = nn.Softmax(dim=2)

    def forward(self, x, hidden):
        batchSize = x.size(0)

        x = x.long()

        embeds = self.embedding(x)

        lstm_out, hidden = self.lstm(embeds, hidden)

        #Tensor changes here from 250,33,524 to 8250,524
        # lstm_out = lstm_out.contiguous().view(-1,self.hiddenDimensions)

        out = self.dropout(lstm_out)
        out = self.fc(out)
        out = self.fc2(out)
        out = self.fc3(out)
        out = self.fc4(out)
        out = self.fc5(out)
        out = self.fc6(out)

        out = self.softmax(out)

        out = out[:,-1,:]

        # myTensor = torch.Tensor([0,0,0])
        # newOut = torch.zeros(batchSize, self.output)
        # count = 0
        # row = 0

        # for tensor in out:
        #     if(count == 33):
        #         newOut[row] = myTensor/33
        #         myTensor = torch.Tensor([0,0,0])
        #         row += 1
        #         count = 0
        #     myTensor += tensor
        #     count += 1
        return out, hidden

    def init_hidden(self, batchSize, device):
        weight = next(self.parameters()).data

        hidden = (weight.new(self.numLayers, batchSize, self.hiddenDimensions).zero_().to(device), weight.new(self.numLayers, batchSize, self.hiddenDimensions).zero_().to(device))

        return hidden


title: Writing tensor to a file in a visually readable manner

body: In pytorch, I want to write a tensor to a file and visually read the file contents. For example, consider T = torch.tensor([3,4,5,6]). I want to write the tensor T to a file, say file_T.txt, and want to visually read the contents of the file_T.txt, which will be 3,4,5 and 6. How can I achieve this?


title: Why torch.nn package not supporting inputs that are a single sample

body:
I am trying to understand deep-learning with pytorch. I read the pytorch tutorial: https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html, and its written:

''torch.nn only supports mini-batches. The entire torch.nn package only supports inputs that are a mini-batch of samples, and not a single sample.
For example, nn.Conv2d will take in a 4D Tensor of nSamples x nChannels x Height x Width.
If you have a single sample, just use input.unsqueeze(0) to add a fake batch dimension.''

I am not sure to understand what it means.
Indeed, I have made a simple feed-forward neural network (cf below code), on which I used a really small dataset (the idea is to learn how it works without mini-batches first, not really to have anything useful), and hence don't need to use mini-batches. Hence, I introduce directly all the sample at each epochs. If I understand it correctly, I should add 'train_data = train_data.unsqueeze(0) '. But I am not sure where, as it seems to change the data size to 1. Also, it works without adding this line so why should I really use it?

Any help would be highly appreciated!

import torch
import torchvision
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import pandas as pd
import os
import numpy as np

# Download data
#...

# Construct network
len_input = len(data[0])
len_output = nbr_class
print('There is %d classes used for classification'%nbr_class)

#defining a new class: Net, that extended nn.Module
#setup the “skeleton” of our network architecture
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        #creation of fully connected. A fully connected neural network layer is represented by the nn.Linear object,
        #with the first argument in the definition being the number of nodes in layer l and the next argument being
        #the number of nodes in layer l+1
        self.fc1 = nn.Linear(len_input, 200)
        self.fc2 = nn.Linear(200, 200)
        self.fc3 = nn.Linear(200, len_output)

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return F.log_softmax(x)

# Create initial network
epochs = 3000
model = Net()
criterion = nn.NLLLoss()
optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.9)

# Train & test network
def train(data, target, epoch):
    model.train() # set the model in "training mode"
    # run the main training loop
    #no need batch size as small number of sample! like this the process is more exact
    #zero our gradients before calling .backward() which is necessary for new sum of gradients
    optimizer.zero_grad()
    target_pred = model(data)
    loss = criterion(target_pred, target)
    #Propagate the gradients back through the network
    loss.backward()
    #update the weights
    #tell our optimizer to “step”, meaning that the weights will be updated using the calculated gradients
    #according to our rule. perform model parameter update (update weights)
    optimizer.step()
    # for graphing puposes
    loss_array.append(loss.data[0])

    if epoch in list(range(0,20000,100)):
        print('Train Epoch: {} \tLoss: {:.6f}'.format(epoch, loss.data[0]))

def test(epoch, test_data, test_target):
    #eval mode to turn Dropout and BatchNorm off
    model.eval()
    test_loss = 0
    correct = 0
    test_target_pred = model(test_data)
    criterion = nn.NLLLoss()
    # sum up batch loss
    test_loss += criterion(test_target_pred, test_target).data[0]
    pred = test_target_pred.data.max(1)[1]  # get the index of the max log-probability
    correct += pred.eq(test_target.data).sum()
    if epoch in list(range(0,20000,100)):
        print('\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\n'.format(
            test_loss, correct, len(test_target), 100. * correct / len(test_target)))

if __name__ == '__main__':
    for epoch in range(epochs):
        train(data = train_data, target=train_target, epoch=epoch)
        test(epoch, test_data, test_target)


title: Outer sum, etc. in pytorch

body:
Numpy offers optimized outer operations for any RxR -> R function, like np.multiply.outer or np.subtract.outer, with the behaviour:

>>> np.subtract.outer([6, 5, 4], [3, 2, 1])
array([[3, 4, 5],
       [2, 3, 4],
       [1, 2, 3]])

Pytorch  does not seem to offer such a feature (or I have missed it).
What is the best / usual / fastest / cleanest way to do so with torch tensors?


title: Is there any way to convert pretrained model from PyTorch to ONNX?

body:
I trained StarGAN model on my custom dataset.
And I need to convert this model from .pth(Pytorch) to .pb for using on Android studio.
I searched a lot and I found some ways for conversion.
However all solutions don't work on my case.

I tried on small network that includes only one nn.Linear layer.
On this network, solutions work very well!

I think, my network includes Conv2D layer and MaxPooling2D layer so conversion processing doesn't work.

First, this is my network(StarGAN).

import torch
import torch.nn as nn
import numpy as np

class ResidualBlock(nn.Module):
    def __init__(self, dim_in, dim_out):
        super(ResidualBlock, self).__init__()
        self.main = nn.Sequential(
            nn.Conv2d(dim_in, dim_out, kernel_size=3, stride=1, padding=1, bias=False),
            nn.InstanceNorm2d(dim_out, affine=True, track_running_stats=True),
            nn.ReLU(inplace=True),
            nn.Conv2d(dim_out, dim_out, kernel_size=3, stride=1, padding=1, bias=False),
            nn.InstanceNorm2d(dim_out, affine=True, track_running_stats=True))

    def forward(self, x):
        return x + self.main(x)

class Generator(nn.Module):
    def __init__(self, conv_dim=64, c_dim=5, repeat_num=6):
        super(Generator, self).__init__()

        layers = []
        layers.append(nn.Conv2d(3 + c_dim, conv_dim, kernel_size=7, stride=1, padding=3, bias=False))
        layers.append(nn.InstanceNorm2d(conv_dim, affine=True, track_running_stats=True))
        layers.append(nn.ReLU(inplace=True))

        curr_dim = conv_dim
        for _ in range(2):
            layers.append(nn.Conv2d(curr_dim, curr_dim * 2, kernel_size=4, stride=2, padding=1, bias=False))
            layers.append(nn.InstanceNorm2d(curr_dim * 2, affine=True, track_running_stats=True))
            layers.append(nn.ReLU(inplace=True))
            curr_dim = curr_dim * 2

        for _ in range(repeat_num):
            layers.append(ResidualBlock(dim_in=curr_dim, dim_out=curr_dim))

        for _ in range(2):
            layers.append(nn.ConvTranspose2d(curr_dim, curr_dim // 2, kernel_size=4, stride=2, padding=1, bias=False))
            layers.append(nn.InstanceNorm2d(curr_dim // 2, affine=True, track_running_stats=True))
            layers.append(nn.ReLU(inplace=True))
            curr_dim = curr_dim // 2

        layers.append(nn.Conv2d(curr_dim, 3, kernel_size=7, stride=1, padding=3, bias=False))
        layers.append(nn.Tanh())
        self.main = nn.Sequential(*layers)

    def forward(self, x, c):
        c = c.view(c.size(0), c.size(1), 1, 1)
        c = c.repeat(1, 1, x.size(2), x.size(3))
        x = torch.cat([x, c], dim=1)
        return self.main(x)

class Discriminator(nn.Module):
    def __init__(self, image_size=128, conv_dim=64, c_dim=5, repeat_num=6):
        super(Discriminator, self).__init__()
        layers = []
        layers.append(nn.Conv2d(3, conv_dim, kernel_size=4, stride=2, padding=1))
        layers.append(nn.LeakyReLU(0.01))

        curr_dim = conv_dim
        for _ in range(1, repeat_num):
            layers.append(nn.Conv2d(curr_dim, curr_dim * 2, kernel_size=4, stride=2, padding=1))
            layers.append(nn.LeakyReLU(0.01))
            curr_dim = curr_dim * 2

        kernel_size = int(image_size / np.power(2, repeat_num))
        self.main = nn.Sequential(*layers)
        self.conv1 = nn.Conv2d(curr_dim, 1, kernel_size=3, stride=1, padding=1, bias=False)
        self.conv2 = nn.Conv2d(curr_dim, c_dim, kernel_size=kernel_size, bias=False)

    def forward(self, x):
        h = self.main(x)
        out_src = self.conv1(h)
        out_cls = self.conv2(h)
        return out_src, out_cls.view(out_cls.size(0), out_cls.size(1))

And this is the error message.

TypeError: object of type 'torch._C.Value' has no len() (occurred when translating repeat)

Is there any way for conversion? Help me.


title: GroupNorm is considerably slower and consumes higher GPU memory than BatchNorm in Pytorch

body:
I use GroupNorm in pytorch instead of BatchNorm and keep all the others (network architecture) unchanged. It shows that in Imagenet dataset, using resnet50 architecture, GroupNorm is 40% slower than BatchNorm, and consumes 33% more GPU memory than BatchNorm. I am really confused because GroupNorm shouldn’t need more calculation than BatchNorm. The details are listed below.

For details of Group Normalization, one can see this paper: https://arxiv.org/pdf/1803.08494.pdf

For BatchNorm, one minibatch consumes 12.8 seconds with GPU memory 7.51GB;

For GroupNorm, one minibatch consumes 17.9 seconds with GPU memory 10.02GB.

I use the following code to convert all the BatchNorm layers to GroupNorm layers.

def convert_bn_model_to_gn(module, num_groups=16):
"""
Recursively traverse module and its children to replace all instances of
``torch.nn.modules.batchnorm._BatchNorm`` with :class:`torch.nn.GroupNorm`.
Args:
    module: your network module
    num_groups: num_groups of GN
"""
mod = module
if isinstance(module, nn.modules.batchnorm._BatchNorm):
    mod = nn.GroupNorm(num_groups, module.num_features,
                       eps=module.eps, affine=module.affine)
    # mod = nn.modules.linear.Identity()
    if module.affine:
        mod.weight.data = module.weight.data.clone().detach()
        mod.bias.data = module.bias.data.clone().detach()
for name, child in module.named_children():
    mod.add_module(name, convert_bn_model_to_gn(
        child, num_groups=num_groups))
del module
return mod


title: Feeding Multiple Inputs to LSTM for Time-Series Forecasting using PyTorch

body:
I'm currently working on building an LSTM network to forecast time-series data using PyTorch. Following Roman's blog post, I implemented a simple LSTM for univariate time-series data, please see the class definitions below. However, it's been a few days since I ground to a halt on adding more features to the input data, say an hour of the day, day of the week, week of the year, and sorts.
class Model(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(Model, self).__init__()
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.output_size = output_size
        self.lstm = nn.LSTMCell(self.input_size, self.hidden_size)
        self.linear = nn.Linear(self.hidden_size, self.output_size)

    def forward(self, input, future=0, y=None):
        outputs = []

        # reset the state of LSTM
        # the state is kept till the end of the sequence
        h_t = torch.zeros(input.size(0), self.hidden_size, dtype=torch.float32)
        c_t = torch.zeros(input.size(0), self.hidden_size, dtype=torch.float32)

        for i, input_t in enumerate(input.chunk(input.size(1), dim=1)):
            h_t, c_t = self.lstm(input_t, (h_t, c_t))
            output = self.linear(h_t)
            outputs += [output]

        for i in range(future):
            if y is not None and random.random() > 0.5:
                output = y[:, [i]]  # teacher forcing
            h_t, c_t = self.lstm(output, (h_t, c_t))
            output = self.linear(h_t)
            outputs += [output]
        outputs = torch.stack(outputs, 1).squeeze(2)
        return outputs

class Optimization:
    "A helper class to train, test and diagnose the LSTM"

    def __init__(self, model, loss_fn, optimizer, scheduler):
        self.model = model
        self.loss_fn = loss_fn
        self.optimizer = optimizer
        self.scheduler = scheduler
        self.train_losses = []
        self.val_losses = []
        self.futures = []

    @staticmethod
    def generate_batch_data(x, y, batch_size):
        for batch, i in enumerate(range(0, len(x) - batch_size, batch_size)):
            x_batch = x[i : i + batch_size]
            y_batch = y[i : i + batch_size]
            yield x_batch, y_batch, batch

    def train(
        self,
        x_train,
        y_train,
        x_val=None,
        y_val=None,
        batch_size=100,
        n_epochs=20,
        dropout=0.2,
        do_teacher_forcing=None,
    ):
        seq_len = x_train.shape[1]
        for epoch in range(n_epochs):
            start_time = time.time()
            self.futures = []

            train_loss = 0
            for x_batch, y_batch, batch in self.generate_batch_data(x_train, y_train, batch_size):
                y_pred = self._predict(x_batch, y_batch, seq_len, do_teacher_forcing)
                self.optimizer.zero_grad()
                loss = self.loss_fn(y_pred, y_batch)
                loss.backward()
                self.optimizer.step()
                train_loss += loss.item()
            self.scheduler.step()
            train_loss /= batch
            self.train_losses.append(train_loss)

            self._validation(x_val, y_val, batch_size)

            elapsed = time.time() - start_time
            print(
                "Epoch %d Train loss: %.2f. Validation loss: %.2f. Avg future: %.2f. Elapsed time: %.2fs."
                % (epoch + 1, train_loss, self.val_losses[-1], np.average(self.futures), elapsed)
            )

    def _predict(self, x_batch, y_batch, seq_len, do_teacher_forcing):
        if do_teacher_forcing:
            future = random.randint(1, int(seq_len) / 2)
            limit = x_batch.size(1) - future
            y_pred = self.model(x_batch[:, :limit], future=future, y=y_batch[:, limit:])
        else:
            future = 0
            y_pred = self.model(x_batch)
        self.futures.append(future)
        return y_pred

    def _validation(self, x_val, y_val, batch_size):
        if x_val is None or y_val is None:
            return
        with torch.no_grad():
            val_loss = 0
            batch = 1
            for x_batch, y_batch, batch in self.generate_batch_data(x_val, y_val, batch_size):
                y_pred = self.model(x_batch)
                loss = self.loss_fn(y_pred, y_batch)
                val_loss += loss.item()
            val_loss /= batch
            self.val_losses.append(val_loss)

    def evaluate(self, x_test, y_test, batch_size, future=1):
        with torch.no_grad():
            test_loss = 0
            actual, predicted = [], []
            for x_batch, y_batch, batch in self.generate_batch_data(x_test, y_test, batch_size):
                y_pred = self.model(x_batch, future=future)
                y_pred = (
                    y_pred[:, -len(y_batch) :] if y_pred.shape[1] > y_batch.shape[1] else y_pred
                )
                loss = self.loss_fn(y_pred, y_batch)
                test_loss += loss.item()
                actual += torch.squeeze(y_batch[:, -1]).data.cpu().numpy().tolist()
                predicted += torch.squeeze(y_pred[:, -1]).data.cpu().numpy().tolist()
            test_loss /= batch
            return actual, predicted, test_loss

    def plot_losses(self):
        plt.plot(self.train_losses, label="Training loss")
        plt.plot(self.val_losses, label="Validation loss")
        plt.legend()
        plt.title("Losses")

You can find some of the helper functions that help me split and format data before feeding it to my LSTM network.
def to_dataframe(actual, predicted):
    return pd.DataFrame({"value": actual, "prediction": predicted})

def inverse_transform(scaler, df, columns):
    for col in columns:
        df[col] = scaler.inverse_transform(df[col])
    return df

def split_sequences(sequences, n_steps):
    X, y = list(), list()
    for i in range(len(sequences)):
        # find the end of this pattern
        end_ix = i + n_steps
        # check if we are beyond the dataset
        if end_ix > len(sequences):
            break
        # gather input and output parts of the pattern
        seq_x, seq_y = sequences[i:end_ix, :-1], sequences[end_ix-1, -1]
        X.append(seq_x)
        y.append(seq_y)
    return array(X), array(y)

def train_val_test_split_new(df, test_ratio=0.2, seq_len = 100):
    y = df['value']
    X = df.drop(columns = ['value'])
    tarin_ratio = 1 - test_ratio
    val_ratio = 1 - ((train_ratio - test_ratio) / train_ratio)

    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=test_ratio)
    X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=val_ratio)

    return X_train, y_train, X_val, y_val, X_test, y_test

I use the following data frames to train my model.
# df_train
value   weekday monthday    hour
timestamp
2014-07-01 00:00:00 10844   1   1   0
2014-07-01 00:30:00 8127    1   1   0
2014-07-01 01:00:00 6210    1   1   1
2014-07-01 01:30:00 4656    1   1   1
2014-07-01 02:00:00 3820    1   1   2
... ... ... ... ...
2015-01-31 21:30:00 24670   5   31  21
2015-01-31 22:00:00 25721   5   31  22
2015-01-31 22:30:00 27309   5   31  22
2015-01-31 23:00:00 26591   5   31  23
2015-01-31 23:30:00 26288   5   31  23
10320 rows × 4 columns

# x_train
weekday monthday    hour
timestamp
2014-08-26 16:30:00 1   26  16
2014-08-18 16:30:00 0   18  16
2014-10-22 20:00:00 2   22  20
2014-12-10 08:00:00 2   10  8
2014-07-27 22:00:00 6   27  22
... ... ... ...
2014-08-24 05:30:00 6   24  5
2014-11-24 12:00:00 0   24  12
2014-12-18 06:00:00 3   18  6
2014-07-27 17:00:00 6   27  17
2014-12-05 21:00:00 4   5   21
6192 rows × 3 columns

# y_train
timestamp
2014-08-26 16:30:00    14083
2014-08-18 16:30:00    14465
2014-10-22 20:00:00    25195
2014-12-10 08:00:00    21348
2014-07-27 22:00:00    16356
                       ...
2014-08-24 05:30:00     2948
2014-11-24 12:00:00    16292
2014-12-18 06:00:00     7029
2014-07-27 17:00:00    18883
2014-12-05 21:00:00    26284
Name: value, Length: 6192, dtype: int64

After transforming and splitting time-series data into smaller batches, the training data set for X and y becomes as follows:
X_data shape is (6093, 100, 3)
y_data shape is (6093,)
tensor([[[-1.0097,  1.1510,  0.6508],
         [-1.5126,  0.2492,  0.6508],
         [-0.5069,  0.7001,  1.2238],
         ...,
         [ 1.5044, -1.4417, -1.6413],
         [ 1.0016, -0.0890,  0.7941],
         [ 1.5044, -0.9908, -0.2087]],

        [[-1.5126,  0.2492,  0.6508],
         [-0.5069,  0.7001,  1.2238],
         [-0.5069, -0.6526, -0.4952],
         ...,
         [ 1.0016, -0.0890,  0.7941],
         [ 1.5044, -0.9908, -0.2087],
         [ 0.4988,  0.5874,  0.5076]],

        [[-0.5069,  0.7001,  1.2238],
         [-0.5069, -0.6526, -0.4952],
         [ 1.5044,  1.2637,  1.5104],
         ...,
         [ 1.5044, -0.9908, -0.2087],
         [ 0.4988,  0.5874,  0.5076],
         [ 0.4988,  0.5874, -0.6385]],

        ...,

        [[ 1.0016,  0.9255, -1.2115],
         [-1.0097, -0.9908,  1.0806],
         [-0.0041,  0.8128,  0.3643],
         ...,
         [ 1.5044,  0.9255, -0.9250],
         [-1.5126,  0.9255,  0.0778],
         [-0.0041,  0.2492, -0.7818]],

        [[-1.0097, -0.9908,  1.0806],
         [-0.0041,  0.8128,  0.3643],
         [-0.5069,  1.3765, -0.0655],
         ...,
         [-1.5126,  0.9255,  0.0778],
         [-0.0041,  0.2492, -0.7818],
         [ 1.5044,  1.2637,  0.7941]],

        [[-0.0041,  0.8128,  0.3643],
         [-0.5069,  1.3765, -0.0655],
         [-0.0041, -1.6672, -0.4952],
         ...,
         [-0.0041,  0.2492, -0.7818],
         [ 1.5044,  1.2637,  0.7941],
         [ 0.4988, -1.2163,  1.3671]]])
tensor([ 0.4424,  0.1169,  0.0148,  ..., -1.1653,  0.5394,  1.6037])

Finally, just to check if the dimensions of all these training, validation, and test datasets are correct, I print out their shapes.
train shape is: torch.Size([6093, 100, 3])
train label shape is: torch.Size([6093])
val shape is: torch.Size([1965, 100, 3])
val label shape is: torch.Size([1965])
test shape is: torch.Size([1965, 100, 3])
test label shape is: torch.Size([1965])

When I try to build the model as follows, I end up getting a RuntimeError pointing at inconsistent input sizes.
model_params = {'train_ratio': 0.8,
                'validation_ratio': 0.2,
                'sequence_length': 100,
                'teacher_forcing': False,
                'dropout_rate': 0.2,
                'batch_size': 100,
                'num_of_epochs': 5,
                'hidden_size': 24,
                'n_features': 3,
                'learning_rate': 1e-3
               }

train_ratio = model_params['train_ratio']
val_ratio = model_params['validation_ratio']
seq_len = model_params['sequence_length']
teacher_forcing = model_params['teacher_forcing']
dropout_rate = model_params['dropout_rate']
batch_size = model_params['batch_size']
n_epochs = model_params['num_of_epochs']
hidden_size = model_params['hidden_size']
n_features = model_params['n_features']
lr = model_params['learning_rate']

model = Model(input_size=n_features, hidden_size=hidden_size, output_size=1)
loss_fn = nn.MSELoss()
optimizer = optim.Adam(model.parameters(), lr=lr)
scheduler = optim.lr_scheduler.StepLR(optimizer, step_size=5, gamma=0.1)
optimization = Optimization(model, loss_fn, optimizer, scheduler)

start_time = datetime.now()
optimization.train(x_train, y_train, x_val, y_val,
                     batch_size=batch_size,
                     n_epochs=n_epochs,
                     dropout=dropout_rate,
                     do_teacher_forcing=teacher_forcing)

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-192-6fc406c0113d> in <module>
      6
      7 start_time = datetime.now()
----> 8 optimization.train(x_train, y_train, x_val, y_val,
      9                      batch_size=batch_size,
     10                      n_epochs=n_epochs,

<ipython-input-189-c18d20430910> in train(self, x_train, y_train, x_val, y_val, batch_size, n_epochs, dropout, do_teacher_forcing)
     68             train_loss = 0
     69             for x_batch, y_batch, batch in self.generate_batch_data(x_train, y_train, batch_size):
---> 70                 y_pred = self._predict(x_batch, y_batch, seq_len, do_teacher_forcing)
     71                 self.optimizer.zero_grad()
     72                 loss = self.loss_fn(y_pred, y_batch)

<ipython-input-189-c18d20430910> in _predict(self, x_batch, y_batch, seq_len, do_teacher_forcing)
     93         else:
     94             future = 0
---> 95             y_pred = self.model(x_batch)
     96         self.futures.append(future)
     97         return y_pred

~\Anaconda3\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

<ipython-input-189-c18d20430910> in forward(self, input, future, y)
     17
     18         for i, input_t in enumerate(input.chunk(input.size(1), dim=1)):
---> 19             h_t, c_t = self.lstm(input_t, (h_t, c_t))
     20             output = self.linear(h_t)
     21             outputs += [output]

~\Anaconda3\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

~\Anaconda3\lib\site-packages\torch\nn\modules\rnn.py in forward(self, input, hx)
    963
    964     def forward(self, input: Tensor, hx: Optional[Tuple[Tensor, Tensor]] = None) -> Tuple[Tensor, Tensor]:
--> 965         self.check_forward_input(input)
    966         if hx is None:
    967             zeros = torch.zeros(input.size(0), self.hidden_size, dtype=input.dtype, device=input.device)

~\Anaconda3\lib\site-packages\torch\nn\modules\rnn.py in check_forward_input(self, input)
    789     def check_forward_input(self, input: Tensor) -> None:
    790         if input.size(1) != self.input_size:
--> 791             raise RuntimeError(
    792                 "input has inconsistent input_size: got {}, expected {}".format(
    793                     input.size(1), self.input_size))

RuntimeError: input has inconsistent input_size: got 1, expected 3

I suspect my current LSTM model class does not support data with multiple features, and I've been trying out different approaches lately with no luck so far. Feel free to share your thoughts or point me in the right direction that could help me solve this problem.
As suggested by @stackoverflowuser2010, I printed out the shapes of the tensors input_t, h_t and c_t that is fed into the forward step before the error is thrown.
input_t
torch.Size([100, 1, 3])
h_t
torch.Size([100, 24])
c_t
torch.Size([100, 24])


title: PyTorch runtime error: expected argument to have type long, but got CPUType instead

body:
I'm new to PyTorch and going through this tutorial on the transformer model. I'm using PyCharm on Win10.
For now, I've basically just copy-pasted the example code, but I'm getting the following error:

  RuntimeError: Expected tensor for argument #1 'indices' to have scalar type Long; but got CPUType instead (while checking arguments for embedding)

It seems to be coming from this line

  def encode(self, src, src_mask):
         return self.encodder(self.src_embed(src), src_mask)

Tbh, I'm not even sure what this means, let alone how I should go about fixing it.
What's a CPUType? When did I create a variable of such type? From looking at the code I'm only using tensors (or numpy arrays)

here's the full error message:

  C:...\Python\Python37\lib\site-packages\torch\nn_reduction.py:46: UserWarning: size_average and reduce args will be deprecated, please use reduction='sum' instead.
    warnings.warn(warning.format(ret))
  C:/.../PycharmProjects/Transformer/all_the_code.py:263: UserWarning: nn.init.xavier_uniform is now deprecated in favor of nn.init.xavier_uniform_.
    nn.init.xavier_uniform(p)
  Traceback (most recent call last):
   File "C:/.../PycharmProjects/Transformer/all_the_code.py", line 421, in
     SimpleLossCompute(model.generator, criterion, model_opt))
   File "C:/.../PycharmProjects/Transformer/all_the_code.py", line 297, in run_epoch
    batch.src_mask, batch.trg_mask)
   File "C:/.../PycharmProjects/Transformer/all_the_code.py", line 30,  in forward
    return self.decode(self.encode(src, src_mask), src_mask,
  File "C:/.../PycharmProjects/Transformer/all_the_code.py", line 34, in encode
     return self.encoder(self.src_embed(src), src_mask)
  File "C:...\Python\Python37\lib\site-packages\torch\nn\modules\module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "C:...\Python\Python37\lib\site-packages\torch\nn\modules\container.py", line 92, in forward
    input = module(input)
  File "C:...\Python\Python37\lib\site-packages\torch\nn\modules\module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "C:/.../PycharmProjects/Transformer/all_the_code.py", line 218, in forward
    return self.lut(x) * math.sqrt(self.d_model)
  File "C:...\Python\Python37\lib\site-packages\torch\nn\modules\module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "C:...\Python\Python37\lib\site-packages\torch\nn\modules\sparse.py", line 117, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "C:...\Python\Python37\lib\site-packages\torch\nn\functional.py", line 1506, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)


title: How do you change require_grad to false for each parameters in your model in pytorch?

body:
My code is below which I thought would do what I want but the output shows require_grad didn't change to false.

import torch
import torch.nn as nn

encoder = nn.Sequential( nn.Conv2d(1, 4, 1), nn.Sigmoid())

for params in encoder.parameters():

    params.require_grad = False
    print(params.requires_grad) # prints two True statements?

What am I doing wrong?


title: OverflowError: (34, 'Numerical result out of range') in PyTorch

body:
I am getting the following error (see the stacktrace) when I ran my code in a different GPU (Tesla K-20, cuda 7.5 installed, 6GB memory). Code works fine if I run in GeForce 1080 or Titan X GPU.

Stacktrace:

File "code/source/main.py", line 68, in <module>
    train.train_epochs(train_batches, dev_batches, args.epochs)
  File "/gpfs/home/g/e/geniiexe/BigRed2/code/source/train.py", line 34, in train_epochs
    losses = self.train(train_batches, dev_batches, (epoch + 1))
  File "/gpfs/home/g/e/geniiexe/BigRed2/code/source/train.py", line 76, in train
    self.optimizer.step()
  File "/gpfs/home/g/e/geniiexe/BigRed2/anaconda3/lib/python3.5/site-packages/torch/optim/adam.py", line 70, in step
    bias_correction1 = 1 - beta1 ** state['step']
OverflowError: (34, 'Numerical result out of range')

So, what can be the reason to get such error in a different GPU (Tesla K-20) while it works fine in GeForce or Titan X GPU? Moreover what the error means? Is it related to memory overflow which I don't think so.


title: Seq2seq multiple input features (Passing multiple word/word tokens as input)

body:
Is there a way to pass extra feature tokens along with the existing word token (training features/source file vocabulary) and feed it to the encoder RNN of seq2seq?. Since, it currently accepts only one word token from the sentence at a time.

Let me put this in a more concrete fashion; Consider the example of machine translation/nmt - say I have 2 more feature columns for the corresponding source vocabulary set( Feature1 here ). For example, consider this below:

+---------+----------+----------+
|Feature1 | Feature2 | Feature3 |
+---------+----------+----------+
|word1    |    x     |     a    |
|word2    |    y     |     b    |
|word3    |    y     |     c    |
|.        |          |          |
|.        |          |          |
+---------+----------+----------+

To summarise, currently seq2seq dataset is the parallel data corpora has a one-to one mapping between he source feature(vocabulary,i.e Feature1 alone) and the target(label/vocabulary). I'm looking for a way to map more than one feature(i.e Feature1, Feature2,Feature3) to the target(label/vocabulary).

Moreover, I believe this is glossed over in the seq2seq-pytorch tutorial(https://github.com/spro/practical-pytorch/blob/master/seq2seq-translation/seq2seq-translation.ipynb) as quoted below:

  When using a single RNN, there is a one-to-one relationship between
  inputs and outputs. We would quickly run into problems with different
  sequence orders and lengths that are common during translation…….With
  the seq2seq model, by encoding many inputs into one vector, and
  decoding from one vector into many outputs, we are freed from the
  constraints of sequence order and length. The encoded sequence is
  represented by a single vector, a single point in some N dimensional
  space of sequences. In an ideal case, this point can be considered the
  "meaning" of the sequence.

Furthermore, I tried tensorflow and took me a lot of time to debug and make appropriate changes and got nowhere. And heard from my colleagues that pytorch would have the flexibility to do so and would be worth checking out.

Please share your thoughts on how to achieve the same in tensorflow or pytorch. Would be great of anyone tells how to practically implement/get this done. Thanks in advance.


title: pytorch predictions stability

body:
This is my predict function. is there anything wrong with this? Predictions are not stable, everytime I run on same data, I get different predictions.

def predict(model, device, inputs, batch_size=1024):
    model = model.to(device)
    dataset = torch.utils.data.TensorDataset(*inputs)
    loader = torch.utils.data.DataLoader(
                    dataset,
                    batch_size=batch_size,
                    pin_memory=False
                )

    predictions = []

    for i, batch in enumerate(loader):
        with torch.no_grad():
            pred = model(*(item.to(device) for item in batch))
            pred = pred.detach().cpu().numpy()
        predictions.append(pred)
    return np.concatenate(predictions)


While running smdataparallel, I see following error
# python
Python 3.6.10 |Anaconda, Inc.| (default, May  8 2020, 02:54:21)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import smdistributed.dataparallel.torch.distributed as dist
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/conda/lib/python3.6/site-packages/smdistributed/dataparallel/__init__.py", line 16, in <module>
    import smddpcommon as hc
ImportError: libc10.so: cannot open shared object file: No such file or directory


title: How can I load CSV data for a PyTorch neural network?

body:
This may be a simple question, and I apologize if it's too simple. But I have some data in a CSV:

Date,Open,High,Low,Close,Adj Close,Volume
1993-01-29,43.968750,43.968750,43.750000,43.937500,26.453930,1003200
1993-02-01,43.968750,44.250000,43.968750,44.250000,26.642057,480500
1993-02-02,44.218750,44.375000,44.125000,44.343750,26.698507,201300
1993-02-03,44.406250,44.843750,44.375000,44.812500,26.980742,529400
1993-02-04,44.968750,45.093750,44.468750,45.000000,27.093624,531500
1993-02-05,44.968750,45.062500,44.718750,44.968750,27.074818,492100
1993-02-08,44.968750,45.125000,44.906250,44.968750,27.074818,596100
1993-02-09,44.812500,44.812500,44.562500,44.656250,26.886669,122100
....

I want to create a "training set", which is basically a random vector of 10 rows of data (I can figure out the normalizing, etc) randomly sampled from anywhere in the file. I think I'll have to use pandas to do the loading maybe?

If what I'm trying to ask is unclear, please add comments and I will adjust the question accordingly. Thank you.


title: How do I use one-hot encoding with cross-entropy loss?

body:
I have tried using one-hot encoding for a multiclass classification of about 120 classes using a dog breed dataset. Also using resnet18. But the following error is shown when I run the code. Please help me solve the issue.

The code of my model is shown below:

model = torchvision.models.resnet18()
op = torch.optim.Adam(model.parameters(),lr=0.001)
crit = nn.NLLLoss()
model.fc = nn.Sequential(
    nn.Linear(512,120),
    nn.Dropout(inplace=True),
    nn.ReLU(),
    nn.LogSoftmax())

for i,(x,y) in enumerate(train_dl):
    # prepare one-hot vector
    y_oh=torch.zeors(y.shape[0],120)
    y_oh.scatter_(1, y.unsqueeze(1), 1)

    # do the prediction
    y_hat=model(x)
    y_=torch.max(y_hat)

    loss=crit(y,y_)
    op.zero_grad()
    loss.backward()
    op.step()

The error:

RuntimeError Traceback (most recent call last) <ipython-input-190-46a21ead759a> in <module>
       6
       7     y_hat=model(x)
 ----> 8     loss=crit(y_oh,y_hat)
       9     op.zero_grad()
      10     loss.backward()

***RuntimeError: 1D target tensor expected, multi-target not supported***


I am trying to generate a number from the normal distribution using:

from torch.distributions import Normal
noise = Normal(th.tensor([0]), th.tensor(3.20))
noise = noise.sample()

But I am getting this error: RuntimeError: _th_normal not supported on CPUType for Long


title: PyTorch tensor slice and memory usage

body:
import torch
T = torch.FloatTensor(range(0,10 ** 6)) # 1M

#case 1:
torch.save(T, 'junk.pt')
# results in a 4 MB file size

#case 2:
torch.save(T[-20:], 'junk2.pt')
# results in a 4 MB file size

#case 3:
torch.save(torch.FloatTensor(T[-20:]), 'junk3.pt')
# results in a 4 MB file size

#case 4:
torch.save(torch.FloatTensor(T[-20:].tolist()), 'junk4.pt')
# results in a 405 Bytes file size

My questions are:

In case 3 the resulting file size seems surprising as we are creating a new tensor. Why is this new tensor not just the slice?

Is case 4, the optimal method for saving just part (slice) of a tensor?

More generally, if I want to 'trim' a very large 1-dimensional tensor by removing the first half of its values in order to save memory, do I have to proceed as in case 4, or is there a more direct and less computationally costly way that does not involve creating a python list.


title: Error training RNN with pytorch : RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

body:
Helo everyone, I am trying to create a model using the PyTorch RNN class and to train this model using minibatches. My dataset is a simple timeserie (one input one output). Here is what my model looks like :

class RNN_pytorch(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(RNN_pytorch, self).__init__()

        self.hidden_size = hidden_size
        self.input_size = input_size

        self.rnn = nn.RNN(input_size, hidden_size, num_layers=1)
        self.linear =  nn.Linear(hidden_size, output_size)

    def forward(self, x, hidden):

        batch_size = x.size(1)
#         print(batch_size)
        hidden = self.init_hidden(batch_size)

        out, hidden = self.rnn(x, hidden)
#         out = out.view(out.size(1), out.size(2))
        print("Input linear : ", out.size())
        out = self.linear(out)

        return out, hidden

    def init_hidden(self, batch_size):

        hidden = torch.zeros(1, batch_size, self.hidden_size)
#         print(hidden.size())

        return hidden

Then I process my dataset and split it like so :

batch_numbers = 13
batch_size = int(len(train_signal[:-1])/batch_numbers)
print("Train sample total size =", len(train_signal[:-1]))
print("Number of batches = ", batch_numbers)
print("Size of batches = {} (train_size / batch_numbers)".format(batch_size))

train_signal_batched = train_signal[:-1].reshape(batch_numbers, batch_size, 1)
train_label_batched = train_signal[1:].reshape(batch_numbers, batch_size, 1)

print("X_train shape =", train_signal_batched.shape)
print("Y_train shape =", train_label_batched.shape)

Returning :

Train sample total size = 829439
Number of batches =  13
Size of batches = 63803 (train_size / batch_numbers)
X_train shape = (13, 63803, 1)
Y_train shape = (13, 63803, 1)

So far so good, but then I try to train my model :

rnn_mod = RNN_pytorch(1, 16, 1)

criterion = nn.MSELoss()
optimizer = torch.optim.RMSprop(rnn_mod.parameters(), lr=0.01)

n_epochs = 3
hidden = rnn_mod.init_hidden(batch_size)
for epoch in range(1, n_epochs):
    for i, batch in enumerate(train_signal_batched):
        optimizer.zero_grad()
        x = torch.Tensor([batch]).float()
        print("Input : ",x.size())
        out, hidden = rnn_mod.forward(x, hidden)
        print("Output : ",out.size())
        label = torch.Tensor([train_label_batched[i]]).float()
        print("Label : ", label.size())
        loss = criterion(output, label)
        print("Loss : ", loss)
        loss.backward(retain_graph=True)
        optimizer.step()
        print("*", end="")

#     if epoch % 100 == 0:
    print("Step {} --- Loss {}".format(epoch, loss))

Which leads to the error :

Input :  torch.Size([1, 63803, 1])
Input linear :  torch.Size([1, 63803, 16])
Output :  torch.Size([1, 63803, 1])
Label :  torch.Size([1, 63803, 1])
Loss :  tensor(0.0051)

/home/kostia/.virtualenvs/machine-learning/lib/python3.6/site-packages/torch/nn/modules/loss.py:431: UserWarning: Using a target size (torch.Size([1, 63803, 1])) that is different to the input size (torch.Size([1, 1, 1])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
  return F.mse_loss(input, target, reduction=self.reduction)

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-217-d019358438ff> in <module>
     17         loss = criterion(output, label)
     18         print("Loss : ", loss)
---> 19         loss.backward(retain_graph=True)
     20         optimizer.step()
     21         print("*", end="")

~/.virtualenvs/machine-learning/lib/python3.6/site-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph)
    116                 products. Defaults to ``False``.
    117         """
--> 118         torch.autograd.backward(self, gradient, retain_graph, create_graph)
    119
    120     def register_hook(self, hook):

~/.virtualenvs/machine-learning/lib/python3.6/site-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
     91     Variable._execution_engine.run_backward(
     92         tensors, grad_tensors, retain_graph, create_graph,
---> 93         allow_unreachable=True)  # allow_unreachable flag
     94
     95

RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

Can anyone tell me what is the problem here because I honestly doesn't have a clue ?

Thanks in advance


title: Pytorch tutorial code error: "NameError: name 'net' is not defined"

body:
The code comes from a tutorial for PyTorch. I'm using Google Collabs notebook to run the code.
import torch
import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):

  def __init__(self):
    super(Net, self).__init__()
    self.conv1 = nn.Conv2d(1, 6, 3)
    self.conv2 = nn.Conv2d(6, 16, 3)
    self.fc1 = nn.Linear(16 * 6 * 6, 120)
    self.fc2 = nn.Linear(120, 84)
    self.fc3 = nn.Linear(84, 10)

  def forward(self, x):
    x = F.max_pool2d(F.relu(self.conv1(x)), (2,2))
    x = F.max_pool2d(F.relu(self.conv2(x)), 2)
    x = x.view(-1, self.num_flat_features(x))
    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(x))
    x = self.fc3(x)
    return x

  def num_flat_features(self, x):
    size = x.size()[1:]
    num_features = 1
    for s in size:
      num_features *= s
    return num_features

  net = Net()
  print(net)

# The code works up until here. It's the following chunk that returns an error.
params = list(net.parameters())
print(len(params))
print(params[0].size())

The error is:
NameError                                 Traceback (most recent call last)

<ipython-input-17-ad79a1eff4f3> in <module>()
     32   print(net)
     33
---> 34 params = list(net.parameters())
     35 print(len(params))
     36 print(params[0].size())

NameError: name 'net' is not defined

The tutorial says the output should be this instead:
10
torch.Size([6, 1, 3, 3])

It looks to me like net is defined, so I'm unclear why this error is happening. I'm not expert with Python to begin with, so maybe there is something obvious I'm missing.


It can run epoch 1 and eval successfully, but it fails when run epoch 2.

Train Epoch:1[655200/655800(100%)] loss:26.4959 lr:0.2050
Test Epoch:1 acc:0.973 val:0.895

Train Epoch:2[0/655800(0%)] loss:26.8068 lr:0.2051
File "train_11w.py", line 244, in main
    train(train_loader, model, optimizer, epoch, lr_decay_type, logger, args.log_interval, args)
  File "train_11w.py", line 305, in train
    prediction, ex, exnorm = model(img, mode=6, y=label)
  File "/home/luban/anaconda2/lib/python2.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/luban/anaconda2/lib/python2.7/site-packages/torch/nn/parallel/data_parallel.py", line 143, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/home/luban/anaconda2/lib/python2.7/site-packages/torch/nn/parallel/data_parallel.py", line 153, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/home/luban/anaconda2/lib/python2.7/site-packages/torch/nn/parallel/parallel_apply.py", line 83, in parallel_apply
    raise output
RuntimeError: CUDA error: invalid argument

Driver Version: 418.67
CUDA Version 10.0.130
python 2.7.3
torch 1.0.0


title: PyTorch: Move Weights Between GPU and CPU on the fly

body:
I have a large architecture which does not fit into GPU memory, but there is a nice property of this architecture where only subsets of the architecture run at any given time for a stretch of time. Therefore, I would like to dynamically load/unload the weights of layers which are not being utilized between the CPU and GPU. How can this be achieved?

The first thing one might try is call .cpu() or .cuda() on the parameters I wish to move. Unfortunately, that would cause training problems with the optimizer as stated in the docs:

cuda(device=None)
Moves all model parameters and buffers to the GPU.

This also makes associated parameters and buffers different objects. So it should be called before constructing optimizer if the module will live on GPU while being optimized.

One example use case would be implementing ProxylessNAS, however only final trained models are available at the time of writing and the architecture search implementation is not available.


title: Loading FITS images with PyTorch

body:
I'm trying to create a CNN using PyTorch but my images need importing from the FITS format rather than conventional .png or .jpeg etc.

Is there a way to accomplish this easily using torch.utils.data.DataLoader or is there a place in the source code where I can put in a clause which will handle FITS files while loading in?

I have looked in the documentation and the most relevant thing I've found is the ToPILImage transformer which converts a tensor or ndarray into a PIL Image.

Currently I'm using an image loading routine as follows:

import torch
from torch.autograd import Variable
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import torchvision.datasets as dset
import torchvision.transforms as transforms
import torchvision

batch_size = 4

transform = transforms.Compose(
                   [transforms.Resize((32,32)),
                    transforms.ToTensor(),
                    ])

trainset = dset.ImageFolder(root="Documents/Image_data",transform=transform)
train_loader = torch.utils.data.DataLoader(trainset, batch_size=batch_size,shuffle=True)

Astropy: http://www.astropy.org/

Pytorch: https://pytorch.org/

torch.utils: https://pytorch.org/docs/master/data.html

UPDATE: Perhaps using torchvision.datasets.DatasetFolder instead of DataLoader, an inserting in my own FITS handler would work?

When trying to use this class I get the following error:

AttributeError: module 'torchvision.datasets' has no attribute 'DatasetFolder'

Is DatasetFolder actually supported by torchvision at this point in time?


title: PyTorch gradient differs from manually calculated gradient

body:
I'm trying to compute the gradient of 1/x without using Pytorch's autograd. I use the formula grad(1/x, x) = -1/x**2. When I compare my result with this formula to the gradient given by Pytorch's autograd, they're different.

Here is my code:

a = torch.tensor(np.random.randn(), dtype=dtype, requires_grad=True)
loss = 1/a
loss.backward()
print(a.grad - (-1/(a**2)))

The output is:

tensor(5.9605e-08, grad_fn=<ThAddBackward>)

Can anyone explain to me what the problem is?


title: UNet Model accuracy is stuck at exact 0.5 (neither more or less) (No class imbalance, tried tuning learning rate)

body:
This is using PyTorch

I have been trying to implement UNet model on my images, however, my model accuracy is always exact 0.5. Loss does decrease.

I have also checked for class imbalance. I have also tried playing with learning rate. Learning rate affects loss but not the accuracy.

My architecture below ( from here )

""" `UNet` class is based on https://arxiv.org/abs/1505.04597

The U-Net is a convolutional encoder-decoder neural network.
Contextual spatial information (from the decoding,
expansive pathway) about an input tensor is merged with
information representing the localization of details
(from the encoding, compressive pathway).

Modifications to the original paper:
(1) padding is used in 3x3 convolutions to prevent loss
    of border pixels
(2) merging outputs does not require cropping due to (1)
(3) residual connections can be used by specifying
    UNet(merge_mode='add')
(4) if non-parametric upsampling is used in the decoder
    pathway (specified by upmode='upsample'), then an
    additional 1x1 2d convolution occurs after upsampling
    to reduce channel dimensionality by a factor of 2.
    This channel halving happens with the convolution in
    the tranpose convolution (specified by upmode='transpose')

    Arguments:
        in_channels: int, number of channels in the input tensor.
                     Default is 3 for RGB images. Our SPARCS dataset is 13 channel.
              depth: int, number of MaxPools in the U-Net. During training, input size needs to be
                     (depth-1) times divisible by 2
        start_filts: int, number of convolutional filters for the first conv.
            up_mode: string, type of upconvolution. Choices: 'transpose' for transpose convolution

"""

class UNet(nn.Module):

    def __init__(self, num_classes, depth, in_channels, start_filts=16, up_mode='transpose', merge_mode='concat'):

        super(UNet, self).__init__()

        if up_mode in ('transpose', 'upsample'):
            self.up_mode = up_mode
        else:
            raise ValueError("\"{}\" is not a valid mode for upsampling. Only \"transpose\" and \"upsample\" are allowed.".format(up_mode))

        if merge_mode in ('concat', 'add'):
            self.merge_mode = merge_mode
        else:
            raise ValueError("\"{}\" is not a valid mode for merging up and down paths.Only \"concat\" and \"add\" are allowed.".format(up_mode))

        # NOTE: up_mode 'upsample' is incompatible with merge_mode 'add'
        if self.up_mode == 'upsample' and self.merge_mode == 'add':
            raise ValueError("up_mode \"upsample\" is incompatible with merge_mode \"add\" at the moment "
                             "because it doesn't make sense to use nearest neighbour to reduce depth channels (by half).")

        self.num_classes = num_classes
        self.in_channels = in_channels
        self.start_filts = start_filts
        self.depth = depth

        self.down_convs = []
        self.up_convs = []

        # create the encoder pathway and add to a list
        for i in range(depth):
            ins = self.in_channels if i == 0 else outs
            outs = self.start_filts*(2**i)
            pooling = True if i < depth-1 else False

            down_conv = DownConv(ins, outs, pooling=pooling)
            self.down_convs.append(down_conv)

        # create the decoder pathway and add to a list
        # - careful! decoding only requires depth-1 blocks
        for i in range(depth-1):
            ins = outs
            outs = ins // 2
            up_conv = UpConv(ins, outs, up_mode=up_mode, merge_mode=merge_mode)
            self.up_convs.append(up_conv)

        self.conv_final = conv1x1(outs, self.num_classes)

        # add the list of modules to current module
        self.down_convs = nn.ModuleList(self.down_convs)
        self.up_convs = nn.ModuleList(self.up_convs)

        self.reset_params()

    @staticmethod
    def weight_init(m):
        if isinstance(m, nn.Conv2d):

            #https://prateekvjoshi.com/2016/03/29/understanding-xavier-initialization-in-deep-neural-networks/
            ##Doc: https://pytorch.org/docs/stable/nn.init.html?highlight=xavier#torch.nn.init.xavier_normal_
            init.xavier_normal_(m.weight)
            init.constant_(m.bias, 0)

    def reset_params(self):
        for i, m in enumerate(self.modules()):
            self.weight_init(m)

    def forward(self, x):
        encoder_outs = []

        # encoder pathway, save outputs for merging
        for i, module in enumerate(self.down_convs):
            x, before_pool = module(x)
            encoder_outs.append(before_pool)

        for i, module in enumerate(self.up_convs):
            before_pool = encoder_outs[-(i+2)]
            x = module(before_pool, x)

        # No softmax is used. This means we need to use
        # nn.CrossEntropyLoss is your training script,
        # as this module includes a softmax already.
        x = self.conv_final(x)
        return x

Parameters are :

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
x,y = train_sequence[0] ; batch_size = x.shape[0]
model = UNet(num_classes = 2, depth=5, in_channels=5, merge_mode='concat').to(device)
optim = torch.optim.Adam(model.parameters(),lr=0.01, weight_decay=1e-3)
criterion = nn.BCEWithLogitsLoss() #has sigmoid internally
epochs = 1000

The function for training is :

import torch.nn.functional as f

def train_model(epoch,train_sequence):
    """Train the model and report validation error with training error
    Args:
        model: the model to be trained
        criterion: loss function
        data_train (DataLoader): training dataset
    """
    model.train()
    for idx in range(len(train_sequence)):
        X, y = train_sequence[idx]
        images = Variable(torch.from_numpy(X)).to(device) # [batch, channel, H, W]
        masks = Variable(torch.from_numpy(y)).to(device)

        outputs = model(images)
        print(masks.shape, outputs.shape)
        loss = criterion(outputs, masks)
        optim.zero_grad()
        loss.backward()
        # Update weights
        optim.step()
    # total_loss = get_loss_train(model, data_train, criterion)

My function for calculating loss and accuracy is below:

def get_loss_train(model, train_sequence):
    """
        Calculate loss over train set
    """
    model.eval()
    total_acc = 0
    total_loss = 0
    for idx in range(len(train_sequence)):
        with torch.no_grad():
            X, y = train_sequence[idx]
            images = Variable(torch.from_numpy(X)).to(device) # [batch, channel, H, W]
            masks = Variable(torch.from_numpy(y)).to(device)

            outputs = model(images)
            loss = criterion(outputs, masks)
            preds = torch.argmax(outputs, dim=1).float()
            acc = accuracy_check_for_batch(masks.cpu(), preds.cpu(), images.size()[0])
            total_acc = total_acc + acc
            total_loss = total_loss + loss.cpu().item()
    return total_acc/(len(train_sequence)), total_loss/(len(train_sequence))

Edit : Code which runs (calls) the functions:

for epoch in range(epochs):
    train_model(epoch, train_sequence)
    train_acc, train_loss = get_loss_train(model,train_sequence)
    print("Train Acc:", train_acc)
    print("Train loss:", train_loss)

Can someone help me identify as why is accuracy always exact 0.5?


title: How to decide which mode to use for 'kaiming_normal' initialization

body:
I have read several codes that do layer initialization using nn.init.kaiming_normal_() of PyTorch. Some codes use the fan in mode which is the default. Of the many examples, one can be found here and shown below.

init.kaiming_normal(m.weight.data, a=0, mode='fan_in')

However, sometimes I see people using the fan out mode as seen here and shown below.

if isinstance(m, nn.Conv2d):
                nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')

Can someone give me some guidelines or tips to help me decide which mode to select? Further I am working on image super resolutions and denoising tasks using PyTorch and which mode will be more beneficial.


title: How to set random seed when it is in distributed training in PyTorch?

body:
Now I am training a model using torch.distributed, but I am not sure how to set the random seeds. For example, this is my current code:

def main():
    np.random.seed(args.seed)
    torch.manual_seed(args.seed)
    torch.cuda.manual_seed(args.seed)

    cudnn.enabled = True
    cudnn.benchmark = True
    cudnn.deterministic = True

    mp.spawn(main_worker, nprocs=args.ngpus, args=(args,))

And should I move the

    np.random.seed(args.seed)
    torch.manual_seed(args.seed)
    torch.cuda.manual_seed(args.seed)

    cudnn.enabled = True
    cudnn.benchmark = True
    cudnn.deterministic = True

into the function main_worker() to make sure every process has the correct seed and cudnn settings? By the way, I have tried this, and this behavior will make the training 2 times slower, which really confused me.

Thank you very much for any help!


title: What is the machine precision in pytorch and when should one use doubles?

body:
I am running experiments on synthetic data (e.g. fitting a sine curve) and I get errors in pytorch that are really small. One if about 2.00e-7. I was reading about machine precision and it seems really close to the machine precision. How do I know if this is going to cause problems (or if perhaps it already has e.g. I can't differentiate between the different errors since they are "machine zero").
errors:
p = np.array([2.3078539778125768e-07,
               1.9997889411762922e-07,
               2.729681222011256e-07,
               3.2532371115080884e-07])

m = np.array([3.309504692539563e-07,
                 4.1058904888091606e-06,
                 6.8326703386053605e-06,
                 7.4616147721799645e-06])

what confuses me is that I tried adding what I thought was to small of a number so that it returned no difference but it did return a difference (i.e. I tried to do a+eps = a using eps = smaller than machine precision):
import torch

x1 = torch.tensor(1e-6)
x2 = torch.tensor(1e-7)
x3 = torch.tensor(1e-8)
x4 = torch.tensor(1e-9)

eps = torch.tensor(1e-11)

print(x1.dtype)
print(x1)
print(x1+eps)

print(x2)
print(x2+eps)

print(x3)
print(x3+eps)

print(x4)
print(x4+eps)

output:
torch.float32
tensor(1.0000e-06)
tensor(1.0000e-06)
tensor(1.0000e-07)
tensor(1.0001e-07)
tensor(1.0000e-08)
tensor(1.0010e-08)
tensor(1.0000e-09)
tensor(1.0100e-09)

I expected everything to be zero but it wasn't. Can someone explain to me what is going on? If I am getting losses close to 1e-7 should I use double rather than float? googling it seems that single is the precision for float afaik.
If I want to use doubles what are cons/pros + what is the least error prone way to change my code? Is a single change to double type enough or is there a global flag?

Useful reminder:
recall machine precision:
Machine precision is the smallest number ε such that the difference between 1 and 1 + ε is nonzero, i.e., it is the smallest difference between these two numbers that the computer recognizes. For IEEE-754 single precision this is 2-23 (approximately 10-7) while for IEEE-754 double precision it is 2-52 (approximately 10-16) .

Potential solution:
Ok let’s see if this is a good summary of what I think is correct (modulo ignoring some details that I don’t fully understand right now of floats, like the bias).
But I’ve concluded that the best thing for me is to make sure my errors/numbers have two properties:
they are within 7decimals of each other (due to the mantissa being 24 bigs like you pointed out the log_10(2^24) = 7.225)
they are far enough from the edges. For this I take the mantissa to be 23 bits away from the lower edge (point position about -128+23) and the same for the largest edge but 127-23.
As long we satisfy that more or less we avoid adding two numbers that are too small for the machine to distinguish (condition 1) and avoid overflows/underflows (condition 2).
Perhaps there is a small detail I might be missing with the bias or some other float detail (like representing infinity, NaN). But I believe that is correct.
If anyone can correct the details, that would be fantastic.

useful links:

https://www.cfd-online.com/Wiki/Machine_precision
https://discuss.pytorch.org/t/what-is-the-machine-precision-of-pytorch-with-cpus-or-gpus/9384/3
Should I use double or float?
Double precision floating values in Python?
Difference between Python float and numpy float32


title: Pytorch load model

body: I used tensorflow some days ago. To build conv layers with fixed weights is easy, just pass the weight kernel to conv2d(). And it is convenient to load pretrained models such as VGG19. But I found it did't work that way using pytorch, because conv2d() doesn't accept an explicit kernel but a kernel size. So I wonder if there is any possibility that we can reuse the weights in VGG19 by simply passing it to a method like conv2d(). Any reply will be appreciated.


title: How to create sub network reference in pytorch?

body:
The simple explanation for what I aim to do is the following

Given network O with structure of

---------           ----------            ----------            ----------            ----------
| Input | -> (x) -> | Blob A | -> (xa) -> | Blob B | -> (xb) -> | Blob C | -> (xc) -> | Output |
---------           ----------            ----------            ----------            ----------

I want to create a sub network to calculate a noise loss function for Blob C. The operation is given input xb with original output xc, and pass xb + noise through Blob C again to get xc'. Then the mse_loss is compute between xc and xc'

I have tried creating nn.Sequential from the original model. But I am not sure that it created a new deep copy or a reference.

If I have missed anything, please comment

Thank you


title: Python - importing fastai results in SyntaxError

body:
I am trying to setup an ML model using fastai and have to do the following imports:

import fastai.models
import fastai.nlp
import fastai.dataset

However, it gives me the following error by the fastai imports.

Python 2.7.15rc1 (default, Apr 15 2018, 21:51:34)
[GCC 7.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import fastai.nlp
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/fastai/nlp.py", line 172
    if os.path.isdir(path): paths=glob(f'{path}/*.*')
                                                   ^
SyntaxError: invalid syntax

Apparently, the character f in glob(f'{path}/*.*') is causing the error. I fixed the error by removing f, but it seems that there are lots of these errors in the fastai library.

My current thought is that I am using an incorrect python version. Could anyone give me some pointer?


title: Word embeddings with multiple categorial features for a single word

body:
I'm looking for a method to implement word embedding network with LSTM layers in Pytorch such that the input to the nn.Embedding layer has a different form than vectors of words IDs.

Each word in my case has a corresponding vector and the sentence in my corpus is consequently a vector of vectors. So, for example, I may have the word "King" with vector [500, 3, 18] where 500 is the Word ID, 3 is the word color, and 18 is the font size, etc. The embedding layer role here is to do some automatic feature reduction/extraction.

How can I feed the embedding layer with such form data? Or do you have any better suggestions?