title: Using LSTM stateful for passing context b/w batches; may be some error in context passing, not getting good results?

body:
I have checked the data before giving it to the network. The data is correct.
Using LSTM and passing the context b/w batches. per_class_accuracy is changing, but the loss is not going down. Been stuck for long, not sure if there is an error in the Code?
I have multi-class classification problem based upon an imbalanced dataset
Dataset_type: CSV
Dataset_size: 20000
Based upon CSV data of sensors
X = 0.6986111111111111,0,0,1,0,1,0,0,0,1,0,0,0,0,1,0,0,0,1,1,0,0,0
Y = leaveHouse
Per class accuracy:
{'leaveHouse': 0.34932855, 'getDressed': 1.0, 'idle': 0.8074534, 'prepareBreakfast': 0.8, 'goToBed': 0.35583413, 'getDrink': 0.0, 'takeShower': 1.0, 'useToilet': 0.0, 'eatBreakfast': 0.8857143}
Training:
# Using loss weights, the inverse of class frequency

criterion = nn.CrossEntropyLoss(weight = class_weights)

 hn, cn = model.init_hidden(batch_size)
            for i, (input, label) in enumerate(trainLoader):
                hn.detach_()
                cn.detach_()
                input = input.view(-1, seq_dim, input_dim)

                if torch.cuda.is_available():
                    input = input.float().cuda()
                    label = label.cuda()
                else:
                    input = input.float()
                    label = label

                # Forward pass to get output/logits
                output, (hn, cn) = model((input, (hn, cn)))

                # Calculate Loss: softmax --> cross entropy loss
                loss = criterion(output, label)#weig pram
                running_loss += loss
                loss.backward()  # Backward pass
                optimizer.step()  # Now we can do an optimizer step
                optimizer.zero_grad()  # Reset gradients tensors

Network

class LSTMModel(nn.Module):
    def init_hidden(self, batch_size):
        self.batch_size = batch_size
        if torch.cuda.is_available():
            hn = torch.zeros(self.layer_dim, self.batch_size, self.hidden_dim).cuda()
            # Initialize cell state
            cn = torch.zeros(self.layer_dim, self.batch_size, self.hidden_dim).cuda()
        else:
            hn = torch.zeros(self.layer_dim, self.batch_size, self.hidden_dim)
            # Initialize cell state
            cn = torch.zeros(self.layer_dim, self.batch_size, self.hidden_dim)
        return hn, cn

    def __init__(self, input_dim, hidden_dim, layer_dim, output_dim, seq_dim):
        super(LSTMModel, self).__init__()
        # Hidden dimensions
        self.hidden_dim = hidden_dim

        # Number of hidden layers
        self.layer_dim = layer_dim

        self.input_dim = input_dim
        # Building your LSTM
        # batch_first=True causes input/output tensors to be of shape
        # (batch_dim, seq_dim, feature_dim)
        self.lstm = nn.LSTM(self.input_dim, hidden_dim, layer_dim, batch_first=True)

        # Readout layer
        self.fc = nn.Linear(hidden_dim, output_dim)
        self.relu = nn.ReLU()
        self.softmax = nn.Softmax(dim=1)
        self.seq_dim = seq_dim

    def forward(self, inputs):
        # Initialize hidden state with zeros
        input, (hn, cn) = inputs
        input = input.view(-1, self.seq_dim, self.input_dim)

        # time steps
        out, (hn, cn) = self.lstm(input, (hn, cn))

        # Index hidden state of last time step
        out = self.fc(out[:, -1, :])
        out = self.softmax(out)
        return out, (hn,cn)


title: Get information on torch.tensor in Spyder

body:
I have simple project that uses Pytorch in my Spyder 4.1.3
Trying to get documentation by pressing CTRL+I on torch.tensor:

import torch
x=torch.tensor(2.0,requires_grad=True)

Got error:

No further documentation available

How to solve this problem?


title: How to split train test in order to categories data in PyTorch

body:
I have an image dataset with 36406 images, 52 categories. They are imbalance data. I want to split train and test data in order to proportion of categories in the dataset.
for example:

'Tshirts': 3534, 'Shirts': 3213, 'Casual Shoes': 2846,...

so in train dataset, I want 20% of each category:

train dataset: 'Tshirts': 3534 * 0.2, 'Shirts': 3213*0.2, 'Casual Shoes': 2846*0.2,...

I do not know how to implement it with PyTorch.

my dataset:

dataset = ImageFolder("/content/gdrive/My Drive/categorized_products"
                      , transform=transform)
classes = dataset.classes

I want to use:

train_loader = torch.utils.data.DataLoader(dataset, batch_size=batch_size, sampler=train_sampler, num_workers=16)
test_loader = torch.utils.data.DataLoader(dataset, batch_size=batch_size, sampler=test_sampler, num_workers=16)


title: Is it possible to write a PyTorch layer where the parameters are different for each training sample?

body:
I am trying to implement a hybrid timeseries forecasting model for a set of 3,000 similar timeseries (weekly sales data for each store in a large organisation's network) which uses a combination of Exponential Smoothing (ES) and a LSTM. The idea is that the ES parameters are unique for each store to capture its individual variability, while the LSTM parameters are learnt across the entire dataset. At each timestep for each store, the ES level and seasonality are used to normalise the input and output data.
Currently I have implemented this algorithm using fixed ES parameters for each store (optimized on the training set of each individual store) but would like to instead perform gradient descent on these parameters at the same time as the LSTM.
I want to write a custom PyTorch layer which transforms the unnormalised data using the ES parameters for each store, but I'm not sure how to implement this given that the ES parameters will be different for each store (ie training sample). The ES parameters unique to each store are the smoothing parameters a and b, and the initial levels and seasonality lt and st
My current training function is below:
def fit_model(model, X, y, optimizer, loss_function, a, b, lt, st):

    # lt and st should be rolled forwards to t=52

    n_stores = len(X)

    for i in range(n_epochs):

        total_loss=0
        count=0
        start = time.time()

        for store, (Xi, yi) in enumerate(zip(X, y)):

            # Find the number of timesteps
            n_timesteps, batchsize, n = Xi.shape

            # Reset the hidden state for each training sample
            model.hidden = model.init_hidden()

            for t in range(n_timesteps):

                ######### I want to turn this part into a pytorch layer #########
                # Find the level and seasonality of the current timestep
                lt[store] = level(Xi[t,:,-1], lt[store], st[store, t%52], a[store])
                st[store, t%52] = season(Xi[t,:,-1], lt[store], st[store, t%52], b[store])
                # Rescale the input vector
                Xit_scaled = torch.log(Xi[t,:,:]/lt[store]/st[store,:])
                ######### /end I want to turn this part into a pytorch layer #########

                # Predict on the scaled data
                yit_pred_scaled = model(Xit_scaled)
                # Unscale the data
                yit_pred = torch.exp(yit_pred_scaled)*lt[store]*torch.cat((st[store,(t%52)+1:],st[store,:(t%52)+1]))

                # Optimisation step
                optimizer.zero_grad()
                single_loss = loss_function(yit_pred, yi[t,:])
                single_loss.backward()
                optimizer.step()

                total_loss+=single_loss.item()
                count+=1

            time_taken = time.time() - start
            eta = (n_stores/(store+1)) * time_taken

            sys.stdout.write("\r" + 'N: '+ str(store+1)+'/'+str(n_stores)+' Average loss: '+str(total_loss/count)+' Time estimate: '+ str(eta))

        print(f'\nepoch: {i+1:3} loss: {total_loss/count:10.8f}\n')

    return model

n=52
h=52
n_hidden_neurons=100
batch_size=1
n_epochs=3
num_layers=1
print_steps=0
l_n = levels[n,:].clone()
s_n = seasons[n,:,:].clone()
dataset = sales_data

# Set up training data, model, loss and optimizer
train_X, train_y, test_X, test_y = get_supervised_data(dataset, n, h)
model = LSTM(n, n_hidden_neurons, batch_size, h, num_layers).double()
loss_function = SMAPELoss
optimizer = torch.optim.SGD(model.parameters(), lr=0.1)

# Fit model
model = fit_model(model, train_X, train_y, optimizer, loss_function, alpha, beta, l_n, s_n)


title: Trouble installing torch and torchvision on Windows 10

body:
I am having issues installing torch==0.4.1 on windows 10.
I am getting the following error, "Could not find a version that satisfies the requirement torch==0.4.1 (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)
ERROR: No matching distribution found for torch==0.4.1.
This is weird because torch 0.4.1 is available on PyPi
I am using python 2.7.15 and pip 20.1.1


title: GPU Memory lack even with model parallelism on PyTorch

body:
I was testing the U-GAT-IT model and found out that the model is to heavy for a single GPU even with batch size 1, and tried model parallelism.
I'm using TiTan X GPU, Cuda 10.0, Pytorch 1.2.0.
class ResnetGenerator(nn.Module):
    def __init__(self, input_nc, output_nc, ngf=64, n_blocks=6, img_size=256, light=False, split_gpus=True):
        assert(n_blocks >= 0)
        super(ResnetGenerator, self).__init__()
        self.input_nc = input_nc
        self.output_nc = output_nc
        self.ngf = ngf
        self.n_blocks = n_blocks
        self.img_size = img_size
        self.light = light
        self.split_gpus = split_gpus

        DownBlock = []
        DownBlock += [nn.ReflectionPad2d(3),
                      nn.Conv2d(input_nc, ngf, kernel_size=7, stride=1, padding=0, bias=False),
                      nn.InstanceNorm2d(ngf),
                      nn.ReLU(True)]

        # Down-Sampling
        n_downsampling = 2
        for i in range(n_downsampling):
            mult = 2**i
            DownBlock += [nn.ReflectionPad2d(1),
                          nn.Conv2d(ngf * mult, ngf * mult * 2, kernel_size=3, stride=2, padding=0, bias=False),
                          nn.InstanceNorm2d(ngf * mult * 2),
                          nn.ReLU(True)]

        # Down-Sampling Bottleneck
        mult = 2**n_downsampling
        for i in range(n_blocks):
            DownBlock += [ResnetBlock(ngf * mult, use_bias=False)]

        # Class Activation Map
        self.gap_fc = nn.Linear(ngf * mult, 1, bias=False).to('cuda:1')
        self.gmp_fc = nn.Linear(ngf * mult, 1, bias=False).to('cuda:1')
        self.conv1x1 = nn.Conv2d(ngf * mult * 2, ngf * mult, kernel_size=1, stride=1, bias=True).to('cuda:1')
        self.relu = nn.ReLU(True).to('cuda:1')

        # Gamma, Beta block
        if self.light:
            FC = [nn.Linear(ngf * mult, ngf * mult, bias=False),
                  nn.ReLU(True),
                  nn.Linear(ngf * mult, ngf * mult, bias=False),
                  nn.ReLU(True)]
        else:
            FC = [nn.Linear(img_size // mult * img_size // mult * ngf * mult, ngf * mult, bias=False),
                  nn.ReLU(True),
                  nn.Linear(ngf * mult, ngf * mult, bias=False),
                  nn.ReLU(True)]
        self.gamma = nn.Linear(ngf * mult, ngf * mult, bias=False).to('cuda:1')
        self.beta = nn.Linear(ngf * mult, ngf * mult, bias=False).to('cuda:1')

        # Up-Sampling Bottleneck
        for i in range(n_blocks):
            setattr(self, 'UpBlock1_' + str(i+1), ResnetAdaILNBlock(ngf * mult, use_bias=False).to('cuda:1'))

        # Up-Sampling
        UpBlock2 = []
        for i in range(n_downsampling):
            mult = 2**(n_downsampling - i)
            UpBlock2 += [nn.Upsample(scale_factor=2, mode='nearest'),
                         nn.ReflectionPad2d(1),
                         nn.Conv2d(ngf * mult, int(ngf * mult / 2), kernel_size=3, stride=1, padding=0, bias=False),
                         ILN(int(ngf * mult / 2)),
                         nn.ReLU(True)]

        UpBlock2 += [nn.ReflectionPad2d(3),
                     nn.Conv2d(ngf, output_nc, kernel_size=7, stride=1, padding=0, bias=False),
                     nn.Tanh()]

        self.DownBlock = nn.Sequential(*DownBlock).to('cuda:0')
        self.FC = nn.Sequential(*FC).to('cuda:1')
        self.UpBlock2 = nn.Sequential(*UpBlock2).to('cuda:1')

    def forward(self, input):

        x = self.DownBlock(input)
        x = x.cuda(1)

        print('A')
        gap = torch.nn.functional.adaptive_avg_pool2d(x, 1)
        gap_logit = self.gap_fc(gap.view(x.shape[0], -1))
        gap_weight = list(self.gap_fc.parameters())[0]
        gap = x * gap_weight.unsqueeze(2).unsqueeze(3)

        gmp = torch.nn.functional.adaptive_max_pool2d(x, 1)
        gmp_logit = self.gmp_fc(gmp.view(x.shape[0], -1))
        gmp_weight = list(self.gmp_fc.parameters())[0]
        gmp = x * gmp_weight.unsqueeze(2).unsqueeze(3)

        cam_logit = torch.cat([gap_logit, gmp_logit], 1)
        x = torch.cat([gap, gmp], 1)
        x = self.relu(self.conv1x1(x))

        heatmap = torch.sum(x, dim=1, keepdim=True)

        if self.light:
            x_ = torch.nn.functional.adaptive_avg_pool2d(x, 1)
            x_ = self.FC(x_.view(x_.shape[0], -1))
        else:
            x_ = self.FC(x.view(x.shape[0], -1))

        gamma, beta = self.gamma(x_), self.beta(x_)
        gamma = gamma.cuda(1)
        beta = beta.cuda(1)
        for i in range(self.n_blocks):
            x = getattr(self, 'UpBlock1_' + str(i+1))(x, gamma, beta)

        out = self.UpBlock2(x)

        out = out.cuda(0)
        cam_logit = cam_logit.cuda(0)
        heatmap = heatmap.cuda(0)

        return out, cam_logit, heatmap

but still gives the following error
RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 1; 11.91 GiB total capacity; 11.12 GiB already allocated; 8.56 MiB free; 160.91 MiB cached)
The funny thing is that by my code of model parallelism, the model gets through 1 iteration then gets the error, so it isn't useless at all.
So I think there could be two possibilities

Something is wrong with my Model Parallelism or GPU settings(e.g. memory dynamic allocation is not working)
U-GAT-IT is super heavy so it needs even more memory

Could anyone answer which possibility is true? Any help is appreciated.


title: Scheduling GPU in Pytorch

body: How to implement the following task in Pytorch:  I need a Linux environment where I can share GPU resources between multiple users. I have two servers with each 4 GPU cards. How to report scheduling behavior for each user: let say for user 1, user 2 and so on? Need suggestions about the implementation, please. Thank you for understanding!


title: Conv3d error on dimensions of a 5d tensor

body:
I have a tensor shaped ([5, 1, 3, 126, 126]), which represents a video (5 frames each 126x126 rgb).
I need to forward it into a
self.resnet = nn.Sequential(
        nn.Conv3d(5,5,1),
        nn.UpsamplingBilinear2d(size=None, scale_factor=0.5)
    )

but i get
RuntimeError: Given groups=1, weight of size [5, 5, 1, 1, 1], expected input[5, 1, 3, 126, 126] to have 5 channels, but got 1 channels instead

I think that I have probably misunderstood how the conv3d works but I can't really understand why the expected dimensions are so different from the ones that my 5d tensor has at that moment


title: Looking for a more efficient way to implement scalar multiplication for word2vec

body:
I would like to implement myself word2vec in pytorch and the forward method has the following signature:
def forward(self, input_index_batch, output_indices_batch):
#input_index_batch - Tensor of ints, shape: (batch_size, )
#output_indices_batch - Tensor if ints, shape: (batch_size, num_negative_samples+1)

In the word2vec approach one calculates the dot product between the input in the U - embedding matrix, and each of the num_negative_samples+1 outputs, which correspond to the context words, and the first one is the positive context word, and all remaining - the negative. I have tackled the problem with the following way:
    U_batch = self.input.weight.T[input_index_batch]
    for i, output_index in enumerate(output_indices_batch.T):
        V_batch = self.output.weight[output_index]
        dot_prod = torch.einsum('bi,bj->b', U_batch, V_batch)
        if i == 0:
            predictions[:, i] = torch.sigmoid(dot_prod)
        else:
            predictions[:, i] = 1 - torch.sigmoid(dot_prod)

    return predictions

This works correctly, however, rather inefficiently and slow. What would be more robust way to implement this?
In the init method self.input and self.output are defined as (10 - word2vec dim):
self.input = nn.Linear(num_tokens, 10, bias=False)
self.output = nn.Linear(10, num_tokens, bias=False)


title: Question on restoring training after loading model

body:
Having trained for 24 hours, the training process saved the model files via torch.save. There was a power-off or other issues caused the process exited. Normally, we can load the model and continue training from the last step.
Why should not we load the states of optimizers (Adam, etc), is it necessary?


title: How much deep a Neural Network Required for 12 inputs of ranging from -5000 to 5000 in a3c Reinforcement Learning

body:
I am trying to use A3C with LSTM for an environment where states has 12 inputs ranging from -5000 to 5000.
I am using an LSTM layer of size 12 and then 2 fully connected hidden layers of size 256, then 1 fc for 3 action dim and 1 fc for 1 value function.
The reward is in range (-1,1).
However during initial training I am unable to get good results.
My question is- Is this Neural Network good enough for this kind of environment.
Below is the code for Actor Critic
class ActorCritic(torch.nn.Module):

    def __init__(self, params):
        super(ActorCritic, self).__init__()

        self.state_dim = params.state_dim
        self.action_space = params.action_dim
        self.hidden_size = params.hidden_size
        state_dim = params.state_dim
        self.lstm = nn.LSTMCell(state_dim, state_dim)
        self.lstm.bias_ih.data.fill_(0)
        self.lstm.bias_hh.data.fill_(0)
        lst = [state_dim]
        for i in range(params.layers):
            lst.append(params.hidden_size)

        self.hidden = nn.ModuleList()
        for k in range(len(lst)-1):
            self.hidden.append(nn.Linear(lst[k], lst[k+1]))
        for layer in self.hidden:
            layer.apply(init_weights)

        self.critic_linear = nn.Linear(params.hidden_size, 1)
        self.critic_linear.apply(init_weights)
        self.actor_linear = nn.Linear(params.hidden_size, self.action_space)
        self.actor_linear.apply(init_weights)
        self.train()

    def forward(self, inputs):
        inputs, (hx, cx) = inputs
        inputs = inputs.reshape(1,-1)
        hx, cx = self.lstm(inputs, (hx, cx))
        x = hx
        for layer in self.hidden:
            x = torch.tanh(layer(x))
        return self.critic_linear(x), self.actor_linear(x), (hx, cx)

class Params():
    def __init__(self):
        self.lr = 0.0001
        self.gamma = 0.99
        self.tau = 1.
        self.num_processes = os.cpu_count()
        self.state_dim = 12
        self.action_dim = 3
        self.hidden_size = 256
        self.layers = 2
        self.epochs = 10
        self.lstm_layers = 1
        self.lstm_size = self.state_dim
        self.num_steps = 20
        self.window = 50


title: Pytorch precomputing part of a loss function

body:
I'm trying to use pytorch to calculate a complex function but still require the gradient of the output as a function of the inputs. For example:
a=torch.tensor([1,2,3], dtype=torch.float32, requires_grad=True)
b=torch.tensor([3,2,1], dtype=torch.float32, requires_grad=True)
v=torch.tensor([0.2], dtype=torch.float32, requires_grad=True)

# here precalc represents some (fairly expensive) sequence of operations
precalc = a.dot(b)+a*b+a*a+b*b

def calc(precalc, v):
    z=torch.randn(3,1000)
    batch=v*precalc.matmul(z)
    return torch.relu(batch).mean()

So the precalc tensor is a fixed function of a and b (but expensive to compute).
When I call calc for the first time as x=calc(precalc, v) and then x.backward(), I get the correct gradients for a,b and v. However, if I call calc a second time, i.e. x2=calc(precalc, v) and then x2.backward(), I get the pytorch error that I'm going backward through the graph a second time and I should retain the graph.
I would ideally like to free the graph as precalc is expensive to compute but small in memory and the calc itself is easier to compute but will use a lot of memory.
Is there a way to achieve this?
Thanks.


title: CNN trained model doesn't appear to be working

body:
I've trained a CNN model and I would like to run the trained model against new data. However, it seems that the trained model isn't predicting the count correctly as it done during the training. I have a feeling that the model is not using the PTH file. Could someone please advise what I am doing wrong, please?
import argparse
import datetime
import glob
import os
import random
import shutil
import time
from os.path import join

import numpy as np
import pandas as pd
import torch
import torch.nn as nn
from torch.utils.data import DataLoader
from torch.utils.tensorboard import SummaryWriter
from torchvision.transforms import ToTensor
from tqdm import tqdm
import torch.optim as optim

from convnet3_eval import Convnet
from dataset2_eval import CellsDataset

parser = argparse.ArgumentParser('Predicting hits from pixels')
parser.add_argument('name',type=str,help='Name of experiment')
parser.add_argument('data_dir',type=str,help='Path to data directory containing images and gt.csv')
parser.add_argument('--weight_decay',type=float,default=0.0,help='Weight decay coefficient (something like 10^-5)')
parser.add_argument('--lr',type=float,default=0.0001,help='Learning rate')
args = parser.parse_args()

metadata = pd.read_csv(join(args.data_dir,'gt.csv'))
metadata.set_index('filename', inplace=True)

dataset = CellsDataset(args.data_dir,transform=ToTensor(),return_filenames=True)
dataset = DataLoader(dataset,num_workers=4,pin_memory=True)
model_path = '/base_model.pth'

model = Convnet()
optimizer = torch.optim.Adam(model.parameters(),lr=args.lr,weight_decay=args.weight_decay)

for images, paths in tqdm(dataset):

    targets = torch.tensor([metadata['count'][os.path.split(path)[-1]] for path in paths]) # B
    targets = targets.float()

    # code to print training data to a csv file
    filename=CellsDataset(args.data_dir,transform=ToTensor(),return_filenames=True)
    output = model(images) # B x 1 x 9 x 9 (analogous to a heatmap)
    preds = output.sum(dim=[1,2,3]) # predicted cell counts (vector of length B)
    print(preds)
    paths_test = np.array([paths])
    names_preds = np.hstack(paths)
    print(names_preds)
    df=pd.DataFrame({'Image_Name':names_preds, 'Target':targets.detach(), 'Prediction':preds.detach()})
    print(df)
    # save image name, targets, and predictions
    df.to_csv(r'model.csv', index=False, mode='a')

model.load_state_dict(torch.load(model_path))
model.eval()


title: Pytorch tensor to numpy conversion imprecision

body:
I am using pytorch to build a network model that outputs a probabilistic distribution, after obtaining predictions i want to transform them into a numpy array for further processing, although, when doing that, the converted array is not precise, that is, the values differ from the original (see example below). This is specially bad in my case because, considering the imprecision, the rows does not sum exactly 1 anymore, which is causing me issues further into the code, any ideas on why this is happening? Any help would be much appreciated.
>> predictions
Out[34]:
tensor([[0.0155, 0.0260, 0.2671, 0.6820, 0.0093],
        [0.1231, 0.1076, 0.1660, 0.5376, 0.0658],
        [0.0734, 0.0501, 0.1683, 0.6602, 0.0480],
        ...,
        [0.0260, 0.0287, 0.0465, 0.8830, 0.0159],
        [0.0251, 0.0327, 0.3148, 0.5643, 0.0632],
        [0.0105, 0.0116, 0.2723, 0.6702, 0.0354]], device='cuda:0',
       grad_fn=<CopySlices>)

>> predictions.cpu().detach().numpy()
Out[33]:
array([[0.01549727, 0.02597333, 0.26714763, 0.68203545, 0.00934631],
       [0.12310678, 0.10758312, 0.16596784, 0.53758585, 0.06575638],
       [0.07338995, 0.05007047, 0.16834012, 0.6601813 , 0.04801816],
       ...,
       [0.02603309, 0.02868567, 0.046453  , 0.8829591 , 0.01586908],
       [0.02508399, 0.03266559, 0.31480333, 0.564255  , 0.06319207],
       [0.01048695, 0.01161059, 0.2722554 , 0.6702131 , 0.03543395]],
      dtype=float32)


title: Get exact formula used by Pytorch autograd to compute gradients

body:
I am implementing a custom CNN with some custom modules in it. I have implemented only the forward pass for the custom modules and left their backward pass to autograd.
I have manually computed the correct formulae for backpropagation through the parameters of the custom modules, and I wished to see whether they match with the formulae used internally by autograd to compute the gradients.
Is there any way to see this?
Thanks
Edit (To add a test case) :-
I have a complex affine layer where the weights and inputs are complex-valued matrices, and the operation is a matrix multiplication of the weight and input matrices.
The multiplication of two complex numbers is given by -
(a+ib)(c+id) = (ac-bd)+i(ad+bc)
I computed the backpropagation formula for this layer given we have the incoming gradient from the higher layer.
It comes out to be dL/dI(n) = (hermitian(W(n))).matmul(dL/dI(n+1))
where I(n) and W(n) are the input and weight of nth layer and I(n+1) is input of (n+1)th layer.
So I wished to check whether autograd is also computing dL/dI(n) using the same formula that I derived.
(Since Pytorch doesn't support complex-valued tensors backpropagation as for now, I have created my own representation of complex numbers by dealing with separate real and imaginary tensors)


title: google colab pytorch version error (course 231n)

body:
I'm going through cs231n exercises. Now I have a problem with it!
In assignment 2 in PyTorch.ipynb (The link of the file : https://github.com/israfelsr/CS231n/blob/master/assignment2/PyTorch.ipynb)
As usual, I run the first cell which mounts the drive and in the second cell it checks PyTorch version to be 1.4 by using assert command in python. That's where the error comes out. Does anybody know why this is happening and how can I fix that?

first cell

A Bit of information. It says if you're in colab you don't need to do anything

The second cell where it asserts to check PyTorch version is 1.4 and it's not and so shows error.
For more information please refer to the GitHub link I provided above.


title: Cannot use stable_baselines library, any ideas?

body:
When I try to run some code I get:

SyntaxError: invalid syntax straight from the import line
Traceback (most recent call last):
File "C:/Users/User/Documents/GitHub/Stock-Trading-Visualization/main.py", line 3, in  from stable_baselines.common.policies import MlpPolicy

The import line is from the official site of the library so it should work.
Probably, I have some problems with the environment, but I can't figure out what's wrong, any ideas?
I use:

PyCharm 2019.1.1 (Community Edition)
Build #PC-191.6605.12, built on April 3, 2019
JRE: 11.0.2+9-b159.34 amd64
JVM: OpenJDK 64-Bit Server VM by JetBrains s.r.o

Windows 10 10.0

Python 3.8.3

stable-baselines  2.10.1

pip   20.2.3


title: How to update parameter by hand during training without disturbing autograd in Pytorch?

body:
Here is my code,
class model(torch.nn.Module):
    def __init__(self, config):
        super(model, self).__init__()
        self.memory_matrix = torch.zeros((user_num, self.k, self.embedding_dim), device=self.device,
                                         requires_grad=False)

    def forward(self, users, items):
        for clicks in items:
            # ...
            self.memory_matrix[users] = self.memory_matrix[users] * (1 - z.unsqueeze(2) @ erase.unsqueeze(1)) \
                                    + z.unsqueeze(2) @ add.unsqueeze(1)
            a = a * self.memory_matrix

The model is about the recommendation system and a memory matrix is maintained for every user. I feed the model with user history clicks, that is, a sequence containing items that users interact with. When processing a user's history clicks, the memory matrix should be updated according to the current click. To reach this, I initialize the memory matrix as a tensor in __init__ function and update that memory matrix during the training. After finishing processing the whole history clicks, the model's parameters will be updated by the backward function. And here is the problem, the statement a = a * self.memory_matrix showing that the gradient of a depends on the memory matrix at that time which means the gradient of a depends on the old version of the memory matrix (e.g. the memory matrix when the loop is executed exactly once), not the final version of the memory matrix. This is because the memory matrix is updated during the following training. As a result, the autograd cannot work, because it doesn't store the old version of the memory matrix.
The memory matrix is always updated by hand and it is no need to calculate the gradient.
I find scatter_update in tensorflow can do this, how to achieve this in Pytorch?


title: How can I mask a fraction of the parameters while training a Pytorch model?

body:
I am training a simple pytorch model, initialised as usual:
           model = nn.Sequential(
            nn.Linear(input_dim, output_dim)).

I also have a specific matrix W of dimension (input_dim,output_dim) made by zeros and ones that I want to use as a mask during the whole training and scoring process to mask different parameters of the network. How can I do it?
In other words, I want to only train the non-zero parameters of
nn.Linear(input_dim, output_dim)*W

And keep all the others equal to zero.


title: How can I add zeroes to the end of a PyTorch tensor based on a modulo operation?

body:
I have a tensor with size: torch.Size([1, 305760])
Since 305760 is not divisible by 400, I want to change the size to: [1, 306000] filling the remaining spaces with 0. How can I do this with PyTorch?


title: Why are input data transferrd to Variable type?

body:
I'm reading code that implementing YOLOv3 with Pytorch, and coming with a line like this:
    for batch_i, (_, imgs, targets) in enumerate(dataloader):
            batches_done = len(dataloader) * epoch + batch_i
            imgs = Variable(imgs.to(device))  # ??
            targets = Variable(targets.to(device), requires_grad=False)

imgs is the input data, and I can't understand why there exits the transform: Variable(imgs.to(device)))
Does this mean that the input data should be trained(since the default option is that requires_grad=true) or is there another reason?


title: Why the trained neural network output the same results for each row in the last layer?

body:
I am trying to train a graph neural network using pytorch, but the problem is the final output contain the same values for each row,  here is the  layer of my network
self.linear_pre = nn.Linear(input_dim, feature_dim)  # (transform features)
self.linear_hidden = nn.Linear(input_dim, output_dim)  # -get different result
self.act = nn.ReLU()  # -get different result
self.linear_out_ = nn.Linear(output_dim, 1)  # same results for each row, example output is shown below

currently I am using one hot encoding as features for each graph, Adam optimizer, and weight initialization using xavier_uniform
I tried to change the learning rate, use random features, use scheduler still the problem persist
What else should I do? Any idea will be much appreciated


title: How to feed back LSTM output to input in pytorch dataloader

body: I’m doing video prediction frame prediction using LSTM. How do I implement this in my custom dataloader?


title: Why pytorch gradients are array but not vector?

body:
I am trying to calculate the dot of gradients of the same layer of two different epochs but when I am using print(model.layer1[0].weight.grad) it returns
tensor([[[[-1.1855e-03, -3.7884e-03, -2.8973e-03, -2.8847e-03, -9.6510e-04],
          [-2.0213e-03, -4.4927e-03, -5.4852e-03, -6.6060e-03, -3.5726e-03],
          [ 7.4499e-04, -1.8440e-03, -5.0472e-03, -5.6322e-03, -1.9532e-03],
          [-4.5696e-04,  9.6445e-04, -1.4923e-03, -2.9467e-03, -1.4610e-03],
          [ 2.4987e-04,  2.2086e-03, -7.6576e-04, -2.7009e-03, -2.8571e-03]]],

        [[[ 2.1447e-03,  3.1090e-03,  6.8175e-03,  6.4778e-03,  3.0501e-03],
          [ 2.0214e-03,  3.9936e-03,  7.9528e-03,  6.0224e-03,  1.7545e-03],
          [ 3.8781e-03,  5.6659e-03,  6.6901e-03,  5.4041e-03,  7.8014e-04],
          [ 4.4273e-03,  3.4548e-03,  5.7185e-03,  4.1650e-03,  9.9067e-04],
          [ 4.6075e-03,  4.1176e-03,  6.8392e-03,  3.4005e-03,  1.0009e-03]]],

        [[[-3.8654e-04, -2.9567e-03, -6.1341e-03, -8.3991e-03, -8.2343e-03],
          [-2.9113e-03, -5.4605e-03, -6.3008e-03, -8.2075e-03, -9.6702e-03],
          [-1.5218e-03, -4.4105e-03, -5.5651e-03, -6.8926e-03, -6.6076e-03],
          [-6.0357e-04, -3.1118e-03, -4.4441e-03, -4.0519e-03, -3.9733e-03],
          [-2.8683e-04, -1.6281e-03, -4.2213e-03, -5.5304e-03, -5.0142e-03]]],

        [[[-3.7607e-04, -1.7234e-04, -1.4569e-03, -3.5825e-04,  1.4530e-03],
          [ 2.6226e-04,  8.5076e-04,  1.2195e-03,  2.7885e-03,  2.5953e-03],
          [-7.7404e-04,  1.0984e-03,  7.8208e-04,  5.1286e-03,  4.6842e-03],
          [-1.8183e-03,  8.9730e-04,  1.0955e-03,  4.9259e-03,  6.4677e-03],
          [ 1.1674e-03,  4.0651e-03,  4.5886e-03,  8.3678e-03,  8.9893e-03]]],

Are that the gradients? If yes, why they are not vector? Below there is my neural network
class ConvNet(nn.Module):
    def __init__(self):
        super(ConvNet, self).__init__()
        self.layer1 = nn.Sequential(
            nn.Conv2d(1, 32, kernel_size=5, stride=1, padding=2),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2))
        self.layer2 = nn.Sequential(
            nn.Conv2d(32, 64, kernel_size=5, stride=1, padding=2),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2))
        self.drop_out = nn.Dropout()
        self.fc1 = nn.Linear(7 * 7 * 64, 1000)
        self.fc2 = nn.Linear(1000, 10)

    def forward(self, x):
        out = self.layer1(x)
        out = self.layer2(out)
        out = out.reshape(out.size(0), -1)
        out = self.drop_out(out)
        out = self.fc1(out)
        out = self.fc2(out)
        return out

Below is the code of how I train and compute the gradients
model = ConvNet()
klisi=[]
# Loss and optimizer
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

# Train the model
total_step = len(train_loader)
loss_list = []
acc_list = []
for epoch in range(num_epochs):
    for i, (images, labels) in enumerate(train_loader):
        # Run the forward pass
        outputs = model(images)
        loss = criterion(outputs, labels)
        loss_list.append(loss.item())

        # Backprop and perform Adam optimisation
        optimizer.zero_grad()
        loss.backward()

        optimizer.step()

        # Track the accuracy
        total = labels.size(0)
        _, predicted = torch.max(outputs.data, 1)
        correct = (predicted == labels).sum().item()
        acc_list.append(correct / total)

        if (i + 1) % 100 == 0:
            print('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}, Accuracy: {:.2f}%'
                  .format(epoch + 1, num_epochs, i + 1, total_step, loss.item(),
                          (correct / total) * 100))
    print(model.layer1[0].weight.grad)
    klisi.append(model.layer1[0].weight.grad)
    print(optimizer.param_groups[0]['lr'])
    optimizer.param_groups[0]['lr'] *= 0.9999


title: Using _scatter() to replace values in matrix

body:
Given the following two tensors:
x = torch.tensor([[[1, 2],
                   [2, 0],
                   [0, 0]],

                  [[2, 2],
                   [2, 0],
                   [3, 3]]]) # [batch_size x sequence_length x subseq_length]
y = torch.tensor([[2, 1, 0],
                  [2, 1, 2]]) # [batch_size x sequence_length]

I would like to sort the sequences in x based on their sub-sequence lengths (0 corresponds to padding in the sequence). y corresponds to the lengths of the sub-sequences in x. I have tried the following:
y_sorted, y_sort_idx = y.sort(dim=1, descending=True)
print(x.scatter_(dim=1, index=y_sort_idx.unsqueeze(2), src=x))

This results in:
tensor([[[1, 2],
         [2, 0],
         [0, 0]],

        [[2, 2],
         [2, 0],
         [2, 3]]])

However what I would like to achieve is:
tensor([[[1, 2],
         [2, 0],
         [0, 0]],

        [[2, 2],
         [3, 3],
         [2, 0]]])


title: Model does not train when using KFold

body:
I have run into what may be the most bizarre error of my life.
I have written a function to train a BERT model as follows :
def process_data(path, enc_pos = None, enc_tags = None):
    df = pd.read_csv(path, index_col=0).dropna(subset=['tag'])

    if enc_pos is None:
        enc_pos = LabelEncoder()
        df.loc[:, 'pos'] = enc_pos.fit_transform(df['pos'])
    else:
        df.loc[:, 'pos'] = enc_pos.transform(df['pos'])

    if enc_tags is None:
        enc_tags = LabelEncoder()
        df.loc[:, 'tag'] = enc_tags.fit_transform(df['tag'])
    else:
        df.loc[:, 'tag'] = enc_tags.transform(df['tag'])

    sentences = df.groupby("sentence")['word'].apply(list).values
    tags = df.groupby("sentence")['tag'].apply(list).values
    pos = df.groupby("sentence")['pos'].apply(list).values

    for i in range(df.shape[0]):
        assert len(sentences) == len(tags) == len(pos)

    return sentences, tags, pos, enc_tags, enc_pos

def _train(
    train_sentences,
    valid_sentences,
    train_pos,
    valid_pos,
    train_tags,
    valid_tags,
    num_pos,
    num_tags,
    save=False,
    early_stopping=None):

    num_unimproved_steps = 0

    train_dataset = Dataset(train_sentences, train_pos, train_tags)
    train_loader = torch.utils.data.DataLoader(
        train_dataset, batch_size=config.TRAIN_BATCH_SIZE)

    valid_dataset = Dataset(valid_sentences, valid_pos, valid_tags)
    valid_loader = torch.utils.data.DataLoader(
        valid_dataset, batch_size=config.VALID_BATCH_SIZE)

    device = torch.device(config.DEVICE)
    model = NERModel(num_tags, num_pos)
    model.to(device)

    param_optimizer = list(model.named_parameters())
    no_decay = ["bias", "LayerNorm.bias", "LayerNorm.weight"]
    optimizer_parameters = [
        {
            "params": [
                p for n, p in param_optimizer if not any(
                    nd in n for nd in no_decay
                )
            ],
            "weight_decay": 0.001,
        },
        {
            "params": [
                p for n, p in param_optimizer if any(
                    nd in n for nd in no_decay
                )
            ],
            "weight_decay": 0.0,
        },
    ]

    num_steps = int(len(train_sentences)/ config.TRAIN_BATCH_SIZE * config.EPOCHS)
    optimizer = AdamW(optimizer_parameters, lr=7e-5)

    scheduler = get_linear_schedule_with_warmup(
        optimizer, num_warmup_steps=0, num_training_steps= num_steps)

    best_loss = np.inf

    train_graphs = []
    valid_graphs = []

    for epoch in range(config.EPOCHS):
        train_loss, train_graph = train_fn(train_loader, model, device, scheduler, optimizer)
        valid_loss, valid_graph = eval_fn(valid_loader, model, device)

        train_graphs.append(train_graph)
        valid_graphs.append(valid_graph)

        print(f"For epoch {epoch}, Train Loss: {train_loss:.04f} , Valid Loss: {valid_loss:.04f}")

        if valid_loss < best_loss:
            num_unimproved_steps = 0
            best_loss = valid_loss
            if save:
                print("Validation loss improved. Now saving model ...")
                torch.save(model.state_dict(), config.MODEL_FILE)
        else:
            num_unimproved_steps += 1
            if num_unimproved_steps == early_stopping :
                print(f"Early Stopping: The model did not improve in the last {early_stopping} epochs.")
                break

    return best_loss, train_graphs, valid_graphs

# training section
def train_with_cv(early_stopping=None, n_folds=5):
    print(f"Now starting model training with {n_folds} fold cross validation...")
    sentences, tags, pos, enc_tags, enc_pos = process_data(config.TRAINING_FILE)
    num_unimproved_steps = 0

    num_pos = len(enc_pos.classes_)
    num_tags = len(enc_tags.classes_)

    kf = KFold(shuffle=True, random_state=42, n_splits=10)
    cv_losses = []
    fold = 0

    for train_i, valid_i in kf.split(sentences):
        train_sentences = sentences[train_i]
        valid_sentences = sentences[valid_i]

        train_pos = pos[train_i]
        valid_pos = pos[valid_i]

        train_tags = tags[train_i]
        valid_tags = tags[valid_i]

        # (
        #     train_sentences,
        #     valid_sentences,
        #     train_pos,
        #     valid_pos,
        #     train_tags,
        #     valid_tags
        # ) = train_test_split(
        #     sentences,
        #     pos,
        #     tags,
        #     test_size = 0.1
        # )
        print(f"Fold {fold+1} starting ...")

        loss, _, _ = _train(
            train_sentences=train_sentences,
            valid_sentences=valid_sentences,
            train_pos=train_pos,
            valid_pos=valid_pos,
            train_tags=train_tags,
            valid_tags=valid_tags,
            num_pos=num_pos,
            num_tags=num_tags,
            early_stopping=early_stopping
        )

        print(f"Fold {fold} finished with best loss: {loss}")
        cv_losses.append(loss)
        fold +=1
        if fold == n_folds:
            break

    return  cv_losses

If I comment out the K-fold code and uncomment the train_test_split code, the model trains fine and the validation loss decreases as expected to about 0.15. However, when I use K Fold, the validation loss does not improve at all! It is stuck around the 0.4 - 0.45 range.
I even tried printing out the sentences and the dtypes, but they seem to be identical. I don't understand what the difference is here.
Any help would be appreciated!


title: PyTorch multiprocessing: are ACER implementations moving gradients correctly?

body:
I am implementing the ACER architecture, which performs asynchronous parameter update as A3C. Looking at existing PyTorch implementations, I often see the parameter update performed as
# Transfers gradients from thread-specific model to shared model
def _transfer_grads_to_shared_model(model, shared_model):
  for param, shared_param in zip(model.parameters(), shared_model.parameters()):
    if shared_param.grad is not None:
      return
    shared_param._grad = param.grad

I understand that if shared_param.grad is not None it means another thread added a gradient to the model, thus we return to not overwrite it.
My questions are:

This way we are "throwing away" some of the gradients computed, am I right?

Why don't we use a lock mechanism when updating the shared model parameters? It would avoid two threads updating the parameters at the same time. Since the threads spend most of their time performing steps in the environment and updating their local models, adding a lock on the shared model update may have a negligible impact on the performances. Am I right on this? If yes, why isn't it done in practice?

What happens if a thread calls optimizer.step() on the shared_model while another thread is copying gradients to it?

Another implementation I found does it this way:
def copy_gradients_from(model, shared_model):
  for shared_parameter, parameter in zip(shared_model.parameters(), model.parameters()):
    shared_parameter._grad = parameter.grad

which does not check for existing gradients, and is also called without any Lock mechanism.


title: Runtime error in loading state_dict for VGG16 Model in Pytorch

body:
I download the code from this repo,I want to load their models and test them. However, I get this error:
Missing key(s) in state_dict: "vgg.model.0.weight", "vgg.model.0.bias", "vgg.model.2.weight", "vgg.model.2.bias", "vgg.model.5.weight", "vgg.model.5.bias", "vgg.model.7.weight", "vgg.model.7.bias", "vgg.model.10.weight", "vgg.model.10.bias", "vgg.model.12.weight", "vgg.model.12.bias", "vgg.model.14.weight", "vgg.model.14.bias", "vgg.model.17.weight", "vgg.model.17.bias", "vgg.model.19.weight", "vgg.model.19.bias", "vgg.model.21.weight", "vgg.model.21.bias".

Unexpected key(s) in state_dict: "vgg.features.0.weight", "vgg.features.0.bias",
"vgg.features.2.weight", "vgg.features.2.bias", "vgg.features.5.weight", "vgg.features.5.bias", "vgg.features.7.weight", "vgg.features.7.bias", "vgg.features.10.weight", "vgg.features.10.bias", "vgg.features.12.weight", "vgg.features.12.bias", "vgg.features.14.weight", "vgg.features.14.bias", "vgg.features.17.weight", "vgg.features.17.bias", "vgg.features.19.weight", "vgg.features.19.bias", "vgg.features.21.weight", "vgg.features.21.bias".

Please help me fix this.


title: How to define a Python Class which uses R code, but called from rTorch?

body:
I am using rTorch (e.g. https://f0nzie.github.io/rtorch-minimal-book/). I want to define a DataSet Class from within rTorch. I can do that with py_run_file. Yet, I want this DataSet Class use R code, because the data preparation is easier for me in tidyR. In Python I use rpy2 for this, and it works fine, when call from within Python.
Yet, when I do this embedded from rTorch or reticulate, it does not work (R studio crashes; on the command line I get "R is already initialized"), maybe because R-->Python-->R is a bit circular (cf. https://github.com/rstudio/reticulate/issues/56). [But I'd thought rpy2 can open another R session instance?]
Here is what I think would be a minimal example:
R code to prep data (in prepDataSet.R):
library(tidyverse)
prepDataSetFunc <- function(parameter) {
df <- tibble(name=parameter, x=1:10, y=10:1)
df
### Of course the real function does much more ...
}

Python code in file "PyClasses.py":
import numpy as np
import pandas as pd
import torch
from torch.utils.data import Dataset, DataLoader
import rpy2.robjects as ro
from rpy2.robjects.packages import importr
from rpy2.robjects import pandas2ri
pandas2ri.activate()

ro.r('''source('prepDataSet.R')''')
prepFunc = ro.globalenv["prepDataSetFunc"]

class RDataSet(DataSet):
    def __init__(self, py_parameter):
        self.df = prepFunc(parameter=py_parameter)
    def __len__(self):
        return len(self.dfy.index)

    def __getitem__(self, i):
        return self.df.iloc[i:(i+1)].values.astype('float32')

Main R code to do all the rTorch stuff:
library(rTorch)
library(reticulate)
library(tidyverse)
library(magrittr)

reticulate::use_condaenv("mrTorchEnv") ## This was another problem, i.e. I could not install rpy2 in rTorch environment, so I change the environment here

main <- py_run_file("PyClasses.py") ## This already fails

##### Then I want to do more stuff, i.e. Dataloader etc....

Or is there any way to define from rTorch a PyTorch DataSet Class, which then uses R code? Or pass an R object into the Class via the string???


title: How can I pass one image through Resnet50 using Pytorch?

body:
I would simply like to pass one image through the Resnet50 model and receive a tensor or classification value as an output, but I get an error TypeError: 'list' object is not callable.
I am able to pass my image through the VGG16 model so I wonder what I have to do differently.
I select one image to convert to tensor for Pytorch and format it for CUDA.
from torchvision.models import *
from torchvision.transforms import ToTensor

import glob
import PIL

vgg_m = vgg16_bn(True).features.cuda().eval()
resnet50 = resnet50(pretrained=True).cuda().eval()

path = '/home/iii/tor/course-v3/pets_set/images/*'

img_path = glob.glob(path)[385]

image1 = PIL.Image.open(img_path)
image2 = ToTensor()(image1).unsqueeze(0)
image3 = image2.cuda()
image3.shape

returns
torch.Size([1, 3, 376, 500])
When I input this image3 tensor in the VGG16 model I get a tensor output. (I am showing this working case first to illustrate exactly what I am trying to make happen).
VGG16(image3).shape
returns
torch.Size([1, 512, 7, 7])
I was expecting the same thing with Resnet50, but when I try, I get...
resnet50(image3)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-780-6b0696a7d406> in <module>
----> 1 resnet50(image3)

~/miniconda3/envs/ml/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

~/miniconda3/envs/ml/lib/python3.7/site-packages/torchvision/models/resnet.py in forward(self, x)
    218
    219     def forward(self, x):
--> 220         return self._forward_impl(x)
    221
    222

~/miniconda3/envs/ml/lib/python3.7/site-packages/torchvision/models/resnet.py in _forward_impl(self, x)
    201     def _forward_impl(self, x):
    202         # See note [TorchScript super()]
--> 203         x = self.conv1(x)
    204         x = self.bn1(x)
    205         x = self.relu(x)

~/miniconda3/envs/ml/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    729                 _global_forward_hooks.values(),
    730                 self._forward_hooks.values()):
--> 731             hook_result = hook(self, input, result)
    732             if hook_result is not None:
    733                 result = hook_result

TypeError: 'list' object is not callable

I've also passing the image tensor with a [0] index in case there happened to be extra brackets around the tensor for some reason but that just returns an error that Resnet50 was expecting an object of four dimensions.
resnet50(image3[0])
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-781-9a0d05885a20> in <module>
----> 1 resnet50(image3[0])

~/miniconda3/envs/ml/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

~/miniconda3/envs/ml/lib/python3.7/site-packages/torchvision/models/resnet.py in forward(self, x)
    218
    219     def forward(self, x):
--> 220         return self._forward_impl(x)
    221
    222

~/miniconda3/envs/ml/lib/python3.7/site-packages/torchvision/models/resnet.py in _forward_impl(self, x)
    201     def _forward_impl(self, x):
    202         # See note [TorchScript super()]
--> 203         x = self.conv1(x)
    204         x = self.bn1(x)
    205         x = self.relu(x)

~/miniconda3/envs/ml/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

~/miniconda3/envs/ml/lib/python3.7/site-packages/torch/nn/modules/conv.py in forward(self, input)
    421
    422     def forward(self, input: Tensor) -> Tensor:
--> 423         return self._conv_forward(input, self.weight)
    424
    425 class Conv3d(_ConvNd):

~/miniconda3/envs/ml/lib/python3.7/site-packages/torch/nn/modules/conv.py in _conv_forward(self, input, weight)
    418                             _pair(0), self.dilation, self.groups)
    419         return F.conv2d(input, weight, self.bias, self.stride,
--> 420                         self.padding, self.dilation, self.groups)
    421
    422     def forward(self, input: Tensor) -> Tensor:

RuntimeError: Expected 4-dimensional input for 4-dimensional weight [64, 3, 7, 7], but got 3-dimensional input of size [3, 224, 224] instead


title: Pytorch, standard layer to convert sequential output to binary?

body:
I am working on a new Pytorch model which takes sequential data as input and I need to output just a single value, which I will then use a binary cross-entropy function to evaluate as a probability of 1 or 0.
To be more concrete, lets say my sequence is 1000 time steps and only 2 dimensions, like a 2-dimensional sine wave, so the data shape would be 1000 x 2.
I have done something like this before using an RNN, which there is a lot of content online. Because of the recurrent structure of the RNN, in order to do this we just look at final output of the RNN after processing the sequence. In this way the the final step output would be 2 dimensions, then we can apply a linear layer to convert 2 -> 1 dimension, et voila, its done.
MY PROBLEM:
What I am attempting to do now is not using a recurrent network, but instead an encoder with attention (Transformer). So the output of the encoder is now still 1000 steps long and whatever my embedded dimension is, likes say 8. So the output of the sequential encoder is shape 1000 x 8. So my issue is that I need to convert this output to a single value, to which I can apply the binary cross-entropy function. I am not finding an obvious way to do this.
IDEAS:
Traditionally with this kind of sequential model, the encoder feeds into a decoder and the decoder can then output a variable length sequence (this is used to language translation problems). My problem is different in that I don't want to output another sequence but just a single value. Maybe I need to convert the decoder in such a way where this works? The decoder usually takes a target value as well as the output from the encoder as input. The output from the decoder then has the same shape as this target value. An idea would be to use the traditional decoder and give a 1 length target, I would then get a 1 length output and I could use a traditional linear layer to convert this to my desired output. However this doesn't seem entirely logical because I really am not interested in outputting a sequence but just 1 value.
Anyways just looking for some more ideas from the community, if you have any. Thanks!


title: PyTorch map 3D feature into 2D according to arbitrary slicing planes

body:
In PyTorch, I wonder how I can build a function map_feat(feat, plane_mask) that can map a 3D feature [B, C, D, H, W] into a stack of 2D feature [B, C, H', W'] based on a few pre-defined slicing planes. Here is a toy example,
import torch
import numpy as np

def coordinates(voxel_dim, device=torch.device('cpu')):
    nx, ny, nz = voxel_dim
    x = torch.arange(nx, dtype=torch.long, device=device)
    y = torch.arange(ny, dtype=torch.long, device=device)
    z = torch.arange(nz, dtype=torch.long, device=device)
    x, y, z = torch.meshgrid(x, y, z)
    return torch.stack((x.flatten(), y.flatten(), z.flatten()))

def get_plane_mask(verts, n1, n2, n3):
    a =  verts[0] * n1 + verts[1] * n2 + verts[2] * n3
    return a == 1

I have a 3D feature from network. To make things simple, let's say batch size = 1 and channel = 5
voxel_dim = [10, 10, 5]
feat = torch.randn([1, 5] + voxel_dim)

And I have a few pre-defined slicing planes
verts = coordinates(voxel_dim)
plane_mask = []
plane_mask.append(get_plane_mask(verts, 0.1, 0.2, 0.))
plane_mask.append(get_plane_mask(verts, 0.2, 0, 0))
plane_mask.append(get_plane_mask(verts, 0, 0, 0.5))

Next, I wish to index the feature values lying on these planes and interpolate them into a new 3D feature volume, where the last dimension corresponding to the index of the plane masks, and the first 4 dimensions are the [B, C, H', W'] as described above.
new_feat = torch.zeros([1, 3, 8, 8, len(plane_mask)])
for i in range(len(plane_mask)):
    new_feat[:,:,:,:, i] = map_feat(feat, plane_mask[i])

This mapping is sort of similar to the CT scan, in which, given a slicing plane, we get a 2D image show the cross section of a 3D volume. Any idea to implement this function using Pytorch or Numpy is appreciated! It would be even better if we can do it in one pass without for loop. I can save the plane mask into a sparse tensor.
Currently, I am only considering the case where cross_section is a rectangle, but if your idea  can deal with more general cases (triangle, pentagon, etc), it would be perfect!


I want to train my dataset with training data and validation data.
Total data is 2048, train data is 1638, and validation data is 410 (20% of total).
Here are my codes

loading data (org: total training data)
 org_x = train_csv.drop(['id', 'digit', 'letter'], axis=1).values
 org_x = org_x.reshape(-1, 28, 28, 1)
 org_x = org_x/255
 org_x = np.array(org_x)
 org_x = org_x.reshape(-1, 1, 28, 28)
 org_x = torch.Tensor(org_x)

 x_test = test_csv.drop(['id','letter'], axis=1).values
 x_test = x_test.reshape(-1, 28, 28, 1)
 x_test = x_test/255
 x_test = np.array(x_test)
 x_test = x_test.reshape(-1, 1, 28, 28)
 x_test = torch.Tensor(x_test)

 y = train_csv['digit']
 y = list(y)
 print(len(y))
 org_y = np.zeros([len(y), 1])
 for i in range(len(y)):
   org_y[i] = y[i]
 org1 = np.array(org_y, dtype=object)

splitting data (org: total training data)
 from sklearn.model_selection import train_test_split

 x_train, x_valid, y_train, y_valid = train_test_split(
    org, org1, test_size=0.2, random_state=42)

transform
 transform = transforms.Compose([transforms.ToPILImage(),
                         transforms.ToTensor(),
                         transforms.Normalize((0.5, ), (0.5, )) ])

dataset
 class kmnistDataset(data.Dataset):
 def __init__(self, images, labels=None, transforms=None):
     self.x = images
     self.y = labels
     self.transforms = transforms

 def __len__(self):
     return (len(self.x))

 def __getitem__(self, idx):
     data = np.asarray(self.x[idx][0:]).astype(np.uint8)

     if self.transforms:
         data = self.transforms(data)

     if self.y is not None:
         return (data, self.y[i])
     else:
         return data

 train_data = kmnistDataset(x_train, y_train, transform)
 valid_data = kmnistDataset(x_valid, y_valid, transform)

 train_loader = DataLoader(train_data, batch_size=16, shuffle=True)
 valid_loader = DataLoader(valid_data, batch_size=16, shuffle = False)

I'll skip the model structure.

training(Here, I got the error message)
n_epochs = 30

valid_loss_min = np.Inf

for epoch in range(1, n_epochs+1):
    train_loss = 0
    valid_loss = 0

###################
# train the model #
###################
model.train()
for data in train_loader:
    inputs, labels = data[0], data[1]
    optimizer.zero_grad()
    output = model(inputs)
    loss = criterion(output, labels)
    loss.backward()
    optimizer.step()
    train_loss += loss.item()*data.size(0)

#####################
# validate the model#
#####################
model.eval()
for data in valid_loader:
    inputs, labels = data[0], data[1]
    output = model(inputs)
    loss = criterion(output, labels)
    valid_loss += loss.item()*data.size(0)

train_loss = train_loss/ len(train_loader.dataset)
valid_loss = valid_loss / len(valid_loader.dataset)

print('Epoch: {} \tTraining Loss: {:.6f} \tValidation Loss: {:.6f}'.format(
    epoch, train_loss, valid_loss))

Although I checked the data size, I got the error message below.

index 2047 is out of bounds for axis 0 with size 1638

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-42-b8783819421f> in <module>
     11     ###################
     12     model.train()
---> 13     for data in train_loader:
     14         inputs, labels = data[0], data[1]
     15         optimizer.zero_grad()

/opt/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py in __next__(self)
    433         if self._sampler_iter is None:
    434             self._reset()
--> 435         data = self._next_data()
    436         self._num_yielded += 1
    437         if self._dataset_kind == _DatasetKind.Iterable and \

/opt/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py in_next_data(self)
    473     def _next_data(self):
    474         index = self._next_index()  # may raise StopIteration
--> 475         data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
    476         if self._pin_memory:
    477             data = _utils.pin_memory.pin_memory(data)

/opt/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py in fetch(self,
possibly_batched_index)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---> 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

/opt/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py in <listcomp(.0)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---> 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

<ipython-input-38-e5c87dd8a7ff> in __getitem__(self, idx)
     17
     18         if self.y is not None:
---> 19             return (data, self.y[i])
     20         else:
     21             return data

IndexError: index 2047 is out of bounds for axis 0 with size 1638

Can you explain why and how to solve it?


title: What is a tensor argument to Normal supposed to mean in Distributions Package of Pytorch?

body:
I understand torch.Normal(loc, scale) is a class corresponding to univariate normal distribution in pytorch.
I understand how it works when loc and scale are numbers.
The problem is when the inputs to torch.Normal are tensors as opposed to numbers. In that case I do not understand it well. What is the exact interpretation/usage of such tensor arguments?
See for example y_dist in code below. loc and scale are tensors for y_dist. What does this exactly mean? I do not think this converts the univariate distribution to multivariate, does it? Does it instead form a group of univariate distributions?
import torch as pt
ptd = pt.distributions
x_dist = ptd.Normal(loc = 2, scale = 3)
x_samples = x_dist.sample()

batch_size = 256
y_dist = ptd.Normal(loc = 0.25 * pt.ones(batch_size, dtype=pt.float32), scale = pt.ones(batch_size, dtype=pt.float32))


title: How to compute standard deviation of predcited tensors?

body: I have an undirected graph data where G = (V, E). I implemented graph convolution to calculate node embedding. For each fold, I have 38 testing samples. My model computes node embedding and transforms them into adjacency matrix format. My question how can I compute the standard deviation of these 38 predicted adjacency matrices in each fold? Should I compute the standard deviation from node embedding? If so, what is the true way to calculate?


title: Changing Dropout value during training

body:
How can I change Dropout during training? For example
Dropout= [0.1, 0.2, 0.3]

I tried passing it as as a list but I couldn't make it work.


title: Is there a way to classify a set of data as a whole via Pytorch?

body:
I'm currently dealing with a classification task on a CT dataset. In CT datasets, multiple slices belong to one single patient, while setting up my dataset, I arrange my data as follows:
dataset/0/patient_1/1.png,2.png...
dataset/0/patient_2/1.png,2.png...
I wonder is there a way to let my network to classify by patient instead of by slices?
thank you


title: Image classification Using Pytorch

body:
this is the code where I was working on Image Classification using Pytorch and I'm not able to get the  accuracy right.
the accuracy is exceeding 100 ,can anyone help me to find the error.
     def trained_model(criterion, optimizer, epochs=5):

      epoch_loss = 0.0
      epoch_accuracy = 0
      running_loss = 0
      running_accuracy = 0
      total = 0

      for epoch in range(epochs):
        print('epoch : {}/{}'.format(epoch+1, epochs))

        for images, labels in train_loader:
          images, labels = images.to(device), labels.to(device)

          optimizer.zero_grad()

          outputs = model(images)
          loss = criterion(outputs, labels)

          _, predictions = torch.max(outputs, dim=1)

          loss.backward()
          optimizer.step()

          running_loss += loss.item()

          running_accuracy += torch.sum(predictions == labels.data)

        epoch_loss = running_loss / len(train_dataset)
        epoch_accuracy = running_accuracy / len(train_dataset)

        print('Loss:{:.4f} , Accuracy : {:.4f} '.format(epoch_loss, epoch_accuracy))

      return model


title: Pytorch can't access cuda, every torch.cuda code returns nothing

body:
it's my first time posting here.
I'm trying to set up Pulse AI which matches a face with another face. I used it on an older PC previously which worked nicely, but after I try to install it on a new PC, it doesn't work anymore, and I pinpointed the problem at torch.cuda. (Please skip to the bold lines if you're not interested in the background.)
Pulse AI has two major files, align_face.py and PULSE.py.
align_face.py detects a face and transforms it into a small file. It used to run in a few seconds.
PULSE.py matches the file generated by align_face.py with another face. It used to run in 90 seconds and regardless of its quality, provided results.
Now, align_face.py takes 5 minutes to run, and PULSE.py takes 15 minutes to run and never provided any result.
After some debugging attempt (by adding print("test") after every suspicious line, if the test doesn't print or takes very long to print, then that line has problem; yes, I'm a newbie in Python), I found that in align_face.py

face_tensor =
torchvision.transforms.ToTensor()(face).unsqueeze(0).cuda()

is what slows it down.
In PULSE.py it is, surprise,

self.synthesis = G_synthesis().cuda()

After hours of googling I found that cuda could be the issue. And it is proven because in the test.py file I made
import torch

torch.cuda.current_device()
torch.cuda.device(0)
torch.cuda.device_count()
torch.cuda.get_device_name(0)
torch.cuda.is_available()

return nothing. Google says at least torch.cuda.is_available() should return true or false, but it returns really nothing. Type python test.py and hit enter, then it is like this
(pulse) E:\pulse>python test.py

(pulse) E:\pulse>

Tried installing cuda manually then also reinstalled Pytorch. Same result. Obviously, torch or cuda was not installed properly or cuda has some problem that I wasn't sure about, but I've tried every way to install them that I found on google and other websites and no one seems to have encountered such an issue.
Would appreciate your help with this.


title: How to load MoCo model weights that are stored as an state_dict?

body:
I am new to PyTorch and I am trying to load the MoCo model in order to use it.
In the following repo I have found the code and also, I downloaded the pre-trained model (moco_v2_800ep_pretrain.pth.tar) which is a state_dict with the model’s weights.
I know that in order to use a model I need to create a model instance first and then load the state_dict.
My problem is that I can not create an instance model of MoCo. I saw the code on GiHub and I end up with the following code:
    moco = torch.load('moco_v2_800ep_pretrain.pth.tar')
    moco_state_dict = moco['state_dict']

    my_moco_model = MoCo(base_encoder, dim=128, K=65536, m=0.999, T=0.07, mlp=False)
    my_moco_model.load_state_dict(new_dict)

I do not know what to put in the base_encoder parameter or tell me if there is another way to load the model.
Can anyone help me please?


title: Train simple RNN from my own dataset in pytorch

body:
edited after @Nerveless_child answer
I have a file with words-bitstrings as key and True/False for value which represents if word is in my dictionary.

010000101010000, False

10100010110010001011, True

The words represents kind of pattern and I want to train rnn network that can identify if word is in the language or not (simple binary classifier).
my Dataset:
class myDataset(T.utils.data.Dataset):
# WORD  T/f
# 010000101010000  FALSE

    tmp_x = np.loadtxt(src_file, max_rows=m_rows,
                       usecols=[0], delimiter=",", skiprows=0,
    dtype=np.int64)

    tmp_y = np.genfromtxt(src_file, max_rows=m_rows,
                       usecols=[1], delimiter=",", dtype=bool)

    tmp_y = tmp_y.reshape(-1, 1)  # 2-D required

    self.x_data = T.from_numpy(tmp_x).to(device)

def __getitem__(self, index):
    return self.x_data[index], self.y_data[index]

def __len__(self):
    return len(self.x_data)

and when I'm trying to train the network
    net.train()  # set mode
for epoch in range(0, max_epochs):
    T.manual_seed(1 + epoch)  # recovery reproducibility
    epoch_loss = 0  # for one full epoch

    for (batch_idx, batch) in enumerate(train_ldr):
        (X, Y) = batch  # (predictors, targets)
        optimizer.zero_grad()  # prepare gradients
        oupt = net(X)  # predicted prices
        loss_val = loss_func(oupt, Y)  # avg per item in batch
        epoch_loss += loss_val.item()  # accumulate avgs
        loss_val.backward()  # compute gradients
        optimizer.step()  # update wts

I get the error

OverflowError: Python int too large to convert to C long


title: Python: append images in for loop use a lot of memory

body:
I'm actually facing a problem while working on a python project.
I'm appending some images in a for loop and it uses a lot of RAM memory.
If you guys, have any solution to optimize this for loop, It'll help me a lot.
Thanks!
augment_img = []
augment_label = []
augment_weight = []
for i in range(4):
    for j in range(len(train_dataset)):
        single_img, single_label, single_weight = train_dataset[j]
        augment_img.append(single_img)
        augment_label.append(single_label)
        augment_weight.append(single_weight)
        if j % 1000==0:
            print(j)```


title: Error while importing pytorch in anaconda prompt

body:
Microsoft Visual C++ Redistributable is not installed, this may lead to the DLL load failure.
Microsoft Visual C++ Redistributable is not installed, this may lead to the DLL load failure.
It can be downloaded at https://aka.ms/vs/16/release/vc_redist.x64.exe
Traceback (most recent call last):
File "", line 1, in
File "C:\ProgramData\Anaconda3\lib\site-packages\torch_init_.py", line 128, in
raise err
OSError: [WinError 126] The specified module could not be found. Error loading "C:\ProgramData\Anaconda3\lib\site-packages\torch\lib\asmjit.dll" or one of its dependencies.
getting this message


title: DCGANs discriminator accuracy metric using PyTorch

body:
I am implementing DCGANs using PyTorch.
It works well in that I can get reasonable quality generated images, however now I want to evaluate the health of the GAN models by using metrics, mainly the ones introduced by this guide https://machinelearningmastery.com/practical-guide-to-gan-failure-modes/
Their implementation uses Keras which SDK lets you define what metrics you want when you compile the model, see https://keras.io/api/models/model/. In this case the accuracy of the discriminator, i.e. percentage of when it successfully identifies an image as real or generated.
With the PyTorch SDK, I can't seem to find a similar feature that would help me easily acquire this metric from my model.
Does Pytorch provide the functionality to be able to define and extract common metrics from a model?


title: How to you select percentiles of Categorical distribution in numpy or Pytorch

body:
I have network which outputs, (9 different) equally spaced categorical distributions (basically  a Tensor of size [batch_size,9,support]).
I want to find an easy way to find percentiles, but I am not sure how to do that. I know I can compute the cdf using torch.cumsum(dim=-1), but I am not sure I know how to extract the indices after that.
Any help would be appreciated.
Edit: As an example say I have the following, which can be interpreted as 64*9 discrete distributions. I want to find the percentiles of these distributions.
dists=F.softmax(torch.randn(64,9,51),dim=-1)
support=torch.linspace(-1,1,51) #or any other equally spaced support


title: Parallelizing in Python

body:
I've got something I'd like to do in parallel in python. The following function:
def _train_users_locally(self):
        for i in self.users:
            self.users[i].train()

Gets a bunch of instances of the 'user' class to train a pytorch neural net:
def train(self):
        self.current_model.train()
        optim = self.optim(self.current_model.parameters(), lr=self.lr)
        criterion = self.criterion()
        for epoch in range(self.epochs):
            loss_per_epoch = 0
            counter = 0
            for i, data in enumerate(self.dataloader):
                x, y = data
                fx = self.current_model(x.unsqueeze(1))
                loss = criterion(fx, y)
                optim.zero_grad()
                loss.backward(retain_graph=True)
                optim.step()
                loss_per_epoch += loss.item()
                print('\rEpoch {}\tBatch: {:.3f}, Loss: {:.3f}'.format(epoch+1, i, loss.item()), end="")
                counter += 1
            print('\nEpoch {}\t Average Loss: {:.3f}'.format(epoch+1, loss_per_epoch / counter))

Nothing is being output; the model assigned to the object is being updated. I would like each user object to do its training simultaneously, but I can't for the life of me figure out how to do this, as all the examples I've been able to find involve some processing of list elements.


title: NLP Transformers - PyTorch RuntimeError: shape [5, -1, 32783] invalid for input of size 655360

body:
I am trying to load a SuperTransformer model once, and perform predictions on the dataset multiple times in a loop - while changing the sampled model configuration each time.
I am using the generate.py file in github.com/mit-han-lab/hardware-aware-transformers to run translations on the WMT'14En-De dataset. The part of the code that changes the sampled model configuration is model.set_sample_config(config) in line 67 of generate.py.
Basically, what I am trying to achieve is to repeat inference on the dataset with different SubTransformer configurations - while only loading the dataset and model once (reducing as much overhead as possible on the second iteration).
I am currently trying to implement the modified code below, with a while loop after the #Load ensemble stage and changing the 'config' variable after each iteration.
 # Load ensemble
print('| loading model(s) from {}'.format(args.path))
models, _model_args = checkpoint_utils.load_model_ensemble(
    args.path.split(':'),
    arg_overrides=eval(args.model_overrides),
    task=task,
)

torch.manual_seed(args.seed)

#loop twice, change 'config' variable on second iteration to have different SubTransformer configuration
while i in range (0,2):

    # Optimize ensemble for generation
    for model in models:
        if use_cuda:
            model.cuda()

        config = utils.get_subtransformer_config(args)

        model.set_sample_config(config)
        model.make_generation_fast_(
            beamable_mm_beam_size=None if args.no_beamable_mm else args.beam,
            need_attn=args.print_alignment,
        )

However, I get the following error: shape is invalid for input of size x
The problem seems to be with the size of the 'lprobs' variable in sequence_generator.py, where the size is always wrong on the second iteration. This error happens even when the code loops with the same SubTransformer configuration again.
I am fairly new to PyTorch libraries, and machine learning in general, so any help I can get with this would be appreciated! Thanks.


title: PyTorch ToTensor scaling to [0,1] discrepancy

body:
If I do this
mnist_train = MNIST('../data/MNIST', download = True,
                transform = transforms.Compose([
                    transforms.ToTensor(),
                ]), train = True)

and
mnist_train.data.max()
why do I get 255? I should get 1, because ToTensor() scales to [0,1], right?
If  I do:
for i in range(0, len(mnist_train)):
    print(mnist_train[i][0].max())

then, I get almost 1?
Could someone please help me understand this?


title: Pytorch operations (adding and average) between layers

body:
I am building a pytorch nn model that uses skip connections between two parallel sequential layers. This model is known as the merge-and-run. I will include an image of the model as given by the paper publication.
merge-and-run model
You can look it up in the literature for more reference on how it works. The thing I'm having problem with is the operation of adding and dividing skip connections from the output of one layer to a point farther down the line in the model. One of the operations involve in these skip connections is an addition. I am doing this addition using the following code:
def forward(self, x):
    x = self.relu1(self.conv1(x))
    print("X size: ", x.size())
    av0 = (x + x)/2

    b1 = self.block1_up(av0)
    b2 = self.block1_up(av0)

    print("av0 size: ", av0.size())
    print("b1 size: ", b1.size())
    add1 = b1 + av0
    add2 = b2 + av0

The print statements project respectively the following:
X size:  torch.Size([128, 64, 32, 32])
av0 size:  torch.Size([128, 64, 32, 32])
b1 size:  torch.Size([128, 128, 1, 1])

But when calculating "add1" I get the following error:
RuntimeError: The size of tensor a (128) must match the size of tensor b (64) at non-singleton dimension 1

I have tried changing the dimensions of av0 so it can fit with b1 using the view() function torch provides, but this didn't help all that much. Any help with this type of problem?
Thank you.


title: How to change tensor like numpy and pytorch do in tensorflow2?

body:
I want to implement Informer(a model in deep-learning) in Tensorflow2.1.
There are some codes like this:
Q = tensor.shape(B,H,L,D) (for example, Q is a tensor and shape is (B,H,L,D) B is batch)
index = tensor.shape(B,C) （C < L）
Q_reduce = Q[torch.arange(B)[:, None, None], torch.arange(H)[None, :, None], index, :]
print(Q_reduce.shape)
# get shape is (B,B,C,D)

How can I get same result in Tensorflow2?
I use tf.gather like this:
Q_reduce = tf.gather(Q,index, axis=-2)
# get shape is (B,H,B,C,D)


title: Torch is not saving my freezed and optimized model

body:
when I start my script it runs fine until it hits the traced_model.save(args.save_path) statement after that the script just stop running.
Could someone please help me out with this?
import argparse
import torch
from model import SpeechRecognition
from collections import OrderedDict

def trace(model):
    model.eval()
    x = torch.rand(1, 81, 300)
    hidden = model._init_hidden(1)
    traced = torch.jit.trace(model, (x, hidden))
    return traced

def main(args):
    print("loading model from", args.model_checkpoint)
    checkpoint = torch.load(args.model_checkpoint, map_location=torch.device('cpu'))
    h_params = SpeechRecognition.hyper_parameters
    model = SpeechRecognition(**h_params)

    model_state_dict = checkpoint['state_dict']
    new_state_dict = OrderedDict()
    for k, v in model_state_dict.items():
        name = k.replace("model.", "") # remove `model.`
        new_state_dict[name] = v

    model.load_state_dict(new_state_dict)

    print("tracing model...")
    traced_model = trace(model)
    print("saving to", args.save_path)
    traced_model.save(args.save_path)
    print("Done!")

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="testing the wakeword engine")
    parser.add_argument('--model_checkpoint', type=str, default='your/checkpoint_file', required=False,
                        help='Checkpoint of model to optimize')
    parser.add_argument('--save_path', type=str, default='path/where/you/want/to/save/the/model', required=False,
                        help='path to save optmized model')

    args = parser.parse_args()
    main(args)

If you start the script you can even see where it stops working because print("Done!") is not executed.
Here is what it looks in the terminal when I run the script:
loading model from C:/Users/supre/Documents/Python Programs/epoch=0-step=11999.ckpt
tracing model...
saving to C:/Users/supre/Documents/Python Programs


title: Reinforcement Learning TypeError: expected CPU (got CUDA) - how to solve

body:
I know this problem has been raised many times, so please forgive me. I'm trying to run my reinforcement model but I get an error TypeError: expected CPU (got CUDA). I found this solution in another thread but don't know how to incorporate it in my code (I'm still a dummy newbie):

Pytorch requires your to do self.module_name = module for things to work correctly. It's okay to keep them in a list. Just do something like setattr(self, 'emb_{}'.format(i), emb) for each step in that loop.

This is my code:
class Policy(nn.Module):
    def __init__(self):
        super(Policy, self).__init__()
        self.input_layer = nn.Linear(11, 128)
        self.hidden_1 = nn.Linear(128, 128)
        self.hidden_2 = nn.Linear(32,31)
        self.hidden_state = torch.tensor(torch.zeros(2,1,32)).cuda()
        self.rnn = nn.GRU(128, 32, 2)
        self.action_head = nn.Linear(31, 5)
        self.value_head = nn.Linear(31, 1)
        self.saved_actions = []
        self.rewards = []

    def reset_hidden(self):
        self.hidden_state = torch.tensor(torch.zeros(2,1,32)).cuda()

    def forward(self, x):
        x = torch.tensor(x).cuda()
        x = torch.sigmoid(self.input_layer(x))
        x = torch.tanh(self.hidden_1(x))
        x, self.hidden_state = self.rnn(x.view(1,-1,128), self.hidden_state.data)
        x = F.relu(self.hidden_2(x.squeeze()))
        action_scores = self.action_head(x)
        state_values = self.value_head(x)
        return F.softmax(action_scores, dim=-1), state_values

        def forward(self, x):
          conv_out = self.conv(x).view(x.size()[0], -1)
          val = self.fc_val(conv_out)
          adv = self.fc_adv(conv_out)
          return val + (adv - adv.mean(dim=1, keepdim=True))

    def act(self, state):
        probs, state_value = self.forward(state)
        m = Categorical(probs)
        action = m.sample()
        if action == 1 and env.state[0] < 1: action = torch.LongTensor([2]).squeeze().cuda
        if action == 4 and env.state[1] < 1: action = torch.LongTensor([2]).squeeze().cuda
        if action == 6 and env.state[2] < 1: action = torch.LongTensor([2]).squeeze().cuda
        self.saved_actions.append((m.log_prob(action), state_value))
        return action.item()

env = TradingEnvironment(max_stride=4, series_length=250, starting_cash_mean=1000, randomize_cash_std=100, starting_shares_mean=100, randomize_shares_std=10)
model = Policy().cuda()
optimizer = optim.Adam(model.parameters(), lr=3e-4)

env.reset()
# In case you're running this a second time with the same model, delete the gradients
del model.rewards[:]
del model.saved_actions[:]

gamma = 0.9
log_interval = 60

def finish_episode():
    R = 0
    saved_actions = model.saved_actions
    policy_losses = []
    value_losses = []
    rewards = []
    for r in model.rewards[::-1]:
        R = r + (gamma * R)
        rewards.insert(0, R)
    rewards = torch.tensor(rewards)

    epsilon = (torch.rand(1) / 1e4) - 5e-5
    # With different architectures, I found the following standardization step sometimes
    # helpful, sometimes unhelpful.
    #
    rewards = (rewards - rewards.mean()) / (rewards.std(unbiased=False) + epsilon)
    # Alternatively, comment it out and use the following line instead:
    rewards += epsilon

    for (log_prob, value), r in zip(saved_actions, rewards):
        reward = torch.tensor(r - value.item()).cuda()
        policy_losses.append(-log_prob * reward)
        value_losses.append(F.smooth_l1_loss(value, torch.tensor([r]).cuda()))

    optimizer.zero_grad()
    loss = torch.stack(policy_losses).sum() + torch.stack(value_losses).sum()
    loss = torch.clamp(loss, -1e-5, 1e5)
    loss.backward()
    optimizer.step()
    del model.rewards[:]
    del model.saved_actions[:]

running_reward = 0
for episode in range(0, 4000):
    state = env.reset()
    reward = 0
    done = False
    msg = None
    while not done:
        action = model.act(torch.Tensor(state.cuda()))
        state, reward, done, msg = env.step(action)
        model.rewards.append(reward)
        if done:
            break
    running_reward = running_reward * (1 - 1/log_interval) + reward * (1/log_interval)
    finish_episode()
    # Resetting the hidden state seems unnecessary - it's effectively random from the previous
    # episode anyway, more random than a bunch of zeros.
    # model.reset_hidden()
    if msg["msg"] == "done" and env.portfolio_value() > env.starting_portfolio_value * 1.1 and running_reward > 500:
        print("Early Stopping: " + str(int(reward)))
        break
    if episode % log_interval == 0:
        print("""Episode {}: started at {:.1f}, finished at {:.1f} because {} @ t={}, \
last reward {:.1f}, running reward {:.1f}""".format(episode, env.starting_portfolio_value, \
              env.portfolio_value(), msg["msg"], env.cur_timestep, reward, running_reward))

The error message is the following:
TypeError                                 Traceback (most recent call last)
<ipython-input-19-59ec54f626db> in <module>()
     46     msg = None
     47     while not done:
---> 48         action = model.act(torch.Tensor(state.cuda()))
     49         state, reward, done, msg = env.step(action)
     50         model.rewards.append(reward)

TypeError: expected CPU (got CUDA)

Any chance you could tell me where and how I need to make changes? Of course if you know a different, better solution I'd be extremaly grateful.
Joanna


title: Semantic Segmentation Post Processing Optimization

body:
I am currently deploying my semantic segmentation model in Nvidia AGX Xavier(using pytorch), the bottleneck in my code is when I transfer the output of the model to CPU in order to  convert it to numpy arrays, before doing some post processing and saving the image.
Is there any efficient way to do the same job?
The following line of the code is the bottleneck
output = output.cpu().data[0].numpy()


title: Apply vectorised function to Cartesian product of two ranges in PyTorch

body:
I am kind of new to pytorch, and have a very simple question. Let's say we have a scalar function f():
def f(x,y):
    return np.cos(x)+y

What I want to do is use the GPU to generate all pairs of data-points from two ranges x and y. For simple case, take x=y=[0,1,2].
Can I do that without changing the function? If not, how would you change the function?


title: PyTorch classifier for simple images of letters: CNN model design questions

body:
Looking for tips on building a simple image classifier for CAPTCHA images of text where there are only two possible fonts per letter. Here's an example image:

Approach thus far has been to try to break up the image into 6 equal-size images to try to get individual character images, and build a classifier for these (example below).

Is there a simpler way to go about this? Any tips on how to design the actual model? (A relatively simple CNN should suffice here perhaps?)
Edit: questions on building a suitable model below.
I've tried to build a cursory model on top of resnet50 to subpar effect... this seems like the kind of image classification task that should be relatively trivial.
Any tips on model design greatly appreciated.
Code below:
model = models.resnet50(pretrained=True)

for param in model.parameters():
    param.requires_grad = False

model.fc = nn.Sequential(nn.Linear(2048, 512),
                                 nn.ReLU(),
                                 nn.Dropout(0.2),
                                 nn.Linear(512, 26),
                                 nn.LogSoftmax(dim=1))
criterion = nn.NLLLoss()
optimizer = optim.Adam(model.fc.parameters(), lr=0.003)
model.to(device)

epochs = 10
steps = 0
running_loss = 0
print_every = 10
train_losses, test_losses = [], []
for epoch in range(epochs):
    for inputs, labels in train_loader:
        steps += 1
        inputs, labels = inputs.to(device), labels.to(device)
        optimizer.zero_grad()
        logps = model.forward(inputs)
        loss = criterion(logps, labels)
        loss.backward()
        optimizer.step()
        running_loss += loss.item()

        if steps % print_every == 0:
            test_loss = 0
            accuracy = 0
            model.eval()
            with torch.no_grad():
                for inputs, labels in val_loader:
                    inputs, labels = inputs.to(device), labels.to(device)
                    logps = model.forward(inputs)
                    batch_loss = criterion(logps, labels)
                    test_loss += batch_loss.item()

                    ps = torch.exp(logps)
                    top_p, top_class = ps.topk(1, dim=1)
                    equals = top_class == labels.view(*top_class.shape)
                    accuracy += torch.mean(equals.type(torch.FloatTensor)).item()
            train_losses.append(running_loss/len(train_loader))
            test_losses.append(test_loss/len(val_loader))
            print(f"Epoch {epoch+1}/{epochs}.. "
                  f"Train loss: {running_loss/print_every:.3f}.. "
                  f"Test loss: {test_loss/len(val_loader):.3f}.. "
                  f"Test accuracy: {accuracy/len(val_loader):.3f}")
            running_loss = 0
            model.train()

Results from the above look as follows, hitting <50% accuracy:


title: How can I set a minimum learning rate in lr_scheduler LambdaLR?

body:
I'm using LambdaLR as a learning rate function:
import torch
import torch.nn as nn
import matplotlib.pyplot as plt

model = torch.nn.Linear(2, 1)
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
lambda1 = lambda epoch: 0.99 ** epoch
scheduler = torch.optim.lr_scheduler.LambdaLR(optimizer, lr_lambda=lambda1, last_epoch = -1)

lrs = []

for i in range(2001):
    optimizer.step()
    lrs.append(optimizer.param_groups[0]["lr"])
    scheduler.step()

plt.plot(lrs)

I'm trying to set a min learning rate so it won't go to 0. How can I do that?