title: PyTorch LeNet-5 do not training

body:
I am trying to train an adaptation of the LeNet-5 network for 1-channel images of size 30x30. Each sample matrix consists of the z-score values of some parameter for each source RGB image (size 30x30x3). Thus, along with the positive values, there are negative values.
According to this article, I use padding for the input layer, the same layer order, and Z-score calculation for all samples, with the mean and standard deviation calculated for all samples (training and test), but the network is not training.
Here is the network code:
#LeNet-5 class
class Model(nn.Module):

    def __init__(self):
        super(Model, self).__init__()
        self.lay1_2=nn.Sequential( nn.Conv2d(1, 6, kernel_size=5, stride=1, padding=3),
            nn.ReLU(), nn.AvgPool2d(kernel_size=2, stride=2))
        self.lay3_4=nn.Sequential( nn.Conv2d(6, 16, kernel_size=5, stride=1, padding=0),
            nn.ReLU(), nn.AvgPool2d(kernel_size=2, stride=2))
        '2D to 1D'
        self.trans=nn.Sequential(nn.Flatten())
        self.lay5_6 = nn.Linear(576, 100)
        self.lay7_8 = nn.Linear(100, 2)

        self.opt=opt.RMSprop(self.parameters(),lr=0.001,momentum=0.9) #optimizer
        self.criterion=nn.CrossEntropyLoss() #loss function - negative logarithm of likelihood

    def forward(self, x):
        x = self.lay1_2(x)
        x = self.lay3_4(x)
        x = self.trans(x)
        x = self.lay5_6(x)
        return fnc.softmax(self.lay7_8(x))

Training parameters:
number of epochs-500, package size-50, number of classes-2.
Training main loop:
'''
for h in range(start_ep,len(epochs)):
for i in range(epochs[h]-epochs[h-1]):
        x1, y1 = prep_batch(train_mat,train_ans,start_posit%train_size,batch_size)
        model.opt.zero_grad()
        react=model(x1) #reaction calculation
        y1=y1.squeeze_()
        loss=model.criterion(react,y1)
        loss_copy=loss.clone()
        loss.backward()
        model.opt.step()
        start_posit=start_posit+batch_size

    print('Epoch: '+str(epochs[h])+" ",end="")
    msg='Loss_train: '+str(loss_copy.detach().numpy())+" "
    print(msg,end="")

    #error calculation
    t_react=model(t_mat)
    t_loss=model.criterion(t_react,t_ans)
    msg='Loss_test: '+str(t_loss.detach().numpy())
    print(msg)

    #F-measure calculation
    t_react_n=np.argmax(t_react.detach().numpy(),axis=1)
    t_ans_n=t_ans.detach().numpy()
    F_m_res=F_mea(t_ans_n,t_react_n)

    #save results and model
    all_res['ep'].append(epochs[h])
    all_res['res'].append(F_m_res)

    checkpoint={
        'state_dict': model.state_dict(),
        'optimizer': model.opt.state_dict()
    }
    torch.save(checkpoint,os.path.join(path_nn,'model-ep'+str(epochs[h])+'.pt'))

'''
The learning process proceeds as follows:

What could be wrong?


title: Why there are some 1s, 0s, and some NAN in the metrics of evaluating the quantized model?

body:
I am doing CNN quantization, I use the following code to calculate some metrics.
        FP = confusion_matrix.sum(axis=0) - np.diag(confusion_matrix)
        FN = confusion_matrix.sum(axis=1)  - np.diag(confusion_matrix)
        TP = np.diag(confusion_matrix)
        TN = confusion_matrix.sum() - (FP + FN + TP)

        FP = torch.Tensor(FP)
        FN = torch.Tensor(FN)
        TP = torch.Tensor(TP)
        TN = torch.Tensor(TN)

        # Sensitivity, hit rate, recall, or true positive rate
        TPR = TP/(TP+FN)
        # Specificity or true negative rate
        TNR = TN/(TN+FP)
        # Precision or positive predictive value
        PPV = TP/(TP+FP)
        # # Negative predictive value
        # NPV = TN/(TN+FN)
        # # Fall out or false positive rate
        # FPR = FP/(FP+TN)
        # # False negative rate
        # FNR = FN/(TP+FN)
        # # False discovery rate
        # FDR = FP/(TP+FP)
        F1 = 2 * (TPR * PPV) / (TPR+PPV)

        # Overall accuracy
        ACC = (TP+TN)/(TP+FP+FN+TN)
        # print('Sensitivity for each class',confusion_matrix.diag()/confusion_matrix.sum(1))
        print('Acc',ACC)
        print('Sensitivity',TPR)
        print('Specificity',TNR)
        print('Precision',PPV)
        print('F1 score',F1)

But I got the results like this:
Test_Quantization Epoch #0: 100% 129/129 [00:02<00:00, 57.30it/s, Acc=0.0785, F1 score=0.1589, Loss=3.7637, Sensitivity(recall)=1.0000, Specificity=0.0000, precision=0.0870]
/content/drive/My Drive/ECG_quantization/trainer.py:219: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at  /pytorch/torch/csrc/utils/tensor_numpy.cpp:141.)
  TP = torch.Tensor(TP)
Acc tensor([0.1743, 0.9743, 0.9350, 0.9924, 0.0760])
Sensitivity tensor([0., 0., 0., 0., 1.])
Specificity tensor([1., 1., 1., 1., 0.])
Precision tensor([   nan,    nan,    nan,    nan, 0.0760])
F1 score tensor([   nan,    nan,    nan,    nan, 0.1412])
Normalized confusion matrix

I don't know why all metrics except accuracy are so strange. Anybody knows what's the matter?


title: What is a good use of the intermediate hidden states of an RNN?

body:
So I've used RNN/LSTMs in three different capacities:

Many to many: Use every output of the final layer to predict the next. Could be classification or regression.
Many to one: Use the final hidden state to perform regression or classification.
One to many: Take a latent space vector, perhaps the final hidden state of an LSTM encoder and use it to generate a sequence (I've done this in the form of an autoencoder).

In none of these cases do I use the intermediate hidden states to generate my final output. Only the last layer outputs in case #1 and only the last layer hidden state in case #2 and #3. However, PyTorch nn.LSTM/RNN returns a vector containing the final hidden state of every layer, so I assume they have some uses.
I'm wondering what some use cases of those intermediate layer states are?


title: getting an RunTime Error (mat1 and mat2....) when trying to use a trained FNN model in pytorch

body:
In my model there are 5 float inputs and one output (LWS= 1\0).
while training the model there was no problem, but when I tried to load the model and check it on new data I got the "mat1 and mat2..." error.
i used this website to build my model(https://curiousily.com/posts/build-your-first-neural-network-with-pytorch/).
the dataset (LWS is the output)
import torch
import torch.nn as nn
from torch.autograd import Variable
import pandas as pd
import collections #to test itterabillity
import numpy as np
import os
from tqdm import tqdm
import seaborn as sns
from pylab import rcParams
import matplotlib.pyplot as plt
from matplotlib import rc
from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix, classification_report
from torch import nn, optim
import torch.nn.functional as F

df = pd.read_csv(r'C:\Users\ofer\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Anaconda3 (64-bit)\DEW_Data_NNN.csv')
#train = pd.read_excel(r'C:\Users\ofer\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Anaconda3 (64-bit)\DEW_Data_NN_train.xlsx')
#test = pd.read_csv(r'C:\Users\ofer\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Anaconda3 (64-bit)\DEW_Data_NN_test.csv')

#df.drop(labels='Time',axis=1,inplace=True)
df.drop(labels='Rad',axis=1,inplace=True)
df.drop(labels='Baro',axis=1,inplace=True)
input_col_names = df.columns

RANDOM_SEED = 42
np.random.seed(RANDOM_SEED)
torch.manual_seed(RANDOM_SEED)

#dividing the df to test and train df's
# X = inputs; Y = output
X = df[input_col_names[:(len(a)-1)]]
y = df[['LWS']]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2,random_state=RANDOM_SEED)

#converting df's to tensors
X_train = torch.from_numpy(X_train.to_numpy()).float()
y_train = torch.squeeze (torch.from_numpy(y_train.to_numpy()).float())
X_test = torch.from_numpy(X_test.to_numpy()).float()
y_test = torch.squeeze (torch.from_numpy(y_test.to_numpy()).float())

#BUilding a neural network

class Net(nn.Module):
  def __init__(self, n_features):
    super(Net, self).__init__()
    self.fc1 = nn.Linear(n_features, 5)
    self.fc2 = nn.Linear(5, 3)
    self.fc3 = nn.Linear(3, 1)
  def forward(self, x):
    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(x))
    return torch.sigmoid(self.fc3(x))

net = Net(X_train.shape[1])

#training - a loss function, closer to 0 the better
criterion = nn.BCELoss()
#optimization - to reduce the loss function
#lr=learning rate
optimizer = optim.Adam(net.parameters(), lr=0.001)

#converting every value below 0.5 to 0. Otherwise, we set it to 1. Finally, we calculate the percentage of correct values
def calculate_accuracy(y_true, y_pred):
  predicted = y_pred.ge(.5).view(-1)
  return (y_true == predicted).sum().float() / len(y_true)

def round_tensor(t, decimal_places=3):
  return round(t.item(), decimal_places)
for epoch in range(2000):
    y_pred = net(X_train)
    y_pred = torch.squeeze(y_pred)
    train_loss = criterion(y_pred, y_train)
    if epoch % 100 == 0:
      train_acc = calculate_accuracy(y_train, y_pred)
      y_test_pred = net(X_test)
      y_test_pred = torch.squeeze(y_test_pred)
      test_loss = criterion(y_test_pred, y_test)
      test_acc = calculate_accuracy(y_test, y_test_pred)
      print(
f'''epoch {epoch}
Train set - loss: {round_tensor(train_loss)}, accuracy: {round_tensor(train_acc)}
Test  set - loss: {round_tensor(test_loss)}, accuracy: {round_tensor(test_acc)}
''')
    optimizer.zero_grad()
    train_loss.backward()
    optimizer.step()

model_path = 'model.pth'
torch.save(net, model_path)

classes = ['No dew', 'dew']
y_pred = net(X_test)
y_pred = y_pred.ge(.5).view(-1).cpu()
y_test = y_test.cpu()
print(classification_report(y_test, y_pred, target_names=classes))

cm = confusion_matrix(y_test, y_pred)
df_cm = pd.DataFrame(cm, index=classes, columns=classes)
hmap = sns.heatmap(df_cm, annot=True, fmt="d")
hmap.yaxis.set_ticklabels(hmap.yaxis.get_ticklabels(), rotation=0, ha='right')
hmap.xaxis.set_ticklabels(hmap.xaxis.get_ticklabels(), rotation=30, ha='right')
plt.ylabel('True label')
plt.xlabel('Predicted label');

def will_it_dew(Time, Wind_Speed, Air_Temp, RH):
  t = torch.as_tensor([Time, Wind_Speed, Air_Temp, RH]) \
      .float() \
      .to(device)
  output = net(t)
  return output.ge(0.5).item()

#will_it_dew(1,2,3,4)


title: Why flask does not return result?

body:
I am new to deep learning and flask deployment and I am trying to deploy a pytorch model(resnet34) with flask. When I try to upload an image to check the predictions, it does not show any predictions. It does not show any errors in my code.
The HTML page where I upload images.
<form method ='post' enctype=multipart/form-data>
        <input type="file" name="file" >
        <input type="submit" value="upload">

The html page where I want to see the predictions.

Prediction

Currency Name: {{ currency }}

App.py for flask
from flask import Flask, request, render_template, jsonify
from commons import get_tensor
from inference import get_currency_name
app = Flask(__name__)

@app.route('/', methods=['GET', 'POST'])
def hello_world():
    if request.method == 'GET':
        return render_template('index.html', value='hi')
    if request.method == 'POST':
        try:
            file = request.files['file']
            image = file.read()
            print(request.files)
            if 'file' not in request.files:
                print('file not uploaded')
                return
            file = request.files['file']
            image = file.read()
            currency_name = get_currency_name(image_bytes=image)
            category, currency_name = get_currency_name(image_bytes=image)
            get_currency_name(image_bytes=image)
            tensor = get_tensor(image_bytes=image)
            print(get_tensor(image_bytes=image))
            print(currency_name)
            return render_template('result.html', currency=currency_name, category=category)
            #return jsonify({'category': category, 'currency_name': currency_name})

        except:

            return render_template('result.html')

if __name__ == '__main__':
    app.run(debug=True)

commons.py code
import io

import torch
import torch.nn as nn
from torchvision import models, transforms
from PIL import Image

def get_model():
    checkpoint_path = 'checkpoint.pth'
    model = models.resnet34(pretrained=True)
    model.classifier = nn.Sequential(nn.Linear(512, 6))
    model.load_state_dict(torch.load(
        checkpoint_path, map_location='cpu'), strict=False)
    model.eval()
    print("model loaded!!")
    return model

def get_tensor(image_bytes):
    my_transforms = transforms.Compose([transforms.Resize(224),
                                        transforms.ToTensor(),
                                        transforms.Normalize(mean=[0.5, 0.5, 0.5],
                                                                std=[0.5, 0.5, 0.5])
                                        ])
    image = Image.open(io.BytesIO(image_bytes))
    return my_transforms(image).unsqueeze(0)

code for inference.py
import torch
from commons import get_model, get_tensor

class_names = ['Rs. 20', 'Rs. 100', 'Rs. 500', 'Rs. 10', 'fake 100', 'Fake 500']
model = get_model()
def get_currency_name(image_bytes):
    tensor = get_tensor(image_bytes)
    outputs = model(tensor)
    _, preds = outputs.max(2)
    category = preds.item()
    currency_name = class_names[category]

    return currency_name

When I select an image and submit, it does not return any result to result.html and does not show any errors either.
What could have possibly gone wrong??


title: Pytorch LBFGS with implicit gradient calculation

body:
I am using pytorch to optimize a certain function f(x). SO, I created a function f, That gives me the gradient as following:
f  = lambda x: gradient(x)
"gradient (x) is a function to calculate the gradient"

Then, I try to use LBFGS to optimize this function as follwoing:
opt = optim.LBFGS([X],lr=.1)
for _ in range(1000):
    def closure():
        opt.zero_grad()
        X.grad = -f(X)
        return X
    opt.step(closure)

But, I got this error:
RuntimeError: assigned grad has data of a different size

How can I solve this issue, with the same setup.


title: Indexing using pyTorch tensors along one specific dimension with 3 dimensional tensor

body:
I have 2 tensors:
A with shape (batch, sequence, vocab)
and B with shape (batch, sequence).
A = torch.tensor([[[ 1.,  2.,  3.],
     [ 5.,  6.,  7.]],

    [[ 9., 10., 11.],
     [13., 14., 15.]]])

B = torch.tensor([[0, 2],
    [1, 0]])

I want to get the following:
C = torch.zeros_like(B)
for i in range(B.shape[0]):
   for j in range(B.shape[1]):
      C[i,j] = A[i,j,B[i,j]]

But in a vectorized way. I tried torch.gather and other stuff but I cannot make it work.
Can anyone please help me?


title: coo_matrix TypeError: data type not understood

body:
Steps
import torch
import numpy as np
from scipy.sparse import coo_matrix

row = torch.ones([10000], dtype=torch.float32)
col = torch.ones([10000], dtype=torch.float32)
data = torch.ones([10000], dtype=torch.float32)
fn = 5120
bm = 10000

coo_matrix((data, (row, col)), shape=(fn, bm))

Error Produced
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib64/python3.8/site-packages/scipy/sparse/coo.py", line 161, in init
self.data = np.array(obj, copy=copy, dtype=data_dtype)
TypeError: data type not understood
Versions
Python 3.8.8
numpy  1.17.5
torch  1.7.1


title: Choosing bach size and learning rate for large datastet

body:
The dataset has around 2.5 million rows, and I'm using a 80/20 train test split.
I read some answers here and some papers regarding this which suggest batch sizes of 32 or 64. But wouldn't that be extremely small relative to the size of the dataset?
I previously trained with a bath size of10000 and LR of 1e-2 but did not get a good accuracy.
What values would be optimal for a dataset this large?


title: Pixel operations in batches

body:
I have a batch of depth images, shape -> [B, 1, H, W]. For each pixel in each image of the batch I need to perform:
X = d * Kinverse @ [u, v, 1]   #therefore X is in R^3
where d is float tensor[0;1] representing depth at pixel u,v; Kinverse is a constant 3X3 matrix and u, v refer to the pixel column and row respectively.
Is there some way I can vectorize the operation to obtain X(u+1,v), X(u,v) and X(u,v+1) for all the images in the batch.
I eventually need to take this cross product:
{X(u+1,v) - X(u,v)} x {X(u, v+1) - X(u,v)}
Thanks for the help!


title: Why I reload model and continue training the loss raise rapidly?

body:
here's a similar problem, but I think torch is good enough to deal with initialization problem.

Let's see loss first
# first time save and the loss
epoch:8  loss:2.6038

# second time reload, and save model when epoch is 10
epoch:8  loss:7.8922
epoch:10 loss:4.8546

# third time reload, and save model when epoch is 27
epoch:10 loss:6.0920
epoch:27 loss:2.1613

# fourth time reload and training
epoch:27 loss:6.0920

as you can see, once I reload the model and run a epoch training, the loss raising instead of descending. It's really strange.
I use step method that because of my machine cannot support big batch.

The first time I use four GPU with 16 batch size and backward the loss every 2 step..
The second time I use two GPU with 8 batch size and backward the loss every 4 step
The third time I use four GPU with 16 batch size and backward the loss every 2 step..
The fourth time I use four GPU with 16 batch size and backward the loss every 2 step..

According to the first to the third time, I think it's batchnorm problem cause the batch size is different that the param alpha and beta is not suitable for another set.
But surprisingly, the fourth time still get a similar situation.So it's really confuse me.Did I miss some param should save? Here's my load and reload code.
# saving script
    def save_model(self, path, epoch, model, optimizer):
        #if epoch % 4 != 0:
        #    return
        if isinstance(model, torch.nn.DataParallel):
            state_dict = model.module.state_dict()
        else:
            state_dict = model.state_dict()
        torch.save({
            'epoch': epoch,
            'model_state_dict': state_dict,
            'optimizer_state_dict': optimizer.state_dict(),
            'learning_rate':optimizer.state_dict()['param_groups'][0]['lr']
            # 'loss': loss
        }, path)

# reloading script
    def load_model(self, model, optimizer, resume, strict=True):
        checkpoint = torch.load(resume, map_location=lambda storage, loc: storage)
        print('loaded weights from {}, epoch {}'.format(resume, checkpoint['epoch']))
        state_dict_ = checkpoint['model_state_dict']
        state_dict = {}
        for k in state_dict_:
            if k.startswith('module') and not k.startswith('module_list'):
                state_dict[k[7:]] = state_dict_[k]
            else:
                state_dict[k] = state_dict_[k]
        model_state_dict = model.state_dict()
        if not strict:
            for k in state_dict:
                if k in model_state_dict:
                    if state_dict[k].shape != model_state_dict[k].shape:
                        print('Skip loading parameter {}, required shape{}, ' \
                              'loaded shape{}.'.format(k, model_state_dict[k].shape, state_dict[k].shape))
                        state_dict[k] = model_state_dict[k]
                else:
                    print('Drop parameter {}.'.format(k))
            for k in model_state_dict:
                if not (k in state_dict):
                    print('No param {}.'.format(k))
                    state_dict[k] = model_state_dict[k]
        model.load_state_dict(state_dict, strict=False)
        optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
        for state in optimizer.state.values():
            for k, v in state.items():
                if isinstance(v, torch.Tensor):
                    state[k] = v.cuda()
        epoch = checkpoint['epoch']
        # loss = checkpoint['loss']
        return model, optimizer, epoch


title: Average Pooling layer in Deep Learning and gradient artifacts

body:
I know that in Convolution layers the kernel size needs to be a multiplication of stride or else it will produce artefacts in gradient calculations like the checkerboard problem.
Now does it also work like that in Pooling layers? I read somewhere that max pooling can also cause problems like that. Take this line in the discriminator for example:
  self.downsample = nn.AvgPool2d(3, stride=2, padding=1, count_include_pad=False)

I have a model (MUNIT) with it, and this is the image it produced:

It looks like the checkerboard problem, or at least a gradient problem but I checked my Convolution layers and didn't found the error described above. They all are of size 4 with stride 2 or an uneven size with stride of 1.


title: How can I set the diagonal of an N-dim tensor to 0 along given dims?

body:
I’m trying to figure out a way to set the diagonal of a 3-dimensional Tensor (along 2 given dims) equal to 0. An example of this would be, let’s say I have a Tensor of shape [N,N,N] and I wanted to set the diagonal along dim=1,2 equal to 0? How exactly could that be done?
I tried using fill_diagonal_ but that only does the k-th diagonal element for each sub-array, i.e:
>>> data = torch.ones(3,4,4)
>>> data.fill_diagonal_(0)

tensor([[[0, 1, 1, 1],
         [1, 1, 1, 1],
         [1, 1, 1, 1],
         [1, 1, 1, 1]],

        [[1, 1, 1, 1],
         [1, 0, 1, 1],
         [1, 1, 1, 1],
         [1, 1, 1, 1]],

        [[1, 1, 1, 1],
         [1, 1, 1, 1],
         [1, 1, 0, 1],
         [1, 1, 1, 1]]])

whereas I would want the entire diagonal for each sub-matrix to be equal to 0 here. So, the desired outcome would be,
tensor([[[0, 1, 1, 1],
         [1, 0, 1, 1],
         [1, 1, 0, 1],
         [1, 1, 1, 0]],

        [[0, 1, 1, 1],
         [1, 0, 1, 1],
         [1, 1, 0, 1],
         [1, 1, 1, 0]],

        [[0, 1, 1, 1],
         [1, 0, 1, 1],
         [1, 1, 0, 1],
         [1, 1, 1, 0]]])

Secondly, the reason I state for a given pair of dimension is, I need to repeat this `zeroing’ along 2 different pairs of dimensions (e.g. dim=(1,2) then dim=(0,1)) to get the required masking I need.
Is there a way to mask a given diagonal over 2 arbitrary dimensions for a 3D-tensor?


title: Getting an error in Activation function "TypeError: 'Tensor' object is not callable"

body:
I am trying to use the relu activation function in pytorch LSTM but getting the error on " Tensor object is not callable. any guideline or help? Can i use the different activation function in forward propogation? but i am using the same activation function in hidden layers and forward propogation. your kind review will be more help full.
class LSTM(nn.Module):
  def __init__(self, input_size=1, hidden_layer_size=20, output_size=1):
    super().__init__()

    self.hidden_layer_size = hidden_layer_size
    self.lstm = nn.LSTM(input_size, hidden_layer_size)
    self.relu = nn.functional.relu(torch.FloatTensor(hidden_layer_size), torch.FloatTensor(output_size))
    self.hidden_cell = (torch.zeros(1, 1, self.hidden_layer_size),
                        torch.zeros(1, 1, self.hidden_layer_size))
     def forward(self, input_seq):
    lstm_out, self.hidden_cell = self.lstm(input_seq.view(len(input_seq), 1, -1), self.hidden_cell)
    predictions = self.relu(lstm_out.view(len(input_seq), -1))
    return predictions[-1]

    model = LSTM()
    loss_function = nn.MSELoss()
    optimizer = torch.optim.Adam(model.parameters(), lr = 0.0001)

    epochs = 150

     for i in range(epochs):
       for seq, labels in train_inout_seq:
        optimizer.zero_grad()
        model.hidden_cell = (torch.zeros(1, 1, model.hidden_layer_size),
                            torch.zeros(1, 1, model.hidden_layer_size))
       y_pred = model(seq)

    single_loss = loss_function(y_pred, labels)
    single_loss.backward()
    optimizer.step()

     if i%25 == 1:
    print(f'epoch: {i:3} loss: {single_loss.item():10.8f}')

    print(f'epoch: {i:3} loss: {single_loss.item():10.10f}')

After that i am getting an error:`
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-108-5fcfb471ed9a> in <module>
      6     model.hidden_cell = (torch.zeros(1, 1, model.hidden_layer_size),
      7                          torch.zeros(1, 1, model.hidden_layer_size))
----> 8     y_pred = model(seq)
      9
     10     single_loss = loss_function(y_pred, labels)

/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--> 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

<ipython-input-105-221892d3f487> in forward(self, input_seq)
     12   def forward(self, input_seq):
     13     lstm_out, self.hidden_cell = self.lstm(input_seq.view(len(input_seq), 1, -1), self.hidden_cell)
---> 14     predictions = self.relu(lstm_out.view(len(input_seq), -1))
     15     return predictions[-1]

TypeError: 'Tensor' object is not callable


title: Displaying images loaded with pytorch dataloader

body:
I am working with some lidar data images that I cannot post here due to a reputation restriction on posting images. However, when loading the same images using pytorch ImageFolder and Dataloader with the only transform being converting the images to tensors there seems to be some extreme thresholding and I can't seem to locate the cause of this.
Below is how I'm displaying the first image:
dataset = gdal.Open(dir)

print(dataset.RasterCount)
img = dataset.GetRasterBand(1).ReadAsArray()

f = plt.figure()
plt.imshow(img)
print(img.shape)
plt.show()

and here is how I am using the data loader and displaying the thresholded image:
data_transforms = {
        'train': transforms.Compose([
            transforms.ToTensor(),
        ]),
        'val': transforms.Compose([
            transforms.ToTensor(),
        ]),
    }

image_datasets = {x: datasets.ImageFolder(os.path.join(data_dir, x),
                                              data_transforms[x]) for x in ['train', 'val']}
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

dataloders = {x: torch.utils.data.DataLoader(image_datasets[x],
                                                 batch_size=1,
                                                 shuffle=True,
                                                 num_workers=2) for x in ['train', 'val']}
dataset_sizes = {x: len(image_datasets[x]) for x in ['train', 'val']}

for image in dataloders["train"]:
  f = plt.figure()
  print(image[0].shape)
  plt.imshow(image[0].squeeze()[0,:,:])
  plt.show()
  break

Any help on an alternative way to display the images or any mistakes I am making would be greatly appreciated.


title: When using torch.backward() for a GANs generator, why doesn't discriminator losses change in Pytorch?

body:
My understanding of GANs is:

When training your generator, you need to back-propagate through the discriminator first so you can follow the chain rule. As a result, we can't use a .detach()  when working on our generators loss calculation.

When updating discriminator, since your generator weight update doesn't affect discriminator weight updates, we can .detach() the generator output from your computation, my understanding tells me that since this generator is no longer a part of computation graph, we don't update it during back prop any longer.

So when we update your discriminator losses with:
disc_loss.backward(retain_graph=True) at each mini-batch, we don't have to worry about your generator being part of the pipeline thanks to the .detach() function call.
But what about when we work on our generator? What is keeping our model from constantly changing our discriminator weights in accordance to generator? That isn't what we want after all right? The discriminator shouldn't learn when it is told that fake samples are real.
Why does such a model work in the first place


title: PyTorch: concat and flatten inputs of different shape

body:
I have multiple inputs of different shape: (7,), (), (6,), how can I concatenate and flatten them to a single flatten input. My desired output shape is (14,).
For example: arr1= [1, 2, 3], arr2=6, arr3=[6,7], output=[1,2,3,6,6,7]. Although I can use multiple numpy.append, but that would be ugly.


title: Neural Network on GPU using system RAM instead of GPU memory

body:
I built a basic chatbot using PyTorch, and in the training code, I moved both the neural network as well as the training data to the gpu. However, when I run the program, it uses up to 2GB of my ram. There is a little gpu memory that is used, but not that much. When i run the same program, but this time on the cpu, it takes only about 900mb of my ram.
Can anyone tell me why is is happening? i have attached my code, as well as a couple of screenshots.
Sorry if the answer is pretty obvious, I'm new to deep learning.
My code:
import torch
import torch.nn as nn
from torch.utils.data import Dataset, DataLoader
import numpy as np
from nltk_utils import tokenizer, stem, bag_of_words
import pandas as pd

device = torch.device("cuda")

#Neural Network
class NeuralNetwork(nn.Module):
    def __init__(self, input_size, hidden_size, num_classes):
        super().__init__()
        self.l1 = nn.Linear(input_size, hidden_size)
        self.l2 = nn.Linear(hidden_size, hidden_size)
        self.l3 = nn.Linear(hidden_size, hidden_size)
        self.l4 = nn.Linear(hidden_size, num_classes)
        self.relu = nn.ReLU()

    def forward(self, x):
        out = self.relu(self.l1(x))
        out = self.relu(self.l2(out))
        out = self.relu(self.l3(out))
        out = self.l4(out)
        return out

#data initialization
data = pd.read_csv("path_to_train_data")
data = data.dropna(axis=0)

allwords = []
taglist = []
xy = []
ignorewords = ["?", "!", "'", ","]

taglist.extend(x for x in data["tag"] if x not in taglist)

#developing vocabulary
for x in data["pattern"]:
    w = tokenizer(x)
    allwords.extend(stem(y) for y in w if y not in ignorewords)

#making training data
for indx, x in enumerate(data["pattern"]):
    w = tokenizer(x)
    bag = bag_of_words(w, allwords)
    tag = taglist.index(data["tag"].iloc[indx])
    xy.append((bag, tag))

xtrain = np.array([x[0] for x in xy])
ytrain = np.array([x[1] for x in xy])

class TestDataset(Dataset):
    def __init__(self):
        self.num_classes = len(xtrain)
        self.xdata = torch.from_numpy(xtrain.astype(np.float32))
        self.ydata = torch.from_numpy(ytrain.astype(np.float32))

    def __getitem__(self, index):
        return self.xdata[index], self.ydata[index]

    def __len__(self):
        return self.num_classes

dataset = TestDataset()
train_data = DataLoader(dataset=dataset,
                        batch_size=8,
                        shuffle=True,
                        num_workers=0,
                        pin_memory=True)

inputSize = len(xtrain[0])
hiddenSize = 32
outputSize = len(taglist)

model = NeuralNetwork(inputSize, hiddenSize, outputSize).to(device)

criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters())

epochs = 5000

for epoch in range(epochs):
    for (words, labels) in train_data:
        words = words.to(device)
        labels = labels.type(torch.LongTensor)
        labels = labels.to(device)

        y_pred = model(words)

        loss = criterion(y_pred, labels)

        optimizer.zero_grad(set_to_none=True)
        loss.backward()
        optimizer.step()

    with torch.no_grad():
        if (epoch + 1) % 10 == 0:
            print(f"epoch: {epoch + 1}, loss: {loss:.30f}")

While running on GPU:
GPU utilization,
RAM utilization
While running on CPU:
RAM utilization


title: Subclass of PyTorch dataset class cannot find dataset files

body:
I'm trying to create a subclass of the PyTorch MNIST dataset class, which I call  CustomMNISTDataset, as follows:
import torchvision.datasets as datasets

class CustomMNISTDataset(datasets.MNIST):

    def __init__(self, root='/home/psando'):
        super().__init__(root=root,
                         download=False)

but when I execute:
dataset = CustomMNISTDataset()

it fails with error: "RuntimeError: Dataset not found. You can use download=True to download it".
However, when I run the following in the same file:
dataset = datasets.MNIST(root='/home/psando', download=False)
print(len(dataset))

it succeeds and prints "60000", as expected.
Since CustomMNISTDataset subclasses datasets.MNIST why is the behavior different? I've verified that the path '/home/psando' contains the MNIST directory with raw and processed subdirectories (otherwise, explicitly calling the constructor for datasets.MNIST() would have failed). The current behavior implies that the call to super().__init__() within CustomMNISTDataset is not calling the constructor for datasets.MNIST which is very strange!
Other details: I'm using Python 3.6.8 with torch==1.6.0, torchtools==0.0.2, and
torchvision==0.7.0. Any help would be appreciated!


title: How to load and pre-process data from pandas dataframe for PyTorch

body:
I have a fully pre-processed Pandas dataframe and I want to run NLP on it using pytorch.
Here is the code which uses TabularDataset library to load and process the data from files:
import pandas as pd
import torch
from torchtext.legacy import data
from torchtext.legacy.data import TabularDataset

TEXT = data.Field(sequential=True)
LABEL = data.LabelField(dtype = torch.float, use_vocab = False, preprocessing = int)

datafields = [("source", None),
                 ("label", LABEL),
                 ('notes', None),
                 ("text", TEXT)]

train_data, valid_data, test_data = TabularDataset.splits(path='/Folder/',
                                                          train = 'train.csv',
                                                          validation = 'validation.csv',
                                                          test = 'test.csv',
                                                          format = 'csv', skip_header=True, fields=datafields)

TEXT.build_vocab(train_data, max_size = MAX_VOCAB_SIZE, min_freq = 1)

However, I already have the data as Pandas dataframes, and I don't want to write them to .csv and then load them again using TablularDataset.
What is the equivalent code by which I can get train_data, valid_data, test_data from their dataframes (e.g. train_df, valid_df, test_df)?
The example of data in my Pandas dataframes are as follows:

label
text

1
text important processing data nlp process predict

0
visual model success time simulate large fast

Note that the text data are some tokens " ".joined together.


title: mat1 dim 1 must match mat2 dim 0 -Pytorch

body:
I have attempted to solve this error but it has been to no avail. My CNN model is below:
The shape of X_train and X_test are:
X_train shape: torch.Size([12271, 3, 100, 100]) | X_test shape: torch.Size([3068, 3, 100, 100])
class Net(nn.Module):
    def __init__(self, num_classes=7):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(in_channels=3, out_channels=32, kernel_size= 5,stride=1,padding=2)
        self.conv2 = nn.Conv2d(in_channels=32, out_channels=64, kernel_size=5,stride=1,padding=2)
        self.conv3 = nn.Conv2d(in_channels=64, out_channels=128, kernel_size=5,stride=1,padding=2)

        self.pool = nn.MaxPool2d(2, 2)
        self.drop = nn.Dropout2d(p=0.2)

        self.fc1 = nn.Linear(9216, 1000)
        self.fc2 = nn.Linear(1000, 500)
        self.fc3 = nn.Linear(500, out_features=num_classes)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = self.pool(F.relu(self.conv3(x)))

        x = x.reshape(x.size(0), -1)

        x = F.dropout(self.drop(x), training=self.training)

        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

net = Net()

print(net)

The training on the model on data code as required is show below:
The required traceback is below:
count = 0
total_step = len(trainloader)
loss_list = []
acc_list = []
iteration_list = []
for epoch in range(20):
    for i, data  in enumerate(trainloader,0):
      images, labels = data[0].to(device), data[1].to(device)
        # Run the forward pass
      outputs = net(images.float())
      loss = criterion(outputs, labels)
      loss_list.append(loss.item())
      iteration_list.append(i)
      # Backprop and perform Adam optimisation
      optimizer.zero_grad()
      loss.backward()
      optimizer.step()

      #count += 1

      #if count % 100 == 0:
        # Track the accuracy
      total = labels.size(0)
      _, predicted = torch.max(outputs.data, 1)
      correct = (predicted == labels).sum().item()
      acc_list.append(correct / total)

      if (i + 1) % 1000 == 0:
          print('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}, Accuracy: {:.2f}%'
                .format(epoch + 1, num_epochs, i + 1, total_step, loss.item(),
                        (correct / total) * 100))


title: Pytorch cosine similarity NxN elements

body:
I have 128 vectors of embeddings
image.shape = torch.Size([128, 512])
text.shape = torch.Size([128, 512])

And I want to calculate the tensor containing the cosine similarity between all elements (i.e:
cosine.shape = torch.Size([128, 128])

Where the first row is the cosine similarity between the 1st image and all text (128), etc.
At the moment I'm only doing this, but the result is a one-dimension array containing only N cosine similarities.
cosine_similarity = torch.nn.CosineSimilarity()

cosine = cosine_similarity(image, text)

How can I do it? I tried to transpose text but didn't work


title: How can I solve RuntimeError: cuda runtime error (59) : device-side assert triggered error

body:
There is a part that I cannot understand when I train the deep learning model on colab.
When I first train the model, measuring training loss and valid loss at the same time, it works for every batch and epoch.
However, when I train again in the same cell, or train another model in a different cell, RuntimeError: cuda runtime error (59) : device-side assert triggered error happens suddenly.
I also have tried this one at the very front of Colab window, but it didn't work
import os
os.environ['CUDA_LAUNCH_BLOCKING'] = "1"

if os.environ.get('COLAB_GPU', '0') == '1':
  os.environ['GPU_NUM_DEVICES'] = '1'
  os.environ['XLA_FLAGS'] = '--xla_gpu_cuda_data_dir=/usr/local/cuda/'

I don't think it is because of index error from loss function, because
if it was an index error, the first trial should not work!
Here is be the link where I got information, but not being my case.
Is here anybody who can solve this problem?


title: attn_output_weights in MultiheadAttention

body:
I wanna know if the matrix of the attn_output_weight can demonstrate the relationship between every word-pair in the input sequence.
In my project, I draw the heat map based on this output and it shows like this:

However, I can hardly see any information from this heat map.
I refer to other people's work, their heat map is like this.  At least the diagonal of the matrix should have the deep color.
Then I wonder if my method to draw the heat map is correct or not (i.e. directly using the output of the attn_output_weight ) If this is not the correct way, could you please tell me how to draw the heat map?


title: Returning indices from pytorch Dataset: Function to alter __getitem__ results in metaclass conflict

body:
I have multiple classes (for different datasets) that inherit from pytorch's Dataset class. They have a general structure, like so:
from torch.utils.data import Dataset

class SomeDataset(Dataset):

    def __init__(self, data, labels):
        super(SomeDataset, self).__init__()
        self.data = data
        self.labels = labels
        self.__name__ = 'SomeDataset'

    def __getitem__(self, index):
        return {'data': self.data[index], 'label': self.labels[index]}

    def __len__(self):
        return len(data)

Recently I have realised that it would be beneficial to keep track of the labels passed into the Dataloader when batching, so upon googling how to do this I came across this thread, which is where I have adapted the code to write this function:
def return_indices(dataset_class):

    def __getitem__(self, index):
        return {'index':1, **dataset_class.__getitem__(self, index)}

    return type(dataset_class.__name__, (dataset_class, ), {'__getitem__': __getitem__})

I had never seen type used like this before, but after some googling, it made some sense, so I tried it out. Unfortunately this led to this error:
TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases

which led to a whole lot more googling, and even though I'm beginning to grasp what a metaclass is and how they're used I still can't figure out what is wrong with this approach or how to solve it - and I'm starting to think that maybe it would be easier to rewrite this functionality into my dataset classes instead of having some neat wrapper that does it for me. Can anyone weigh in with whatever it is I'm missing?


title: How I can convert model.pt to model.h5?

body:
I want to insert my pretrained pytorch (model.pt) to keras for transfer learning.
So I want to convert (model.pt) to (model.h5).
can you help me how?


title: PyTorch LSTM for Daily Stock Return Prediction - Train loss is consistently lower than test loss

body:
I was wondering if someone could share some ideas for why my training loss begins at a higher level than the test loss?
I am trying to run an LSTM on daily stock return data as the only input and using the 10 previous days to predict the price on the next day. Training/test/validation sets do not overlap, so there is no leakage. Not using any regularisation that would impact the training data only.
Really confused at the moment as I cannot seem to find the error.
I will include the code below but its quite long
    # Defining the LSTM class

import torch
import torch.nn as nn
from sklearn import preprocessing
from sklearn.preprocessing import StandardScaler, MinMaxScaler

class LSTM(nn.Module):
  def __init__(self, n_inputs, n_hidden, num_layers, n_outputs):
    super(LSTM, self).__init__()
    self.D = n_inputs
    self.M = n_hidden
    self.K = n_outputs
    self.L = num_layers

    self.rnn = nn.LSTM(
        input_size=self.D,
        hidden_size=self.M,
        num_layers=self.L,
        batch_first=True)
    self.fc = nn.Linear(self.M, self.K)

  def forward(self, X):
    # initial hidden states
    h0 = torch.zeros(self.L, X.size(0), self.M).to(device)
    c0 = torch.zeros(self.L, X.size(0), self.M).to(device)

    # get RNN unit output
    out, _ = self.rnn(X, (h0, c0))

    # we only want h(T) at the final time step
    out = self.fc(out[:, -1, :])
    return out

# Defining a function to train the LSTM

def full_gd(model,
            loss_function,
            optimizer,
            X_train,
            y_train,
            X_test,
            y_test,
            no_epochs):

  # Stuff to store
  train_losses = np.zeros(no_epochs)
  test_losses = np.zeros(no_epochs)

  for it in range(no_epochs):
    # zero the parameter gradients
    optimizer.zero_grad()

    # Forward pass
    outputs = model(X_train)
    loss = loss_function(outputs, y_train)

    # Backward and optimize
    loss.backward()
    optimizer.step()

    # Save losses
    train_losses[it] = loss.item()

    # Test loss
    test_outputs = model(X_test)
    test_loss = loss_function(test_outputs, y_test)
    test_losses[it] = test_loss.item()

    if (it + 1) % 10 == 0:
      print(f'Epoch {it+1}/{no_epochs}, Train Loss: {loss.item():.4f}, Test Loss: {test_loss.item():.4f}')

  return train_losses, test_losses

# Import sklearn's StandardScaler to scale the returns data
scaler = StandardScaler()
scaler.fit(data[:3*len(data)//5])
historical_returns = scaler.transform(data)

# Creating the dataset to train the LSTM. D is the number of input features. T is the number of data points used in forecasting

T = 10
D = 1
X = []
Y = []

for t in range(len(historical_returns) - T):
    x = historical_returns[t:t+T]
    X.append(x)
    y = historical_returns[t+T]
    Y.append(y)

X_historical = np.array(X).reshape(-1, T, 1)
Y_historical = np.array(Y).reshape(-1, 1)

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

# Splitting the data into a 60/20/20 train/validation/test split. No random split is used here as this is a time series dataset

x_train1 = torch.from_numpy(X_historical[:3*len(historical_returns)//5].astype(np.float32))
y_train1 = torch.from_numpy(Y_historical[:3*len(historical_returns)//5].astype(np.float32))

x_val1 = torch.from_numpy(X_historical[-2*len(historical_returns)//5: -1*len(historical_returns)//5].astype(np.float32))
y_val1 = torch.from_numpy(Y_historical[-2*len(historical_returns)//5: -1*len(historical_returns)//5].astype(np.float32))

x_test1 = torch.from_numpy(X_historical[-1*len(historical_returns)//5:].astype(np.float32))
y_test1 = torch.from_numpy(Y_historical[-1*len(historical_returns)//5:].astype(np.float32))

# move data to GPU
x_train1, y_train1 = x_train1.to(device), y_train1.to(device)
x_val1, y_val1 = x_val1.to(device), y_val1.to(device)
x_test1, y_test1 = x_test1.to(device), y_test1.to(device)

x_train1 = x_train1.reshape(-1, T, 1)
x_test1 = x_test1.reshape(-1, T, 1)
x_val1 = x_val1.reshape(-1, T, 1)

# Define the model parameters
Hidden = 10
model = LSTM(1, Hidden, 1, 1)
model.to(device)
loss_function = nn.MSELoss()

learning_rate = 0.01
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

# Train the model

no_epochs = 200
train_losses, validation_losses = full_gd(model,
                                    loss_function,
                                    optimizer,
                                    x_train1,
                                    y_train1,
                                    x_val1,
                                    y_val1,
                                    no_epochs)

# Plot training and validation loss

plt.figure(figsize=(12,8))
plt.plot(train_losses, label='train loss')
plt.plot(validation_losses, label='test loss')
plt.legend()
plt.show()


title: Does pytorch Dataset.__getitem__ have to return a dict?

body:
EDIT:  This is not about the general __getitem__ method but the usage of __getitem__ in the Pytorch Dataset-subclass
I'm trying to implement the usage of Pytorchs Dataset-class.
The guide e.g here is really good, but I struggle to figure out Pytorch requirements for the return value of __getitem__. In the Pytorch documentation I cannot find anything about what it should return; is it any object which is iterable with size 2 e.g [sample,target], (sample,target)? In some guides they return a dict, but they do not specify if it has to be a dict which is returned.


title: How can I use map or a loop for decoding a transformer dataset?

body:
I load a dataset containing columns of text, I want to translate them.
To speed the process I tried to use transformer datasets.
model_size = "base"
model_name = f"persiannlp/mt5-{model_size}-parsinlu-translation_en_fa"
tokenizer = MT5Tokenizer.from_pretrained(model_name)
model = MT5ForConditionalGeneration.from_pretrained(model_name)
dataset = load_dataset('csv', data_files=dfname, split='train')

dataset = dataset.map(lambda e: tokenizer(e['input_text'], padding='longest'))

dataset.set_format(type='torch', columns=['input_ids'])

# map for generating translation
#dataset = dataset.map(lambda e: {"trans":model.generate(e['input_ids'])})

dataloader = torch.utils.data.DataLoader(dataset, batch_size=32)
for batch in tqdm(dataloader):
    input_ids = batch["input_ids"]
    res = model.generate(input_ids)
    target = tokenizer.batch_decode(res, skip_special_tokens=True)

First, I tried to call model.generate in another map which give this error (commented in code):
File "/home/pouramini/miniconda3/lib/python3.7/site-packages/transformers/generation_utils.py", line 378, in _prepare_encoder_decoder_kwargs_for_generation
    model_kwargs["encoder_outputs"]: ModelOutput = encoder(input_ids, return_dict=True, **encoder_kwargs)
  File "/home/pouramini/miniconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/pouramini/miniconda3/lib/python3.7/site-packages/transformers/models/t5/modeling_t5.py", line 877, in forward
    batch_size, seq_length = input_shape
ValueError: not enough values to unpack (expected 2, got 1)

Then I tried to call it in a loop, but it gives the following error for the loop:
Traceback (most recent call last):
  File "prepare_natural.py", line 146, in <module>
    for batch in tqdm(dataloader):
  File "/home/pouramini/miniconda3/lib/python3.7/site-packages/tqdm/std.py", line 1129, in __iter__
    for obj in iterable:
  File "/home/pouramini/miniconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 435, in __next__
    data = self._next_data()
  File "/home/pouramini/miniconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 475, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/home/pouramini/miniconda3/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
    return self.collate_fn(data)
  File "/home/pouramini/miniconda3/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py", line 73, in default_collate
    return {key: default_collate([d[key] for d in batch]) for key in elem}
  File "/home/pouramini/miniconda3/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py", line 73, in <dictcomp>
    return {key: default_collate([d[key] for d in batch]) for key in elem}
  File "/home/pouramini/miniconda3/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py", line 55, in default_collate
    return torch.stack(batch, 0, out=out)
RuntimeError: stack expects each tensor to be equal size, but got [17] at entry 0 and [15] at entry 1


title: PyTorch's FFT not preserving Linearity

body:
I'm not sure if this is a bug or I'm misunderstanding something about the indexing/shifting. When I try to calculate the difference in the Fourier domain, something is going wrong in the calculation. For a simple case, consider a identity kernel K, such that for any signal X it follows that X - K*X = 0. Further, x - k o x = 0. However, when I compute the residual in the Fourier domain using pytorch, I don't get a numerically 0 solution.
import torch
from torch.fft import rfftn, irfftn
from PIL import Image
from torchvision.transforms import ToTensor
from torch.nn.functional import pad
from matplotlib import pyplot as plt

# Let X be an input image.
with Image.open("barbara.png") as im:
    X = ToTensor()(im)[0,...]

# Let K be a convolutional identity kernel.
K = torch.zeros((3,3))
K[1,1] = 1.0

# Full convolution shape.
cnvShape = (X.shape[0] + K.shape[0] - 1, X.shape[1] + K.shape[1] - 1)

# Apply DFT. Uncentered zero-padding to full shape.
Xhat = rfftn(X, s=cnvShape, dim=(0,1))
Khat = rfftn(K, s=cnvShape, dim=(0,1))

# Calculate convolution via Fourier domain.
XKhat = Xhat * Khat
XK = irfftn(XKhat, s=cnvShape)[1:-1,1:-1]
# Difference in spatial domain. Is numerically close to 0.
R = X - XK

# Difference in Fourier domain. This should be numerically close to 0.
Shat = Xhat - XKhat
S = irfftn(Shat)

plt.subplot(1,2,1)
plt.imshow(R)
plt.colorbar()
plt.subplot(1,2,2)
plt.imshow(S)
plt.colorbar()
plt.show()

# Center zeropad image FFT, produces numerically 0 solution.
That = rfftn(pad(X, (1,1)*2), s=cnvShape) - XKhat
T = irfftn(That)
plt.imshow(T)
plt.colorbar()
plt.show()

This results in

So from the last plot, I can see there's some sort of shifting going on with operations in the Fourier domain, but it's not intuitive what. I'd also like to figure out how to implement consistent operations in the Fourier domain, as this is just a small sample of what I need to compute. I'm aware of fftshift, but that doesn't seem to be appropriate here, as the shift doesn't seem to be with negative and positive frequencies. So, what is going on in the indexing?


title: how to use collate_fn properly in the code below?

body:
My code is:
model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)
dataset = PennFudanDataset('PennFudanPed', get_transform(train=True))
data_loader = torch.utils.data.DataLoader(
dataset, batch_size=2, shuffle=True, num_workers=4,
collate_fn=utils.collate_fn)
# For Training
images,targets = next(iter(data_loader))
images = list(image for image in images)
targets = [{k: v for k, v in t.items()} for t in targets]
output = model(images,targets)   # Returns losses and detections
# For inference
model.eval()
x = [torch.rand(3, 300, 400), torch.rand(3, 500, 400)]
predictions = model(x)           # Returns predictions

I get the error:

"collate_fn=utils.collate_fn" shows error"name 'utils' is not defined". "module 'torch.utils' has no attribute 'collate_fn' error after adding torch.


title: Can I reduce number of GPUs without terminating the training?

body:
Let's say I am using multiple GPUs (0,1,2,3) on one machine and later someone else also needs to use GPUs on this machine. Is there a way for me to reduce the number of gpu usage (i.e. only use 0 and 1) from my training without terminating the training and start over again? I don't want to waste the training I already did.
This sounds like a common need in a team. Is that possible?


title: Batchnormalization in Keras vs PyTorch vs Numpy are different

body:
I created the BatchNormalization layer in Keras, PyTorch and calculated the same operation using Numpy but I get three different results. Am I making some error here?
Things I assume below: layer.get_weights() in tf.keras for BN layer returns in order gamma, beta, running_mean, running_var. For the BN operation I am using the following operation: gamma * (x - running_mean) / sqrt(running_var + epsilon) + beta
Code snippet to reproduce the issue:
import torch
import tensorflow
from torch.nn import Module, BatchNorm1d, Conv1d
from torch.nn.functional import pad
import numpy as np
from tensorflow.keras.layers import Conv1D, BatchNormalization, Input
from tensorflow.keras.models import Model

torch.backends.cudnn.deterministic = True
np.random.seed(12345)

z = Input((1024, 8), dtype=np.float32)
inp = z
z = Conv1D(64, 16, padding='same', use_bias=False)(z)
z = BatchNormalization(epsilon=0.001)(z)
keras_model = Model(inp, z)
# in order: conv-layer weight, gamma, beta, running_mean, running_var
weights = [np.random.random((16, 8, 64)), np.random.random((64,)), np.random.random((64,)), np.random.random((64,)),
           np.random.random((64,))]
weights = [np.array(x, dtype=np.float32) for x in weights]
keras_model.layers[1].set_weights([weights[0]])
keras_model.layers[2].set_weights(weights[1:])
keras_model_subpart = Model(keras_model.inputs, keras_model.layers[1].output)

class TorchModel(Module):

    def __init__(self):
        super(TorchModel, self).__init__()
        self.l1 = Conv1d(8, 64, 16, bias=False)
        self.l2 = BatchNorm1d(64, 0.001)

    def forward(self, x):
        x = pad(x, (7, 8))
        x = self.l1(x)
        y = x
        x = self.l2(x)
        return y, x

torch_model = TorchModel().to(torch.device('cpu'))
torch_model.l1.weight.data = torch.from_numpy(weights[0].T).float()
torch_model.l2.weight.data = torch.from_numpy(weights[1].T).float()
torch_model.l2.bias.data = torch.from_numpy(weights[2]).float()
torch_model.l2.running_mean = torch.from_numpy(weights[3]).float()
torch_model.l2.running_var = torch.from_numpy(weights[4]).float()
torch_model.eval()

input_value = np.array(np.random.random((1024, 8)), dtype=np.float32)

keras_results = [np.array(keras_model_subpart.predict(input_value[np.newaxis, :, :])),
                 np.array(keras_model.predict(input_value[np.newaxis, :, :]))]
with torch.no_grad():
    torch_results = [x.detach().numpy() for x in torch_model(torch.from_numpy(input_value.T[np.newaxis, :, :]).float())]

keras_results = [np.squeeze(x) for x in keras_results]
torch_results = [np.squeeze(x) for x in torch_results]
numpy_results = weights[1] * (keras_results[0] - weights[3]) / np.sqrt(weights[4] + 0.001) + weights[2]

print(torch.__version__, tensorflow.__version__, np.__version__, sep=",")
print('\nRESULTS:')
print('\tLayer 1 difference:', np.mean(np.abs(keras_results[0] - torch_results[0].T).flatten()))
print('\tLayer 2 difference:', np.mean(np.abs(keras_results[1] - torch_results[1].T).flatten()))
print('\tLayer 2 keras - numpy:', np.mean(np.abs(keras_results[1] - numpy_results).flatten()))
print('\tLayer 2 torch - numpy:', np.mean(np.abs(torch_results[1] - numpy_results.T).flatten()))

The output I get (after all the initialization printing of tensorflow)
1.7.1+cu110,2.4.1,1.19.5

RESULTS:
        Layer 1 difference: 0.0
        Layer 2 difference: 6.8671216e-07
        Layer 2 keras - numpy: 2.291581e-06
        Layer 2 torch - numpy: 1.8929532e-06


title: Is there a way to see what's going wrong with a training session in Pytorch?

body:
I'm training a triplet convolution neural network in Jupyter.
When I execute the cell I just get the * symbol and nothing happens.

I'm not asking for help finding a problem with the code. I would just like to know if there is a troubleshooting possibility that might let me see what is happening.
There is probably something wrong with my data loader, or data format, or model. I will find it myself if Pytorch or somebody has a method to find a clue here. It is not giving me an error. It is just working ad infinitum on something wrong.

I saw a function called 'set_trace()' that could be typed into the block that is supposed to be able to give a clue about the problem. But after putting it in the for loop I get

NameError: name 'set_trace' is not defined

for batch_idx in range(1):
    for batch_idx, (data, target) in enumerate(triplet_train_loader):
        model.train()
        metrics = []
        losses = []
        total_loss = 0
        data = tuple(d.cuda() for d in data)
        optimizer.zero_grad()
        outputs = model(*data)

        loss_outputs = loss_fn(*outputs)
        loss = loss_outputs[0] if type(loss_outputs) in (tuple, list) else loss_outputs
        losses.append(loss.item())
        total_loss += loss.item()
        loss.backward()
        optimizer.step()
        set_trace()
        if batch_idx % log_interval == 0:
            message = 'Train: [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                batch_idx * len(data[0]), len(triplet_train_loader.dataset),
                100. * batch_idx / len(triplet_train_loader), np.mean(losses))
            for metric in metrics:
                message += '\t{}: {}'.format(metric.name(), metric.value())

            print(message)
            losses = []

"Describe expected and actual results"

LOVE FOR IT TO TRAIN. IN ACTUAL RESULT WORLD IT DOES NOT TRAIN.


title: By picture that is displayed in Jupyter notebook isn't shown when running the same code in IDE?

body:
I was trying to run code from an online tutorial on my local machine by copying code from Jupiter notebook to my IDE (pycharm).

This part

import numpy as np
import matplotlib.pyplot as plt
import matplotlib
from PIL import Image
from torchvision import transforms
import matplotlib.pyplot as plt

pig_img = Image.open("pig.jpg")
preprocess = transforms.Compose([
   transforms.Resize(224),
   transforms.ToTensor(),
])
pig_tensor = preprocess(pig_img)[None,:,:,:]

plt.imshow(pig_tensor[0].numpy().transpose(1,2,0))

While Jupiter notebook shows the imagine - I cannot get it displayed when running in terminal or IDE.

Any ideas why?

Thank you


title: Nonexistant pytorch gradients when dotting tensors in loss function

body:
For the purposes of this MWE I'm trying to fit a linear regression using a custom loss function with multiple terms. However, I'm running into strange behavior when trying to weight the different terms in my loss function by dotting a weight vector with my losses. Just summing the losses works as expected; however, when dotting the weights and losses the backpropagation gets broken somehow and the loss function doesn't decrease.

I've tried enabling and disabling requires_grad on both tensors, but have been unable to replicate the expected behavior.

import torch
import torch.nn as nn
import numpy as np
import matplotlib.pyplot as plt

# Hyper-parameters
input_size = 1
output_size = 1
num_epochs = 60
learning_rate = 0.001

# Toy dataset
x_train = np.array([[3.3], [4.4], [5.5], [6.71], [6.93], [4.168],
                    [9.779], [6.182], [7.59], [2.167], [7.042],
                    [10.791], [5.313], [7.997], [3.1]], dtype=np.float32)

y_train = np.array([[1.7], [2.76], [2.09], [3.19], [1.694], [1.573],
                    [3.366], [2.596], [2.53], [1.221], [2.827],
                    [3.465], [1.65], [2.904], [1.3]], dtype=np.float32)

# Linear regression model
model = nn.Linear(input_size, output_size)

# Loss and optimizer
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)

def loss_fn(outputs, targets):
    l1loss = torch.norm(outputs - targets, 1)
    l2loss = torch.norm(outputs - targets, 2)

    # This works as expected
    # loss = 1 * l1loss + 1 * l2loss
    # Loss never changes, no matter what combination of
    # requires_grad I set
    loss = torch.dot(torch.tensor([1.0, 1.0], requires_grad=False),
            torch.tensor([l1loss, l2loss], requires_grad=True))
    return loss

# Train the model
for epoch in range(num_epochs):
    # Convert numpy arrays to torch tensors
    inputs = torch.from_numpy(x_train)
    targets = torch.from_numpy(y_train)

    # Forward pass
    outputs = model(inputs)
    loss = loss_fn(outputs, targets)

    # Backward and optimize
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    if (epoch+1) % 5 == 0:
        print ('Epoch [{}/{}], Loss: {:.4f}'.format(epoch+1, num_epochs, loss.item()))

# Plot the graph
predicted = model(torch.from_numpy(x_train)).detach().numpy()
plt.plot(x_train, y_train, 'ro', label='Original data')
plt.plot(x_train, predicted, label='Fitted line')
plt.legend()
plt.show()

Expected result: loss function decreases and the linear regression is fitted (see output below)

Epoch [5/60], Loss: 7.9943
Epoch [10/60], Loss: 7.7597
Epoch [15/60], Loss: 7.6619
Epoch [20/60], Loss: 7.6102
Epoch [25/60], Loss: 7.4971
Epoch [30/60], Loss: 7.4106
Epoch [35/60], Loss: 7.3942
Epoch [40/60], Loss: 7.2438
Epoch [45/60], Loss: 7.2322
Epoch [50/60], Loss: 7.1012
Epoch [55/60], Loss: 7.0701
Epoch [60/60], Loss: 6.9612

Actual result: no change in loss function

Epoch [5/60], Loss: 73.7473
Epoch [10/60], Loss: 73.7473
Epoch [15/60], Loss: 73.7473
Epoch [20/60], Loss: 73.7473
Epoch [25/60], Loss: 73.7473
Epoch [30/60], Loss: 73.7473
Epoch [35/60], Loss: 73.7473
Epoch [40/60], Loss: 73.7473
Epoch [45/60], Loss: 73.7473
Epoch [50/60], Loss: 73.7473
Epoch [55/60], Loss: 73.7473
Epoch [60/60], Loss: 73.7473

I'm pretty confused as to why such a simple operation is breaking the backpropagation gradients and would really appreciate it if anyone had some insights on why this isn't working.


title: Understanding the use of classes when training a neural network with Pytorch

body:
I'm currently reading about using the cross-entropy loss to train a neural network in the pytorch documentation. The criterion that is to be used to calculate the loss is called as follows

criterion = nn.CrossEntropyLoss()

According to the torch.nn documentation, CrossEntropyLoss is a class. From my understanding this would mean that criterion is an object of type nn.CrossEntropyLoss.

When training the neural network, criterion is used to calculate the loss in the following way

loss = criterion(input, target)

This is somewhat confusing to me. If criterion is an object, then how can it be used as a function? What I would expect is something like

loss = criterion.calculate_loss(input, target)

where calculate_loss() would be a method defined in the nn.CrossEntropyLoss class. Furthermore, the documentation also uses the following line of code

running_loss += loss.item()

Where does this item() method come from? I can't find a source mentioning item() online.


title: Stacked two LSTMs with different hidden layers

body:
I would like to create LSTM layers which contain different hidden layers to predict time series data, for the 1st layer of LSTM_1 contains 10 hidden layers, LSTM_2 contains 1 hidden layer, the proposed neural network architecture is illustrated following

proposed neural network architecture

def __init__(self, nb_features=1, hidden_size_1=100, hidden_size_2=100, nb_layers_1 =10, nb_layers_2 = 1, dropout=0.5): #(self, nb_features=1, hidden_size=100, nb_layers=10, dropout=0.5):
super(Sequence, self).__init__()
self.nb_features = nb_features
self.hidden_size_1 = hidden_size_1
self.hidden_size_2 = hidden_size_2
self.nb_layers_1 =nb_layers_1
self.nb_layers_2 = nb_layers_2
self.lstm_1 = nn.LSTM(self.nb_features, self.hidden_size_1, self.nb_layers_1, dropout=dropout)
self.lstm_2 = nn.LSTM(self.hidden_size_1, self.hidden_size_2, self.nb_layers_2, dropout=dropout)
self.lin = nn.Linear(self.hidden_size_2, 1)

def forward(self, input):
h_t1 = Variable(torch.zeros(self.nb_layers_1, input.size()[1], self.hidden_size_1))
c_t1 = Variable(torch.zeros(self.nb_layers_1, input.size()[1], self.hidden_size_1))
h_t2 = Variable(torch.zeros(self.nb_layers_2, input.size()[1], self.hidden_size_2))
c_t2 = Variable(torch.zeros(self.nb_layers_2, input.size()[1], self.hidden_size_2))
outputs = []

for i, input_t in enumerate(input.chunk(input.size(1))):
    h_t1, c_t1 = self.lstm_1(input_t, (h_t1, c_t1))
    h_t2, _ = self.lstm_2(h_t1, (h_t2, c_t2))
    output = self.lin(h_t2[-1])
    outputs += [output]

outputs = torch.stack(outputs, 1).squeeze(2)

return outputs

However, I got runtime error for the above code “RuntimeError: Expected hidden[0] size (3, 60, 100), got (1, 60, 100)”. is it correct for the above code, Could you please give some suggestions? Many thanks


title: In pytorch what is ([3,128,128]) and just(3,128,128)

body:
I'm so confused about the image shape.
I just read the image and it shape with

  img[0].shape = ([3,128,128])

what is  ([3,128,128]) and (3,128,128)?
How can I convert it?

trying to reshpae the

img.shape = ([3,128,128])
img = cv2.resize(img,(32,32))
iamge_shape->torch.Size([1, 3, 128, 128])

  following error

img = cv2.resize(img,(32,32)) TypeError: Expected Ptr<cv::UMat> for argument '%s'


title: Denoising Autoencoder Testing mode for Multiclass Classification

body:
I am training an autoencoder for a multiclass classification problem where I transmit 16 equiprobable messages and send them through a denoising autoencoder to receive them. I am trying to implement the result (modification of Fig. 3b) in this paper, to be specific: Please refer to Fig. 2 in https://arxiv.org/pdf/1702.00832.pdf for the model.

Here is my autoencoder class:

class FullyConnectedAutoencoder(nn.Module):
    def __init__(self, k, n_channel, EbN0_dB):
        self.k = k
        self.n_channel = n_channel
        self.EbN0_dB = EbN0_dB

        super(FullyConnectedAutoencoder, self).__init__()
        self.transmitter = nn.Sequential(
            nn.Linear(in_features=2 ** k, out_features=2 ** k, bias=True),
            nn.ReLU(inplace=True),
            nn.Linear(in_features=2 ** k, out_features=n_channel, bias=True) )
        self.receiver = nn.Sequential(
            nn.Linear(in_features=n_channel, out_features=2 ** k, bias=True),
            nn.ReLU(inplace=True),
            nn.Linear(in_features=2 ** k, out_features=2 ** k, bias=True),)

    def forward(self, x):

        x = self.transmitter(x)
        # Normalization
        n = (x.norm(dim=-1)[:,None].view(-1,1).expand_as(x))
        x = sqrt(7)*(x / n)
        training_SNR = 10 ** (self.EbN0_dB / 10)  # Train at 3 dB
        R = k / n_channel
        noise = torch.randn(x.size()) / ((2*R*training_SNR) ** 0.5)
        x += noise

        x = self.receiver(x)
        return x

and my training loop is as follows:

# TRAINING
for epoch in range(epochs):
    for step, (x, y) in enumerate(trainloader):  # gives batch data, normalize x when iterate train_loader

        # Forward pass
        output = net(x)  # output
        y = (y.long()).view(-1)
        loss = loss_func(output, y)  # cross entropy loss

        # Backward and optimize
        optimizer.zero_grad()  # clear gradients for this training step
        loss.backward()  # backpropagation, compute gradients
        optimizer.step()  # apply gradients

        if step % 100 == 0:
            train_output = net(train_data)
            pred_labels = torch.max(train_output, 1)[1].data.squeeze()
            accuracy = sum(pred_labels == train_labels) / float(train_labels.size(0))
            print('Epoch: ', epoch, '| train loss: %.4f' % loss.item(), '| train accuracy: %.4f' % accuracy)

The training loop works well. However, I want to test my approach across different signal-to-noise ratios. I am having some problems doing that. Here are the two approaches I am trying

Approach 1: Declare a new object every time I test the autoencoder

for p in range(len(EbNo_test)):
with torch.no_grad():
    for test_data, test_labels in testloader:

        net = FullyConnectedAutoencoder(k, n_channel, EbNo_test[p])
        decoded_signal = net(test_data)

        # encoded_signal = net.transmitter(test_data)
        # noisy_signal = encoded_signal + test_noise
        # decoded_signal =  net.receiver(noisy_signal)

        pred_labels = torch.max(decoded_signal, 1)[1].data.squeeze()
        test_BLER[p] = sum(pred_labels != test_labels) / float(test_labels.size(0))

print('Eb/N0:',EbNo_test[p].numpy(), '| test BLER: %.4f' % test_BLER[p])

Approach 2: This is more intutive. Use the transmitter and receiver part separately and add noise after I transmit the signal.

for p in range(len(EbNo_test)):
    EcNo_test_sqrt[p] = 1/(2*R*(10**(EbNo_test[p]/20)))
    test_noise = EcNo_test_sqrt[p] * torch.randn(batch_size, n_channel)
    with torch.no_grad():
        for test_data, test_labels in testloader:

            encoded_signal = net.transmitter(test_data)
            noisy_signal = encoded_signal + test_noise
            decoded_signal =  net.receiver(noisy_signal)

            pred_labels = torch.max(decoded_signal, 1)[1].data.squeeze()
            test_BLER[p] = sum(pred_labels != test_labels) / float(test_labels.size(0))

    print('Eb/N0:',EbNo_test[p].numpy(), '| test BLER: %.4f' % test_BLER[p])

Strangely, I am getting wrong answers - meaning the errors are 90% where as they should be following a trend like this. (Fig. 3b in the paper cited above)

Am I doing something wrong? Any help is much appreciated.


title: pytorch: model.forward() effects the training process even if the result is not part of the loss function

body:
I have the following code to train a simple model:

train_loader, another_loader = get_loaders()
model = torch_utils.get_model()

ce = nn.CrossEntropyLoss()
optimizer = optim.SGD(model.parameters(), lr=0.001, momentum=0.9)
another_loader_iter = iter(another_loader)

for epoch in range(10):
    for i, (X, y) in enumerate(train_loader):
        model.train()
        X, y = X.to(device), y.to(device)
        pred = model.forward(X)
        loss1 = ce(pred, y)
        for j in range(2):
            X2, y2 = next(another_loader)
            pred2 = model.forward(X2) #**This line effects the training process - why???**

        new_loss = loss1
        optimizer.zero_grad()
        new_loss.backward()
        optimizer.step()

When I comment out pred2 = model.forward(X2) the model trains as it should but I want to use pred2 to add a new expression to the loss function.


title: Extract all 3x3 blocks from a tensor

body:
If I have a 5x5 tensor, then, how do I get all 9 3x3 blocks from it, so that resulting tensor is of shape [9, 3, 3] or if these 3x3 blocks are flattened then [9, 9] shape.
for example,

x = torch.randn(5, 5)

suppose x is

tensor([[ 0.5756,  0.2463,  1.3940,  0.8473, -0.8371],
        [ 0.9690,  1.4913, -0.2129,  0.8331, -0.6322],
        [-0.0348, -1.6920, -0.0157,  0.6159,  0.1038],
        [-1.0790,  1.4303,  0.3861,  0.1293,  0.4582],
        [ 0.2815, -1.1944, -0.7612,  0.6595,  1.4611]])

then resulting tensor should be like,

tensor([[0.5756,  0.2463,  1.3940, 0.9690,  1.4913, -0.2129, -0.0348, -1.6920, -0.0157],
 [0.2463, 1.3940,  0.8473, 1.4913, -0.2129,  0.8331, -1.6920, -0.0157,  0.6159],
...
[-0.0157,  0.6159,  0.1038, 0.3861,  0.1293,  0.4582, -0.7612,  0.6595,  1.4611]])


title: Can I program a GPU when it is the system's only graphics card?

body: Can I run PyTorch or Tensorflow on Windows on a GPU that is also acting as the system's graphics card (e.g. there is no graphics built-in to a Ryzen 3600 CPU)? If so, is there any downside, or would I be better off getting a CPU with built-in graphics?


title: Opposite to expected behavior [PyTorch]

body:
I have a dataset that has severe class imbalance (bank-additional-full). I'm using it without making any changes except assigning values to categorical variable. It has about 89% classes of no(0) and 11% class of yes(1).

My model always predicts yes (lower count of one) and also changing learning rate is having no effect. It should predict the class that has larger count more often i.e, no

I'm learning pytorch so let me know my mistake as i'm having trouble finding it.

class LogisticRegressionModel(nn.Module):
  def __init__(self, input_dim, output_dim):
      super(LogisticRegressionModel, self).__init__()
      self.linear = nn.Linear(input_dim, output_dim)
  def forward(self, x):

      out = F.softmax(self.linear(x),dim=1)
      return out

 input_dim = 1*20
 output_dim = 2

 model = LogisticRegressionModel(input_dim, output_dim)
 criterion = nn.CrossEntropyLoss()
 learning_rate = 0.01
 optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)

 iter = 0
num_epochs= 10
train_losses, val_losses = [], []
for epoch in range(num_epochs):
  running_loss = 0
  for i, (x, labels) in enumerate(train_loader):

      x = Variable(x.view(-1, height*width))
      labels = Variable(labels)
      optimizer.zero_grad()
      outputs = model(x.float())
      loss = criterion(outputs, labels)
      loss.backward()

      optimizer.step()

      running_loss += loss.item()

      iter += 1

  else:
      val_loss = 0

      correct = 0
      total = 0
      with torch.no_grad():

          for x, labels in val_loader:

              x = Variable(x.view(-1,height*width))
              outputs = model(x.float())

              val_loss += criterion(outputs, labels)

              values, predicted = torch.max(outputs.data, 1)
              #print(values.data)

              total += labels.size(0)

              correct += (predicted == labels).sum()

      accuracy = 100 * correct / total

      train_losses.append(running_loss/len(train_loader))
      val_losses.append(val_loss/len(val_loader))

      print("Epoch: {}/{}.. ".format(epoch+1, num_epochs),
                "Training Loss: {:.3f}.. ".format(running_loss/len(train_loader)),
                "Validation Loss: {:.3f}.. ".format(val_loss/len(val_loader)),
                "Validation Accuracy: {:.3f}".format(accuracy))

      print("\n")

Results:
Epoch: 1/10..  Training Loss: 1.201..  Validation Loss: 1.202..  Validation Accuracy: 11.000

loss and accuracy remain same and do not change


title: how to decide which decision should be taken from four given options?

body:
For my snake AI, I'm using pytorch whose structure is:

class SnakeAI(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc = nn.Sequential(
            nn.Linear(24, 16, bias=True),
            nn.ReLU(),
            nn.Linear(16, 8, bias=True),
            nn.ReLU(),
            nn.Linear(8, 4, bias=True),
            nn.Sigmoid()
        )

    def forward(self, inputs):
        x = self.fc(inputs)
        return x

Based on the inputs it gives me 4 values which to my understanding should be the probability of which decision to be, whose sum should be equal to 1. which is not the case here as the output is:

[0.59388083 0.5833764  0.47855872 0.5388371 ]

based on the given output should i use argmax to decide which direction to go with? is there any better way to do that?


title: How to know node/feature contributions?

body:
I'm working on GCN (Graph Convolutional Network) in PyTorch, in my application: a patient is a graph, nodes represent its genes, for each gene I have 2 features (gene structure and expression value).

The task is I'm doing a regression model to predict the risk of each patient to get a disease.

My question is,

1- how to know which nodes (genes) contribute to the prediction?

2- and which feature of the 2 that I have (gene structure and expression value) contribute to the prediction?

Any suggestions/ideas would be helpful, thanks.


title: Got `UnboundLocalError` error when measuring time for `torch.where`

body:
When try to %timeit in Jupyter Notebook got error; Without it working fine.

  UnboundLocalError: local variable 'a' referenced before assignment

import torch
a = torch.rand(10)
b = torch.rand(10)

%timeit a = torch.where(b > 0.5, torch.tensor(0.), a)

What is happening here?


title: Tensor slicing conditions for assignement

body:
Why does this
'''tensor_ques[1:3,1]=0'''
work
and this
'''tensor_ques[1:3,1] =[0,0]'''
does not work
tensor_ques is a torch.tensor()

I know [0,0] is not a tensor then why does =0 work fine


title: PyTorch CNN gives different results

body:
I am trying to make a face recognition model with PyTorch. My model performs well with both loss scores for training and validation close to 0. The problem is when I tested it using same input, it gives me different results like below:

and below

The left side is the true label and the right one is the prediction output. I set random seed and ran model.eval() before feeding the input to the model.

Does anyone know how to solve this?

Please advise


title: FileNotFoundError: No such file or directory. '=/models/test'

body:
I get following error and I totally don't know what is wrong.
The full path with p (dir_path) should look something like this:  /home/user/ulmfit4pl/plhater/models/test while the error shows No such file or directory: '=/models/test'  It looks like dir_path does not work in this case.
Full traceback:
Loading /home/user/ulmfit4pl/plhater/models/fwd_nl-4-v1_best_clas_1.h5
Converting mutli classification in to binary classificaiton
Traceback (most recent call last):
  File "/home/user/ulmfit4pl/ulmfit/evaluate.py", line 129, in <module>
    if __name__ == '__main__': fire.Fire(evaluate)
  File "/opt/conda/envs/ulmfit4pl/lib/python3.6/site-packages/fire/core.py", line 138, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/opt/conda/envs/ulmfit4pl/lib/python3.6/site-packages/fire/core.py", line 468, in _Fire
    target=component.__name__)
  File "/opt/conda/envs/ulmfit4pl/lib/python3.6/site-packages/fire/core.py", line 672, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/home/user/ulmfit4pl/ulmfit/evaluate.py", line 124, in evaluate
    (p / "models" / test_file ).mkdir(exist_ok=True)
  File "/opt/conda/envs/ulmfit4pl/lib/python3.6/pathlib.py", line 1248, in mkdir
    self._accessor.mkdir(self, mode)
  File "/opt/conda/envs/ulmfit4pl/lib/python3.6/pathlib.py", line 387, in wrapped
    return strfunc(str(pathobj), *args)
FileNotFoundError: [Errno 2] No such file or directory: '=/models/test'

The problem occurs in this function:
def evaluate(dir_path, clas_id, test_file='test1', cuda_id=0, nl=4, classes=3, bs=120, squeeze_bin=False, backwards=False, dump_preds=None, labels=False):
    if not hasattr(torch._C, '_cuda_setDevice'):
        print('CUDA not available. Setting device=-1.')
        cuda_id = -1
    torch.cuda.set_device(cuda_id)

    p = Path(dir_path)
    spp = sp.SentencePieceProcessor()
    spp.Load('/home/user/ulmfit4pl/plhater/tmp/sp.model')
    vs = spp.GetPieceSize()  # len(itos)
    spp.SetEncodeExtraOptions("bos:eos")

    # In[14]:
    dps = np.array([0.4,0.5,0.05,0.3,0.4])
    bptt,em_sz,nh,nl = 70,400,1150,nl
    c=classes
    m = get_rnn_classifer(bptt, 20*70, c, vs, emb_sz=em_sz, n_hid=nh, n_layers=nl, pad_token=1,
              layers=[em_sz*3, 50, c], drops=[dps[4], 0.1],
              dropouti=dps[0], wdrop=dps[1], dropoute=dps[2], dropouth=dps[3])
    model_path = "/home/user/ulmfit4pl/plhater/models/fwd_nl-4-v1_best_clas_1.h5"
    load_model(m, model_path)
    print("Loading", model_path)
    m = to_gpu(m)
    direction="bwd" if backwards else "fwd"
    preds, metrics = evaluate_model(test_file, m, p/"tmp", spp, bs, squeeze_bin, backwards, labels)
    if dump_preds is not None:
        with open(dump_preds, 'w') as f:
            f.write('\n'.join([str(int(x>=0)) for x in preds[:, 1]]))
    (p / "models" / test_file ).mkdir(exist_ok=True)
    np.save(p/"models"/ test_file/ f"{direction}_{clas_id}_clas_1-results.npy", preds)

    with open(p / "models" / test_file / f"{direction}_{clas_id}_clas_1-results.json", 'w') as fp:
        json.dump(metrics, fp)
if __name__ == '__main__': fire.Fire(evaluate)

Directory path:
p = Path(dir_path)
dir_path = /home/user/ulmfit4pl/plhater


title: Get variable from a Numpy array

body:
For a image classification problem with Pytorch, I read in my data as follows:
import scipy .io
emnist = scipy.io.loadmat(DATA_DIR + '/emnist-letters.mat')
data = emnist ['dataset']
X_train = data ['train'][0, 0]['images'][0, 0]
X_train = X_train.reshape((-1,28,28), order='F')

y_train = data ['train'][0, 0]['labels'][0, 0]

X_test = data ['test'][0, 0]['images'][0, 0]
X_test = X_test.reshape((-1,28,28), order = 'F')

y_test = data ['test'][0, 0]['labels'][0, 0]

I aim to create a dataset, using:
train_dataset = torch.utils.data.TensorDataset(torch.from_numpy(X_train), torch.from_numpy(y_train))

Currently, when I run an instance of y_train, the output is an array:
y_train[0]
>>> array([23], dtype=uint8)

However, I want train_dataset to contain only the number that's inside the array at the 0th index (in this case 23) instead of the entire array.
How can I change my code so that the TensorDataset that is created contains only the first element of the array of y_train, instead of the entire array?


title: How to mix categorical and continuous inputs in a PyTorch sequential model

body:
I am using PyTorch and I want to mix categorical and continuous inputs in a sequential neural network model. Up to now I only use continuous inputs as follows:
  model = torch.nn.Sequential(
    torch.nn.Linear(D_in, H),
    torch.nn.Tanh(),
    torch.nn.Dropout(0.125),
    torch.nn.Linear(H, H),
    torch.nn.Tanh(),
    torch.nn.Linear(H, D_out),
  )

Is there a "quick and dirty" way of incorporating categorical inputs on such a sequential model (maybe with embedding)?
I want to model a chemical process. The categories are some types of chemical compounds (I don't know any properties of them, just their names). The continuous inputs are the temperature, pressure ...
Thanks in advance!


title: How to judge whether the sample in a new data set belongs to original data set in PyTorch?

body:
I am new to PyTorch. Now, I have two data sets named A and B (eg: MNIST). I want to mix A and B together to form a new data set. And I want to shuffle this new data set. During the period of training, I need to determine whether the sample in batch is belongs to A. How can I do this?
The two problems are as follows:
1)How can I mix two datasets and shuffle it?
2)How can I determine whether the sample in new data set is belongs to original data set A?


title: Is Pytorch nn.Embedding supposed to give identical results with the same weights?

body:
I'm having an issue where after loading from my checkpoint my model is not giving me the same results. I've tested the same exact code in training and that works as expected, but after loading from a checkpoint I get a completely different result.
I think I have narrowed it down to the nn.embedding layer; I've checked to ensure that the embedding layer gives the same weights, but somehow when I look for a specific vocab in the embedding layer, I get different results during training vs inference with the same code.
For example, here I am saving the weights

and I get the embedding for vocab 7:

But after loading even though my weights are the same (titled Parameter containing: ), my embedding for vocab 7 is different!

Is there something I'm not aware of or is this expected behavior (some precision issue?) and this is a red herring.


title: How can I parallelize this bottleneck in my code?

body:
I am using pytorch for data processing, not for machine learning, etc.  This has resulted in a substantial performance boost, but I still have a bottleneck in the process.  Most of the time I am doing simple operations on large arrays, including boolean indexing.  However, at some point I need to compare the "final" array with values from another data set and find matches.  Something like this:
import numpy as np
import torch

device = torch.device("cuda:0")

# make couple tensors
data1 = torch.rand((1000, 10), device=device)
data2 = torch.rand((1000, 10), device=device)

# do some arithmetic operations on tensors (real code has a lot more operations)
data3 = data1 * data2

# a dataframe was built previously and must be loaded.  Here a dummy dataframe is created
df = pd.DataFrame(data=np.random.randint(0, 10, (10, 5)), columns=list("ABCDE"))

# define flag array
matchFound = np.full((8000, 1), False)

# now the bottleneck
for row, val in df.A.unique():
    # determine if anything in data3 is within 10% of "val"
    idx = (val >= 0.9 * data3) & (val <= 1.1 * data3)

    # if there are any matches then set flag value
    if idx.sum() > 0:
        matchFound[row, 0] = True

Everything up to the bottleneck point takes ~3-4 ms, but the bottleneck requires ~750 ms.  I need to pass through this ~18E6 times, making the total run time a couple weeks.  Is there any way I can speed this up?
What I have tried so far: I wrote a function to encapsulate the bottleneck region and pass it off to multiple CPUs using Process from multiprocessing (I also tried using Process from torch.multiprocessing).  The idea is to break this dataframe up into 15 chunks and send each chunk to a CPU thereby reducing the bottleneck time from 750 ms to 50 ms and my total runtime to 1d.  However, I keep getting a runtime error (RuntimeError: CUDA error: initialization error).  I don't understand what is causing this error (does CUDA just get angry when I pass tensors to multiple CPUs?).  Also, I think that by sending the data from the GPU to CPU takes more time than it saves.


The pytorch code is given below. How can the variable computed in the previous epoch be used for
computing the loss function in the next epoch?
for epoch in range((args.start_epoch+1), args.epochs):

    for input, target in train_loader:
        target = target.cuda()
        input = input.cuda()
        input_var = torch.autograd.Variable(input)
        target_var = torch.autograd.Variable(target)
        outputs, feature = model(input_var)

        if epoch > 0:
            l = criterion.forward(feature,target_var, Fea)

    Fea = function(model,train_loader)


title: The tensor can't be autograded with unsqueeze(2)

body:
rho = base[0][:,1:,:]
mu = base[1][:,1:,:]
diff_time = (time[:, 1:] - time[:, :-1]) * non_pad_mask[:, 1:]
temp_time = diff_time.unsqueeze(2) * \
           torch.rand([*diff_time.size(), num_samples], device=data.device)
temp_time /= (time[:, :-1] +1).unsqueeze(2)
all_base = torch.zeros(rho.size())
for i in range(rho.size()[2]):
    slice_of_mu = mu.clone()[:,:,i]
    slice_of_mu = slice_of_mu.unsqueeze(2)
    slice_of_rho = rho.clone()[:,:,i]
    slice_of_rho = slice_of_rho.unsqueeze(2)
    all_base[:,:,i] = all_base[:,:,i] + ( slice_of_mu * slice_of_rho * temp_time ^ ( slice_of_rho - 1 ) )

size of rho and mu are [16,99,5], and size of temp_time is [16,99,100]，I want to calculate the value of all_base as shown in the for loop, I sliced, unsquezeed and deformed these tensors to ensure their dimensions are consistent, but I get an error in the last line:
bitwise_xor(): functions with out=... arguments don't support automatic differentiation, but one of the arguments requires grad.
File ".......", line 47, in compute_integral_unbiased
all_base[:,:,i] = all_base[:,:,i] + ( slice_of_mu * slice_of_rho * temp_time ^ ( slice_of_rho - 1 ) )
I don't know how to solve this problem. Can you give me some help? Thank you very much.


title: How can I convert SmoothedValue into float for plotting using pyplot?

body:
I am trying to plot the loss vs epoch graph from my neural network using matplotlib.pyplot, but I am encountering a problem. After each epoch, the losses are gathered in an array and are SmoothedValue objects (from utils.py). Right now I am trying to get the loss after each epoch and storing it in an array so that later I can use it as an axis for pyplot. I do this the following way:
for epoch in range(num_epochs):
    epoch_list.append(epoch)
    # train for one epoch, printing every 10 iterations
    metric_logger = train_one_epoch(model, optimizer, data_loader, device, epoch, print_freq=10)

    loss_list = metric_logger.meters.get('loss')
    loss_axis.append(loss_list)

    print("Loss: ", loss_list.value)
    print("Loss: ", loss_list.deque)
    # update the learning rate
    lr_scheduler.step()
    # evaluate on the test dataset
    coco_evaluator = evaluate(model, data_loader_test, device=device)

After training in each epoch, I get the value corresponding to the key 'loss' in the dictionary metric_logger.meters, which I then append to my array loss_axis. However, the elements in metric_logger.meters are of the type SmoothedValue, which then my program can't interpret to plot it. How can I convert this Smoothed Value type to float so that I can plot it?


title: How to concatenate specific tensors in a list of torch.tensors along a particular dimension

body:
Suppose I have a list of torch tensors: 16 tensors in a list, each tensor of size (10,5).
I need to concatenate every 8th tensor in the list along dim=1.
At the end I want a list with 8 tensors, each tensor of size (10, 10).
I have tried this using for loop, torch.cat, list.append functions, but is time consuming.
Is there a more efficient way to do this?


title: Multi-Scale Cnn model from pytorch to keras

body:
I have this model in pytorch, which basically is a Multi-scale CNN model which returns the feature maps concatenated in the end for that image. I'm not really sure how to write the same thing in tensorflow 2 using keras. Any pointers is appreciated.
class VGG16ScaledFeatures(object):

def __init__(self, last_layer=22):
    self.vgg16_features = torch.nn.ModuleList(
        list(models.vgg16(pretrained=True).features)[:last_layer]
    ).eval()

def __call__(self, org):
    x_ = torch.tensor([])
    with torch.no_grad():
        for s in range(3):
            x = F.max_pool2d(org, (2 ** s, 2 ** s))
            for i, f in enumerate(self.vgg16_features):
                x = f(x)
                if (
                    (s == 0 and i == 21)
                    or (s == 1 and i == 14)
                    or (s == 2 and i == 7)
                ):
                    x_ = torch.cat([x_, x], dim=1)
                    break

    x_ = (x_ - x_.mean(dim=(2, 3), keepdim=True)) / x_.std(dim=(2, 3), keepdim=True)

    return x_


title: Apply a coordinate mask on a grid in Numpy or Pytorch

body:
As part of a reinforcement learning problem, I need to save gpu memory.
The memory size of the grid is quadratic (in number of pixels). However, only the pixels where the agents are located are important. By using "sparse" representations of these grids we save a lot of memory.
To save this memory I created another matrix format.
I have a mask M shape [Batch,N,2]:

B : Batch size
N : Number of agents
2 : x and y coordinates of the agents.

I also have a shape action grid [Batch,C,H,W]:

B : Batch size
C: Number of actions
H : Height of the grid
W: Grid width

The goal is to mask this action grid to obtain only the actions at the coordinate of the mask to obtain a masked grid of shape: [B,N,C].

Here is an example of what I would like to get:
>>> import numpy as np
>>> # Example of dimension
>>> BATCH_SIZE = 2
>>> N_AGENT = 3
>>> NB_ACTION = 2
>>> H_GRID = 3
>>> W_GRID = 3

>>> action_grid_batch.shape # [BATCH_SIZE, NB_ACTION, H_GRID, W_GRID]
(2, 2, 3, 3)

>>> action_grid_batch
array( [[[[0.4000, 0.5000, 0.7000], # Probability  of the action 1 on the action_grid 1 in the batch
          [0.3000, 0.2000, 0.1000],
          [0.9000, 0.8000, 0.7000]],

         [[0.6000, 0.5000, 0.3000], # Probability  of the action 2 on the action_grid 1 in the batch
          [0.7000, 0.8000, 0.9000],
          [0.1000, 0.2000, 0.3000]]],

        [[[0.3000, 0.2000, 0.1000], # Probability  of the action 1 on the action_grid 2 in the batch
          [0.6000, 0.7000, 0.4000],
          [0.9000, 0.8000, 0.1000]],

         [[0.7000, 0.8000, 0.9000], # Probability  of the action 2 on element 2 in the batch
          [0.4000, 0.3000, 0.6000],
          [0.1000, 0.2000, 0.9000]]]])

>>> batch_mask_agent_position
array( [[[0, 1], # Position (H, W) of the agent 1 on element 1 in the batch
         [1, 1], # Position (H, W) of the agent 2 on element 1 in the batch
         [2, 0]],# Position (H, W) of the agent 3 on element 1 in the batch

        [[1, 1], # Position (H, W) of the agent 1 on element 2 in the batch
         [1, 2], # Position (H, W) of the agent 2 on element 2 in the batch
         [2, 2]]]) # Position (H, W) of the agent 3 on element 2 in the batch

>>> output = apply_mask_on_grid(action_grid_batch,batch_mask_agent_position) # the function I would like

>>> output.shape
(2, 3, 2) # [Batch_size, N_AGENT, N_ACTION]
>>> output
array( [[[0.5000, 0.5000], # Probability of the action 1 and 2 for the agent position 1 on element 1 in the batch
         [0.2000, 0.8000], # Probability of the action 1 and 2 for the agent position 2 on element 1 in the batch
         [0.9000, 0.1000]], # Probability of the action 1 and 2 for the agent position 3 on element 1 in the batch

        [[0.7000, 0.3000], # Probability of the action 1 and 2 for the agent position 1 on element 2 in the batch
         [0.4000, 0.6000], # Probability of the action 1 and 2 for the agent position 2 on element 2 in the batch
         [0.1000, 0.9000]]]) # Probability of the action 1 and 2 for the agent position 3 on element 2 in the batch

(I will reimplement the proposed solution in Pytorch.)
Thanking you in advance !


title: PyTorch Tensors

body:
I am new to PyTorch. Can anyone explain to me why we are supposed to convert the data frame to a torch tensor, before performing Regression? (I've split the dataset, as you can see). And I also need an explanation to this piece of code:
x_train_tensor = torch.tensor(x_train.values.reshape(-1, 1), dtype = torch.float)
y_train_tensor = torch.tensor(y_train.values.reshape(-1, 1), dtype = torch.float)

x_test_tensor = torch.tensor(x_test.values.reshape(-1, 1), dtype = torch.float)
y_test_tensor = torch.tensor(y_test.values.reshape(-1, 1), dtype = torch.float)

Thank you!


title: Fail to run Pytorch NGC container image in browser

body:
I am new to docker and pull Pytorch container image to my local docker host using command docker pull nvcr.io/nvidia/pytorch:20.10-py3 in Windows Powershell.
Then I check the image in my docker registry and try to run the container from that image.
My command is docker container run -dit -v C:/Users/GGPC/Desktop/Pytorch:/root -p 0.0.0.0:6242:8888  --name pytorch 896906697db3.
I create a folder "Pytorch" at the desktop and specify it as the local dir and I use -v volume command and specify \root as the container dir. In this step, I am not sure if I specify both dir correctly and how to check the container dir that can be used for -v parameter?
After running my command above, a container is running successful in docker host.

Then I open it in browser, the connection failed. Run command  curl 0.0.0.0:6242, the error is curl : Unable to connect to the remote server. I suspect I didn't get the connection between container and local host working in the right way. How can I fix this problem?


title: I have an error on Pytorch and in particular with nllloss

body:
I want to appply the criterion, where
criterion = nn.NLLLoss()
I apply it on output and labels
loss = criterion(output.view(-1,1), labels.long())

where:
*the shape of the labels
labels
tensor([ 1,  4,  1,  1,  4,  1,  2,  3,  2,  4,  2,  3,  3,  4,
         0,  4])
output
tensor([ 0.1829,  0.1959,  0.1909,  0.1895,  0.1914,  0.1883,  0.1895,
         0.1884,  0.1865,  0.1931,  0.1883,  0.1917,  0.1942,  0.1937,
         0.1897,  0.1934])

the shape of the output
torch.Size([16])
On the following line:
loss = criterion(output.view(-1,1), labels.long())
I get this error:
The error is:

RuntimeError: Assertion `cur_target >= 0 && cur_target < n_classes' failed.  at /opt/conda/conda-bld/pytorch_1524584710464/work/aten/src/THNN/generic/ClassNLLCriterion.c:97

Any ideas?


title: Pass weights into CrossEntropyLoss in correct order

body:
I'm trying to use weight in torch.nn.CrossEntropyLoss,
but I'm not sure which order should I put
e.g.
    weight = torch.tensor([1.0, 52337/34649, 52337/11066]).to(device)

    criterion = nn.CrossEntropyLoss(weight=weight)

My class0 has 52337 examples it's labeled as 0 in target value, so I take 1.0
My class1 has 34649 examples it's labeled as 2 in target value, so I take 52337/34649
My class2 has 34649 examples it's labeled as 1 in target value, so I take 52337/11066
but I'm not sure which order should put in weight array,
My question is
Is there a way to show what is class0, class1 and class2 in CrossEntropy?
or the CrossEntropy will figure out the weights by itself?


title: How to make PyTorch DataLoader from scratch?

body:
Is it possible to recreate a simple version of PyTorch DataLoader from scratch?
The class should be able to return current batch based on the batch size.
For example, the code bellow only allows me to return one example at the time
X = np.array([[1,2],[3,4],[5,6],[6,7]])

class DataLoader:
    def __init__(self, X, b_size):
        self.X = X
        self.b_size = b_size

    def __len__(self):
        return len(self.X)

    def __getitem__(self, index):
        return self.X[index]

But what I want to achieve is that if I specify b_size=2, it would return:
Iteration 0: [[1,2],[3,4]]
Iteration 1: [[5,6],[7,8]]

Is it possible to do something like that in Python? I can't use the DataLoader class.