title: Pytorch, get rid of a for loop when adding permutation of one vector to entries of a matrix?

body:
I'm trying to implement this paper, and stuck with this simple step. Although this is to do with attention, the thing I'm stuck with is just how to implement a permutation of a vector added to a matrix without using for loops.
The attention scores have a learned bias vector added to them, the theory is that it encodes relative position (j-i) of the two tokens the score represents
so alpha is a T x T matrix,T depends on the batch being forwarded, and B is a learned bias vector whose length has to be fixed and as large as 2T. My current implementation which I believe does what the paper suggests is:
    def __init__(...):
       ...
        self.bias = torch.nn.Parameter(torch.randn(config.n),requires_grad = True)
        stdv = 1. / math.sqrt(self.bias.data.size(0))
        self.bias.data.uniform_(-stdv, stdv)
     def forward(..)
        ...
        #n = 201  (2* max_seq_len + 1)

        B_matrix = torch.zeros(self.T, self.T) # 60 x 60
        for i in range(self.T):
          B_matrix[i] = self.bias[torch.arange(start=n//2-i, end=n//2-i+T)])]

        attention_scores = attention_scores + B_matrix.unsqueeze(0)
        # 64 x 60 x 60
        ...

This is the only relevant part
B_matrix = torch.zeros(self.T, self.T) # 60 x 60
        for i in range(self.T):
          B_matrix[i] = self.bias[torch.arange(start=n//2-i, end=n//2-i+T)])]

basically trying to not use a for loop to go over each row.
but I know this must be really inefficient, and costly when this model is very large. I'm doing an explicit for loop over each row to get a permutation of the learned bias vector.
Can anyone help me out with a better way, through smart broadcasting perhaps?
After thinking about it, I don't need to instantiate a zero matrix, but still can't get rid of the for loop? and can't use gather as the B_matrix is a different size than a tiled b vector.
functor = lambda i : bias[torch.arange(start=n//2-i, end=n//2-i+T)]
B_matrix = torch.stack([functor(i) for i in torch.arange(T)])


title: Tensor size mismatch in Resnet50 with PyTorch with CIFAR-10

body:
I'm getting the following error with my code and I'm not really sure what I'm doing wrong. I've tried changing a few things around but I can't seem to find where the mistake is.
Is there an issue with the definition of the forward function? I'm assuming that part is fine, but I can't figure out what else could be causing this issue. Hope its not a stupid mistake.
Thanks
line 52, in forward
    return out + x
RuntimeError: The size of tensor a (5) must match the size of tensor b (4) at non-singleton dimension 3

import torch.nn as nn

# 1x1 convolution
def conv1x1(in_channels, out_channels, stride, padding):
   model = nn.Sequential(nn.Conv2d(in_channels, out_channels, kernel_size=1, stride=stride, padding=padding),
       nn.BatchNorm2d(out_channels),
       nn.ReLU(inplace=True))
   return model

# 3x3 convolution
def conv3x3(in_channels, out_channels, stride, padding):
   model = nn.Sequential(nn.Conv2d(in_channels, out_channels, kernel_size=3, stride=stride, padding=padding),
       nn.BatchNorm2d(out_channels),
       nn.ReLU(inplace=True))
   return model

###########################################################################
# Question 1 : Implement the "bottle neck building block" part.
# Hint : Think about difference between downsample True and False.  How we make
# the difference by code?
class ResidualBlock(nn.Module):
   def __init__(self, in_channels, middle_channels, out_channels, downsample=False):
       super(ResidualBlock, self).__init__()
       self.downsample = downsample

       if self.downsample:
           self.layer = nn.Sequential(##########################################
               ############## fill in here (20 points)
               # Hint : use these functions (conv1x1, conv3x3)
               conv1x1(in_channels, middle_channels, 1, 0),
               conv3x3(middle_channels, middle_channels, 1, 0),
               conv1x1(middle_channels, out_channels, 1, 0)
               #########################################
           )
           self.downsize = conv1x1(in_channels, out_channels, 2, 0)

       else:
           self.layer = nn.Sequential(##########################################
               ############# fill in here (20 points)
               conv1x1(in_channels, middle_channels, 2, 0),
               conv3x3(middle_channels, middle_channels, 2, 0),
               conv1x1(middle_channels, out_channels, 2, 0)
               #########################################
           )
           self.make_equal_channel = conv1x1(in_channels, out_channels, 1, 0)

   def forward(self, x):
       if self.downsample:
           out = self.layer(x)
           x = self.downsize(x)
           return out + x
       else:
           out = self.layer(x)
           if x.size() is not out.size():
               x = self.make_equal_channel(x)
           return out + x
###########################################################################

###########################################################################
# Question 2 : Implement the "class, ResNet50_layer4" part.
# Understand ResNet architecture and fill in the blanks below.  (25 points)
# (blank : #blank#, 1 points per blank )
# Implement the code.
class ResNet50_layer4(nn.Module):
   def __init__(self, num_classes=10): # Hint : How many classes in Cifar-10 dataset?
       super(ResNet50_layer4, self).__init__()
       self.layer1 = nn.Sequential(nn.Conv2d(stride=2, kernel_size=7, in_channels=3, padding=3, out_channels=64),
               # Hint : Through this conv-layer, the input image size is
               # halved.
               #        Consider stride, kernel size, padding and input &
               #        output channel sizes.
           nn.BatchNorm2d(64),
           nn.ReLU(inplace=True),
           nn.MaxPool2d(stride=2, kernel_size=3, padding=0))
       self.layer2 = nn.Sequential(ResidualBlock(in_channels=64, middle_channels=64, out_channels=256, downsample=False),
           ResidualBlock(in_channels=256, middle_channels=64, out_channels=256, downsample=False),
           ResidualBlock(in_channels=256, middle_channels=64,out_channels=256, downsample=True))
       self.layer3 = nn.Sequential(##########################################
           ############# fill in here (20 points)
           ####### you can refer to the 'layer2' above
           ResidualBlock(in_channels=256, middle_channels=128, out_channels=512, downsample=False),
           ResidualBlock(in_channels=512, middle_channels=128, out_channels=512, downsample=False),
           ResidualBlock(in_channels=512, middle_channels=128, out_channels=512, downsample=False),
           ResidualBlock(in_channels=512, middle_channels=128, out_channels=512, downsample=True)
           #########################################
       )
       self.layer4 = nn.Sequential(##########################################
           ############# fill in here (20 points)
           ####### you can refer to the 'layer2' above
           ResidualBlock(in_channels=512, middle_channels=256, out_channels=1024, downsample=False),
           ResidualBlock(in_channels=1024, middle_channels=256, out_channels=1024, downsample=False),
           ResidualBlock(in_channels=1024, middle_channels=256, out_channels=1024, downsample=False),
           ResidualBlock(in_channels=1024, middle_channels=256, out_channels=1024, downsample=False),
           ResidualBlock(in_channels=1024, middle_channels=256, out_channels=1024, downsample=False),
           ResidualBlock(in_channels=1024, middle_channels=256, out_channels=1024, downsample=False)
           #########################################
       )

       self.fc = nn.Linear(1024, 10) # Hint : Think about the reason why fc layer is needed
       self.avgpool = nn.AvgPool2d(7, stride=1)

       for m in self.modules():
           if isinstance(m, nn.Linear):
               nn.init.xavier_uniform_(m.weight.data)
           elif isinstance(m, nn.Conv2d):
               nn.init.xavier_uniform_(m.weight.data)

   def forward(self, x):

       out = self.layer1(x)
       out = self.layer2(out)
       out = self.layer3(out)
       out = self.layer4(out)
       out = self.avgpool(out)
       out = out.view(out.size()[0], -1)
       out = self.fc(out)

       return out
###########################################################################


title: Is there a method regarding skipping of batch which raise errors in pytorch?

body:
Nowadays, I have trouble with model training.
Some of my training data set raise some errors regarding index select (during LSTM network).
I couldn't find the solution and decided to skip the batch which raise the errors.
To do so, I used try-except but I couldn't do skipping.
Could you help me?
Thank you so much!
for epoch in tqdm(range(args.load_epoch + 1, args.epoch)):
    dataset = DataFolder(args.train, args.batch_size)

    for batch in dataset:
        total_step += 1
        model.zero_grad()
        try:
            loss, kl_div, wacc, iacc, tacc, sacc = model(*batch, beta=beta)
        except:
            continue

        loss.backward()
        nn.utils.clip_grad_norm_(model.parameters(), args.clip_norm)
        optimizer.step()

        meters = meters + np.array([kl_div, loss.item(), wacc * 100, iacc * 100, tacc * 100, sacc * 100])

        if total_step % args.print_iter == 0:
            meters /= args.print_iter
            print("[%d] Beta: %.3f, KL: %.2f, loss: %.3f, Word: %.2f, %.2f, Topo: %.2f, Assm: %.2f, PNorm: %.2f, GNorm: %.2f" % (total_step, beta, meters[0], meters[1], meters[2], meters[3], meters[4], meters[5], param_norm(model), grad_norm(model)))
            sys.stdout.flush()
            meters *= 0

        if args.save_iter >= 0 and total_step % args.save_iter == 0:
            n_iter = total_step // args.save_iter - 1
            torch.save(model.state_dict(), args.save_dir + "/model." + str(n_iter))
            scheduler.step()
            print("learning rate: %.6f" % scheduler.get_lr()[0])

    del dataset
    if args.save_iter == -1:
        torch.save(model.state_dict(), args.save_dir + "/model." + str(epoch))
        scheduler.step()
        print("learning rate: %.6f" % scheduler.get_lr()[0])

Please refer to below error message.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [11,0,0], thread: [90,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [11,0,0], thread: [91,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [11,0,0], thread: [92,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [11,0,0], thread: [93,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [11,0,0], thread: [94,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [11,0,0], thread: [95,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
  0%|                                                                                | 0/30 [04:31<?, ?it/s]
Traceback (most recent call last):
  File "vae_train.py", line 82, in <module>
    model.zero_grad()
  File "/home/sejeong/anaconda3/envs/PSJ/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1340, in zero_grad
    p.grad.zero_()
RuntimeError: CUDA error: device-side assert triggered


title: Predicting next 60 days using pytorch (LSTM)

body:
I am trying to use pytorch to make predictions on time-series dataset.

1- First, I splitted the dataset into training and test.

2- Then, I created the model.

import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()

        self.rnn = nn.LSTM(input_size=1,hidden_size=50,num_layers=3,batch_first=True,dropout=0.2)
        self.out = nn.Linear(50, 1)

    def forward(self, x):

        r_out, (h_n, h_c) = self.rnn(x, None)
        x = r_out[:,-1,:]                    #last hidden output!
        x = self.out(x)
        return x

model = Model()

if cuda:
    model.cuda()

3- After training the model, I made the validation step.

mse = mean_squared_error(dataset['y'].iloc[60:].values,predicted)
print("RMSE:", np.sqrt(mse))

# RMSE: 0.10680269716212222

Question: I would like to know, how can I use this model to predict the next 60 days ?

I read that is necessary to: Take the last prediction value(i.e predicted[-1]) from my predictions and add it to the last array of total_x, excluding the first timestep.

So, I tried this:

#array --> List
today = total_x[-1].reshape(-1).tolist()

# scaling last price
last_price = scaler.transform(infered['y_pred'][-1].reshape(-1, 1))

# adding last price to list.
today.append(last_price[0])

# Exclude first(0th index) element
today = today[1:]

#reshape
today = np.array(today).reshape(-1,60,1)

today = torch.Tensor(list(today))

#predict!
tomorrow = predict_with_pytorch(model, today)

#inverse transform.
tomorrow = scaler.inverse_transform(tomorrow)[0]

This Variable tomorrow is my tomorrow prediction, however, how to do this, for the next 60 days ?


title: Display result of convolution in PyTorch

body:
PyTorch newbie here. I wrote a script (code below) that performs the following operations: load an image, perform a 2D convolution operation and then display the output and the input.

At present I have the image below, which seems off. How can I plot the feature map correctly?

import numpy as np
import torch
import torchvision
import torchvision.transforms as transforms
import torch.nn as nn
import matplotlib.pyplot as plt
import imageio
import sys

A = imageio.imread('LiT.png')
# Define how the convolution operation works
conv2 = nn.Conv2d(in_channels=3, out_channels=3, kernel_size=3, stride=1, padding=1)

image_d = torch.FloatTensor(np.asarray(A.reshape(1, 3, A.shape[0] , A.shape[1])))
fc = conv2(image_d)
fc1 = fc.permute(0, 2, 3, 1).reshape([516, 780, 3])

plt.figure(figsize=(16,8))
plt.subplot(1,2,1)
plt.imshow(A)
plt.subplot(1,2,2)
plt.imshow(fc1.data.numpy())

plt.show()


title: How to revert BERT/XLNet embeddings?

body:
I've been experimenting with stacking language models recently and noticed something interesting: the output embeddings of BERT and XLNet are not the same as the input embeddings. For example, this code snippet:

bert = transformers.BertForMaskedLM.from_pretrained("bert-base-cased")
tok = transformers.BertTokenizer.from_pretrained("bert-base-cased")

sent = torch.tensor(tok.encode("I went to the store the other day, it was very rewarding."))
enc = bert.get_input_embeddings()(sent)
dec = bert.get_output_embeddings()(enc)

print(tok.decode(dec.softmax(-1).argmax(-1)))

Outputs this for me:

,,,,,,,,,,,,,,,,,

I would have expected the (formatted) input sequence to be returned since I was under the impression that the input and output token embeddings were tied.

What's interesting is that most other models do not exhibit this behavior. For example, if you run the same code snippet on GPT2, Albert or Roberta, it outputs the input sequence.

Is this a bug? Or is it expected for BERT/XLNet?


title: Argmax indexing in pytorch with 2 tensors of equal shape

body:
Summarize the problem
I am working with high dimensional tensors in pytorch and I need to index one tensor with the argmax values from another tensor. So I need to index tensor y of dim [3,4] with the results from the argmax of tensor xwith dim [3,4]. If tensors are:
import torch as T
# Tensor to get argmax from
# expected argmax: [2, 0, 1]
x = T.tensor([[1, 2, 8, 3],
              [6, 3, 3, 5],
              [2, 8, 1, 7]])

# Tensor to index with argmax from preivous
# expected tensor to retrieve [2, 4, 9]
y = T.tensor([[0,  1,  2,  3],
              [4,  5,  6,  7],
              [8,  9, 10, 11]])
# argmax
x_max, x_argmax = T.max(x, dim=1)

I would like an operation that given the argmax indexes of x, or x_argmax, retrieves the values in tensor y in the same indexes x_argmax indexes.
Describe what you’ve tried
This is what I have tried:
# What I have tried
print(y[x_argmax])
print(y[:, x_argmax])
print(y[..., x_argmax])
print(y[x_argmax.unsqueeze(1)])

I have been reading a lot about numpy indexing, basic indexing, advanced indexing and combined indexing. I have been trying to use combined indexing (since I want a slice in first dimension of the tensor and the indexes values on the second one). But I have not been able to come up with a solution for this use case.


title: PyTorch: How to use `torch.einsum()` to find the trace between the dot product of a nested tensor and another tensor

body:
Suppose I have a nested tensor A:
import torch.nn as nn
X = np.array([[1, 3, 2], [2, 3, 5], [1, 2, 3]])
X = torch.DoubleTensor(X)

rows = X.shape[0]
cols = X.shape[1]

A = torch.matmul(X.view(rows, cols, 1),
                 X.view(rows, 1, cols))

A

Output:
tensor([[[ 1.,  3.,  2.],
         [ 3.,  9.,  6.],
         [ 2.,  6.,  4.]],

        [[ 4.,  6., 10.],
         [ 6.,  9., 15.],
         [10., 15., 25.]],

        [[ 1.,  2.,  3.],
         [ 2.,  4.,  6.],
         [ 3.,  6.,  9.]]], dtype=torch.float64)

And I have another tensor B:
B = torch.DoubleTensor([[11., 21, 31], [31, 51, 31], [41, 51, 21]])
B

Output:
tensor([[11., 21., 31.],
        [31., 51., 31.],
        [41., 51., 21.]])

How do I use torch.einsum() to find the trace value between the dot product of each of the nested tensor in A and tensor B. For eg. the trace value of the dot product between the 1st nested tensor in A:
[[ 1.,  3.,  2.],
 [ 3.,  9.,  6.],
 [ 2.,  6.,  4.]]

and B:
tensor([[11., 21., 31.],
        [31., 51., 31.],
        [41., 51., 21.]])

and similarly with the other 2 nested tensors in A.
My results tensor will be a tensor with just 3 trace values. Is there a way to do this without looping over each of the nested tensor in A (with say a for loop)?
Ps:
I know the code to find the trace value between the dot product of 2 tensors is:
torch.einsum('ij,ji->', X, Y).item()

If you know how to do this with numpy.einsum(), please let me know too. I might just need to tweak numpy.einsum() a little to make it work for PyTorch tensors.


title: pytorch: How to use the output of the GRU model?

body:
The GRU model in pytorch outputs two objects: the output features as well as the hidden states. I understand that for classification one uses the output features, but I'm not entirely sure which of them. Specifically, in a typical decoder-encoder architecture that uses a GRU in the decoder part, one would typically only pass the last (time-wise, i.e., t = N, where N is the length of the input sequence) output to the encoder. Which part of the output tensor refers to this time-wise last output?
The GRU is created like so (note that it is bidirectional):
self.gru = nn.GRU(
            700,
            700,
            bidirectional=True,
            batch_first=True,
        )

Given some embedding vector representing a piece of text of size 150x700, I use the GRU like so (150 is the sequence length, 700 the embedding dimension):
gru_out, gru_hidden = self.gru(embedding)

gru_out will be of shape 150x1400, where 150 is again the sequence length and 1400 is double the embedding dimension, which is because of the GRU being a bidirectional one (in terms of pytorch's documentation, hidden_size*num_directions).
If I only want to access the time-wise last output, do I need to access it like so?
tmp = gru_out.view(150, 2, 700)
last_out_first_direction = tmp[149, 0, :]
last_out_second_direction = tmp[149, 1, :]

While this technically seems right and is similar to the answer posted here, it would also require that the actual input sequence is always of length 150, whereas typically you have also shorter actual input sequences that are simply padded to be of length 150. However, in GRU one is typically interested in the last actual input token, which can thus also be at a position <150. What is a common way to access the actual last token or time-step (<=150) instead of only the technically last step (always =150)?
Side question: Is the output of the second direction reversed (since the direction in which information is passed through the GRU is also reversed compared to the first direction) so I should actually access last_out_second_direction = tmp[0, 1, :] instead of tmp[149, 1, :]?


title: How can I calculate FLOPs and Params without 0 weights neurons affected?

body:
My Prune code is shown below, after running this, I will get a file named 'pruned_model.pth'.
import torch
from torch import nn
import torch.nn.utils.prune as prune
import torch.nn.functional as F
from cnn import net

ori_model = '/content/drive/My Drive/ECG_weight_prune/checkpoint_dir/model.pth'
save_path = '/content/drive/My Drive/ECG_weight_prune/checkpoint_dir/pruned_model.pth'
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

model = net().to(device)
model.load_state_dict(torch.load(ori_model))

module = model.conv1
print(list(module.named_parameters()))
print(list(module.named_buffers()))

prune.l1_unstructured(module, name="weight", amount=0.3)
prune.l1_unstructured(module, name="bias", amount=3)
print(list(module.named_parameters()))
print(list(module.named_buffers()))
print(module.bias)
print(module.weight)
print(module._forward_pre_hooks)
prune.remove(module, 'weight')
prune.remove(module, 'bias')
print(list(module.named_parameters()))
print(model.state_dict())
torch.save(model.state_dict(), save_path)

and results is :
[('weight', Parameter containing:
tensor([[[-0.0000, -0.3137, -0.3221,  ...,  0.5055,  0.3614, -0.0000]],

        [[ 0.8889,  0.2697, -0.3400,  ...,  0.8546,  0.2311, -0.0000]],

        [[-0.2649, -0.1566, -0.0000,  ...,  0.0000,  0.0000,  0.3855]],

        ...,

        [[-0.2836, -0.0000,  0.2155,  ..., -0.8894, -0.7676, -0.6271]],

        [[-0.7908, -0.6732, -0.5024,  ...,  0.2011,  0.4627,  1.0227]],

        [[ 0.4433,  0.5048,  0.7685,  ..., -1.0530, -0.8908, -0.4799]]],
       device='cuda:0', requires_grad=True)), ('bias', Parameter containing:
tensor([-0.7497, -1.3594, -1.7613, -2.0137, -1.1763,  0.4150, -1.6996, -1.5354,
         0.4330, -0.9259,  0.4156, -2.3099, -0.4282, -0.5199,  0.1188, -1.1725,
        -0.9064, -1.6639, -1.5834, -0.3655, -2.0727, -2.1078, -1.6431, -0.0694,
        -0.5435, -1.9623,  0.5481, -0.8255, -1.5108, -0.4029, -1.9759,  0.0522,
         0.0599, -2.2469, -0.5599,  0.1039, -0.4472, -1.1706, -0.0398, -1.9441,
        -1.5310, -0.0837, -1.3250, -0.2098, -0.1919,  0.4600, -0.8268, -1.0041,
        -0.8168, -0.8701,  0.3869,  0.1706, -0.0226, -1.2711, -0.9302, -2.0696,
        -1.1838,  0.4497, -1.1426,  0.0772, -2.4356, -0.3138,  0.6297,  0.2022,
        -0.4024,  0.0000, -1.2337,  0.2840,  0.4515,  0.2999,  0.0273,  0.0374,
         0.1325, -0.4890, -2.3845, -1.9663,  0.2108, -0.1144,  0.0544, -0.2629,
         0.0393, -0.6728, -0.9645,  0.3118, -0.5142, -0.4097, -0.0000, -1.5142,
        -1.2798,  0.2871, -2.0122, -0.9346, -0.4931, -1.4895, -1.1401, -0.8823,
         0.2210,  0.4282,  0.1685, -1.8876, -0.7459,  0.2505, -0.6315,  0.3827,
        -0.3348,  0.1862,  0.0806, -2.0277,  0.2068,  0.3281, -1.8045, -0.0000,
        -2.2377, -1.9742, -0.5164, -0.0660,  0.8392,  0.5863, -0.7301,  0.0778,
         0.1611,  0.0260,  0.3183, -0.9097, -1.6152,  0.4712, -0.2378, -0.4972],
       device='cuda:0', requires_grad=True))]

There are many zero weights existing. How can I calculate FLOPs and Params without counting calculations associated with these zero values?
I use the following code to calculate FLOPs and Params.
import torch
from cnn import net
from ptflops import get_model_complexity_info

ori_model = '/content/drive/My Drive/ECG_weight_prune/checkpoint_dir/model.pth'
pthfile = '/content/drive/My Drive/ECG_weight_prune/checkpoint_dir/pruned_model.pth'

model = net()
# model.load_state_dict(torch.load(ori_model))
model.load_state_dict(torch.load(pthfile))
# print(model.state_dict())

macs, params = get_model_complexity_info(model, (1, 260), as_strings=False,
                                         print_per_layer_stat=True, verbose=True)
print('{:<30}  {:<8}'.format('Computational complexity: ', macs))
print('{:<30}  {:<8}'.format('Number of parameters: ', params))

The output of both ori_model nad pthfile is the same, as follows.
Warning: module Dropout2d is treated as a zero-op.
Warning: module Flatten is treated as a zero-op.
Warning: module net is treated as a zero-op.
net(
  0.05 M, 100.000% Params, 0.001 GMac, 100.000% MACs,
  (conv1): Conv1d(0.007 M, 13.143% Params, 0.0 GMac, 45.733% MACs, 1, 128, kernel_size=(50,), stride=(3,))
  (conv2): Conv1d(0.029 M, 57.791% Params, 0.001 GMac, 50.980% MACs, 128, 32, kernel_size=(7,), stride=(1,))
  (conv3): Conv1d(0.009 M, 18.619% Params, 0.0 GMac, 0.913% MACs, 32, 32, kernel_size=(9,), stride=(1,))
  (fc1): Linear(0.004 M, 8.504% Params, 0.0 GMac, 0.404% MACs, in_features=32, out_features=128, bias=True)
  (fc2): Linear(0.001 M, 1.299% Params, 0.0 GMac, 0.063% MACs, in_features=128, out_features=5, bias=True)
  (bn1): BatchNorm1d(0.0 M, 0.515% Params, 0.0 GMac, 1.793% MACs, 128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (bn2): BatchNorm1d(0.0 M, 0.129% Params, 0.0 GMac, 0.114% MACs, 32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (dropout): Dropout2d(0.0 M, 0.000% Params, 0.0 GMac, 0.000% MACs, p=0.5, inplace=False)
  (faltten): Flatten(0.0 M, 0.000% Params, 0.0 GMac, 0.000% MACs, )
)
Computational complexity:       1013472.0
Number of parameters:           49669


title: The result is empty when prediction of Faster RCNN model (Pytorch)

body:
I'm trying to train Faster RCNN model. After training, I try to predict the result of image but the result is empty.
My data is w: 1600, h: 800, c: 3, classes: 7, bounding boxes:(x1, y1, x2, y2)
My model is below.

My model
import torchvision
from torchvision.models.detection.faster_rcnn import FastRCNNPredictor
from torchvision.models.detection.mask_rcnn import MaskRCNNPredictor
from torchvision.models.detection import FasterRCNN
from torchvision.models.detection.rpn import AnchorGenerator

def get_instance_segmentation_model(num_classes):
    backbone = torchvision.models.vgg16(pretrained=True).features
    backbone.out_channels = 512

    anchor_generator = AnchorGenerator(sizes=((32, 64, 128, 256, 512),),
                                    aspect_ratios=((0.5, 1.0, 2.0),))

    roi_pooler = torchvision.ops.MultiScaleRoIAlign(featmap_names=['0'],
                                                    output_size=7,
                                                    sampling_ratio=2)
    model = FasterRCNN(backbone,
                    num_classes=2,
                    rpn_anchor_generator=anchor_generator,
                    box_roi_pool=roi_pooler)
    in_features = model.roi_heads.box_predictor.cls_score.in_features
    model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)

    return model

device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')

num_classes = 2

model = get_instance_segmentation_model(num_classes)

model.to(device)

params = [p for p in model.parameters() if p.requires_grad]
optimizer = torch.optim.SGD(params, lr=0.005,
                            momentum=0.9, weight_decay=0.0005)

lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer,
                                               step_size=3,
                                               gamma=0.1)

training
# let's train it for 10 epochs
num_epochs = 10

for epoch in range(num_epochs):
    # train for one epoch, printing every 10 iterations
    train_one_epoch(model, optimizer, train_data_loader, device, epoch, print_freq=10)
    # update the learning rate
    lr_scheduler.step()
    # evaluate on the test dataset
    evaluate(model, valid_data_loader, device=device)

prediction:
prediction
[{'boxes': tensor([], device='cuda:0', size=(0, 4)),
  'labels': tensor([], device='cuda:0', dtype=torch.int64),
  'scores': tensor([], device='cuda:0')}]


title: TypeError: backward() got an unexpected keyword argument 'grad_tensors' in pytorch

body:
I have the following
w = torch.tensor([1.], requires_grad=True)
x = torch.tensor([2.], requires_grad=True)

a = torch.add(w, x)
b = torch.add(w, 1)

y0 = torch.mul(a, b)    # y0 = (x+w) * (w+1)
y1 = torch.add(a, b)    # y1 = (x+w) + (w+1)

loss = torch.cat([y0, y1], dim=0)       # [y0, y1]

weight = torch.tensor([1., 2.])

loss.backward(grad_tensors=weight)

The above give me TypeError: backward() got an unexpected keyword argument 'grad_tensors'
I check the website , the grad_tensors does live in the backward.
However, when I use
loss.backward(gradient=weight)

It works. gradient is not a parameters in backward. Any idea of that? my pytorch version is 1.7.0. Thanks.


title: Getting nan as loss value

body:
I have implemented focal loss in Pytorch with using of this paper. And ran into a problem with loss - got nan as loss function value.
This is implementation of focal loss:
def focal_loss(y_real, y_pred, gamma = 2):
    y_pred = torch.sigmoid(y_pred)
    return -torch.sum((1 - y_pred)**gamma * y_real * torch.log(y_pred) +
                       y_pred**gamma * (1 - y_real) * torch.log(1 - y_pred))

Train loop and my SegNet are working, I think so, because I have tested them with dice and bce losses.
I think errors occurs in backprop. Why can it be? Maybe my implementation is wrong?


title: How to convert pytorch (CNN+LSTM) model to tflite?

body:
I am trying to convert CNN+LSTM model mentioned in the following blog Image Captioning using Deep Learning (CNN and LSTM). Related github repo is : Pytorch image captioning
I want to convert this pytorch model to tflite. It has both encoder and decoder checkpoints. As far as i understand both of them have to be converted to tflite (correct me if i am wrong)
approach:  using the example mentioned in onnx2keras library, onnx2keras i was able to convert encoder to tflite. but with decoder i am facing the following issue.
Not sure what is the right approach. can anyone suggest better approach and help me achieve a tflite model

File “convert_pytorch_tf.py”, line 63, in change_ordering=False) File
“/root/anaconda3/envs/pyt2tf/lib/python3.7/site-packages/pytorch2keras/converter.py”,
line 53, in pytorch_to_keras dummy_output = model(*args) File
“/root/anaconda3/envs/pyt2tf/lib/python3.7/site-packages/torch/nn/modules/module.py”,
line 550, in call result = self.forward(*input, **kwargs) TypeError:
forward() missing 2 required positional arguments: ‘captions’ and
‘lengths’

Please let me know which approach to follow and help me fix the issue related to the approach i have taken


title: cifar100 dataset consists of repeated image?

body:
I randomly browsed some images in cifar100, and found many images like this:
.

Anything went wrong? Or cifar100 indeed consist of such images?


title: understanding seed of a ByteTensor in PyTorch

body:
I understand that a seed is a number used to initialize pseudo-random number generator. in pytorch, torch.get_rng_state documentation states as follows "Returns the random number generator state as a torch.ByteTensor.". and when i print it i get a 1-d tensor of size 5048 whose values are as shown below

  tensor([ 80,  78, 248,  ...,   0,   0,   0], dtype=torch.uint8)

why does a seed have 5048 values and how is this different from usual seed which we can get using  torch.initial_seed


title: Sparse tensors to decrease training time

body:
I've learning about PyTorch sparse tensors : https://pytorch.org/docs/stable/sparse.html

From the docs (https://pytorch.org/docs/stable/sparse.html) : "Torch supports sparse tensors in COO(rdinate) format, which can efficiently store and process tensors for which the majority of elements are zeros."

Is one of the intended use of sparse tensors instead of regular PyTorch tensors to decrease training time ?


title: Autoencoder model either oscillates or doesn't converge on MNIST dataset

body:
Already ran the code 3 months ago with intended results. Changed nothing. Tried troubleshooting by using codes from (several) earlier versions, including among the earliest (which definitely worked). The problem persists.

# 4 - Constructing the undercomplete architecture
class autoenc(nn.Module):
    def __init__(self, nodes = 100):
        super(autoenc, self).__init__() # inheritence
        self.full_connection0 = nn.Linear(784, nodes) # encoding weights
        self.full_connection1 = nn.Linear(nodes, 784) # decoding weights
        self.activation = nn.Sigmoid()

    def forward(self, x):
        x = self.activation(self.full_connection0(x)) # input encoding
        x = self.full_connection1(x) # output decoding
        return x

# 5 - Initializing autoencoder, squared L2 norm, and optimization algorithm
model = autoenc().cuda()
criterion = nn.MSELoss()
optimizer = optim.Adam(model.parameters(),
                          lr = 1e-3, weight_decay = 1/2)

# 6 - Training the undercomplete autoencoder model
num_epochs = 500
batch_size = 32
length = int(len(trn_data) / batch_size)

loss_epoch1 = []

for epoch in range(num_epochs):
    train_loss = 0
    score = 0.

    for num_data in range(length - 2):
        batch_ind = (batch_size * num_data)
        input = Variable(trn_data[batch_ind : batch_ind + batch_size]).cuda()

        # === forward propagation ===
        output = model(input)
        loss = criterion(output, trn_data[batch_ind : batch_ind + batch_size])

        # === backward propagation ===
        loss.backward()

        # === calculating epoch loss ===
        train_loss += np.sqrt(loss.item())
        score += 1. #<- add for average loss error instead of total
        optimizer.step()

    loss_calculated = train_loss/score
    print('epoch: ' + str(epoch + 1) + '   loss: ' + str(loss_calculated))
    loss_epoch1.append(loss_calculated)

When plotting the loss now, it oscillates oscillates wildly (at lr = 1e-3). Whereas 3 months ago, it was steadily converging (at lr = 1e-3).

Can't upload pictures yet due to recently created account.

How it looks like now.

Though this is when I reduce the learning rate to 1e-5. When it's at 1e-3, it's just all over the places.

How it should look like, and used to look like at lr = 1e-3.


title: Regarding the consistency of convolutional neural networks

body:
I am currently building a 2-channel (also called double-channel) convolutional neural network in order to classify 2 binary images (containing binary objects) as 'similar' or 'different'.

The problem I am having is that it seems as though the network doesn't always converge to the same solution. For example, I can use exactly the same ordering of training pairs and all the same parameters and so forth, and when I run the network multiple times, each time produces a different solution; sometimes converging to below 2% error rates, and other times I get 50% error rates.

I have a feeling that it has something to do with the random initialization of the weights of the network, which results in different optimization paths each time the network is executed. This issue even occurs when I use SGD with momentum, so I don't really know how to 'force' the network to converge to the same solution (global optima) every time?

Can this have something to do with the fact that I am using binary images instead of grey-scale or color images, or is there something intrinsic to neural networks that is causing this issue?


title: How to set up loss function in PyTorch for Soft-Actor-Critic

body:
I'm trying to implement a custom loss function for a soft Q-learning, actor-critic policy gradient algorithm in PyTorch. This comes from the following paper Learning from Imperfect Demonstrations. The structure of the algorithm is similar to deep q-learning, in that we are using a network to estimate Q-values, and we use a target network to stabilize results. Unlike DQN, however, we calculate V(s) from Q(s) by:

This is simple enough to calculate with PyTorch. My main question has to do with how to set up the loss function. Part of the update equation is expressed as:

Note that Q_hat comes from the target network. How can I go about putting something like this into a loss function? I can compute values for V and Q, but how can I handle the gradients in this case? If anyone can point me towards a similar example that would be much appreciated.


title: what should I do if my regression model stuck at a high value loss?

body:
I'm using neural nets for a regression problem where I have 3 features and I'm trying to predict one continuous value. I noticed that my neural net start learning good but after 10 epochs it get stuck on a high loss value and could not improve anymore.

I tried to use Adam and other adaptive optimizers instead of SGD but that didn't work. I tried a complex architectures like adding layers, neurons, batch normalization and other activations etc.. and that also didn't work.
I tried to debug and try to find out if something is wrong with the implementation but when I use only 10 examples of the data my model learn fast so there are no errors.  I start to increase the examples of the data and monitoring my model results as I increase the data examples. when I reach 3000 data examples my model start to get stuck on a high value loss.
I tried to increase layers, neurons and also to try other activations, batch normalization. My data are also normalized between [-1, 1], my target value is not normalized since it is regression and I'm predicting a continuous value. I also tried using keras but I've got the same result.

My real dataset have 40000 data, I don't know what should I try, I almost try all things that I know for optimization but none of them worked. I would appreciate it if someone can guide me on this. I'll post my Code but maybe it is too messy to try to understand, I'm sure there is no problem with my implementation, I'm using skorch/pytorch and some SKlearn functions:

# take all features as an Independant variable except the bearing and distance
# here when I start small the model learn good but from 3000 data points as you can see the model stuck on a high value. I mean the start loss is 15 and it start to learn good but when it reach 9 it stucks there
# and if I try to use the whole dataset for training then the loss start at 47 and start decreasing until it reach 36 and then stucks there too
X = dataset.iloc[:3000, 0:-2].reset_index(drop=True).to_numpy().astype(np.float32)

# take distance and bearing as the output values:
y = dataset.iloc[:3000, -2:].reset_index(drop=True).to_numpy().astype(np.float32)
y_bearing = y[:, 0].reshape(-1, 1)
y_distance = y[:, 1].reshape(-1, 1)

# normalize the input values
scaler = StandardScaler()
X_norm = scaler.fit_transform(X, y)

X_br_train, X_br_test, y_br_train, y_br_test = train_test_split(X_norm,
                                                                y_bearing,
                                                                test_size=0.1,
                                                                random_state=42,
                                                                shuffle=True)

X_dis_train, X_dis_test, y_dis_train, y_dis_test = train_test_split(X_norm,
                                                                    y_distance,
                                                                    test_size=0.1,
                                                                    random_state=42,
                                                                    shuffle=True)
bearing_trainset = Dataset(X_br_train, y_br_train)
bearing_testset = Dataset(X_br_test, y_br_test)

distance_trainset = Dataset(X_dis_train, y_dis_train)
distance_testset = Dataset(X_dis_test, y_dis_test)

def root_mse(y_true, y_pred):
    return np.sqrt(mean_squared_error(y_true, y_pred))

class RMSELoss(nn.Module):
    def __init__(self):
        super().__init__()
        self.mse = nn.MSELoss()

    def forward(self, yhat, y):
        return torch.sqrt(self.mse(yhat, y))

class AED(nn.Module):
    """custom average euclidean distance loss"""
    def __init__(self):
        super().__init__()

    def forward(self, yhat, y):
        return torch.dist(yhat, y)

def train(on_target,
          hidden_units,
          batch_size,
          epochs,
          optimizer,
          lr,
          regularisation_factor,
          train_shuffle):

    network = None
    trainset = distance_trainset if on_target.lower() == 'distance' else bearing_trainset
    testset = distance_testset if on_target.lower() == 'distance' else bearing_testset
    print(f"shape of trainset.X = {trainset.X.shape}, shape of trainset.y = {trainset.y.shape}")
    print(f"shape of testset.X = {testset.X.shape}, shape of testset.y = {testset.y.shape}")

    mse = EpochScoring(scoring=mean_squared_error, lower_is_better=True, name='MSE')
    r2 = EpochScoring(scoring=r2_score, lower_is_better=False, name='R2')
    rmse = EpochScoring(scoring=make_scorer(root_mse), lower_is_better=True, name='RMSE')

    checkpoint = Checkpoint(dirname=f'results/{on_target}/checkpoints')
    train_end_checkpoint = TrainEndCheckpoint(dirname=f'results/{on_target}/checkpoints')

    if on_target.lower() == 'bearing':
        network = BearingNetwork(n_features=X_norm.shape[1],
                                 n_hidden=hidden_units,
                                 n_out=y_distance.shape[1])

    elif on_target.lower() == 'distance':
        network = DistanceNetwork(n_features=X_norm.shape[1],
                                  n_hidden=hidden_units,
                                  n_out=1)

    model = NeuralNetRegressor(
        module=network,
        criterion=RMSELoss,
        device='cpu',
        batch_size=batch_size,
        lr=lr,
        optimizer=optim.Adam if optimizer.lower() == 'adam' else optim.SGD,
        optimizer__weight_decay=regularisation_factor,
        max_epochs=epochs,
        iterator_train__shuffle=train_shuffle,

        train_split=predefined_split(testset),
        callbacks=[mse, r2, rmse, checkpoint, train_end_checkpoint]
    )

    print(f"{'*' * 10} start training the {on_target} model {'*' * 10}")
    history = model.fit(trainset, y=None)

    print(f"{'*' * 10} End Training the {on_target} Model {'*' * 10}")

if __name__ == '__main__':

    args = parser.parse_args()

    train(on_target=args.on_target,
          hidden_units=args.hidden_units,
          batch_size=args.batch_size,
          epochs=args.epochs,
          optimizer=args.optimizer,
          lr=args.learning_rate,
          regularisation_factor=args.regularisation_lambda,
          train_shuffle=args.shuffle)

and this is my network declaration:

class DistanceNetwork(nn.Module):
    """separate NN for predicting distance"""
    def __init__(self, n_features=5, n_hidden=16, n_out=1):
        super().__init__()
        self.model = nn.Sequential(
            nn.Linear(n_features, n_hidden),
            nn.LeakyReLU(),
            nn.Linear(n_hidden, 5),
            nn.LeakyReLU(),
            nn.Linear(5, n_out)
        )

here is the log while training:


title: How to rename Pytorch object name?

body:
My Pytorch Model:

EfficientDet(
  (backbone): EfficientNetFeatures(
    (conv_stem): Conv2d(4, 48, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
    (bn1): BatchNorm2d(48, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)
...
...

Is there a way to rename backbone object to some other name?


title: Identical random crop on two images Pytorch transforms

body: I am trying to feed two images into a network and I want to do identical transform between these two images. transforms.Compose() takes one image at a time and produces output independent to each other but I want same transformation. I did my own coding for hflip() now I am interested to get the random crop. Is there any way to do that without writing custom functions?


title: pytorch: how to multiply 3d tensor with 2d tensor

body:
I got a 3D tensor three and a 2D tensor two, which need to be multiplied. For example, the dimensions are:
three.shape = 4x100x700
two.shape = 4x100

Output shape should be:
output.shape = 4x100x700

So basically, in output[a,b] there should be 700 scalars which were computed by multiplying all 700 scalars from three[a,b] with the single scalar from two[a,b].


title: load and freeze one model and train others in PyTorch

body:
I have a model A that including three submodels model1, model2, model3.
the model flow: model1 --> model2 --> model3
I have trained model1 in an independent project.
The question is how to use the pre-trained model1 when training the model A?
Now, I try to implement this as follow:
I load the checkpoint of model1 by `model1.load_state_dict(torch.load(model1.pth)) and then set the requires_grad of the model1’s parameters as False?
Is it right?


title: tensor transformation in pytorch?

body:
I have a tensor of shape (size, 1) and I want to convert it into of shape (size, lookback, 1) by shifting its values. A pandas equivalent is below
size = 7
lookback = 3

data = pd.DataFrame(np.arange(size), columns=['out'])  # input
y = np.full((len(data), lookback, 1), np.nan)          # required/output
for j in range(lookback):
    y[:, j, 0] = data['out'].shift(lookback - j - 1).fillna(method="bfill")

How can I acheive similar in pytorch?
Example input:
[0, 1, 2, 3, 4, 5, 6]

Desired output:
[[0. 0. 0.]
 [0. 0. 1.]
 [0. 1. 2.]
 [1. 2. 3.]
 [2. 3. 4.]
 [3. 4. 5.]
 [4. 5. 6.]]


title: When using torchvision.ops.roi_align, I got the error:Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)

body: My environment:torch-1.2, torchvision-0.3, python-3.7. I also tried other visions like torchvision-0.5, torchvision-0.6, but always got this error. What should I do could sovle this?


title: Is it possible to feed the output back to input in artificial neural network?

body:
I am currently designing a artificial neural network for a problem with a decay curve.
For example, building a model for predicting the durability of the some material. It may includes the environment condition like temperature and humidity.
However, it is not adequate to predict the durability of the material. For such a problem, I think it is better to using the output durability of previous time slots as one of the current input to predict the durability of next time slot.
Moreover, I do not know how to train a model which feed the output back to input as one of the input columns has only the initial value before training.
For this case,
Method 1 (fail)
I have tried to fill the predicted output durability of current row to the input durability of next row. Nevertheless, it will prevent the model from "loss.backward()" so we cannot compute and update the gradient if we do so. The gradient function used was "CopySlices" instead of "MSELoss" when I copied the predicted output to the next row of the input data.
Feed output to input
gradient function -copy-
Method 2 "fill the input column with expected output"
In this method, I fill the blank input column with expected output (row-1) before training the model. Filling the input column with expected output of previous row is only done for training. For real prediction, I will feed the predicted output to the input. In this case, I am successful to train a overfitting model with MSELoss.
Moreover, I do not believe it is a right method as it uses the expected output as the input no matter how bad it predict. I strongly believed that it is not a right method.

Therefore, I want to ask whether it is possible to feed output to input in linear regression problem using artificial neural network.
I apologize for uploading no code here as I am not convenient to upload the full code here. It may be confidential.


title: pyTorch gradient becomes none when dividing by scalar

body:
Consider the following code block:
import torch as torch

n=10
x = torch.ones(n, requires_grad=True)/n
y = torch.rand(n)
z = torch.sum(x*y)
z.backward()
print(x.grad) # results in None
print(y)

As written, x.grad is None. However, if I change the definition of x by removing the scalar multiplication (x = torch.ones(n, requires_grad=True)) then indeed I got a non-None gradient that is equivalent to y.
I've googled a bunch looking for this issue, and I think it reflects something fundamental in what I don't understand about how the computational graph in torch. I'd love some clarification. Thanks!


title: Augmenting PIL images with borders in Pytorch efficiently

body:
I am trying to classify images where some of the evaluation images have a border of one of the following two types: White rectangles or black circle.
I would therefore like to see if I can improve the performance of my model by randomly augmenting some of the training images with a border. The images are loaded as PIL images and I am using Pytorch.
At the moment, I have the following code:
class BorderTransformer(torch.nn.Module):
  def __init__(self, black_p=0.05, white_p=0.01, white_border_prop=0.05):
    """
    Args:
      black_p (float): Probability of adding a black circle border
      white_p (float): Probability of adding a white rectangle border if
                       black circle border was not added
      white_border_prop (float): Proportion of white border to add in relation
                                 to img. If 0.1, will add a white border spanning
                                 10% of the image to each side.
    """
    super().__init__()
    self.white_p = white_p
    self.black_p = black_p
    assert white_border_prop < 0.5, "Can't add white border bigger than full image"
    self.white_border_prop = white_border_prop
    self.black_indices = None
    self.to_tensor = transforms.ToTensor()
    self.to_pil = transforms.ToPILImage()

  def white_square_border(self, img):
    """
    Args:
      img (PIL Image): Input image.

    Returns:
      PIL Image: White rectangle bordered image.
    """
    img_tensor = self.to_tensor(img).to(device)
    _, width, height = img_tensor.size()

    h_border = int(self.white_border_prop * width)
    v_border = int(self.white_border_prop * height)

    # Setting left, right, top, bottom border in that order
    img_tensor[:, :h_border, :] = 1.0
    img_tensor[:, width - h_border:, :] = 1.0
    img_tensor[:, :, :v_border] = 1.0
    img_tensor[:, :, height - v_border:] = 1.0

    return self.to_pil(img_tensor)

  def _compute_black_indices(self, dim):
    """
    Args:
      dim (int): Height and width of image

    Returns:
      None, but sets self.black_indices to tensor of size (3, dim, dim)
      where values are True outside the circle and False inside.
    """
    black_indices = torch.zeros([3, dim, dim], dtype=bool)
    center = torch.tensor([dim // 2, dim // 2])
    max_len = torch.tensor(dim**2.0)
    for x in range(dim):
      for y in range(dim):
        vec = center - torch.tensor([x, y])
        is_outside_circle = (vec * vec).sum() > max_len
        black_indices[:, x, y] = is_outside_circle
    self.black_indices = black_indices

  def black_circle_border(self, img):
    """
    Args:
      img (PIL Image): Input image.

    Returns:
      PIL Image: Black circle bordered image.
    """
    img_tensor = self.to_tensor(img)
    _, width, height = img_tensor.size()
    if self.black_indices is None:
      self.black_indices = self._compute_black_indices(width)
    img_tensor[self.black_indices] = 0.0

    return self.to_pil(img_tensor)

  def forward(self, img):
    """
    Args:
      img (PIL Image): Input image.

    Returns:
      PIL Image: Bordered image.
    """
    if torch.rand(1) < self.black_p:
      return self.black_circle_border(img)
    elif torch.rand(1) < self.white_p:
      return self.white_square_border(img)
    return img

While this seems to work, it is also horrendously slow (making training take 3-4x as long as without the augmention). I suspect it is due to the conversion to Tensor followed by the conversion back to PIL.
How would one write code that accomplishes something similar in an efficient manner?


title: BERT model loading not working with pytorch 1.3.1-eia container

body:
I have a custom python file for inference in which I have implemented the functions model_fn, input_fn, predict_fn and output_fn.  I have saved the model as a torchscript using torch.jit.trace, torch.jit.save and loading it using torch.jit.load. The model_fn implementation is as follows:
import torch
import os
import logging

logger = logging.getLogger()
is_ei = os.getenv("SAGEMAKER_INFERENCE_ACCELERATOR_PRESENT") == "true"
logger.warn(f"Elastic Inference enabled: {is_ei}")

def model_fn(model_dir):
    model_path = os.path.join(model_dir, "model_best.pt")
    try:
        loaded_model = torch.jit.load(model_path, map_location=torch.device('cpu'))
        loaded_model.eval()
        return loaded_model
    except Exception as e:
        logger.exception(f"Exception in model fn {e}")
        return None

This implementation works perfectly for the container with pytorch 1.5. But for container with torch 1.3.1 it exits abruptly when loading the pretrained model without any logs. The only line I see in the logs is
algo-1-nvqf7_1  | 2020-11-30 07:17:15,392 [WARN ] W-9000-model com.amazonaws.ml.mms.wlm.BatchAggregator - Load model failed: model, error: Worker died.
algo-1-nvqf7_1  | 2020-11-30 07:17:15,393 [INFO ] W-9000-model com.amazonaws.ml.mms.wlm.WorkerThread - Retry worker: 9000-44f1cd64 in 1 seconds.

The worker dies and tries to restart, and the process repeats till I stop the container.
The model I am using is trained with pytorch 1.5. But since EI support is only supported till 1.3.1, I am using this container.
Things I have tried:

The same code with same model works outside the container with pytorch version 1.3.1. So, I don't think pytorch version compatibility is the issue.
Tried using debug and notset levels for logs. Didn't get any more info as to why model loading fails
Tried loading the original model instead of the traced one. Again this works in 1.5 but not in 1.3.1. Fails at the same point, while loading the BERT pretrained model.
Tried this setup on sagemaker notebook instance with gpu accelerator and sagemaker PytorchModel's deploy() function with framework_version as 1.3.1. Also tried it using the 1.3.1 container without eia. Has same behaviour everywhere.

Am I doing something wrong or missing something crucial from the documentation? Any help would be much appreciated.
**Logs for container with torch 1.3.1-eia **
algo-1-nvqf7_1  | 2020-11-30 07:17:14,333 [INFO ] main com.amazonaws.ml.mms.ModelServer -
algo-1-nvqf7_1  | MMS Home: /opt/conda/lib/python3.6/site-packages
algo-1-nvqf7_1  | Current directory: /
algo-1-nvqf7_1  | Temp directory: /home/model-server/tmp
algo-1-nvqf7_1  | Number of GPUs: 0
algo-1-nvqf7_1  | Number of CPUs: 8
algo-1-nvqf7_1  | Max heap size: 6972 M
algo-1-nvqf7_1  | Python executable: /opt/conda/bin/python
algo-1-nvqf7_1  | Config file: /etc/sagemaker-mms.properties
algo-1-nvqf7_1  | Inference address: http://0.0.0.0:8080
algo-1-nvqf7_1  | Management address: http://0.0.0.0:8080
algo-1-nvqf7_1  | Model Store: /.sagemaker/mms/models
algo-1-nvqf7_1  | Initial Models: ALL
algo-1-nvqf7_1  | Log dir: /logs
algo-1-nvqf7_1  | Metrics dir: /logs
algo-1-nvqf7_1  | Netty threads: 0
algo-1-nvqf7_1  | Netty client threads: 0
algo-1-nvqf7_1  | Default workers per model: 1
algo-1-nvqf7_1  | Blacklist Regex: N/A
algo-1-nvqf7_1  | Maximum Response Size: 6553500
algo-1-nvqf7_1  | Maximum Request Size: 6553500
algo-1-nvqf7_1  | Preload model: false
algo-1-nvqf7_1  | Prefer direct buffer: false
algo-1-nvqf7_1  | 2020-11-30 07:17:14,391 [WARN ] W-9000-model com.amazonaws.ml.mms.wlm.WorkerLifeCycle - attachIOStreams() threadName=W-9000-model
algo-1-nvqf7_1  | 2020-11-30 07:17:14,481 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - model_service_worker started with args: --sock-type unix --sock-name /home/model-server/tmp/.mms.sock.9000 --handler sagemaker_pytorch_serving_container.handler_service --model-path /.sagemaker/mms/models/model --model-name model --preload-model false --tmp-dir /home/model-server/tmp
algo-1-nvqf7_1  | 2020-11-30 07:17:14,482 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Listening on port: /home/model-server/tmp/.mms.sock.9000
algo-1-nvqf7_1  | 2020-11-30 07:17:14,482 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - [PID] 51
algo-1-nvqf7_1  | 2020-11-30 07:17:14,482 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - MMS worker started.
algo-1-nvqf7_1  | 2020-11-30 07:17:14,483 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Python runtime: 3.6.6
algo-1-nvqf7_1  | 2020-11-30 07:17:14,483 [INFO ] main com.amazonaws.ml.mms.wlm.ModelManager - Model model loaded.
algo-1-nvqf7_1  | 2020-11-30 07:17:14,487 [INFO ] main com.amazonaws.ml.mms.ModelServer - Initialize Inference server with: EpollServerSocketChannel.
algo-1-nvqf7_1  | 2020-11-30 07:17:14,496 [INFO ] W-9000-model com.amazonaws.ml.mms.wlm.WorkerThread - Connecting to: /home/model-server/tmp/.mms.sock.9000
algo-1-nvqf7_1  | 2020-11-30 07:17:14,544 [INFO ] main com.amazonaws.ml.mms.ModelServer - Inference API bind to: http://0.0.0.0:8080
algo-1-nvqf7_1  | 2020-11-30 07:17:14,545 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Connection accepted: /home/model-server/tmp/.mms.sock.9000.
algo-1-nvqf7_1  | Model server started.
algo-1-nvqf7_1  | 2020-11-30 07:17:14,547 [WARN ] pool-2-thread-1 com.amazonaws.ml.mms.metrics.MetricCollector - worker pid is not available yet.
algo-1-nvqf7_1  | 2020-11-30 07:17:14,962 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - PyTorch version 1.3.1 available.
algo-1-nvqf7_1  | 2020-11-30 07:17:15,314 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Lock 140580224398952 acquired on /root/.cache/torch/transformers/26bc1ad6c0ac742e9b52263248f6d0f00068293b33709fae12320c0e35ccfbbb.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084.lock
algo-1-nvqf7_1  | 2020-11-30 07:17:15,315 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt not found in cache or force_download set to True, downloading to /root/.cache/torch/transformers/tmpcln39mxo
algo-1-nvqf7_1  | 2020-11-30 07:17:15,344 [WARN ] W-9000-model-stderr com.amazonaws.ml.mms.wlm.WorkerLifeCycle -
algo-1-nvqf7_1  | 2020-11-30 07:17:15,349 [WARN ] W-9000-model-stderr com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Downloading:   0%|          | 0.00/232k [00:00<?, ?B/s]
algo-1-nvqf7_1  | 2020-11-30 07:17:15,349 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - storing https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt in cache at /root/.cache/torch/transformers/26bc1ad6c0ac742e9b52263248f6d0f00068293b33709fae12320c0e35ccfbbb.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084
algo-1-nvqf7_1  | 2020-11-30 07:17:15,349 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - creating metadata file for /root/.cache/torch/transformers/26bc1ad6c0ac742e9b52263248f6d0f00068293b33709fae12320c0e35ccfbbb.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084
algo-1-nvqf7_1  | 2020-11-30 07:17:15,349 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Lock 140580224398952 released on /root/.cache/torch/transformers/26bc1ad6c0ac742e9b52263248f6d0f00068293b33709fae12320c0e35ccfbbb.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084.lock
algo-1-nvqf7_1  | 2020-11-30 07:17:15,350 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - loading file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt from cache at /root/.cache/torch/transformers/26bc1ad6c0ac742e9b52263248f6d0f00068293b33709fae12320c0e35ccfbbb.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084
algo-1-nvqf7_1  | 2020-11-30 07:17:15,378 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Created tokenizer
algo-1-nvqf7_1  | 2020-11-30 07:17:15,378 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Elastic Inference enabled: True
algo-1-nvqf7_1  | 2020-11-30 07:17:15,378 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - inside model fn
algo-1-nvqf7_1  | 2020-11-30 07:17:15,379 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - /.sagemaker/mms/models/model
algo-1-nvqf7_1  | 2020-11-30 07:17:15,379 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - /.sagemaker/mms/models/model/model.pt
algo-1-nvqf7_1  | 2020-11-30 07:17:15,379 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - ['model.pt', 'model.tar.gz', 'code', 'model_tn_best.pth', 'MAR-INF']
algo-1-nvqf7_1  | 2020-11-30 07:17:15,379 [INFO ] W-9000-model-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Loading torch script
algo-1-nvqf7_1  | 2020-11-30 07:17:15,392 [INFO ] epollEventLoopGroup-4-1 com.amazonaws.ml.mms.wlm.WorkerThread - 9000-44f1cd64 Worker disconnected. WORKER_STARTED
algo-1-nvqf7_1  | 2020-11-30 07:17:15,392 [WARN ] W-9000-model com.amazonaws.ml.mms.wlm.BatchAggregator - Load model failed: model, error: Worker died.
algo-1-nvqf7_1  | 2020-11-30 07:17:15,393 [INFO ] W-9000-model com.amazonaws.ml.mms.wlm.WorkerThread - Retry worker: 9000-44f1cd64 in 1 seconds.
algo-1-nvqf7_1  | 2020-11-30 07:17:16,065 [INFO ] W-9000-model ACCESS_LOG - /172.18.0.1:45110 "GET /ping HTTP/1.1" 200 8


title: PyTorch: Shuffle DataLoader

body:
There are several scenarios that make me confused about shuffling the data loader, which are as follows.
I set the “shuffle” parameter to False on both train_loader and valid_loader. then the results I get are as follows
Epoch 1/4    loss=0.8802     val_loss=0.8202     train_acc=0.63      val_acc=0.63
Epoch 2/4    loss=0.6993     val_loss=0.6500     train_acc=0.66      val_acc=0.72
Epoch 3/4    loss=0.5363     val_loss=0.5385     train_acc=0.76      val_acc=0.80
Epoch 4/4    loss=0.4055     val_loss=0.5130     train_acc=0.85      val_acc=0.81

I set the “shuffle” parameter to True on train_loader and False to valid_loader. then the results I get are as follows
Epoch 1/4    loss=0.8928     val_loss=0.8284     train_acc=0.63      val_acc=0.63
Epoch 2/4    loss=0.7308     val_loss=0.6263     train_acc=0.61      val_acc=0.73
Epoch 3/4    loss=0.5594     val_loss=0.5046     train_acc=0.54      val_acc=0.81
Epoch 4/4    loss=0.4304     val_loss=0.4525     train_acc=0.49      val_acc=0.82

Based on that result, my training accuracy has a worse performance when I shuffle train_loader.
And this is a snippet of my code.
for epoch in range(n_epochs):
    model.train()
    avg_loss = 0.
    train_preds = np.zeros((len(train_X),len(le.classes_)))

    for i, (x_batch, y_batch) in enumerate(train_loader):
        y_pred = model(x_batch)
        loss = loss_fn(y_pred, y_batch)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        avg_loss += loss.item() / len(train_loader)
        train_preds[i * batch_size:(i+1) * batch_size] = F.softmax(y_pred).cpu().detach().numpy()

    train_accuracy = sum(train_preds.argmax(axis=1) == y_train)/len(y_train)

    model.eval()
    avg_val_loss = 0.
    val_preds = np.zeros((len(x_cv),len(le.classes_)))

    for i, (x_batch, y_batch) in enumerate(valid_loader):
        y_pred = model(x_batch).detach()
        avg_val_loss += loss_fn(y_pred, y_batch).item() / len(valid_loader)
        val_preds[i * batch_size:(i+1) * batch_size] =F.softmax(y_pred).cpu().numpy()
    val_accuracy = sum(val_preds.argmax(axis=1)==y_test)/len(y_test)

Did I make a mistake calculating the training accuracy? Thanks in advance


title: calculating mIoU for multi-class semantic segmentation

body:
I want to calculate mIoU for multi-class semantic segmentation. i have trained u-net with mIoU metric and  I want to test this model with different dataset but in this stage i have to ignore some pixels value but i can't run it with my mIoU code and i find the shown below. Bu i didn't understand the its input/output.
for preds I give (1, NumberofClass, w,h)
for labes I give (1, w, h)
for C I give number of my class and I define ignore as a list(ignore = [0])
def iou(preds, labels, C, EMPTY=1., ignore=None, per_image=False):
"""
Array of IoU for each (non ignored) class
"""
if not per_image:
    preds, labels = (preds,), (labels,)
ious = []
for pred, label in zip(preds, labels):
    iou = []
    for i in range(C):
        if i != ignore:
            intersection = ((label == i) & (pred == i)).sum()
            union = ((label == i) | ((pred == i) & (label != ignore))).sum()
            if not union:
                iou.append(EMPTY)
            else:
                iou.append(float(intersection) / float(union))
    ious.append(iou)
ious = [mean(iou) for iou in zip(*ious)]  # mean across images if per_image
return 100 * np.array(ious)

def mean(l, ignore_nan=False, empty=0):
"""
nanmean compatible with generators.
"""
l = iter(l)
if ignore_nan:
    l = ifilterfalse(isnan, l)
try:
    n = 1
    acc = next(l)
except StopIteration:
    if empty == 'raise':
        raise ValueError('Empty mean')
    return empty
for n, v in enumerate(l, 2):
    acc += v
if n == 1:
    return acc
return acc / n

When i run the code the my output is looking that. I also notice my intersection always 0.
Can anyone got better solution. It will be very helpful.

loss =  [100.   0.   0. 100.   0.   0.   0.   0.   0. 100.   0.   0. 100.   0.
0.]


title: PyTorch one of the variables needed for gradient computation has been modified by an inplace operation

body:
I'm doing a policy gradient method in PyTorch. I wanted to move the network update into the loop and it stopped working. I'm still a PyTorch newbie so sorry if the explanation is obvious.
Here is the original code that works:
self.policy.optimizer.zero_grad()
G = T.tensor(G, dtype=T.float).to(self.policy.device)

loss = 0
for g, logprob in zip(G, self.action_memory):
    loss += -g * logprob

loss.backward()
self.policy.optimizer.step()

And after the change:
G = T.tensor(G, dtype=T.float).to(self.policy.device)

loss = 0
for g, logprob in zip(G, self.action_memory):
    loss = -g * logprob
    self.policy.optimizer.zero_grad()

    loss.backward()
    self.policy.optimizer.step()

I get the error:
File "g:\VScode_projects\pytorch_shenanigans\policy_gradient.py", line 86, in learn
    loss.backward()
  File "G:\Anaconda3\envs\pytorch_env\lib\site-packages\torch\tensor.py", line 185, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "G:\Anaconda3\envs\pytorch_env\lib\site-packages\torch\autograd\__init__.py", line 127, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [128, 4]], which is output 0 of TBackward, is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

I read that this RuntimeError often has to do with having to clone something, because we're using the same tensor to compute itself but I can't make heads of tails of what is wrong in my case.


title: Pytorch: backwards differentiation on a function of a Hessian: "

body:
I just finished debugging a crazy error in my pytorch code, so I thought I'd share my experience.

As a minimal example, here's some code that crashes:

import torch
ts = torch.tensor
t = ts([i+.5 for i in range(5)], requires_grad=True) #some tensor
f = torch.sum(t**3) #some function of t
[lg] = torch.autograd.grad(f,t,create_graph=True,retain_graph=True) #the gradient of f wrt t
[lgg] = torch.autograd.grad(lg[1],t,create_graph=True,retain_graph=True) #one row of the Hessian of f wrt t
lgg2 = lgg.contiguous().view(-1)
lgg3 = torch.zeros(6,6,requires_grad=True) #if we were calculating the full Hessian, it would be a matrix...
lgg3[1,1:].add_(lgg2.type_as(lgg3)) #...and we'd fill it in row by row
l2 = torch.sum(lgg3) #a loss that's a function of the Hessian
l2.backward() #we want the gradient of that loss
print(t.grad)
print(lgg3.requires_grad)

This gives the error:

RuntimeError: leaf variable has been moved into the graph interior

How to fix it? I will give you the answer...


title: Advantage of TensorWatch over TensorBoardX?

body:
I work with PyTorch.

I was wondering what's the advantage of TensorWatch over TensorBoardX ?

From what I understand, the only advantage is the ease of use of TensorWatch with notebooks.

Since I don't really work with notebooks, I was wondering if there is any other advantage.


title: Should the embedding layer be changed during training a neural network?

body:
I'm a new one for the field of deep learning and Pytorch.

Recently when I learn one of the pytorch tutorial example for NER task, I found the embedding of nn.Embedding changed during the training.

So my question is should the embedding be changed during training the network?

And if I want to load a pre-trained embedding (for example, trained word2vec embedding) into a PyTorch embedding layer, should the pre-trained embedding also be changed during the train process?

Or how could I prevent updating the embeddings?

Thank you.


title: Using pytorch built in derivative as part of custom autograd function

body:
I'm looking to implement a custom autograd. A function where the backward pass is a mix of a custom function and the derivative of a function which torch should be able to find by itself.

For a simple example, say I wanted to create a function for y = x * exp(x)

def custom_function(torch.autograd.Function):
    @staticmethod
    def forward(ctx, x):
       ctx.save_for_backward(x)
       return x * exp(x)

    @staticmethod
    def backward(ctx, grad_output):
       x = ctx.saved_tensors[0]

       custom_derivative = x * [d/dx torch.exp(x)] + torch.exp(x)
       return grad_output * custom_derivative

How would one call the derivative of a known pytorch function in the backward pass?


title: How to concatenate the lists within a tensor to a single list?

body:
suppose that I have the following tensor:

>> outputs
tensor([[-0.1260,  0.0463, -0.3362,  ...,  0.1089, -0.2454,  0.0140],
        [ 0.5050, -0.0750, -0.1639,  ..., -0.0020, -0.0521, -0.3224],
        [-0.5311,  0.4526,  0.0079,  ..., -0.0654, -0.1255, -0.0012],
        [ 0.0728, -0.1219,  0.0905,  ...,  0.1354,  0.2730, -0.1186],
        [-0.0680, -0.5570,  0.0295,  ..., -0.2411, -0.1690,  0.0331]],
       grad_fn=<SliceBackward>)
>> outputs.shape
Out[20]: torch.Size([5, 1024])

How can I concatenate the 5 lists that are contained in the tensor outputs and make it into a single list?

Thank you,


title: Differentially generate sentences with Huggingface Library for adversarial training (GANs)

body:
I have the following goal, which I have been trying to achieve with the Huggingface Library but I encountered some roadblocks.
The Problem:
I want to generate sentences in a differentiable way at training time. Why am I doing this? I want to apply a discriminator to this output to generate sentences with certain properties, which are "enforced" by the discriminator. These sentences will also be conditioned on a input sentence, so I need a Encoder Decoder Model.
To get around the non differentiability of argmax, I simply take the softmax output of the decoder and multiply it with my embedding matrix. Then I am taking this embedded input and feed it into a transformer discriminator, which simply classifies the input as original/fake. Then I backpropagate through the encoder decoder. Just as one would do it with a normal GAN.
So far I have tried to use the EncoderDecoderModel from Huggingface. This class has a method named generate, which generates sentences in a non differentiable way (greedy or beam-search). So I dug through the source code and tried to build my own differentiable generate method. I didn't get it to work though.
Questions:

Is there a reasonably easy way to do this with the Huggingface Library, as I really want to use the pretrained models and everything else that comes with it?
Is there a way to invoke the forward method of the decoder and only generate one new token, not the whole sequence again?

Thanks for your help, I would really appreciate it, I have been stuck on this for quiet a while now.


title: Beginner: loss.backwards() doesn't work if input had no impact on output

body:
I am a beginner with pytorch and have the following problem.
I want to optimize a complex problem that uses torch.min() multiple times, but even with a simple toy example I can't get it to work. My code has these lines:
output = net(input)
loss = g(output)
loss = torch.min(loss, torch.ones(1))
loss.backward()

To minimize this loss the net should ascertain that the output minimizes g:R^2->R. Here g is a very simple function that has a zero at {-1,-2} and if I delete the third line the neural network finds the solution just fine. However with the posted code and bad initial weights the minimum is attained by 1. This leads to the backward() function not updating the weights and no learning happening at all over arbitrary many epochs.
Is there a way to detect/fix this behaviour in more complex cases? My task uses the minimum function multiple times and in more complex ways, so I think it would be quite hard to track every single one and ascertain that learning actually takes place.
Edit: If I start the optimizer multiple times it rarely happens that the optimization works just fine (e.g. converges to {-1,-2}). My interpretation is that in those cases the inital weights randomly lead to the minimum beeing attained in the first component.


title: PyTorch value thresholding and zeroing all other values

body:
I have 2D tensor in PyTorch, representing model confidences. I want:

if 2nd value in row is greater or equal to threshold, all other values should be changed to 0
else values should not change

The simple approach would be:

iterate through rows
check 2nd value
if value is greater or equal, create row of zeroes, change 2nd value to the 2nd value from row and replace row
else don't do anything

It is inefficient, however. Is there a vectorized / tensorized way to do this?


title: PyTorch: How to check if some weights are not changed during training?

body:
How can I check if some weights are not changed during training in PyTorch?
As I understand one option can be just dump model weights at some epochs and check if they are changed iterating over weights, but maybe there is some simpler way?


title: Using Captum with Pytorch Lightning?

body:
So I tried to use Captum with PyTorch Lightning. I am having issues when passing the Module to Captum, since it seems to do weird reshaping of the tensors.
For example in the below minimal example, the lightning code works easy and well.
But when I use IntegratedGradient with "n_step>=1" I get an issue.
The code of the LighningModule is not that important I would say, I wonder more at the code line at the very bottom.
Does anyone know how to work around this?
from captum.attr import IntegratedGradients
from torch import nn, optim, rand, sum as tsum, reshape, device
import torch.nn.functional as F
from pytorch_lightning import seed_everything, LightningModule, Trainer
from torch.utils.data import DataLoader, Dataset

SAMPLE_DIM = 3

class CustomDataset(Dataset):
    def __init__(self, samples=42):
        self.dataset = rand(samples, SAMPLE_DIM).cuda().float() * 2 - 1

    def __getitem__(self, index):
        return (self.dataset[index], (tsum(self.dataset[index]) > 0).cuda().float())

    def __len__(self):
        return self.dataset.size()[0]

class OurModel(LightningModule):
    def __init__(self):
        super(OurModel, self).__init__()
        # Network layers
        self.linear = nn.Linear(SAMPLE_DIM, 2048)
        self.linear2 = nn.Linear(2048, 1)
        self.output = nn.Sigmoid()
        # Hyper-parameters, that we will auto-tune using lightning!
        self.lr = 0.001
        self.batch_size = 512

    def forward(self, x):
        x = self.linear(x)
        x = self.linear2(x)
        output = self.output(x)
        return reshape(output, (-1,))

    def configure_optimizers(self):
        return optim.Adam(self.parameters(), lr=self.lr)

    def train_dataloader(self):
        loader = DataLoader(CustomDataset(samples=1000), batch_size=self.batch_size, shuffle=True)
        return loader

    def training_step(self, batch, batch_nb):
        x, y = batch
        loss = F.binary_cross_entropy(self(x), y)
        return {'loss': loss, 'log': {'train_loss': loss}}

if __name__ == '__main__':
    seed_everything(42)
    device = device("cuda")
    model = OurModel().to(device)
    trainer = Trainer(max_epochs=2, min_epochs=1, auto_lr_find=False,
                      progress_bar_refresh_rate=10)
    trainer.fit(model)
    # ok Now the Problem
    test_input = CustomDataset(samples=1).__getitem__(0)[0].requires_grad_()
    ig = IntegratedGradients(model)
    attr, delta = ig.attribute(test_input, target=1, return_convergence_delta=True)


title: Give randomCrop augmentation and loss become explode

body:
I'm trying to test the different kind of augmentation,
but when I gave option with RandomCrop it gives loss value NaN or infinity.
Here is my random augmentation optims
def mapper2(dataset_dict):
    dataset_dict = copy.deepcopy(dataset_dict)  # it will be modified by code below
    image = utils.read_image(dataset_dict["file_name"], format="BGR")
    transform_list = [
                     T.RandomFlip(prob=0.5, horizontal=True, vertical=False),
                     T.ResizeShortestEdge(short_edge_length=(640, 672, 704, 736, 768, 800), max_size=1333, sample_style='choice')

                     ,T.RandomCrop('relative_range', (0.9, 0.9))
                      ]
    image, transforms = T.apply_transform_gens(transform_list, image)
    dataset_dict["image"] = torch.as_tensor(image.transpose(2, 0, 1).astype("float32"))

    annos = [
        utils.transform_instance_annotations(obj, transforms, image.shape[:2])
        for obj in dataset_dict.pop("annotations")
        if obj.get("iscrowd", 0) == 0
    ]

    instances = utils.annotations_to_instances(annos, image.shape[:2])
    dataset_dict["instances"] = instances
    return dataset_dict

Will this code apply augmentation randomly to any input batch images

And why it explode the loss when I gave RandomCrop ?

FloatingPointError: Predicted boxes or scores contain Inf/NaN. Training has diverged.


title: Why are the MNIST images 1x28x28 tensors?

body:
I made the MNIST images which are 28x28 pixel images into tensors with
dataset = MNIST(root='data/', train=True, transform=transforms.ToTensor())

and when I run
img_tensor, label = dataset[0]
print(img_tensor.shape, label)

It says the shape is torch.Size([1, 28, 28]).
Why is it a 1x28x28? What does the first dimension mean? and what is the point of a 1x28x28 opposed to 28x28?


title: How to use an embedding layer as a linear layer in PyTorch?

body:
I'm currently working on a personal reimplementation of the Transformer paper and had a question.
On page 5 in section "3.4 Embeddings and Softmax," it states:

In our model, we share the same weight matrix between the two embedding layers and the pre-softmax linear transformation.

I've currently implemented my model to use just one embedding layer for both source and target tensors, but I'm wondering if there would be a way that I could use the weights of the embedding layer as a linear layer. What I've currently done is something like:
output = previous_layer(previous_input)
final_output = torch.matmul(output, embedding_layer.embedding.weight.transpose(1, 0))

I've transposed the weight matrix before matrix multiplication because it's of shape (vocab_size, embedding_dim) and the shape of output is (batch_size, seq_len, embedding_dim). Is this the proper way to use an embedding layer as a linear layer? If not, I'd like some tips on what I should be doing.
Thanks.
Edit
Specific line for weight sharing from link in answer.


title: why can't I find libc++.1.dylib on imac?

body:
I want to use pytorch torchtext but failed. Mac system says:

library not loaded: @rpath/libc++.1.dylib

I fond a lot of solutions about this problem which told me I needed to copy libc++ file to my virtual environment. But suddenly I find I don't have this file in usr/lib, and beyond that, I don't have a lot of files compared with my original mac.
I installed xcode and llvm but still can't find the file.
which software should I install to make the file libc++.1.dylib exist?


title: BERT regression implementation is not training

body:
I'm trying to implement scibert model for a regression task. I trained the model and predict, however, model predicts 0 for each instance.
In a nutshell, task was predicting the positivity/negativity of a scientific article between -1 and 1.
Obviously the problem turns out from train (model does not learn) because validation loss does not decrease as expected. I will share the code of my model and look forward to hearing any suggestions. Please, let me know if you need to see other parts of the code.
Thanks!
class CustomSciBERTModel(nn.Module):
    def __init__(self):
      super(CustomSciBERTModel, self).__init__()
      self.scibert = AutoModel.from_pretrained('allenai/scibert_scivocab_uncased')
      self.num_labels = 1
      self.linear_layer = nn.Linear(768, 1)

    def forward(self, ids, mask,labels):
      output = self.scibert(
            input_ids=ids,
            attention_mask=mask)

      logits = self.linear_layer(output[1])

      loss = None
      # while training return loss and while validation/ testing resturn logits
      if labels is not None:
        loss_fct = nn.MSELoss()
        #loss_fct = nn.CrossEntropyLoss() *************
        loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))
        return loss
      else:
        return logits

TRAINING
import random
import gc
# Set the seed value all over the place to make this reproducible.
seed_val = 42

random.seed(seed_val)
np.random.seed(seed_val)
torch.manual_seed(seed_val)
torch.cuda.manual_seed_all(seed_val)

# Store the average loss after each epoch so we can plot them.
loss_values = []

# store predictions on validation data
valid_preds = []

# For each epoch...
for epoch_i in range(0, epochs):

    # ========================================
    #               Training
    # ========================================

    # Perform one full pass over the training set.

    print("")
    print('======== Epoch {:} / {:} ========'.format(epoch_i + 1, epochs))
    print('Training...')

    # Measure how long the training epoch takes.
    t0 = time.time()

    # Reset the total loss for this epoch.
    total_loss = 0

    # Put the model into training mode. Don't be mislead--the call to
    model.train()

    # For each batch of training data...
    for step, batch in enumerate(train_dataloader):

        # Progress update every 40 batches.
        if step % 40 == 0 and not step == 0:
            # Calculate elapsed time in minutes.
            elapsed = format_time(time.time() - t0)

            # Report progress.
            print('  Batch {:>5,}  of  {:>5,}.    Elapsed: {:}.'.format(step, len(train_dataloader), elapsed))

        # Unpack this training batch from our dataloader.
        #
        # As we unpack the batch, we'll also copy each tensor to the GPU using the
        # `to` method.
        #
        # `batch` contains three pytorch tensors:
        #   [0]: input ids
        #   [1]: attention masks
        #   [2]: labels
        b_input_ids = batch[0].to(device)
        b_input_mask = batch[1].to(device)
        b_labels = batch[2].to(device)

        #modified******
        b_labels = b_labels.to(torch.float32)

        # backward pass. PyTorch doesn't do this automatically because
        model.zero_grad()

        # Forward pass
        outputs = model(ids = b_input_ids, mask = b_input_mask, labels = b_labels)

        # forward pass gives loss as output
        loss = outputs

        # accumulate loss
        total_loss += loss.item()

        # backward pass to calculate the gradients.
        loss.backward()

        # Clip the norm of the gradients to 1.0. to avoid exploding gradients
        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)

        # Update parameters
        optimizer.step()

        # Update the learning rate.
        scheduler.step()

    # Calculate the average loss over the training data.
    avg_train_loss = total_loss / len(train_dataloader)

    # Store the loss value for plotting the learning curve.
    loss_values.append(avg_train_loss)

    print("")
    print("  Average training loss: {0:.2f}".format(avg_train_loss))
    print("  Training epcoh took: {:}".format(format_time(time.time() - t0)))

    # ========================================
    #               Validation
    # ========================================

    print("")
    print("Running Validation...")

    t0 = time.time()

    # evaluation mode.
    model.eval()

    # evaluation loss and accuracy
    eval_loss, eval_accuracy = 0, 0
    nb_eval_steps, nb_eval_examples = 0, 0

    # Evaluate data for one epoch
    for batch in validation_dataloader:

        # Add batch to GPU
        batch = tuple(t.to(device) for t in batch)

        # Unpack the inputs from our dataloader
        b_input_ids, b_input_mask, b_labels = batch

        # Telling the model not to compute or store gradients, saving memory and
        # speeding up validation
        with torch.no_grad():

            # It gives logits when labels are not given to model
            outputs = model(b_input_ids,
                            b_input_mask, labels = None)

        # Get the "logits" output by the model. The "logits" are the output
        logits = outputs

        # Move logits and labels to CPU
        logits = logits.detach().cpu().numpy()
        label_ids = b_labels.to('cpu').numpy()

        valid_preds.extend(np.argmax(logits, axis=1).flatten())
        # Calculate the accuracy for this batch of test sentences.
        tmp_eval_accuracy = accuracy(logits, label_ids)

        # Accumulate the total accuracy.
        eval_accuracy += tmp_eval_accuracy

        # Track the number of batches
        nb_eval_steps += 1

    # Report the final accuracy for this validation run.
    print("  Accuracy: {0:.2f}".format(eval_accuracy/nb_eval_steps))
    print("  Validation took: {:}".format(format_time(time.time() - t0)))
    gc.collect()
    torch.cuda.empty_cache()

print("")
print("Training complete!")


I have recently upgraded the PyTorch version from 1.4 to 1.8. Now the mean function isn't working.
class loss_e(nn.Module):

def __init__(self, patch_size, mean_val):
    super(L_exp, self).__init__()
    # print(1)
    self.pool = nn.AvgPool2d(patch_size)
    self.mean_val = mean_val

def forward(self, x):
    # b, c, h, w = x.shape
    print(x)
    x = torch.mean(x, 1, True) # I got the error in this line
    mean = self.pool(x)

    d = torch.mean(torch.pow(mean - torch.FloatTensor([self.mean_val]).cuda(), 2))
    return d

Here's the error I got.
File "/home/nemo/Desktop/AICE/Myloss.py", line 86, in forward
    x = torch.mean(x, 1, True)
TypeError: mean() received an invalid combination of arguments - got (tuple, int, bool), but expected one of:
 * (Tensor input, *, torch.dtype dtype)
      didn't match because some of the arguments have invalid types: (!tuple!, !int!, !bool!)
 * (Tensor input, tuple of names dim, bool keepdim, *, torch.dtype dtype, Tensor out)
 * (Tensor input, tuple of ints dim, bool keepdim, *, torch.dtype dtype, Tensor out)

I have printed the x t make sure If it's a tensor or not.
tensor([[[[0.9581, 0.9584, 0.9575,  ..., 0.6204, 0.6114, 0.6179],
          [0.9608, 0.9616, 0.9611,  ..., 0.6609, 0.6543, 0.6619],
          [0.9606, 0.9612, 0.9606,  ..., 0.7257, 0.6903, 0.6772],
          ...,

Then I have printed x.shape and got this.
torch.Size([8, 3, 256, 256])

How this tensor is converted to a tuple?
Edit-
I have found that function invert causing that issue.
I have fed a inv_imag and img to the same model.
inv_image = torchvision.transforms.functional.invert(img)

There's no issue when I fed only the img.


title: How to install the module pytorch_lightning.metrics in Raspberry pi3

body:
I am trying to execute a python file which has pytorch with lightning and torchvision modules. But after I downloaded and successfully installed whl file of pytorch in pi3 I am getting same error again and again.
The error is
 ModuleNotFoundError: No module named 'pytorch_lightning.metrics'

Help would be highly appreciated as I am stuck for more than 3 days.
I have installed the modules using pip.


title: How to Log Metrics (eg. Validation Loss) To TensorBoard when using Pytorch Lightning?

body:
I am using Pytorch Lightning to train my models (on GPU devices, using DDP) and TensorBoard is the default logger used by Lightning.
My code is setup to log the training and validation loss on each training and validation step respectively.
class MyLightningModel(pl.LightningModule):

    def training_step(self, batch):
        x, labels = batch
        out = self(x)
        loss = F.mse_loss(out, labels)
        self.log("train_loss", loss)
        return loss

    def validation_step(self, batch):
        x, labels = batch
        out = self(x)
        loss = F.mse_loss(out, labels)
        self.log("val_loss", loss)
        return loss

TensorBoard correctly plots both the train_loss and val_loss charts in the SCALERS tab. However, in the HPARAMS tab, on the left side bar, only hp_metric is visible under Metrics.

However, in the HPARAMS tab, on the left side bar, only hp_metric is visible under Metrics.

How can we add train_loss and val_loss to the Metrics section? This way, we will be able to use val_loss in the PARALLEL COORDINATES VIEW instead of hp_metric.
Image showing hp_metric and no val_loss:

Using Pytorch 1.8.1, Pytorch Lightning 1.2.6, TensorBoard 2.4.1


title: Why/how can model.forward() succeed both on input being mini-batch vs single item?

body:
Why and how does this work?

When I run the forward phase on input

being mini-batch tensor
or alternatively being a single input item

model.__call__() (which AFAIK is calling forward() ) swallows that and spills out adequate output (i.e. a tensor of mini-batch of estimates or a single item of estimate)

Adopting testcode from the Pytorch NN example shows what I mean, but I don't get it.

I would expect it to create problems and me forced to transform the single item input into a mini-batch of size 1( reshape (1,xxx)) or likewise, like I did in the code below.

( I did variations of the test to be sure it is e.g. not depending on execution order )

# -*- coding: utf-8 -*-
import torch

# N is batch size; D_in is input dimension;
# H is hidden dimension; D_out is output dimension.
#N, D_in, H, D_out = 64, 1000, 100, 10
N, D_in, H, D_out = 64, 10, 4, 3

# Create random Tensors to hold inputs and outputs
x = torch.randn(N, D_in)
y = torch.randn(N, D_out)

# Use the nn package to define our model as a sequence of layers. nn.Sequential
# is a Module which contains other Modules, and applies them in sequence to
# produce its output. Each Linear Module computes output from input using a
# linear function, and holds internal Tensors for its weight and bias.
model = torch.nn.Sequential(
    torch.nn.Linear(D_in, H),
    torch.nn.ReLU(),
    torch.nn.Linear(H, D_out),
)

# The nn package also contains definitions of popular loss functions; in this
# case we will use Mean Squared Error (MSE) as our loss function.
loss_fn = torch.nn.MSELoss(reduction='sum')

learning_rate = 1e-4
for t in range(1):
    # Forward pass: compute predicted y by passing x to the model. Module objects
    # override the __call__ operator so you can call them like functions. When
    # doing so you pass a Tensor of input data to the Module and it produces
    # a Tensor of output data.
    model.eval()
    print ("###########")
    print ("x[0]",x[0])
    print ("x[0].size()", x[0].size())
    y_1pred = model(x[0])
    print ("y_1pred.size()", y_1pred.size())
    print (y_1pred)

    model.eval()
    print ("###########")
    print ("x.size()", x.size())
    y_pred = model(x)
    print ("y_pred.size()", y_pred.size())
    print ("y_pred[0]", y_pred[0])

    print ("###########")
    model.eval()
    input_item = x[0]
    batch_len1_shape = torch.Size([1,*(input_item.size())])
    batch_len1 = input_item.reshape(batch_len1_shape)
    y_pred_batch_len1 = model(batch_len1)

    print ("input_item",input_item)
    print ("input_item.size()", input_item.size())
    print ("y_pred_batch_len1.size()", y_pred_batch_len1.size())
    print (y_1pred)

    raise Exception

This is the output it generates:

###########
x[0] tensor([-1.3901, -0.2659,  0.4352, -0.6890,  0.1098, -0.3124,  0.6419,  1.1004,
        -0.7910, -0.5389])
x[0].size() torch.Size([10])
y_1pred.size() torch.Size([3])
tensor([-0.5366, -0.4826,  0.0538], grad_fn=<AddBackward0>)
###########
x.size() torch.Size([64, 10])
y_pred.size() torch.Size([64, 3])
y_pred[0] tensor([-0.5366, -0.4826,  0.0538], grad_fn=<SelectBackward>)
###########
input_item tensor([-1.3901, -0.2659,  0.4352, -0.6890,  0.1098, -0.3124,  0.6419,  1.1004,
        -0.7910, -0.5389])
input_item.size() torch.Size([10])
y_pred_batch_len1.size() torch.Size([1, 3])
tensor([-0.5366, -0.4826,  0.0538], grad_fn=<AddBackward0>)


title: Tensor reduction based off index vector

body: As an example, I have 2 tensors: A = [1;2;3;4;5;6;7] and B = [2;3;2]. The idea is that I want to reduce A based off B - such that B's values represent how to sum A's values- such that B = [2;3;2] means the reduced A shall be the sum of the first 2 values, next 3, and last 2: A' = [(1+2);(3+4+5);(6+7)]. It is apparent that the sum of B shall always be equal to the length of A. I'm trying to do this as efficiently as possible - preferably specific functions or matrix operations contained within pytorch/python. Thanks!


title: How to Build Simple LSTM Model for Anomally Detection

body:
I want to create a LSTM model in PyTorch that will be used for anomally detection, but I'm having trouble understanding the details in doing so.

Note, my training-data consists of sets with 16 features in 80 time-steps. Here is what I've written for the model below:

class AutoEncoder(torch.nn.Module):
    def __init__(self, input_dim, hidden_dim, layer_dim):
        super(AutoEncoder, self).__init__()
        self.hidden_dim = hidden_dim
        self.layer_dim = layer_dim

        self.lstm = nn.LSTM(input_dim, hidden_dim, layer_dim, batch_first=True)
        self.fc1 = torch.nn.Linear(hidden_dim, hidden_dim)
        self.fc2 = torch.nn.Linear(hidden_dim, input_dim)
    def forward(self, x):
        h0 = torch.zeros(self.layer_dim, x.size(0), self.hidden_dim).requires_grad_()
        c0 = torch.zeros(self.layer_dim, x.size(0), self.hidden_dim).requires_grad_()

        out, (hn, cn) = self.lstm(x, (h0.detach(), c0.detach()))
        out = self.fc1(out[:, -1, :])
        out = self.fc2(out)
        return out

input_dim = 16
hidden_dim = 8
layer_dim = 2
model = AutoEncoder(input_dim, hidden_dim, layer_dim)

I don't think I've built the model correctly. How does it know that I'm feeding it 80 time-steps of data? How will the auto-encoder reconstruct those 80 time-steps of data?

I'm having a hard time understanding the material online. What would the final layer have to be?


title: How to perform self convolution (convolution with itself) in PyTorch

body:
PyTorch offers a functional 2D convolution operation torch.nn.functional.conv2d which may take an input and a weight parameter (https://pytorch.org/docs/stable/nn.html#conv2d).

I want to perform a self convolution, meaning that given a tensor X. I want to calculate:

conv2d(X, X)

However the shape of the filter is expected to be of the form [out_ch, in_channel/groups, kernel_size, kernel_size].

Let's say X is of the form [32, 3, 16, 16,] I have tried to permute X to try to match the expected form of the weight parameter, such as:

W = X.permute(1, 0, 2, 3) # 3, 32, 16, 16
conv2d(X, W)

But that does not work since 32 != in_channel/groups == 3 / 1 == 3

The complete operation would essentially look like this (to see the dimensions requirements):

X + conv2d(X, X)

Is it possible to do this with PyTorch?

Other packages such as SciPy offers a convolve2d operation (https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.convolve2d.html) but that only supports 2D arrays, and appears to do what I'm trying to do above, but without support for batches and channels.