I have two images, one model output and its correponding ground truth. I want to compare the two and calculate precision of my model. but i am getting an error.
Error:
Traceback (most recent call last):
File "storagelocation.py", line 58, in
numpydata_in = numpydata_in1.reshape(numpydata_in1.shape[1:])
ValueError: cannot reshape array of size 2359296 into shape (1024,3)
My code is below:
img50 = Image.open('0050.jpg')
numpydata_in1 = np.asarray(img50)
print(numpydata_in1.shape) #output= [3,768,1024]
numpydata_in = numpydata_in1.reshape(numpydata_in1.shape[1:])
numpydata_in = transforms.ToTensor()(numpydata_in)
imgT50 = Image.open('0050_1.png')
numpydata_out1 = np.asarray(imgT50)
print(numpydata_out1.shape) #output [1, 768, 1024]
numpydata_out = transforms.ToTensor()(numpydata_out1)
prec = precision(numpydata_in, numpydata_out)
print("precision is",prec)

The output of the image.shape shows 3 as one dimenion length in ground truth image. But they mention in the dataset detail it is a binary image.


title: How does a Pytorch neural network load dataset into GPU

body:
When loading a dataset into the GPU for training, would a Pytorch NN load the entire dataset or just the batch?
I have a 33GB dataset that fits comfortably on my normal RAM (64GB) but i only have a 16GB of GPU RAM (T4). As long as Pytorch only loads one batch at a time into the GPU, that should work fine without any memory problems?


title: Combining multiple models trained in different parts of the dataset in PyTorch

body:
In PyTorch, is it theoretically possible to 'merge' multiple models together into one model - effectively combining all the data learnt so far? The models are exactly identical, however, are trained with different parts of the training data.
If so, would it be possible to split a dataset into equal parts and distribute training between many computers in a similar manner to folding@home? Would the new model be as good as if it was not distributed?


title: In pytorch, can we conditionally train a model based on the label of a sample?

body:
For example, we have labels 1,2 and 3 in a dataset. If a sample is associated with label 3 then we don't train the model using this sample. Otherwise, we use this sample to calculate the loss along with the gradient. I wonder how we can do this during the training phase and whether the following code is correct or not.
for i in range(len(outputs)):
    if targets[i].item() != 3:
       targets_.append(targets[i].item())
       outputs_.append(outputs[i].cpu().detach().numpy())

loss = loss_fn(torch.Tensor(outputs_).requires_grad_().to(DEVICE, dtype=torch.float),
               torch.Tensor(targets_).to(DEVICE, dtype=torch.long))

loss.backward()
optimizer.step()


title: Python ValueError: Inserting a list of values using df.loc

body:
I have a dataframe df with paths of images. I am performing object detection and want to save the names of the detected objects in a new column (named objects) of the dataframe.
However the code works fine when only 1 object is detected, when there is a list of multiple objects the code gives an error.
ValueError: cannot set using a multi-index selection indexer with a different length than the value
I think there is some problem with my way of inserting values in the dataframe. I have actually created an empty column first and then using loop inserting vaues in the objects column.
Code
train_image_paths = 'data/train_images/' + df['image']    # train_images is the folder containing images and
                                                          #  df['image'] contains all image paths

df['objects'] = ''     # Creating an empty column for storing detected objects

for idx, img in enumerate(train_image_paths):
    ..
    #object detection code    # Performs object detection and
                              # stores the detected objects in a list named detected_objects
    ..

    df.loc[idx, 'objects'] = detected_objects   # Adding the detected objects to the dataframe

return df

Output of df.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 34250 entries, 0 to 34249
Data columns (total 1 columns):
 #   Column  Non-Null Count  Dtype
---  ------  --------------  -----
 0   image   34250 non-null  object
dtypes: object(1)
memory usage: 267.7+ KB


title: Is there a way to specifically underutilize your gpu?

body:
I find that when I'm training models but still working on my computer I get slower responsiveness on other programs that make use of the GPU. One good example might be Xournal++ which I use to take notes with a stylus.
I'm guessing the way the GPU is accessed by programs is in a time-shared fashion like with a CPU. But is there any way I can specifically tell PyTorch to only use the GPU on a reduced duty cycle, thereby giving more space for my side applications to use it before whatever manager has to prioritise competing needs?


title: save model output in pytorch

body:
dic = []
for step, batch in tqdm(enumerate(train_dataloader)):
inpt = batch[0].to(device)
msks = batch[1].to(device)
#Run the sentences through the model
outputs = model_obj(inpt, msks)
dic.append( {
    'hidden_states': outputs[2],
    'pooled_output': outputs[1]})

I want to save the model output in each iteration but I got the below error for a small set of datasets.
RuntimeError: CUDA out of memory.
notice that without the below code my model works correctly.
dic.append( { 'hidden_states': outputs[2], 'pooled_output': outputs[1]})
How can I save these outputs in each iteration?


I'm trying to train a CNN using PyTorch's example with my own data. I have the following training loop which is identical to PyTorch:
def train_model(model, criterion, optimizer, scheduler, num_epochs=25):
    since = time.time()

    best_model_wts = copy.deepcopy(model.state_dict())
    best_acc = 0.0

    for epoch in range(num_epochs):
        print('Epoch {}/{}'.format(epoch, num_epochs - 1))
        print('-' * 10)

        # Each epoch has a training and validation phase
        for phase in ['train', 'val']:
            if phase == 'train':
                model.train()  # Set model to training mode
            else:
                model.eval()   # Set model to evaluate mode

            running_loss = 0.0
            running_corrects = 0

            # Iterate over data.
            for i, batch in enumerate(loaders[phase]):
                inputs = batch["image"].type(torch.cuda.LongTensor).to(device)
                labels = batch["label"].type(torch.cuda.LongTensor).to(device)

                # zero the parameter gradients
                optimizer.zero_grad()

                # forward
                # track history if only in train
                with torch.set_grad_enabled(phase == 'train'):
                    outputs = model(inputs.type(torch.cuda.LongTensor).to(device))
                    _, preds = torch.max(outputs, 1)
                    loss = criterion(outputs, labels)

                    # backward + optimize only if in training phase
                    if phase == 'train':
                        loss.backward()
                        optimizer.step()

                # statistics
                running_loss += loss.item() * inputs.size(0)
                running_corrects += torch.sum(preds == labels.data)
            if phase == 'train':
                scheduler.step()

            epoch_loss = running_loss / dataset_sizes[phase]
            epoch_acc = running_corrects.double() / dataset_sizes[phase]

            print('{} Loss: {:.4f} Acc: {:.4f}'.format(
                phase, epoch_loss, epoch_acc))

            # deep copy the model
            if phase == 'val' and epoch_acc > best_acc:
                best_acc = epoch_acc
                best_model_wts = copy.deepcopy(model.state_dict())

        print()

    time_elapsed = time.time() - since
    print('Training complete in {:.0f}m {:.0f}s'.format(
        time_elapsed // 60, time_elapsed % 60))
    print('Best val Acc: {:4f}'.format(best_acc))

    # load best model weights
    model.load_state_dict(best_model_wts)
    return model

However, I get the error:
Epoch 0/24
----------
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-24-79684c739f29> in <module>()
----> 1 model_ft = train_model(resnet_cnn, criterion, optimizer_ft, exp_lr_scheduler, num_epochs=25)

6 frames
<ipython-input-21-393aa43e7b06> in train_model(model, criterion, optimizer, scheduler, num_epochs)
     30                 # track history if only in train
     31                 with torch.set_grad_enabled(phase == 'train'):
---> 32                     outputs = model(inputs.type(torch.cuda.LongTensor).to(device))
     33                     _, preds = torch.max(outputs, 1)
     34                     loss = criterion(outputs, labels)

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    887             result = self._slow_forward(*input, **kwargs)
    888         else:
--> 889             result = self.forward(*input, **kwargs)
    890         for hook in itertools.chain(
    891                 _global_forward_hooks.values(),

/usr/local/lib/python3.7/dist-packages/torchvision/models/resnet.py in forward(self, x)
    247
    248     def forward(self, x: Tensor) -> Tensor:
--> 249         return self._forward_impl(x)
    250
    251

/usr/local/lib/python3.7/dist-packages/torchvision/models/resnet.py in _forward_impl(self, x)
    230     def _forward_impl(self, x: Tensor) -> Tensor:
    231         # See note [TorchScript super()]
--> 232         x = self.conv1(x)
    233         x = self.bn1(x)
    234         x = self.relu(x)

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    887             result = self._slow_forward(*input, **kwargs)
    888         else:
--> 889             result = self.forward(*input, **kwargs)
    890         for hook in itertools.chain(
    891                 _global_forward_hooks.values(),

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/conv.py in forward(self, input)
    397
    398     def forward(self, input: Tensor) -> Tensor:
--> 399         return self._conv_forward(input, self.weight, self.bias)
    400
    401 class Conv3d(_ConvNd):

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/conv.py in _conv_forward(self, input, weight, bias)
    394                             _pair(0), self.dilation, self.groups)
    395         return F.conv2d(input, weight, bias, self.stride,
--> 396                         self.padding, self.dilation, self.groups)
    397
    398     def forward(self, input: Tensor) -> Tensor:

RuntimeError: Input type (torch.cuda.LongTensor) and weight type (torch.cuda.FloatTensor) should be the same

I've tried to convert my data using torch.cuda.LongTensor as seen from above however it doesn't work for some reason. Does anybody have any ideas? Thank you greatly in advance!
Edit 1:
def train_model(model, criterion, optimizer, scheduler, num_epochs=25):
    since = time.time()

    best_model_wts = copy.deepcopy(model.state_dict())
    best_acc = 0.0

    for epoch in range(num_epochs):
        print('Epoch {}/{}'.format(epoch, num_epochs - 1))
        print('-' * 10)

        # Each epoch has a training and validation phase
        for phase in ['train', 'val']:
            if phase == 'train':
                model.train()  # Set model to training mode
            else:
                model.eval()   # Set model to evaluate mode

            running_loss = 0.0
            running_corrects = 0

            # Iterate over data.
            for i, batch in enumerate(loaders[phase]):
                inputs = batch["image"].type(torch.cuda.FloatTensor).to(device)
                labels = batch["label"].type(torch.cuda.FloatTensor).to(device)

                # zero the parameter gradients
                optimizer.zero_grad()

                # forward
                # track history if only in train
                with torch.set_grad_enabled(phase == 'train'):
                    outputs = model(inputs.to(device))
                    _, preds = torch.max(outputs, 1)
                    loss = criterion(outputs, labels)

                    # backward + optimize only if in training phase
                    if phase == 'train':
                        loss.backward()
                        optimizer.step()

                # statistics
                running_loss += loss.item() * inputs.size(0)
                running_corrects += torch.sum(preds == labels.data)
            if phase == 'train':
                scheduler.step()

            epoch_loss = running_loss / dataset_sizes[phase]
            epoch_acc = running_corrects.double() / dataset_sizes[phase]

            print('{} Loss: {:.4f} Acc: {:.4f}'.format(
                phase, epoch_loss, epoch_acc))

            # deep copy the model
            if phase == 'val' and epoch_acc > best_acc:
                best_acc = epoch_acc
                best_model_wts = copy.deepcopy(model.state_dict())

        print()

    time_elapsed = time.time() - since
    print('Training complete in {:.0f}m {:.0f}s'.format(
        time_elapsed // 60, time_elapsed % 60))
    print('Best val Acc: {:4f}'.format(best_acc))

    # load best model weights
    model.load_state_dict(best_model_wts)
    return model

This returns the new error:
Epoch 0/24
----------
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-38-79684c739f29> in <module>()
----> 1 model_ft = train_model(resnet_cnn, criterion, optimizer_ft, exp_lr_scheduler, num_epochs=25)

4 frames
<ipython-input-36-9b4381de034f> in train_model(model, criterion, optimizer, scheduler, num_epochs)
     32                     outputs = model(inputs.to(device))
     33                     _, preds = torch.max(outputs, 1)
---> 34                     loss = criterion(outputs, labels)
     35
     36                     # backward + optimize only if in training phase

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    887             result = self._slow_forward(*input, **kwargs)
    888         else:
--> 889             result = self.forward(*input, **kwargs)
    890         for hook in itertools.chain(
    891                 _global_forward_hooks.values(),

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/loss.py in forward(self, input, target)
   1046         assert self.weight is None or isinstance(self.weight, Tensor)
   1047         return F.cross_entropy(input, target, weight=self.weight,
-> 1048                                ignore_index=self.ignore_index, reduction=self.reduction)
   1049
   1050

/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction)
   2691     if size_average is not None or reduce is not None:
   2692         reduction = _Reduction.legacy_get_string(size_average, reduce)
-> 2693     return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
   2694
   2695

/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)
   2386         )
   2387     if dim == 2:
-> 2388         ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
   2389     elif dim == 4:
   2390         ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)

RuntimeError: Expected object of scalar type Long but got scalar type Float for argument #2 'target' in call to _thnn_nll_loss_forward


title: Gradient of neural network with respect to inputs

body:
I am working on a NN with Pytorch which simply maps points from the plane into real numbers, for example
model = nn.Sequential(nn.Linear(2,2),nn.ReLU(),nn.Linear(2,1))

What I want to do, since this network defines a map h:R^2->R, is to compute the gradient of this mapping h in the training loop. So for example
 for it in range(epochs):
      pred = model(X_train)
      grad = torch.autograd.grad(pred,X_train)
      ....

The training set has been defined as a tensor requiring the gradient. My problem is that even if the output, for each fixed point, is a scalar, since I am propagating a set of N=100 points, the output is actually a Nx1 tensor. This brings to the error: autograd can compute the gradient just of scalar functions.
In fact, trying with the little change
pred = torch.sum(model(X_train))

everything works perfectly. However I am interested in all the single gradients so, is there a way to compute all these gradients together?
Actually computing the sum as presented above gives exactly the same result I expect of course, but I wanted to know if this is the only possiblity.


title: A question about pytorch change tensor shape

body:
When I want to split an image into multiple patches, the first thing I thought was using pytorch view() function. For example, An image of shape (1,3,256,256)(pytorch style), and split it into 8x8=64 patches, each patch height and width is 32. So for this image, we can get 256/32=8 for row and 8 for column patches, so we total have 8x8=64 patches.
I want to split the image (1,3,256,256) into patches, each patch shape is (1,3,32,32), and reshape these tensor into shape of (1,8x8,32x32x3), here 8x8 is the number of patches, 32x32x3 is patch height * width * channel.
Code below use einops rearrange function can get the correct answer, but when I use view function, I got correct shape but incorrect value tensor. Who can tell me how to implement this operation by view function?
from einops.layers.torch import Rearrange
img = torch.randn(1, 3, 256, 256)
import copy
img2 = copy.deepcopy(img)
b, c, h, w = img.size()
p=32
to_patch_embedding = nn.Sequential(
    Rearrange('b c (h p1) (w p2) -> b (h w) (p1 p2 c)', p1=32, p2=32),
)
img2 = img2.view(b, h // p * w // p, c * p * p)

print(img2.shape)
print(img2==to_patch_embedding(img))

----------------------------------------output--------------------------------------------
torch.Size([1, 64, 3072])
tensor([[[ True, False, False,  ..., False, False, False],
     [False, False, False,  ..., False, False, False],
     [False, False, False,  ..., False, False, False],
     ...,
     [False, False, False,  ..., False, False, False],
     [False, False, False,  ..., False, False, False],
     [False, False, False,  ..., False, False,  True]]])


I'm using python 3.7 and pytorch in google colab.
I installed
Pip install alennlp==2.4.0
pip install allennlp-models

In google colab, but when I run this code:
allennlp train experiments/oie_labeler_crf.json -s ~/Desktop/PHD/NLP/large-scale-oie/results/classic_train_crf --include-package large_scale_oie
I get this error:
File "/content/large-scale-oie/large_scale_oie/dataset_readers/oie_reader.py", line 12, in
from allennlp.data.dataset_readers.dataset_utils import Ontonotes, OntonotesSentence
ImportError: cannot import name 'Ontonotes' from 'allennlp.data.dataset_readers.dataset_utils' (/usr/local/lib/python3.7/dist-packages/allennlp/data/dataset_readers/dataset_utils/init.py)

Please help me! What do I do?


title: How can I implement the map function on the matrix in PyTorch?

body:
In PyTorch, I would like to implement a calculation as shown in the following figure:

In other words, this is a map function that applied to each vector in the matrix, and each function is different such as torch.sin, torch.cos, and torch.tan. How can I implement such calculation in PyTorch (i.e., without for loop on GPU).


title: Gradient computation error in inplace operation

body:
I have an encoder-decode model and a function that calculating something iteratively:
f(a,b,c):
   # a: [batchsize, n, channel]
   res_a = torch.empty(a.shape)
   res_b = torch.empty(b.shape)
   for i in range(a.shape[1]):
        res_a[:,i] = res_a[:,i-1] * xxx + xxx
        res_b[:,i] = res_a[:,i] * res_b[i-1] + xxx
        # something like the two lines above
   return torch.cat([res_a,res_b,...],-1)

This is the main procedure of forward function:
def forward(x):
    a, b, c = x[0:i], x[i:j], x[j:k]
    q = f(a,b,c)
    x1 = decoder(encoder(x))
    a, b, c = x1[0:i], x1[i:j], x1[j:k]
    q1 = f(a,b,c)
    return q, q1

The loss is the difference of q and q1
Then an error occurs when backward for the computation of q1:
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [64, 3]], which is output 0 of SelectBackward, is at version 21; expected version 20 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

I really have no idea where I modify the variables for gradient computation, all I do in function f is just indexing some data from input and conduct some computations (add, multiply), then store the result to a new tensor.


title: How to save and load nn.Parameter() in pytorch, such that one can continue optimisation over it?

body:
I know how to store and load nn.Model, but can not find how to make a checkpoint for nn.Parameter. I tried this version, but the optimizer is not changing the nn.Parameter value after restoring.
from torch import nn as nn
import torch
from torch.optim import Adam

alpha = torch.ones(10)
lr = 0.001
alpha = nn.Parameter(alpha)
print(alpha)
alpha_optimizer = Adam([alpha], lr=lr)

for i in range(10):
   alpha_loss = - alpha.mean()
   alpha_optimizer.zero_grad()
   alpha_loss.backward()
   alpha_optimizer.step()
   print(alpha)
path = "./test.pt"
state = dict(alpha_optimizer=alpha_optimizer.state_dict(),
               alpha=alpha)
torch.save(state, path)
checkpoint = torch.load(path)
alpha = checkpoint["alpha"]
alpha_optimizer.load_state_dict(checkpoint["alpha_optimizer"])
for i in range(10):
   alpha_loss = - alpha.mean()
   alpha_optimizer.zero_grad()
   alpha_loss.backward()
   alpha_optimizer.step()
   print(alpha)


title: How to make code run on GPU on Windows 10?

body: I want to run my code run on gpu in windows 10, like for google colab, we can just change the runtime option which is pretty easy to do to shift to gpu. Is there a possibility to do the same for jupyter notebook in windows.


title: Documenting jit-compiled PyTorch Class Method (Sphinx)

body:
I am having a problem trying to document custom PyTorch models with Sphinx: methods that are jit-compiled show up without docstrings in the documentation. How do I fix this? I checkoed out Python Sphinx autodoc and decorated members and How to autodoc decorated methods with sphinx? but the proposed solutions don't seem to work. When I try using ..automethod I get
AttributeError: '_CachedForward' object has no attribute '__getattr__'

Here's a MWE; at the moment I circumvent the problem by writing a my_forward and calling it in the forward.
from torch import jit, Tensor

class DummyModel(jit.ScriptModule):
    """My dummy model"""

    def __init__(self, const: float):
        super(DummyModel, self).__init__()
        self.const = Tensor(const)

    @jit.script_method
    def forward(self, x: Tensor) -> Tensor:
        """This method does show as a :undoc-member: in the documentation"""
        return self.my_forward(x)

    def my_forward(self,  x: Tensor) -> Tensor:
        """This method shows up as a :member: in the documentation"""
        return x + self.const


title: PyTorch: Error>> expected scalar type float but found double

body:
I've just started using pytorch and I am trying a simple multi-layer perceptron . My ReLU Activation Function is the following:
def ReLU_activation_func(outputs):
    print(type(outputs))
    result = torch.where(outputs > 0, outputs, 0.)
    result = float(result)
    return result

So I am trying to maintain the value which is greater than 0 and change the value to 0 if the value is smaller than 0.
And this is a part of the main code where I use the ReLU Function (where I have the error):
def forward_pass(train_loader):
    for batch_idx, (image, label) in enumerate(train_loader):
        print(image.size())
        x = image.view(-1, 28 * 28)
        print(x.size())

        input_node_num = 28 * 28
        hidden_node_num = 100
        output_node_num = 10
        W_ih = torch.rand(input_node_num, hidden_node_num)
        W_ho = torch.rand(hidden_node_num, output_node_num)
        final_output_n = ReLU_activation_func(torch.matmul(x, W_ih))

and when I run the code, I get the following error:
RuntimeError:
1 forward_pass(train_loader)

in forward_pass(train_loader)
-----14         W_ih = torch.rand(input_node_num, hidden_node_num)
-----15         W_ho = torch.rand(hidden_node_num, output_node_num)
---->16         final_output_n = ReLU_activation_func(torch.matmul(x, W_ih))

in ReLU_activation_func(outputs)
-----10     print(type(outputs))
---->11     result = torch.where(outputs > 0, outputs, 0.)
-----12     result = float(result)
-----13     return result

RuntimeError: expected scalar type float but found double

Any help?


title: Is it possible to use a machine learning library with streaming inputs and outputs?

body:
I want to incorporate machine learning into a project ive been working on but i havent seen anything about my intended use case. It seems like the old pandoras box project did something like this but with textual input and output.
I want to train a model in real time as well as use it (and then switch it from testing to live api endpoints when it works well enough.)
But every library ive found works like "feed in a datablob, get an answer"
I want to be able to stream data into it:
instead of giving it "5,4,3,4,3,2,3,4,5" and it says "1" or "-1" or "0"
I want to give it "5" then "4" then "3" then "4" etc and each time it responds.

Im not even sure if "streaming" is the right word for this. Please help!


title: What’s the difference between the two operations in back-propagation with pytorch?

body:
I have a tensor x and x.shape=(batch_size,10)

I want to add one to all of the elements, and take two different operations

x=x+1
for i in range(0,batch_size):
  x[i]=x[i]+1

I got the same tensors with the two operations,but when I call loss.backward(), (2) takes much more time than (1) in back propagation.

What’s the difference between them???


title: How to calculate high order derivative of the square of adjacency matrix?

body:
I would like to calculate high order derivatives in Pytorch, but the results are nan.

X = torch.tensor([[[0.,0.,0.],[1.,0.,0.],[3.,0.,0.]]],requires_grad=True)
adjacency_matrix = (X.unsqueeze(2)-X.unsqueeze(1)).norm(dim=3)
Y = torch.matmul(adjacency_matrix,adjacency_matrix)
grad1 = torch.autograd.grad(Y.sum(),X,create_graph=True)[0]
grad2 = torch.autograd.grad(grad1.sum(),X,create_graph=True)[0]
print(grad2)

The output grad2 is all nan. Could you please tell me how to get the correct result？


title: Image format in segmentation via neural networks

body:
I am doing segmentation via deep learning in pytorch. My dataset is a .raw/.mhd format ultrasound images.
I want to input my dataset into the system via data loader.

I faced few important questions:

Does changing the format of the dataset to either .png or .jpg make the segmentation inaccurate?(I think I lost some information in this way!)
Which format is less data lossy?
How should I make a dumpy array if I don't convert the original image format, i.e., .raw/.mhd?
How should I load this dataset?


title: How to save python code (part of the notebook) to file in GDrive from code

body:
I am using Google Colabs for my research in machine learning.
I do many variations on a network and run them saving the results.

I have a part of my notebook that used to be separate file (network.py) At the start of a training session I used to save this file in a directory that has the results and logs etc. Now that this part of the code is in the notebook it is easier to edit etc, BUT I do not have a file to copy to the output directory that describes the model. how to i take a section of a google colab notebook and save the raw code as a python file?

Things I have tried:
%%writefile "my_file.py" - is able to write the file however the classes are not available to the runtime.


title: PyTorch - a functional equivalent of nn.Module

body: As we know we can wrap arbitrary number of stateful building blocks into a class which inherits from nn.Module. But how is it supposed to be done when you want to wrap a bunch of stateless functions (from nn.Functional), in order to fully utilize things which nn.Module allows you to, like automatic moving of tensors between CPU and GPU with just model.to(device)?


title: Can not load image segmentation model partially in pytorch

body:
I'm trying to load a model partially (i.e., instead of loading all the layers at once, I'm just trying to load the first couple layers of the network). Here is my code:

import torch

unet = my_unet(in_ch=5, out_ch=1).cuda()
enc = torch.nn.Sequential(*list(unet.children())[:10])
del unet # Comment this line if you wnat to load unet and see output later
print(torch.cuda.empty_cache())
x = np.random.randn(1, 5, 320, 320)
x = (x - np.min(x))/(np.max(x) - np.min(x))
out = enc(torch.FloatTensor(x).cuda())
# out = unet(torch.FloatTensor(x).cuda())
print(out.size())

If I run this script, it is interrupted by CUDA out of memory error. But if I load the whole my_unet
model(by commenting the 3rd and the 7th line and uncommenting the 8th line) it works fine and prints the out.size() (1, 1, 320, 320) as expected. What am I doing wrong here? Why can I load the whole model but can not load part of the same model which should require less memory? Is there any workaround?


title: Slicing in Pytorch gives a wrong answer

body:
When I run the following lines, I see the following result which I can't explain. I expected to see [1., 2.] as the answer to a*x[:,0] and indeed if I do the same thing in numpy, I get that answer but I don't know why pytorch gives me a different answer:

>>> x = torch.tensor([[1,0],[1,1]])
>>> a = torch.tensor([1.,2.])
>>> a*x
tensor([[1., 0.], [1., 2.]])
>>> a*x[:,0]
tensor([1., nan])


title: pytorch ImageFolder datasize does not match with the original imgs number?

body:
I have 21,522 items, totalling 609.7 MB in train folder

train
  -1(3600 png)
  -2(3600 png)
   .
   .
  -6

train_trans = transforms.Compose([
        vision.transforms.Resize(target_size),
        vision.transforms.RandomHorizontalFlip(),
        vision.transforms.RandomRotation(20),
        #CIFAR10Policy(),
        vision.transforms.ToTensor(),
        vision.transforms.Normalize(
            [0.485, 0.456, 0.406],
            [0.229, 0.224, 0.225])
])
valid_trans = transforms.Compose([
        vision.transforms.Resize(target_size),
        vision.transforms.RandomResizedCrop(target_size, scale=(0.8,1.0)),
        vision.transforms.RandomHorizontalFlip(),
        vision.transforms.ToTensor(),
        vision.transforms.Normalize(
            [0.485, 0.456, 0.406],
            [0.229, 0.224, 0.225])
])
test_trans = transforms.Compose([
        vision.transforms.Resize((size,size)),
        vision.transforms.RandomResizedCrop(target_size, scale=(0.8,1.0)),
        vision.transforms.ToTensor(),
        vision.transforms.Normalize(
            [0.485, 0.456, 0.406],
            [0.229, 0.224, 0.225])
])

but when I load the dataset with

train_dataset = torchvision.datasets.ImageFolder(root= path + 'train/',transform=trans)

somehow

  print(len(train_dataset)) = 4109

somehow it does not load the entire image files


title: Train part of the network

body:
I am new to Deep Learning and Pytorch. I have the following problem:

My overall architecture consists of a network that I define (NN1), and another pretrained network (NN2), so that the output of NN1 is the input of NN2. I want to define the loss of NN1 using the difference (RMSE) between the output of NN2 and a know ground-truth.
I need to back-propagate through NN2 and NN1 (to train NN1), without changing NN2.

I can use requires_grad=False on NN2, but will it disable the back-propagation through NN2? How do I specify this requirement in Pytorch?

Thanks.


title: Is there any essential difference between * and + for Pytorch autograd?

body:
I was trying to understand the autograd mechanism in more depth. To test my understanding, I tried to write the following code which I expected to produce an error (i.e., Trying to backward through the graph a second time).

b = torch.Tensor([0.5])
for i in range(5):
    b.data.zero_().add_(0.5)
    b = b + a
    c = b*a
    c.backward()

Apparently, it should report an error when c.backward() is called for the second time in the for loop, because the history of b has been freed， however, nothing happens.

But when I tried to change b + a to b * a as follows,

b = torch.Tensor([0.5])
for i in range(5):
    b.data.zero_().add_(0.5)
    b = b * a
    c = b*a
    c.backward()

It did report the error I was expecting.
This looks pretty weird to me. I don't understand why there is no error evoked for the former case, and why it makes a difference to change from + to *.


title: how to use Tensorboard in Pytorch just draw a simple net

body:
I am a new pytorcher.
I want to draw a very very simple net just one input with constant weight and bias.
This net just a = w * x + b. Every input is scalar.
I want to use pytorch and tensorboard to show this net.

I write code below.

class Net(nn.Module):
    def __init__(self, w, b):
        super(Net, self).__init__()
        self.w = w
        self.b = b

    def forward(self, x):
        x = self.w * x + self.b
        return x

net = Net(1.3, 3.0)

writer = SummaryWriter('runs/CH1_2_1')
input = torch.tensor([1.0], dtype=torch.float)
input.requires_grad_(True)
writer.add_graph(net, input)
writer.close()

The net I want shouldn't be like this
what's the problem


title: Copying values from one tensor to another tensor based on index mapping and averaging values mapped to same index

body:
I have one input tensor and an index mapping. I want to create output tensor based on that index mapping.

For example:
input = torch.tensor([12, 56, 45, 37], dtype=torch.float)

index_map = torch.tensor([-1, 1, 2, 1, 3, -1])
(-1 in index_map represents it is not mapped to anything)

grad = torch.tensor([4, 2, 5, 3, 6, 7], dtype=torch.float)

I want to copy values from grad to input based on index_map. The values which are mapped to same index should be averaged. For e.g. grad[1] and grad[3] are both mapped to index 1, so final value at index 1 should be 1.5

Output tensor should be like :
tensor([12.,  1.5,  5.,  6.])

My code is:

input = torch.tensor([12, 56, 45, 37], dtype=torch.float)
input = torch.cat([input, torch.tensor([-1], dtype=torch.float)])
index_map = torch.tensor([-1, 1, 2, 1, 3, -1])
grad_out = torch.tensor([4, 2, 5, 3, 6, 7], dtype=torch.float)
input[index_map] = grad_out
input = input[:4]
print(input)

Above code copies the latest value but does not average it out.


title: In pytorch, how could we save model's weight to a file in my github

body:
I am new to pytorch and google colab. Right now, I am running the python script on colab. I tried to save my trained model to a model.pth file that I just created for the model in my github. However, it reports an error
"Is a directory: 'https://github.com/***/mygithub/blob/master/model.pth'"


title: Why torch.FloatTensor([[[0,1,2],[3,4,5]],[[6,7,8],[9,10,11]]]) size is [2,2,3]?

body:
>>> ft = torch.FloatTensor([[[0,1,2],[3,4,5]],[[6,7,8],[9,10,11]]])
>>> print(ft.shape)
torch.Size([2, 2, 3])

I can't understand this result.
I think the torch size should be [2,3,2], but the result is [2,2,3].


title: How can i extract Trained model information from its weight files?

body: I had got pre-trained model weight files of a model for audio classification. How can i extract model information from that weights given (For examples no of layers used to build architecture)? This question asked in the interview. is if possible to extract model information from its weights.


title: Getting this error RuntimeError: size mismatch, m1: [10 x 33], m2: [34 x 10]

body:
I am building a neural network to solve my classification problem, but for some reason Im getting the error RuntimeError: size mismatch, m1: [10 x 33], m2: [34 x 10]. I dont understand what im doing wrong.
Here is my code.

X_train= torch.tensor(X_train, dtype= torch.float)
X_test= torch.tensor(X_test, dtype= torch.float )
Y_train= torch.tensor(Y_train, dtype= torch.float).view(-1,1)
Y_test= torch.tensor(Y_test, dtype= torch.float).view(-1,1)

dataset= torch.utils.data.TensorDataset(X_train, Y_train)

train_iter = torch.utils.data.DataLoader(dataset, batch_size=10, shuffle=True)

import torch.nn.functional as F

class Network(nn.Module):
    def __init__(self):
        super().__init__()

        # Inputs to hidden layer linear transformation
        self.hidden = nn.Linear(34, 10)
        # Output layer, 10 units - one for each digit
        self.output = nn.Linear(10, 1)

        # Define sigmoid activation and softmax output
        self.sigmoid_act = nn.Sigmoid()
        self.ReLU_act= nn.ReLU()

    def forward(self, x):
        # Pass the input tensor through each of our operations
        x = self.hidden(x)
        x = self.ReLU_act(x)
        x = self.output(x)
        x = self.sigmoid_act(x)

        return x

model= Network()
criterion = nn.BCELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.003)

epochs = 30
steps = 0

train_losses, test_losses = [], []
for e in range(epochs):
    running_loss = 0
    for images, labels in train_iter:

        optimizer.zero_grad()

        **log_ps = model(images)**
        loss = criterion(log_ps, labels)
        loss.backward()
        optimizer.step()

        running_loss += loss.item()

    else:
        test_loss = 0
        accuracy = 0

Im getting the error on the line in bold. Please help me resolve this. Im stuck on it for almost a day now. Im a beginner at neural networks.


title: Getting Correct Accuracy Per Epoch

body:
I am running SGD over a batch of data being fed by below:

my_dataset = data.TensorDataset(train_x,train_y)
trainloader = data.DataLoader(my_dataset,batch_size=32,
                                      shuffle=True)

This is my model and loss function:

net = Net(n_x, n_h, n_y)
criterion = nn.BCELoss()
optimizer = optim.ASGD(net.parameters(), lr=lr)

after defining my model and the loss function I run the code below for the training phase:

for epoch in range(2):
count=0
    for i, data in enumerate(trainloader, 0):
        # get the inputs
        inputs, labels = data

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = net(inputs)
        loss = criterion(outputs, labels)
        preds = outputs > 0.5
        nb_correct = (preds == labels).sum()
        count+=nb_correct.item()
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()

print('Finished Training')

The problem is I cannot find a good way to find the accuracy per epoch. Initially, my goal was to use the found method I came up with and divide the count by the size of my dataset (happens to be 209 test cases). However to my surprise diving by the dataset length results in strange outputs. What is a good way to get an accuracy percentage epoch given the way I have defined my problem?


title: Linear Regression Speed Increase

body:
I am new in  machine learning and it is my first time to create a linear regression model on a dataset(which is big step for me). I have created my reference rows and reshaped them. Only problem is  It is too slow. Is there an any code or better way that I can use. It would be great if you have chance to revise my code.
Thank you.

import pandas as pd
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.utils.data import DataLoader
from torch.utils.data import TensorDataset
db = pd.read_csv("Melbourne_housing_FULL.csv")
db2 = pd.read_csv("MELBOURNE_HOUSE_PRICES_LESS.csv")

"""['Suburb', 'Address',
'Rooms', 'Type', 'Price',
'Method', 'SellerG',
'Date', 'Distance',
'Postcode', 'Bedroom2', 'Bathroom', 'Car',
'Landsize', 'BuildingArea',
'YearBuilt', 'CouncilArea',
'Lattitude','Longtitude',
'Regionname', 'Propertycount']

column names for Full.csv"""

rooms_properties = db[["Rooms","Landsize","Bathroom","Car","YearBuilt"]].copy()
rooms_properties.fillna(rooms_properties.mean(),inplace=True)
rooms_price = db[["Price"]].copy()
rooms_price.fillna(rooms_price.mean(),inplace=True)

room_array_properties = rooms_properties.to_numpy()
room_array_price = rooms_price.to_numpy()

##Splitted list with percentage
def indice_splitter(array_prop,x=0.2):
       val = np.random.permutation(len(array_prop))
       percent_1 = val[:int(len(val) * x)]
       percent_2 = val[int(len(val)*x):]
       return percent_2,percent_1
## Converted df as tensor

train_indices,validation_indices = indice_splitter(room_array_price)
train_data,targets1 = room_array_properties[train_indices], room_array_price[train_indices]

validation_data ,targets2 = torch.from_numpy(room_array_properties[validation_indices]).float(), torch.from_numpy(room_array_price[validation_indices]).float()
t_data, tar1 = torch.tensor(train_data,requires_grad=True).float(), torch.tensor(targets1,requires_grad=True).float()
# rooms_price = rooms_price[rooms_price.notnull()]
# r_nonull = [rooms_properties.loc[rooms_properties[i].notnull()]  for i in rooms_properties.columns]
# r_nonull = r_nonull[len(r_nonull)-1]
# r_array = r_nonull.to_numpy()

# weight = torch.rand(5,1, dtype=float,requires_grad=True)
# bias = torch.randn(len(train_data),dtype=float,requires_grad=True)

## my model and result
model = nn.Linear(5,1)
weight, bias = model.parameters()

train_ds = TensorDataset(t_data,tar1)
batch_size = 10
train_dl = DataLoader(train_ds,batch_size,shuffle=True)

preds = model(t_data)
loss_fn = F.mse_loss
opt = torch.optim.SGD(model.parameters(), lr= 1e-5)

#
def fit(num_epochs, model, loss_fn, opt):
       for epochy in range(num_epochs):
              for xb,yb in train_dl:
                     #increase the model accuracy
                     pred = model(xb)
                     loss = loss_fn(pred,yb)
                     loss.backward()
                     #upgrade the stoachaistic grad descent
                     opt.step()
                     #refresh the data
                     opt.zero_grad()
              if (epochy +1) % 10 == 0:
                     print("{}/{}, Loss:{:.4f}".format(epochy+1,num_epochs,loss.item()))

fit(10,model, loss_fn,opt)

Output is = 10/10, Loss:nan

My expected output should decrease my loss function value every time.

I want to iterate this regression at least 1000 times.

I have 1660Ti , i7 9th Gen, 16gb ram laptop


title: is there any similar function with clamp_ in tensorflow > 2.0

body:
I'm converting torch code to tensorflow 2.0

prior_boxes = torch.FloatTensor(prior_boxes).to(device)  # (8732, 4)
prior_boxes.clamp_(0, 1)  # (8732, 4)

is there any replacement of clamp_(0,1) in tensorflow > 2.0?


title: Removing tensors for optimising a for loop in python

body:
I am working on a large code that I'm trying to optimise.
The part of code you see below is a for loop that returns encodings in a tensor. How do I output these numbers in a regular list instead, without going to tensors?

def _make_batches(self, lines):
        tokens = [self._tokenize(line) for line in lines]
        lengths = np.array([t.numel() for t in tokens])
        indices = np.argsort(-lengths, kind=self.sort_kind)  # pylint: disable=invalid-unary-operand-type

        def batch(tokens, lengths, indices):
            toks = tokens[0].new_full((len(tokens), tokens[0].shape[0]),
                                      self.pad_index)
            for i in range(len(tokens)):
                toks[i, -tokens[i].shape[0]:] = tokens[i]
            return Batch(srcs=None,
                         tokens=toks,
                         lengths=torch.LongTensor(lengths)), indices

        batch_tokens, batch_lengths, batch_indices = [], [], []
        ntokens = nsentences = 0
        for i in indices:
            if nsentences > 0 and ((self.max_tokens is not None
                                    and ntokens + lengths[i] > self.max_tokens)
                                   or (self.max_sentences is not None
                                       and nsentences == self.max_sentences)):
                yield batch(batch_tokens, batch_lengths, batch_indices)
                ntokens = nsentences = 0
                batch_tokens, batch_lengths, batch_indices = [], [], []
            batch_tokens.append(tokens[i])
            batch_lengths.append(lengths[i])
            batch_indices.append(i)
            ntokens += tokens[i].shape[0]
            nsentences += 1
        if nsentences > 0:
            yield batch(batch_tokens, batch_lengths, batch_indices)

This is how I call this function:

if __name__ == '__main__':
    s = SentenceEncoder("data/model.pt")
    input = [args.string_enc]
    make_batches = s._make_batches
    print([batch[1] for batch, indexes in make_batches(input)])

The output is:

[tensor([[29733, 20720,     2]])]

The desired output is:

[29733, 20720,     2]


title: Should I import a python module if I don't use it directly

body:
I am working on a project with a few files, and one of my helper modules has a function that takes as argument a criterion that measures MSE from the torch library. This module does postprocessing (displaying/saving plots and saving relevant data) and doesn't explicitly use any functionality from torch other than using the passed criterion to compute the mean squared error for two of the parameters.
Is it best practice to import torch, or is it ok not to? Similarly, should I import numpy when dealing implicitly with numpy arrays?
Here is the code in question:
import matplotlib.pyplot as plt

# use latex for plt labels/diagrams
plt.rc('text', usetex=True)
plt.rc('font', family='serif')

columns_tex = [r'$x$ (m)', r'$\dot{x}$ (m/s)', r'$\theta$ (rad)', r'$\dot{\theta}$ (rad/s)']
columns_file = ['x', 'x_dot', 'theta', 'theta_dot']

def process_plots(test_y, predicted_y, total_error, criterion, system, m, displaying_plots, saving_plots):
    '''
    m: number of training examples
    '''
    for i, col in enumerate(columns_tex):
        # criterion is given by torch.nn.MSELoss()
        mse = criterion(test_y[:,i],predicted_y[:,i])
        error = total_error[:,i].detach().numpy() # plt.scatter complains about being passed torch tensor
        plt.scatter([j for j in range(len(test_y))], error, s=1)
        plt.ylabel(f"{col} error: exp - actual")
        plt.xlabel(r'prediction $i$')
        plt.title(f"mean squared error for predictions of {col} at t+1 is {mse}", pad=20)
        plt.tight_layout()
        if saving_plots:
            plt.savefig(f'./data/processed/imgs/torch_lin_reg/{system}/{columns_file[i]}_error_{m}')
        if displaying_plots:
            plt.show()
        plt.clf()


title: PyTorch dictionary keys not matching

body:
I am trying to implement a convolutional LSTM I found online, and it seems that the dictionary keys are not matching:
The pre-trained weights are in a pickled dictionary with the following keys:
pkl_load = torch.load(trained_model_dir)
print(pkl_load.keys())

odict_keys(['module.E.conv1.weight', 'module.E.bn1.weight', 'module.E.bn1.bias', ....

However, the keys in the state_dict for the actual NN model are:
"E.conv1.weight", "E.bn1.weight", "E.bn1.bias", ....

I am getting an error when trying to load the pre-trained weights into the state_dict because the keys don't match. What are ways to work around this? (Sorry if this is easy, I am new to PyTorch).


title: Gradient accumulation with BatchNorm2d layers

body:
When I am doing gradient accumulation, the BatchNorm2d layers are not properly accumulated, right? Though, I don't entirely understand exactly what is going on. The running mean and std deviation are reset on each consecutive batch, correct?
If the BatchNorm2d layers are not properly accumulated, does that also disrupt the gradient accumulation in the accompanying Conv2d layers?
From what I've found online the "solution" is to use InstanceNorm2d.
I used InstanceNorm2d layers, but my model would not improve and drives predictions to nan even with clamping gradients between (-1,1).
I'm using these layers in a custom UNet model creating segmentation masks.
Does anyone have any insight to this?


title: How to handle model creep

body:
In machine learning, we are constantly updating the models. But we still want a way to use our old trained model checkpoints. These checkpoints will not load if the model class has been changed with different keyword arguments etc.
What's the best way to manage "model creep" or models gradually changing?
One way is, we can associated each pytorch checkpoint with a github commit. Then load that commit to run our model again.
Another way is, every time we change the model, we implement backwards compatibility (could get messy).


title: pytorch training loss invariant with varying forward pass implementations

body:
The following code (MNIST MLP in PyTorch) delivers approximately the same training loss regardless of having the last layer in the forward pass returning:

F.log_softmax(x)
(x)

Option 1 is incorrect because I use criterion = nn.CrossEntropyLoss() and yet the results are almost identical. Am I missing anything?
import torch
import numpy as np
import time
from torchvision import datasets
import torchvision.transforms as transforms
# number of subprocesses to use for data loading
num_workers = 0
# how many samples per batch to load
batch_size = 20

# convert data to torch.FloatTensor
transform = transforms.ToTensor()

# choose the training and test datasets
train_data = datasets.MNIST(root='data', train=True,
                                   download=True, transform=transform)
test_data = datasets.MNIST(root='data', train=False,
                                  download=True, transform=transform)

# prepare data loaders
train_loader = torch.utils.data.DataLoader(train_data, batch_size=batch_size,
    num_workers=num_workers)
test_loader = torch.utils.data.DataLoader(test_data, batch_size=batch_size,
    num_workers=num_workers)

import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        # linear layer (784 -> 1 hidden node)
        self.fc1 = nn.Linear(28 * 28, 512)
        self.dropout1= nn.Dropout(p=0.2, inplace= False)
        self.fc2 = nn.Linear(512, 256)
        self.dropout2= nn.Dropout(p=0.2, inplace= False)
        self.dropout = nn.Dropout(p=0.2, inplace= False)
        self.fc3 = nn.Linear(256, 10)

    def forward(self, x):
        # flatten image input
        x = x.view(-1, 28 * 28)
        # add hidden layer, with relu activation function
        x = F.relu(self.fc1(x))
        x = self.dropout1(x)
        x = F.relu(self.fc2(x))
        x = self.dropout2(x)
        x = self.fc3(x)
#        return F.log_softmax(x)
        return x

# initialize the NN
model = Net()
print(model)
model.to('cuda')
criterion = nn.CrossEntropyLoss()

optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
n_epochs = 10

model.train() # prep model for training

for epoch in range(n_epochs):
    # monitor training loss
    train_loss = 0.0

    start = time.time()
    for data, target in train_loader:
        data, target = data.to('cuda'), target.to('cuda')

        optimizer.zero_grad()
        output = model(data)
        loss = criterion(output, target)
        loss.backward()
        optimizer.step()
        train_loss += loss.item()*data.size(0)

    train_loss = train_loss/len(train_loader.dataset)

    print('Epoch: {} \tTraining Loss: {:.6f} \ttime: {:.6f}'.format(
        epoch+1,
        train_loss,
        time.time()-start
        ))


title: Loss value in Linear regression

body:
I have done a linear regression problem with boston dataset and I have obtained the next results:
Loss value does not change with increasing number of value. What is the reason of this mistake? Please, help me
import pandas as pd
import torch
import numpy as np
import torch.nn as nn
from sklearn import preprocessing
training_set=pd.read_csv('boston_data.csv')
training_set=training_set.to_numpy()
test_set=test_set.to_numpy()
inputs=training_set[:,0:13]
inputs=preprocessing.normalize(inputs)
target=training_set[:,13:14]
target=preprocessing.normalize(target)
inputs=torch.from_numpy(inputs)
target=torch.from_numpy(target)
test_set=torch.from_numpy(test_set)
w=torch.randn(13,1,requires_grad=True)
b=torch.randn(404,1,requires_grad=True)
def model(x):
    return x@w+b
pred=model(inputs.float())
def loss_MSE(x,y):
    ras=x-y
    return torch.sum(ras * ras) / ras.numel()
for i in range(100):
    pred=model(inputs.float())
    loss=loss_MSE(target,pred)
    loss.backward()
    with torch.no_grad():
        w -= w.grad * 1e-5
        b -= b.grad * 1e-5
        w.grad.zero_()
        b.grad.zero_()
    print(loss)


title: How to print out sample images from the training data using a class?

body:
The code is the following:
class Dataset(Dataset):

# Constructor
def __init__(self, csv_file, data_dir, transform=None):

    # Image directory
    self.data_dir=data_dir

    # The transform is going to be used on an image.
    self.transform = transform

    # Load the CSV file that contians image info
    self.data_name= pd.read_csv(csv_file)

    # Number of images in dataset
    self.len=self.data_name.shape[0]

# Get the length
def __len__(self):
    return self.len

# Getter
def __getitem__(self, idx):

    # Image file path
    img_name=self.data_dir + self.data_name.iloc[idx, 2]

    # Open image file
    image = Image.open(img_name)

    # The class label for the image
    y = self.data_name.iloc[idx, 3]

    # If there is any transform method, apply it onto the image
    if self.transform:
        image = self.transform(image)

    return image, y
#=================
# Create the dataset objects

train_dataset = Dataset(csv_file=train_csv_file
                        , data_dir='/resources/data/training_data_pytorch/')

I would like to answer the following question:
Question: Print out three sample images and their classes form the training data.
samples = [53, 23, 10]
Thank you!


title: Optimizer for an RNN using pytorch

body:
The pytorch RNN tutorial uses
for p in net.parameters():
    p.data.add_(p.grad.data, alpha = -learning_rate)

as optimizer. Does anyone know the difference between doing that or doing the classical optimizer.step(), once an optimizer has been defined explicitly? Is there some special consideration one has to take into when training RNNs in regards to the optimizer?


title: How to read video with generator in parallel?

body:
I don't really know much about multiprocessing in general.
I have the following code that generates a sequence of frames from the given video:
def read_video(filename: str) -> np.array:
    w, h = get_resolution(filename)

    command = [
        "ffmpeg",
        "-i",
        filename,
        "-f",
        "image2pipe",
        "-pix_fmt",
        "bgr24",
        "-vsync",
        "0",
        "-vcodec",
        "rawvideo",
        "-",
    ]

    pipe = sp.Popen(command, stdout=sp.PIPE, bufsize=-1)
    while True:
        data = pipe.stdout.read(w * h * 3)
        if not data:
            break
        yield np.frombuffer(data, dtype="uint8").reshape((3, h, w))

Then there is a function that later uses a Pytorch model and the generator defined above to make inference on video frames and what I want here is to populate batch with video frames and then call the model once to get predictions for the whole batch to utilize the power of gpu:
def inference_step(img_generator: Iterable, model: nn.Module, batch_size: int) -> None:
    inputs = []
    for i, img in enumerate(img_generator):
        if len(inputs) < batch_size:
            inputs.append(torch.from_numpy(img))
            # append is slow so one can use inputs[i % batch_size]
            continue
        outputs = model(inputs)
        inputs = []
        ...

So, my question is how to adapt this code so that it loads the frames in parallel, but at the same time preserves its order?


title: Getting the weight of a Layer

body:
I've been working on the MNIST data set using PyTorch and I am having trouble in accessing the weights and biases that is generated in my code.
This is my code
from torch import nn
import torch.nn.functional as F

class Neural(nn.Module):
  def __init__(self):
      super().__init__()

      self.hidden1 = nn.Linear(784,128)
      self.hidden2 = nn.Linear(128,64)
      self.output  = nn.Linear(64,10)

  def forward(self,x):
      x=F.relu(self.hidden1(x))
      x=F.relu(self.hidden2(x))
      x=F.softmax(self.output(x))
      return x

model= Neural()

and to access the weight when i use
print(model.fc1.weight)
print(model.fc1.bias)

this is the error i get
AttributeError                            Traceback (most recent call last)
<ipython-input-58-e92de631c798> in <module>()
----> 1 model.fc1.weight
      2 print(model.fc1.bias)

/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py in __getattr__(self, name)
    530                 return modules[name]
    531         raise AttributeError("'{}' object has no attribute '{}'".format(
--> 532             type(self).__name__, name))
    533
    534     def __setattr__(self, name, value):

AttributeError: 'Neural' object has no attribute 'fc1'


title: Is there a way that feeding a input to layers parallelly in PyTorch?

body:
I'm implementing ELMo with PyTorch. I want to feed ELMo's CNNs, which have different filter map sizes, on word matrices but I'm considering an efficent way. Here's my code:
# fill the empty tensor iteratively
batch_size = word.size(0)
y = torch.zeros(batch_size, self.kernel_dim)

cnt = 0

for kernel in self.kernels:
    temp = kernel(word)
    pooled = torch.max(temp, dim=2)[0]
    y[:, cnt:cnt+pooled.size(1)] = pooled
    cnt += pooled.size(1)

# Using torch.cat
y = []
for kernel in kernels:
    temp = kernel(a)
    y.append(torch.max(temp, dim=2)[0]) # max pooling
    y = torch.cat(y, dim=1)

I have two questions. The first one is "Is there a way that feeding input to layers in parallel?". It makes me avoiding for loop iteration so that make my code more efficient. The second one is "Which one is faster between using torch.cat and just filling an empty tensor.


title: How does one install torchmeta using conda?

body:
I wanted to install torchmeta with conda because it's giving me some other isses as here: How does one install torchmeta for a ppc64le architecture in pytorch?
so I hope by removing the pip variable I have less things that might be causing my errors...

related:

https://github.com/tristandeleu/pytorch-meta/issues/63


title: Mismatch of output dimensions and whats reported from model summary in pytorch

body:
I am struggeling to determine a cause for following behavior of 3 class classification code I have below. The classifier takes a 2D matrix of torch.Size([1, 1, 512, 512]) . The output is torch.Size([256, 3]) .
class Classifier(nn.Module):
    def __init__(self):
        super(Classifier, self).__init__()
        self.layer1 = nn.Sequential(
            nn.Conv2d(1, 32, kernel_size=5, stride=1, padding=2),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2))
        self.layer2 = nn.Sequential(
            nn.Conv2d(32, 64, kernel_size=5, stride=1, padding=2),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2))
        self.drop_out = nn.Dropout()
        self.fc1 = nn.Linear(4096, 1024)
        self.fc2 = nn.Linear(1024, 3)

    def forward(self, x):
        out = self.layer1(x)
        out = self.layer2(out)
        out = out.view(-1, 4096)
        out = self.drop_out(out)
        out = self.fc1(out)
        out = self.fc2(out)
        return out

The problem is same model provides following dimensions when model summary is printed. My problem is I need to compare this with a torch.Size([1]) where the class labels are stored. Everything is showing that this is designed for 3 class classification problem but cannot determine where this value of 256 in the output tensor comes from.
summary(activity_recognizer,(1,512,512))
==========================================================================================
Layer (type:depth-idx)                   Output Shape              Param #
==========================================================================================
├─Sequential: 1-1                        [-1, 32, 256, 256]        --
|    └─Conv2d: 2-1                       [-1, 32, 512, 512]        832
|    └─ReLU: 2-2                         [-1, 32, 512, 512]        --
|    └─MaxPool2d: 2-3                    [-1, 32, 256, 256]        --
├─Sequential: 1-2                        [-1, 64, 128, 128]        --
|    └─Conv2d: 2-4                       [-1, 64, 256, 256]        51,264
|    └─ReLU: 2-5                         [-1, 64, 256, 256]        --
|    └─MaxPool2d: 2-6                    [-1, 64, 128, 128]        --
├─Dropout: 1-3                           [-1, 4096]                --
├─Linear: 1-4                            [-1, 1024]                4,195,328
├─Linear: 1-5                            [-1, 3]                   3,075
==========================================================================================
Total params: 4,250,499
Trainable params: 4,250,499
Non-trainable params: 0
Total mult-adds (G): 3.57
==========================================================================================
Input size (MB): 1.00
Forward/backward pass size (MB): 96.01
Params size (MB): 16.21
Estimated Total Size (MB): 113.22
==========================================================================================

Spend few hours but seems like I hit a deadend. Appreciate any help I can have here.


title: Trying to use Tensorboard on Google Colab

body:
The page below gives informations about Tensorboard:
https://pytorch.org/docs/stable/tensorboard.html
I am using Google Colab and when i write the following
instructions(which are in the link above):
!pip install tensorboard
tensorboard --logdir=runs

it sends me the following error message:
 File "<ipython-input-111-949c7e8e565e>", line 2
tensorboard --logdir=runs
                         ^
SyntaxError: can't assign to operator

so when i copy paste their
own example:
from torch.utils.tensorboard import SummaryWriter
writer = SummaryWriter()
x = range(100)
for i in x:
    writer.add_scalar('y=2x', i * 2, i)
writer.close()

it does not return the expected graph.
Could someone help me fix this problem?Thank you in advance!


title: Broken Pipe Error while using Loading Text Data in Pytorch

body:
I'm trying to preprocess some text data but after creating a pytorch dataloader, and looping through to check if it is working, I get a Broken Pipe Error. However when trying again in Google Colab, the code works so I think it may be a problem with my setup.
(The Collate class is useless, I just haven't removed it yet.)
import numpy as np
import pandas as pd

data = pd.read_csv("imdb.csv")

import torch
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F
from torch.utils.data import Dataset, DataLoader

import spacy
spacy_eng = spacy.load("en")

class Vocabulary():
    def __init__(self, freq_threshold=4):
        self.word_to_index = {"<PAD>":0, "<SOS>":1, "<EOS>":2, "<UNK>":3}

        self.freq_threshold = freq_threshold
        self.max_length = 0

    def __len__(self):
        return len(self.word_to_index)

    @staticmethod
    def tokenizer_eng(text):
        return [tok.text.lower() for tok in spacy_eng.tokenizer(text)]

    def build_vocabulary(self, sentence_list):
        frequencies = {}
        idx = 4

        longest_length = 0

        for sentence in sentence_list:
            if len(sentence) > longest_length:
                self.max_length = len(sentence)
                longest_length = self.max_length

            for word in self.tokenizer_eng(sentence):
                if word not in frequencies:
                    frequencies[word] = 1
                else:
                    frequencies[word] += 1

                if frequencies[word] == self.freq_threshold:
                    self.word_to_index[word] = idx
                    idx += 1

        self.max_length += 25

    def numericalize(self, text):
        tokenized_text = self.tokenizer_eng(text)

        vector_text = []

        for token in tokenized_text:
            if token in self.word_to_index:
                vector_text.append(self.word_to_index[token])
            else:
                vector_text.append(self.word_to_index["<UNK>"])

        vector_text.append(self.word_to_index["<EOS>"])

        pad_length = self.max_length - len(vector_text)
        for i in range(0, pad_length):
            vector_text.append(self.word_to_index["<PAD>"])

        return vector_text

class IMDBDataset(Dataset):
    def __init__(self):
        data = pd.read_csv("imdb.csv").to_numpy()

        self.target = []
        for data_point in data[:, 2]:
            if data_point == "neg":
                self.target.append(0)
            else:
                self.target.append(1)

        self.text = data[:, 4]

        self.vocab = Vocabulary()
        self.vocab.build_vocabulary(self.text)

    def __len__(self):
        return self.text.shape[0]

    def __getitem__(self, idx):
        review = self.text[idx]

        vector_text = [self.vocab.word_to_index["<SOS>"]]
        vector_text += self.vocab.numericalize(review)

        target = self.target[idx]

        return torch.tensor(vector_text), torch.tensor(target)

class Collate:
    def __init__(self, pad_idx):
        self.pad_idx = pad_idx

    def __call__(self, batch):
        text = [item[0] for item in batch]
        text = nn.utils.rnn.pad_sequence(text, batch_first=False, padding_value=self.pad_idx)

        return text, batch[1]

def get_loader(batch_size=32, num_workers=4, shuffle=True, pin_memory=True):
    dataset = IMDBDataset()
    pad_idx = dataset.vocab.word_to_index["<PAD>"]

    loader = DataLoader(
        dataset=dataset,
        batch_size=batch_size,
        num_workers=num_workers,
        shuffle=shuffle,
        pin_memory=pin_memory,
        collate_fn=Collate(pad_idx=pad_idx) # Redundant now
    )

    return loader, dataset

train_dl, train_ds = get_loader()

for idx, (data, target) in  enumerate(train_dl):
    print(data.shape)


title: Pytorch CrossEntropyLoss from single dimensional Tensors

body:
I am missing something here: why does CrossEntropyLoss not work with single dimensional Tensors?
from  torch  import Tensor
X =Tensor([1.0,2.0,3.0])
labs = Tensor([2,2,3])
loss = nn.CrossEntropyLoss().forward(X,labs)

_stacklevel, dtype)
   1315         dim = _get_softmax_dim('log_softmax', input.dim(), _stacklevel)
   1316     if dtype is None:
-> 1317         ret = input.log_softmax(dim)
   1318     else:
   1319         ret = input.log_softmax(dim, dtype=dtype)

IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

Why does that fail and what should be changed to get the desired result?


title: Inreproducible loss/results when training bert with same setting for >=2 times

body:
I am using my customized bert script to train a model. However, everything even I keep the same setting for lr, AdamW weight decay and epoch, and run on the same platform (cuda on SageMaker) with same torch (1.5.0) and transformers (2.11.0) versions,  the results still change a lot in terms of the loss. This make my different experiments not comparable.
Can someone who has experienced this before or have any ideas please advice me on what should I do? I really want to solve this inreproducible issue so that I can continue on my experiments. Super appreciated for your help!
Details as below:
For example, if I set epoch = 4, lr = 1e-5, decay for AdamW as 0.01.
For one run I got this result for the first epoch only showing the last complete 100 batches result:
2020-10-19 03:45:29,032 - utils - INFO - | epoch   1 |  1300/ 1320 batches | lr 2.261e-05 | loss 0.267 | Elapsed 0:12:29
2020-10-19 03:45:40,550 - utils - INFO -   Training epoch took: 0:12:41
2020-10-19 03:45:40,550 - utils - INFO - Validating...
2020-10-19 03:46:14,588 - utils - INFO - | loss 0.019 | Elapsed 0:00:34
              precision    recall  f1-score      support
False          0.906472  0.979875  0.941745  2087.000000
True           0.475000  0.152610  0.231003   249.000000
accuracy       0.891695  0.891695  0.891695     0.891695
macro avg      0.690736  0.566243  0.586374  2336.000000
weighted avg   0.860480  0.891695  0.865986  2336.000000
2020-10-19 03:46:15,403 - utils - INFO - Testing...
2020-10-19 03:46:55,182 - utils - INFO - use model: 1 batch / 1319 step
              precision  recall  f1-score   support
False             0.906   0.984     0.944  2344.000
True              0.413   0.098     0.159   265.000
accuracy          0.894   0.894     0.894     0.894
macro avg         0.659   0.541     0.551  2609.000
weighted avg      0.856   0.894     0.864  2609.000
2020-10-19 03:46:55,188 - utils - INFO - best test F1 score: 0.8638224640164368

And for the second attempt I got this for the first epoch:
2020-11-07 17:08:08,821 - utils - INFO - | epoch   1 |  1300/ 1320 batches | lr 2.261e-05 | loss 0.286 | Elapsed 0:12:25
2020-11-07 17:08:20,487 - utils - INFO -   Training epoch took: 0:12:37
2020-11-07 17:08:20,487 - utils - INFO - Validating...
2020-11-07 17:08:54,609 - utils - INFO - | loss 0.018 | Elapsed 0:00:34
              precision    recall  f1-score      support
False          0.893408  1.000000  0.943703  2087.000000
True           0.000000  0.000000  0.000000   249.000000
accuracy       0.893408  0.893408  0.893408     0.893408
macro avg      0.446704  0.500000  0.471852  2336.000000
weighted avg   0.798177  0.893408  0.843112  2336.000000
2020-11-07 17:08:55,313 - utils - INFO - Testing...
2020-11-07 17:09:34,934 - utils - INFO - use model: 1 batch / 1319 step
              precision  recall  f1-score   support
False             0.898   1.000     0.946  2344.000
True              0.000   0.000     0.000   265.000
accuracy          0.898   0.898     0.898     0.898
macro avg         0.449   0.500     0.473  2609.000
weighted avg      0.807   0.898     0.850  2609.000
2020-11-07 17:09:34,938 - utils - INFO - best test F1 score: 0.8503599608647853

Note that, the last used lr rate per 100 batches are the same, while the average loss per 100 batches are slightly different. But this result in the predictions for the validation and testing data set very different.
I already set the seed during my model with this function below:
def set_seed(seed):
    """ Set all seeds to make results reproducible (deterministic mode).
        When seed is a false-y value or not supplied, disables deterministic mode. """
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False

And my model script is like this:
class ReviewClassification(BertPreTrainedModel):
    def __init__(self, config,
                 add_agent_text, agent_text_heads):
        """
        :param config: Bert configuration, can set up some parameters, like  output_attention, output_hidden_states
        :param add_agent_text: whether to use the non text feature, and how.
                It can have three options: None, "concat" and "attention"
        :param agent_text_heads: number of the heads in agent attention mechanism. Only useful if add_agent_text are set to
                "attention"
        """
        super().__init__(config)
        # self.num_labels = 2
        self.add_agent_text = add_agent_text

        self.bert = BertModel(config)
        self.dropout = nn.Dropout(config.hidden_dropout_prob)

        embedding_size = config.hidden_size

        if self.add_agent_text == "concat":
            embedding_size = 2 * embedding_size
        elif self.add_agent_text == "attention":
            self.agent_attention = nn.MultiheadAttention(embedding_size, num_heads=agent_text_heads)
        else:
            # don't use the information in Agent text
            pass

        self.classifier = nn.Linear(embedding_size, 1) # self.classifier = nn.Linear(embedding_size, len(LABEL_NAME)) # bias: If set to False, the layer will not learn an additive bias
        self.init_weights()

        print(
            """
            add agent text         :{}
            agent text multi-head  :{}
            """.format(self.add_agent_text, agent_text_heads)
        )

    def forward(
            self,
            review_input_ids=None,
            review_attention_mask=None,
            review_token_type_ids=None,
            agent_input_ids=None,
            agent_attention_mask=None,
            agent_token_type_ids=None,
            labels=None,
    ):

        review_outputs = self.bert(
            review_input_ids,
            attention_mask=review_attention_mask,
            token_type_ids=review_token_type_ids,
            position_ids=None,
            head_mask=None,
            inputs_embeds=None,
        )
        if self.add_agent_text is not None:
            # means that self.add_agent_text is "concat" or "attention"
            # TODO: we can try that agent_outputs do not share the same parameter
            agent_outputs = self.bert(
                agent_input_ids,
                attention_mask=agent_attention_mask,
                token_type_ids=agent_token_type_ids,
                position_ids=None,
                head_mask=None,
                inputs_embeds=None,
            )

        if self.add_agent_text == "attention":
            review_hidden_states = review_outputs[0].transpose(0, 1)  # before trans: (bs, seq_len, hidden_size)

            # want to take it as query, we need the it has the shape (#target_seq_len, batch_size, embedding_size)
            agent_hidden_states = agent_outputs[0].mean(axis=1).unsqueeze(dim=0)  # (1, batch_size, hidden_size)

            attn_output, _ = self.agent_attention(agent_hidden_states, review_hidden_states, review_hidden_states)
            feature = attn_output.squeeze()  # (batch_size, seq_len)
        else:
            feature = review_outputs[1]  # (batch_size, seq_len) -? Should it be (batch_size, hidden_size)

        if self.add_agent_text == "concat":
            feature = torch.cat([feature, agent_outputs[1]], axis=1)

        logits = self.classifier(feature).squeeze()

        outputs = (logits,)  # + outputs[2:]  # add hidden states and attention if they are here

        if labels is not None:
            loss_fct = nn.BCEWithLogitsLoss().cuda() #pos_weight=pos_weight
            loss = loss_fct(logits, labels)
            outputs = (loss,) + outputs

        return outputs  # (loss, logits, hidden_states, attentions)

The loss is calculated using BCEWithLogitsLoss() from torch.nn.
The train, validation and test part script is as below:
import time
import pickle
from path import Path
import numpy as np
import pandas as pd

from sklearn.metrics import precision_recall_fscore_support, classification_report, confusion_matrix
import torch
import torch.nn as nn

from utils import LABEL_NAME, isnotebook, set_seed, format_time

if isnotebook():
    from tqdm.notebook import tqdm
else:
    from tqdm import tqdm

def model_train(model, train_data_loader, valid_data_loader, test_data_loader,
                logger, optimizer, scheduler, num_epochs, seed, out_dir):
    # move model to gpu
    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
    model.to(device)
    if torch.cuda.device_count() > 1:
        model = nn.DataParallel(model)

    num_gpus = torch.cuda.device_count()
    logger.info("Let's use {} GPUs!".format(num_gpus))

    # Set the seed value all over the place to make this reproducible.
    set_seed(seed=seed)

    # We'll store a number of quantities such as training and validation loss,
    # validation accuracy, and timings.
    training_stats = []
    print_interval = 100

    # Measure the total training time for the whole run.
    total_t0 = time.time()
    batch_size = train_data_loader.batch_size
    num_batch = len(train_data_loader)
    best_f1_score = {
        "weighted": 0,
        "averaged": 0
    }
    best_test_f1_score = 0

    # For each epoch...
    for epoch_i in range(0, num_epochs):

        # ========================================
        #               Training
        # ========================================

        # Perform one full pass over the training set.
        logger.info("")
        logger.info('======== Epoch {:} / {:} ========'.format(epoch_i + 1, num_epochs))
        logger.info('Training...')

        # Reset the total loss for this epoch.
        total_train_loss = 0

        # Measure how long the training epoch takes.
        t_train = time.time()

        model.train()

        # For each batch of training data...
        for step, batch in tqdm(enumerate(train_data_loader), desc="Training Iteration", total=num_batch):
            # Progress update every 100 batches.
            if step % print_interval == 0 and not step == 0:
                # Calculate elapsed time in minutes.
                elapsed = format_time(time.time() - t_train)
                avg_train_loss = total_train_loss / print_interval

                # Report progress.
                logger.info('| epoch {:3d} | {:5d}/{:5d} batches | lr {:.3e} | loss {:5.3f} | Elapsed {:s}'.format(
                    epoch_i+1, step, num_batch, scheduler.get_last_lr()[0], avg_train_loss, elapsed)
                )
                total_train_loss = 0
                training_stats.append(
                    {
                        'epoch': epoch_i + 1,
                        'step': step,
                        'train loss': avg_train_loss,
                    }
                )

            # Unpack this training batch from our dataloader.
            #
            # As we unpack the batch, we'll also copy each tensor to the GPU using the
            # `to` method.
            #
            # `batch` contains four pytorch tensors:
            #   "input_ids"
            #   "attention_mask"
            #   "token_type_ids"
            #   "binarized_labels"

            b_review_input_ids = batch["review_input_ids"].to(device)
            b_review_attention_mask = batch["review_attention_mask"].to(device)
            b_review_token_type_ids = batch["review_token_type_ids"].to(device)
            b_agent_input_ids = batch["agent_input_ids"].to(device)
            b_agent_attention_mask = batch["agent_attention_mask"].to(device)
            b_agent_token_type_ids = batch["agent_token_type_ids"].to(device)

            b_binarized_label = batch["binarized_label"].to(device)

            model.zero_grad()
            (loss, _) = model(review_input_ids=b_review_input_ids,
                              review_attention_mask=b_review_attention_mask,
                              review_token_type_ids=b_review_token_type_ids,
                              agent_input_ids=b_agent_input_ids,
                              agent_attention_mask=b_agent_attention_mask,
                              agent_token_type_ids=b_agent_token_type_ids,

                              labels=b_binarized_label
                              )

            # Accumulate the training loss over all of the batches so that we can
            # calculate the average loss at the end. `loss` is a Tensor containing a
            # single value; the `.item()` function just returns the Python value
            # from the tensor.

            if num_gpus > 1:
                total_train_loss += loss.mean().item()
                loss.mean().backward()  # use loss.mean().backward() instead of loss.backward() for multiple gpu trainings
            else:
                total_train_loss += loss.item()
                loss.backward()

            # Clip the norm of the gradients to 1.0.
            # This is to help prevent the "exploding gradients" problem.
            torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)

            # Update parameters and take a step using the computed gradient.
            # The optimizer dictates the "update rule"--how the parameters are
            # modified based on their gradients, the learning rate, etc.
            optimizer.step()
            scheduler.step()
        # End of training epoch

        # Measure how long this epoch took.
        training_time = format_time(time.time() - t_train)

        logger.info("")
        logger.info("  Training epoch took: {:s}".format(training_time))

        # evaluate the model after one epoch.

        # ========================================
        #               Validation
        # ========================================
        # After the completion of each training epoch, measure our performance on
        # our validation set.

        logger.info("")
        logger.info("Validating...")

        t_valid = time.time()
        model.eval()
        ave_valid_loss, valid_f1_table, cm_table, f1_score = model_validate(model=model, data_loader=valid_data_loader)
        # Measure how long this epoch took.
        validation_time = format_time(time.time() - t_valid)

        logger.info("")
        logger.info('| loss {:5.3f} | Elapsed {:s}'.format(ave_valid_loss, validation_time))
        logger.info("  \n{:s}".format(valid_f1_table.to_string()))
        logger.info("")
        logger.info("  \n{:s}".format(cm_table.to_string()))

        # need to store the best model
        for key in best_f1_score.keys():
            if best_f1_score[key] < f1_score[key]:
                # remove the old model:
                file_list = [f for f in out_dir.files() if f.name.endswith(".pt") and f.name.startswith(key)]
                for f in file_list:
                    Path.remove(f)
                model_file = out_dir.joinpath('{:s}_epoch_{:02d}-f1_{:.3f}.pt'.format(
                    key, epoch_i + 1, f1_score[key])
                )
                best_f1_score[key] = f1_score[key]
                if num_gpus > 1:
                    torch.save(model.module.state_dict(), model_file)
                else:
                    torch.save(model.state_dict(), model_file)

        # ========================================
        #               Test
        # ========================================
        logger.info("")
        logger.info("Testing...")

        result_df = model_test(model=model, data_loader=test_data_loader)

        y_true = np.array(result_df["review_label"], dtype=np.bool) # This part may need double check
        y_pred = result_df["Probability"] > 0.5

        report = classification_report(y_true, y_pred, output_dict=True)
        metrics_df = pd.DataFrame(report).transpose()

        metrics_df = metrics_df.sort_index()

        weighted_f1_score = metrics_df.loc['weighted avg', 'f1-score']
        averaged_f1_score = metrics_df.loc['macro avg', 'f1-score']

        best_test_f1_score = metrics_df.loc['weighted avg', 'f1-score'] \
            if best_test_f1_score < metrics_df.loc['weighted avg', 'f1-score'] else best_test_f1_score

        metrics_df = metrics_df.astype(float).round(3)

        # Calculate confusion matrix
        tn, fp, fn, tp  = confusion_matrix(y_true, y_pred).ravel()
        cm_df = pd.DataFrame(columns = ['Predicted No', 'Predicted Yes'],
                       index = ['Actual No', 'Actual Yes'])
        # adding rows to an empty
        # dataframe at existing index
        cm_df.loc['Actual No'] = [tn,fp]
        cm_df.loc['Actual Yes'] = [fn,tp]

        logger.info("use model: {} batch / {} step".format(epoch_i + 1, step))
        logger.info("\n" + "=" * 50)
        logger.info("\n" + metrics_df.to_string())
        logger.info("\n" + "=" * 50)
        logger.info("\n" + cm_df.to_string())
        logger.info("best test F1 score: {}".format(best_test_f1_score))
        logger.info("\n" + "=" * 50)
        # Below is to save the result files
        result_filename = "result_df_epoch_" + str(epoch_i + 1) + ".xlsx"
        result_df.to_excel(out_dir.joinpath(result_filename), index=False)

    logger.info("")
    logger.info("Training complete!")
    logger.info("Total training took {:} (h:mm:ss)".format(format_time(time.time() - total_t0)))

    # Save training_stats to csv file
    pd.DataFrame(training_stats).to_csv(out_dir.joinpath("model_train.log"), index=False)
    return model, optimizer, scheduler

def model_validate(model, data_loader):
    # Put the model in evaluation mode--the dropout layers behave differently
    # during evaluation.
    model.eval()
    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
    model.to(device)
    if torch.cuda.device_count() > 1:
        model = nn.DataParallel(model)

    label_prop = data_loader.dataset.dataset.label_prop()

    total_valid_loss = 0

    batch_size = data_loader.batch_size
    num_batch = len(data_loader)

    y_pred, y_true = [], []

    # Evaluate data
    for step, batch in tqdm(enumerate(data_loader), desc="Validation...", total=num_batch):
        b_review_input_ids = batch["review_input_ids"].to(device)
        b_review_attention_mask = batch["review_attention_mask"].to(device)
        b_review_token_type_ids = batch["review_token_type_ids"].to(device)
        b_agent_input_ids = batch["agent_input_ids"].to(device)
        b_agent_attention_mask = batch["agent_attention_mask"].to(device)
        b_agent_token_type_ids = batch["agent_token_type_ids"].to(device)

        b_binarized_label = batch["binarized_label"].to(device)

        # Tell pytorch not to bother with constructing the compute graph during
        # the forward pass, since this is only needed for backprop (training).
        with torch.no_grad():
            (loss, logits,) = model(review_input_ids=b_review_input_ids,
                                    review_attention_mask=b_review_attention_mask,
                                    review_token_type_ids=b_review_token_type_ids,
                                    agent_input_ids=b_agent_input_ids,
                                    agent_attention_mask=b_agent_attention_mask,
                                    agent_token_type_ids=b_agent_token_type_ids,

                                    labels=b_binarized_label)

        total_valid_loss += loss.item()
        ### The sigmoid function is used for the two-class logistic regression,
        ### whereas the softmax function is used for the multiclass logistic regression

        # Version 1
        # numpy_probas = logits.detach().cpu().numpy()
        # y_pred.extend(np.argmax(numpy_probas, axis=1).flatten())
        # y_true.extend(b_binarized_label.cpu().numpy())

        # Version 2
        # transfored_logits = F.log_softmax(logits,dim=1)
        # numpy_probas = transfored_logits.detach().cpu().numpy()
        # y_pred.extend(np.argmax(numpy_probas, axis=1).flatten())
        # y_true.extend(b_binarized_label.cpu().numpy())

        # Version 3
        # transfored_logits = torch.sigmoid(logits)
        # numpy_probas = transfored_logits.detach().cpu().numpy()
        # y_pred.extend(np.argmax(numpy_probas, axis=1).flatten())
        # y_true.extend(b_binarized_label.cpu().numpy())

        # New version - for num_label = 1
        transfored_logits = torch.sigmoid(logits)
        numpy_probas = transfored_logits.detach().cpu().numpy()
        y_pred.extend(numpy_probas)
        y_true.extend(b_binarized_label.cpu().numpy())

    # End of an epoch of validation

    # put model to train mode again.
    model.train()

    ave_loss = total_valid_loss / (num_batch * batch_size)

    y_pred = np.array(y_pred)
    y_pred[y_pred < 0.5] = 0
    y_pred[y_pred >= 0.5] = 1

    # Below is in case the input and target are not the same data format
    y_pred = np.array(y_pred, dtype=np.bool)
    y_true = np.array(y_true, dtype=np.bool)

    # compute the various f1 score for each label
    report = classification_report(y_true, y_pred, output_dict=True)
    metrics_df = pd.DataFrame(report).transpose()
    # metrics_df = pd.DataFrame(0, index=LABEL_NAME, columns=["Precision", "Recall", "F1","support"])
    # metrics_df.Precision = precision_recall_fscore_support(y_true, y_pred)[0]
    # metrics_df.Recall = precision_recall_fscore_support(y_true, y_pred)[1]
    # metrics_df.F1 = precision_recall_fscore_support(y_true, y_pred)[2]
    # metrics_df.support = precision_recall_fscore_support(y_true, y_pred)[3]

    # y_pred = np.array(y_pred)
    # y_pred[y_pred < 0] = 0
    # y_pred[y_pred > 0] = 1
    # y_pred = np.array(y_pred, dtype=np.bool)
    # y_true = np.array(y_true, dtype=np.bool)

    # metrics_df = pd.DataFrame(0, index=LABEL_NAME, columns=["Precision", "Recall", "F1"], dtype=np.float)
    # # or_y_pred = np.zeros(y_pred.shape[0], dtype=np.bool)
    # # or_y_true = np.zeros(y_true.shape[0], dtype=np.bool)
    # for i in range(len(LABEL_NAME)):
    #     metrics_df.iloc[i] = precision_recall_fscore_support(
    #         y_true=y_true[:, i], y_pred=y_pred[:, i], average='binary', zero_division=0)[0:3]

        # or_y_pred = or_y_pred | y_pred[:, i]
        # or_y_true = or_y_true | y_true[:, i]

    metrics_df = metrics_df.sort_index()
    # metrics_df.loc['Weighted Average'] = metrics_df.transpose().dot(label_prop)
    # metrics_df.loc['Average'] = metrics_df.mean()

    # metrics_df.loc['Weighted Average', 'F1'] = 2 / (1/metrics_df.loc['Weighted Average', "Recall"] +
    #                                                 1/metrics_df.loc['Weighted Average', "Precision"])
    # metrics_df.loc['Average', 'F1'] = 2 / (1/metrics_df.loc['Average', "Recall"] +
    #                                        1/metrics_df.loc['Average', "Precision"])

    weighted_f1_score = metrics_df.loc['weighted avg', 'f1-score']
    averaged_f1_score = metrics_df.loc['macro avg', 'f1-score']

    # Calculate confusion matrix
    tn, fp, fn, tp  = confusion_matrix(y_true, y_pred).ravel()
    cm_df = pd.DataFrame(columns = ['Predicted No', 'Predicted Yes'],
                   index = ['Actual No', 'Actual Yes'])
    # adding rows to an empty
    # dataframe at existing index
    cm_df.loc['Actual No'] = [tn,fp]
    cm_df.loc['Actual Yes'] = [fn,tp]

    # pooled_f1_score = f1_score(y_pred=or_y_pred, y_true=or_y_true)

    return ave_loss, metrics_df, cm_df,{
        "weighted": weighted_f1_score,
        "averaged": averaged_f1_score,
    }

def model_test(model, data_loader):
    # Put the model in evaluation mode--the dropout layers behave differently
    # during evaluation.
    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
    model.eval()
    model.to(device)
    if torch.cuda.device_count() > 1:
        model = nn.DataParallel(model)

    num_batch = len(data_loader)
    # Below need to modify if change the input
    review_id, review_label, hmd_text, head_cust_text = [], [], [], []
    agent = []
    pred_logits = []

    # Evaluate data
    for step, batch in tqdm(enumerate(data_loader), desc="Inference...", total=num_batch):
        if "anecdote_lead_final" in batch.keys():
            review_label.extend(batch["anecdote_lead_final"])
        review_id.extend(batch["_id"].tolist())
        hmd_text.extend(batch["hmd_comments"])
        head_cust_text.extend(batch["head_cust"])
        agent.extend(batch["new_transcript_agent"])

        b_review_input_ids = batch["review_input_ids"].to(device)
        b_review_attention_mask = batch["review_attention_mask"].to(device)
        b_review_token_type_ids = batch["review_token_type_ids"].to(device)
        b_agent_input_ids = batch["agent_input_ids"].to(device)
        b_agent_attention_mask = batch["agent_attention_mask"].to(device)
        b_agent_token_type_ids = batch["agent_token_type_ids"].to(device)

        # Tell pytorch not to bother with constructing the compute graph during
        # the forward pass, since this is only needed for backprop (training).
        with torch.no_grad():
            (logits,) = model(review_input_ids=b_review_input_ids,
                              review_token_type_ids=b_review_token_type_ids,
                              review_attention_mask=b_review_attention_mask,
                              agent_input_ids=b_agent_input_ids,
                              agent_token_type_ids=b_agent_token_type_ids,
                              agent_attention_mask=b_agent_attention_mask
                              )

        if logits.detach().cpu().numpy().size == 1:
            pred_logits.extend(logits.detach().cpu().numpy().reshape(1,))
        else:
            pred_logits.extend(logits.detach().cpu().numpy())

    # End of an epoch of validation
    # put model to train mode again.
    model.train()
    pred_logits = np.array(pred_logits)
    pred_prob = np.exp(pred_logits)
    pred_prob = pred_prob / (1 + pred_prob)
    pred_label = pred_prob.copy()
    pred_label[pred_label < 0.5] = 0
    pred_label[pred_label >= 0.5] = 1
    # compute the f1 score for each tags
    d = {'Probability':pred_prob,'Anecdotes Prediction':pred_label}
    pred_df = pd.DataFrame(d, columns=['Probability','Anecdotes Prediction'])
    result_df = pd.DataFrame(
        {
            "review_id": review_id,
            "hmd_text": hmd_text,
            "head_cust_text": head_cust_text,
            "agent": agent
        }
    )
    if len(review_label) != 0:
        result_df["review_label"] =  [x.item() for x in review_label]
    return pd.concat([result_df, pred_df], axis=1).set_index("review_id")

optimizer and scheduler part are defined as below:
if args.full_finetuning:
    param_optimizer = list(model.named_parameters())
    no_decay = ['bias', 'gamma', 'beta']
    optimizer_grouped_parameters = [
        {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)],
         'weight_decay_rate': args.decay},
        {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)],
         'weight_decay_rate': 0.0}
    ]
else:
    param_optimizer = list(model.classifier.named_parameters())
    optimizer_grouped_parameters = [
        {"params": [p for n, p in param_optimizer]}
    ]

# Note: AdamW is a class from the huggingface library (as opposed to pytorch)
optimizer = AdamW(optimizer_grouped_parameters,  # or param_optimizer
                  lr=args.lr,  # args.learning_rate - default is 5e-5, our notebook had 1e-5
                  eps=1e-8)  # args.adam_epsilon  - default is 1e-8.

# Create the learning rate scheduler.
scheduler = get_linear_schedule_with_warmup(
    optimizer,
    num_warmup_steps=0,
    num_training_steps=len(data_loader["train"]) * args.num_epochs
)

And to run the model, I use below script:
model_train(model=model, train_data_loader=data_loader["train"], valid_data_loader=data_loader["valid"],
            test_data_loader=data_loader["test"], optimizer=optimizer, scheduler=scheduler,
            num_epochs=args.num_epochs, seed=args.seed, logger=logger, out_dir=out_dir)


title: Pytorch affine grid and grid sample preserve aspect ration during rotation

body:
I am using affine grid and grid sample to rotate an image by 90 degrees. This is the original image

This is the output of grid sample. The aspect ratio is not maintained.

This is the expected output with padding set to zeros.

Is there anything else that needs to be considered?