title: Calculate the shape in fully connected layer after cnn

body:
I am trying to learn PyTorch. But I am really confused about the shape in a fully connected layer after convolution and max pooling.

Case 1. How we calculate the number 5408 in nn.Linear
I think 5408 = 32 * m * m, where 32 comes from the nn.Conv2d(3, **32**, kernel_size=7, stride=2), but then m would equals 13. However, where is the 13 comes from?

simple_model = nn.Sequential(
                nn.Conv2d(3, 32, kernel_size=7, stride=2),
                nn.ReLU(inplace=True),
                Flatten(), # see above for explanation
                nn.Linear(5408, 10), # affine layer
              )

Case 2 How we get the number 4*4 in fc = nn.Linear(64*4*4, 10), the same problem in case 1. I don't know where the number 4 comes from...

# (conv -> batchnorm -> relu -> maxpool) * 3 -> fc

layer1 = nn.Sequential(
    nn.Conv2d(3, 16, kernel_size = 5, padding = 2),
    nn.BatchNorm2d(16),
    nn.ReLU(),
    nn.MaxPool2d(2)
)
layer2 = nn.Sequential(
    nn.Conv2d(16, 32, kernel_size = 3, padding = 1),
    nn.BatchNorm2d(32),
    nn.ReLU(),
    nn.MaxPool2d(2)
)
layer3 = nn.Sequential(
    nn.Conv2d(32, 64, kernel_size = 3, padding = 1),
    nn.BatchNorm2d(64),
    nn.ReLU(),
    nn.MaxPool2d(2)
)
fc = nn.Linear(64*4*4, 10)


I am trying to run SSD-pytorch on google colabe and it is showing this error:

File
"/usr/local/lib/python3.6/dist-packages/torch/autograd/function.py",
line 145, in call "Legacy autograd function with non-static forward
method is deprecated. " RuntimeError: Legacy autograd function with
non-static forward method is deprecated. Please use new-style autograd
function with static forward method.

Example
How to solve it?


title: Error with F1 score: AttributeError: 'float' object has no attribute 'item'

body:
I try to train NN for binary labels, here is my data:
 Train: {'0': 126315, '1': 2915}
    Val  : {'0': 31579, '1': 729}
    Test : {'0': 27864, '1': 643}

Here is my weights:
tensor([7.9167e-06, 3.4305e-04], device='cuda:0')

I get this warnings:
C:\Users\___\venv\lib\site-packages\torch\utils\data\sampler.py:115: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  self.weights = torch.tensor(weights, dtype=torch.double)
NN_FullyConnected(
  (layer_1): Linear(in_features=611, out_features=512, bias=True)
  (layer_2): Linear(in_features=512, out_features=256, bias=True)
  (layer_3): Linear(in_features=256, out_features=128, bias=True)
  (layer_4): Linear(in_features=128, out_features=64, bias=True)
  (layer_5): Linear(in_features=64, out_features=32, bias=True)
  (layer_out): Linear(in_features=32, out_features=2, bias=True)
  (relu): ReLU()
  (dropout): Dropout(p=0.45)
  (batchnorm1): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (batchnorm2): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (batchnorm3): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (batchnorm4): BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (batchnorm5): BatchNorm1d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)

Begin training.
C:\Users\__\PycharmProjects\myproject\network_creator_nn.py:50: UserWarning: Implicit dimension choice for log_softmax has been deprecated. Change the call to include dim=X as an argument.
  y = nn.LogSoftmax()(x)
C:\Users\__\PycharmProjects\__\venv\lib\site-packages\sklearn\metrics\classification.py:1145: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 due to no true samples.
  'recall', 'true', average, warn_for)

Epoch 000: | Train Loss: 0.66666 | Val Loss: 0.68134 | Train Acc: 0.835| Val Acc: 0.498 | Train F_score: 4.892| Val F_score: 7.191
2020-07-14 08:47:07
Epoch 001: | Train Loss: 0.48715 | Val Loss: 0.73657 | Train Acc: 0.850| Val Acc: 0.040 | Train F_score: 18.034| Val F_score: 4.398

Traceback (most recent call last):
  File "C:/Users/__/PycharmProjects/__/complete_proccess.py", line 239, in <module>
    val_epoch_f_score += val_f_score.item()
AttributeError: 'float' object has no attribute 'item'

Why I got this error after 2 epochs? my F1 score function is:
def binary_f1_score(y_pred, y_test):
    y_pred_tag = torch.log_softmax(y_pred, dim=1)
    _, y_pred_tags = torch.max(y_pred_tag, dim=1)
    f_result = f1_score(y_test.cpu().numpy(), y_pred_tags.cpu().numpy())
    f_result = f_result * 100
    return f_result

If I add to the f1 score function the argument "macro" it will works well, but I don't know why
Thank you


title: How to make custom code in python utilize GPU while using Pytorch tensors and matrice functions

body:
I've created a CNN from scratch only using Pytorch tensors and matrix operation functions in the hope of utilizing GPU. To my surprise, the GPU stays 0% utilized and my training doesn't seem to be faster than running on my cpu.
Before Training:

While Training:

I've double checked whether CUDA is available and have installed it already.
Graphics card: Nvidia GEFORCE 2070 SUPER
Processor: Intel i5 10400
Coding Environment: Jupyter Notebook
Cuda & Cudnn Version: 11.0
Pytorch version: 1.6.0


title: pytorch: the dropout layer after LayerNorm, There are some magical phenomena

body:
When I add a dropout layer after LayerNorm，the validation set loss reduction at 1.5 epoch firstly，then the loss Substantially increase，and the acc becomes 0; when I remove the dropout layer, it works; when I remove the layernorm, it changes , not zero, but results was very poor.
the model code:
class Model(nn.Module):
    def __init__(self, config):
        super(Model, self).__init__()
        if config.embedding_pretrained is not None:
            #self.embedding = nn.Embedding.from_pretrained(config.embedding_pretrained, freeze=False)
            self.embedding = nn.Embedding.from_pretrained(torch.FloatTensor(config.embedding_pretrained.vectors), freeze=False)
        else:
            self.embedding = nn.Embedding(config.n_vocab, config.embed, padding_idx=config.n_vocab - 1)
        self.lstm = nn.LSTM(config.embed, config.hidden_size, config.num_layers,
                            bidirectional=True, batch_first=True, dropout=config.dropout)
        self.maxpool = nn.MaxPool1d(config.pad_size)
        self.fc = nn.Linear(config.hidden_size * 2 + config.embed, config.num_classes)
        self.dropout = nn.Dropout(config.dropout)

        self.ln_emb = nn.LayerNorm(config.embed)
        self.ln_lstm = nn.LayerNorm(config.hidden_size * 2)

    def forward(self, x):
        x, _ = x
        embed = self.embedding(x)  # [batch_size, seq_len, embeding]
        embed = self.ln_emb(embed)
        out, _ = self.lstm(embed)
        out = self.ln_lstm(out)

        out = torch.cat((embed, out), 2)
        out = self.dropout(out)
        #out = F.relu(out)
        out = out.permute(0, 2, 1)
        out = self.maxpool(out).squeeze()
        out = self.fc(out)
        return out

the acc tensorboard:

the loss tensorboard:


I want to classify cat and dog with pytorch. So I downloaded dataset from Kaggle, and seperate train/validate set. I changed the file name from 00001.jpg to cat.00001.jpg..
But when I try to use enumerate(dataset), this error occurs:
My dataset code is:
class TrainImageFolder(Dataset):
    def __init__(self, path, transform=None):
        self.transform = transform
        self.path = path
        self.image = []
        self.label = []
        for i in os.listdir(self.path):
            self.image.append(i)
            if i.startswith("cat"):
                self.label.append(0)
            elif i.startswith("dog"):
                self.label.append(1)
        assert len(self.label) == len(self.image)

    def __len__(self):
        len(self.image)

    def __getitem__(self, index):
        label = self.label[index]
        img = Image.open(self.image[index]).convert("RGB")
        if self.transform:
            img = self.transform(img)
        return img, label

train_transform = transforms.Compose([transforms.Resize((224, 224)),
                                transforms.RandomHorizontalFlip(),
                                transforms.ToTensor(),
                                transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])])

train_dataset = TrainImageFolder('train', transform=train_transform)
train_dataloader = torch.utils.data.DataLoader(train_dataset, batch_size=32, shuffle=False)

for i, (imgs, labels) in tqdm(enumerate(train_dataloader)):
    print(labels)

and error is:
Traceback (most recent call last):
  File "C:/Users/ge971/PycharmProjects/myVGG16/dataset.py", line 148, in <module>
    for i, (imgs, labels) in tqdm(enumerate(train_dataloader)):
  File "C:\Users\ge971\miniconda3\envs\torch17\lib\site-packages\tqdm\std.py", line 1166, in __iter__
    for obj in iterable:
  File "C:\Users\ge971\miniconda3\envs\torch17\lib\site-packages\torch\utils\data\dataloader.py", line 435, in __next__
    data = self._next_data()
  File "C:\Users\ge971\miniconda3\envs\torch17\lib\site-packages\torch\utils\data\dataloader.py", line 474, in _next_data
    index = self._next_index()  # may raise StopIteration
  File "C:\Users\ge971\miniconda3\envs\torch17\lib\site-packages\torch\utils\data\dataloader.py", line 427, in _next_index
    return next(self._sampler_iter)  # may raise StopIteration
  File "C:\Users\ge971\miniconda3\envs\torch17\lib\site-packages\torch\utils\data\sampler.py", line 227, in __iter__
    for idx in self.sampler:
  File "C:\Users\ge971\miniconda3\envs\torch17\lib\site-packages\torch\utils\data\sampler.py", line 67, in __iter__
    return iter(range(len(self.data_source)))
TypeError: 'NoneType' object cannot be interpreted as an integer

Process finished with exit code 1

Could you let me know how to fix the error? Please help me


title: Is Feature Scaling recommended for AutoEncoder?

body:
Problem:

The Staked Auto Encoder is being applied to a dataset with 25K rows and 18 columns, all float values.
SAE is used for feature extraction with encoding & decoding.

When I train the model without feature scaling, the loss is around 50K, even after 200 epochs. But, when scaling is applied the loss is around 3 from the first epoch.

My questions:

Is it recommended to apply feature scaling when SAE is used for feature extraction
Does it impact accuracy during decoding?


title: Why does training accuracy fall after reaching perfect training fit?

body:
I am training a NN in pytorch on MNIST data. The model start well, improves, reaches good accuracy for both training and testing data, stabilizes for a while and then both the testing and training accuracy collapse, as shown on the following base results graph.

As for MNIST, I use 60000 training images, 10000 testing, training batch size 100 and learning rate 0.01. Neural network consist of two fully connected hidden layers, each with 100 nodes, nodes having ReLU activation functions. F.cross_entropy is used for loss and SGD for gradients calculations.

This is not the over-fitting problem, as it is both the training and testing accuracy which collapse. I suspected that it has something to do with too large learning rate. In the base case I have used 0.01 but when I lower it to 0.001 the whole pattern repeats, just later, as shown on the following graph (please notice x-axis scale change, the pattern is happening roughly 10 times later, which is intuitive). Similar results were obtained using even lower learning rates.

I have tried unit testing, checking the individual parts and making the model smaller. Here are results when I am using only 6 data points in the training set, batch size 2. Perfect fit on training data (here clearly different from the test accuracy, as expected) is unsurprisingly reached, but it still collapses from 100% to 1/6 so no better than a random pick. What needs to happen for the network to spin out from a perfect fit on the training set, can anybody tell me?

Here is the structure of the network (the relevant libraries are added before), although I hope that the aforementioned symptoms will be sufficient for you to recognize what the problem is without it:

class Network(nn.Module):
def __init__(self):
    # call to the super class Module from nn
    super(Network, self).__init__()

    # fc strand for 'fully connected'
    self.fc1 = nn.Linear(in_features=28*28, out_features=100)
    self.fc2 = nn.Linear(in_features=100, out_features=100)
    self.out = nn.Linear(in_features=100, out_features=10)

def forward(self, t):

    # (1) input layer (redundant)
    t = t

    # (2) hidden linear layer
    # As my t consists of 28*28 bit pictures, I need to flatten them:
    t = t.reshape(-1, 28*28)
    # Now having this reshaped input, add it to the linear layer
    t = self.fc1(t)
    # Again, apply ReLU as the activation function
    t = F.relu(t)

    # (3) hidden linear layer
    # As above, but reshaping is not needed now
    t = self.fc2(t)
    t = F.relu(t)

    # (4) output layer
    t = self.out(t)
    t = F.softmax(t, dim=1)

    return t

The main execution of the code:

for b in range(epochs):
print('***** EPOCH NO. ', b+1)
# getting a batch iterator
batch_iterator = iter(batch_train_loader)
# For loop for a single epoch, based on the length of the training set and the batch size
for a in range(round(train_size/b_size)):
    print(a+1)
    # get one batch for the iteration
    batch = next(batch_iterator)
    # decomposing a batch
    images, labels = batch[0].to(device), batch[1].to(device)
    # to get a prediction, as with individual layers, we need to equate it to the network with the samples as input:
    preds = network(images)
    # with the predictions, we will use F to get the loss as cross_entropy
    loss = F.cross_entropy(preds, labels)
    # function for counting the number of correct predictions
    get_num_correct(preds, labels))
    # calculate the gradients needed for update of weights
    loss.backward()
    # with the known gradients, we will update the weights according to stochastic gradient descent
    optimizer = optim.SGD(network.parameters(), lr=learning_rate)
    # with the known weights, step in the direction of correct estimation
    optimizer.step()
    # check if the whole data check should be performed (for taking full training/test data checks only in evenly spaced intervals on the log scale, pre-calculated later)
    if counter in X_log:
        # get the result on the whole train data and record them
        full_train_preds = network(full_train_images)
        full_train_loss = F.cross_entropy(full_train_preds, full_train_labels)
        # Record train loss
        a_train_loss.append(full_train_loss.item())
        # Get a proportion of correct estimates, to make them comparable between train and test data
        full_train_num_correct = get_num_correct(full_train_preds, full_train_labels)/train_size
        # Record train accuracy
        a_train_num_correct.append(full_train_num_correct)
        print('Correct predictions of the dataset:', full_train_num_correct)
        # Repeat for test predictions
        # get the results for the whole test data
        full_test_preds = network(full_test_images)
        full_test_loss = F.cross_entropy(full_test_preds, full_test_labels)
        a_test_loss.append(full_test_loss.item())
        full_test_num_correct = get_num_correct(full_test_preds, full_test_labels)/test_size
        a_test_num_correct.append(full_test_num_correct)
    # update counter
    counter = counter + 1

I have googled and checked here for answers to this questions but people either ask about over-fitting or their NNs do not increase accuracy on the training set at all (i.e. they simply don't work), not about finding a good training fit and then completely loosing it, also on the training set. I hope I did not post something obvious, I am relatively new to NN but I did my best to research the topic before posting it here, thank you for your help and understanding!


title: Error in Converting Pytorch model to Keras using Pytorch2Keras

body:
I am using Pytorch2Keras to convert Signature Verification
(https://github.com/luizgh/sigver) available in Pytorch to Keras using following code

from pytorch2keras.converter import pytorch_to_keras
state_dict, _, _ = torch.load('models/signet_f_lambda_0.95.pth')
base_model = SigNet().to(device).eval()
input_np = np.random.uniform(0, 1, (1,1,150,220))
input_var = Variable(torch.FloatTensor(input_np))
k_model = pytorch_to_keras(base_model, input_var, [(1, 150, 220)], verbose=True)

I am getting the following Error

TypeError                                 Traceback (most recent call last)
----> 6 k_model = pytorch_to_keras(base_model, input_var, [(1, 150, 220)], verbose=True)

/pytorch2keras/converter.py in pytorch_to_keras(model, args, input_shapes, change_ordering, verbose, name_policy)
 71     k_model = onnx_to_keras(onnx_model=onnx_model, input_names=input_names,
 72                             input_shapes=input_shapes, name_policy=name_policy,
---> 73                             verbose=verbose, change_ordering=change_ordering)
 74
 75     return k_model

/onnx2keras/converter.py in onnx_to_keras(onnx_model, input_names, input_shapes, name_policy, verbose, change_ordering)
144             node_params,
145             layers,
--> 146             node_name
147         )
148

/onnx2keras/reshape_layers.py in convert_unsqueeze(node, params, layers, node_name)
151         layers[node_name] = np.expand_dims(layers[node.input[0]], params['axes'][0])
152     else:
--> 153         if len(params['axes'][0]) != 0:
154             raise AttributeError('Axes is not 0. Cannot unsqueeze')
155

 TypeError: object of type 'int' has no len()

Help me to solve the issue


title: Use numpy in script class (torch.jit.script)

body:
I was wondering if I can use numpy APIs in a function which is going to be scripted by torch.jit.script. I have this simple function which does not work:

import torch
import torch.nn as nn

class MyModule(nn.Module):

def __init__(self):
    super(MyModule, self).__init__()

@torch.jit.ignore
def call_np():
    return torch.jit.export(np.random.choice(2, p=[.95,.05]))

def forward(self):
    pass

@torch.jit.export
def func(self):
    done = self.call_np()
    print (done)

  scripted_module = torch.jit.script(MyModule())
  scripted_module.func()

which results in:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-133-ab1ce37d6edc> in <module>()
     18         print (done)
     19
---> 20 scripted_module = torch.jit.script(MyModule())
     21 scripted_module.func()

C:\ProgramData\Anaconda3\lib\site-packages\torch\jit\__init__.py in script(obj, optimize, _frames_up, _rcb)
   1201
   1202     if isinstance(obj, torch.nn.Module):
-> 1203         return torch.jit.torch.jit._recursive.recursive_script(obj)
   1204
   1205     qualified_name = _qualified_name(obj)

C:\ProgramData\Anaconda3\lib\site-packages\torch\jit\_recursive.py in recursive_script(mod, exclude_methods)
    171     filtered_methods = filter(ignore_overloaded, methods)
    172     stubs = list(map(make_stub, filtered_methods))
--> 173     return copy_to_script_module(mod, overload_stubs + stubs)
    174
    175

C:\ProgramData\Anaconda3\lib\site-packages\torch\jit\_recursive.py in copy_to_script_module(original, stubs)
     93             setattr(script_module, name, item)
     94
---> 95     torch.jit._create_methods_from_stubs(script_module, stubs)
     96
     97     # Now that methods have been compiled, take methods that have been compiled

C:\ProgramData\Anaconda3\lib\site-packages\torch\jit\__init__.py in _create_methods_from_stubs(self, stubs)
   1421     rcbs = [m.resolution_callback for m in stubs]
   1422     defaults = [get_default_args(m.original_method) for m in stubs]
-> 1423     self._c._create_methods(self, defs, rcbs, defaults)
   1424
   1425 # For each user-defined class that subclasses ScriptModule this meta-class,

RuntimeError: Unable to cast Python instance of type <class 'int'> to C++ type 'unsigned __int64'

I appreciate any help or comment.


title: Size of the training data of GPT2-XL pre-trained model

body: In huggingface transformer, it is possible to use the pre-trained GPT2-XL language model. But I don't find, on which dataset it is trained? Is it the same trained model which OpenAI used for their paper (trained on 40GB dataset called webtext) ?


title: Albumentations RandomCrop with mask different size as image

body:
is it possible to RandomCrop an image with the size 256x256 and the mask with the size 100x100?

Or RandomGridShuffle and RandomSizedCrop?

https://albumentations.readthedocs.io/en/latest/api/augmentations.html

Thank you


title: Reduce the number of hidden units in hugging face transformers (BERT)

body:
I have been given a large csv each line of which is a set of BERT tokens made with hugging face BertTokenizer (https://huggingface.co/transformers/main_classes/tokenizer.html).
1 line of this file looks as follows:

101, 108, 31278, 90939, 70325, 196, 199, 71436, 10107, 29190, 10107, 106, 16680, 68314, 10153, 17015, 15934, 10104, 108, 10233, 12396, 14945, 10107, 10858, 11405, 13600, 13597, 169, 57343, 64482, 119, 119, 119, 100, 11741, 16381, 10109, 68830, 10110, 20886, 108, 10233, 11127, 21768, 100, 14120, 131, 120, 120, 188, 119, 11170, 120, 12132, 10884, 10157, 11490, 12022, 10113, 10731, 10729, 11565, 14120, 131, 120, 120, 188, 119, 11170, 120, 162, 11211, 11703, 12022, 11211, 10240, 44466, 100886, 102

and there are 9 million lines like this

Now, I am trying to get embeddings from these tokens like this:

def embedding:
    tokenizer = BertTokenizer.from_pretrained('bert-base-multilingual-cased', do_lower_case=False)
    model = BertModel.from_pretrained('bert-base-multilingual-cased')
    input_ids = torch.tensor([101, 108, 31278, 90939, 70325, 196, 199, 71436, 10107, 29190, 10107, 106, 16680, 68314, 10153, 17015, 15934, 10104, 108, 10233, 12396, 14945, 10107, 10858, 11405, 13600, 13597, 169, 57343, 64482, 119, 119, 119, 100, 11741, 16381, 10109, 68830, 10110, 20886, 108, 10233, 11127, 21768, 100, 14120, 131, 120, 120, 188, 119, 11170, 120, 12132, 10884, 10157, 11490, 12022, 10113, 10731, 10729, 11565, 14120, 131, 120, 120, 188, 119, 11170, 120, 162, 11211, 11703, 12022, 11211, 10240, 44466, 100886, 102]).unsqueeze(0)  # Batch size 1
    outputs = model(input_ids)
    last_hidden_states = outputs[0][0][0]  # The last hidden-state is the first element of the output tuple

Output of this is embedding correspending to the line. The size is 768*1 tensor. Semantically, everything is ok. But, when I do this for the full file the output is 768 * 9,0000,000 torch tensors. So I get a memory error even with large machine with a 768 GB of RAM.
Here is how I call this function:
tokens['embeddings'] = tokens['text_tokens'].apply(lambda x: embedding(x))

tokens is the pandas data frame with 9 million lines each of which contains BERT tokens.

Is it possible to reduce the default size of the hidden units, which is 768 here: https://huggingface.co/transformers/main_classes/model.html

Thank you for your help.


title: My train accuracy remains at 10% when I add weight_decay parameter to my optimizer in PyTorch. I am using CIFAR10 dataset and LeNet CNN model

body:
I am training CIFAR10 dataset on LeNet CNN model. I am using PyTorch on Google Colab. The code runs only when I use Adam optimizer with model.parameters() as the only parameter. But when I change my optimizer or use weight_decay parameter then the accuracy remains at 10% through all the epochs. I cannot understand the reason why it is happening.

# CNN Model - LeNet
class LeNet_ReLU(nn.Module):
    def __init__(self):
        super().__init__()
        self.cnn_model = nn.Sequential(nn.Conv2d(3,6,5),
                                       nn.ReLU(),
                                       nn.AvgPool2d(2, stride=2),
                                       nn.Conv2d(6,16,5),
                                       nn.ReLU(),
                                       nn.AvgPool2d(2, stride=2))
        self.fc_model = nn.Sequential(nn.Linear(400, 120),
                                      nn.ReLU(),
                                      nn.Linear(120,84),
                                      nn.ReLU(),
                                      nn.Linear(84,10))

    def forward(self, x):
        x = self.cnn_model(x)
        x = x.view(x.size(0), -1)
        x = self.fc_model(x)
        return x

# Importing dataset and creating dataloader
batch_size = 128
trainset = torchvision.datasets.CIFAR10(root='./data', train=True, download=True,
                                    transform=transforms.ToTensor())
trainloader = utils_data.DataLoader(trainset, batch_size=batch_size, shuffle=True)
testset = torchvision.datasets.CIFAR10(root='./data', train=False, download=True,
                                    transform=transforms.ToTensor())
testloader = utils_data.DataLoader(testset, batch_size=batch_size, shuffle=False)

# Creating instance of the model
net = LeNet_ReLU()

# Evaluation function
def evaluation(dataloader):
    total, correct = 0, 0
    for data in dataloader:
        inputs, labels = data

        outputs = net(inputs)
        _, pred = torch.max(outputs.data, 1)
        total += labels.size(0)
        correct += (pred==labels).sum().item()
    return correct/total * 100

# Loss function and optimizer
loss_fn = nn.CrossEntropyLoss()
opt = optim.Adam(net.parameters(), weight_decay = 0.9)

# Model training
loss_epoch_arr = []
max_epochs = 16

for epoch in range(max_epochs):
    for i, data in enumerate(trainloader, 0):
        inputs, labels = data

        outputs = net(inputs)
        loss = loss_fn(outputs, labels)
        loss.backward()
        opt.step()

        opt.zero_grad()

    loss_epoch_arr.append(loss.item())

    print('Epoch: %d/%d, Test acc: %0.2f, Train acc: %0.2f'
    % (epoch,max_epochs, evaluation(testloader), evaluation(trainloader)))

plt.plot(loss_epoch_arr)


title: A Problem with Pytorch-Trying to backward through the graph a second time, but the buffers have already been freed

body:
This is the code

# The definition of some value
batch_size = 2
seq_len = 35
hidden_num = 32
vocab_len = len(corpus_id_index)

# The definition of the RNNmodule
class RNN(nn.Module):
    def __init__(self):
        super(RNN, self).__init__()
        self.rnn = nn.RNN(input_size=vocab_len,hidden_size=hidden_num,batch_first=True)
        self.dense = nn.Linear(hidden_num,vocab_len)
        self.hn = None
    def forward(self, x, hn):
        x = F.one_hot(x,vocab_len)
        x = x.float()
        output,self.hn = self.rnn(x,hn)
        output = self.dense(output)
        return output,self.hn

#The process of training model
model = RNN()
hn = torch.zeros(1,seq_len,hidden_num)
epochs = 250
clip_value = 100
loss = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(),lr=0.001)

for epoch in range(epochs):
    for x,y in data_collect(corpus_indice,batch_size,seq_len):
        output,hn = model(x,hn)
        y = y.transpose(1,0).contiguous().view(-1)
        ls = loss(output.view(-1,vocab_len),y)

        optimizer.zero_grad()
        ls.backward()

        torch.nn.utils.clip_grad_value_(model.parameters(), clip_value)

        optimizer.step()

This is the error information.

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-15-f3c48893b07e> in <module>
     14
     15         optimizer.zero_grad()
---> 16         ls.backward()
     17
     18         torch.nn.utils.clip_grad_value_(model.parameters(), clip_value)

~/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph)
    196                 products. Defaults to ``False``.
    197         """
--> 198         torch.autograd.backward(self, gradient, retain_graph, create_graph)
    199
    200     def register_hook(self, hook):

~/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
     98     Variable._execution_engine.run_backward(
     99         tensors, grad_tensors, retain_graph, create_graph,
--> 100         allow_unreachable=True)  # allow_unreachable flag
    101
    102

RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.

At first,I can train the neural networks correctly，but after I change the size of the hn, the strange thing happen,the Jupyter Notebook begin to report error,then I change back the hn and restart the Notebook kernel，but it still report the same error. I want to know why it reports this error.I wish someone can help me.Thanks a lot.


title: Numpy/PyTorch method for partial tiling

body:
I seem to recall encountering a Numpy or PyTorch method similar to numpy.tile, except that it allowed partial tiling to reach specified dimensions. So if I had

a = np.array([[5, 6, 7, 8],
              [1, 2, 3, 4]])

(or, correspondingly, t = torch.Tensor(a) if this is a PyTorch method),

then

a.mystery_method((3,4))

would produce output

array([[5, 6, 7, 8],
       [1, 2, 3, 4],
       [5, 6, 7, 8]])

Unfortunately, I now can't find this method in the Numpy or Pytorch docs. Does it actually exist, and what is it if so?


title: How can I change this GPU code in CPU code?

body:
All.

This is my simple GPU code in PyTorch.

How can I change this in CPU code?

filename=r’./test/bees/1.jpg’
img = skimage.io.imread(filename)

x = V(centre_crop(img).unsqueeze(0), volatile=True).cuda()

model = models.dict(’resnet18’)
model = torch.nn.DataParallel(model).cuda()

model = torch.load(‘model5.pth’)

logit = model(x)
print(logit)

Thanks in advance .


title: Am I using polynomial regression right in pytorch?

body:
I am trying to do polynomial regression by pytorch. First I just tried only linear regression (b + wx).

model_1 = RegressionModel()
W = torch.zeros(1, requires_grad=True)
b = torch.zeros(1, requires_grad = True)
optimizer_1 = torch.optim.SGD([W, b], lr = 0.001)

x_train = torch.FloatTensor(dataset.x_data['LSTAT'])
y_train = torch.FloatTensor(dataset.data['target'])

nb_epochs = 10000

for epoch in range(nb_epochs + 1):
    hypothesis = x_train * W + b
    cost = torch.nn.functional.mse_loss(hypothesis, y_train.float())

    optimizer_1.zero_grad()
    cost.backward()
    optimizer_1.step()

    print('Epoch {:4d}/{} W: {:.3f}, b: {:.3f}, Cost: {:.6f}'.format(epoch,
    nb_epochs, W.item(), b.item(), cost.item()))

Then I changed and added some variables to do polynomial regression (b + w1x + w2x^2)

model_2 = RegressionModel()
W1 = torch.zeros(1, requires_grad=True)
W2 = torch.zeros(1, requires_grad=True)
b = torch.zeros(1, requires_grad = True)
optimizer_2 = torch.optim.SGD([W2, W1, b], lr = 0.0000099)

x_train = torch.FloatTensor(dataset.x_data['LSTAT'])
y_train = torch.FloatTensor(dataset.data['target'])

nb_epochs = 10000

for epoch in range(nb_epochs + 1):
    hypothesis = b + x_train * W1 + x_train * x_train * W2
    cost = torch.nn.functional.mse_loss(hypothesis, y_train.float())

    optimizer_2.zero_grad()
    cost.backward()
    optimizer_2.step()

    print('Epoch {:4d}/{} W1: {:.3f}, W2: {:.3f}, b: {:.3f}, Cost:
    {:.6f}'.format(epoch, nb_epochs, W1.item(), W2.item(), b.item(),
    cost.item()))

Can I try polynomial regression like this? If not, I would be really appreciate if you let me know. I'm really noob to pytorch...


title: Model behaves differently after saving and loading

body:
I want to use torch.save() to save a trained model for inference. However, with either torch.load_state_dict() or torch.load(), I can't get the saved model. The loss computed by the loaded model is just different from the loss computed by the saved model.

The relevant Libraries:

import numpy as np
import torch
import torch.nn as nn
from torch.autograd import Variable
from torch.nn import functional as F

The model:

class nn_block(nn.Module):

    def __init__(self, feats_dim):

        super(nn_block, self).__init__()

        self.linear = nn.Linear(feats_dim, feats_dim)
        self.bn = nn.BatchNorm1d(feats_dim)
        self.softplus1 = nn.Softplus()
        self.softplus2 = nn.Softplus()

    def forward(self, rep_mat):

        transformed_mat = self.linear(rep_mat)
        transformed_mat = self.bn(transformed_mat)
        transformed_mat = self.softplus1(transformed_mat)
        transformed_mat = self.softplus2(transformed_mat + rep_mat)

        return transformed_mat

class test_nn(nn.Module):

    def __init__(self, in_feats, feats_dim, num_conv, num_classes):

        super(test_nn, self).__init__()

        self.linear1 = nn.Linear(in_feats, feats_dim)
        self.convs = [nn_block(feats_dim) for _ in range(num_conv)]
        self.linear2 = nn.Linear(feats_dim, num_classes)
        self.softmax = nn.Softmax()

    def forward(self, rep_mat):

        h = self.linear1(rep_mat)
        for conv_func in self.convs:
            h = conv_func(h)
        h = self.linear2(h)
        h = self.softmax(h)

        return h

Train, save, and reload a model:

# fake a classification task
num_classes = 2; input_dim = 8
one = np.random.multivariate_normal(np.zeros(input_dim),np.eye(input_dim),20)
two = np.random.multivariate_normal(np.ones(input_dim),np.eye(input_dim),20)
inputs = np.concatenate([one, two], axis=0)
labels = np.concatenate([np.zeros(20), np.ones(20)])

inputs = Variable(torch.Tensor(inputs))
labels = torch.LongTensor(labels)

# build a model
net = test_nn(input_dim, 5, 2, num_classes)
optimizer = torch.optim.Adam(net.parameters(), lr=0.01)
net.train()

losses = []
best_score = 1e10
for epoch in range(25):

    preds = net(inputs)
    loss = F.cross_entropy(preds, labels)

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    state_dict = {'state_dict': net.state_dict()}

    if loss.item()-best_score<-1e-4:
        # save only parameters
        torch.save(state_dict, 'model_params.torch')
        # save the whole model
        torch.save(net, 'whole_model.torch')
    best_score = np.min([best_score, loss.item()])

    losses.append(loss.item())

net_params = test_nn(input_dim, 5, 2, num_classes)
net_params.load_state_dict(torch.load('model_params.torch')['state_dict'])
net_params.eval()
preds_params = net_params(inputs)
loss_params = F.cross_entropy(preds_params, labels)
print('reloaded params %.4f %.4f' % (loss_params.item(), np.min(losses)))

net_whole = torch.load('whole_model.torch')
net_whole.eval()
preds_whole = net_whole(inputs)
loss_whole = F.cross_entropy(preds_whole, labels)
print('reloaded whole %.4f %.4f' % (loss_whole.item(), np.min(losses)))

As you can see by running the code, the losses computed by the two loaded models are different, while the two loaded models are exactly the same. Not just the two losses are different, they are also different from the loss computed by the best model that was saved in the first place.

Why this can happen?


title: PyTorch Dataset / Dataloader batching

body:
I'm a little confused regarding the 'best practise' to implement a PyTorch data pipeline on time series data.

I have a HD5 file which I read using a custom DataLoader. It seems that I should return the data samples as a (features,targets) tuple with the shape of each being (L,C) where L is seq_len and C is number of channels - i.e. don't preform batching in the data loader, just return as a table.

PyTorch modules seem to require a batch dim, i.e. Conv1D expects (N, C, L).

I was under the impression that the DataLoader class would prepend the batch dimension but it isn't, I'm getting data shaped (N,L).

dataset = HD5Dataset(args.dataset)

dataloader = DataLoader(dataset,
                        batch_size=N,
                        shuffle=True,
                        pin_memory=is_cuda,
                        num_workers=num_workers)

for i, (x, y) in enumerate(train_dataloader):
    ...

In the code above the shape of x is (N,C) not (1,N,C), which results in the code below (from a public git repo) to fail on the first line.

def forward(self, x):
    """expected input shape is (N, L, C)"""
    x = x.transpose(1, 2).contiguous() # input should have dimension (N, C, L)

The documentation states When automatic batching is enabled It always prepends a new dimension as the batch dimension which leads me to believe that automatic batching is disabled but I don't understand why?


title: How to assign labels from DataFrame to PyTorch ImageFolder Dataset

body:
I am loading images from folder using ImageFolder, and i want to assign labels to images from DataFrame.
DataFrame contains a image name in folder and label to image
All images in folder sorted by alphabet.
My code:
def load_data(data_path, targets):
    train_data = torchvision.datasets.ImageFolder(
        root=data_path,
        transform=torchvision.transforms.ToTensor()
)
    train_data.classes = ['FAKE', 'REAL']
    train_data.class_to_idx = {k: v for k, v in zip(train_data.classes, [0, 1])}
    train_data.targets = targets
    return train_data

metadata = metadata.sort_values(by='image_name') # metadata it's my dataframe with labels
targets = np.array(metadata.label.astype('category').cat.codes)
# And i'm try to reshape my array with labels
# targets = targets.reshape(targets.shape[0], 1)
targets = torch.from_numpy(targets)

train_dataset = load_data('../input/deepfake-faces/', targets)

For example, in my dataset expected output of this code is should be 'Label: 1', but i'm get 0.
img, label = train_dataset[10]
print('Label: ', label)
plt.imshow(img.permute(1, 2, 0))

But, when i'm count a images with label 1, i get a correct result.
np.count_nonzero(train_dataset.targets == 1)


title: Best Loss Function for Multi-Class Multi-Target Classification Problem

body:
I have a classification problem and I don't know how to categorize this classification problem. As per my understanding,

A Multiclass classification problem is where you have multiple mutually exclusive classes and each data point in the dataset can only be labelled by one class. For example, in an Image Classification task for fruits, a fruit data point labelled as an apple cannot be an orange and an orange cannot be a banana and so on. Each data point, in this case can only be any one of the fruits of the fruits class and so is labelled accordingly.

Where as ...

A Multilabel classification is a problem where you have multiple sets of mutually exclusive classes of which the data point can be labelled simultaneously. For example, in an Image Classification task for Cars, a car data point labelled as a sedan cannot be a hatchback and a hatchback cannot be a SUV and so on for the type of car. At the same time, the same car data point can be labelled one from VW, Ford, Mercedes, etc. as the car manufacturer. So in this case, the car data point is labeled from two different  sets of mutually exclusive classes.

Please correct my understanding if I am thinking in a wrong way here.
Now to my problem, my classification problem with multiple classes, lets say A, B, C, D and E. Here each data point can have one or more than one classes from the set as shown below on the left:
|-------------|----------|              |-------------|-----------------|
|      X      |     y    |              |      X      |    One-Hot-Y    |
|-------------|----------|              |-------------|-----------------|
|     DP1     |   A, B   |              |     DP1     | [1, 1, 0, 0, 0] |
|-------------|----------|              |-------------|-----------------|
|     DP2     |   C      |              |     DP2     | [0, 0, 1, 0, 0] |
|-------------|----------|              |-------------|-----------------|
|     DP3     |   B, E   |              |     DP3     | [0, 1, 0, 0, 1] |
|-------------|----------|              |-------------|-----------------|
|     DP4     |   A, C   |              |     DP4     | [1, 0, 1, 0, 0] |
|-------------|----------|              |-------------|-----------------|
|     DP5     |   D      |              |     DP5     | [0, 0, 0, 1, 0] |
|-------------|----------|              |-------------|-----------------|

I One-Hot Encoded the labels for training as shown above on the right. My question is:

What Loss function (preferably in PyTorch) can I use for training the model to optimize for the One-Hot encoded output
What do we call such a classification problem? Multi-label or Multi-class?

Thank you for your answers!


title: Pytorch Loss Function for making embeddings similar

body:
I am working on an embedding model, where there is a BERT model, which takes in text inputs and output a multidimensional vector. The goal of the model is to find similar embeddings (high cosine similarity) for texts which are similar and different embeddings (low cosine similarity) for texts that are dissimilar.
When training in mini-batch mode, the BERT model gives a N*D dimensional output where N is the batch size and D is the output dimension of the BERT model.
Also, I have a target matrix of dimension N*N, which contains 1 in the [i, j] th position if the sentence[i] and sentence[j] are similar in sense and -1 if not.
What I want to do is find the loss/error for the entire batch by finding the cosine similarity of all embeddings in the BERT output and comparing it to the target matrix.
What I did was simply multiply the tensor with its transpose and then take elementwise sigmoid.
scores = torch.matmul(document_embedding, torch.transpose(document_embedding, 0, 1))
scores = torch.sigmoid(scores)

loss = self.bceloss(scores, targets)

But this does not seem to work.
Is there any other way to do this?
P.S. What I want to do is similar to the method described in this paper.


title: how to append a singleton numpy array item into a list?

body:
I am attaching the code first which will give you a better idea

`   prediction = prediction.data.max(1)[1] #gives a tensor value
    prediction = (prediction.cpu().numpy().item()) #converts that tensor into a numpy array
    result.append(int_to_word[prediction])`

I am using pytorch for word generation. The line prediction = prediction.data.max(1)[1] gives us the class label with the maximum probability, which turns out to be a  tensor value.

prediction = (prediction.cpu().numpy().item()) this statement converts the tensor into a numpy array and the funtion item() extracts the value from the array.

Now when I try to append this value using pattern.append(prediction)
I am getting the following error
'numpy.ndarray' object has no attribute 'append'

I am not able to understand as to why I am getting this error. I have already converted the numpy array into a scalr value, didn't I?? Still, why am I getting that error?

Can anybody please explain and provide a solution to this. I would be most grateful.


title: In Torch, module = nn.Padding(dim, pad, nInputDim, value, index), What does the nInputDim do?

body:
Like the documentation stated 'When nInputDim is provided, inputs larger than that value will be considered batches where the actual dim to be padded will be dimension dim + 1.'
My understanding is the inputs dimensional larger than the provided nInputDim, the dimension dim will become dim + 1, but why is it ?


title: pytorch autoencoder model evaluation fail

body:
I am literally a beginner of PyTorch.
I trained an autoencoder network so that I can plot the distribution of the latent vectors (the result of encoders).

This is the code that I used for network training.

import torch
import torchvision
from torch import nn
from torch.utils.data import DataLoader
from torchvision import transforms
from torchvision.utils import save_image
from torch.utils.data import Dataset
from PIL import Image
import os
import glob

dir_img_decoded = '/media/dohyeong/HDD/mouth_autoencoder/dc_img_2'
if not os.path.exists(dir_img_decoded):
    os.mkdir(dir_img_decoded)

dir_check_point = '/media/dohyeong/HDD/mouth_autoencoder/ckpt_2'
if not os.path.exists(dir_check_point):
    os.mkdir(dir_check_point)

device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')

num_epochs = 200
batch_size = 150  # up -> GPU memory increase
learning_rate = 1e-3

dir_dataset = '/media/dohyeong/HDD/mouth_autoencoder/mouth_crop/dir_normalized_mouth_cropped_images'
images = glob.glob(os.path.join(dir_dataset, '*.png'))
train_images = images[:-113]
test_images = images[-113:]

train_images.sort()
test_images.sort()

class TrumpMouthDataset(Dataset):
    def __init__(self, images):
        super(TrumpMouthDataset, self).__init__()
        self.images = images

        self.transform = transforms.Compose([
            # transforms.Resize((28, 28)),
            transforms.ToTensor(),
            transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
        ])

    def __getitem__(self, index):
        image = Image.open(self.images[index])

        return self.transform(image)

    def __len__(self):
        return len(self.images)

train_dataset = TrumpMouthDataset(train_images)
test_dataset = TrumpMouthDataset(test_images)

train_dataloader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
test_dataloader = DataLoader(test_dataset, batch_size=batch_size, shuffle=True)

class Autoencoder(nn.Module):
    def __init__(self):
        super(Autoencoder, self).__init__()
        self.encoder = nn.Sequential(
            nn.Linear(60000, 60),
            nn.ReLU(True),
            nn.Linear(60, 3),
            nn.ReLU(True),
        )
        self.decoder = nn.Sequential(
            nn.Linear(3, 60),
            nn.ReLU(True),
            nn.Linear(60, 60000),
            nn.Tanh()
        )

    def forward(self, x):
        x = x.view(x.size(0), -1)
        encoded = self.encoder(x)
        decoded = self.decoder(encoded)

        return encoded, decoded

model = Autoencoder().cuda()
if torch.cuda.device_count() > 1:
    model = nn.DataParallel(model)
    model.to(device)

criterion = nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(),
                             lr=learning_rate,
                             weight_decay=1e-5)

for epoch in range(num_epochs):

    total_loss = 0

    for index, imgs in enumerate(train_dataloader):
        imgs = imgs.to(device)

        # ===================forward=====================
        outputs = model(imgs)

        imgs_flatten = imgs.view(imgs.size(0), -1)
        loss = criterion(outputs, imgs_flatten)

        # ===================backward====================
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        total_loss += loss.item()

        print('{} Epoch, [{}/{}] batch, loss: {:.4f}'.format(epoch, index + 1, len(train_dataloader), loss.item()))

    avg_loss = total_loss / len(train_dataset)
    print('{} Epoch, avg_loss: {:.4f}'.format(epoch, avg_loss))

    if epoch % 10 == 0:
        check_point_file = os.path.join(dir_check_point, str(epoch) + ".pth")
        torch.save(model.state_dict(), check_point_file)

After training, I tried to get encoded values using this code.

device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')

check_point = '/media/dohyeong/HDD/mouth_autoencoder/290.pth'
model = torch.load(check_point)

for index, imgs in enumerate(train_dataloader):

    imgs = imgs.to(device)

    # ===================evaluate=====================
    encoded, _ = model(imgs)

It finished with this error message.
"TypeError: 'collections.OrderedDict' object is not callable"
May I get some help?


title: Tensor shape mismatch error in PyTorch on MNIST dataset, but no error on synthetic data

body:
I am trying to implement a Deep Learning paper (https://github.com/kiankd/corel2019) and having a weird error when supplying real data (MNIST) to it, but no error when using the same synthetic data as the authors used.
The error happens in this function:

def get_armask(shape, labels, device=None):
    mask = torch.zeros(shape).to(device)
    arr = torch.arange(0, shape[0]).long().to(device)
    mask[arr, labels] = -1.
    return mask

More specifically this line:

mask[arr, labels] = -1.

The error is:

RuntimeError: The shape of the mask [500] at index 0 does not match the shape of the indexed tensor [500, 10] at index 1

The weird thing is, that if I use the synthetic data, there is no error and it works perfectly. If I print out the shapes, I get the following (both with synthetic data and with MNIST):

mask torch.Size([500, 10])
arr torch.Size([500])
labels torch.Size([500])

The code used to generate the synthetic data is the following:

X_data = (torch.rand(N_samples, D_input) * 10.).to(device)
labels = torch.LongTensor([i % N_classes for i in range(N_samples)]).to(device)

While the code to load MNIST is this:

train_images = mnist.train_images()
X_data_all = train_images.reshape((train_images.shape[0], train_images.shape[1] * train_images.shape[2]))
X_data = torch.tensor(X_data_all[:500,:]).to(device)
X_data = X_data.type(torch.FloatTensor)

labels = torch.tensor(mnist.train_labels()[:500]).to(device)

get_armask is used the following way:

def forward(self, predictions, labels):
    mask = get_armask(predictions.shape, labels, device=self.device)

    # make the attractor and repulsor, mask them!
    attraction_tensor = mask * predictions
    repulsion_tensor = (mask + 1) * predictions

    # now, apply the special cosine-COREL rules, taking the argmax and squaring the repulsion
    repulsion_tensor, _ = repulsion_tensor.max(dim=1)
    repulsion_tensor = repulsion_tensor ** 2

    return arloss(attraction_tensor, repulsion_tensor, self.lam)

The actual error seems to be different from what is in the error message, but I have no idea where to look. I tried a few things, like changing the learning rate, normalizing the MNIST data to be more or less in the same range as the test data but nothing seems to work.

Any suggestions? Thanks a lot in advance!


title: What is nuclear norm of a tensor?

body:
When I do

import torch
x = torch.ones(3, 4)
x.norm(p='nuc')

it gives

tensor(3.4641)

how is this value calculated?


title: Is there a memory leak? Pytorch

body:
I am trying to run this code. I get high memory usage such that my ram hits 100% and my PC will freeze for hours. I see little to no gpu usage while monitoring with Nvidia-Smi. The cpu usage is around 20-30%. Can anyone tell me what's wrong? Is the code running on cuda?
class RL_Agents:
    def __init__(self, building_info, observation_spaces=None, action_spaces=None):
# Parameters
        self.device = "cpu"
        self.time_step = 0
        self.building_info = building_info  # Can be used to create different RL agents based on basic building attributes or climate zones
        self.observation_spaces = observation_spaces
        self.action_spaces = action_spaces
        self.n_buildings = len(observation_spaces)
        self.buffer = {i: Buffer() for i in range(self.n_buildings)}
        self.networks_initialized = False


title: Pytorch inference time difference between CUDA 10.0 & 10.2

body:
We have a working library that uses LibTorch 1.5.0, built with CUDA 10.0 which runs as expected.
We are working on upgrading to CUDA 10.2 for various non-PyTorch related reasons. We noticed that when we run LibTorch inference on the newly compiled LibTorch (compiled exactly the same, except changing to CUDA 10.2), the runtime is about 20x slower.
We also checked it using the precompiled binaries.
This was tested on 3 different machines using 3 different GPUs (Tesla T4, GTX980 & P1000) and all gives consistent ~20x slower on CUDA 10.2
(Both on Windows 10 & Ubuntu 16.04), all with the latest drivers and on 3 different torch scripts (of the same architecture)
I've simplified the code to be extremely minimal without external dependencies other than Torch
int main(int argc, char** argv)
{
    // Initialize CUDA device 0
    cudaSetDevice(0);

    std::string networkPath = DEFAULT_TORCH_SCRIPT;
    if (argc > 1)
    {
        networkPath = argv[1];
    }
    auto jitModule = std::make_shared<torch::jit::Module>(torch::jit::load(networkPath, torch::kCUDA));
    if (jitModule == nullptr)
    {
        std::cerr << "Failed creating module" << std::endl;
        return EXIT_FAILURE;
    }

    // Meaningless data, just something to pass to the module to run on
    // PATCH_HEIGHT & WIDTH are defined as 256
    uint8_t* data = new uint8_t[PATCH_HEIGHT * PATCH_WIDTH * 3];
    memset(data, 0, PATCH_HEIGHT * PATCH_WIDTH * 3);

    auto stream = at::cuda::getStreamFromPool(true, 0);

    bool res = infer(jitModule, stream, data, PATCH_WIDTH, PATCH_HEIGHT);

    std::cout << "Warmed up" << std::endl;

    res = infer(jitModule, stream, data, PATCH_WIDTH, PATCH_HEIGHT);

    delete[] data;
    return 0;
}

// Inference function

bool infer(std::shared_ptr<JitModule>& jitModule, at::cuda::CUDAStream& stream, const uint8_t* inputData, int width, int height)
{
    std::vector<torch::jit::IValue> tensorInput;
    // This function simply uses cudaMemcpy to copy to device and create a torch::Tensor from that data
    // I can paste it if it's relevant but didn't now to keep as clean as possible
    if (!prepareInput(inputData, width, height, tensorInput, stream))
    {
        return false;
    }
    // Reduce memory usage, without gradients
    torch::NoGradGuard noGrad;
    {
        at::cuda::CUDAStreamGuard streamGuard(stream);

        auto totalTimeStart = std::chrono::high_resolution_clock::now();

        jitModule->forward(tensorInput);
        // The synchronize here is just for timing sake, not use in production
        cudaStreamSynchronize(stream.stream());

        auto totalTimeStop = std::chrono::high_resolution_clock::now();
        printf("forward sync time = %.3f milliseconds\n",
            std::chrono::duration<double, std::milli>(totalTimeStop - totalTimeStart).count());
    }

    return true;
}

When compiling this with Torch that was compiled using CUDA 10.0 we get a runtime of 18 ms and when we run it with Torch compiled with CUDA 10.2, we get a runtime of 430 ms
Any thoughts on that?

This issue was also posted on PyTorch Forums.

Issue on GitHub

UPDATE
I profiled this small program using both CUDAs
It seems that both use very different kernels
96.5% of the 10.2 computes are conv2d_grouped_direct_kernel which takes ~60-100ms on my P1000
where as the top kernels in the 10.0 run are
47.1% - cudnn::detail::implicit_convolve_sgemm (~1.5 ms)
23.1% - maxwell_scudnn_winograd_128x128_ldg1_ldg4_tile148n_nt (~0.4 ms)
8.5%  - maxwell_scudnn_128x32_relu_small_nn (~0.4ms)

so it's easy to see where the time difference comes from. Now the question is, why.


title: How to reduce the inference time of Helsinki-NLP/opus-mt-es-en (translation model) from transformer

body:
Currently Helsinki-NLP/opus-mt-es-en model takes around 1.5sec for inference from transformer. How can that be reduced?
Also when trying to convert it to onxx runtime getting this error:

ValueError: Unrecognized configuration class <class 'transformers.models.marian.configuration_marian.MarianConfig'> for this kind of AutoModel: AutoModel.
Model type should be one of RetriBertConfig, MT5Config, T5Config, DistilBertConfig, AlbertConfig, CamembertConfig, XLMRobertaConfig, BartConfig, LongformerConfig, RobertaConfig, LayoutLMConfig, SqueezeBertConfig, BertConfig, OpenAIGPTConfig, GPT2Config, MobileBertConfig, TransfoXLConfig, XLNetConfig, FlaubertConfig, FSMTConfig, XLMConfig, CTRLConfig, ElectraConfig, ReformerConfig, FunnelConfig, LxmertConfig, BertGenerationConfig, DebertaConfig, DPRConfig, XLMProphetNetConfig, ProphetNetConfig, MPNetConfig, TapasConfig.

Is it possible to convert this to onxx runtime?


title: Validation and training loss per batch and epoch

body:
I am using Pytorch to run some deep learning models. I am currently keeping track of training and validation loss per epoch, which is pretty standard. However, what is the best way of going about keeping track of training and validation loss per batch/iteration?
For training loss, I could just keep a list of the loss after each training loop. But, validation loss is calculated after a whole epoch, so I’m not sure how to go about the validation loss per batch. The only thing I can think of is to run the whole validation step after each training batch and keeping track of those, but that seems overkill and a lot of computation.
For example, the training is like this:
for epoch in range(2):  # loop over the dataset multiple times
running_loss = 0.0
for i, data in enumerate(trainloader, 0):
    # get the inputs; data is a list of [inputs, labels]
    inputs, labels = data

    # zero the parameter gradients
    optimizer.zero_grad()

    # forward + backward + optimize
    outputs = net(inputs)
    loss = criterion(outputs, labels)
    loss.backward()
    optimizer.step()

    # print statistics
    running_loss += loss.item()

And for validation loss:
with torch.no_grad():
    for data in testloader:
        images, labels = data
        outputs = net(images)
        _, predicted = torch.max(outputs.data, 1)
        total += labels.size(0)
        correct += (predicted == labels).sum().item()
        # validation loss
        batch_loss = error(outputs.float(), labels.long()).item()
        loss_test += batch_loss
    loss_test /= len(testloader)

The validation loss/test part is done per epoch. I’m looking for a way to get the validation loss per batch, which is my point above.
Any tips?


title: RuntimeError: CUDA error: device-side assert triggered on loss function

body:
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [18,0,0], thread: [54,0,0] Assertion input_val >= zero && input_val <= one failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [18,0,0], thread: [55,0,0] Assertion input_val >= zero && input_val <= one failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [18,0,0], thread: [56,0,0] Assertion input_val >= zero && input_val <= one failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [18,0,0], thread: [57,0,0] Assertion input_val >= zero && input_val <= one failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [18,0,0], thread: [58,0,0] Assertion input_val >= zero && input_val <= one failed.
/pytorch/aten/src/ATen/native/cuda/Loss.cu:102: operator(): block: [18,0,0], thread: [59,0,0] Assertion input_val >= zero && input_val <= one failed.
Traceback (most recent call last):
File "run_toys.py", line 215, in
loss = criterion(torch.reshape(out, [-1, dataset.out_dim]), torch.reshape(target, [-1, dataset.out_dim]))
File "/usr/local/python3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/usr/local/python3/lib/python3.6/site-packages/torch/nn/modules/loss.py", line 530, in forward
return F.binary_cross_entropy(input, target, weight=self.weight, reduction=self.reduction)
File "/usr/local/python3/lib/python3.6/site-packages/torch/nn/functional.py", line 2526, in
binary_cross_entropy
input, target, weight, reduction_enum)
RuntimeError: CUDA error: device-side assert triggered
The Code
criterion = nn.CrossEntropyLoss()
loss = criterion(torch.reshape(out, [-1, dataset.out_dim]), torch.reshape(target, [-1, dataset.out_dim]))
loss = torch.mean(loss)

The shape of the target and output is the same # torch.Size([640, 32])
The model runs on my CPU OK, but running on GPU is the issue


title: Passing word2vec embedding to a custom LSTM pytorch model

body:
I have a set of input sentences. I am using the pretrained word2vec model from gensim to get the embedding of the input sentences. I want to pass these embeddings as input to a custom pytorch LSTM model

hidden_size = 32
num_layers = 1
num_classes = 2

class customModel(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers, num_classes):
        super(customModel, self).__init__()
        self.hidden_size = hidden_size
        self.num_layers = num_layers
        self.bilstm = nn.LSTM(input_size, hidden_size, num_layers, batch_first=False, bidirectional=True)
        self.fcl = nn.Linear(hidden_size*2, num_classes)

    def forward(self, x):
        # Set initial hidden and cell states
        h0 = torch.zeros(self.num_layers*2, x.size(0), self.hidden_size).to(device)
        c0 = torch.zeros(self.num_layers*2, x.size(0), self.hidden_size).to(device)

        # Forward propagate LSTM
        out, hidden = self.bilstm(x, (h0, c0))
        fw_bilstm = out[-1, :, :self.hidden_size]
        bk_bilstm = out[0, :, :self.hidden_size]
        concat_fw_bw = torch.cat((fw_bilstm, bk_bilstm), dim = 1)
        fc = self.fcl(concat_fw_bw)
        x = F.softmax(F.relu(fc))
        return x

Now I initialize the model object.

model = customModel(300, hidden_size, num_layers, num_classes)

Get embedding for the input sentences

sentences = [['my', 'name', 'is', 'nad'], ['i', 'love', 'nlp', 'proc']]
embedding = create_embedding(sentences)
embedding_torch = torch.FloatTensor(embedding)

Now I want to pass these embeddings to the model to get the prediction

for item in embedding_torch:
    item = item.view((1, item.size()[0], item.size()[1]))
    for epoch in range(1):
        tag_scores = model(item)
        print (tag_scores)

Which throws me runtime error

RuntimeError: Expected hidden[0] size (2, 4, 32), got (2, 1, 32)

I am not sure why this is happening. My understanding is h0 = torch.zeros(self.num_layers*2, x.size(0), self.hidden_size).to(device) line is calculating the hidden dimension properly.

What am I missing? Please suggest.


title: Fast.Ai EarlyStoppingCallback does not work

body:
callbacks = [EarlyStoppingCallback(learn, monitor='error_rate', min_delta=1e-5, patience=5)]
learn.fit_one_cycle(30, callbacks=callbacks, max_lr=slice(1e-5,1e-3))

As you can see, I use patience = 5 and min_delta=1e-5 and monitor='error_rate'

  My understanding is: patience tells how many epochs it waits if improvement is less than min_delta on the monitored value, in this case it's error_rate.

So if my understanding was correct, then it would not stop at Epoch 6.

So is this my understanding wrong or the debug in fast.ai lib ?


title: Pytorch memory model: how does "torch.from_numpy()" work?

body:
I'm trying to have an in-depth understanding of how torch.from_numpy() works.

import numpy as np
import torch

arr = np.zeros((3, 3), dtype=np.float32)
t = torch.from_numpy(arr)
print("arr: {0}\nt: {1}\n".format(arr, t))

arr[0,0]=1
print("arr: {0}\nt: {1}\n".format(arr, t))

print("id(arr): {0}\nid(t): {1}".format(id(arr), id(t)))

The output looks like this:

arr: [[0. 0. 0.]
 [0. 0. 0.]
 [0. 0. 0.]]
t: tensor([[0., 0., 0.],
        [0., 0., 0.],
        [0., 0., 0.]])

arr: [[1. 0. 0.]
 [0. 0. 0.]
 [0. 0. 0.]]
t: tensor([[1., 0., 0.],
        [0., 0., 0.],
        [0., 0., 0.]])

id(arr): 2360964353040
id(t): 2360964352984

This is part of the doc from torch.from_numpy():

  from_numpy(ndarray) -> Tensor

  Creates a :class:Tensor from a :class:numpy.ndarray.

  The returned tensor and :attr:ndarray share the same memory. Modifications to
  the tensor will be reflected in the :attr:ndarray and vice versa. The returned
  tensor is not resizable.

And this is taken from the doc of id():

  Return the identity of an object.

  This is guaranteed to be unique among simultaneously existing objects.
  (CPython uses the object's memory address.)

So here comes the question:
Since the ndarray arr and tensor t share the same memory, why do they have different memory addresses?

Any ideas/suggestions?


title: Cannot import Pytorch

body:
While importing installed pytorch I am facing below issue

>>> import torch
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "C:\Anaconda3\lib\site-packages\torch\__init__.py", line 81, in <module>
        ctypes.CDLL(dll)
      File "C:\Anaconda3\lib\ctypes\__init__.py", line 364, in __init__
        self._handle = _dlopen(self._name, mode)
    OSError: [WinError 126] The specified module could not be found


title: Zero Division Error When Using Pytorch Mask RCNN

body:
I am using the pretrained torchvision MaskRCNN model on a dataset containing several videos, by passing the videos to the model frame by frame. However in the forward pass of the MaskRCNN model I get the following error statement:

RuntimeError                              Traceback (most recent call last)

----> 1 output = model(images,targets)

4 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--> 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

/usr/local/lib/python3.6/dist-packages/torchvision/models/detection/generalized_rcnn.py in forward(self, images, targets)
     64             original_image_sizes.append((val[0], val[1]))
     65
---> 66         images, targets = self.transform(images, targets)
     67         features = self.backbone(images.tensors)
     68         if isinstance(features, torch.Tensor):

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--> 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

/usr/local/lib/python3.6/dist-packages/torchvision/models/detection/transform.py in forward(self, images, targets)
     42                 raise ValueError("images is expected to be a list of 3d tensors "
     43                                  "of shape [C, H, W], got {}".format(image.shape))
---> 44             image = self.normalize(image)
     45             image, target_index = self.resize(image, target_index)
     46             images[i] = image

/usr/local/lib/python3.6/dist-packages/torchvision/models/detection/transform.py in normalize(self, image)
     62         mean = torch.as_tensor(self.image_mean, dtype=dtype, device=device)
     63         std = torch.as_tensor(self.image_std, dtype=dtype, device=device)
---> 64         return (image - mean[:, None, None]) / std[:, None, None]
     65
     66     def torch_choice(self, l):

RuntimeError: ZeroDivisionError

My images in this code is a list of size 431, denoting the number of frames in the video, with each item of the list being a torch tensor of shape 3 x 256 x 320, denoting channels x height x width, respectively. Targets here contains the labels, masks and bounding boxes coordinates in the form of a dictionary of lists. The length of the targets list is also 431, with each item containing information about each individual frame.

The code I am using is similar to the one found in this tutorial, except for the fact that that tutorial deals with static images, and I am working with videos, so I made only some minor changes to that code.


title: How do I install **Pytorch** with conda? Is anaconda.org down temporarily?

body:
I just installed Anaconda and now I'm trying to install pytorch via conda install pytorch torchvision cudatoolkit=10.2 -c pytorch. But I'm getting the error message
Collecting package metadata (current_repodata.json): failed

CondaHTTPError: HTTP 503 SERVICE UNAVAILABLE: BACK-END SERVER IS AT CAPACITY for url <https://conda.anaconda.org/pytorch/win-64/current_repodata.json>
Elapsed: 00:00.635763
CF-RAY: 5c7c36945d34c4a4-DUS

A remote server error occurred when trying to retrieve this URL.

A 500-type error (e.g. 500, 501, 502, 503, etc.) indicates the server failed to
fulfill a valid request.  The problem may be spurious, and will resolve itself if you
try your request again.  If the problem persists, consider notifying the maintainer
of the remote server.

How can I resolve this and install pytorch? Is anaconda.org just down temporarily?


title: PyTorch: 'CrossEntropyLoss" object has no attribute 'item'

body:
Currently deploying a CNN model.
model = CNN(height=96, width=96, channels=3)

and looking to observe its cross entropy loss.
criterion = nn.CrossEntropyLoss()

The Trainer class is given as follows,
class Trainer:
def __init__(
    self,
    model: nn.Module,
    train_loader: DataLoader,
    val_loader: DataLoader,
    criterion: nn.Module,
    optimizer: Optimizer,
    summary_writer: SummaryWriter,
    device: torch.device,
):
    self.model = model.to(device)
    self.device = device
    self.train_loader = train_loader
    self.val_loader = val_loader
    self.criterion = criterion
    self.optimizer = optimizer
    self.summary_writer = summary_writer
    self.step = 0

def train(
        self,
        epochs: int,
        val_frequency: int,
        print_frequency: int = 20,
        log_frequency: int = 5,
        start_epoch: int = 0
):
    self.model.train()
    for epoch in range(start_epoch, epochs):
        self.model.train()
        data_load_start_time = time.time()
        for batch, labels in self.train_loader:
            batch = batch.to(self.device)
            labels = labels.to(self.device)
            data_load_end_time = time.time()
            loss=self.criterion
            logits=self.model.forward(batch)

            with torch.no_grad():
                preds = logits
                accuracy = compute_accuracy(labels, preds)

            data_load_time = data_load_end_time - data_load_start_time
            step_time = time.time() - data_load_end_time
            if ((self.step + 1) % log_frequency) == 0:
                self.log_metrics(epoch, accuracy, loss, data_load_time, step_time)
            if ((self.step + 1) % print_frequency) == 0:
                self.print_metrics(epoch, accuracy, loss, data_load_time, step_time)

            self.step += 1
            data_load_start_time = time.time()

        self.summary_writer.add_scalar("epoch", epoch, self.step)
        if ((epoch + 1) % val_frequency) == 0:
            self.validate()
            self.model.train()

The function to log the loss is,
    def log_metrics(self, epoch, accuracy, loss, data_load_time, step_time):
    self.summary_writer.add_scalar("epoch", epoch, self.step)
    self.summary_writer.add_scalars(
            "accuracy",
            {"train": accuracy},
            self.step
    )
    self.summary_writer.add_scalars(
            "loss",
            {"train": float(loss.item())},
            self.step
    )
    self.summary_writer.add_scalar(
            "time/data", data_load_time, self.step
    )
    self.summary_writer.add_scalar(
            "time/data", step_time, self.step
    )

I have been receiving an attribute error "' CrossEntropyLoss' object has no attribute 'item'". I have tried removing several ways such as removing "item()" from different parts of code and trying different types of loss functions like MSELoss etc.
Any solution or direction would be highly appreciated. Thank you.
Edit-1:
Here is the error traceback
Traceback (most recent call last):

 File "/Users/xyz/main.py", line 316, in <module>
main(parser.parse_args())
 File "/Users/xyz/main.py", line 128, in main
    log_frequency=args.log_frequency,
  File "/Users/xyz/main.py", line 198, in train
    self.log_metrics(epoch, accuracy, loss, data_load_time, step_time)
  File "/Users/xyz/main.py", line 232, in log_metrics
    {"train": float(loss.item)},
  File "/Users/xyz/main.py", line 585, in __getattr__
    type(self).__name__, name))
AttributeError: 'CrossEntropyLoss' object has no attribute 'item'


title: Install pytorch 1.7 using python3.8 to Xavier NX (JetPack 4.4) on Ubuntu 18.04

body:
I am installing Torch 1.7 with python3.8 to Xavier NX (JetPack 4.4) on Ubuntu 18.04 using this reference. I got an error when I ran python3.8 setup.py install after python3.8 setup.py build.

    Install the project...
    -- Install configuration: "Release"
    -- Set runtime path of "/home/costar/Documents/pytorch/torch/lib/libtorch_cuda.so" to "$ORIGIN:/usr/local/cuda/lib64:/usr/lib/aarch64-linux-gnu/openmpi/lib"
    -- Set runtime path of "/home/costar/Documents/pytorch/torch/lib/libtorch.so" to "$ORIGIN:/usr/local/cuda/lib64"
    -- Set runtime path of "/home/costar/Documents/pytorch/torch/test/cuda_atomic_ops_test" to "$ORIGIN:/usr/local/cuda/lib64"
    -- Set runtime path of "/home/costar/Documents/pytorch/torch/test/cuda_complex_test" to "$ORIGIN:/usr/local/cuda/lib64"
    -- Set runtime path of "/home/costar/Documents/pytorch/torch/test/cuda_complex_math_test" to "$ORIGIN:/usr/local/cuda/lib64"
    -- Set runtime path of "/home/costar/Documents/pytorch/torch/test/cuda_integer_divider_test" to "$ORIGIN:/usr/local/cuda/lib64"
    -- Set runtime path of "/home/costar/Documents/pytorch/torch/test/cuda_half_test" to "$ORIGIN:/usr/local/cuda/lib64"
    -- Set runtime path of "/home/costar/Documents/pytorch/torch/test/cuda_distributions_test" to "$ORIGIN:/usr/local/cuda/lib64"
    -- Set runtime path of "/home/costar/Documents/pytorch/torch/test/cuda_optional_test" to "$ORIGIN:/usr/local/cuda/lib64"
    -- Set runtime path of "/home/costar/Documents/pytorch/torch/test/cuda_packedtensoraccessor_test" to "$ORIGIN:/usr/local/cuda/lib64"
    -- Set runtime path of "/home/costar/Documents/pytorch/torch/test/cuda_vectorized_test" to "$ORIGIN:/usr/local/cuda/lib64"
    -- Set runtime path of "/home/costar/Documents/pytorch/torch/test/cuda_generator_test" to "$ORIGIN:/usr/local/cuda/lib64"
    -- Set runtime path of "/home/costar/Documents/pytorch/torch/lib/libtorchbind_test.so" to "$ORIGIN:/usr/local/cuda/lib64"
    -- Set runtime path of "/home/costar/Documents/pytorch/torch/lib/libjitbackend_test.so" to "$ORIGIN:/usr/local/cuda/lib64"
    CMake Error at test_jit/cmake_install.cmake:90 (file):
      file RPATH_CHANGE could not write new RPATH:

        $ORIGIN:/usr/local/cuda/lib64:/usr/local/cuda/lib64/stubs

      to the file:

        /home/costar/Documents/pytorch/torch/bin/test_jit

      No valid ELF RPATH or RUNPATH entry exists in the file; Failed to load
      section headers.
    Call Stack (most recent call first):
      caffe2/cmake_install.cmake:3191 (include)
      cmake_install.cmake:111 (include)

    Makefile:61: recipe for target 'install' failed
    make: *** [install] Error 1
    Traceback (most recent call last):
      File "setup.py", line 727, in
        build_deps()
      File "setup.py", line 314, in build_deps
        build_caffe2(version=version,
      File "/home/costar/Documents/pytorch/tools/build_pytorch_libs.py", line 62, in build_caffe2
        cmake.build(my_env)
      File "/home/costar/Documents/pytorch/tools/setup_helpers/cmake.py", line 346, in build
        self.run(build_args, my_env)
      File "/home/costar/Documents/pytorch/tools/setup_helpers/cmake.py", line 141, in run
        check_call(command, cwd=self.build_dir, env=env)
      File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
        raise CalledProcessError(retcode, cmd)
    subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--target', 'install', '--config', 'Release', '--', '-j', '2']' returned non-zero exit status 2.

Thanks.


title: Tensorflow Equivalent of Elastic deformation for pytorch

body:
I am trying to implement elastic deformation by sampling control points on a regularly spaces 100*100 grid with σ = 20 with bilinear variant for the image and nearest neighbour for the masks.
I found this implementation for the TensorFlow by DeepMind.
So far, I was using the below implementation as a lambda transformation-
def elastic_transform(image, alpha=1000, sigma=20, spline_order=1, mode='nearest', random_state=np.random):
"""Elastic deformation of image as described in [Simard2003]_.
.. [Simard2003] Simard, Steinkraus and Platt, "Best Practices for
   Convolutional Neural Networks applied to Visual Document Analysis", in
   Proc. of the International Conference on Document Analysis and
   Recognition, 2003.
"""
#assert image.ndim == 3
image = np.array(image)
assert image.ndim == 3
shape = image.shape[:2]

dx = gaussian_filter((random_state.rand(*shape) * 2 - 1),
                     sigma, mode="constant", cval=0) * alpha
dy = gaussian_filter((random_state.rand(*shape) * 2 - 1),
                     sigma, mode="constant", cval=0) * alpha

x, y = np.meshgrid(np.arange(shape[0]), np.arange(shape[1]), indexing='ij')
indices = [np.reshape(x + dx, (-1, 1)), np.reshape(y + dy, (-1, 1))]
result = np.empty_like(image)
for i in range(image.shape[2]):
    result[:, :, i] = map_coordinates(
        image[:, :, i], indices, order=spline_order, mode=mode).reshape(shape)
result = Image.fromarray(result)
return result

But the issue with the above implementation is it uses spline interpolation.
Is there any TensorFlow equivalent(kind of which i specified above) for PyTorch or is there any other way I can implement my requirement without switching to Tensorflow?


title: PyTorch get indices of value in two-dimensional tensor

body:
Given the following tensor (or any random tensor with two dimension), I want to get the index of '101':

tens = tensor([[  101,   146,  1176, 21806,  1116,  1105, 18621,   119,   102,     0,
             0,     0,     0],
        [  101,  1192,  1132,  1136,  1184,   146,  1354,  1128,  1127,   117,
          1463,   119,   102],
        [  101,  6816,  1905,  1132, 14918,   119,   102,     0,     0,     0,
             0,     0,     0]])

From the related answers I know that I can do something like this:

idxs = torch.tensor([(i == 101).nonzero() for i in tens])

But this seems messy and potentially quite slow. Is there a better way to do this that is fast and more torch-y?

Related questions discussing only one-dimensional tensor:

How Pytorch Tensor get the index of specific value
How Pytorch Tensor get the index of elements?


title: How to embed Sequence of Sentences in RNN?

body:
I am trying to make a RNN model (in Pytorch), that takes couple of sentences and then classifies it to be either Class 0 or Class 1.

For the sake of this question let's assume that the max_len of the sentence is 4 and max_amount of time steps is 5. Thus, each datapoint is on the form (0  is a value that used for padding padded value):

    x[1] = [
    # Input features at timestep 1
    [1, 48, 91, 0],
    # Input features at timestep 2
    [20, 5, 17, 32],
    # Input features at timestep 3
    [12, 18, 0, 0],
    # Input features at timestep 4
    [0, 0, 0, 0],
    # Input features at timestep 5
    [0, 0, 0, 0]
    ]
    y[1] = [1]

When I have just one sentence per target: I simply pass each word to the embedding layer and then to the LSTM or GRU, but I am a bit stuck on what to do when I have a sequence of sentences per target?

How do I build an embedding that can handle sentences?


I am trying to update/change the parameters of a neural net model and then having the forward pass of the updated neural net be in the computation graph (no matter how many changes/updates we do).

I tried this idea but whenever I do it pytorch sets my updated tensors (inside the model) to be leafs, which kills the flow of gradients to the networks I want to receive gradients. It kills the flow of gradients because leaf nodes are not part of the computation graph the way I want them to be (since they aren't truly leafs).

I've tried multiple things but nothing seems to work. I created a dummy code that is self contained that prints the gradients of the networks I desire to have gradients:

import torch
import torch.nn as nn

import copy

from collections import OrderedDict

# img = torch.randn([8,3,32,32])
# targets = torch.LongTensor([1, 2, 0, 6, 2, 9, 4, 9])
# img = torch.randn([1,3,32,32])
# targets = torch.LongTensor([1])
x = torch.randn(1)
target = 12.0*x**2

criterion = nn.CrossEntropyLoss()

#loss_net = nn.Sequential(OrderedDict([('conv0',nn.Conv2d(in_channels=3,out_channels=10,kernel_size=32))]))
loss_net = nn.Sequential(OrderedDict([('fc0', nn.Linear(in_features=1,out_features=1))]))

hidden = torch.randn(size=(1,1),requires_grad=True)
updater_net = nn.Sequential(OrderedDict([('fc0',nn.Linear(in_features=1,out_features=1))]))
print(f'updater_net.fc0.weight.is_leaf = {updater_net.fc0.weight.is_leaf}')
#
nb_updates = 2
for i in range(nb_updates):
    print(f'i = {i}')
    new_params = copy.deepcopy( loss_net.state_dict() )
    ## w^<t> := f(w^<t-1>,delta^<t-1>)
    for (name, w) in loss_net.named_parameters():
        print(f'name = {name}')
        print(w.size())
        hidden = updater_net(hidden).view(1)
        print(hidden.size())
        #delta = ((hidden**2)*w/2)
        delta = w + hidden
        wt = w + delta
        print(wt.size())
        new_params[name] = wt
        #del loss_net.fc0.weight
        #setattr(loss_net.fc0, 'weight', nn.Parameter( wt ))
        #setattr(loss_net.fc0, 'weight', wt)
        #loss_net.fc0.weight = wt
        #loss_net.fc0.weight = nn.Parameter( wt )
    ##
    loss_net.load_state_dict(new_params)
#
print()
print(f'updater_net.fc0.weight.is_leaf = {updater_net.fc0.weight.is_leaf}')
outputs = loss_net(x)
loss_val = 0.5*(target - outputs)**2
loss_val.backward()
print()
print(f'-- params that dont matter if they have gradients --')
print(f'loss_net.grad = {loss_net.fc0.weight.grad}')
print('-- params we want to have gradients --')
print(f'hidden.grad = {hidden.grad}')
print(f'updater_net.fc0.weight.grad = {updater_net.fc0.weight.grad}')
print(f'updater_net.fc0.bias.grad = {updater_net.fc0.bias.grad}')

if anyone knows how to do this please give me a ping...I set the the number of times to update to be 2 because the update operation should be in the computation graph an arbitrary number of times...so it MUST work for 2.

Strongly related post:

SO: How does one have parameters in a pytorch model not be leafs and be in the computation graph?
pytorch forum: https://discuss.pytorch.org/t/how-does-one-have-the-parameters-of-a-model-not-be-leafs/70076

Cross-posted:

Quora: https://www.quora.com/unanswered/How-does-one-have-parameters-in-a-PyTorch-model-not-be-leaves-and-be-in-the-computation-graph
reddit: https://www.reddit.com/r/pytorch/comments/f5gu3g/how_does_one_have_parameters_in_a_pytorch_model/


title: AttributeError: ‘NoneType’ object has no attribute ‘register_forward_hook’

body:
I’m trying to register a forward hook function to the last conv layer of my network. I first printed out the names of the modules via:

for name, _ in model.named_modules():
    print(name)

Which gave me "0.conv" as the module name. However, when I tried to do the following, the above error was triggered by line 4:

def hook_feature(module, in_, out_):
    features.append(out_.cpu().data.numpy())

model._modules.get("0.conv").register_forward_hook(hook_feature)

Here is my named_modules() output:

...
0.decoder1.dec1conv2
0.decoder1.dec1norm2
0.decoder1.dec1relu2
0.conv
1
1.outc1
1.outc2

What am I doing wrong and how do I fix it? Thanks!


title: how to assign value to a tensor using index

body:
I defined four tensors that represent index_x,index_y,index_z,and value,respectively and assigned value to a new tensor using these three index. Why were the results of the two assignments different?

import torch
import numpy as np
import random
import os

def seed_torch(seed=0):
    random.seed(seed)
    np.random.seed(seed)
    os.environ['PYTHONHASHSEED'] = str(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False

seed_torch(1)
a_list, b_list, c_list = [], [], []
for i in range(0, 512*512):
    a_ = random.randint(0, 399)
    b_ = random.randint(0, 399)
    c_ = random.randint(0, 199)
    a_list.append(a_)
    b_list.append(b_)
    c_list.append(c_)
a = torch.tensor(a_list)
b = torch.tensor(b_list)
c = torch.tensor(c_list)
v = torch.rand(512*512)
matrix1 = torch.zeros(400,400,200)
matrix2 = torch.zeros(400,400,200)
index=[a,b,c]
matrix1[index]=v
matrix2[index]=v
m = matrix1 - matrix2
print(m.sum())

print(m.sum()) is not zero


title: Torch optimisers with different scaled parameters

body:
I am trying to optimise parameter values using a torch optimiser but the parameters are on vastly different scales. i.e., one parameter has values in the thousands while others are between 0 and 1. For example in this made up case there are two parameters - one has an optimal value of 0.1 and the other an optimal value of 20. How can I modify this code so it applies a sensible learning rate to each parameter say 1e-3 and 0.1?

import torch as pt
# Objective function
def f(x, y):
    return (10 - 100 * x) ** 2 + (y - 20) ** 2
# Optimal parameters
print("Optimal value:", f(0.1, 20))
# Initial parameters
hp = pt.Tensor([1, 10])
print("Initial value", f(*hp))
# Optimiser
hp.requires_grad = True
optimizer = pt.optim.Adam([hp])
n = 5
for i in range(n):
    optimizer.zero_grad()
    loss = f(*hp)
    loss.backward()
    optimizer.step()
hp.requires_grad = False
print("Final parameters:", hp)
print("Final value:", f(*hp))


title: Pytorch TypeError - eq() received an invalid combination of arguments

body:
I'm working on a text classification problem with BERT. When training on the local machine everything works just fine, but when switching to the server, I get the following error:

<ipython-input-28-508d35ac5f5f> in flat_accuracy(preds, labels)
      5     pred_flat = np.argmax(preds, axis=1).flatten()
      6     labels_flat = labels.flatten()
----> 7     return np.sum(pred_flat == labels_flat) / len(labels_flat)
      8
      9 # Function to calculate the f1_score of our predictions vs labels

TypeError: eq() received an invalid combination of arguments - got (numpy.ndarray), but expected one of:
 * (Tensor other)
      didn't match because some of the arguments have invalid types: (numpy.ndarray)
 * (Number other)
      didn't match because some of the arguments have invalid types: (numpy.ndarray)

Code:

def flat_accuracy(preds, labels):
    pred_flat = np.argmax(preds, axis=1).flatten()
    labels_flat = labels.flatten()
    return np.sum(pred_flat == labels_flat) / len(labels_flat)

Torch version on local machine: 1.4.0

Torch version on the server: 1.3.1

Any help would be greatly appreciated!


title: Finding the top k matches in Pytorch

body:
I'm using the following code to find the topk matches using pytorch:
def find_top(self, x, y, n_neighbors, unit_vectors=False, cuda=False):
    if not unit_vectors:
        x = __to_unit_torch__(x, cuda=cuda)
        y = __to_unit_torch__(y, cuda=cuda)
    with torch.no_grad():
        d = 1. - torch.matmul(x, y.transpose(0, 1))
        values, indices = torch.topk(d, n_neighbors, dim=1, largest=False, sorted=True)
        return indices.cpu().numpy()

Unfortunately, it is throwing the following error:
values, indices = torch.topk(d, n_neighbors, dim=1, largest=False, sorted=True)
RuntimeError: invalid argument 5: k not in range for dimension at /pytorch/aten/src/THC/generic/THCTensorTopK.cu:23

The size of d is (1793,1)  . What am I missing?


title: What is the equivalent of keras NonNeg weight constraint?

body:
Keras has an option to force the weights of the learned model to be positive:
tf.keras.constraints.NonNeg()

But I couldn't find the equivalent of this in pytorch, does anyone know how can I force my linear model's weights to be all positives?
Tried asking this on other forums but the answers were not helpful.
Let's say I have a very simple linear model as shown below, how should I change it?
class Classifier(nn.Module):

    def __init__(self,input , n_classes):
        super(Classifier, self).__init__()

        self.classify = nn.Linear( input  , n_classes)

     def forward(self, h ):

        final = self.classify(h)
        return final

I want to do exactly what the NonNeg() does but in pytorch, don't want to change what its doing.
This is the implementation of NonNeg in keras:
class NonNeg(Constraint):
    """Constrains the weights to be non-negative.
    """

    def __call__(self, w):
        w *= K.cast(K.greater_equal(w, 0.), K.floatx())
        return w


title: Size mismatch error with PyTorch toturial

body:
I am learning the tutorial called Deep Learning with PyTorch: A 60 Minute Blitz on PyTorch website. My codes are the same as those of it, but there is a size mismatch error as shown below. Could anyone tell me why and how to solve it? Thank you:)

  RuntimeError: size mismatch, m1: [80 x 5], m2: [400 x 120] at
  c:\a\w\1\s\tmp_conda_3.7_110509\conda\conda-bld\pytorch_1544094576194\work\aten\src\th\generic/THTensorMath.cpp:940

import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):
def __init__(self):
    super(Net,self).__init__()
    self.conv1=nn.Conv2d(1,6,5)
    self.conv2=nn.Conv2d(6,16,5)
    self.fc1=nn.Linear(16*5*5,120)
    self.fc2=nn.Linear(120,84)
    self.fc3=nn.Linear(84,10)

def forward(self,x):
    x=F.max_pool2d(F.relu(self.conv1(x)),(2,2))
    x=F.max_pool2d(F.relu(self.conv2(x)),2)
    x.view(-1,self.num_flat_features(x))
    x=F.relu(self.fc1(x))
    x=F.relu(self.fc2(x))
    x=self.fc3(x)
    return x

def num_flat_features(self,x):
    size=x.size()[1:]
    num_features=1
    for s in size:
        num_features*=s
    return num_features

net=Net()
input=torch.randn(1,1,32,32)
out=net(input)
print(out)


title: U-net with pre-trained backbones: where to make skip connections?

body:
I'm trying to implement U-Net with PyTorch, using pre-trained networks in the encoder path.

The original U-Net paper trained a network from scratch. Are there any resources or principles on where skip connections should be placed, when a pre-trained backbone is used instead?

I already found some examples (e.g. this repo), but without any justification for the feature selection.


title: Why i can't improve the performance of a CNN on the Fashion MNIST dataset?

body:
I am playing around with Pytorch and i implemented a CNN on MNIST dataset which has 99+% accuracy on both train and test sets.

I decided to switch to Fashion MNIST in order to see how the architecture of my network performs.  I got 95% accuracy on the train set and 91% on test set.

Then, i started trying to improve that performance by tuning the model.

Briefly, my model looks like this:

         Conv -> ReLU -> Batch norm -> Max pool ->
         Conv -> ReLU -> Batch norm -> Max pool ->
         Conv -> ReLU -> Batch norm -> Max pool ->
         Conv -> ReLU -> Batch norm -> Max pool ->
         Linear -> ReLu -> Linear -> Output

         Optimizer: Stochastic Gradient Descent
         Transformations: ToTensor() only

My tests where removing the last Conv layer, adding average pooling instead of max pooling in the last Conv layer, inspecting the train loss curve in order to adjust the learning rate statically or dynamically and change the batch size.

However with the above combinations either my model will overfit (eg. 97% train, 89% test) or it will not have the best performance (eg 91% train, 89% test).

Am i missing something?  Am i doing something wrong?  Are there any other tuning parameters that i need to adjust that i didn't think of?

Thank you


title: Pytorch batch matrix-matrix outer product

body:
Similarly to the question in Pytorch batch matrix vector outer product I have two matrices and would like to compute their outer product, or in other words the pairwise elementwise product.

Shape example:
If we have X1 and X2 of shapes of torch.Size([32, 300, 8])
The result should be of size torch.Size([32, 300, 300, 8])


Is it possible to disable running average calculation in tf.layers.batch_normalization? i.e. I want BN work the same for training and testing, estimating stats only on input tensor.

According to this equation momentum = 1.0 is sufficient for this behaviour?

x_new_est = (1-momentum) * x_est + momentum * (x_t)

or/and setting training=True in training and inference is sufficient?

  training: Either a Python boolean, or a TensorFlow boolean scalar
  tensor (e.g. a placeholder). Whether to return the output in training
  mode (normalized with statistics of the current batch) or in inference
  mode (normalized with moving statistics). NOTE: make sure to set this
  parameter correctly, or else your training/inference will not work
  properly.

Seems this is possible in PyTorch
https://pytorch.org/docs/stable/nn.html#torch.nn.BatchNorm2d

  If track_running_stats is set to False, this layer then does not keep
  running estimates, and batch statistics are instead used during
  evaluation time as well.


title: How to obtain sequence of submodules from a pytorch module?

body:
For a pytorch module, I suppose I could use .named_children, .named_modules, etc. to obtain a list of the submodules. However, I suppose the list is not given in order, right? An example:

In [19]: import transformers

In [20]: model = transformers.DistilBertForSequenceClassification.from_pretrained('distilb
    ...: ert-base-cased')

In [21]: [name for name, _ in model.named_children()]
Out[21]: ['distilbert', 'pre_classifier', 'classifier', 'dropout']

The order of .named_children() in the above model is given as distilbert, pre_classifier, classifier, and dropout. However, if you examine the code, it is evident that dropout happens before classifier. So how do I get the order of these submodules?


title: How to load multi-image input in PyTorch?

body:
I have a dataset with 2 RGB images per data sample (6 channels). How to read such a dataset in PyTorch?
For one RGB  image I used:
data_transform = transforms.Compose([
    transforms.RandomHorizontalFlip(),
    transforms.RandomVerticalFlip(),
    transforms.ToTensor()
])
mel_dataset_train = datasets.ImageFolder(root='./ftrain',
                                         transform=data_transform)
train_sampler = torch.utils.data.distributed.DistributedSampler(mel_dataset_train)
dataset_loader_train = torch.utils.data.DataLoader(mel_dataset_train,
                                                   batch_size=64, shuffle=True, sampler=train_sampler,
                                                   num_workers=config.workers)

Yet I do not see how to modify it to read two images per data sample instead of one.


title: Which deep learning library support the compression of the deep learning models to be used on the phones?

body:
I want to build an advanced deep learning model (for example: a model that uses attention) and use it on android phones (without training of course) i will only use it for inference.
and i want a library that can do that and can compress the model size to be used on the phone or android.
and do you know any projects or apps similar to my goal?


title: Pytorch: executiin on GPU: RuntimeError: Expected object of backend CUDA but got backend CPU for argument #3 'index'

body:
How to make model execute on GPU in Pytorch?

I load model like:

def load_tocotron_2_model():
    model = Tacotron2(num_chars=n_chars, r=CONFIG.r, attn_win=CONFIG.windowing, attn_norm=CONFIG.attention_norm,
                      prenet_type=CONFIG.prenet_type, forward_attn=CONFIG.use_forward_attn,
                      trans_agent=CONFIG.transition_agent)

    if use_cuda:
        cp = torch.load(MODEL_PATH)
    else:
        cp = torch.load(MODEL_PATH, map_location='cpu')

    model.load_state_dict(cp['model'])

    if use_cuda:
        model.cuda()

    model.eval() # Set eval mode

    return model

If use_cuda is False then model runs ok, but if it's True I get error:

  File "/data/user/external_projects/text-to-speech/mozila-tts-wavernn-cpu/TTS/models/tacotron2.py", line 50, in inference
    embedded_inputs = self.embedding(text).transpose(1, 2)
  File "/data/user/external_projects/text-to-speech/mozila-tts-wavernn-cpu/TTS/my_env_gpu/lib/python3.5/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/data/user/external_projects/text-to-speech/mozila-tts-wavernn-cpu/TTS/my_env_gpu/lib/python3.5/site-packages/torch/nn/modules/sparse.py", line 117, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/data/user/external_projects/text-to-speech/mozila-tts-wavernn-cpu/TTS/my_env_gpu/lib/python3.5/site-packages/torch/nn/functional.py", line 1506, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: Expected object of backend CUDA but got backend CPU for argument #3 'index'

self.embedding is self.embedding = nn.Embedding(num_chars, 512), do I need to specify .cuda() on some tensor or model.cuda() is sufficient?


title: How to give the output of craft( character - region awareness for text detection ) to crnn

body: I am doing text detection and recognition. I have used Craft(character-Region Awareness for text detection)https://github.com/clovaai/CRAFT-pytorch. it gave me an output of the image with a bounding box and a text file with numbers. I want to feed the output to CRNN https://github.com/bgshih/crnn . but I need ROI of text image to make CRNN work.how to connect craft and CRNN


title: PyTorch broadcast multiplication of 4D and 2D matrix?

body:
How do I broadcast to multiply these two matrices together?

x: torch.Size([10, 120, 180, 30]) # (N, H, W, C)
W: torch.Size([64, 30]) # (Y, C)

The output should be:

(10, 120, 180, 64) == (N, H, W, Y)


title: HuggingFace ByteLevelBPETokenizer encoding issue in merge.txt file

body:
I am using HuggingFace ByteLevelBPETokenizer to generate vocabulary for marathi language
("आपण हवे ते लेख शोधू शकता")

When I train marathi language data, I end up with such outputs in merges.text file  which is not part of vocabulary or training dataset.

à ¤

à ¥

Ġ à¤

à¤ ¾

Below is the code used for training

from tokenizers import ByteLevelBPETokenizer

filename= "marathi.txt"

# Initialize a tokenizer
tokenizer = ByteLevelBPETokenizer()

# Customize training
tokenizer.train(files=filename, vocab_size=20_000, min_frequency=3, special_tokens=[
    "<s>",
    "<pad>",
    "</s>",
    "<unk>",
    "<mask>",
])

I also tried setting ByteLevelBPETokenizer's unicode_normalizer parameter by setting each of the values to ['nfc', 'nfd', 'nfkc', 'nfkd'].

How do I get correct words in vocabulary, how to handle it?