title: Yolo5 model training fails with CUDA out of memory even with an AWS P8 instance

body:
I am training a yolo5 on a custom dataset but I keep running out of memory for GPU as it only uses one of the 8 GPUs. How should I run it in order for it to use all of the GPUs?
YOLOv5 v4.0-83-gd2e754b torch 1.7.1 CUDA:0 (Tesla K80, 11441.1875MB)
CUDA:1 (Tesla K80, 11441.1875MB)
CUDA:2 (Tesla K80, 11441.1875MB)
CUDA:3 (Tesla K80, 11441.1875MB)
CUDA:4 (Tesla K80, 11441.1875MB)
CUDA:5 (Tesla K80, 11441.1875MB)
CUDA:6 (Tesla K80, 11441.1875MB)
CUDA:7 (Tesla K80, 11441.1875MB)
RuntimeError: CUDA out of memory. Tried to allocate 16.00 MiB (GPU 0; 11.17 GiB total capacity; 10.66 GiB already allocated; 2.31 MiB free; 10.72 GiB reserved in total by PyTorch
Thanks
Ganesh


title: HuggingFace - GPT2 Tokenizer configuration in config.json

body:
The GPT2 finetuned model is uploaded in huggingface-models for the inferencing
Below error is observed during the inference,
Can't load tokenizer using from_pretrained, please update its configuration: Can't load tokenizer for 'bala1802/model_1_test'. Make sure that: - 'bala1802/model_1_test' is a correct model identifier listed on 'https://huggingface.co/models' - or 'bala1802/model_1_test' is the correct path to a directory containing relevant tokenizer files
Below is the configuration - config.json file for the Finetuned huggingface model,
{
  "_name_or_path": "gpt2",
  "activation_function": "gelu_new",
  "architectures": [
    "GPT2LMHeadModel"
  ],
  "attn_pdrop": 0.1,
  "bos_token_id": 50256,
  "embd_pdrop": 0.1,
  "eos_token_id": 50256,
  "gradient_checkpointing": false,
  "initializer_range": 0.02,
  "layer_norm_epsilon": 1e-05,
  "model_type": "gpt2",
  "n_ctx": 1024,
  "n_embd": 768,
  "n_head": 12,
  "n_inner": null,
  "n_layer": 12,
  "n_positions": 1024,
  "resid_pdrop": 0.1,
  "summary_activation": null,
  "summary_first_dropout": 0.1,
  "summary_proj_to_labels": true,
  "summary_type": "cls_index",
  "summary_use_proj": true,
  "task_specific_params": {
    "text-generation": {
      "do_sample": true,
      "max_length": 50
    }
  },
  "transformers_version": "4.3.2",
  "use_cache": true,
  "vocab_size": 50257
}

Should I configure the GPT2 Tokenizer just like the "model_type": "gpt2" in the config.json file


I iam trying to run a simple code as below.
I have features of two graphs of type sparse tensor, stored in a list
feature= [tensor(indices=tensor([[   0,    1,    2,  ..., 1896, 1897, 1898],
                           [   0,    1,    2,  ..., 1896, 1897, 1898]]),
           values=tensor([1., 1., 1.,  ..., 1., 1., 1.]),
           size=(1899, 1899), nnz=1899, layout=torch.sparse_coo), tensor(indices=tensor([[   0,    1,    2,  ..., 1896, 1897, 1898],
                           [   0,    1,    2,  ..., 1896, 1897, 1898]]),
           values=tensor([1., 1., 1.,  ..., 1., 1., 1.]),
           size=(1899, 1899), nnz=1899, layout=torch.sparse_coo)]

when I convert the feature above to dense tensor, i get the following result
print('this is feature 0', feature[0].to_dense())
print('this is feature 1', feature[1].to_dense())
this is feature 0 tensor([[1., 0., 0.,  ..., 0., 0., 0.],
        [0., 1., 0.,  ..., 0., 0., 0.],
        [0., 0., 1.,  ..., 0., 0., 0.],
        ...,
        [0., 0., 0.,  ..., 1., 0., 0.],
        [0., 0., 0.,  ..., 0., 1., 0.],
        [0., 0., 0.,  ..., 0., 0., 1.]])
this is feature 1 tensor([[1., 0., 0.,  ..., 0., 0., 0.],
        [0., 1., 0.,  ..., 0., 0., 0.],
        [0., 0., 1.,  ..., 0., 0., 0.],
        ...,
        [0., 0., 0.,  ..., 1., 0., 0.],
        [0., 0., 0.,  ..., 0., 1., 0.],
        [0., 0., 0.,  ..., 0., 0., 1.]])

how ever when I do the same in the loop as below, i am getting surprising result
for i in range(2):
   feature=feature[i]
    print('these are new features: ',feature )

result to
tensor(indices=tensor([[   0,    1,    2,  ..., 1896, 1897, 1898],
                       [   0,    1,    2,  ..., 1896, 1897, 1898]]),
       values=tensor([1., 1., 1.,  ..., 1., 1., 1.]),
       size=(1899, 1899), nnz=1899, layout=torch.sparse_coo)
tensor(indices=tensor([[1]]),
       values=tensor([1.]),
       size=(1899,), nnz=1, layout=torch.sparse_coo)

and when I convert the same to dense tensor in the loop, like this
for i in range(2):
   feature=feature[i]
    print('these are new features: ',feature )
i get this error
RuntimeError: Could not run 'aten::to_dense' with arguments from the 'CPU' backend

What is the problem? how can I solve it? Any help will be much appeciated


title: Pytorch: Updating numpy array not updating the corresponding tensor

body:
When I run the following code,

import numpy as np
a = np.ones(3)
b = torch.from_numpy(a)
np.add(a, 1, out=a)
print(a)
print(b)

Both a and b are 2s.

However, When I run:

import numpy as np
a = np.ones(3)
b = torch.from_numpy(a)
a = a+1
print(a)
print(b)

b remains as 1s while a has been updated to 2s.

Is this an expected behaviour?


title: pix2pixHD in cpu mode illegal instruction

body:
I am trying to test pix2pixHD. I don't have a GPU. I am using pytorch==0.4.1. I am running the test for cityscape model with the following command:

python test.py --name label2city_1024p --netG local --ngf 32 --gpu_ids -1 --resize_or_crop none

But I am getting Illegal instruction (core dumped). I tried the backtrace with gdb and found this:

#0  0x00007fcd623175e6 in THFloatVector_normal_fill_AVX2 () from /home/momen/nvidia_vid2vid/lib/python3.6/site-packages/torch/lib/libcaffe2.so
#1  0x00007fcd621d06f0 in THFloatTensor_normal () from /home/momen/nvidia_vid2vid/lib/python3.6/site-packages/torch/lib/libcaffe2.so
#2  0x00007fcd61c85156 in at::CPUFloatType::normal_(at::Tensor&, double, double, at::Generator*) const ()
   from /home/momen/nvidia_vid2vid/lib/python3.6/site-packages/torch/lib/libcaffe2.so
#3  0x00007fcdb8724895 in torch::autograd::VariableType::normal_ (this=0x27d4fc0, self=..., mean=0, std=0.02, generator=0x0)
at torch/csrc/autograd/generated/VariableType.cpp:10819
#4  0x00007fcdb89e6300 in at::Tensor::normal_ (generator=0x0, std=0.02, mean=0, this=0x7fcd921cc958) at /pytorch/torch/lib/tmp_install/include/ATen/TensorMethods.h:700

How do I solve this issue? Any type of help is welcome. Thanks.

Additional Info:
I tried py-bt and the problem occurs with m.weight.data.normal_(0.0, 0.02) for first Conv2d layer.


title: Pytorch convering LSTM Model to Dataparallel for Multiple GPU Usage

body:
I need to convert my existing model to use DataParallel to run across multiple GPU's, I have been reading the docs for ages and cant get my head around what i need to change exactly so i need some help, Thanks!

class LSTM(nn.Module):
def __init__(self, input_size=1, hidden_layer_size=100, output_size=1):
    super().__init__()
    self.hidden_layer_size = hidden_layer_size

    self.lstm = nn.LSTM(input_size, hidden_layer_size)

    self.linear = nn.Linear(hidden_layer_size, output_size)

    self.hidden_cell = (torch.zeros(1,1,self.hidden_layer_size),
                        torch.zeros(1,1,self.hidden_layer_size))

def forward(self, input_seq):
    lstm_out, self.hidden_cell = self.lstm(input_seq.view(len(input_seq) ,1, -1), self.hidden_cell)
    predictions = self.linear(lstm_out.view(len(input_seq), -1))
    return predictions[-1]

model = LSTM()
loss_function = nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.00001)

epochs = 1

 for i in range(epochs):
  count=0
  for seq, labels in train_inout_seq:
    optimizer.zero_grad()
    model.hidden_cell = (torch.zeros(1, 1, model.hidden_layer_size),
                    torch.zeros(1, 1, model.hidden_layer_size))
    y_pred = model(seq)

    single_loss = loss_function(y_pred, labels)
    single_loss.backward()
    optimizer.step()

print("Epoch: %d, loss: %1.5f" % (i, single_loss.item()))

model.eval()

 testPredictions=[]
 realValues=[]
 for seq, labels in test_input_seq:

  with torch.no_grad():
    model.hidden = (torch.zeros(1, 1, model.hidden_layer_size),
                    torch.zeros(1, 1, model.hidden_layer_size))
    testPredictions.append(model(seq).item())
    print(model(seq).item())
    realValues.append(labels)


title: Pytorch CUDA out of Memory if GPU is not “warmed up”?

body:
I have a simple script as follows:

import torch
LOSS_WEIGHTS = [1,2,3]
LOSS_WEIGHTS = torch.Tensor(LOSS_WEIGHTS)
LOSS_WEIGHTS = LOSS_WEIGHTS.to(0)

If I start the script while the computer is idle, I often get “CUDA error: out of memory” yet the GPU is completely empty.

The error somehow always goes away after I repeatedly relaunch the script several times. Does anyone know what can I do to prevent this error? Am I suppose to initialise my cuda device before starting the script?

pytorch 1.2.0 (Tried several versions)
cuda 10.1 (Also tried cuda 9)
python 3.7
Nvidia Driver 430
Hardware: 1 x GTX 1070
Ubuntu 18.04


title: issue while import pytorch in Anaconda

body:
(test_pytorch) C:\Users\kuntal chowdhury>python
Python 3.8.3 (default, May 19 2020, 06:50:17) [MSC v.1916 64 bit (AMD64)] :: Anaconda, Inc. on win32
Type "help", "copyright", "credits" or "license" for more information.

      import torch
      Traceback (most recent call last):
        File "", line 1, in
        File "C:\Users\kuntal chowdhury\Anaconda3\envs\test_pytorch\lib\site-packages\torch__init__.py", line 81, in
          ctypes.CDLL(dll)
        File "C:\Users\kuntal chowdhury\Anaconda3\envs\test_pytorch\lib\ctypes__init__.py", line 373, in init
          self._handle = _dlopen(self._name, mode)
      FileNotFoundError: Could not find module 'C:\Users\kuntal chowdhury\Anaconda3\envs\test_pytorch\lib\site-packages\torch\lib\caffe2_nvrtc.dll' (or one of its dependencies). Try using the full path with constructor syntax.


title: Dropout is not working in train mode on Pytorch

body:
I am trying to do Monte Carlo sampling to estimate uncertainty. Even though I turn on the train mode by model.train(), dropout is not working and all the samples are getting the same value. What am I doing wrong here?

Model looks like something like below:

LSTM_Base(
  (lstm): LSTM(167, 256, num_layers=3, batch_first=True, dropout=0.5)
  (fc): Linear(in_features=12800, out_features=256, bias=True)
  (dropout): Dropout(p=0.5, inplace=False)
  (fc2): Linear(in_features=256, out_features=43, bias=True)
)

-My sampling function looks like something like this

def sampler(X, model, T=128):
    y_dist = []
    model.train()
    hidden_init = (torch.zeros(n_layers, 1, hidden_dim).cuda(),
                       torch.zeros(n_layers, 1, hidden_dim).cuda())
    if model.training:
        for i in range(T):
            sample_out, _ = model.train().forward(X, hidden_init)
            y_dist.append(torch.clone(sample_out))

    return y_dist

Note that model is already trained, I am trying to do sampling in train mode, after I trained the model.

EDIT: I just realized that there is no problem about the working of train mode. Because before training, the model is able to predict different outputs in train mode. But after training with more than 10 epochs, all outputs become the same. After realizing that, I also tried to train for just one epoch, and see that outputs are different. So it is clear that the problem is not about train mode. Can it be vanishing gradients?


title: RuntimeError: Expected 4-dimensional input for 4-dimensional weight X, but got 3-dimensional input of size Y instead

body:
I am building a CNN to do image classification on the EMNIST dataset.

To do so, I have the following datasets:

import scipy .io
emnist = scipy.io.loadmat(DIRECTORY + '/emnist-letters.mat')
data = emnist ['dataset']
X_train = data ['train'][0, 0]['images'][0, 0]
X_train = X_train.reshape((-1,28,28), order='F')

y_train = data ['train'][0, 0]['labels'][0, 0]

X_test = data ['test'][0, 0]['images'][0, 0]
X_test = X_test.reshape((-1,28,28), order = 'F')

y_test = data ['test'][0, 0]['labels'][0, 0]

With shape:

X_train = (124800, 28, 28)
y_train = (124800, 1)
X_test = (20800, 28, 28)
y_test = (20800, 1)

Note that the pictures are grayscale, so the colors are represented with only one number.

Which I prepare further as follows:

train_dataset = torch.utils.data.TensorDataset(torch.from_numpy(X_train), torch.from_numpy(y_train))
test_dataset = torch.utils.data.TensorDataset(torch.from_numpy(X_test), torch.from_numpy(y_test))

train_loader = torch.utils.data.DataLoader(dataset=train_dataset,
                                           batch_size=batch_size,
                                           shuffle=True)

test_loader = torch.utils.data.DataLoader(dataset=test_dataset,
                                          batch_size=batch_size,
                                          shuffle=False)

My model looks as follows:

class CNNModel(nn.Module):
    def __init__(self):
        super(CNNModel, self).__init__()

        self.cnn_layers = Sequential(
            # Defining a 2D convolution layer
            Conv2d(1, 4, kernel_size=3, stride=1, padding=1),
            BatchNorm2d(4),
            ReLU(inplace=True),
            MaxPool2d(kernel_size=2, stride=2),
            # Defining another 2D convolution layer
            Conv2d(4, 4, kernel_size=3, stride=1, padding=1),
            BatchNorm2d(4),
            ReLU(inplace=True),
            MaxPool2d(kernel_size=2, stride=2),
        )

        self.linear_layers = Sequential(
            Linear(4 * 7 * 7, 10)
        )

    # Defining the forward pass
    def forward(self, x):
        x = self.cnn_layers(x)
        x = x.view(x.size(0), -1)
        x = self.linear_layers(x)
        return x

model = CNNModel()

The code below is part of the code that I use to train my model:

for epoch in range(num_epochs):
    for i, (images, labels) in enumerate(train_loader):

    images = Variable(images)
    labels = Variable(labels)

    # Forward pass to get output/logits
    outputs = model(images)

However, by excuting my code, I get the following error:

RuntimeError: Expected 4-dimensional input for 4-dimensional weight [4, 1, 3, 3], but got 3-dimensional input of size [100, 28, 28] instead

So a 4D input is expected while my input is 3D. What can I do so a 3D model is expected rather than a 4D model?

Here a similar question is asked, however I dont see how I can translate this to my code


title: Runtime error: CUDA out of memory: Cant train SEGAN

body:
I am currently trying to run a SEGAN for speech enhancement but can't seem to get the network to start training since it runs the following error:
Runtime error: CUDA out of memory: Tried to allocate 30.00 MiB (GPU 0; 3.00 GiB total capacity; 2.00 GiB already allocated; 5.91 MiB free; 2.03 GiB reserved in total by PyTorch
I have already tried to include torch.cuda.empty_cache() but that did not seem to have solved the issue
This is the script I am currently running
import argparse
import os

import torch
import torch.nn as nn
from scipy.io import wavfile
from torch import optim
from torch.autograd import Variable
from torch.utils.data import DataLoader
from tqdm import tqdm

from data_preprocess import sample_rate
from model import Generator, Discriminator
from utils import AudioDataset, emphasis

if __name__ == '__main__':
    parser = argparse.ArgumentParser(description='Train Audio Enhancement')
    parser.add_argument('--batch_size', default=50, type=int, help='train batch size')
    parser.add_argument('--num_epochs', default=86, type=int, help='train epochs number')

    opt = parser.parse_args()
    BATCH_SIZE = opt.batch_size
    NUM_EPOCHS = opt.num_epochs

    # load data
    torch.cuda.empty_cache()

    print('loading data...')
    train_dataset = AudioDataset(data_type='train')
    test_dataset = AudioDataset(data_type='test')
    train_data_loader = DataLoader(dataset=train_dataset, batch_size=BATCH_SIZE, shuffle=True, num_workers=4)
    test_data_loader = DataLoader(dataset=test_dataset, batch_size=BATCH_SIZE, shuffle=False, num_workers=4)
    # generate reference batch
    ref_batch = train_dataset.reference_batch(BATCH_SIZE)

    # create D and G instances
    discriminator = Discriminator()
    generator = Generator()
    if torch.cuda.is_available():
        discriminator.cuda()
        generator.cuda()
        ref_batch = ref_batch.cuda()
    ref_batch = Variable(ref_batch)
    print("# generator parameters:", sum(param.numel() for param in generator.parameters()))
    print("# discriminator parameters:", sum(param.numel() for param in discriminator.parameters()))
    # optimizers
    g_optimizer = optim.RMSprop(generator.parameters(), lr=0.0001)
    d_optimizer = optim.RMSprop(discriminator.parameters(), lr=0.0001)

    for epoch in range(NUM_EPOCHS):
        train_bar = tqdm(train_data_loader)
        for train_batch, train_clean, train_noisy in train_bar:

            # latent vector - normal distribution
            z = nn.init.normal(torch.Tensor(train_batch.size(0), 1024, 8))
            if torch.cuda.is_available():
                train_batch, train_clean, train_noisy = train_batch.cuda(), train_clean.cuda(), train_noisy.cuda()
                z = z.cuda()
            train_batch, train_clean, train_noisy = Variable(train_batch), Variable(train_clean), Variable(train_noisy)
            z = Variable(z)

            # TRAIN D to recognize clean audio as clean
            # training batch pass
            discriminator.zero_grad()
            outputs = discriminator(train_batch, ref_batch)
            clean_loss = torch.mean((outputs - 1.0) ** 2)  # L2 loss - we want them all to be 1
            clean_loss.backward()

            # TRAIN D to recognize generated audio as noisy
            generated_outputs = generator(train_noisy, z)
            outputs = discriminator(torch.cat((generated_outputs, train_noisy), dim=1), ref_batch)
            noisy_loss = torch.mean(outputs ** 2)  # L2 loss - we want them all to be 0
            noisy_loss.backward()

            # d_loss = clean_loss + noisy_loss
            d_optimizer.step()  # update parameters

            # TRAIN G so that D recognizes G(z) as real
            generator.zero_grad()
            generated_outputs = generator(train_noisy, z)
            gen_noise_pair = torch.cat((generated_outputs, train_noisy), dim=1)
            outputs = discriminator(gen_noise_pair, ref_batch)

            g_loss_ = 0.5 * torch.mean((outputs - 1.0) ** 2)
            # L1 loss between generated output and clean sample
            l1_dist = torch.abs(torch.add(generated_outputs, torch.neg(train_clean)))
            g_cond_loss = 100 * torch.mean(l1_dist)  # conditional loss
            g_loss = g_loss_ + g_cond_loss

            # backprop + optimize
            g_loss.backward()
            g_optimizer.step()

            train_bar.set_description(
                'Epoch {}: d_clean_loss {:.4f}, d_noisy_loss {:.4f}, g_loss {:.4f}, g_conditional_loss {:.4f}'
                    .format(epoch + 1, clean_loss.data[0], noisy_loss.data[0], g_loss.data[0], g_cond_loss.data[0]))

        # TEST model
        test_bar = tqdm(test_data_loader, desc='Test model and save generated audios')
        for test_file_names, test_noisy in test_bar:
            z = nn.init.normal(torch.Tensor(test_noisy.size(0), 1024, 8))
            if torch.cuda.is_available():
                test_noisy, z = test_noisy.cuda(), z.cuda()
            test_noisy, z = Variable(test_noisy), Variable(z)
            fake_speech = generator(test_noisy, z).data.cpu().numpy()  # convert to numpy array
            fake_speech = emphasis(fake_speech, emph_coeff=0.95, pre=False)

            for idx in range(fake_speech.shape[0]):
                generated_sample = fake_speech[idx]
                file_name = os.path.join('results',
                                         '{}_e{}.wav'.format(test_file_names[idx].replace('.npy', ''), epoch + 1))
                wavfile.write(file_name, sample_rate, generated_sample.T)

        # save the model parameters for each epoch
        g_path = os.path.join('epochs', 'generator-{}.pkl'.format(epoch + 1))
        d_path = os.path.join('epochs', 'discriminator-{}.pkl'.format(epoch + 1))
        torch.save(generator.state_dict(), g_path)
        torch.save(discriminator.state_dict(), d_path)


title: Convert Pytorch Batch Normalization weights to Tensorflow Batch Normalization

body: I have been working on converting ResNeXt50 weights from Pytorch into Tensorflow. I seem to have transferred all other weights from the convolution layers and fully connected head, however, I can't seem to correctly transfer the values inside the batch normalization layers. From what I understand, Pytorch batch normalization layers have a weight, bias, running_mean, and running_var. These correspond to the gamma, beta, moving_mean, and moving_variance in Tensorflow's batch normalization. I have inspected the weights of the batch normalization layer when I convert them into Tensorflow, and it seems that everything worked. However, when I go to make predictions, the output values of the Tensorflow batch normalization are way off from the output values of the Pytorch batch normalization. Is this discrepancy due to an incorrect conversion, or a difference in the behavior of batch normalization at inference in both libraries?


title: How To Get The Pixel Count Of A Segmented Area in an Image I used Vgg16 for Segmentation

body: I am new to deep learning but have succeeded in semantic segmentation of the image I am trying to get the pixel count of each class in the label. As an example in the image I want to get the pixel count of the carpet, or the chandelier or the light stand. How do I go about? Thanks any suggestions will help.


title: Add blocks of values to a tensor at specific locations in PyTorch

body:
I have a list of indices:
indx = torch.LongTensor([
    [ 0,  2,  0],
    [ 0,  2,  4],
    [ 0,  4,  0],
    [ 0, 10, 14],
    [ 1,  4,  0],
    [ 1,  8,  2],
    [ 1, 12,  0]
])

And I have a tensor of 2x2 blocks:
blocks = torch.FloatTensor([
    [[1.5818, 2.3108],
     [2.6742, 3.0024]],

    [[2.0472, 1.6651],
     [3.2807, 2.7413]],

    [[1.5587, 2.1905],
     [1.9231, 3.5083]],

    [[1.6007, 2.1426],
     [2.4802, 3.0610]],

    [[1.9087, 2.1021],
     [2.7781, 3.2282]],

    [[1.5127, 2.6322],
     [2.4233, 3.6836]],

    [[1.9645, 2.3831],
     [2.8675, 3.3770]]
])

What I want to do is to add each block at an index position to another tensor (i.e. so that it starts at that index). Let's assume that I want to add it to the following tensor:
a = torch.ones([2,18,18])

Is there any efficient way to do so? So far I came up only with:
i = 0
for b, x, y in indx:
   a[b, x:x+2, y:y+2] += blocks[i]
   i += 1

It is quite inefficient, I also tried to use index_add, but it did not work properly.


title: PyTorch for Object detection - Image augmentation

body:
I am using PyTorch for object detection and refining an existing model (transfer learning) as described in the following link -
https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html
While different transformations are used for image augmentation (horizontal flip in this tutorial), the tutorial doesnt mention anything on transforming the bounding box/annotation to ensure they are in line with the transformed image. Am I missing something basic here?


I want to convert the Pytorch-trained model to the tensorflow model and use the model on mobile devices. For this, I follow these steps; First I convert the pytorch trained model to onnx format. Then I convert the onnx format to the tensorflow model.
Firstly pytorch trained model to onnx;
import torch
import torch.onnx
from detectron2.modeling import build_model
from detectron2.modeling import build_backbone
from torch.autograd import Variable

model= build_backbone(cfg)
model.eval()

dummy_input = torch.randn(1,3,224, 224,requires_grad=True)

torch.onnx.export(model,dummy_input,"drive/Detectron2/model_final.onnx")

Then onnx to tflite model;
import onnx
import warnings
from onnx_tf.backend import prepare

model = onnx.load("drive/Detectron2/model_final.onnx")
tf_rep = prepare(model)
tf_rep.export_graph("drive/Detectron2/tf_rep.pb")

import tensorflow as tf

## TFLite Conversion
# Before conversion, fix the model input size
model = tf.saved_model.load("drive/Detectron2/tf_rep.pb")
model.signatures[tf.saved_model.DEFAULT_SERVING_SIGNATURE_DEF_KEY].inputs[0].set_shape([1, 3,224, 224])
tf.saved_model.save(model, "saved_model_updated", signatures=model.signatures[tf.saved_model.DEFAULT_SERVING_SIGNATURE_DEF_KEY])
# Convert
converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir='saved_model_updated', signature_keys=['serving_default'])
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS, tf.lite.OpsSet.SELECT_TF_OPS]
tflite_model = converter.convert()

# Save the model.
with open('drive/Detectron2/model.tflite', 'wb') as f:
  f.write(tflite_model)

## TFLite Interpreter to check input shape
interpreter = tf.lite.Interpreter(model_content=tflite_model)
interpreter.allocate_tensors()

# Get input and output tensors.
input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()

# Test the model on random input data.
input_shape = input_details[0]['shape']
print(input_shape)

But when I use the model on mobile devices, I get the following error;
java.lang.AssertionError: Error occurred when initializing ImageSegmenter: The input tensor should have dimensions 1 x height x width x 3. Got 1 x 3 x 224 x 224.

Where am I doing wrong?


title: Efficient way to avoid modifying parameter by inplace operation

body:
I have a model which has noisy linear layers (for which you can sample values from a mu and sigma parameter) and need to create two decorrelated outputs of it.

This means I have something like:

model.sample_noise()
output_1 = model(input)

with torch.no_grad():
     model.sample_noise()
     output_2 = model(input)

sample_noise actually modifies weights attached to the model according to a normal distribution.

But in the end this leads to

  RuntimeError: one of the variables needed for gradient computation has been
  modified by an inplace operation

The question actually is, what's the best way to avoid modifying these parameters. I could actually deepcopy the model every iteration and then use it for the second forward pass, but this does not sound very efficient to me.


title: GPU usage with Cuda and Pytorch

body:
When I started to use GPU for ML applications, i went to the task manager to see how much GPU it used and found six different graphs, in which just 2 were showing activity. Compute_0 and Copy. The Compute_0 had an interval between 85-95 % and Copy 5-15 %, but it just represented a 8-12% of the full use of the GPU.

I would like to know how it exactly works and find if there's a way to increase that 8-12% because it's also other graph with title Compute_1 indicating a constant zero usage, implying (by the name) that it can be used too.

This is the implementation code:

if torch.cuda.is_available():
    device = torch.device('cuda:0')

I'm working with a NVIDIA GeForce 940MX


title: python ValueError: non-broadcastable output operand with shape (389,1) doesn't match the broadcast shape (389,7)

body:
I am new to python programming language I am using Pytorch neural network LSTM to predict the feature price of the stock i know it's a common question but I am not being able to fix the issue

ValueError: non-broadcastable output operand with shape (389,1) doesn't match the broadcast shape (389,7)

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import torch.nn as nn
import torch

from torch.autograd import Variable
# Importing the training set
dataset_train = pd.read_csv('data/trraining_set.csv')
training_set = dataset_train.iloc[:,[1,2,5,6,7,8,9]].values

from sklearn.preprocessing import MinMaxScaler
sc = MinMaxScaler(feature_range = (0, 1))
training_set_scaled = sc.fit_transform(training_set)

X_train = []
y_train = []
for i in range(INPUT_SIZE, 766):
    X_train.append(training_set_scaled[i-INPUT_SIZE:i, 0])
    y_train.append(training_set_scaled[i, 0])
X_train, y_train = np.array(X_train), np.array(y_train)

# Reshaping
X_train = np.reshape(X_train, (X_train.shape[0], 1, X_train.shape[1]))

class RNN(nn.Module):
    def __init__(self, i_size, h_size, n_layers, o_size):
        super(RNN, self).__init__()

        self.rnn = nn.LSTM(
            input_size=i_size,
            hidden_size=h_size,
            num_layers=n_layers
        )
        self.out = nn.Linear(h_size, o_size)

    def forward(self, x, h_state):
        r_out, hidden_state = self.rnn(x, h_state)

        hidden_size = hidden_state[-1].size(-1)
        r_out = r_out.view(-1, hidden_size)
        outs = self.out(r_out)

        return outs, hidden_state
rnn = RNN(INPUT_SIZE, HIDDEN_SIZE, NUM_LAYERS, OUTPUT_SIZE)

optimiser = torch.optim.Adam(rnn.parameters(), lr=learning_rate)
criterion = nn.MSELoss()

hidden_state = None

for epoch in range(num_epochs):
    inputs = Variable(torch.from_numpy(X_train).float())
    labels = Variable(torch.from_numpy(y_train).float())

    output, hidden_state = rnn(inputs, hidden_state)

    loss = criterion(output.view(-1), labels)
    optimiser.zero_grad()
    loss.backward(retain_graph=True)
    optimiser.step()

    print('epoch {}, loss {}'.format(epoch,loss.item()))

dataset_test = pd.read_csv('data/test.csv')
real_stock_price = dataset_test.iloc[:,[1,2,5,6,7,8,9]].values

dataset_total = pd.concat((dataset_train['Open'], dataset_test['Open']), axis = 0)
inputs = dataset_total[len(dataset_total) - len(dataset_test) - INPUT_SIZE :].values
inputs = inputs.reshape(-1,1)
inputs = sc.transform(inputs)
X_test = []
for i in range(INPUT_SIZE, 329):
    X_test.append(inputs[i-INPUT_SIZE:i, 0])
X_test = np.array(X_test)
X_test = np.reshape(X_test, (X_test.shape[0], 1, X_test.shape[1]))

here is the following error

 ---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-117-249ca96baddc> in <module>
      8 inputs = dataset_total[len(dataset_total) - len(dataset_test) - INPUT_SIZE :].values
      9 inputs = inputs.reshape(-1,1)
---> 10 inputs = sc.transform(inputs)
     11 X_test = []
     12 for i in range(INPUT_SIZE, 329):

~\Anaconda3\lib\site-packages\sklearn\preprocessing\_data.py in transform(self, X)
    412                         force_all_finite="allow-nan")
    413
--> 414         X *= self.scale_
    415         X += self.min_
    416         return X

ValueError: non-broadcastable output operand with shape (389,1) doesn't match the broadcast shape (389,7)


title: Python not finding torch

body:
I want to use deepsaber.
For that I have to Install Torch.

I did this using pip install torch torchvision.

After running cd scripts/generation and ./script_generate.sh [path to song] python tells me:

Traceback (most recent call last):
  File "generate_stage1.py", line 16, in <module>
    from models import create_model
  File "/home/server/deepsaber/models/__init__.py", line 2, in <module>
    from .base_model import BaseModel
  File "/home/server/deepsaber/models/base_model.py", line 2, in <module>
    import torch
ModuleNotFoundError: No module named 'torch'
Traceback (most recent call last):
  File "generate_stage2.py", line 16, in <module>
    from models import create_model
  File "/home/server/deepsaber/models/__init__.py", line 2, in <module>
    from .base_model import BaseModel
  File "/home/server/deepsaber/models/base_model.py", line 2, in <module>
    import torch
ModuleNotFoundError: No module named 'torch'

using import torch works like a charm when using python

any ideas of what I can do?

The repos says:

  pytorch (installed as torch or via https://pytorch.org/get-started/locally/)


title: Vector-Tensor element-wise multiplication in Pytorch

body:
I am trying to extract the luminance from a tensor representing an image in Pytorch, and so I need to multiply element-wise a vector of size 3 (for the three RGB value weights) by a 3xNxN tensor representing the image such that I obtain a NxN matrix in the end where the three channels of the tensor have been summed with the weights given in the vector.

I guess there exists Pytorch operations that would help me do that without loops but I haven't found them.


title: Efficient batch derivative operations in PyTorch

body:
I am using Pytorch to implement a neural network that has (say) 5 inputs and 2 outputs

class myNetwork(nn.Module):
   def __init__(self):
      super(myNetwork,self).__init__()
      self.layer1 = nn.Linear(5,32)
      self.layer2 = nn.Linear(32,2)
   def forward(self,x):
      x = torch.relu(self.layer1(x))
      x = self.layer2(x)
      return x

Obviously, I can feed this an (N x 5) Tensor and get an (N x 2) result,

net = myNetwork()
nbatch = 100
inp = torch.rand([nbatch,5])
inp.requires_grad = True
out = net(inp)

I would now like to compute the derivatives of the NN output with respect to one element of the input vector (let's say the 5th element), for each example in the batch. I know I can calculate the derivatives of one element of the output with respect to all inputs using torch.autograd.grad, and I could use this as follows:

deriv = torch.zeros([nbatch,2])
for i in range(nbatch):
   for j in range(2):
      deriv[i,j] = torch.autograd.grad(out[i,j],inp,retain_graph=True)[0][i,4]

However, this seems very inefficient: it calculates the gradient of out[i,j] with respect to every single element in the batch, and then discards all except one. Is there a better way to do this?


title: How to translate the neural network of MLP from tensorflow to pytorch

body:
I have built up an MLP neural network using 'Tensorflow', which is stated as follow:
model_mlp=Sequential()
model_mlp.add(Dense(units=35, input_dim=train_X.shape[1], kernel_initializer='normal', activation='relu'))
model_mlp.add(Dense(units=86, kernel_initializer='normal', activation='relu'))
model_mlp.add(Dense(units=86, kernel_initializer='normal', activation='relu'))
model_mlp.add(Dense(units=10, kernel_initializer='normal', activation='relu'))
model_mlp.add(Dense(units=1))

I want to convert the above MLP code using pytorch. How to do it? I try to do it as follows:
    class MLP(nn.Module):
    def __init__(self):
        super(MLP, self).__init__()
        self.fc1 = nn.Linear(train_X.shape[1],35)
        self.fc2 = nn.Linear(35, 86)
        self.fc3 = nn.Linear(86, 86)
        self.fc4 = nn.Linear(86, 10)
        self.fc5 = nn.Linear(10, 1)
    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.relu(self.fc3(x))
        x = F.relu(self.fc4(x))
        x = self.fc5(x)
        return x
    def predict(self, x_test):
        x_test = torch.from_numpy(x_test).float()
        x_test = self.forward(x_test)
        return x_test.view(-1).data.numpy()
model = MLP()

I use the same dataset but the two codes give two different answers. Code written in Tensorflow always produce a much better results than using the code written in Pytorch. I wonder if my code in pytorch is not correct.
In case my written code in PyTorch is correct, I wonder how to explain the differences. I am looking forward to any replies.


title: cuda is not available on my pytorch, but I can't find anything wrong with the version

body:
for some reason I have to use cuda version10.0 instead of upgrading it
The version of DriverAPI is higher than RunTimeAPI but somebody told me thats OK
Others who asked the same question at last found their version was not match. emmmm not like me
details here
OS:Windows10
Python 3.7&3.8 both tried
result of 'nvcc -V':
nvcc: release 10.0, V10.0.130
nvidia-smi:
NVIDIA-SMI 451.67       Driver Version: 451.67       CUDA Version: 11.0
conda list: cudatoolkit  10.0.130  0
import torch
print(torch.version.cuda)-----None
torch.cuda.is_available()-----False


title: Pytorch's Autograd does not support complex matrix inversion, does anyone have a workaround?

body:
Somewhere in my loss function, I invert a complex matrix of size 64*64. Although complex matrix inversion is supported for torch.tensor, the gradient cannot be computed in the training loop as I get this error:
RuntimeError: inverse does not support automatic differentiation for outputs with complex type.
Does anyone have a workaround for this issue? a custom function instead of torch.inverse maybe?


title: Pytorch Global Pruning is not reducing the size of the model

body:
I am trying to Prune my Deep Learning model via Global Pruning. The original UnPruned model is about 77.5 MB. However after pruning, when I am saving the model, the size of the model is the same as the original. Can anyone help me with this issue?
Below is the Pruning code:-
import torch.nn.utils.prune as prune

parameters_to_prune = (
(model.encoder[0], ‘weight’),
(model.up_conv1[0], ‘weight’),
(model.up_conv2[0], ‘weight’),
(model.up_conv3[0], ‘weight’),
)
print(parameters_to_prune)

prune.global_unstructured(
parameters_to_prune,
pruning_method=prune.L1Unstructured,
amount=0.2,
)

print(
“Sparsity in Encoder.weight: {:.2f}%”.format(
100. * float(torch.sum(model.encoder[0].weight == 0))
/ float(model.encoder[0].weight.nelement())
)
)
print(
“Sparsity in up_conv1.weight: {:.2f}%”.format(
100. * float(torch.sum(model.up_conv1[0].weight == 0))
/ float(model.up_conv1[0].weight.nelement())
)
)
print(
“Sparsity in up_conv2.weight: {:.2f}%”.format(
100. * float(torch.sum(model.up_conv2[0].weight == 0))
/ float(model.up_conv2[0].weight.nelement())
)
)
print(
“Sparsity in up_conv3.weight: {:.2f}%”.format(
100. * float(torch.sum(model.up_conv3[0].weight == 0))
/ float(model.up_conv3[0].weight.nelement())
)
)

print(
“Global sparsity: {:.2f}%”.format(
100. * float(
torch.sum(model.encoder[0].weight == 0)
+ torch.sum(model.up_conv1[0].weight == 0)
+ torch.sum(model.up_conv2[0].weight == 0)
+ torch.sum(model.up_conv3[0].weight == 0)
)
/ float(
model.encoder[0].weight.nelement()
+ model.up_conv1[0].weight.nelement()
+ model.up_conv2[0].weight.nelement()
+ model.up_conv3[0].weight.nelement()
)
)
)

**Setting Pruning to Permanent**
prune.remove(model.encoder[0], “weight”)
prune.remove(model.up_conv1[0], “weight”)
prune.remove(model.up_conv2[0], “weight”)
prune.remove(model.up_conv3[0], “weight”)

**Saving the model**
PATH = “C:\PrunedNet.pt”
torch.save(model.state_dict(), PATH)


I'm using rtx 3080, python 3.7, torch 1.7.1, torchvision 0.8.2
i tried chaning python version and batch size etc. But doesn't work..
what can i do to fix this problem
Traceback (most recent call last):
  File "run_captioning.py", line 882, in <module>
    main()
  File "run_captioning.py", line 861, in main
    global_step, avg_loss = train(args, train_dataset, val_dataset, model, tokenizer)
  File "run_captioning.py", line 424, in train
    for step, (img_keys, batch) in enumerate(train_dataloader):
  File "C:\Users\User\.conda\envs\py37\lib\site-packages\torch\utils\data\dataloader.py", line 352, in __iter__
    return self._get_iterator()
  File "C:\Users\User\.conda\envs\py37\lib\site-packages\torch\utils\data\dataloader.py", line 294, in _get_iterator
    return _MultiProcessingDataLoaderIter(self)
  File "C:\Users\User\.conda\envs\py37\lib\site-packages\torch\utils\data\dataloader.py", line 801, in __init__
    w.start()
  File "C:\Users\User\.conda\envs\py37\lib\multiprocessing\process.py", line 112, in start
    self._popen = self._Popen(self)
  File "C:\Users\User\.conda\envs\py37\lib\multiprocessing\context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Users\User\.conda\envs\py37\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
  File "C:\Users\User\.conda\envs\py37\lib\multiprocessing\popen_spawn_win32.py", line 65, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Users\User\.conda\envs\py37\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: cannot serialize '_io.TextIOWrapper' object

(py37) D:\Oscar\oscar>Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\User\.conda\envs\py37\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\Users\User\.conda\envs\py37\lib\multiprocessing\spawn.py", line 115, in _main
    self = reduction.pickle.load(from_parent)
EOFError: Ran out of input


title: Pytorch: test loss becoming nan after some iteration

body: I am trying to train a deep learning architecture, the model trains perfectly. I am testing after each epoch. For 7 epoch all the loss and accuracy seems okay but at 8 epoch during the testing test loss becomes nan. I have checked my data, it got no nan. Also my test accuracy is higher than train which is weird. Train data size is 37646 and test is 18932 so it should be enough. Before becoming nan test started to become very high around 1.6513713663602217e+30. This is really weird and I don't understand why is happening. Any help or suggestion is much appreciated.


title: How to add layers to a pretrained model in PyTorch?

body:
I want to add layer normalization function just after AdaptiveAvgPool2d layer and L2 normalization after fc layer. i wanted my fc layer  output to be 200 so tried not to include fc layer instead of it make new fc layer, but it did not remove fc layers comes with pretrained model, i am using googlenet.
my code:
class GoogleNet(nn.Module):
    def __init__(self):
        super(GoogleNet,self).__init__()
        self.model = googlenet_pytorch.GoogLeNet.from_pretrained('googlenet')

        self.fc = nn.Linear(1024,200, bias=False),

    def forward(self, x):
        batch_size ,_,_,_ =x.shape
        x = self.model.extract_features(x)
        x = self.model._avg_pooling(x)
        x = F.layer_norm(x,x.size[1:],elementwise_affine=False)
        x = self.fc(x)
        x = F.normalize(x, p=2, dim=1)
        return x

Output I am getting:
      .....
      .....
      .....

(aux1): InceptionAux(
      (conv): BasicConv2d(
        (conv): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn): BatchNorm2d(128, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)
      )
      (fc1): Linear(in_features=2048, out_features=1024, bias=True)
      (fc2): Linear(in_features=1024, out_features=1000, bias=True)
    )
    (aux2): InceptionAux(
      (conv): BasicConv2d(
        (conv): Conv2d(528, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn): BatchNorm2d(128, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)
      )
      (fc1): Linear(in_features=2048, out_features=1024, bias=True)
      (fc2): Linear(in_features=1024, out_features=1000, bias=True)
    )
    (avgpool): AdaptiveAvgPool2d(output_size=(1, 1))
    (dropout): Dropout(p=0.2, inplace=False)
    (fc): Linear(in_features=1024, out_features=1000, bias=True)
  )
)

Output I want:
  ......
  ......
  ......

  (aux1): None
  (aux2): None
  (avgpool): AdaptiveAvgPool2d(output_size=(1, 1))
  ** layer normalization here**
  (dropout): Dropout(p=0.2, inplace=False)
  (fc): Linear(in_features=1024, out_features=200, bias=False)
  **L2 normalization here**
)

If someone needs to know the solution code for this problem, with the help of iacob's answer I solved it, I added it as an answer.


title: Setting custom loss causes RuntimeError: Variable data has to be a tensor, but got Variable in Pytorch

body:
I have custom class for the net definition:

class PyTorchUNet(Model):
    ....
    def set_loss(self):
            if self.activation_func == 'softmax': #this is working example
                 loss_function = partial(mixed_dice_cross_entropy_loss,
                                dice_loss=multiclass_dice_loss,
                                cross_entropy_loss=nn.CrossEntropyLoss(),
                                dice_activation='softmax',
                                dice_weight=self.architecture_config['model_params']['dice_weight'],
                                cross_entropy_weight=self.architecture_config['model_params']['bce_weight']
                                )

            elif self.activation_func == 'sigmoid':
                    loss_function = designed_loss #setting will cause error on validation

            else:
                raise Exception('Only softmax and sigmoid activations are allowed')
            self.loss_function = [('mask', loss_function, 1.0)]

    def designed_loss(output, target):
       target = target.long() # this should make variable to tensor
       return lovasz_hinge(output, target)

    # this is just as it from github
    def lovasz_hinge(logits, labels, per_image=True, ignore=None):
        """
        Binary Lovasz hinge loss
          logits: [B, H, W] Variable, logits at each pixel (between -\infty and +\infty)
          labels: [B, H, W] Tensor, binary ground truth masks (0 or 1)
          per_image: compute the loss per image instead of per batch
          ignore: void class id
        """
        if per_image:
            loss = mean(lovasz_hinge_flat(*flatten_binary_scores(log.unsqueeze(0), lab.unsqueeze(0), ignore))
                              for log, lab in zip(logits, labels))
        else:
            loss = lovasz_hinge_flat(*flatten_binary_scores(logits, labels, ignore))
        return loss

    def lovasz_hinge_flat(logits, labels):
        """
        Binary Lovasz hinge loss
          logits: [P] Variable, logits at each prediction (between -\infty and +\infty)
          labels: [P] Tensor, binary ground truth labels (0 or 1)
          ignore: label to ignore
        """
        if len(labels) == 0:
            # only void pixels, the gradients should be 0
            return logits.sum() * 0.
        signs = 2. * labels.float() - 1.
        errors = (1. - logits * Variable(signs))
        errors_sorted, perm = torch.sort(errors, dim=0, descending=True)
        perm = perm.data
        gt_sorted = labels[perm]
        grad = lovasz_grad(gt_sorted)
        loss = torch.dot(F.elu(errors_sorted), Variable(grad))
        return loss
    def mean(l, ignore_nan=False, empty=0):
        """
        nanmean compatible with generators.
        """
        l = iter(l)
        if ignore_nan:
            l = ifilterfalse(np.isnan, l)
        try:
            n = 1
            acc = next(l)
        except StopIteration:
            if empty == 'raise':
                raise ValueError('Empty mean')
            return empty
        for n, v in enumerate(l, 2):
            acc += v
        if n == 1:
            return acc
        return acc / n

Working example:

def mixed_dice_cross_entropy_loss(output, target, dice_weight=0.5, dice_loss=None,
                                  cross_entropy_weight=0.5, cross_entropy_loss=None, smooth=0,
                                  dice_activation='softmax'):
    num_classes_without_background = output.size(1) - 1
    dice_output = output[:, 1:, :, :]
    dice_target = target[:, :num_classes_without_background, :, :].long()
    cross_entropy_target = torch.zeros_like(target[:, 0, :, :]).long()
    for class_nr in range(num_classes_without_background):
        cross_entropy_target = where(target[:, class_nr, :, :], class_nr + 1, cross_entropy_target)
    if cross_entropy_loss is None:
        cross_entropy_loss = nn.CrossEntropyLoss()
    if dice_loss is None:
        dice_loss = multiclass_dice_loss
    return dice_weight * dice_loss(dice_output, dice_target, smooth,
                                   dice_activation) + cross_entropy_weight * cross_entropy_loss(output,
                                                                                cross_entropy_target)

def multiclass_dice_loss(output, target, smooth=0, activation='softmax'):
    """Calculate Dice Loss for multiple class output.

    Args:
        output (torch.Tensor): Model output of shape (N x C x H x W).
        target (torch.Tensor): Target of shape (N x H x W).
        smooth (float, optional): Smoothing factor. Defaults to 0.
        activation (string, optional): Name of the activation function, softmax or sigmoid. Defaults to 'softmax'.

    Returns:
        torch.Tensor: Loss value.

    """
    if activation == 'softmax':
        activation_nn = torch.nn.Softmax2d()
    elif activation == 'sigmoid':
        activation_nn = torch.nn.Sigmoid()
    else:
        raise NotImplementedError('only sigmoid and softmax are implemented')

    loss = 0
    dice = DiceLoss(smooth=smooth)
    output = activation_nn(output)
    num_classes = output.size(1)
    target.data = target.data.float()
    for class_nr in range(num_classes):
        loss += dice(output[:, class_nr, :, :], target[:, class_nr, :, :])
    return loss / num_classes

As result I keep getting:

RuntimeError: Variable data has to be a tensor, but got Variable

How to fix the problem?


title: Pytorch : Results of vector multiplications are different for same input

body:
I did not understand why these multiplication outputs are different.

print(features*weights)
print('------------')
print(features*weights.view(5,1))
print('------------')
print(torch.mm(features,weights.view(5,1)))

outputs:

tensor([[ 0.1314, -0.2796,  1.1668, -0.1540, -2.8442]])
------------
tensor([[ 0.1314, -0.7035, -0.8472,  0.9971, -1.5130],
        [ 0.0522, -0.2796, -0.3367,  0.3963, -0.6013],
        [-0.1809,  0.9688,  1.1668, -1.3733,  2.0837],
        [-0.0203,  0.1086,  0.1308, -0.1540,  0.2336],
        [ 0.2469, -1.3224, -1.5927,  1.8745, -2.8442]])
------------
tensor([[-1.9796]])


title: How to stabilize the output size of TransposedConv2d in PyTorch

body:
I have an encoder-decoder regression network written in PyTorch which takes RGB images as input. Output is a 2D image that supposed to have the same size as the input.

For the case of patch-based analysis that I use 224x224 sized rgb patches, everything works well. But when it comes to variable sized inputs such as 681x1024 or 777x900, things start to get messy. Because the output size is almost always unequal to the input size and this is hinders me from implementing the structure I intend to. Here is why:

Lets say I would like to build a multi-branch, hybrid fusion network like above (taken from this paper). In this network, one of the outputs of the global branch becomes a part of the input of local branch. In other words, size of GUIDANCE MAP should be equal to LIDAR image.

Now, my intention is to have the same functionality with variable sized input. My two branches has the structures

self.global = nn.Sequential(Conv2d(3, 16, 9, same_padding=True, bn=bn),
                            nn.MaxPool2d(2),
                            Conv2d(16, 32, 7, same_padding=True, bn=bn),
                            nn.MaxPool2d(2),
                            Conv2d(32, 16, 7, same_padding=True, bn=bn),
                            Conv2d(16, 8, 7, same_padding=True, bn=bn),

                            DeConv2d(8, 8, 3, stride=2, same_padding=True),
                            Conv2d(8, 4, 7, same_padding=True, bn=bn),
                            DeConv2d(4, 4, 3, stride=2, same_padding=True),
                            Conv2d(4, 1, 4, same_padding=False, bn=bn))

self.local = nn.Sequential(Conv2d(3, 24, 5, same_padding=True, bn=bn),
                        nn.MaxPool2d(2),
                        Conv2d(24, 48, 3, same_padding=True, bn=bn),
                        nn.MaxPool2d(2),
                        Conv2d(48, 24, 3, same_padding=True, bn=bn),
                        Conv2d(24, 12, 3, same_padding=True, bn=bn),

                        DeConv2d(12, 12, 3, stride=2, same_padding=True),
                        Conv2d(12, 6, 3, same_padding=True, bn=bn),
                        DeConv2d(6, 6, 3, stride=2, same_padding=True),
                        Conv2d(6, 1, 4, same_padding=False, bn=bn))

They work properly when I need to fuse their output, since their output have the same size. But when I want to fuse the input image with the output of the global network, I get the

runtime error : invalid argument 0: Sizes of tensors must match
except in dimension 1.
Got 294 and 292 in dimension 2 at ...

I am aware that these size changes are caused by TransposedConv2d functions inability to restore the resolution that have been decreased by MaxPool2d with an exact precision. But I wonder if there is a way to restore the size explicitly and dynamically.

TLDR; I would like to restore the image resolution decreased by MaxPool2d using TransposedConv2d for inputs with uneven sizes such as 681x1024 or 777x900.

I appreciate any help or suggestion.


title: somehow my accuracy is very low on cifar10?

body:
with torch.no_grad():
    for data in test_loader:
        images,labels = data
        images, labels = images.to(device), labels.to(device)
        outputs, features = net(images)
        _ , predicted = torch.max(outputs,1)
        total += labels.size(0)
        correct += (predicted==labels).sum().item()
    print('Accuracy of the network on the 10000 test images: %d %%' % (
            100 * (correct / total)))

How do I obtain GPU results with a label?
I got almost 10% accuracy but my original training is accuracy is 70%.


title: Define a neural network with PyTorch to train images with labels

body:
I have a 3 folders (train, validation and test) where the folder contains 1000, 100 and 100 images respectively with sizes 128x128 and a labels.txt file which contains the image name, center points (as x,y) and radius of the circles for all images in "images" directory. Here are the first few examples of the details in the labels.txt file:

0000,   0.68 ,   0.61 ,   0.24
0001,   0.7  ,   0.33 ,   0.26
0002,   0.52 ,   0.61 ,   0.22
0003,   0.5  ,   0.49 ,   0.46
0004,   0.43 ,   0.43 ,   0.36
0005,   0.36 ,   0.38 ,   0.19
0006,   0.4  ,   0.61 ,   0.3
0007,   0.81 ,   0.5  ,   0.15
0008,   0.7  ,   0.36 ,   0.26
0009,   0.29 ,   0.56 ,   0.2

And this is an example of how the image looks like (First training image in the training dataset).

The image is fed into a neural network, whose outputs are the center and radius of the circle within the image (the image contains a rectangle, triangle and a circle). The domain considered is "[0,1]x[0,1] "box". The goal is to paint the circle. The images are RGB so therefore I do know that the shape of the image read is (3,128,128). I have created a dataset class and a data loader by doing some research, but as I am new to pytorch I am unsure of how to define the neural network. The code is as follows:

Dataset class

class ShapesDataset(Dataset):

    def __init__(self, dataset_dir):
        """
        Initializing dataset by generating a dicitonary of labels, where an image file name is the key
        and its labels are the contents of that entry in the dictionary. Images are not loaded. This way it
        is possible to iterate over arbitrarily large datasets (limited by labels dicitonary fitting
        in memory, which is not a problem in practice)

        Args:
            dataset_dir : path to directory with images and labels. In this directory we expect to find
                          a directory called "images" containing the input images, and a file called
                          "labels.txt" containing desired labels (coefficients)
        """

        self.dataset_dir = dataset_dir
        self.labels_dict = self.gen_labels_dict()
        self.images_keys = list(self.labels_dict)  # getting the keys of the dictionary as list
        self.images_keys.sort()                    # sorting so as to have in alphabetical order

    def __len__(self):
        return len(self.labels_dict)

    def __getitem__(self, index):
        """
        This funtion makes it possible to iterate over the ShapesDataset
        Args:
            index: running index of images

        Returns:
            sample: a dicitionary with three entries:
                    1. 'image'  contains the image
                    2. 'labels' contains labels (coeffs) corresponding to image
                    3. 'fname'  contains name of file (image_key) - may be useful for debugging
        """
        image_key = self.images_keys[index]     # recall - key is the file name of the corresponding image
        image = np.array(Image.open(image_key)) # image has shape: (128, 128, 3)
        image = image/255.0                     # simple normalization - just to maintain small numbers
        image = np.transpose(image, (2, 0, 1))  # network needs RGB channels to be first index
        labels = self.labels_dict[image_key]
        sample = {'image': image, 'labels': labels, 'fname':image_key}

        return sample

    def gen_labels_dict(self):
        """
        This fucntion generates a dictionary of labels

        Returns:
            labels_dict: the key is image file name and the value is the corresponding
            array of labels
        """

        labels_fname = self.dataset_dir + "/labels.txt"
        labels_dict = {}
        with open(labels_fname, "r") as inp:
            for line in inp:
                line = line.split('\n')[0]                                      # remove '\n' from end of line
                line = line.split(',')
                key  = self.dataset_dir + '/images/' + line[0].strip() + ".png" # image file name is the key
                del line[0]

                list_from_line = [float(item) for item in line]
                labels_dict[key] = np.asarray(list_from_line, dtype=np.float32)

        return labels_dict

Data loader

train_dir      = "./train/"
validation_dir = "./validation/"
test_dir       = "./test/"

train_dataset = ShapesDataset(train_dir)

train_loader = DataLoader(train_dataset,
                          batch_size=32,
                          shuffle=True)

validation_dataset = ShapesDataset(validation_dir)

validation_loader = DataLoader(validation_dataset,
                               batch_size=1,
                               shuffle=False)

test_dataset = ShapesDataset(test_dir)

test_loader = DataLoader(test_dataset,
                          batch_size=1,
                          shuffle=False)

print("train loader examples     :", len(train_dataset))
print("validation loader examples:", len(validation_dataset))
print("test loader examples      :", len(test_dataset))

#Output:
#train loader examples     : 1000
#validation loader examples: 100
#test loader examples      : 100

And finally the definition of the neural network:

**EDIT: Definition of NN **

class CircleNet(nn.Module):    # nn.Module is parent class
    #Not sure if I am on the right track.
    def __init__(self):
        self.conv1 = nn.Conv2d(in_channels=3, out_channels=12, kernel_size=3, stride=1, padding=1)

        # We'll apply max pooling with a kernel size of 2
        self.pool = nn.MaxPool2d(kernel_size=2)

        # A second convolutional layer takes 12 input channels, and generates 12 outputs
        self.conv2 = nn.Conv2d(in_channels=12, out_channels=12, kernel_size=3, stride=1, padding=1)

        # A third convolutional layer takes 12 inputs and generates 24 outputs
       self.conv3 = nn.Conv2d(in_channels=12, out_channels=24, kernel_size=3, stride=1, padding=1)

        # A drop layer deletes 20% of the features to help prevent overfitting
        self.drop = nn.Dropout2d(p=0.2)

        # Our 128x128 image tensors will be pooled twice with a kernel size of 2. 128/2/2 is 32.
        # So our feature tensors are now 32 x 32, and we've generated 24 of them, so the array is 32x32x24
        # We need to flatten these and feed them to a fully-connected layer
        # to map them to  the probability for each class
        self.fc = nn.Linear(in_features=32 * 32 * 24, out_features=num_classes)

    #I am struggling to move forward from here
    def forward(self, x):
        """
        Feed forward through network
        Args:
            x - input to the network

        Returns "out", which is the network's output
        """
        # Use a relu activation function after convolution 1 and pool
        x = F.relu(self.pool(self.conv1(x)))

        # Use a relu activation function after convolution 2 and pool
        x = F.relu(self.pool(self.conv2(x)))

        # Select some features to drop after the 3rd convolution to prevent overfitting
        x = F.relu(self.drop(self.conv3(x)))

        # Only drop the features if this is a training pass
        x = F.dropout(x, training=self.training)

        # Flatten
        x = x.view(-1, 32 * 32 * 24)
        # Feed to fully-connected layer to predict class
        x = self.fc(x)
        # Return class probabilities via a softmax function

        return F.log_softmax(x, dim=1)


title: Dimension mismatch CNN LibTorch/PyTorch

body:
I have a CNN structure in LibTorch but the dimensions are not ok. My objective is to input a 3 channel 64x64 image and output a logistic regression float for a DGAN. Last layer I set as input channels 36 because if I remove that layer the output neuron had 6x6 dimension so I guesses that was the required dimension for the input of the fully connected. I would like to know:

What do you normally do to check dimensions in LibTorch or Pytorch (i.e. check the required size for the last module, check how many trainable parameters has each layer ...)
What is the error in this case

   #include <torch/torch.h>
   #include "parameters.h"
   using namespace torch;

    class DCGANDiscriminatorImpl: public nn::Module {

    private:
        nn::Conv2d conv1, conv2, conv3, conv4;
        nn::BatchNorm2d batch_norm1, batch_norm2;
        nn::Linear fc1;

    public:
        DCGANDiscriminatorImpl()
            :conv1(nn::Conv2dOptions(3, 64, 4).stride(2).padding(1).bias(false)),

             conv2(nn::Conv2dOptions(64, 128, 4).stride(2).padding(1).bias(false)),

             batch_norm1(128),

             conv3(nn::Conv2dOptions(128, 256, 4).stride(2).padding(1).bias(false)),

             batch_norm2(256),

             conv4(nn::Conv2dOptions(256, 1, 3).stride(1).padding(0).bias(false)),

             fc1(6*6, 1)

        {
            register_module("conv1", conv1);
            register_module("conv2", conv2);
            register_module("conv3", conv3);
            register_module("conv4", conv4);
            register_module("batch_norm1", batch_norm1);
            register_module("batch_norm2", batch_norm2);
            register_module("fc1", fc1);

        }

        Tensor forward(torch::Tensor x)
        {
            x = leaky_relu(conv1(x), cte::NEGATIVE_SLOPE);
            x = leaky_relu(batch_norm1(conv2(x)), cte::NEGATIVE_SLOPE);
            x = leaky_relu(batch_norm2(conv3(x)), cte::NEGATIVE_SLOPE);
            x = sigmoid(fc1(x));
            return x;
        }

    };

    TORCH_MODULE(DCGANDiscriminator);

The error I get is:

libc++abi.dylib: terminating with uncaught exception of type std::runtime_error: size mismatch, m1: [131072 x 8], m2: [36 x 1] at ../aten/src/TH/generic/THTensorMath.cpp:136


title: Should Decoder Prediction Be Detached in PyTorch Training?

body:
Hi guys I have recently started to use PyTorch for my research that needs the encoder-decoder framework. PyTorch's tutorials on this are wonderful, but there's a little problem: when training the decoder without teacher forcing, which means the prediction of the current time step is used as the input to the next, should the prediction be detached?

In this PyTorch tutorial, detach is used (decoder_input = topi.squeeze().detach()  # detach from history as input
), but it is not the case in this one (top1 = output.max(1)[1]; output = (trg[t] if teacher_force else top1)).

Both tutorials are RNN-based, so I am not sure about Transformer-based architectures. Would be grateful if someone could point out which one is the better practice :).


title: converting pytorch 2d padding to tensorflow keras

body:
What would be the equivalent of this:

nn.ReflectionPad2d(1)

in TensorFlow 2? The above line is from PyTorch.

In TensorFlow 2 Keras, I'm currently looking into using tf.pad() as a TF version of that, but it seems PyTorch is able to handle varying dimensions with that single integer 1. For instance, if it gets an input of shape [batch size, 1, 1, 100], nn.ReflectionPad2D will handle that well, but in TensorFlow, I get an error if I try to use

tf.pad(t, tf.constant([0,0], [1,1], [1,1], [0,0]]), 'REFLECT')

Any suggestions on how to replicate nn.ReflectinPad2d in TensorFlow 2 keras? Thanks!


title: Single shot multi-dimension indexing in torch - perhaps with index_select or gather?

body:
I am performing a multi-index re-arrangement of a matrix based upon its correspondence data. Right now, I and doing this with a pair of index_select calls, but this is very memory inefficient (n^2 in terms of memory usage), and is not exactly ideal in terms of computation efficiency either. Is there some way that I can boil my operation down into a single .gather or .index_select call?
What I essentially want to do is when given a source array of shape (I,J,K), and an array of indices of shape (I,J,2), produce a result which meets the condition:
result[i][j][:] = source[idx[i][j][0]] [idx[i][j][1]] [:]

Here's a runnable toy example of how I'm doing things right now:
source = torch.tensor([[1,2,3], [4,5,6], [7,8,9], [10,11,12]])
indices = torch.tensor([[[2,2],[3,1],[0,2]],[[0,2],[0,1],[0,2]],[[0,2],[0,1],[0,2]],[[0,2],[0,1],[0,2]]])

ax1 = torch.index_select(source,0,indices[:,:,0].flatten())
ax2 = torch.index_select(ax1, 1, indices[:,:,1].flatten())

result = ax2.diagonal().reshape(indices.shape(0), indices.shape(1))

This approach works for me only because my images are rather small, so they fit into memory even with the diagonalization issue. Regardless, I am producing a pretty massive amount of data that doesn't need to be. Furthermore, if K becomes large, then this issue gets worse exponentially. Perhaps I'm just missing something obvious in the documentation, but I feel like this is a problem somebody else has to have run into before that can help me out!


title: is binary cross entropy an additive function?

body:
I am trying to train a machine learning model where the loss function is binary cross entropy, because of gpu limitations i can only do batch size of 4 and i'm having lot of spikes in the loss graph. So I'm thinking to back-propagate after some  predefined batch size(>4). So it's like i'll do 10 iterations of batch size 4 store the losses, after 10th iteration add the losses and back-propagate. will it be similar to batch size of 40.
TL;DR
f(a+b) = f(a)+f(b) is it true for binary cross entropy?


title: Using zero_grad() after loss.backward(), but still receives RuntimeError: "Trying to backward through the graph a second time..."

body:
Below is my implementation of a2c using PyTorch. Upon learning about backpropagation in PyTorch, I have known to zero_grad() the optimizer after each update iteration. However, there is still a RunTime error on second-time backpropagation.
def torchworker(number, model):
  worker_env = gym.make("Taxi-v3").env
  max_steps_per_episode = 2000
  worker_opt = optim.Adam(lr=5e-4, params=model.parameters())

  p_history = []
  val_history = []
  r_history = []

  running_reward = 0
  episode_count = 0
  under = 0

  start = time.time()

  for i in range(2):
    state = worker_env.reset()
    episode_reward = 0
    penalties = 0
    drop = 0
    print("Episode {} begins ({})".format(episode_count, number))
    worker_env.render()
    criterion = nn.SmoothL1Loss()

    time_solve = 0

    for _ in range(1, max_steps_per_episode):
      #worker_env.render()
      state = torch.tensor(state, dtype=torch.long)
      action_probs = model.forward(state)[0]
      critic_value = model.forward(state)[1]
      val_history.append((state, critic_value[0]))

      # Choose action
      action = np.random.choice(6, p=action_probs.detach().numpy())
      p_history.append(torch.log(action_probs[action]))

      # Apply chosen action
      state, reward, done, _ = worker_env.step(action)
      r_history.append(reward)
      episode_reward += reward
      time_solve += 1

      if reward == -10:
        penalties += 1

      elif reward == 20:
        drop += 1

      if done:
        break

    # Update running reward to check condition for solving
    running_reward = (running_reward * (episode_count) + episode_reward) / (episode_count + 1)

    # Calculate discounted returns
    returns = deque(maxlen=3500)
    discounted_sum = 0
    for r in r_history[::-1]:
      discounted_sum = r + gamma * discounted_sum
      returns.appendleft(discounted_sum)

    # Calculate actor losses and critic losses
    loss_actor_value = 0
    loss_critic_value = 0
    history = zip(p_history, val_history, returns)
    for log_prob, value, ret in history:
      diff = ret - value[1]
      loss_actor_value += -log_prob * diff
      ret_tensor = torch.tensor(ret, dtype=torch.float32)
      loss_critic_value += criterion(value[1], ret_tensor)

    loss = loss_actor_value + 0.1 * loss_critic_value
    print(loss)

    # Update params
    loss.backward()
    worker_opt.step()
    worker_opt.zero_grad()

    # Log details
    end = time.time()
    episode_count += 1
    if episode_count % 1 == 0:
        worker_env.render()

    if running_reward > -50:  # Condition to consider the task solved
        under += 1

    if under > 5:
        print("Solved at episode {} !".format(episode_count))
        break

I believe there may be something to do with the architecture of my AC model, so I also include it here for reference.
class ActorCriticNetwork(nn.Module):
  def __init__(self, num_inputs, num_hidden, num_actions):
    super(ActorCriticNetwork, self).__init__()
    self.embed = nn.Embedding(500, 10)
    self.fc1 = nn.Linear(10, num_hidden * 2)
    self.fc2 = nn.Linear(num_hidden * 2, num_hidden)
    self.c = nn.Linear(num_hidden, 1)
    self.fc3 = nn.Linear(num_hidden, num_hidden)
    self.a = nn.Linear(num_hidden, num_actions)

  def forward(self, x):
    out = F.relu(self.embed(x))
    out = F.relu(self.fc1(out))
    out = F.relu(self.fc2(out))
    critic = self.c(out)
    out = F.relu(self.fc3(out.detach()))
    actor = F.softmax(self.a(out), dim=-1)
    return actor, critic

Would you please tell me what the mistake here is? Thank you in advance.


title: differing results when using model to infer on a batch vs individual with pytorch

body:
I have a neural network which takes input tensor of dimension (batch_size, 100, 1, 1) and produces an output tensor of dimension (batch_size, 3, 64, 64).
I have differing results when using model to infer on a batch of two elements and on inferring on  elements individually.
With the below code I initialize a pytorch tensor of dimension (2, 100, 1, 1). I pass this tensor through the model and  I take the first element of the model output and store in variable result1. For result2  I just directly run the first element of my original input tensor through my model.
inputbatch=torch.randn(2, Z_DIM, 1, 1, device=device)
inputElement=inputbatch[0].unsqueeze(0)

result1=model(inputbatch)[0]
result2=model(inputElement)

My expectation was result1 and result2 would be same. But result1 and result2 are entirely different. Could anyone explain why the two outputs differ.


title: BrokenPipe ERROR File "/usr/lib/python3.6/multiprocessing/connection.py", line 368, in _send

body:
I'm a chatbot that uses model DistilBertForQuestionAnswering but during processing the question I get Broken Pipe Error
self._writer.send_bytes(obj)
Aug 25 15:02:57 ubuntu-s-4vcpu-8gb-fra1-01 gunicorn[8237]:   File "/usr/lib/python3.6/multiprocessing/connection.py", line 200, in send_bytes
Aug 25 15:02:57 ubuntu-s-4vcpu-8gb-fra1-01 gunicorn[8237]:     self._send_bytes(m[offset:offset + size])
Aug 25 15:02:57 ubuntu-s-4vcpu-8gb-fra1-01 gunicorn[8237]:   File "/usr/lib/python3.6/multiprocessing/connection.py", line 404, in _send_bytes
Aug 25 15:02:57 ubuntu-s-4vcpu-8gb-fra1-01 gunicorn[8237]:     self._send(header + buf)
Aug 25 15:02:57 ubuntu-s-4vcpu-8gb-fra1-01 gunicorn[8237]:   File "/usr/lib/python3.6/multiprocessing/connection.py", line 368, in _send
Aug 25 15:02:57 ubuntu-s-4vcpu-8gb-fra1-01 gunicorn[8237]:     n = write(self._handle, buf)
Aug 25 15:02:57 ubuntu-s-4vcpu-8gb-fra1-01 gunicorn[8237]: BrokenPipeError: [Errno 32] Broken pipe

I tried multiprocessing and here is my code
from transformers import pipeline
from functools import lru_cache

import multiprocessing
import codecs

class Model():

    context = "./sw_merge.txt"

    @lru_cache(maxsize=10000)
    def __init__(self):
        print('processing - iniit in model')
        self.model = pipeline('question-answering')
        with codecs.open(self.context, 'rb', errors = 'ignore', encoding='utf-8') as f:
            self.lines = f.read()

    def run_qa(self, qn):
        print('run_qa - on processing')
        ans = self.model(context = self.lines, question = qn)
        return ans

class Conversation():
    #incoming messages - receives an input from the user
    def incoming(self, question):
        usr_qn = []
        usr_qn.append(question)
        return usr_qn

    #model prediction
    def model_ans(self, input_qn):
        y = Model()
        ans = y.run_qa(input_qn)
        ans_text = ans.get("answer")
        print('model_ans - running')
        return ans_text

if __name__ == '__main__':

    p1 = multiprocessing.Process(target=Conversation)
    p2 = multiprocessing.Process(target=Model)

    p1.start()
    p2.start()

    p1.join()
    p2.join()

    # check if processes are alive
    print("Process p1 is alive: {}".format(p1.is_alive()))
    print("Process p2 is alive: {}".format(p2.is_alive()))

    # question = ''
    # chat_conv = Conversation()

    # incoming_text = chat_conv.incoming(question)
    # outgoing_text = chat_conv.model_ans(incoming_text)

My Environment is:

tensorflow==2.3.0
transformers==3.0.2
Python 3.6

Function model_ans takes the question and return the question, so the application execution starts on model_ans the execution ends in class Model() with Pipe Error.


title: Remove the weight_orig in Pytorch after Pruning a model

body:
After a model is pruned in Pytorch, the saved model contains both the pruned weights and weight_orig. This causes the pruned model size to be greater than the unpruned model.
Is there a way to remove the weight_orig and reduce the pruned model size?


title: Image augmentation in Pytorch

body:
I like to augment image alternately.
I have pytorch transform code as follows.
import torchvision.transforms as tt
from torchvision.datasets import ImageFolder
#Data transform (normalization & data augmentation)
stats = ((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010))
train_tfms = tt.Compose([tt.RandomCrop(32, padding = 4, padding_mode = 'reflect'),
                         tt.RandomHorizontalFlip(),
                         tt.RandomAffine(degrees=(10, 30),
                                         translate=(0.1, 0.3),
                                         scale=(0.7, 1.3),
                                         shear=0.1,
                                         resample=Image.BICUBIC)
                         tt.ToTensor(),
                         tt.Normalize(*stats)])

When I create dataset as follow and do training, all images will be augmented.
train_ds = ImageFolder('content/train', train_tfms)

But I want alternately. First image, just train as original image. But the next image is augmented.
How can I do that?


title: How to get indices of top-K values from a numpy array

body:
Let suppose I have probabilities from a Pytorch or Keras predictions and result is with the softmax function
from scipy.special import softmax
probs = softmax(np.random.randn(20,10),1) # 20 instances and 10 class probabilities
probs

I want to find top-5 indices from this numpy array. All I want to do is to run a loop on the results something like:
for index in top_5_indices:
    if index in result:
        print('Found')

I'll get if my results are in top-5 results.
Pytorch has top-k function and I have seen numpy.argpartition but I have no idea how to get this done?


title: Torch distributed broadcast and reduce between CPU/ GPU devices

body:
Using the torch.distributed package. I am trying to move tensors from CPU -> GPU0, GPU1 in two separate processes and update the master version (on CPU).
Assume I have two GPU's connected. One on Device0, the other on Device1.

Store a very large array on CPU (something that can't fit onto a single device/gpu)
X = [1,2,3,4,5,6] for example.

Broadcast part of the array to GPU device 0 and GPU device 1. 0,1 have different chunks of that array.
GPU0 Inds = [0,1] GPU0 data = [1,2]
GPU1 Inds = [2,3] GPU1 data = [2,3]

Run a process on GPU0 and GPU1 independently. For this purpose a simple Add() function will do.

Update the CPU version with GPU data where necessary (for the inds that GPU grabbed). This is where I would probably use reduce to get both tensors from the devices. I would probably store it in a key-value dict where key is the device ID (0 for GPU 0, 1 for GPU 1) and have the inds and data stored in a tuple. Then I need to update the CPU version and run the whole process again.

What I am trying to do can be seen in this diagram.
I plan on using an NCCL backend which apparently supports broadcast and reduce.
My code should look something like this:
Main() function spawns two processes and holds the cpu tensor
Foo() starts the two processes and allows broadcasting, and updating between them (what I want to do in diagram)
def main():
  args.world_size=2
  args.backend='nccl'
  os.environ['MASTER_ADDR'] = '127.0.0.2'
  os.environ['MASTER_PORT'] = '29500'
  args.data = t.Tensor([1,2,3,4,5,6])
  mp.spawn(foo, nprocs=2, args=(args,), join=True)

def foo(rank):
  data = args.data # cpu data
  dist.init_process_group(backend=args.backend, init_method='env://', world_size=args.world_size, rank=rank)
  if rank==0:
    inds=[0,1]
  elif rank == 1:
    inds=[2,3]

  gpu_data = data[inds].cuda(rank) # will send to GPU0 or GPU1. # probably need to use the torch.dist.broadcast operation here but idk how.
  data[inds].data=gpu.data # this would be the update step.

  gpu_data +=1
  dist.destroy_process_group()
  print('data: ', data) >> is not [2,3,4,5,5,6] which is what it should be


title: Memory leak when using lambda in Python class

body:
I have detected a memory leak in Python if I use lambda function inside a class.
Here's the code to reproduce the leak:
import torch
# import numpy as np

class Class1(object):
    def __init__(self, x0):
        self.x0 = x0
        self.obj2 = Class2()
        self._leak_fcn = lambda: self.obj2.fcn()  # source of memory leak!

class Class2(object):
    def fcn(self):
        pass

def fcn(x0):
    obj1 = Class1(x0)
    return x0

def test_fcn():
    shape = (50000000, 3)
    y0 = torch.randn(shape).to(torch.double)
    # y0 = np.random.randn(*shape)
    y = fcn(y0)
    return y

for i in range(1000):
    print(i)
    test_fcn()

The memory leak happens even if I changed it to numpy (without using pytorch).
No memory leak is detected if comment the line containing self._leak_fcn or if I write the _leak_fcn as a method instead of a lambda.
What is happening here?
I am not sure if this leak is from Python, or both PyTorch and NumPy are suffering the same leak.
FYI: I am using Python 3.8.5.
EDIT:
I know there is a memory leak here because if I run it for a long period, my memory is filled up (observed with htop) and the process is killed when it is run out of memory.


How to solve this problem?
  File "train.py", line 98, in validation_epoch_end
    self.scheduler.step(avg_loss)
AttributeError: 'dict' object has no attribute 'step'

My source code:
import os
import ast
import torch
import torchaudio
import torch.nn as nn
from torch.nn import functional as F
import torch.optim as optim
from torch.utils.data import DataLoader
from pytorch_lightning.core.lightning import LightningModule
from pytorch_lightning import Trainer
from argparse import ArgumentParser
from model import SpeechRecognition
from dataset import Data, collate_fn_padd
from pytorch_lightning.callbacks import ModelCheckpoint

nodes = int(1)
gpus = int(1)
data_workers = int(0)
train_file  = 'path/to/your/train.json'
valid_file = 'path/to/your/test.json'
valid_every = int(1000)
save_model_path = 'path/where/you/want/to/save/your/model'
logdir = 'path/where/you/want/to/save/your/logs'
epochs = int(10)
batch_size = int(64)
pct_start = float(0.3)
div_factor = int(100)
dparams_override = str("{}")
hparams_override = str("{}")
load_model_from = None
resume_from_checkpoint = None

class SpeechModule(LightningModule):
    def __init__(self, model, args):
        super(SpeechModule, self).__init__()
        self.model = model
        self.criterion = nn.CTCLoss(blank=28, zero_infinity=True)
        self.args = args

    def forward(self, x, hidden):
        return self.model(x, hidden)

    def configure_optimizers(self):
        self.optimizer = optim.AdamW(self.model.parameters(), self.args.learning_rate)
        lr_scheduler = optim.lr_scheduler.ReduceLROnPlateau(
            self.optimizer, mode='min',
            factor=0.50, patience=6
        )
        self.scheduler = {
            'scheduler': lr_scheduler,
            'reduce_on_plateau': True,
            'monitor': 'val_checkpoint_on'
        }

        return [self.optimizer], [self.scheduler]

    def step(self, batch):
        spectrograms, labels, input_lengths, label_lengths = batch
        bs = spectrograms.shape[0]
        hidden = self.model._init_hidden(bs)
        hn, c0 = hidden[0].to(self.device), hidden[1].to(self.device)
        output, _ = self(spectrograms, (hn, c0))
        output = F.log_softmax(output, dim=2)
        loss = self.criterion(output, labels, input_lengths, label_lengths)
        return loss

    def training_step(self, batch, batch_idx):
        loss = self.step(batch)
        logs = {'loss': loss, 'lr': self.optimizer.param_groups[0]['lr'] }
        return {'loss': loss, 'log': logs}

    def train_dataloader(self):
        d_params = Data.parameters
        d_params.update(dparams_override)
        train_dataset = Data(json_path=self.train_file, **d_params)
        return DataLoader(dataset=train_dataset,
                            batch_size= batch_size,
                            num_workers= data_workers,
                            pin_memory=True,
                            collate_fn=collate_fn_padd)

    def validation_step(self, batch, batch_idx):
        loss = self.step(batch)
        return {'val_loss': loss}

    def validation_epoch_end(self, outputs):
        avg_loss = torch.stack([x['val_loss'] for x in outputs]).mean()
        self.scheduler.step(avg_loss)
        tensorboard_logs = {'val_loss': avg_loss}
        return {'val_loss': avg_loss, 'log': tensorboard_logs}

    def val_dataloader(self):
        d_params = Data.parameters
        d_params.update(dparams_override)
        test_dataset = Data(json_path=valid_file, **d_params, valid=True)
        return DataLoader(dataset=test_dataset,
                            batch_size= batch_size,
                            num_workers= data_workers,
                            collate_fn=collate_fn_padd,
                            pin_memory=True)

def checkpoint_callback():
    return ModelCheckpoint(
        filepath= save_model_path,
        save_top_k=True,
        verbose=True,
        monitor='val_loss',
        mode='min',
        prefix=''
    )

def main(args):
    h_params = SpeechRecognition.hyper_parameters
    h_params.update(hparams_override)
    model = SpeechRecognition(**h_params)
    if load_model_from:
        speech_module = SpeechModule.load_from_checkpoint(load_model_from, model=model)
    else:
        speech_module = SpeechModule(model, args)
    trainer = Trainer()

    trainer = Trainer(
        max_epochs= epochs, gpus= gpus,
        num_nodes= nodes, distributed_backend=None,
        gradient_clip_val=1.0,
        val_check_interval= valid_every,
        checkpoint_callback=checkpoint_callback,
        resume_from_checkpoint= resume_from_checkpoint
    )
    trainer.fit(speech_module)

if __name__ == "__main__":
    parser = ArgumentParser()
    parser.add_argument('--learning_rate', default=1e-3, type=float, help='learning rate')

    args = parser.parse_args()
    hparams_override = ast.literal_eval(hparams_override)
    dparams_override = ast.literal_eval(dparams_override)
    if save_model_path:
        if not os.path.isdir(os.path.dirname(save_model_path)):
            raise Exception("the directory for path {} does not exist".format(save_model_path))

    main(args)