title: Using torch.lstm rises nans unless I initialize it every iteration

body:
I try to use torch.nn.lstm to predict future retail prices. My data cover different stores over time. In each iteration I draw batch_size of 2 stores, then I train my LSTM sequentially over the periods which are covered in the data.
My problem is this - in each iteration I create a month X 2-stores dataset (for the sake of this post), and run it through the model.
Before I start, I initialize it using the init function. In that case, the resulting tensor is full of nans. On the other hand, if I'm using init every time, the results are actual numbers.
The lstm model is
n_features = 68
n_steps = 3
batch_size = 2
seq_len = n_steps*batch_size
n_hidden = 2 # number of hidden states
n_layers = 2 # number of LSTM layers (stacked)

# 2. Build the Model
class SmallLSTM(torch.nn.Module):
    def __init__(self,n_features,seq_len, n_hidden, n_layers, n_steps, batch_size):
        super(SmallLSTM, self).__init__()
        self.n_features = n_features
        self.seq_len = seq_len
        self.n_hidden = n_hidden # number of hidden states
        self.n_layers = n_layers # number of LSTM layers (stacked)
        self.n_steps = n_steps
        self.batch_size = batch_size

        self.l_lstm = torch.nn.LSTM(input_size = self.n_features,
                       hidden_size = self.n_hidden,
                       num_layers = self.n_layers,
                       batch_first = True,
                       dropout = 0.1)
        # according to pytorch docs LSTM output is
        # (batch_size,seq_len, num_directions * hidden_size)
        # when considering batch_first = True
        self.l_linear = torch.nn.Linear(self.n_steps*self.batch_size * self.n_hidden, self.batch_size)

    def init_hidden(self, batch_size):
        # even with batch_first = True this remains same as docs
        hidden_state = torch.zeros(self.n_layers,1,self.n_hidden)
        cell_state = torch.zeros(self.n_layers,1,self.n_hidden)
        self.hidden = (hidden_state, cell_state)

    def forward(self, x):
        #batch_size, seq_len, _ = x.size()

        lstm_out, self.hidden = self.l_lstm(x,self.hidden)
        # lstm_out(with batch_first = True) is
        # (batch_size,seq_len,num_directions * hidden_size)
        # for following linear layer we want to keep batch_size dimension and merge rest
        # .contiguous() -> solves tensor compatibility error
        lstm_out = lstm_out.reshape((1,self.n_steps*self.batch_size * self.n_hidden))
        lstm_out = self.l_linear(lstm_out)

        #self.hidden = [elem.detach_() for elem in self.hidden]

        return lstm_out

Without the per-iteration initialization the code looks like that
batch_size = 2
stores_drawn_idx = 2
Stores_train_batch = Stores_train.iloc[stores_drawn_idx:stores_drawn_idx+batch_size]
Stores_train_batch.reset_index(inplace=True, drop = True)
stores_drawn_idx += batch_size
Months = Xy['Month'].sort_values().unique()
n_steps = 3

mv_net = SmallLSTM(n_features = n_features,\
                    seq_len = seq_len, \
                    n_hidden=n_hidden, \
                    n_layers = n_layers, \
                    n_steps = n_steps, \
                    batch_size = batch_size)
mv_net.init_hidden(1)

train_batch = pd.DataFrame(columns = Xy.columns)
for j in range(Stores_train_batch.shape[0]):
  X_ = Xy_temp_month.drop(['origin_address', 'Retailer_origin', 'Month', 'target'], axis = 1).values.astype('float32')
  y_ = Xy_temp_month[Xy_temp_month['Month'] == months_temp[-1]]['target']

  X_ = torch.from_numpy(X_.reshape(1,n_steps*batch_size,X_.shape[1]))
  X_ = torch.tensor(X_,dtype=torch.float32)
  y_ = torch.tensor(y_.values.astype('float32'), dtype=torch.float32).reshape([len(y_)])

  n_features = 68
  seq_len = n_steps*batch_size
  n_hidden = 2 # number of hidden states
  n_layers = 2 # number of LSTM layers (stacked)

  criterion = torch.nn.MSELoss(reduction='sum')
  optimizer = torch.optim.Adam(mv_net.parameters(), lr=0.05, weight_decay=1500)
  valid_loss_min = 1000

  #mv_net.init_hidden(1)
  output = mv_net(X_)

  loss = criterion(output, y_)
  print(loss.item())

Under this scenario the results are nan for each time the line print(loss.item()) runs.
If I add that line mv_net.init_hidden(1) I get values for the loss.
What should I do?
Thanks!!!


title: colab sessions crashs after "to(device)"

body:
I am using google colab pro.
I am running on GPU mode:
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print(device)
print(torch.cuda.get_device_name())

returns:
cuda
Tesla P100-PCIE-16GB

I am able to create a model and run it perfectly fine, and then save it.
when I try to load it without to(device), it works.
but when I run:
notes_model = LSTM_notes(em_dim).to(device)

the session crashes:
Your session crashed for an unknown reason. View runtime logs

I have tried looking at the logs, but they don't seem to provide meaningful info:
May 20, 2021, 2:37:08 PM    WARNING WARNING:root:kernel 53812acc-3a19-48e2-a41b-1acaab9dfe61 restarted
May 20, 2021, 2:37:08 PM    INFO    KernelRestarter: restarting kernel (1/5), keep random ports
May 20, 2021, 2:36:44 PM    WARNING WARNING:root:kernel 53812acc-3a19-48e2-a41b-1acaab9dfe61 restarted
May 20, 2021, 2:36:44 PM    INFO    KernelRestarter: restarting kernel (1/5), keep random ports

any ideas on why is this happening and what can I do to solve it?
it doesn't, I think, an out-of-ram issue.


title: Convert an Array to PyTorch IValue on Android

body:
Simple question.
I have an array of ints in Android (Kotlin). I need to convert it to an org.pytorch.IValue. I believe that such a conversion can be done, but I can't figure out how to do it. This is my array
val array = Array(34)     { IntArray(1) }
How do I get this wrapped in an IValue?


title: model.eval() should use to measure accuracy on training set pytorch

body:
During training phase, at the epoch 69, I got the following result

train :     Loss: 0.201644404 Acc: 0.887417219(670/755)
validation: Loss: 0.371976876 Acc: 0.880000000(66/75)

For testing phase, I load model at this epoch, and test again on this train and validation set:

train set: Got 735 / 755 with accuracy 97.35
validation: accuracy same as training phase

During training phase:

I call model.eval() before calculating accuracy on validation set, but not for train set

During test phase:

I call model.eval() before calculating accuracy on both train and validation set

The question is

which training accuracy should I rely on? 0.88 during training phase, or 0.97 during test phase?

As Andrew Ng has pointed out, the difference between training accuracy and validation accuracy may imply overfit. If I choose 0.88, then my model is not overfit, but if I choose 0.97, my model may be overfit?


title: AttributeError: 'Sequential' object has no attribute 'send' in Keras

body:
hook = sy.TorchHook(torch)
node1 = sy.VirtualWorker(hook, id="node1")

model = tf.keras.models.Sequential([
tf.keras.layers.Flatten(input_shape=(28,28)),
tf.keras.layers.Dense(128, activation='relu'),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.Dense(10, activation='softmax')])

model.compile(optimizer='adam',
         loss='sparse_categorical_crossentropy',
         metrics=['accuracy'])

I tried like that code. But I had an AttributeError like that image. I don't know what is the error means. Why Sequential object has ho attribute 'send'? What should I do to solve this problem?


title: issue in loading Model using PyTorch in google-collaboratory

body:
I am trying to Load the Model in google_collaboratory to get evaluate it and generate all the statistics results.
My trying
import torch
import torch.nn.functional as F
import torch.optim as optim
import torch.backends.cudnn as cudnn
import torch.backends.cudnn as cudnn
import numpy as np
import torch.nn as nn
import os

def load_checkpoint(filepath):
    checkpoint = torch.load(filepath)
    model = fc_model.Network(checkpoint['input_size'],
                             checkpoint['output_size'],
                             checkpoint['hidden_layers'])
    model.load_state_dict(checkpoint['state_dict'])

    return model

PATH = "/content/gdrive/MyDrive/best.pt"
state_dict = load_checkpoint(PATH)

The Error
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-24-0515f2edfa1a> in <module>()
     18
     19 PATH = "/content/gdrive/MyDrive/best.pt"
---> 20 state_dict = load_checkpoint(PATH)

2 frames
/usr/local/lib/python3.7/dist-packages/torch/serialization.py in _load(zip_file, map_location, pickle_module, pickle_file, **pickle_load_args)
    849     unpickler = pickle_module.Unpickler(data_file, **pickle_load_args)
    850     unpickler.persistent_load = persistent_load
--> 851     result = unpickler.load()
    852
    853     torch._utils._validate_loaded_sparse_tensors()

ModuleNotFoundError: No module named 'models'

---------------------------------------------------------------------------
NOTE: If your import is failing due to a missing package, you can
manually install dependencies using either !pip or !apt.

To view examples of installing some common dependencies, click the
"Open Examples" button below.
---------------------------------------------------------------------------

I tried to install some library but it gives me the same is there anyways to load models inside google collaboratory.


title: Pytorch TabularDataset taking too much time to load csv

body: I’m trying to build a neural machine translation system and trying to load train and Val CSV, but it’s more than an hour and TabularDataset is still loading can you please tell me the solution for it or alternative methods to load the dataset. I have uploaded the image of the code please click on it to see it in better quality, please help with this. Thankyou


title: Huggingface tokenizer always encodes as unknown - Conversion of vocab.txt?

body:
I have used google's implementation of electra to train a model from scratch.
For the pretraining I have followed this tutorial -- with modifications of course, since it uses google's implementation for BERT. After having finished the training & also having converted the electra discriminator checkpoint to huggingface format using this script, I am trying to load the model in order to get the embeddings for some sentences.
First, I create the ElectraConfig, and then I create a tokenizer using the vocab.txt I have created (based on the aforementioned tutorial)
config = ElectraConfig(vocab_size=100000,
                       embedding_size=768,
                       hidden_size=768,
                       num_hidden_layers=12,
                       num_attention_heads=4,
                       intermediate_size=3072,
                       hidden_act="gelu",
                       hidden_dropout_prob=0.1,
                       attention_probs_dropout_prob=0.1,
                       max_position_embeddings=512,
                       position_embedding_type="absolute")

tokenizer = ElectraTokenizer.from_pretrained(path_to_vocab)

My problem is that the tokenizer encodes all tokens in all sentences as unknown.
Do I need to convert the vocab.txt file? Is the creation of the vocab.txt file described in the tutorial incompatible with huggingface?


title: Pytorch Conditioned Dataloader

body:
Let’s say I have a dataloader which returns video samples where each video sample is of the following shape : [number_of_frames X 2048] (number of frames can vary from the following set : [60 , 80 , 100 , 120].

I want in each batch returned from the dataloader to have samples with the same number of frames.
for example if the batchsize=16 . I want batches of the following shape : [16 X 80 X 2048] , [16 X 60 X 2048]…
I have tried to sort the data samples such that we will have continuous samples of the same lengths. but I don't think this is a good solution.
Anyidea how to do it?
Thanks


title: Does pytorch use the average of the training examples in SGD calculation?

body:
I have trouble understanding the result of the gradient in a simple linear model.
Let's consider two points X = [1,2] and y=[1,2]. Let's consider a simple linear model where pred = w*x + b
The loss is equal to 1/2 * ((w * 1 +b - 1)^2 + (w * 2 + b - 2)^2) = 2.5
And the derivatives :

d loss / db = (w * 1 +b - 1) + (w * 2 +b - 2)
d loss / dw = (w * 1 +b - 1) * 1 + (w * 2 +b - 2) * 2

Now let's suppose we initialize the bias and the weight at 0. we have

d loss / db = - 1 -2 = -3
d loss / dw = - 1 * 1 - 2 * 2 = -5

However in pytorch, d loss / dw seems to be equal to -4.5. After quick calculations, i noticed that if i replace *1 and *2 by *(1+2)/2, in dloss/dw, i obtain the same result. Hence my question.
To reproduce the behavior, i have the following code :
import torch
import torch.nn as nn

X = torch.tensor([[1],[2.]])
y = torch.tensor([1,2.])
model = nn.Linear(in_features=1, out_features=1)
criterion = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(),lr = 0.1)
model.weight.data.fill_(0)
model.bias.data.fill_(0)
loss = criterion(model(X), y)
optimizer.zero_grad()
loss.backward()
print("loss", loss.item())
print("bias gradient", model.bias.grad)
print("weight gradient",model.weight.grad)

The output I get is the following :
loss 2.5
bias gradient tensor([-3.])
weight gradient tensor([[-4.5000]])


title: NLP language to launage translation with sentence embedding instead of word embedding

body:
I am following this tutorial, and could replace the embeddings with glove. I am wondering if it would make sense to use sentence embeddings instead of word embeddings in that case. options for sentence embeddings that I found are: this and this.
Any help in this regard would be appreciated.


title: From where can i get dataset of Virtual machine behavior?

body: i want to entrain my convolutional neural network for VMs classification


title: Can TorchServer support dynamic batch_size?

body: When register a model in TorchServer, batch_size need to be set, otherwise batch_size is set as 1, can TorchServer support dynamic batch_size?


title: Building a custom Cifar10 or any image dataset by including their augmented images

body:
I would like to build a custom image dataset. Let’s say my dataset contains 100 images and its labels. I would like to create a new dataset with 2 augmentations for every image and include its labels too.
so, the dataset should contain 100 original images and labels + 100 images of some augmentation and labels + 100 images of some other augmentation and labels.
so, each label is repeated 3 times i.e. one for original and 2 for augmented images.


title: Pytorch print what happens (temporary matrices) within imported Model

body: I am trying to better understand a model that I import (model.py) and train in main.py. The model only returns the loss which I optimize but I'd like to look at different elements within the model. Print statements placed within the model.py file don't seem to work when running main.py. What can I do?


title: Passing ouput from model as input pytorch dataloader

body:
I’m doing video prediction frame prediction using LSTM. I have trained the network using input frame at time (t) and target frame at time (t+1).
Now for testing set I want the LSTM to use the output frame from time (t) as a input frame at time (t+1), is it possible to do this using a dataloader?


title: Using remoter interpreter python

body: I need tu use Ryu controller and Pytorch on remote server. Can I use the python interpreter remotle server and remote interpreter?


title: How to gives range the ratation for RandomRotation in pytorch

body:
I try to give the roate wite random ratate
def mapper(dataset_dict):
    dataset_dict = copy.deepcopy(dataset_dict)  # it will be modified by code below
    image = utils.read_image(dataset_dict["file_name"], format="BGR")
    transform_list = [

                     T.ResizeShortestEdge(short_edge_length=(640, 672, 704, 736, 768, 800), max_size=1333, sample_style='choice')
                     ,T.RandomRotation([-10,0,10])

                      ]
    image, transforms = T.apply_transform_gens(transform_list, image)
    dataset_dict["image"] = torch.as_tensor(image.transpose(2, 0, 1).astype("float32"))

    annos = [
        utils.transform_instance_annotations(obj, transforms, image.shape[:2])
        for obj in dataset_dict.pop("annotations")
        if obj.get("iscrowd", 0) == 0
    ]

    instances = utils.annotations_to_instances(annos, image.shape[:2])
    dataset_dict["instances"] = instances
    #dataset_dict["instances"] = utils.filter_empty_instances(instances)
    return dataset_dict

is this correct way to give the random rotate?
T.RandomRotation([-10,0,10]) will choose 10 or 0 or 10?
 How should I give then a range like
-10 to 10  randomly??


title: How compute loss for character embedding in ConvNet?

body:
I am trying to follow the following paper on CNN Grapheme-to-phoneme model
For example, if the dimensions are [batchsize, channels, max_word_length, max_onehotvector_length]
input: torch.Size([1, 1, 22, 31])
output: torch.Size([1, 1, 22, 72])
predicted: torch.Size([1, 1, 22, 31])

Would the following be a correct way to compute loss? But the model outputs gibberish
for input, output in loader:
    predicted = model(input)
    pred_y, pred_ids = predicted.squeeze().max(dim=1)
    loss = criterion(output, pred_ids)
    loss.backward()

Since there are total of 31 unique symbols in input, how do we translate to 72 symbols in predicted output?


title: PyTorch - LSTM training

body:
I really beginner at this subject and I am trying to run the train here
https://colab.research.google.com/drive/1Rb8OiF-AZ_Y3uFj1O2S0IyocFMhHoTCV?usp=sharing#scrollTo=7Xyt-Gx8XHtQ
that built with the code provided here: https://github.com/pytorch/examples/tree/master/time_sequence_prediction
running my code not provided the future (it is stop on 1000):

but according to the official doc, I should get something like this:

I tried to change the default param future on the forward function inside class Sequence to something different than 0 but gets an error.
What am I missing. How to predict the future?


title: Unexpected GPU space hike when using list to save predictions in pytorch

body:
num_epochs=1
all_x = []
all_y = []
all_y_hat = []
for epoch in tqdm(range(num_epochs)):
        #Evaluation and training on training dataset

        for i,(image,label) in enumerate(train_loader):
            images= image['image'].to(device)
            labels= label.to(device)
            outputs=model(images.float())
            outputs  = torch.softmax(outputs,1)
            all_x.append(outputs)
            all_y.append(labels)

            _,prediction=torch.max(outputs.data,1)

            all_y_hat.append(prediction)

        for i,(image,label) in enumerate(valid_loader):
            images= image['image'].to(device)
#             images= images.to(device)
            labels= label.to(device)

            all_x.append(outputs)
            all_y.append(labels)

            outputs=model(images.float())
            outputs  = torch.softmax(outputs,1)

            _,prediction=torch.max(outputs.data,1)
            all_y_hat.append(prediction)

If i use this code it is throwing error CUDA out of memory. But if i just comment out all_x,all_y... Then it take only 2GB of ram
How can handle this ?


title: Why is my dynamic pytorch model definition not complete?

body:
I tried to create a custom pytorch model class in a way that would allow variable number of hidden layers. Everything seems to "work" in the code, however even if I set 10 hidden layers, none of them show up in the print out of the model definition. I am wondering why this is happening? I can see obviously it has something to do with the way I use a loop to apply the hidden layers.
Code:
class FFModel(nn.Module):
    def __init__(self, input_dim, hidden_dim, layer_dim, output_dim):
        super(FFModel, self).__init__()
        self.input_layer = nn.Linear(input_dim, hidden_dim).to(device)
        self.hidden_layers = [ nn.Linear(hidden_dim, hidden_dim).to(device) for _ in range(layer_dim) ]
        self.activation = nn.LeakyReLU().to(device)
        self.output_layer = nn.Linear(hidden_dim, output_dim).to(device)

    def forward(self, x):
        y_hat = self.activation( self.input_layer(x) )
        for hidden_layer in self.hidden_layers:
            y_hat = self.activation( hidden_layer(y_hat) )
        out = self.output_layer(y_hat)
        return out

model = FFModel(2, 5, 10, 1)
model

Output:
FFModel(
  (input_layer): Linear(in_features=2, out_features=5, bias=True)
  (activation): LeakyReLU(negative_slope=0.01)
  (output_layer): Linear(in_features=5, out_features=1, bias=True)
)

Indeed, if I inspect deeper and parse through the parameters, even though this model should have 10 layers, the parameters are not correct for a model of 10 hidden layers. Clearly this model doesn't have 10 layers as I thought it would.
Why are the hidden layers all gone?
for param in model.parameters():
    print (param)

out:
Parameter containing:
tensor([[-0.1064, -0.0171],
        [-0.5892, -0.5391],
        [-0.6420, -0.5128],
        [ 0.0128,  0.2155],
        [ 0.3406,  0.1825]], device='cuda:0', requires_grad=True)
Parameter containing:
tensor([-0.5432, -0.0650,  0.6447,  0.4615,  0.4772], device='cuda:0',
       requires_grad=True)
Parameter containing:
tensor([[ 0.0615,  0.0901,  0.3399, -0.3341,  0.2029]], device='cuda:0',
       requires_grad=True)
Parameter containing:
tensor([0.1742], device='cuda:0', requires_grad=True)


title: Time and memory complexity in PyTorch

body:
Suppose I have 9 images that have to be processed by a neural network NN. I have two choices.

Sequentially process 9 images by NN one after the another with batch dim 1.
Stack all images along the batch dimension i.e batch size becomes 9.

Certainly, the output for each image remains the same in the two cases but which is better in terms of execution time and memory consumption?
Does the answer vary w.r.t to execution on CPU and GPU?
Here I am just concerned about inference statistics i.e NN will run with torch.no_grad() wrapper and thus training can be ignored.


title: Speed Issues with shuffling in a torch IterableDataset

body:
On Google Colab I built an IterableDataset for loading the approx. 200K pics of the CELEBA dataset, which are located in form of 4 zip files on my GoogleDrive. I process them further in a GAN training loop. The code performs 1.3 iterations/sec on the GPU as long as I don't shuffle the filenames of the zips, but it gets incredibly slow, when I shuffle them shuffle=True (more than 10 sec. for each iteration).
It is not the shuffling itself fnames = [fnames[i] for i in torch.randperm(len(fnames))] , which I tried out seperately and it is equally fast as without shuffling, while I run an empty training loop, it only slows down together with the full GAN training.
class CELEBA_Dataset_multi(IterableDataset):
  def __init__(self, zipnames, PATH="/content/drive/MyDrive/LargeFiles/celeb_dataset/", shuffle=False):
    self.zipnames = [PATH + zipname for zipname in zipnames]
    self.shuffle = shuffle
    self.len = 0
    for zipname in self.zipnames:
      with ZipFile(zipname, 'r') as ZipImg:
        self.len += len(ZipImg.namelist())
  def parse_zipfiles(self, zipnames):
    trf = transforms.Compose(
        [transforms.Resize(64),
         transforms.CenterCrop(64),
         transforms.ToTensor(),
         transforms.Normalize(mean=[0.5, 0.5, 0.5], std=[0.5, 0.5, 0.5])])
    for zipname in self.zipnames:
      with ZipFile(zipname, 'r') as ZipImg:
        fnames = ZipImg.namelist()
        if self.shuffle: fnames = [fnames[i] for i in torch.randperm(len(fnames))]
        for name in fnames:
          img = ZipImg.open(name)
          yield trf(Image.open(img))
  def __iter__(self):
      return self.parse_zipfiles(self.zipnames)
  def __len__(self):
    return self.len

Does anybody have an idea, what is wrong?


title: how can the the tensor matmul shape error be removed?

body:
I am playing with pytorch ZOO AAE. I replaced the MNIST dataset with my own. However, getting an error which says :  mat1 dim 1 must match mat2 dim 0 eventhough I am resizing the image in my dataloader. Here is my code. I am using ImageFolder to read my custom dataset. I made sure I am using opt.shape and opt.batch_size. There is also a resize on the dataloader.
import argparse
import os
import numpy as np
import math
import itertools

import torchvision.transforms as transforms
from torchvision.utils import save_image

from torch.utils.data import DataLoader
from torchvision import datasets
from torch.autograd import Variable

import torch.nn as nn
import torch.nn.functional as F
import torch

os.makedirs("images", exist_ok=True)

parser = argparse.ArgumentParser()
parser.add_argument("--n_epochs", type=int, default=200, help="number of epochs of training")
parser.add_argument("--batch_size", type=int, default=64, help="size of the batches")
parser.add_argument("--lr", type=float, default=0.0002, help="adam: learning rate")
parser.add_argument("--b1", type=float, default=0.5, help="adam: decay of first order momentum of gradient")
parser.add_argument("--b2", type=float, default=0.999, help="adam: decay of first order momentum of gradient")
parser.add_argument("--n_cpu", type=int, default=8, help="number of cpu threads to use during batch generation")
parser.add_argument("--latent_dim", type=int, default=10, help="dimensionality of the latent code")
parser.add_argument("--img_size", type=int, default=32, help="size of each image dimension")
parser.add_argument("--channels", type=int, default=1, help="number of image channels")
parser.add_argument("--sample_interval", type=int, default=400, help="interval between image sampling")
opt = parser.parse_args()
print(opt)

img_shape = (opt.channels, opt.img_size, opt.img_size)

cuda = True if torch.cuda.is_available() else False

def reparameterization(mu, logvar):
    std = torch.exp(logvar / 2)
    sampled_z = Variable(Tensor(np.random.normal(0, 1, (mu.size(0), opt.latent_dim))))
    z = sampled_z * std + mu
    return z

class Encoder(nn.Module):
    def __init__(self):
        super(Encoder, self).__init__()

        self.model = nn.Sequential(
            nn.Linear(int(np.prod(img_shape)), 512),
            nn.LeakyReLU(0.2, inplace=True),
            nn.Linear(512, 512),
            nn.BatchNorm1d(512),
            nn.LeakyReLU(0.2, inplace=True),
        )

        self.mu = nn.Linear(512, opt.latent_dim)
        self.logvar = nn.Linear(512, opt.latent_dim)

    def forward(self, img):
        img_flat = img.view(img.shape[0], -1)
        x = self.model(img_flat)
        mu = self.mu(x)
        logvar = self.logvar(x)
        z = reparameterization(mu, logvar)
        return z

class Decoder(nn.Module):
    def __init__(self):
        super(Decoder, self).__init__()

        self.model = nn.Sequential(
            nn.Linear(opt.latent_dim, 512),
            nn.LeakyReLU(0.2, inplace=True),
            nn.Linear(512, 512),
            nn.BatchNorm1d(512),
            nn.LeakyReLU(0.2, inplace=True),
            nn.Linear(512, int(np.prod(img_shape))),
            nn.Tanh(),
        )

    def forward(self, z):
        img_flat = self.model(z)
        img = img_flat.view(img_flat.shape[0], *img_shape)
        return img

class Discriminator(nn.Module):
    def __init__(self):
        super(Discriminator, self).__init__()

        self.model = nn.Sequential(
            nn.Linear(opt.latent_dim, 512),
            nn.LeakyReLU(0.2, inplace=True),
            nn.Linear(512, 256),
            nn.LeakyReLU(0.2, inplace=True),
            nn.Linear(256, 1),
            nn.Sigmoid(),
        )

    def forward(self, z):
        validity = self.model(z)
        return validity

# Use binary cross-entropy loss
adversarial_loss = torch.nn.BCELoss()
pixelwise_loss = torch.nn.L1Loss()

# Initialize generator and discriminator
encoder = Encoder()
decoder = Decoder()
discriminator = Discriminator()

if cuda:
    encoder.cuda()
    decoder.cuda()
    discriminator.cuda()
    adversarial_loss.cuda()
    pixelwise_loss.cuda()
from torchvision.datasets import ImageFolder

# Configure data loader
os.makedirs("../../data/mnist", exist_ok=True)
import torchvision
import PIL
dataloader = torch.utils.data.DataLoader(

    ImageFolder('./content', transform=transforms.Compose([
    transforms.Resize(opt.img_size),
    transforms.ToTensor(),
    transforms.Normalize([0.5], [0.5])]))
    ,
    batch_size=opt.batch_size,
    shuffle=True,
)

# Optimizers
optimizer_G = torch.optim.Adam(
    itertools.chain(encoder.parameters(), decoder.parameters()), lr=opt.lr, betas=(opt.b1, opt.b2)
)
optimizer_D = torch.optim.Adam(discriminator.parameters(), lr=opt.lr, betas=(opt.b1, opt.b2))

Tensor = torch.cuda.FloatTensor if cuda else torch.FloatTensor

def sample_image(n_row, batches_done):
    """Saves a grid of generated digits"""
    # Sample noise
    z = Variable(Tensor(np.random.normal(0, 1, (n_row ** 2, opt.latent_dim))))
    gen_imgs = decoder(z)
    save_image(gen_imgs.data, "images/%d.png" % batches_done, nrow=n_row, normalize=True)

# ----------
#  Training
# ----------

for epoch in range(opt.n_epochs):
    for i, (imgs, _) in enumerate(dataloader):

        # Adversarial ground truths
        valid = Variable(Tensor(imgs.shape[0], 1).fill_(1.0), requires_grad=False)
        fake = Variable(Tensor(imgs.shape[0], 1).fill_(0.0), requires_grad=False)

        # Configure input
        real_imgs = Variable(imgs.type(Tensor))

        # -----------------
        #  Train Generator
        # -----------------

        optimizer_G.zero_grad()

        encoded_imgs = encoder(real_imgs)
        decoded_imgs = decoder(encoded_imgs)

        # Loss measures generator's ability to fool the discriminator
        g_loss = 0.001 * adversarial_loss(discriminator(encoded_imgs), valid) + 0.999 * pixelwise_loss(
            decoded_imgs, real_imgs
        )

        g_loss.backward()
        optimizer_G.step()

        # ---------------------
        #  Train Discriminator
        # ---------------------

        optimizer_D.zero_grad()

        # Sample noise as discriminator ground truth
        z = Variable(Tensor(np.random.normal(0, 1, (imgs.shape[0], opt.latent_dim))))

        # Measure discriminator's ability to classify real from generated samples
        real_loss = adversarial_loss(discriminator(z), valid)
        fake_loss = adversarial_loss(discriminator(encoded_imgs.detach()), fake)
        d_loss = 0.5 * (real_loss + fake_loss)

        d_loss.backward()
        optimizer_D.step()

        print(
            "[Epoch %d/%d] [Batch %d/%d] [D loss: %f] [G loss: %f]"
            % (epoch, opt.n_epochs, i, len(dataloader), d_loss.item(), g_loss.item())
        )

        batches_done = epoch * len(dataloader) + i
        if batches_done % opt.sample_interval == 0:
            sample_image(n_row=10, batches_done=batches_done)


title: High and volatile loss, constant RMS

body:
I need some insights on high and volatile loss. It is increasing over the course of a few epochs, and then stays in a very volatile and high range.
I have plotted loss, rms and r² score.
Is there an error in my code? Or do I need more training data?
After simplifying the neural net (only 1 hidden layer before going to the latent layer) the loss isn't that large and R² as well as RMS show slight improvements over the first epochs.
import torch
import numpy as np
from torch.utils.data import DataLoader, TensorDataset, IterableDataset
import os
from torch import optim
from model import Network
from matplotlib import pyplot as plt
from sklearn.metrics import r2_score

path = os.getcwd()

inputs = np.load(path + '/data/in1a.npy')
outputs = np.load(path + '/data/out1a.npy')

batch_size = 400

inputs_0 = torch.Tensor(inputs[0,:])
inputs_1 = torch.Tensor(inputs[1,:])

outputs_0 = torch.Tensor(outputs[0,:])
outputs_1 = torch.Tensor(outputs[1,:])
outputs_2 = torch.Tensor(outputs[2,:])

inputs_tensor = torch.Tensor(inputs)

dataset_in = TensorDataset(inputs_0, inputs_1)
dataset_out = TensorDataset(outputs_0, outputs_1, outputs_2)

dataloader_in = DataLoader(dataset_in, batch_size=batch_size, shuffle=False)
dataloader_out = DataLoader(dataset_out, batch_size=batch_size, shuffle = False)

number_batches_in = int(len(dataset_in)/batch_size)
number_batches_out = int(len(dataset_out)/batch_size)

x = torch.empty(size=(number_batches_in, 800))
y = torch.empty(size=(number_batches_out,1200), dtype=torch.float64)

for index, (x1, x2) in enumerate(dataloader_in):
    batch = torch.cat((x1, x2), 0)
    x[index] = batch

x_mean = torch.mean(x)
x_std = torch.std(x)
x_norm = (x - x_mean) / x_std

for index, (y1, y2, y3) in enumerate(dataloader_out):
    batch = torch.cat((y1, y2, y3), 0)
    y[index] = batch

y_mean = torch.mean(y)
y_std = torch.std(y)
y_norm = (y - y_mean) / y_std

model = Network(800,1200,3,800)
SAVE_PATH = "trained/model.dat"
epochs = 20
learning_rate = 0.001
optimizer = optim.Adam(model.parameters(),lr=learning_rate, eps=1e-08)
hist_error = []
hist_loss = []
hist_r2 = []
beta = 0.5

for epoch in range(epochs):
    epoch_error = []
    epoch_loss = []
    epoch_r2 = []
    for x_batch, y_true in zip(x_norm, y_norm):
        optimizer.zero_grad()
        x_batch = torch.unsqueeze(x_batch, 0)
        y_true = torch.unsqueeze(y_true, 0)
        pred = model.forward(x_batch)
        loss = torch.mean(torch.sum((pred - y_true)) ** 2)
        loss.backward()
        optimizer.step()
        error = torch.mean(torch.sqrt((pred - y_true) ** 2)).detach().numpy()
        r2y = torch.squeeze(y_true, 0)
        r2pred = torch.squeeze(pred, 0)
        r2 = r2_score(r2y.detach().numpy(), r2pred.detach().numpy())
        epoch_error.append(error)
        epoch_loss.append(loss.data.detach().numpy())
        epoch_r2.append(r2)
    hist_error.append(np.mean(epoch_error))
    hist_loss.append(np.mean(epoch_loss))
    hist_r2.append(np.mean(epoch_r2))
    print("Epoch %d -- loss %f, RMS error %f, R² score %f " % (epoch+1, hist_loss[-1], hist_error[-1], hist_r2[-1]))
torch.save(model.state_dict(), SAVE_PATH)
print("Model saved to %s" % SAVE_PATH)

f, (ax1, ax2, ax3) = plt.subplots(3, 1, sharex=True)
ax1.plot(hist_error)
ax1.set_ylabel("Amplitude RMSE")
ax2.plot(hist_loss)
ax2.set_ylabel("Loss")
ax3.plot(hist_r2)
ax3.set_ylabel("R²")
ax3.set_xlabel("Epoch")

plt.show()

import torch
import torch.nn as nn
import torch.nn.functional as F

class Network(nn.Module):
    def __init__(self, input_dim, output_dim, latent_dim, layer_dim1):
        """
        Parameter:
        input_dim (int): number of inputs
        output_dim (int): number of outputs
        latent_dim (int): number of latent neurons
        Layer_dim (int): number of neurons in hidden layers
        """
        super(Network, self).__init__()
        self.latent_dim = latent_dim

        self.enc1 = nn.Linear(input_dim, layer_dim1)

        self.latent = nn.Linear(layer_dim1, latent_dim*2)

        self.dec1 = nn.Linear(latent_dim, layer_dim1)

        self.out = nn.Linear(layer_dim1, output_dim)

    def encoder(self, x):
        z = F.elu(self.enc1(x))

        z = self.latent(z)
        self.mu = z[:,0:self.latent_dim]
        self.log_sigma = z[:,self.latent_dim:]
        self.sigma = torch.exp(self.log_sigma)

        eps = torch.randn(x.size(0), self.latent_dim)
        z_sample = self.mu + self.sigma * eps

        self.kl_loss = kl_divergence(self.mu, self.log_sigma, dim=self.latent_dim)

        return z_sample

    def decoder(self, z):
        x = F.elu(self.dec1(z))

        return self.out(x)

    def forward(self, batch):
        self.latent_rep = self.encoder(batch)
        dec_input = self.latent_rep
        return self.decoder(dec_input)

def kl_divergence(means, log_sigma, dim, target_sigma=0.1):
    """
    Computes Kullback–Leibler divergence for arrays of mean and log(sigma)
    """
    target_sigma = torch.Tensor([target_sigma])
    inner = 1 / target_sigma**2 * means**2 + torch.exp(2 * log_sigma) / target_sigma**2 - 2 * log_sigma + 2 * torch.log(target_sigma)
    inner = torch.mean(inner, dim = 0)
    out = 1 / 2. * torch.mean(inner - dim)
    out = out
    return out

The first 20 epochs:
Epoch 1 -- loss 2592.617822, RMS error 0.769075, R² score -0.216357
Epoch 2 -- loss 1020.093633, RMS error 0.767074, R² score -0.159109
Epoch 3 -- loss 1855.906876, RMS error 0.758663, R² score -0.079950
Epoch 4 -- loss 2494.895288, RMS error 0.760361, R² score -0.097916
Epoch 5 -- loss 1447.344017, RMS error 0.759499, R² score -0.094597
Epoch 6 -- loss 1500.075461, RMS error 0.760647, R² score -0.133853
Epoch 7 -- loss 2715.783510, RMS error 0.760349, R² score -0.113949
Epoch 8 -- loss 4040.605946, RMS error 0.760447, R² score -0.097853
Epoch 9 -- loss 2115.793215, RMS error 0.758570, R² score -0.090718
Epoch 10 -- loss 2927.331250, RMS error 0.758347, R² score -0.090624
Epoch 11 -- loss 5020.661278, RMS error 0.761674, R² score -0.122575
Epoch 12 -- loss 5662.833613, RMS error 0.759100, R² score -0.088969
Epoch 13 -- loss 2570.623434, RMS error 0.757806, R² score -0.048262
Epoch 14 -- loss 1901.247553, RMS error 0.758893, R² score -0.064967
Epoch 15 -- loss 13875.499270, RMS error 0.760261, R² score -0.087031
Epoch 16 -- loss 16458.334455, RMS error 0.758352, R² score -0.057363
Epoch 17 -- loss 17253.359936, RMS error 0.760583, R² score -0.080694
Epoch 18 -- loss 17851.229393, RMS error 0.760331, R² score -0.077037
Epoch 19 -- loss 24284.866304, RMS error 0.758882, R² score -0.066919
Epoch 20 -- loss 20997.437961, RMS error 0.760005, R² score -0.084857

After simplifying the neural net (loss is still in a mid 4-digit range):


Have 2 batches of vectors one matrix is 128 X 120 and the other is 100 X 120 how do I computer euclidean distance?
So I have 128 vectors of length 120 and another 100 vectors of length 120.
I want to compute the Euclidean distance between the 128 with each of the 100 vectors in the other tensor. So my return result should be 128 X 100, as in the first 128 vector's euclidean distance for each of the 100 vectors in the other batch where tensor[0,0] corresponds to the Euclidean distance between vector 0 out of 128 and vector 0 out of 100 and tensor[0,-1] corresponds to the euclidean distance between vector 0 out of the 128 and the last vector out of the 100 in the other batch.
I'm sure there's a way to do this without a double for loop but I'm not sure how.


title: Pytorch: When and why am I supposed to use buffer?

body:
I am using buffer to pass on the hidden state of a LSTM network.
def __init__(self, model, hidden_state1=None, ...somethine else...):
    self.register_buffer('hidden_state1', hidden_state1)
    self.hidden_state1 = hidden_state1
    ....#other codes

In order to avoid the error:
RuntimeError: Trying to backward through the graph a second time,
but the buffers have already been freed.
Specify retain_graph=True when calling backward the first time.

I use .clone().detach() to detach the buffers.
As I need to detached them manually anyway, do I still need to use buffer instead of normal parameters in Pytorch?
Would a normal parameter with "requires_grad=False" be sufficient to replace the usage of buffers?
(Actually, I don't know if it is a good way to pass on the hidden states like this)


title: Pytorch: load checkpoint from batch without iterating over dataset again

body:
Instead of loading from an epoch wise checkpoint I need to be able to load from a batch. I am aware that this is not optimal but since I only have limited training time before my training gets interrupted (google colab free version) I need to be able to load from the batch it stopped or around that batch.
I also do not want to iterate over all data again but continue with the data the model has not seen yet.
My current approach which does not work:
def save_checkpoint(state, file=checkpoint_file):
  torch.save(state, file)

def load_checkpoint(checkpoint):
  model.load_state_dict(checkpoint['state_dict'])
  optimizer.load_state_dict(checkpoint['optimizer'])
  train_loss = checkpoint['train_loss']
  val_loss = checkpoint['val_loss']
  epoch = checkpoint['epoch']
  step = checkpoint['step']
  batch = checkpoint['batch']
  return model, optimizer, train_loss, val_loss, epoch, step, batch

While it does load the weights from where it stopped, it iterates over all data again.
Also, do I even need to capture train_loss and val_loss? I cannot see a difference in the loss being output when I include them or not. Thus, I assume it is already included in model.load_state_dict (?)
I assume capturing step and batch won't work this way and I actually need to include some sort of index tracker within my class DataSet ? I do already have this within the DataSet class
   def __getitem__(self, idx):
    question = self.data_qs[idx]
    answer1 = self.data_a1s[idx]
    answer2 = self.data_a2s[idx]
    target = self.targets[idx]

So, could this be useful?


title: How to improve the reading speed of NYUV-2 data set on semantic segmentation tasks, so as to improve GPU utilization?

body:
Environment: a TESLA V100 GPU, Pytorch framework
Task: semantic segmentation
Data set: NYUV-2 data set
Backbone network: deeplab-resnet18
parameter settings:

The batchsize is set to 16
number_worker =8, pin_memory=True in dataloader
The problem is: GPU memory is almost full, but the utilization rate has been rapidly beating between 20 and 100.
Preliminary verification: It should be a problem of data reading and writing. Every time a batch is read, the utilization rate is reduced by 20, and then quickly to 100, then 20...
So how to effectively improve the data set I\O speed?


title: Flexible word count in Pytorch Embedding

body:
The Embedding class in Pytorch takes in num_embedding as a parameter. According to the doc, num_embedding is "size of the dictionary of embeddings". I am curious about the following two cases when creating an embedding object:

The num_embedding, or word count in a database, is unknown before we
create an embedding.
The num_embedding, or word count, is flexible.
For example, initially I create an embedding with num_embedding, or
word count, 1000. Later I have some new elements added in. For
example, I have 10 new words on top of the existing 1000 words, how
to modify the existing embedding (keep the embedding_dim same) to
adapt the change?


title: pytorch use convolution neural network to map from 2D-tensor to 2D-tensor

body:
I am trying to approximate a blackbox function using network. The function take 2D tensor(169x5) and gives 2D tensor(169x45). The two tensor have different scales. I use a simple linear model as following:
import torch
from torch.autograd import Variable
import torch.nn.functional as F
import torch.utils.data as Data
import matplotlib.pyplot as plt
import numpy as np
import imageio
torch.manual_seed(1)
# data
x_in        = torch.rand(169, 5)
x_out       = torch.rand(169, 45)*23
x_in, x_out = Variable(x_in), Variable(x_out)

# linear network
net = torch.nn.Sequential(
        torch.nn.Linear(5, 200),
        torch.nn.ReLU(),
        torch.nn.Linear(200, 100),
        torch.nn.ReLU(),
        torch.nn.Linear(100, 45),)

optimizer = torch.optim.SGD(net.parameters(), lr=0.2)
loss_func = torch.nn.MSELoss()

for t in range(200):
    prediction = net(x_in)
    loss       = loss_func(prediction, x_out)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

I believe a Convolution neural network would be better for such task. But, how can I use CNN to map from 2D-to-2D. How can I construct such CNN?


title: Loading a dataframe of audio files with segmented index into a pytorch tensor

body:
I'm new to pytorch, and I'm trying to prepare an input for an LSTM model of shape (nb_samples, timesteps, features).
I have a dataframe of features where the index of the dataframe is composed of the (file_name, frame) plus 40 log mel features. Here's an example :
Dataframe example
Any idea if there is some built-up function in pytorch that will let me use this information in order to prepare the input in the right shape, where the timesteps will be the number of frames (it is the same for each file already) ?
Thanks in advance :)


title: Can we have inputs that is more than 1D in Pytorch (e.g word-embedding)

body:
Say I have some text and I want to classify them into three groups food, sports, science. If I have a sentence I dont like to each mushrooms we can use wordembedding (say 100 dimensions) to create a 6x100 matrix for this particular sentense.
Ususally when training a neural-network our data is a 2D array with the dimensions n_obs x m_features
If I want to train a neural network on wordembedded sentences(i'm using Pytorch) then our input is 3D n_obs x (m_sentences x k_words)
e.g
#Say word-embedding is 3-dimensions
I = [1,2,3]
dont = [4,5,6]
eat = [7,8,9]
mushrooms = [10,11,12]

"I dont eat mushrooms" = [I,dont,eat,mushrooms] #First observation

Is the best way, when we have N>2 dimensions, to do some kind of pooling e.g mean, or can we use the actual 2D-features as input?


title: Pytorch Add Custom Backward pass for nn.Module Function

body:
I am re-implementing the Invertible Residual Networks architecture.
class iResNetBlock(nn.Module):
    def __init__(self, input_size, hidden_size):
        self.bottleneck = nn.Sequential(
            LinearContraction(input_size, hidden_size),
            LinearContraction(hidden_size, input_size),
            nn.ReLU(),
        )

    def forward(self, x):
        return x + self.bottleneck(x)

    def inverse(self, y):
        x = y.clone()

        while not converged:
            # fixed point iteration
            x = y - self.bottleneck(x)

        return x

I want to add a custom backward pass to the inverse function. Since it is a fixed point iteration, one can make use of the implicit function theorem to avoid unrolling of the loop, and instead compute the gradient by solving a linear system. This is for example done in the Deep Equilibrium Models architecture.
    def inverse(self, y):
        with torch.no_grad():
            x = y.clone()
            while not converged:
                # fixed point iteration
                x = y - self.bottleneck(x)

            return x

    def custom_backward_inverse(self, grad_output):
        pass

How do I register my custom backwards pass for this function? I want that, when I later define some loss such as  r = loss(y, model.inverse(other_model(model(x)))), that r.backwards() correctly uses my custom gradient for the inverse call.
Ideally the solution should be torchscript-compatible.


I use the latest version of imgaug (0.4.0) and Pytorch 1.8.0 GPU, but a warning comes (though my program runs properly). I read some docs but it seems like there is not related doc expressing this problem.
/home/luonanqing/anaconda3/envs/torch1_8/lib/python3.7/site-packages/imgaug/imgaug.py:184: DeprecationWarning: Function `ContrastNormalization()` is deprecated. Use `imgaug.contrast.LinearContrast` instead.
  warn_deprecated(msg, stacklevel=3)


title: Getting gradient for gradCam in pytorch

body:
I am using forward and backward hook in my pytorch densenet121 model.
I set requires_grad to False at the time of training.
for param in model.features.parameters():
    param.requires_grad = False

and I want to get gradient of last conv layer in Neural Network , for this i define hook like this:
model.features.denseblock4.denselayer16.conv2.register_backward_hook(h1)

but this is giving me empty list. But is i set requires_grad = True for the last layer then i get the gradient.
for param in model.features.denseblock4.denselayer16.conv2.parameters():
    param.requires_grad = True

I use gradient for gradCAM .
is this write way to doing or is there  any other way to get gradient or does i need to set requires_grad = True for whole network ?


title: How to combine bidirectional PyTorch LSTM sequentially

body:
The PyTorch LSTM class includes a num_layers argument which stacks sequential LSTMs. I am wondering how to do this external of this argument. As in, I initialize several bidirectional LSTMs with num_layers=1 and want to put data through them sequentially.
Say the input to the LSTM is of shape [10, 16, 64] where 10 is the sequence length, 16 is the batch size, and 64 is the dimension of the input. When this is passed into a bidirectional LSTM, the output becomes [10, 16, 128] which can also be viewed as [10, 16, 2, 64] where the 2 comes from each bidirectional pass of the input.
Since the input to the LSTM is the same whether it is bidirectional or not, (seq_len, batch, input_size), how should I be passing the bidirectional output from the first LSTM to the second? As of now I am just passing the entire (10, 16, 128) output to the next LSTM and letting each directional workflow access the entire 128 vector from both directions of the previous LSTM.


title: After putting 1 batch CUDA error: device-side assert triggered keeps popping up

body:
I have a simple Conv model for EMNIST-letters:
class CNN(torch.nn.Module):
    @property
    def device(self):
        for p in self.parameters():
            return p.device

    def __init__(self):
        super(CNN, self).__init__()

        self.layers = torch.nn.Sequential()
        self.layers.add_module('conv1', torch.nn.Conv2d(1, 1*6, kernel_size = 5))
        self.layers.add_module('relu1', torch.nn.ReLU())
        self.layers.add_module('pool1', torch.nn.MaxPool2d(kernel_size = 2))
        self.layers.add_module('conv2', torch.nn.Conv2d(1*6, 1*16, kernel_size = 5))
        self.layers.add_module('relu2', torch.nn.ReLU())
        self.layers.add_module('pool2', torch.nn.MaxPool2d(kernel_size = 2))
        self.layers.add_module('flatten', torch.nn.Flatten())
        self.layers.add_module('linear1', torch.nn.Linear(16*4*4, 120))
        self.layers.add_module('relu3', torch.nn.ReLU())
        self.layers.add_module('linear2', torch.nn.Linear(120, 84))
        self.layers.add_module('relu4', torch.nn.ReLU())
        self.layers.add_module('linear3', torch.nn.Linear(84, 26))

    def forward(self, input):
        return self.layers(input)

And I have a dataloader:
batch_generator = torch.utils.data.DataLoader(dataset = EMNIST_test,
                                              batch_size=50)

After I put 1 batch from batch_generator, I get this error:
RuntimeError: CUDA error: device-side assert triggered

And it appears if I try to do anything with device. Even if I put my model to device once again I get this error. After such a mistake device is just unavailable. Colab shows there is a lot of free GPU. Did anyone have such a problem?


title: Not getting great results with my GNN on the MNISTSuperPixels Dataset. Can anyone spot any issues?

body:
I have recently gotten started with studying GNNs.  I have some basic background with Machine Learning but most with CNN classification and Handwriting recognition.  I am trying to learn more about GNNs but I have not been able to get good results when trying graph classification on the MNISTSuperPixel Dataset (I am peaking out at about 13% accuracy when there are only 10 possible labels).  My model and training code are below.
class GNN(torch.nn.Module) :
    def __init__(self, dataset):
        super(GNN, self).__init__()
        resized_embedding = 128
        self.gnc1 = torch_geometric.nn.GCNConv(dataset.num_features, resized_embedding)
        self.gnc2 = torch_geometric.nn.GCNConv(resized_embedding, resized_embedding)
        self.gnc3 = torch_geometric.nn.GCNConv(resized_embedding,  resized_embedding)
        self.fc1 = torch.nn.Linear(resized_embedding * 2, resized_embedding)
        self.fc2 = torch.nn.Linear(resized_embedding, resized_embedding // 2)
        self.fc3 = torch.nn.Linear(resized_embedding // 2, dataset.num_classes)
        self.softmax = torch.nn.Softmax()

    def forward(self, batch):
        x, edge_index, batch_index = batch.x, batch.edge_index, batch.batch
        x = self.gnc1(x, edge_index)
        x = torch.nn.functional.relu(x)
        x = torch.nn.functional.dropout(x, training= self.training)
        x = torch.nn.functional.dropout(x, training= self.training)
        x = self.gnc2(x, edge_index)
        x = torch.nn.functional.relu(x)
        x = torch.nn.functional.dropout(x, training= self.training)
        x = self.gnc3(x, edge_index)
        x = torch.nn.functional.relu(x)
        x = torch.nn.functional.dropout(x, training= self.training)
        x = torch.cat([torch_geometric.nn.global_max_pool(x, batch_index), torch_geometric.nn.global_mean_pool(x, batch_index)], dim=1)
        x = self.fc1(x)
        x = torch.nn.functional.dropout(x, training= self.training)
        x = self.fc2(x)
        x = self.fc3(x)
        #x = torch.nn.functional.log_softmax(x, dim=1)
        x = self.softmax(x)
        return x

def train(train_loader, test_loader, model, plot=False):
    optimizer = torch.optim.Adam(model.parameters(), lr=0.01, weight_decay=5e-4)
    loss_fn = torch.nn.CrossEntropyLoss()
    train_accuracies, test_accuracies, train_losses, test_losses = list(), list(), list(), list()
    EPOCHS = 1
    for epoch in range(EPOCHS):
        t0 = time.time()
        epoch_loss, epoch_accuracy = list(), list()

        for batch in train_loader :
            model.train()

            out = model(batch)
            loss = loss_fn(out, batch.y)
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
            epoch_loss.append(loss.item())
            epoch_accuracy.append(accuracy(out, batch.y))

        train_loss = torch.tensor(epoch_loss).mean()
        train_losses.append(train_loss)
        train_acc = torch.tensor(epoch_accuracy).mean()
        train_accuracies.append(train_acc)

        epoch_loss, epoch_accuracy = list(), list()

        for batch in test_loader :
            model.eval()
            out = model(batch)
            loss = loss_fn(out, batch.y)
            epoch_loss.append(loss.item())
            epoch_accuracy.append(accuracy(out, batch.y))

        test_loss = torch.tensor(epoch_loss).mean()
        test_losses.append(test_loss)
        test_acc = torch.tensor(epoch_accuracy).mean()
        test_accuracies.append(test_acc)

Does anyone have any ideas on what I might be doing wrong?


title: PyTorch Bucket Iterator not found

body: BucketIterator is no longer available under torchtext.data in torchtext 0.9. Has it been removed or changed its location?


title: Is there a way to train data on imprecise target ? What kind of loss should be use?

body:
I am currently trying to train my neural network with an imprecise target. Basically my target data is a curve but I want to fit and approach the trend of the curve considering that it is not the ground truth one but an approximation with certain percentage of confidence.
Is there any loss function to take into account the confidence on a curve we have ? What could be a way to let the neural net know that the target is imprecise ?
I have tried to implement the loss in the part 3.1 of this article without any success (link here : https://jcsadeghi.github.io/assets/papers/neunet.pdf )
Let me know if you have any idea


title: FocalLoss TypeError: expected CPU (got CUDA)

body:
I got this error when training my network. From other posts, I know that I'm supposed to put the tensor on the cuda device, either with .cuda() or .to("cuda:0"). However, the same error keeps appearing when I do this.
---> 23         pt = Tensor(logpt.data.exp())
     24
     25         if self.alpha is not None:

TypeError: expected CPU (got CUDA)

I've tried adding .cuda() after line 23, but also after self.alpha = alpha, and a few other places. Am I missing something? Or is it my model code that I should check. Pretty sure the entire model is on cuda. Thanks for any insights.
import torch
from torch import nn, Tensor
import torch.nn.functional as F
class FocalLoss(nn.Module):
    def __init__(self, gamma=0, alpha=None, size_average=True):
        super(FocalLoss, self).__init__()
        self.gamma = gamma
        self.alpha = alpha.cuda()
        if isinstance(alpha,(float,int)): self.alpha = torch.Tensor([alpha,1-alpha])
        if isinstance(alpha,list): self.alpha = torch.Tensor(alpha)
        self.size_average = size_average

    def forward(self, input, target):
        if input.dim()>2:
            input = input.view(input.size(0),input.size(1),-1)  # N,C,H,W => N,C,H*W
            input = input.transpose(1,2)    # N,C,H*W => N,H*W,C
            input = input.contiguous().view(-1,input.size(2))   # N,H*W,C => N*H*W,C
        target = target.view(-1,1)

        logpt = F.log_softmax(input)
        logpt = logpt.gather(1,target)
        logpt = logpt.view(-1)
        pt = Tensor(logpt.data.exp())

        if self.alpha is not None:
            if self.alpha.type()!=input.data.type():
                self.alpha = self.alpha.type_as(input.data)
            at = self.alpha.gather(0,target.data.view(-1))
            logpt = logpt * Tensor(at)

        loss = -1 * (1-pt)**self.gamma * logpt
        if self.size_average: return loss.mean()
        else: return loss.sum()


title: VS Code remote debegger for python can not start

body:
enter image description here
As is shown above, please help me...


title: How to optimize the custom bilinear sampling alternative to grid_sample for TensorRT inference?

body:
I am trying to covert the model with torch.nn.functional.grid_sample from Pytorch (1.6) to TensorRT (7) through ONNX (opset 11). Opset 11 does not support grid_sample conversion.
Custom alternative I found (https://github.com/pytorch/pytorch/issues/27212) is extremely slow while running in Pytorch and have the problem with converting the main loop to TRT.
My own implementation of bilinear sampling (not just grid_sample, but the whole original sampling, based on grid_sample) performs much faster in Pytorch and is converted to TRT successfully.
But my custom bilinear sampling in TRT is slower, than the one in Pytorch (5.6 ms vs 2.0 ms). It turns out, that Pytorch image[:, ind, y0, x0] indexing produce Gather layer with running time about 0.97 ms. And there are 4 such layers in the TRT version of such bilinear sampling.
So the questions are:

How should I optimize my Pytorch code to get the effective TRT model?
What should I do to make Gather layer perform faster?
Can the creation of this function as a custom TRT plugin help with making it faster?

Here is the code of bilinear sampling function:
def bilinear_sample_noloop(image, grid):
    """
    :param image: sampling source of shape [N, C, H, W]
    :param grid: integer sampling pixel coordinates of shape [N, grid_H, grid_W, 2]
    :return: sampling result of shape [N, C, grid_H, grid_W]
    """
    Nt, C, H, W = image.shape
    grid_H = grid.shape[1]
    grid_W = grid.shape[2]
    xgrid, ygrid = grid.split([1, 1], dim=-1)
    mask = ((xgrid >= 0) & (ygrid >= 0) & (xgrid < W - 1) & (ygrid < H - 1)).float()
    x0 = torch.floor(xgrid)
    x1 = x0 + 1
    y0 = torch.floor(ygrid)
    y1 = y0 + 1
    wa = ((x1 - xgrid) * (y1 - ygrid)).permute(3, 0, 1, 2)
    wb = ((x1 - xgrid) * (ygrid - y0)).permute(3, 0, 1, 2)
    wc = ((xgrid - x0) * (y1 - ygrid)).permute(3, 0, 1, 2)
    wd = ((xgrid - x0) * (ygrid - y0)).permute(3, 0, 1, 2)
    x0 = (x0 * mask).view(Nt, grid_H, grid_W).long()
    y0 = (y0 * mask).view(Nt, grid_H, grid_W).long()
    x1 = (x1 * mask).view(Nt, grid_H, grid_W).long()
    y1 = (y1 * mask).view(Nt, grid_H, grid_W).long()
    ind = torch.arange(Nt, device=image.device) #torch.linspace(0, Nt - 1, Nt, device=image.device)
    ind = ind.view(Nt, 1).expand(-1, grid_H).view(Nt, grid_H, 1).expand(-1, -1, grid_W).long()
    image = image.permute(1, 0, 2, 3)
    output_tensor = (image[:, ind, y0, x0] * wa + image[:, ind, y1, x0] * wb + image[:, ind, y0, x1] * wc + \
                 image[:, ind, y1, x1] * wd).permute(1, 0, 2, 3)
    output_tensor *= mask.permute(0, 3, 1, 2).expand(-1, C, -1, -1)
    image = image.permute(1, 0, 2, 3)
    return output_tensor, mask

Time profiling parameters:

Time profiling experiments were performed on laptop Dell G3 15 (Core i7 8750H 2.2 GHz x12, 16 Gb RAM (2666MHz), NVidia GeForce GTX 1050 Ti).
Pytorch environment for profiling: Python 3.7 Anaconda 3 environment, Pytorch 1.6. Pytorch time profiling is performed via time.time() with torch.synchronize() before each time stamp.
TRT environment for profiling: Docker container http://nvcr.io/nvidia/tensorrt:20.06-py3. Profiling was performed with trtexec, and also with custom C++ and Python code. All three results are close.

A part of TRT model profiling with trtexec:
     Layer   Time (ms)   Avg. Time (ms)   Time %
...
   Mul_146        5.82             0.03      0.5
   Add_147        8.50             0.04      0.7
Gather_148      214.39             0.97     17.3
Gather_174      214.25             0.97     17.3
Gather_201      213.88             0.97     17.3
Gather_228      214.48             0.97     17.3
 Add_237))       25.01             0.11      2.0
   Mul_251        7.84             0.04      0.6
     Total     1238.40             5.60    100.0

Additionally I tried viewing the image as the linear array over all dimensions except C and creating the linear indexes to adress elements in the form image[:, p0]. And for this case Gather becomes even slower (about 1.07 ms). Then I considered C=1 (as it always is in the original model) and address the tensor elements as image[p0]. This time Gather takes about 0.92 ms (still too slow).


title: Self-attention model trained with active learning stops learning after a few iterations

body:
I'm doing some active learning with uncertainty sampling on a self-attention model implemented in PyTorch. The algorithm works as follows (steps 3-7 are repeated for 14 iterations):
1. Take 10% of the data as training set, L
2. Train the model on L
3. Either rank the remaining samples U by a certain informativeness measure and pick a batch B of the top n samples, or randomly pick a batch B
4. Add B to L
5. Remove B from U
6. Re-train the model on L union B, after resetting the weights of the model (this is for cumulative training)
7. Evaluate the model on the test set (accuracy)

Now, the model performs well in the first 2-3 iterations and shows to learn on the validation set. However, in later iterations it completely stops learning and shows the same training and validation accuracy and loss. The test accuracy then always drops to the one of a random classifier (around 33%), as you can see here:

The test accuracy should increase instead, as the model is trained on a larger dataset. I really cannot understand why this happens. The code used in each iteration is exactly the same, and I always reset the weights before re-training. This is the code I use to reset the weights:
def reset_weights(m):
    for layer in m.children():
        if hasattr(layer, 'reset_parameters'):
            layer.reset_parameters()

And then:
model.apply(reset_weights)

Can it have something to do with how PyTorch handles the cache? The same procedure works perfectly fine on models implemented with Keras.


title: How to change/reset individual neurons during training

body:
I would like to test what the impact is of randomly resetting certain neurons during training, in a Pytorch-based model with a number of linear layers like this:
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.linear1 = nn.Linear(D, M)
        self.linear2 = nn.Linear(M, M)
        self.linear3 = nn.Linear(M, M)
        self.linear4 = nn.Linear(M, K)

    def forward(self, x):
        x = F.relu(self.linear1(x))
        x = F.relu(self.linear2(x))
        x = F.relu(self.linear3(x))
        x = self.linear4(x)
        x = F.log_softmax(x, dim=1)

        return x

Ideally, I would like to implement a function/code snippet that resets the weights of 1 or more random neurons in each layer at the start of each epoch.
I tried implementing this via calls to parameters or state_dict, but so far I only get permission and anomaly errors when trying this. I'm also not sure whether this should be implemented in the module itself or in the training part.
Any suggestions would be highly welcome!


I am using a modified Resnet18, with my own pooling function at the end of the Resnet.
Here is my code:
resnet = resnet18().cuda() #a modified resnet

class Model():
    def __init__(self, model, pool):
        self.model = model
        self.pool= pool #my own pool class which has trainable layers

    def forward(self, sample):
        output = self.model(sample)
        output = self.pool(output)
        output = F.normalize(output, p=2, dim=1)
        return output

Now, obviously I need to train not only the resnet part, but also the pool part.
But, when I check:
model = Model(model=resnet, pool= pool)
print(list(model.parameters()))

It gives:
AttributeError: 'Model' object has no attribute 'parameters'

Can anyone help?


title: return torch.ops.torchvision.nms(boxes, scores, iou_threshold) RuntimeError: Could not run 'torchvision::nms' with arguments from the 'CUDA' backend

body:
I'm beginner of Deep Learning and trying to use efficientDet model with ".pth" format file to predict image based on Pytorch.
When the model inferred the image, It got the error as below.
RuntimeError: Could not run 'torchvision::nms' with arguments from the 'CUDA' backend. 'torchvision::nms' is only available for these backends: [CPU, BackendSelect, Named, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, Tracer, Autocast, Batched, VmapMode].

How can I fix this error? Please help me!
thanks


title: Does PyTorch compiler allow vectorized mad() function?

body: Some compilers use mad() multiply-accumulate function for higher performance. Does PyTorch compiler allow using a vectorized version of the mad() function?


title: How can i see the forward function of a model which is set up by loading a .pth file?

body:
I a have a .pth file, and I can simply build a network using model = torch.load("path to .pth")
But I don't know how it forwards. Is there any way to check the forward function of the model? Thx.


title: Unable to run super-AND repository in JupyterLab

body:
I am trying to run this repository in JupyterLab. I am getting below error, while running this command python3 main.py --dataset cifar10 --network resnet18. For installation setup i just followed the steps mentioned in the github link.

Log:

Downloading https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz to ./data/cifar-10-python.tar.gz

100.0%Extracting ./data/cifar-10-python.tar.gz to ./data

Files already downloaded and verified

Traceback (most recent call last):

  File "main.py", line 237, in <module>

    main()

  File "main.py", line 215, in main

    train(r, epoch, net, trainloader, optimizer, npc, criterion, criterion2, ANs_discovery, args.device)

  File "main.py", line 124, in train

    features = net(inputs) # (256, 128)

  File "/srv/conda/envs/notebook/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__

    result = self.forward(*input, **kwargs)

  File "/home/jovyan/super-AND/models/resnet_cifar.py", line 111, in forward

    out = self.layer2(out)

  File "/srv/conda/envs/notebook/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__

    result = self.forward(*input, **kwargs)

  File "/srv/conda/envs/notebook/lib/python3.7/site-packages/torch/nn/modules/container.py", line 100, in forward

    input = module(input)

  File "/srv/conda/envs/notebook/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__

    result = self.forward(*input, **kwargs)

  File "/home/jovyan/super-AND/models/resnet_cifar.py", line 26, in forward

    out = F.relu(self.bn1(self.conv1(x)))

  File "/srv/conda/envs/notebook/lib/python3.7/site-packages/torch/nn/functional.py", line 914, in relu

    result = torch.relu(input)

  File "/srv/conda/envs/notebook/lib/python3.7/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler

    _error_if_any_worker_fails()


title: Validation scores are always off, is there any way around overfitting on pretrained network?

body:
I am trying to train Medical Images, which composes of MRI Images, Volume Data, I have 30 Volumes, each of 50 Images (Slices).

I am using Resnet3D, with batchsize=1 (inputting one volume at a time), I know that dataset isn't much at all, but it's balanced, I am splitting 80:20 Train:Validation Set.

I am doing basic augmentation (Random Affine, RandomElasticDeformation and flip)

I am just not getting the good results, my validation scores are always off, it looks like it's overfitting. I am using Pretrained Resnet3D, is there anyway to modify pretrained networks ?

Can someone please recommend what should I be doing so as to improve validation scores ?

Training/Val Trends


title: How is a "mapped tensor" computed in this context?

body:
I'm reading through the PyTorch Geometric docs here: https://pytorch-geometric.readthedocs.io/en/latest/notes/create_gnn.html

They explain a "basic" code:

import torch
from torch_geometric.nn import MessagePassing
from torch_geometric.utils import add_self_loops, degree

class GCNConv(MessagePassing):
def __init__(self, in_channels, out_channels):
    super(GCNConv, self).__init__(aggr='add')  # "Add" aggregation.
    self.lin = torch.nn.Linear(in_channels, out_channels)

def forward(self, x, edge_index):
    # x has shape [N, in_channels]
    # edge_index has shape [2, E]

    # Step 1: Add self-loops to the adjacency matrix.
    edge_index, _ = add_self_loops(edge_index, num_nodes=x.size(0))

    # Step 2: Linearly transform node feature matrix.
    x = self.lin(x)

    # Step 3: Compute normalization
    row, col = edge_index
    deg = degree(row, x.size(0), dtype=x.dtype)
    deg_inv_sqrt = deg.pow(-0.5)
    norm = deg_inv_sqrt[row] * deg_inv_sqrt[col]

    # Step 4-6: Start propagating messages.
    return self.propagate(edge_index, size=(x.size(0), x.size(0)), x=x,
                          norm=norm)

def message(self, x_j, norm):
    # x_j has shape [E, out_channels]

    # Step 4: Normalize node features.
    return norm.view(-1, 1) * x_j

def update(self, aggr_out):
    # aggr_out has shape [N, out_channels]

    # Step 6: Return new node embeddings.
    return aggr_out

The explanation contains this paragraph:

"In the message() function, we need to normalize the neighboring node features x_j. Here, x_j denotes a mapped tensor, which contains the neighboring node features of each edge. Node features can be automatically mapped by appending _i or _j to the variable name. In fact, any tensor can be mapped this way, as long as they have  entries in its first dimension."

Questions: I know this mapped tensor x_j is a tensor that contains the neighbouring node features of each edge. How did they compute x_j? Does this mean that if I change the name of x to x_i I will automatically have a 'mapped tensor'?


I created a new quantum technique using pytorch!
So I'm trying to see how well this technique works on my server computer, Raspberry Pie B3+ and the Jetson TX2.
But there's a problem.
I don't know how to change 32bit operation to 4bit operation.
Now the value itself is quantified through clipping, but it still operates at 32bit in GPU or CPU.
I've been looking for ways to use OpenCL or CUDA, but I haven't found a clue.
I really don't know where to approach this to solve.
How can I change a 32-bit operating system from CPU and GPU to a 4-bit or 8-bit operating system?
I'd really appreciate it if you let me know.


title: Pytorch: Deglow image

body: I want a code which will Deglow night image. Can anyone provide me with the function, please? Code will take an input night image and reduce the glow of street light and other night light sources.


title: Pytorch's inexact float number when numpy is converted to tensor

body:
when I am working with highly sensitive float numbers, I found that Pytorch can not provide exact converting results.
for Example.
A = 0.12312512312312
Input : torch.tensor([A],dtype=torch.float64)
Output: tensor([0.1231], dtype=torch.float64)
How can I fix it?
This is not a problem of dtype of numpy. I already tested with np.float32, np.float64.
Also, I am using 64bit Ubuntu 20.04 machine.
My python version is 3.6 and it is also based on 64bit.
and all functions in numpy and pytorch for changing dtype (i.e., as_type(), dtype=~~).
This is a really serious problem when a dataset consists of this kinda highly sensitive float numbers.


title: Training Vanilla LSTM to learn Levelset images using pytorch

body: I generated a sequence of images using a active methods called Levelset. I have a vanilla LSTM, and I want to train it to learn the Leveltset function from the images. How should I set up the training and evaluating pipeline? Here is the examples of the Levelset (LS) images.


title: Implmenting pg gan generator in pytorch

body:
I want to create a generator pg gan architcure using pytorch. I have found the following architecture and I have been trying to implement the same.pg gan architcture
Here is my code so far.
<class Generator(nn.Module):
def init(self, ngpu):
super(Generator, self).init()
self.ngpu = ngpu
self.main = nn.Sequential(
nn.Conv2d(nz, ngf * 16, 1, 1, 0, bias=False),
nn.ReLU(True),
nn.Conv2d(ngf * 16, ngf * 16, 1, 1, 0, bias=False),
nn.ReLU(True),
        nn.Upsample(scale_factor=8),
        nn.Conv2d(ngf * 16, ngf * 8, 2, 1, 0, bias=False),
        nn.ReLU(True),
        nn.Conv2d(ngf * 8, ngf * 8, 2, 1, 0, bias=False),
        nn.ReLU(True),

        nn.Upsample(scale_factor=8),
        nn.Conv2d(ngf * 8, ngf * 4, 4, 2, 0, bias=False),
        nn.ReLU(True),
        nn.Conv2d(ngf * 4, ngf * 4, 4, 2, 0, bias=False),
        nn.ReLU(True),

        nn.Upsample(scale_factor=4),
        nn.Conv2d(ngf * 4, ngf * 2, 4, 1, 0, bias=False),
        nn.ReLU(True),
        nn.Conv2d(ngf * 2, ngf * 2, 4, 1, 0, bias=False),
        nn.ReLU(True),

        nn.Upsample(scale_factor=2),
        nn.Conv2d(ngf * 2, ngf * 1, 4, 1, 0, bias=False),
        nn.ReLU(True),
        nn.Conv2d(ngf * 1, ngf * 1, 4, 1, 0, bias=False),
        nn.ReLU(True),

        nn.Upsample(scale_factor=2),
        nn.Conv2d(ngf , 32, 2, bias=False),
        nn.ReLU(True),
        nn.Conv2d(32, 32, 4,1,0, bias=False),
        nn.ReLU(True),

        nn.Conv2d(32, 3, 1,  bias=False),
        nn.ReLU(True),

        nn.Tanh()
    )

def forward(self, input):
  return self.main(input)>

I am pretty sure that I am getting the first part of the code wrong(the one highlighted as red in the image.)
Can someone guide me into what I am doing wrong?
The image I want to output has shape(128,128,3)
Thank you.


title: Columns 2D tenson times rows 2D tensor equals a 3d pytorch tensor

body:
Given 2 tensors 2-D in PyTorch A (a X m) and B (m X b), is there any efficient way to obtain a tensor C (m X a X b), where C[i,:,:] = A[:,i] @ B[i,:]?
Here I will give an example of the problem:
A = torch.FloatTensor([[1,2],[3,4]])
B = torch.FloatTensor([[1,2,3],[4,5,6]])

Result:
C = torch.FloatTensor([[[1,2,3],[3,6,9]],[[12,15,18],[16,20,24]]])

I have done it using a for-loop. However, it is very inefficient.


title: Differentiable rotating calipers process PyTorch

body:
I am trying to use the rotating calipers method as well as the ConvexHull2D method in PyTorch but I can't find an implementation that does it in pytorch. This is for finding the minimum enclosing bounding box of two non axis aligned bounding boxes. I was using ConvexHull and was using this RotatingCalipers implementation but its not in Torch so I'm pretty sure that's why I can't backprop through it!
What I am looking for is something like this
A = bounding box 1 as (x,y,width,height,angle)
B = bounding box 2 as (x,y,width,height,angle)
hull = ConvexHull2D(A,B) # this part is from scipy link above
min_rectangle = RotatingCalipers(hull) # this is what I need!
loss = LossFunction(min_rectangle)
loss.backwards() # and poof!

Also wondering about what the backwards pass would look like. Thanks for any help on this!


title: Align feature map with ego motion (problem of zooming ratio )

body:
I want to align the feature map using ego motion, as mentioned in the paper An LSTM Approach to Temporal 3D Object Detection in LiDAR Point Clouds
I use VoxelNet as backbone, which will shrink the image for 8 times. The size of my voxel is 0.1m x 0.1m x 0.2m(height)
So given an input bird-eye-view image size of 1408 x 1024,
the extracted feature map size would be 176 x 128, shrunk for 8 times.

The ego translation of the car between the "images"(point clouds actually) is 1 meter in both x and y direction. Am I right to adjust the feature map for 1.25 pixels?
1m/0.1m = 10  # meter to pixel
10/8 = 1.25   # shrink ratio of the network

However, though experiments, I found the feature maps align better if I adjust the feature map with only 1/32 pixel for the 1 meter translation in real world.

Ps. I am using the function torch.nn.functional.affine_grid to perform the translation, which takes a 2x3 affine matrix as input.


title: getting Error preprocess.py [options]    python

body:
I have some problem
can you tell me how to resolve this error usage: preprocess.py [options]  <in_dir> <out_dir>
I am working on this https://colab.research.google.com/drive/17TMYl3anIwegk-sBU6jHngpqo_3sjtW8#scrollTo=f_xjMkeizPYI
deepvoice3_pytorch https://github.com/r9y9/deepvoice3_pytorch
I download my dataset from https://keithito.com/LJ-Speech-Dataset/
there are 3 things in dataset But when I unzip my dataset

WAVs folder of audio
redme file
metadat.csv

There is no dataset.json file there is a CSV file is both work same or how?
So how I can run this
!mv -v ./datasets/dataset.json ./datasets/audio/dataset.json

getting Error : No such file or directory
When I run this in colab
#!rm -rf preprocessed
!mkdir preprocessed
!python preprocess.py json_meta \
                      "./datasets/audio/dataset.json" \
                      "./preprocessed" \
                      --preset=presets/20180505_deepvoice3_ljspeech.json \
                      --hparams="sample_rate=48000"

there is an Error usage: preprocess.py [options]  <in_dir> <out_dir>


title: Why does the grid_sample function limit the input values to [-1, 1]?

body: The pytorch doc says that "grid specifies the sampling pixel locations normalized by the input spatial dimensions. Therefore, it should have most values in the range of [-1, 1]". But why do they let grid_sample function specify the normalized pixel locations?


title: Convergence issue with SR-GAN, D(x) = D(G(z)) = 0

body:
Good morning,
I'm currently trying to implement for the first time a GAN : a SR-GAN on "very sparse images" (images from a radar). I took as a basis from my code this repo github : https://github.com/leftthomas/SRGAN. My images are actually on one channel, so to be able to use the vgg loss i duplicated each image on 2 channels and added a mask channel. As my generator seemed to struggle i added a criterion on the loss of the discriminator (if it goes under 0.5 stop training it) and tweeked the learning rate and the batch size. A weird failure mod happened after 30 epochs: D(x) and D(G(z)) go to zero, i havent find any documentation on a such failure mod and i dont really know how to deal with it, before those 30 epochs the training seem to go well and even if i let 20 more epochs for the discriminator to catch up, it doesnt. Do you have an idea of what is actually happening ? Lowering the learning rate of the discriminator maybe ? I'm sorry if the question has already been asked, didnt find anything on it
Thank you very much for your help !


title: GCP AI platform unified: Cannot find bucket when using parser/ command line arguments

body:
I am having this weird bug that the submitted custom job fails due to not finding the bucket I defined for the training output although I can see it exists. This is the error I am getting:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/opt/conda/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/root/.local/lib/python3.7/site-packages/trainer/task.py", line 508, in <module>
    train_loss = train(scheduler, optimizer)
  File "/root/.local/lib/python3.7/site-packages/trainer/task.py", line 390, in train
    torch.save(model.state_dict(), args.model_dir)
  File "/opt/conda/lib/python3.7/site-packages/torch/serialization.py", line 369, in save
    with _open_file_like(f, 'wb') as opened_file:
  File "/opt/conda/lib/python3.7/site-packages/torch/serialization.py", line 230, in _open_file_like
    return _open_file(name_or_buffer, mode)
  File "/opt/conda/lib/python3.7/site-packages/torch/serialization.py", line 211, in __init__
    super(_open_file, self).__init__(open(name, mode))
FileNotFoundError: [Errno 2] No such file or directory: 'gs://machine-learning-us-central1/test_04_30_15_12'

The bucket and directory in the error gs://machine-learning-us-central1/test_04_30_15_12 do exist. They are also being created before the actual training.
It works when I do not use command line arguments but python code. Thus, I assume I have a bug within the parser which I used for the command line arguments.
Parser:
parser = argparse.ArgumentParser()
parser.add_argument('--model-dir', dest='model_dir',
                    help='Model dir.')
parser.add_argument('--model-name', dest='model_name',
                help='Name of the model',
                default='model.pt')
args = parser.parse_args()

How I store the training output:
def save_model(args_dir, args_name):
    """Saves the model to Google Cloud Storage

    Args:
      args: contains name for saved model.
    """
    bucket = storage.Client().bucket(ROOT_BUCKET)
    blob = bucket.blob(args_name)
    blob.upload_from_filename(args_dir)

#within the training run
torch.save(model.state_dict(), args.model_name)
save_model(args.model_name, args.model_dir)

Model dir seems to be correctly defined as I can store files in it, just not with the parser.
Or could it be that there is an issue with pytorch and how I save the model?


I'm using pytorch in google colab. I have python 3.7 and allennlp 2.4.0 and allen-models.
when I run my code, I get this error:
Cannot register srl_bert as Model; name already in use for SrlBert
what do I do?


title: Why do I got error for size of tensor when I want to calculate the likelihood instead of MSE in VAE loss function?

body:
I want to calculate likelihood instead of MSE for loss in VAE
but got this error for size of tensor, the size of decoder logvar(logvar_x) is different from others but I don't know how can fix it

   # L2 = torch.mean((recon_x-x)**2)
    # neg log likelihood of x under normal
  #  loss_rec =  np.log(2.0 * np.pi) + logvar_x + (x - mu_x)**2 / (2*torch.exp(logvar_x))
    loss_rec = -torch.sum((-0.5 * np.log(2.0 * np.pi))+ (-0.5 * logvar_x)+ ((-0.5 / torch.exp(logvar_x)) * (0 - mu_x) ** 2.0))

    # see Appendix B from VAE paper:
    # Kingma and Welling. Auto-Encoding Variational Bayes. ICLR, 2014
    # https://arxiv.org/abs/1312.6114
    # 0.5 * sum(1 + log(sigma^2) - mu^2 - sigma^2)
    KLD = -0.5 * torch.mean(1 + logvar - mu.pow(2) - logvar.exp())
    return loss_rec + KLD

RuntimeError                              Traceback (most recent call last)
<ipython-input-43-87c6826d76cb> in <module>
      2 if __name__ == "__main__":
      3     for epoch in range(1, args.epochs + 1):
----> 4         train(epoch)
      5
      6     gt = next(train_loader)

<ipython-input-42-e7ed67103562> in train(epoch)
      9         optimizer.zero_grad()
     10         recon_batch, mu, logvar, mu_x, logvar_x = model(data)
---> 11         loss = loss_function(recon_batch, data, mu, logvar, mu_x, logvar_x)
     12
     13         loss.backward()

<ipython-input-41-9419478caa6a> in loss_function(reco`enter code here`n_x, x, mu, logvar, mu_x, logvar_x)
      5     # neg log likelihood of x under normal
      6   #  loss_rec =  np.log(2.0 * np.pi) + logvar_x + (x - mu_x)**2 / (2*torch.exp(logvar_x))
----> 7     loss_rec = -torch.sum((-0.5 * np.log(2.0 * np.pi))+ (-0.5 * logvar_x)+ ((-0.5 / torch.exp(logvar_x)) * (0 - mu_x) ** 2.0))
      8
      9     # see Appendix B from VAE paper:

RuntimeError: The size of tensor a (2) must match the size of tensor b (100) at non-singleton dimension 1```


title: using nn.Sparselinear in pytorch

body:
Hi I am implementing sparsely connected neural network using the following code
'This is the part of the code that has the problem'
from torchvision import datasets, transforms
import torch
import torchvision
import torch.nn as nn
import torchvision.transforms as transforms
import torchvision.datasets as dsets
import sys
import os

class FeedforwardNeuralNetModel(nn.Module):
    def __init__(self, input_dim, hidden_dim, output_dim):
        super().__init__()
        # Linear function
        self.fc1 = nn.SparseLinear(input_dim, hidden_dim)

        # Non-linearity
        self.sigmoid = nn.Sigmoid()

        # Linear function (readout)
        self.fc2 = nn.SparseLinear(hidden_dim, output_dim)

    def forward(self, x):

        out = self.fc1(x)
        out = self.sigmoid(out)
        out = self.fc2(out)

        return out

But I am getting this error
AttributeError: module 'torch.nn' has no attribute 'SparseLinear'


title: in_channels and kernel_size for wordembedding using Conv1d (Pytorch)

body:
I'm really strugling about the input to Conv1D - I have read several guides/explanations but something still does not quite add up ...
I have a 1-D signal, a flatten sentence of word-embedding e.g

X = ["its a sunny day", "FC Barcelona had 3 red cards the previous game"]

X = [flatten_and_pad(x) for x in X] #Flatten adn zero-pad

X= [
[[1,2,3],[0,1,0],[4,5,6],[7,8,9],[0,0,0],[0,0,0]],  # "its a sunny day" with 3-dim word-embedding
[[-1,2,3],[0,2,0], ....., ,[11,12,13]]  # "FC Barcelona had 3 red cards the previous game"  3-dim word-embedding
]

i.e my size of X is (n_samples, n_words, embed_dim) with n_samples being the number of sentences, n_words the amount of word in each sentece and embed_dim the dimension of word-embedding.
Say I want to have a conv-kernel that strides over every word (or every bi/tri-gram) -
I assume that in_channels is the number of words (i.e n_words) and since I want to stride over each word, I have set kernel_size=embded_dim and stride=1 i.e
conv1 = nn.Conv1d(in_channels = n_words,
out_channels = 10,
kernel_size = embed_dim,
stride=1)

n_samples = 2
n_words = 3
embed_dim = 12
x = torch.rand(n_samples,n_words,embed_dim)
l =list(conv1.parameters())
print(l[0].size) # [1,3,12]

it works fine, but then I saw this SO where we need to set in_channels=embed_dim and kernel_size=n_words, but this says that my way is correct.
So, how should in_channel be understood in this 1D problem? A lot of places people write "its the number of channels" (which is rather poor explanation).
For an RGB-image, in_channel is the depth e.g 3 but I struggle to wrap my head around, how it is used/understood in my text-classification example


title: c++ call python pytorch model, result is different?

body:
I have a trained model in pytorch, i saved it using:
tsm = torch.jit.trace(mlp, torch.rand(1, num_features).double())
tsm.save('model.pt')
print(mlp(np.array([[0.2] * num_features])))

and i load it in c++ and forward using the same data -> [0.2] * num_features, code is following:
#include <iostream>
#include <memory>
#include <torch/script.h> // One-stop header.
#include <torch/custom_class.h>

int main(int argc, const char* argv[]) {
  if (argc != 2) {
    std::cerr << "usage: example-app <path-to-exported-script-module>\n";
    return -1;
  }

  torch::jit::script::Module module = torch::jit::load(argv[1]);

  std::vector<torch::jit::IValue> inputs;
  // std::vector<double> inputs;
  // inputs.push_back(torch::ones({1, 51}));
  std::vector<double> v(51, 0.2);
  torch::Tensor data = torch::from_blob(v.data(), {1, 51});
  inputs.push_back(data);

  // at::Tensor output = module.forward(inputs).toTensor();
  // std::cout << output.slice(/*dim=*/1, /*start=*/0, /*end=*/5) << '\n';
  auto t = module.forward(inputs).toTensor();
  std::cout << t.sizes() << std::endl;
  std::cout << module.forward(inputs).toTensor()[0][0].item<double>();//.toDouble() << std::endl;
}

The output is:
0.112908% and tensor([0.0010], dtype=torch.float64, grad_fn=<AddBackward0>)
can anyone help on this? why the result is different?


title: Having issues importing PyTorch in PyCharm IDE

body:
I've been getting this error:
ERROR: torch-1.0.1-cp37-cp37m-win_amd64.whl is not a supported wheel on this platform.
after typing this into the PyCharm terminal:
pip3 install https://download.pytorch.org/whl/cpu/torch-1.0.1-cp37-cp37m-win_amd64.whl   to install pytorch. Any way to fix this?
Also, I see PyTorch 1.7.1 has successfully downloaded into my Anaconda, but when I launch pyCharm thorough Anaconda, it can't seem to find Pytorch
I'm using MacOS version 10.15.7 on a 2015 macbook pro


title: Can some please explain me how pixel values of generated image get updated and back prop happens in Neural Style Transfer

body: I'm trying to learn neural style transfer but I'm a little confused with how pixel values of the generated image get updated and back prop happens in Neural Style Transfer.


title: Test a virtual try-on module with mmfashion but I got error

body:
Im trying to test code mmfashion, virtual tryon (https://github.com/open-mmlab/mmfashion)
I typed this
python tools/test_virtual_tryon.py \
>         --config configs/virtual_tryon/cp_vton.py \
>         --stage GMM

and error came out
Use load_from_local loader
The model and loaded state dict do not match exactly

unexpected key in source state_dict: generator.unet.0.weight, generator.unet.0.bias,  \ ~~~~

missing keys in source state_dict: feature_extractora_a.layers.0.weight, feature_extractora_a.layers.0.bias, feature_extractora_a.layers.2.weight,
~~~
feature_regression.linear.bias

load checkpoint from: checkpoint/CPVTON/TOM/latest.pth
Traceback (most recent call last):
  File "tools/test_virtual_tryon.py", line 97, in <module>
    main()
  File "tools/test_virtual_tryon.py", line 75, in main
    logger=logger)
  File "/home/azure/venv/lib/python3.7/site-packages/mmfashion-0.4.0-py3.7.egg/mmfashion/apis/test_virtual_tryon.py", line 22, in test_geometric_matching
    _non_dist_test_gmm(model, dataset, cfg, validate=validate)
  File "/home/azure/venv/lib/python3.7/site-packages/mmfashion-0.4.0-py3.7.egg/mmfashion/apis/test_virtual_tryon.py", line 45, in _non_dist_test_gmm
    for batch, data in enumerate(data_loader):
  File "/home/azure/venv/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 517, in __next__
    data = self._next_data()
  File "/home/azure/venv/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1199, in _next_data
    return self._process_data(data)
  File "/home/azure/venv/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1225, in _process_data
    data.reraise()
  File "/home/azure/venv/lib/python3.7/site-packages/torch/_utils.py", line 429, in reraise
    raise self.exc_type(msg)
RuntimeError: Caught RuntimeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/home/azure/venv/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 202, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/azure/venv/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/azure/venv/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/azure/venv/lib/python3.7/site-packages/mmfashion-0.4.0-py3.7.egg/mmfashion/datasets/CP_VTON.py", line 90, in __getitem__
    shape = self.transform(parse_shape)
  File "/home/azure/venv/lib/python3.7/site-packages/torchvision-0.9.1-py3.7-linux-x86_64.egg/torchvision/transforms/transforms.py", line 60, in __call__
    img = t(img)
  File "/home/azure/venv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/azure/venv/lib/python3.7/site-packages/torchvision-0.9.1-py3.7-linux-x86_64.egg/torchvision/transforms/transforms.py", line 221, in forward
    return F.normalize(tensor, self.mean, self.std, self.inplace)
  File "/home/azure/venv/lib/python3.7/site-packages/torchvision-0.9.1-py3.7-linux-x86_64.egg/torchvision/transforms/functional.py", line 336, in normalize
    tensor.sub_(mean).div_(std)
RuntimeError: output with shape [1, 256, 192] doesn't match the broadcast shape [3, 256, 192]

what should I do?
Is that right, that I got 3 errors?
unexpected key in source state_dict:
RuntimeError: Caught RuntimeError in DataLoader worker process 0.
RuntimeError: output with shape [1, 256, 192] doesn't match the broadcast shape [3, 256, 192]


title: AWS Sagemaker custom PyTorch model inference on raw image input

body:
I am new to AWS Sagemaker. I have custom CV PyTorch model locally and deployed it to Sagemaker endpoint. I used custom inference.py code to define model_fn, input_fn, output_fn and predict_fn methods. So, I'm able to generate predictions on json input, which contains url to the image, the code is quite straigtforward:
def input_fn(request_body, content_type='application/json'):

    logging.info('Deserializing the input data...')

    image_transform = transforms.Compose([
            transforms.Resize(size=(224, 224)),
            transforms.ToTensor(),
            transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
        ])

    if content_type:
        if content_type == 'application/json':
            input_data = json.loads(request_body)
            url = input_data['url']
            logging.info(f'Image url: {url}')
            image_data = Image.open(requests.get(url, stream=True).raw)

        return image_transform(image_data)
    raise Exception(f'Requested unsupported ContentType in content_type {content_type}')

Then I am able to invoke endpoint with code:
client = boto3.client('runtime.sagemaker')
inp = {"url":url}
inp = json.loads(json.dumps(inp))

response = client.invoke_endpoint(EndpointName='ENDPOINT_NAME',
                                  Body=json.dumps(inp),
                                  ContentType='application/json')

The problem is, I see, that locally url request return slightly different image array comparing to the one on Sagemaker. Which is why on the same URL I obtain slightly different predictions. To check that at least model weights are the same I want to generate predictions on image itself, downloaded locally and to Sagemaker. But I fail trying to put image as input to endpoint. E.g.:
def input_fn(request_body, content_type='application/json'):

    logging.info('Deserializing the input data...')

    image_transform = transforms.Compose([
            transforms.Resize(size=(224, 224)),
            transforms.ToTensor(),
            transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
        ])

    if content_type == 'application/x-image':
        image_data = request_body

        return image_transform(image_data)
    raise Exception(f'Requested unsupported ContentType in content_type {content_type}')

Invoking endpoint I experience the error:

ParamValidationError: Parameter validation failed:
Invalid type for parameter Body, value: {'img': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=630x326 at 0x7F78A61461D0>}, type: <class 'dict'>, valid types: <class 'bytes'>, <class 'bytearray'>, file-like object

Does anybody know how to generate Sagemaker predictions by Pytorch model on images?


title: error message when setup monodepth2 network in pycharm

body:
I just wanted to setup my machine (windows 10) for the monodepth2 network from here
When I installed a fresh anaconda-version from their website I wanted to install the dependencies like it can be seen in the readme.txt
conda install pytorch=0.4.1 torchvision=0.2.1 -c pytorch

I got the following error message:
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: /
Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.-
failed

UnsatisfiableError: The following specifications were found to be incompatible with each other:

Output in format: Requested package -> Available versions

Package pytorch conflicts for:
pytorch=0.4.1
torchvision=0.2.1 -> pytorch[version='>=0.4']

Package zlib conflicts for:
torchvision=0.2.1 -> pillow[version='>=4.1.1'] -> zlib[version='1.2.*|1.2.11|1.2.11.*|>=1.2.11,<1.3.0a0|1.2.8']
python=3.8 -> sqlite[version='>=3.33.0,<4.0a0'] -> zlib[version='>=1.2.11,<1.3.0a0']

In pycharm I get the following error message:
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... failed with initial frozen solve. Retrying with flexible solve.

PackagesNotFoundError: The following packages are not available from current channels:

  - torch

Current channels:

  - https://repo.anaconda.com/pkgs/main/win-64
  - https://repo.anaconda.com/pkgs/main/noarch
  - https://repo.anaconda.com/pkgs/r/win-64
  - https://repo.anaconda.com/pkgs/r/noarch
  - https://repo.anaconda.com/pkgs/msys2/win-64
  - https://repo.anaconda.com/pkgs/msys2/noarch
  - https://conda.anaconda.org/conda-forge/win-64
  - https://conda.anaconda.org/conda-forge/noarch

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.


I am trying to import worker_manager_loop as follows:
from torch.utils.data.dataloader import _worker_manager_loop

I get an error that says: cannot import name '_worker_manager_loop' from 'torch.utils.data.dataloader'
I looked up this problem and realized that _worker_manager_loop was replaced with _pin_memory_loop after Pytorch 1.0.0 (according to this).
However, even after doing the recommended fix, I still get this error: cannot import name '_pin_memory_loop' from 'torch.utils.data.dataloader'
I know that the most straightforward thing I can do is downgrade my Pytorch to 1.0, but I would like to know what _pin_memory_loop was renamed to in version 1.7, as I would like to avoid downgrading.
My Pytorch version is 1.7.1 with Python 3.7.6 and cudnn 7.6.5 Cuda 10.0 on Windows 10.


title: Could'nt install libtiff in conda

body: I used the command conda install libtiff=4.0.10 --no-deps, but it showed conflicts with present environment and could not install it