title: pytorch backward error, one of variables for gradient computation modified by an inplace operation

body:
I'm new to pytorch, i've been trying to implement a text summarization network. When i call loss.backward() an error appears.

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [10, 1, 1, 400]], which is output 0 of UnsqueezeBackward0, is at version 98; expected version 97 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

it's a seq2seq model, and i think the problem comes from this code snippet
    final_dists=torch.zeros((batch_size,dec_max_len,extended_vsize)) #to hold the model outputs with extended vocab
    attn_dists=torch.zeros((batch_size,dec_max_len,enc_max_len)) #to retain the attention weights over decoder steps
    coverages=torch.zeros((batch_size,dec_max_len,enc_max_len))  #the coverages are retained to compute coverage loss

    inp=self.emb_dropout(self.embedding(dec_batch[:,0])) #starting input: <SOS> shape [batch_size]
    #self.prev_coverage is the accumulated coverage

    coverage=None #initially none, but accumulates

    with torch.autograd.set_detect_anomaly(True):

        for i in range(1,dec_max_len):
            #NOTE: the outputs, atten_dists, p_gens assignments start from i=1 (DON'T FORGET!)
            vocab_dists,hidden,attn_dists_tmp,p_gen,coverage=self.decoder(inp,hidden,enc_outputs,enc_lens,coverage)

            attn_dists[:,i,:]=attn_dists_tmp.squeeze(1)
            coverages[:,i,:]=coverage.squeeze(1)
            #vocab_dists: [batch_size, 1, dec_vocab_size] Note: this is the normalized probability
            #hidden: [1,batch_size, dec_hid_dim]
            #attn_dists_tmp: [batch_size, 1, enc_max_len]
            #p_gen: [batch_size, 1]
            #coverage: [batch_size, 1, enc_max_len]

            #===================================================================
            #To compute the final dist in pointer-generator network by extending vocabulary
            vocab_dists_p=p_gen.unsqueeze(-1)*vocab_dists  #[batch_size,1,dec_vocab_size] note we want to maintain vocab_dists for teacher_forcing_ratio
            attn_dists_tmp=(1-p_gen).unsqueeze(-1)*attn_dists_tmp #[batch_size, 1, enc_max_len] note we want to maintain attn_dists for later use
            extra_zeros=torch.zeros((batch_size,1,max_art_oovs)).to(self.device)
            vocab_dists_extended=torch.cat((vocab_dists_p,extra_zeros),dim=2) #[batch_size, 1, extended_vsize]
            attn_dists_projected=torch.zeros((batch_size,1,extended_vsize)).to(self.device)
            indices=enc_batch_extend_vocab.clone().unsqueeze(1) #[batch_size, 1,enc_max_size]

            attn_dists_projected=attn_dists_projected.scatter(2,indices,attn_dists_tmp)
            #We need this otherwise we would modify a leaf Variable inplace
            #attn_dists_projected_clone=attn_dists_projected.clone()
            #attn_dists_projected_clone.scatter_(2,indices,attn_dists_tmp) #this will project the attention weights
            #attn_dists_projected.scatter_(2,indices,attn_dists_tmp)
            final_dists[:,i,:]=vocab_dists_extended.squeeze(1)+attn_dists_projected.squeeze(1)
            #===================================================================

            #teacher forcing, whether or not should use pred or dec sequence label
            if random.random()<teacher_forcing_ratio:
                inp=self.emb_dropout(self.embedding(dec_batch[:,i]))
            else:
                inp=self.emb_dropout(self.embedding(vocab_dists.squeeze(1).argmax(1)))

if i remove the for loop, and just do one step of updating attn_dists[:,1,:] etc, with toy loss from the outputs returned by forward, then it works fine.
Anyone has any idea what is wrong here? There is no inplace operation here. Many thanks!


title: load pytorch dataloader into GPU

body:
Is there a way to load a pytorch DataLoader (torch.utils.data.Dataloader) entirely into my GPU?
Now, I load every batch separately into my GPU.
CTX = torch.device('cuda')

train_loader = torch.utils.data.DataLoader(
    train_dataset,
    batch_size=BATCH_SIZE,
    shuffle=True,
    num_workers=0,
)

net = Net().to(CTX)
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=LEARNING_RATE)

for epoch in range(EPOCHS):
    for inputs, labels in test_loader:
        inputs = inputs.to(CTX)        # this is where the data is loaded into GPU
        labels = labels.to(CTX)

        optimizer.zero_grad()

        outputs = net(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

    print(f'training accuracy: {net.validate(train_loader, device=CTX)}/{len(train_dataset)}')
    print(f'validation accuracy: {net.validate(test_loader, device=CTX)}/{len(test_dataset)}')

where the Net.validate() function is given by
def validate(self, val_loader, device=torch.device('cpu')):
    correct = 0
    for inputs, labels in val_loader:
        inputs = inputs.to(device)
        labels = labels.to(device)
        outputs = torch.argmax(self(inputs), dim=1)
        correct += int(torch.sum(outputs==labels))
    return correct

I would like to improve the speed by loading the entire dataset trainloader into my GPU, instead of loading every batch separately. So, I would like to do something like
train_loader.to(CTX)

Is there an equivalent function for this? Because torch.utils.data.DataLoader does not have this attribute .to().
I work with an NVIDIA GeForce RTX 2060 with CUDA Toolkit 10.2 installed.


title: How to format TSV files to use with torchtext?

body:
The way i'm formatting is like:

Jersei  N
atinge  V
média   N
. PU

Programe    V
...

First string in each line is the lexical item, the other is a pos tag. But the empty-line (that i'm using to indicate the end of a sentence) gives me the error AttributeError: 'Example' object has no attribute 'text' when running the given code:

src = data.Field()
trg = data.Field(sequential=False)
mt_train = datasets.TabularDataset(
    path='/path/to/file.tsv',
    fields=(src, trg))
src.build_vocab(train)

How the proper way to indicate EOS to torchtext?


title: How to print a tensor without showing gradients

body:
If I do something like this:

tmp = torch.ones(3, 2, 2, requires_grad=True)
out = tmp ** 2
print("\n{}".format(out))

I get as an output:

tensor([[[1., 1.],
         [1., 1.]],

        [[1., 1.],
         [1., 1.]],

        [[1., 1.],
         [1., 1.]]], grad_fn=<PowBackward0>)

I would like to print out just the values, not the grad_fn part.

However, doing

print("\n{}".format(out[0]))

results in:

tensor([[1., 1.],
        [1., 1.]], grad_fn=<SelectBackward>)

The only way I know is to out.detach() or there is another/better way? Just to clarify, I am happy the gradient is calculated. I just want to show the vector values without additional data.


title: Optional tensors in PyTorch c++ extension

body:
I'm writing  a C++ extension for pytorch, and using the c++ api to do so. To my forward function, I need to pass an optional tensor. Inside the function, I want to do different things based on whether this optional parameter was passed or not. In general, we use NULL for optional pointer arguments in C++ and check inside the function if the pointer is NULL or not. I don't know how to do this for the at::Tensor type of Torch's c++ api.

void xyz_forward(
    const at::Tensor xyz1,
    const at::Tensor xyz2,
    const at::Tensor optional_constraints = something)
{
     if(optional_constraints){
        //do something
     }else{
        //do something else
     }
}

Note that, I can't do const at::Tensor optional_constraints = at::ones or something, because that parameter can take any real value and can be of varying size/shape. I can't assign it a numerical value as an optional argument. Is there a NULL equivalent for this?


title: LSTM implementation / overfitting

body:
I am having a problem on an implementation of LSTM. I am not sure if I have the right implementation or this is just an overfitting problem. I am doing essay grading using a LSTM, scoring text with score from 0 - 10 (or other range of score). I am using the ASAP kaggle competition data as one of the training data.

However, the main goal is to achieve good performance on a private dataset, with around 500 samples. The 500 samples includes validation and training set. I have previously done some experiment and got the model to work, but after fiddling with something, the model doesn't fit anymore. The model does not improve at all. I have also re-implemented the code in a cleaner manner with much more obejct oriented code and still can't reproduce my previous result.

However, I am getting the model to fit to my data, just there is tremendous overfitting. I am not sure if this is an implementation problem of some sort or just overfitting, but I cannot get the model to work. The maximum I can get it to is 0.35 kappa using LSTM on the ASAP data essay set 1. For some bizarre reason, I can get a single layer fully connected model to have 0.75 kappa. I think this is an implementation problem but I am not sure.

Here is my old code:

train.py

import gensim
import numpy as np
import pandas as pd
import torch
from sklearn.metrics import cohen_kappa_score
from torch import nn
import torch.utils.data as data_utils
from torch.optim import Adam

from dataset import AESDataset
from network import Network

from optimizer import Ranger
from qwk import quadratic_weighted_kappa, kappa

batch_size = 32

device = "cuda:0"
torch.manual_seed(1000)
# Load data from csv
file_name = "data/data_new.csv"
data = pd.read_csv(file_name)
arr = data.to_numpy()
text = arr[:, :2]
text = [str(line[0]) + str(line[1]) for line in text]
text = [gensim.utils.simple_preprocess(line) for line in text]

score = arr[:,2]

score = [sco*6 for sco in score]
score = np.asarray(score, dtype=int)

train_dataset = AESDataset(text_arr=text[:400], scores=score[:400])
test_dataset = AESDataset(text_arr=text[400:], scores=score[400:])

score = torch.tensor(score).view(-1,1).long().to(device)

train_loader = data_utils.DataLoader(train_dataset,shuffle=True, batch_size=batch_size, drop_last=True)
test_loader = data_utils.DataLoader(test_dataset,shuffle=True,batch_size=batch_size, drop_last=True)

out_class = 61

epochs = 1000

model = Network(out_class).to(device)
model.load_state_dict(torch.load("model/best_model"))
y_onehot = torch.FloatTensor(batch_size, out_class).to(device)
optimizer = Adam(model.parameters())
criti = torch.nn.CrossEntropyLoss()
# model, optimizer = amp.initialize(model, optimizer, opt_level="O2")

step = 0

for i in range(epochs):
    #Testing
    if i % 1 == 0:
        total_loss = 0
        total_kappa = 0
        total_batches = 0
        model.eval()
        for (text, score) in test_loader:

            out = model(text)
            out_score = torch.argmax(out, 1)
            y_onehot.zero_()
            y_onehot.scatter_(1, score, 1)
            kappa_l = cohen_kappa_score(score.view(batch_size).tolist(), out_score.view(batch_size).tolist())
            score = score.view(-1)
            loss = criti(out, score.view(-1))
            total_loss += loss
            total_kappa += kappa_l
            total_batches += 1
        print(f"Epoch {i} Testing kappa {total_kappa/total_batches} loss {total_loss/total_batches}")
        with open(f"model/epoch_{i}", "wb") as f:
            torch.save(model.state_dict(),f)
        model.train()
    #Training

    for (text, score) in train_loader:

        optimizer.zero_grad()
        step += 1
        out = model(text)
        out_score = torch.argmax(out,1)
        y_onehot.zero_()
        y_onehot.scatter_(1, score, 1)
        kappa_l = cohen_kappa_score(score.view(batch_size).tolist(),out_score.view(batch_size).tolist())
        loss = criti(out, score.view(-1))
        print(f"Epoch {i} step {step} kappa {kappa_l} loss {loss}")
        loss.backward()
        optimizer.step()

dataset.py

import gensim
import torch
import numpy as np

class AESDataset(torch.utils.data.Dataset):
    def __init__(self, text_arr, scores):
        self.data = text_arr
        self.scores = scores
        self.w2v_model = ("w2vec_model_all")
        self.max_len = 500
    def __getitem__(self, item):
        vector = []
        essay = self.data[item]

        pad_vec = [1 for i in range(300)]
        for i in range(self.max_len - len(essay)):
            vector.append(pad_vec)
        for word in essay:
            word_vec = pad_vec
            try:
                word_vec = self.w2v_model[word]
            except:
                #print(f"Skipping word as word {word} not in dictionary")
                word_vec = pad_vec

            vector.append(word_vec)
        #print(len(vector))
        vector = np.stack(vector)
        tensor = torch.tensor(vector[:self.max_len]).float().to("cuda")
        score = self.scores[item]
        score = torch.tensor(score).long().to("cuda").view(1)

        return tensor, score

    def __len__(self):
        return len(self.scores)

network.py

import torch.nn as nn
import torch

import torch.nn.functional as F

class Network(nn.Module):
    def __init__(self, output_size):
        super(Network, self).__init__()
        self.lstm = nn.LSTM(300,500,1, batch_first=True)
        self.dropout = nn.Dropout(p=0.5)
        #self.l2 = nn.L2
        self.linear = nn.Linear(500,output_size)

    def forward(self,x):
        x, _ = self.lstm(x)
        x = x[:,-1,:]
        x = self.dropout(x)
        x = self.linear(x)

        return x

My new code: https://github.com/Clement-Hui/EssayGrading


title: How to use Pytorch Tensor object in Opencv without convert to numpy array?

body:
I’m trying to develop a text detection application with Pytorch and Opencv in Python.
I can use Pytorch tensor with Opencv like below.

val = y[0,:,:,0].data.cpu().numpy()
cv2.threshold(val , 0.4, 1, 0)

But it takes a lot of time. I need to do this operation by using the tensor object. How can I do that?


title: Correct way to register a parameter for model in Pytorch

body:
I tried to define a simple model in Pytorch. The model computes negative log prob for a gaussian distribution:
import torch
import torch.nn as nn

class GaussianModel(nn.Module):

    def __init__(self):
        super(GaussianModel, self).__init__()

        self.register_parameter('mean', nn.Parameter(torch.zeros(1),
                                                     requires_grad=True))

        self.pdf = torch.distributions.Normal(self.state_dict()['mean'],
                                              torch.tensor([1.0]))
    def forward(self, x):
        return -self.pdf.log_prob(x)

model = GaussianModel()

Then I tried to optimize the mean parameter:
optimizer = torch.optim.SGD(model.parameters(), lr=0.002)
for _ in range(5):
  optimizer.zero_grad()
  nll = model(torch.tensor([3.0], requires_grad=True))
  nll.backward()
  optimizer.step()
  print('mean : ', model.state_dict()['mean'],
                 ' - Negative Loglikelihood : ', nll.item())

But it seems the gradient is zero and mean does not change:
mean :  tensor([0.])  - Negative Loglikelihood :  5.418938636779785
mean :  tensor([0.])  - Negative Loglikelihood :  5.418938636779785
mean :  tensor([0.])  - Negative Loglikelihood :  5.418938636779785
mean :  tensor([0.])  - Negative Loglikelihood :  5.418938636779785
mean :  tensor([0.])  - Negative Loglikelihood :  5.418938636779785

Did I register and use the mean parameter correctly? can autograd compute the gradient for torch.distributions.Normal.log_prob or I should implement the backward() for the model?


title: Remove downloaded tensorflow and pytorch(Hugging face) models

body:
I would like to remove tensorflow and hugging face models from my laptop.
I did find one link https://github.com/huggingface/transformers/issues/861
but is there not command that can remove them because as mentioned in the link manually deleting can cause problems because we don't know which other files are linked to those models or are expecting some model to be present in that location or simply it may cause some error.


title: How to concatenate 2 pytorch models and make the first one non-trainable in PyTorch

body:
I've two networks, which I need to concatenate for my full model. However my first model is pre-trained and I need to make it non-trainable when training the full model. How can I achieve this in PyTorch.
I am able to concatenate two models using this answer
class MyModelA(nn.Module):
    def __init__(self):
        super(MyModelA, self).__init__()
        self.fc1 = nn.Linear(10, 2)

    def forward(self, x):
        x = self.fc1(x)
        return x

class MyModelB(nn.Module):
    def __init__(self):
        super(MyModelB, self).__init__()
        self.fc1 = nn.Linear(20, 2)

    def forward(self, x):
        x = self.fc1(x)
        return x

class MyEnsemble(nn.Module):
    def __init__(self, modelA, modelB):
        super(MyEnsemble, self).__init__()
        self.modelA = modelA
        self.modelB = modelB

    def forward(self, x):
        x1 = self.modelA(x)
        x2 = self.modelB(x1)
        return x2

# Create models and load state_dicts
modelA = MyModelA()
modelB = MyModelB()
# Load state dicts
modelA.load_state_dict(torch.load(PATH))

model = MyEnsemble(modelA, modelB)
x = torch.randn(1, 10)
output = model(x)

Basically here, I want to load pre-trained modelA and make it non-trainable when training the Ensemble model.


title: while installing apex extension for pytorch(python environment) the following error is showing, am unable to solve this problem

body:
I want to install apex extension for my pytorch environment, my system is windows 10 and am using python version 3.8.1 and pip version is 20.0.2

I read the instructions from this https://github.com/NVIDIA/apex and I executed the command

pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext

This error is showing.

  c:\users\dell\appdata\local\programs\python\python38\lib\site-packages\pip_internal\commands\install.py:244: UserWarning: Disabling all use of wheels due to the use of --build-option / --global-option / --install-option.
        cmdoptions.check_install_build_global(options)
      Non-user install because site-packages writeable
      Created temporary directory: C:\Users\Dell\AppData\Local\Temp\pip-ephem-wheel-cache-ehoqwpvf
      Created temporary directory: C:\Users\Dell\AppData\Local\Temp\pip-req-tracker-uowlsjqi
      Initialized build tracking at C:\Users\Dell\AppData\Local\Temp\pip-req-tracker-uowlsjqi
      Created build tracker: C:\Users\Dell\AppData\Local\Temp\pip-req-tracker-uowlsjqi
      Entered build tracker: C:\Users\Dell\AppData\Local\Temp\pip-req-tracker-uowlsjqi
      Created temporary directory: C:\Users\Dell\AppData\Local\Temp\pip-install-rivnsaa9
      Cleaning up...
      Removed build tracker: 'C:\Users\Dell\AppData\Local\Temp\pip-req-tracker-uowlsjqi'
      ERROR: You must give at least one requirement to install (see "pip help install")
      Exception information:
      Traceback (most recent call last):
        File "c:\users\dell\appdata\local\programs\python\python38\lib\site-packages\pip_internal\cli\base_command.py", line 186, in _main
          status = self.run(options, args)

Please solve this problem


title: How to add a 2d tensor to every 2d tensor from a 3d tensor

body:
I'm trying to add a 2d tensor to every 2d tensor from a 3d tensor.
Let's say i have a  tensor a with (2,3,2) shape and a tensor b with (2,2) shape.

a = [[[1,2],
      [1,2],
      [1,2]],
     [[3,4],
      [3,4],
      [3,4]]]

b = [[1,2], [3,4]]

#the result i want to get
a[:, 0, :] + b
a[:, 1, :] + b
a[:, 2, :] + b

I wanna know if there is a method in pytorch that can do this.


title: Verify convolution theorem using pytorch

body:
Basically this theorem is formulated as below:

F(f*g) = F(f)xF(g)

I know this theorem but I just simply cannot reproduce the result by using pytorch.

Below is a reproducable code:

import torch
import torch.nn.functional as F

# calculate f*g
f = torch.ones((1,1,5,5))
g = torch.tensor(list(range(9))).view(1,1,3,3).float()
conv = F.conv2d(f, g, bias=None, padding=2)

# calculate F(f*g)
F_fg = torch.rfft(conv, signal_ndim=2, onesided=False)

# calculate F x G
f = f.squeeze()
g = g.squeeze()

# need to pad into at least [w1+w2-1, h1+h2-1], which is 7 in our case.
size = f.size(0) + g.size(0) - 1

f_new = torch.zeros((7,7))
g_new = torch.zeros((7,7))

f_new[1:6,1:6] = f
g_new[2:5,2:5] = g

F_f = torch.rfft(f_new, signal_ndim=2, onesided=False)
F_g = torch.rfft(g_new, signal_ndim=2, onesided=False)
FxG = torch.mul(F_f, F_g)

print(FxG - F_fg)

here is the result for print(FxG - F_fg)

tensor([[[[[ 0.0000e+00,  0.0000e+00],
       [ 4.1426e+02,  1.7270e+02],
       [-3.6546e+01,  4.7600e+01],
       [-1.0216e+01, -4.1198e+01],
       [-1.0216e+01, -2.0223e+00],
       [-3.6546e+01, -6.2804e+01],
       [ 4.1426e+02, -1.1427e+02]],

      ...

      [[ 4.1063e+02, -2.2347e+02],
       [-7.6294e-06,  2.2817e+01],
       [-1.9024e+01, -9.0105e+00],
       [ 7.1708e+00, -4.1027e+00],
       [-2.6739e+00, -1.1121e+01],
       [ 8.8471e+00,  7.1710e+00],
       [ 4.2528e+01,  9.7559e+01]]]]])

and you can see that the difference is not always 0.

can someone tell me why and how to do this properly?

Thanks


title: PyTorch: How to append tensors into a list using loops

body:
I have the following code which outputs 2 arrays in a list:
arr1 = np.array([[1.,2,3], [4,5,6], [7,8,9]])

arr_split = np.array_split(arr1,
                           indices_or_sections = 2,
                           axis = 0)

arr_split

Output:
[array([[1., 2., 3.],
        [4., 5., 6.]]), array([[7., 8., 9.]])]

How do I cast these 2 arrays into PyTorch tensors and put them into a list using for (or while) loops, so that they look like this:
[tensor([[1., 2., 3.],
         [4., 5., 6.]], dtype=torch.float64),
tensor([[7., 8., 9.]], dtype=torch.float64)]

Many thanks in advance!


title: I am trying to install intel optimized pytorch in different ways

body:
I am very first time using pyTorch. I am trying to install it. In how many ways I can do this?
Please provide the steps for that.


title: Calculate binary entropy loss using a function in pytorch

body:
I have a problem about calculating binary cross entropy. The way I know that works out in pytorch is:

import torch
import torch.nn as nn
import torch.nn.functional as F
def lossfunc():
    return F.binary_cross_entropy

criterion = lossFunc()
input = torch.randn((3, 2), requires_grad=True)
target = torch.rand((3, 2), requires_grad=False)
loss = criterion(torch.sigmoid(input),target)

But how to complete the lossfunc() in such way, because I don't know how to pass the arguments to the function:

#the function that add sigmoid to input and calculate the binary cross entropy loss
def lossfunc():
   return

criterion = lossFunc()
input = torch.randn((3, 2), requires_grad=True)
target = torch.rand((3, 2), requires_grad=False)
loss = criterion(input,target)


title: path problem : NameError: name '__file__' is not defined

body:
import os.path as osp
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch_geometric.datasets import MNISTSuperpixels
import torch_geometric.transforms as T
from torch_geometric.data import DataLoader
from torch_geometric.utils import normalized_cut
from torch_geometric.nn import (NNConv, graclus, max_pool, max_pool_x, global_mean_pool)

path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'MNIST')

transform = T.Cartesian(cat=False)
train_dataset = MNISTSuperpixels(path, True, transform=transform)
test_dataset = MNISTSuperpixels(path, False, transform=transform)
train_loader = DataLoader(train_dataset, batch_size=64, shuffle=True)
test_loader = DataLoader(test_dataset, batch_size=64, shuffle=False)
d = train_dataset

I'm trying to use MNISTSuperpixels data for graph convolution, but I have some troubles using the example code.
Most of scripts were using
path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'MNIST')
However, they gave me an error
NameError: name '__file__' is not defined and I don't understand what osp.realpath(__file__) really means.
I'm using Jupyter notebook on Ubuntu, and my working directory is
print(os.getcwd())
/home/hkimlx/GDL/pytorch_geometric/examples

which is the same directory where the sample code mnist_nn_conv.py is located.
Please help me. Thanks!


title: I'm having a problem trying to load a Pytoch model: "Can't find Identity in module"

body:
When trying to load a pytorch model it gives the following attribute error
model = torch.load('../input/melanoma-model/melanoma_model_0.pth')
model = model.to(device)
model.eval()

AttributeError                            Traceback (most recent call
last)  in
1 arch = EfficientNet.from_pretrained('efficientnet-b2')
2 model = Net(arch=arch)
----> 3 torch.load('../input/melanoma-model/melanoma_model_0.pth')
4 model = model.to(device)
5 model.eval()
/opt/conda/lib/python3.7/site-packages/torch/serialization.py in
load(f, map_location, pickle_module, **pickle_load_args)
591                     return torch.jit.load(f)
592                 return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
--> 593         return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
594
595
/opt/conda/lib/python3.7/site-packages/torch/serialization.py in
_legacy_load(f, map_location, pickle_module, **pickle_load_args)
771     unpickler = pickle_module.Unpickler(f, **pickle_load_args)
772     unpickler.persistent_load = persistent_load
--> 773     result = unpickler.load()
774
775     deserialized_storage_keys = pickle_module.load(f, **pickle_load_args)
AttributeError: Can't get attribute 'Identity' on <module
'efficientnet_pytorch.utils' from
'/opt/conda/lib/python3.7/site-packages/efficientnet_pytorch/utils.py'>


title: Why does multi-class classification fails with sigmoid?

body:
MNIST trained with Sigmoid fails while Softmax works fine

I am trying to investigate how different activation affects the final results, so I implemented a simple net for MNIST with PyTorch.

I am using NLLLoss (Negative log likelihood) as it implements Cross Entropy Loss when used with softmax.

When I have softmax as activation of the last layer, it works great.
But when I used sigmoid instead, I noticed that things fall apart

Here is my network code

def forward(self, x):
        x = F.relu(F.max_pool2d(self.conv1(x), 2))
        x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))
        x = x.view(-1, 80)
        x = F.relu(self.fc1(x))
        x = F.dropout(x, training=self.training)
        x = self.fc2(x)
        return F.XXXX(x)

where XXXX is the activation function

both Sigmoid and Softmax output values between (0, 1).
Yes Softmax guarantees the sum of 1 but I am not sure if this answers why the training fails with Sigmoid.
Is there any detail I am not catching here?


title: Regression loss functions incorrect

body:
I'm trying a basic averaging example, but the validation and loss don't match and the network fails to converge if I increase the training time. I'm training a network with 2 hidden layers, each 500 units wide on three integers from the range [0,9] with a learning rate of 1e-1, Adam, batch size of 1, and dropout for 3000 iterations and validate every 100 iterations. If the absolute difference between the label and the hypothesis is less than a threshold, here I set the threshold to 1, I consider that correct. Could someone let me know if this is an issue with the choice of loss function, something wrong with Pytorch, or something I'm doing. Below are some plots:

val_diff = 1
acc_diff = torch.FloatTensor([val_diff]).expand(self.batch_size)

Loop 100 times to during validation:

num_correct += torch.sum(torch.abs(val_h - val_y) < acc_diff)

Append after each validation phase:

validate.append(num_correct / total_val)

Here are some examples of the (hypothesis, and labels):

[...(-0.7043088674545288, 6.0), (-0.15691305696964264, 2.6666667461395264),
 (0.2827358841896057, 3.3333332538604736)]

I tried six of the loss functions in the API that are typically used for regression:

                                                    torch.nn.L1Loss(size_average=False)

                                                                      torch.nn.L1Loss()

                                                     torch.nn.MSELoss(size_average=False)

                                                                   torch.nn.MSELoss()

                                             torch.nn.SmoothL1Loss(size_average=False)

                                                               torch.nn.SmoothL1Loss()

Thanks.

Network code:

class Feedforward(nn.Module):
    def __init__(self, topology):
        super(Feedforward, self).__init__()
        self.input_dim     = topology['features']
        self.num_hidden    = topology['hidden_layers']
        self.hidden_dim    = topology['hidden_dim']
        self.output_dim    = topology['output_dim']
        self.input_layer   = nn.Linear(self.input_dim, self.hidden_dim)
        self.hidden_layer  = nn.Linear(self.hidden_dim, self.hidden_dim)
        self.output_layer  = nn.Linear(self.hidden_dim, self.output_dim)
        self.dropout_layer = nn.Dropout(p=0.2)

    def forward(self, x):
        batch_size = x.size()[0]
        feat_size  = x.size()[1]
        input_size = batch_size * feat_size

        self.input_layer = nn.Linear(input_size, self.hidden_dim).cuda()
        hidden = self.input_layer(x.view(1, input_size)).clamp(min=0)

        for _ in range(self.num_hidden):
            hidden = self.dropout_layer(F.relu(self.hidden_layer(hidden)))

        output_size = batch_size * self.output_dim
        self.output_layer = nn.Linear(self.hidden_dim, output_size).cuda()
        return self.output_layer(hidden).view(output_size)

Training code:

def train(self):
    if self.cuda:
        self.network.cuda()

    dh        = DataHandler(self.data)
    # loss_fn = nn.L1Loss(size_average=False)
    # loss_fn = nn.L1Loss()
    # loss_fn = nn.SmoothL1Loss(size_average=False)
    # loss_fn = nn.SmoothL1Loss()
    # loss_fn = nn.MSELoss(size_average=False)
    loss_fn   = torch.nn.MSELoss()
    losses    = []
    validate  = []
    hypos     = []
    labels    = []
    val_size  = 100
    val_diff  = 1
    total_val = float(val_size * self.batch_size)

    for i in range(self.iterations):
        x, y = dh.get_batch(self.batch_size)
        x = self.tensor_to_Variable(x)
        y = self.tensor_to_Variable(y)

        self.optimizer.zero_grad()
        loss = loss_fn(self.network(x), y)
        loss.backward()
        self.optimizer.step()


title: pytorch how to select channels by mask?

body:
I want to know how do I select channels by the mask in Pytorch.
[channel1 channel2 channel3 channel4] x [1,0,0,1] --> [channel1,channel4]
I tried torch.masked_select() and it did't work.

if the input has a shape like [B,C,H,W] the output's shape should be [B,masked_C,H,W],

import torch
from torch import nn
input = torch.randn((1,5,3,3))
pool = nn.AdaptiveAvgPool2d(1)
w = torch.sigmoid(pool(input)).view(1,-1)
mask = torch.gt(w,0.5)
print(input)
print(w)
print(mask)

the output is as following:

tensor([[[[ 0.9129, -0.9763,  1.4460],
          [ 0.3608,  0.5561, -1.4612],
          [ 1.4953, -1.2474,  0.4069]],

         [[-0.9121,  0.1261,  0.4661],
          [-1.1624, -1.0266, -1.5419],
          [ 1.0644,  1.0039, -0.4022]],

         [[-1.8454, -0.2150,  2.3703],
          [ 0.5224,  0.3366,  1.7545],
          [-0.4624,  1.2639,  1.8032]],

         [[-1.1558, -1.9985, -1.1336],
          [-0.4400, -0.2092,  0.0677],
          [-0.4172, -0.3614, -1.3193]],

         [[-0.9441, -0.2944,  0.3381],
          [ 1.6562, -0.5623,  0.0599],
          [ 0.7229,  0.0472, -0.5122]]]])
tensor([[0.5414, 0.4341, 0.6489, 0.3156, 0.5142]])
tensor([[1, 0, 1, 0, 1]], dtype=torch.uint8)

the result I want is like this:

tensor([[[[ 0.9129, -0.9763,  1.4460],
          [ 0.3608,  0.5561, -1.4612],
          [ 1.4953, -1.2474,  0.4069]],

         [[-1.8454, -0.2150,  2.3703],
          [ 0.5224,  0.3366,  1.7545],
          [-0.4624,  1.2639,  1.8032]],

         [[-0.9441, -0.2944,  0.3381],
          [ 1.6562, -0.5623,  0.0599],
          [ 0.7229,  0.0472, -0.5122]]]])


title: Pysyft Federated learning, Error with Websockets

body:
I am trying to run a federated learning from pysyft (https://github.com/OpenMined/PySyft/blob/dev/examples/tutorials/advanced/websockets-example-MNIST-parallel/Asynchronous-federated-learning-on-MNIST.ipynb) that creates remote workers and connect to them via websockets. however I am getting an error in folllowing evaluation step.

future: <Task finished coro=<WebsocketServerWorker._producer_handler() done, defined at C:\Users\Public\Anaconda\lib\site-packages\syft\workers\websocket_server.py:95> exception=AttributeError("'dict' object has no attribute 'owner'")>
Traceback (most recent call last):
  File "C:\Users\Public\Anaconda\lib\site-packages\syft\generic\frameworks\hook\hook_args.py", line 663, in register_response
    register_response_function = register_response_functions[attr_id]
KeyError: 'evaluate'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Public\Anaconda\lib\site-packages\syft\workers\websocket_server.py", line 113, in _producer_handler
    response = self._recv_msg(message)
  File "C:\Users\Public\Anaconda\lib\site-packages\syft\workers\websocket_server.py", line 124, in _recv_msg
    return self.recv_msg(message)
  File "C:\Users\Public\Anaconda\lib\site-packages\syft\workers\base.py", line 310, in recv_msg
    response = self._message_router[type(msg)](msg.contents)
  File "C:\Users\Public\Anaconda\lib\site-packages\syft\workers\base.py", line 457, in execute_command
    command_name, response, list(return_ids), self
  File "C:\Users\Public\Anaconda\lib\site-packages\syft\generic\frameworks\hook\hook_args.py", line 672, in register_response
    new_response = register_response_function(response, response_ids=response_ids, owner=owner)
  File "C:\Users\Public\Anaconda\lib\site-packages\syft\generic\frameworks\hook\hook_args.py", line 766, in <lambda>
    return lambda x, **kwargs: f(lambdas, x, **kwargs)
  File "C:\Users\Public\Anaconda\lib\site-packages\syft\generic\frameworks\hook\hook_args.py", line 522, in two_fold
    return lambdas[0](args[0], **kwargs), lambdas[1](args[1], **kwargs)
  File "C:\Users\Public\Anaconda\lib\site-packages\syft\generic\frameworks\hook\hook_args.py", line 744, in <lambda>
    else lambda i, **kwargs: register_tensor(i, **kwargs)
  File "C:\Users\Public\Anaconda\lib\site-packages\syft\generic\frameworks\hook\hook_args.py", line 712, in register_tensor
    tensor.owner = owner
AttributeError: 'dict' object has no attribute 'owner'

There are no clear answer from their forum. does anyone have any clue as to what the issue is in this script.

My syft version:

syft : 0.2.3a1
syft-proto : 0.1.1a1.post12
torch : 1.4.0


title: Memory Leak in loop pytorch

body:
The following loop is not discarding any of the tensors it makes after each iteration of the loop leading to a memory leak. It is due to the use of grad_loss.backward() in the below code. Is there anything I'm missing or is there an issue with pytorch.

    for (images, one_hot_labels) in tqdm(batched_train_data):
        # I collect batch size here because the last batch may have a smaller batch_size
        images = images.to(device)
        one_hot_labels = one_hot_labels.to(device)

        batch_size = images.shape[0]

        images.requires_grad = True
        optimizer.zero_grad()
        # as images is not a parameters optimizer.zero_grad() won't reset it's gradient
        if images.grad is not None:
            images.grad.data.zero_()

        probabilities = model.forward(images)

        # I want to use .backward() twice rather than autograd because I want to accumulate the gradients
        loss = loss_func(probabilities, one_hot_labels)
        loss.backward(create_graph=True)
        grad_loss = grad_loss_func(images.grad)
        grad_loss.backward()

        optimizer.step()

        labels = one_hot_labels.detach().argmax(dim=1)
        predictions = probabilities.detach().argmax(dim=1)
        num_correct = int(predictions.eq(labels).sum())

        train_data_length += batch_size
        train_correct += num_correct
        train_loss += float(loss.detach()) * batch_size

        writer.add_graph(model, images)
        writer.close()

        # To stop memory leaks
        del images
        del one_hot_labels
        del probabilities
        del loss
        del grad_loss
        del labels
        del predictions
        del num_correct


title: How to initialize mean and variance of Pytorch BatchNorm2d?

body:
I’m transforming a TensorFlow model to Pytorch. And I’d like to initialize the mean and variance of BatchNorm2d using TensorFlow model.
I’m doing it in this way:

bn.running_mean = torch.nn.Parameter(torch.Tensor(TF_param))

And I get this error:

RuntimeError: the derivative for 'running_mean' is not implemented

But is works for bn.weight and bn.bias. Is there any way to initialize the mean and variance using my pre-trained Tensorflow model? Is there anything like moving_mean_initializer and moving_variance_initializer in Pytorch?

Thanks!


title: Shape of target and predictions tensors in PyTorch loss functions

body:
I am confused with the input shapes for tensors in nn.CrossEntropyLoss.
I am trying to implement a simple autoencoder for text sequences. The core of my problem can be illustrated by the following code

predictions = torch.rand(2, 3, 4)
target = torch.rand(2, 3)
print(predictions.shape)
print(target.shape)
nn.CrossEntropyLoss(predictions.transpose(1, 2), target)

In my case predictions has the shape (time_step, batch_size, vocabulary_size) while target has the shape (time_step, batch_size). Next I am transposing the predictions as per description which says that the second dimension of predictions should be the number of classes - vocabulary_size in my case. The code returns an error RuntimeError: bool value of Tensor with more than one value is ambiguous. Could someone please enlighten me how to use the damn thing? Thank you in advance!


title: How to write custom CrossEntropyLoss

body:
I am learning Logistic Regression within Pytorch and to better understand I am defining a custom CrossEntropyLoss as below:

def softmax(x):
    exp_x = torch.exp(x)
    sum_x = torch.sum(exp_x, dim=1, keepdim=True)

    return exp_x/sum_x

def log_softmax(x):
    return torch.exp(x) - torch.sum(torch.exp(x), dim=1, keepdim=True)

def CrossEntropyLoss(outputs, targets):
    num_examples = targets.shape[0]
    batch_size = outputs.shape[0]
    outputs = log_softmax(outputs)
    outputs = outputs[range(batch_size), targets]

    return - torch.sum(outputs)/num_examples

I also make my own logistic regression (to predict FashionMNIST) as below:

input_dim = 784 # 28x28 FashionMNIST data
output_dim = 10

w_init = np.random.normal(scale=0.05, size=(input_dim,output_dim))
w_init = torch.tensor(w_init, requires_grad=True).float()
b = torch.zeros(output_dim)

def my_model(x):
    bs = x.shape[0]
    return x.reshape(bs, input_dim) @ w_init + b

To validate my custom crossentropyloss, I compared it with nn.CrossEntropyLoss from Pytorch by applying it on FashionMNIST data as below:

criterion = nn.CrossEntropyLoss()

for X, y in trn_fashion_dl:
    outputs = my_model(X)
    my_outputs = softmax(outputs)

    my_ce = CrossEntropyLoss(my_outputs, y)
    pytorch_ce = criterion(outputs, y)

    print (f'my custom cross entropy: {my_ce.item()}\npytorch cross entroopy: {pytorch_ce.item()}')
    break

My question is toward the results my_ce (my cross entropy) vs pytorch_ce (pytorch cross entropy) where they are different:

my custom cross entropy: 9.956839561462402
pytorch cross entroopy: 2.378990888595581

I appreciate your help in advance!


title: PyTorch tensor.to(device) for a List of Dict

body:
I am working on an image object detection application using PyTorch torchvision.models.detection.fasterrcnn_resnet50_fpn. As indicated by the documentation, during training phase, the input to fasterrcnn_resnet50_fpn model should be:

- list of image tensors, each of shape [C, H, W]
- list of target dicts, each with:
    - boxes (FloatTensor[N, 4]): the ground-truth boxes in [x1, y1, x2, y2] format,
                                 with values between 0 and H and 0 and W
    - labels (Int64Tensor[N]): the class label for each ground-truth box

I am expecting training in mini-batches, so there should be more than 1 image tensor and 1 target dict in each list.

With the two lists prepared, I can train with my fasterrcnn_resnet50_fpn model:

model.train()
for image_list, target_list in dataset_loader:
    """
    image_list: list of image tensors
    target_list: list of dicts {boxes, labels}
    """
    # some steps before ...

    # Feed inputs to model in training phase
    outputs = model(image_list, target_list)

    # more steps after ...

Everything works well, but there is one issue: by default, the tensor stored in cpu but I would like to train with gpu.

Naively, I can apply iterate through the lists and apply .to(device) for each tensor:

image_list = [ t.to(device) for t in image_list ]
target_list = [ {'boxes':d['boxes'].to(device), 'labels':d['labels']} for d in target_list ]

However, I doubt it would be the most memory/computation efficient, as I am only looping through the lists.

Therefore, my question is: are there better methods to apply tensor.to(device) to list of tensors, or list of dicts, with better memory/computation efficiency, and probably better readability?


title: How to extract and use BERT encodings of sentences for Text similarity among sentences. (PyTorch/Tensorflow)

body:
I want to make a text similarity model which I tend to use for FAQ finding and other methods to get the most related text. I want to use the highly optimised BERT model for this NLP task .I tend to use the the encodings of all the sentences to get a similarity matrix using the cosine_similarity and return results.
In the hypothetical conditions, if I have two sentences as hello world and hello hello world then I am assuming the BRT would give me something like [0.2,0.3,0], (0 for padding) and [0.2,0.2,0.3] and I can pass these two inside the sklearn's cosine_similarity.
How am I supposed to extract the embeddings the sentences to use them in the model? I found somewhere that it can be extracted like:
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertModel.from_pretrained('bert-base-uncased')
input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute")).unsqueeze(0)  # Batch size 1
outputs = model(input_ids)
last_hidden_states = outputs[0]  # The last hidden-state is the first element of the output tuple
Using Tensorflow:

import tensorflow as tf
from transformers import BertTokenizer, TFBertModel

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = TFBertModel.from_pretrained('bert-base-uncased')
input_ids = tf.constant(tokenizer.encode("Hello, my dog is cute"))[None, :]  # Batch size 1
outputs = model(input_ids)
last_hidden_states = outputs[0]  # The last hidden-state is the first element of the output tuple

Is this the right way? Because I read somewhere that there are different type of embeddings that BERT offers.
ALSO please do suggest any other method to find the text similarity


title: Pytorch C++ RuntimeError: Expected object of device type cuda but got device type cpu for argument #1 'self' in call to _th_index_select

body:
I am calculating word similarity using torch::Embedding module by pretrained wordvector (glove.300d) on Ubuntu 18.04LTS PyTorch C++ (1.5.1, CUDA 10.1). I believe I have moved everything I can to the GPU, but when I execute it, it still says (full error log on the end of the question):
Expected object of device type cuda but got device type cpu for
  argument #1 'self' in call to _th_index_select
  (checked_dense_tensor_unwrap at /pytorch/aten/src/ATen/Utils.h:72)

I have checked my model initialization method in main.cpp, and it is okay if I only do initialization.
SimilarityModel simiModel(args, 400000, 300);
simiModel.to(device);

//model forward
torch::Tensor data = ids.index({Slice(i*batch_size, (i+1)*batch_size), Slice()}).to(torch::kInt64).to(device);        //take a batch
tie(score, indice) = simiModel.forward(data);   //forward and transfer score, indice to cpu for further calculation

and this is how I define SimilarityModel in Similarity.h:
class SimilarityModel : public torch::nn::Module {
    public:
        int64_t topk;       // num of top words;
        Dictionary dict;
        int64_t vocab_size;
        int64_t embedding_dim;
        torch::nn::Embedding embedding{nullptr};
        vector<vector<float> > vec_embed;

        SimilarityModel(unordered_map<string, string> args, int64_t vocab_size, int64_t embed_dim);
        tuple<torch::Tensor, torch::Tensor> forward(torch::Tensor x);
};

At the same time I have done embedding initialization in SimilarityModel function in Similarity.cpp:
SimilarityModel::SimilarityModel(unordered_map<string, string> args, int64_t vocab_size, int64_t embed_dim)
        :embedding(vocab_size, embed_dim) {      //Embedding initialize

    this->topk = stoi(args["topk"]);
    vector<vector<float> > pre_embed;
    tie(pre_embed, dict) = loadwordvec(args);       //load pretrained wordvec from txt file

    this->vocab_size = int64_t(dict.size());
    this->embedding_dim = int64_t(pre_embed[0].size());
    this->vec_embed = pre_embed;
    this->dict = dict;

    vector<float> temp_embed;
    for(const auto& i : pre_embed)      //faltten to 1-d
        for(const auto& j : i)
            temp_embed.push_back(j);
    torch::Tensor data = torch::from_blob(temp_embed.data(), {this->vocab_size, this->embedding_dim}, torch::TensorOptions().dtype(torch::kFloat32)).clone();   //vector to tensor
    register_module("embedding", embedding);
    this->embedding = embedding.from_pretrained(data, torch::nn::EmbeddingFromPretrainedOptions().freeze(true));
}

and forward function in Similarity.cpp:
tuple<torch::Tensor, torch::Tensor> SimilarityModel::forward(torch::Tensor x) {

    auto cuda_available = torch::cuda::is_available();      //copy to gpu
    torch::Device device(cuda_available ? torch::kCUDA : torch::kCPU);

    torch::Tensor wordvec;
    wordvec = this->embedding->forward(x).to(device);      //python:embedding(x)
    torch::Tensor similarity_score = wordvec.matmul(this->embedding->weight.transpose(0, 1)).to(device);
    torch::Tensor score, indice;
    tie(score, indice) = similarity_score.topk(this->topk, -1, true, true);        //Tensor.topk(int64_t k, int64_t dim, bool largest = true, bool sorted = true)

    score = score.to(device);
    indice = indice.to(device);
    score.slice(1, 1, score.size(1));       //Tensor.slice(int64_t dim, int64_t start, int64_t end, int64_t step)
    indice.slice(1, 1, indice.size(1));
    return {score.cpu(), indice.cpu()};   //transfer to cpu for further calculation
}

As for intermediate variables in forward() have been put to GPU too. However, I totally have no idea which one is left in CPU, and the error log does not help so much. I have tried the method in Expected object of device type cuda but got device type cpu for argument #1 'self' in call to _th_index_select to do SimilarityModel().to(device), but that does not work. I am still having a hard time reading this error log and would like some instructions on how to debug such questions.
Error log:
terminate called after throwing an instance of 'c10::Error'
  what():  Expected object of device type cuda but got device type cpu for argument #1 'self' in call to _th_index_select (checked_dense_tensor_unwrap at /pytorch/aten/src/ATen/Utils.h:72)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x46 (0x7fb566a27536 in /home/switchsyj/Downloads/libtorch/lib/libc10.so)
frame #1: <unknown function> + 0x101a80b (0x7fb520fa380b in /home/switchsyj/Downloads/libtorch/lib/libtorch_cuda.so)
frame #2: <unknown function> + 0x105009c (0x7fb520fd909c in /home/switchsyj/Downloads/libtorch/lib/libtorch_cuda.so)
frame #3: <unknown function> + 0xf9d76b (0x7fb520f2676b in /home/switchsyj/Downloads/libtorch/lib/libtorch_cuda.so)
frame #4: <unknown function> + 0x10c44e3 (0x7fb558d224e3 in /home/switchsyj/Downloads/libtorch/lib/libtorch_cpu.so)
frame #5: at::native::embedding(at::Tensor const&, at::Tensor const&, long, bool, bool) + 0x2e2 (0x7fb558870712 in /home/switchsyj/Downloads/libtorch/lib/libtorch_cpu.so)
frame #6: <unknown function> + 0x114ef9d (0x7fb558dacf9d in /home/switchsyj/Downloads/libtorch/lib/libtorch_cpu.so)
frame #7: <unknown function> + 0x1187b4d (0x7fb558de5b4d in /home/switchsyj/Downloads/libtorch/lib/libtorch_cpu.so)
frame #8: <unknown function> + 0x2bfe42f (0x7fb55a85c42f in /home/switchsyj/Downloads/libtorch/lib/libtorch_cpu.so)
frame #9: <unknown function> + 0x1187b4d (0x7fb558de5b4d in /home/switchsyj/Downloads/libtorch/lib/libtorch_cpu.so)
frame #10: <unknown function> + 0x32b63a9 (0x7fb55af143a9 in /home/switchsyj/Downloads/libtorch/lib/libtorch_cpu.so)
frame #11: torch::nn::EmbeddingImpl::forward(at::Tensor const&) + 0x71 (0x7fb55af127b1 in /home/switchsyj/Downloads/libtorch/lib/libtorch_cpu.so)
frame #12: SimilarityModel::forward(at::Tensor) + 0xa9 (0x55c96b8e5793 in ./demo)
frame #13: main + 0xaba (0x55c96b8bfe5c in ./demo)
frame #14: __libc_start_main + 0xe7 (0x7fb51edf5b97 in /lib/x86_64-linux-gnu/libc.so.6)
frame #15: _start + 0x2a (0x55c96b8bd74a in ./demo)

Aborted (core dumped)


title: Pytorch equivalent of tf.map_fn with parallel_iterations?

body: I'm executing a PyTorch function for n_iters times on a GPU. Currently I'm using a for loop for this. However, when n_iters is large, this is very inefficient. I'm wondering whether there is a PyTorch equivalent of tf.map_fn with the parallel_iterations functionality so that I can execute all the iterations in parallel?


title: Building wheel for neural-renderer-pytorch (setup.py) ...installing multiperson and neural mesh renderer doesn't work for pytorch 1.6

body:
I am trying to install a github repo named multiperson for PyTorch 1.6 and I get the following error. How can I make it work for PyTorch 1.6?
(base) mona@mona:~/research$ cd phosa/
(base) mona@mona:~/research/phosa$ mkdir -p external
(base) mona@mona:~/research/phosa$ git clone https://github.com/JiangWenPL/multiperson.git external/multiperson
Cloning into 'external/multiperson'...
remote: Enumerating objects: 752, done.
remote: Counting objects: 100% (752/752), done.
remote: Compressing objects: 100% (566/566), done.
remote: Total 752 (delta 189), reused 723 (delta 173), pack-reused 0
Receiving objects: 100% (752/752), 48.29 MiB | 46.26 MiB/s, done.
Resolving deltas: 100% (189/189), done.
(base) mona@mona:~/research/phosa$ pip install external/multiperson/neural_renderer
Processing ./external/multiperson/neural_renderer
Building wheels for collected packages: neural-renderer-pytorch
  Building wheel for neural-renderer-pytorch (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /home/mona/anaconda3/bin/python3.7 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-ma51z6r7/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-ma51z6r7/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-nl9m5bw0
       cwd: /tmp/pip-req-build-ma51z6r7/
  Complete output (210 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.7
  creating build/lib.linux-x86_64-3.7/neural_renderer
  copying neural_renderer/load_obj.py -> build/lib.linux-x86_64-3.7/neural_renderer
  copying neural_renderer/perspective.py -> build/lib.linux-x86_64-3.7/neural_renderer
  copying neural_renderer/vertices_to_faces.py -> build/lib.linux-x86_64-3.7/neural_renderer
  copying neural_renderer/visibility.py -> build/lib.linux-x86_64-3.7/neural_renderer
  copying neural_renderer/get_points_from_angles.py -> build/lib.linux-x86_64-3.7/neural_renderer
  copying neural_renderer/look.py -> build/lib.linux-x86_64-3.7/neural_renderer
  copying neural_renderer/projection.py -> build/lib.linux-x86_64-3.7/neural_renderer
  copying neural_renderer/rasterize.py -> build/lib.linux-x86_64-3.7/neural_renderer
  copying neural_renderer/save_obj.py -> build/lib.linux-x86_64-3.7/neural_renderer
  copying neural_renderer/look_at.py -> build/lib.linux-x86_64-3.7/neural_renderer
  copying neural_renderer/__init__.py -> build/lib.linux-x86_64-3.7/neural_renderer
  copying neural_renderer/lighting.py -> build/lib.linux-x86_64-3.7/neural_renderer
  copying neural_renderer/mesh.py -> build/lib.linux-x86_64-3.7/neural_renderer
  copying neural_renderer/renderer.py -> build/lib.linux-x86_64-3.7/neural_renderer
  creating build/lib.linux-x86_64-3.7/neural_renderer/cuda
  copying neural_renderer/cuda/__init__.py -> build/lib.linux-x86_64-3.7/neural_renderer/cuda
  running build_ext
  building 'neural_renderer.cuda.load_textures' extension
  creating /tmp/pip-req-build-ma51z6r7/build/temp.linux-x86_64-3.7
  creating /tmp/pip-req-build-ma51z6r7/build/temp.linux-x86_64-3.7/neural_renderer
  creating /tmp/pip-req-build-ma51z6r7/build/temp.linux-x86_64-3.7/neural_renderer/cuda
  Emitting ninja build file /tmp/pip-req-build-ma51z6r7/build/temp.linux-x86_64-3.7/build.ninja...
  Compiling objects...
  Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
  [1/2] /usr/bin/nvcc -I/home/mona/anaconda3/lib/python3.7/site-packages/torch/include -I/home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/home/mona/anaconda3/lib/python3.7/site-packages/torch/include/TH -I/home/mona/anaconda3/lib/python3.7/site-packages/torch/include/THC -I/home/mona/anaconda3/include/python3.7m -c -c /tmp/pip-req-build-ma51z6r7/neural_renderer/cuda/load_textures_cuda_kernel.cu -o /tmp/pip-req-

    copying neural_renderer/look_at.py -> build/lib.linux-x86_64-3.7/neural_renderer
    copying neural_renderer/__init__.py -> build/lib.linux-x86_64-3.7/neural_renderer
    copying neural_renderer/lighting.py -> build/lib.linux-x86_64-3.7/neural_renderer
    copying neural_renderer/mesh.py -> build/lib.linux-x86_64-3.7/neural_renderer
    copying neural_renderer/renderer.py -> build/lib.linux-x86_64-3.7/neural_renderer
    creating build/lib.linux-x86_64-3.7/neural_renderer/cuda
    copying neural_renderer/cuda/__init__.py -> build/lib.linux-x86_64-3.7/neural_renderer/cuda
    running build_ext
    building 'neural_renderer.cuda.load_textures' extension
    creating /tmp/pip-req-build-ma51z6r7/build/temp.linux-x86_64-3.7
    creating /tmp/pip-req-build-ma51z6r7/build/temp.linux-x86_64-3.7/neural_renderer
    creating /tmp/pip-req-build-ma51z6r7/build/temp.linux-x86_64-3.7/neural_renderer/cuda
    Emitting ninja build file /tmp/pip-req-build-ma51z6r7/build/temp.linux-x86_64-3.7/build.ninja...
    Compiling objects...
    Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
    [1/2] c++ -MMD -MF /tmp/pip-req-build-ma51z6r7/build/temp.linux-x86_64-3.7/neural_renderer/cuda/load_textures_cuda.o.d -pthread -B /home/mona/anaconda3/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/mona/anaconda3/lib/python3.7/site-packages/torch/include -I/home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/home/mona/anaconda3/lib/python3.7/site-packages/torch/include/TH -I/home/mona/anaconda3/lib/python3.7/site-packages/torch/include/THC -I/home/mona/anaconda3/include/python3.7m -c -c /tmp/pip-req-build-ma51z6r7/neural_renderer/cuda/load_textures_cuda.cpp -o /tmp/pip-req-build-ma51z6r7/build/temp.linux-x86_64-3.7/neural_renderer/cuda/load_textures_cuda.o -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=load_textures -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
    FAILED: /tmp/pip-req-build-ma51z6r7/build/temp.linux-x86_64-3.7/neural_renderer/cuda/load_textures_cuda.o
    c++ -MMD -MF /tmp/pip-req-build-ma51z6r7/build/temp.linux-x86_64-3.7/neural_renderer/cuda/load_textures_cuda.o.d -pthread -B /home/mona/anaconda3/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/mona/anaconda3/lib/python3.7/site-packages/torch/include -I/home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/home/mona/anaconda3/lib/python3.7/site-packages/torch/include/TH -I/home/mona/anaconda3/lib/python3.7/site-packages/torch/include/THC -I/home/mona/anaconda3/include/python3.7m -c -c /tmp/pip-req-build-ma51z6r7/neural_renderer/cuda/load_textures_cuda.cpp -o /tmp/pip-req-build-ma51z6r7/build/temp.linux-x86_64-3.7/neural_renderer/cuda/load_textures_cuda.o -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=load_textures -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
    cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
    In file included from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/ATen/Parallel.h:149,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/utils.h:3,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/nn/cloneable.h:5,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/nn.h:3,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/all.h:7,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/torch.h:3,
                     from /tmp/pip-req-build-ma51z6r7/neural_renderer/cuda/load_textures_cuda.cpp:1:
    /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/ATen/ParallelOpenMP.h:84: warning: ignoring #pragma omp parallel [-Wunknown-pragmas]
       84 | #pragma omp parallel for if ((end - begin) >= grain_size)
          |
    /tmp/pip-req-build-ma51z6r7/neural_renderer/cuda/load_textures_cuda.cpp: In function ‘at::Tensor load_textures(at::Tensor, at::Tensor, at::Tensor, at::Tensor, int, int)’:
    /tmp/pip-req-build-ma51z6r7/neural_renderer/cuda/load_textures_cuda.cpp:15:39: warning: ‘at::DeprecatedTypeProperties& at::Tensor::type() const’ is deprecated: Tensor.type() is deprecated. Instead use Tensor.options(), which in many cases (e.g. in a constructor) is a drop-in replacement. If you were using data from type(), that is now available from Tensor itself, so instead of tensor.type().scalar_type(), use tensor.scalar_type() instead and instead of tensor.type().backend() use tensor.device(). [-Wdeprecated-declarations]
       15 | #define CHECK_CUDA(x) AT_CHECK(x.type().is_cuda(), #x " must be a CUDA tensor")
          |                                       ^
    /tmp/pip-req-build-ma51z6r7/neural_renderer/cuda/load_textures_cuda.cpp:17:24: note: in expansion of macro ‘CHECK_CUDA’
       17 | #define CHECK_INPUT(x) CHECK_CUDA(x); CHECK_CONTIGUOUS(x)
          |                        ^~~~~~~~~~
    /tmp/pip-req-build-ma51z6r7/neural_renderer/cuda/load_textures_cuda.cpp:28:5: note: in expansion of macro ‘CHECK_INPUT’
       28 |     CHECK_INPUT(image);
          |     ^~~~~~~~~~~
    In file included from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/ATen/Tensor.h:3,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/ATen/Context.h:4,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/ATen/ATen.h:5,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/types.h:3,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader_options.h:4,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/base.h:3,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/stateful.h:3,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader.h:3,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data.h:3,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/all.h:4,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/torch.h:3,
                     from /tmp/pip-req-build-ma51z6r7/neural_renderer/cuda/load_textures_cuda.cpp:1:
    /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/ATen/core/TensorBody.h:268:30: note: declared here
      268 |   DeprecatedTypeProperties & type() const {
          |                              ^~~~
    /tmp/pip-req-build-ma51z6r7/neural_renderer/cuda/load_textures_cuda.cpp:15:23: error: ‘AT_CHECK’ was not declared in this scope; did you mean ‘DCHECK’?
       15 | #define CHECK_CUDA(x) AT_CHECK(x.type().is_cuda(), #x " must be a CUDA tensor")
          |                       ^~~~~~~~
    /tmp/pip-req-build-ma51z6r7/neural_renderer/cuda/load_textures_cuda.cpp:17:24: note: in expansion of macro ‘CHECK_CUDA’
       17 | #define CHECK_INPUT(x) CHECK_CUDA(x); CHECK_CONTIGUOUS(x)
          |                        ^~~~~~~~~~
    /tmp/pip-req-build-ma51z6r7/neural_renderer/cuda/load_textures_cuda.cpp:28:5: note: in expansion of macro ‘CHECK_INPUT’
       28 |     CHECK_INPUT(image);
          |     ^~~~~~~~~~~
    /tmp/pip-req-build-ma51z6r7/neural_renderer/cuda/load_textures_cuda.cpp:15:39: warning: ‘at::DeprecatedTypeProperties& at::Tensor::type() const’ is deprecated: Tensor.type() is deprecated. Instead use Tensor.options(), which in many cases (e.g. in a constructor) is a drop-in replacement. If you were using data from type(), that is now available from Tensor itself, so instead of tensor.type().scalar_type(), use tensor.scalar_type() instead and instead of tensor.type().backend() use tensor.device(). [-Wdeprecated-declarations]
       15 | #define CHECK_CUDA(x) AT_CHECK(x.type().is_cuda(), #x " must be a CUDA tensor")
          |                                       ^
    /tmp/pip-req-build-ma51z6r7/neural_renderer/cuda/load_textures_cuda.cpp:17:24: note: in expansion of macro ‘CHECK_CUDA’
       17 | #define CHECK_INPUT(x) CHECK_CUDA(x); CHECK_CONTIGUOUS(x)
          |                        ^~~~~~~~~~
    /tmp/pip-req-build-ma51z6r7/neural_renderer/cuda/load_textures_cuda.cpp:29:5: note: in expansion of macro ‘CHECK_INPUT’
       29 |     CHECK_INPUT(faces);
          |     ^~~~~~~~~~~
    In file included from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/ATen/Tensor.h:3,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/ATen/Context.h:4,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/ATen/ATen.h:5,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/types.h:3,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader_options.h:4,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/base.h:3,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/stateful.h:3,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader.h:3,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data.h:3,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/all.h:4,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/torch.h:3,
                     from /tmp/pip-req-build-ma51z6r7/neural_renderer/cuda/load_textures_cuda.cpp:1:
    /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/ATen/core/TensorBody.h:268:30: note: declared here
      268 |   DeprecatedTypeProperties & type() const {
          |                              ^~~~
    /tmp/pip-req-build-ma51z6r7/neural_renderer/cuda/load_textures_cuda.cpp:15:39: warning: ‘at::DeprecatedTypeProperties& at::Tensor::type() const’ is deprecated: Tensor.type() is deprecated. Instead use Tensor.options(), which in many cases (e.g. in a constructor) is a drop-in replacement. If you were using data from type(), that is now available from Tensor itself, so instead of tensor.type().scalar_type(), use tensor.scalar_type() instead and instead of tensor.type().backend() use tensor.device(). [-Wdeprecated-declarations]
       15 | #define CHECK_CUDA(x) AT_CHECK(x.type().is_cuda(), #x " must be a CUDA tensor")
          |                                       ^
    /tmp/pip-req-build-ma51z6r7/neural_renderer/cuda/load_textures_cuda.cpp:17:24: note: in expansion of macro ‘CHECK_CUDA’
       17 | #define CHECK_INPUT(x) CHECK_CUDA(x); CHECK_CONTIGUOUS(x)
          |                        ^~~~~~~~~~
    /tmp/pip-req-build-ma51z6r7/neural_renderer/cuda/load_textures_cuda.cpp:30:5: note: in expansion of macro ‘CHECK_INPUT’
       30 |     CHECK_INPUT(is_update);
          |     ^~~~~~~~~~~
    In file included from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/ATen/Tensor.h:3,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/ATen/Context.h:4,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/ATen/ATen.h:5,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/types.h:3,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader_options.h:4,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/base.h:3,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/stateful.h:3,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader.h:3,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data.h:3,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/all.h:4,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/torch.h:3,
                     from /tmp/pip-req-build-ma51z6r7/neural_renderer/cuda/load_textures_cuda.cpp:1:
    /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/ATen/core/TensorBody.h:268:30: note: declared here
      268 |   DeprecatedTypeProperties & type() const {
          |                              ^~~~
    /tmp/pip-req-build-ma51z6r7/neural_renderer/cuda/load_textures_cuda.cpp:15:39: warning: ‘at::DeprecatedTypeProperties& at::Tensor::type() const’ is deprecated: Tensor.type() is deprecated. Instead use Tensor.options(), which in many cases (e.g. in a constructor) is a drop-in replacement. If you were using data from type(), that is now available from Tensor itself, so instead of tensor.type().scalar_type(), use tensor.scalar_type() instead and instead of tensor.type().backend() use tensor.device(). [-Wdeprecated-declarations]
       15 | #define CHECK_CUDA(x) AT_CHECK(x.type().is_cuda(), #x " must be a CUDA tensor")
          |                                       ^
    /tmp/pip-req-build-ma51z6r7/neural_renderer/cuda/load_textures_cuda.cpp:17:24: note: in expansion of macro ‘CHECK_CUDA’
       17 | #define CHECK_INPUT(x) CHECK_CUDA(x); CHECK_CONTIGUOUS(x)
          |                        ^~~~~~~~~~
    /tmp/pip-req-build-ma51z6r7/neural_renderer/cuda/load_textures_cuda.cpp:31:5: note: in expansion of macro ‘CHECK_INPUT’
       31 |     CHECK_INPUT(textures);
          |     ^~~~~~~~~~~
    In file included from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/ATen/Tensor.h:3,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/ATen/Context.h:4,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/ATen/ATen.h:5,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/types.h:3,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader_options.h:4,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/base.h:3,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/stateful.h:3,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader.h:3,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data.h:3,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/all.h:4,
                     from /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/torch.h:3,
                     from /tmp/pip-req-build-ma51z6r7/neural_renderer/cuda/load_textures_cuda.cpp:1:
    /home/mona/anaconda3/lib/python3.7/site-packages/torch/include/ATen/core/TensorBody.h:268:30: note: declared here
      268 |   DeprecatedTypeProperties & type() const {
          |                              ^~~~
    [2/2] /usr/bin/nvcc -I/home/mona/anaconda3/lib/python3.7/site-packages/torch/include -I/home/mona/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/home/mona/anaconda3/lib/python3.7/site-packages/torch/include/TH -I/home/mona/anaconda3/lib/python3.7/site-packages/torch/include/THC -I/home/mona/anaconda3/include/python3.7m -c -c /tmp/pip-req-build-ma51z6r7/neural_renderer/cuda/load_textures_cuda_kernel.cu -o /tmp/pip-req-build-ma51z6r7/build/temp.linux-x86_64-3.7/neural_renderer/cuda/load_textures_cuda_kernel.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=load_textures -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_75,code=sm_75 -std=c++14
    ninja: build stopped: subcommand failed.
    Traceback (most recent call last):
      File "/home/mona/anaconda3/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1515, in _run_ninja_build
        env=env)
      File "/home/mona/anaconda3/lib/python3.7/subprocess.py", line 512, in run
        output=stdout, stderr=stderr)
    subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-req-build-ma51z6r7/setup.py", line 40, in <module>
        cmdclass = {'build_ext': BuildExtension}
      File "/home/mona/anaconda3/lib/python3.7/site-packages/setuptools/__init__.py", line 163, in setup
        return distutils.core.setup(**attrs)
      File "/home/mona/anaconda3/lib/python3.7/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/home/mona/anaconda3/lib/python3.7/distutils/dist.py", line 966, in run_commands
        self.run_command(cmd)
      File "/home/mona/anaconda3/lib/python3.7/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/home/mona/anaconda3/lib/python3.7/site-packages/setuptools/command/install.py", line 61, in run
        return orig.install.run(self)
      File "/home/mona/anaconda3/lib/python3.7/distutils/command/install.py", line 545, in run
        self.run_command('build')
      File "/home/mona/anaconda3/lib/python3.7/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/home/mona/anaconda3/lib/python3.7/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/home/mona/anaconda3/lib/python3.7/distutils/command/build.py", line 135, in run
        self.run_command(cmd_name)
      File "/home/mona/anaconda3/lib/python3.7/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/home/mona/anaconda3/lib/python3.7/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/home/mona/anaconda3/lib/python3.7/site-packages/setuptools/command/build_ext.py", line 87, in run
        _build_ext.run(self)
      File "/home/mona/anaconda3/lib/python3.7/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run
        _build_ext.build_ext.run(self)
      File "/home/mona/anaconda3/lib/python3.7/distutils/command/build_ext.py", line 340, in run
        self.build_extensions()
      File "/home/mona/anaconda3/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 649, in build_extensions
        build_ext.build_extensions(self)
      File "/home/mona/anaconda3/lib/python3.7/site-packages/Cython/Distutils/old_build_ext.py", line 195, in build_extensions
        _build_ext.build_ext.build_extensions(self)
      File "/home/mona/anaconda3/lib/python3.7/distutils/command/build_ext.py", line 449, in build_extensions
        self._build_extensions_serial()
      File "/home/mona/anaconda3/lib/python3.7/distutils/command/build_ext.py", line 474, in _build_extensions_serial
        self.build_extension(ext)
      File "/home/mona/anaconda3/lib/python3.7/site-packages/setuptools/command/build_ext.py", line 208, in build_extension
        _build_ext.build_extension(self, ext)
      File "/home/mona/anaconda3/lib/python3.7/distutils/command/build_ext.py", line 534, in build_extension
        depends=ext.depends)
      File "/home/mona/anaconda3/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 478, in unix_wrap_ninja_compile
        with_cuda=with_cuda)
      File "/home/mona/anaconda3/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1233, in _write_ninja_file_and_compile_objects
        error_prefix='Error compiling objects for extension')
      File "/home/mona/anaconda3/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1529, in _run_ninja_build
        raise RuntimeError(message)
    RuntimeError: Error compiling objects for extension
    ----------------------------------------
  Rolling back uninstall of neural-renderer-pytorch
  Moving to /home/mona/anaconda3/lib/python3.7/site-packages/neural_renderer/
   from /home/mona/anaconda3/lib/python3.7/site-packages/~eural_renderer
  Moving to /home/mona/anaconda3/lib/python3.7/site-packages/neural_renderer_pytorch-1.1.3.dist-info/
   from /home/mona/anaconda3/lib/python3.7/site-packages/~eural_renderer_pytorch-1.1.3.dist-info
ERROR: Command errored out with exit status 1: /home/mona/anaconda3/bin/python3.7 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-ma51z6r7/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-ma51z6r7/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-ul8nk1jn/install-record.txt --single-version-externally-managed --compile --install-headers /home/mona/anaconda3/include/python3.7m/neural-renderer-pytorch Check the logs for full command output.

I have:
$ python
Python 3.7.6 (default, Jan  8 2020, 19:59:22)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.__version__
'1.6.0'
$ lsb_release -a
LSB Version:    core-11.1.0ubuntu2-noarch:security-11.1.0ubuntu2-noarch
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.1 LTS
Release:    20.04
Codename:   focal

$ nvidia-smi
Sun Dec  6 16:36:36 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.80.02    Driver Version: 450.80.02    CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce RTX 2070    Off  | 00000000:01:00.0 Off |                  N/A |
| N/A   49C    P8    21W /  N/A |   1546MiB /  7982MiB |      8%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243


title: Cannot manually assign new parameters via layer name

body:
I'm trying to manually assign new weights to my pytorch model. I can assign new weights like this:

import scipy.io as sio
import torch

caffe_params = sio.loadmat('export_conv1_1.mat')
net.conv1_1.weight = torch.nn.Parameter(torch.from_numpy(caffe_params['w']))
net.conv1_1.bias = torch.nn.Parameter(torch.from_numpy(caffe_params['b']))

caffe_params = sio.loadmat('export_conv2_1.mat')
net.conv2_1.weight = torch.nn.Parameter(torch.from_numpy(caffe_params['w']))
net.conv2_1.bias = torch.nn.Parameter(torch.from_numpy(caffe_params['b']))

Since I have a lot of layers I don't want to manually assign every layer via it's name. Instead I want rather to loop over a list of layer names and assign them automatically. Something like this:

varList = ['conv2_1','conv2_2']

for name in varList:
    caffe_params = sio.loadmat(rootDir + 'export_' + name + '.mat')
    setattr(net, name + '.weight' ,torch.nn.Parameter(torch.from_numpy(caffe_params['w'])))
    setattr(net, name + '.bias' ,torch.nn.Parameter(torch.from_numpy(caffe_params['b'])))

Unfortunately this doesn't work. I guess setattr doesn't work with either pytorch weigths or with attributes of type layername.weight, which means the attribute to assign has depth 2 with respect to net.


title: What dtype should I use for PyTorch parameters in a neural network that inputs and outputs arrays of integers?

body:
I'm currently building a neural network in PyTorch that accepts tensors of integers and outputs tensors of integers.  There is only a small number of positive integers that are "allowed"  (like 0, 1, 2, 3, and 4) as elements of the input and output tensors.

Neural networks usually work in continuous space.
For example, the nonlinear activation functions between layers are continuous and map integers to real numbers (including non-integers).

Is it best to use unsigned integers like torch.uint8 internally for the weights and biases of the network plus some custom activation function that maps ints to ints?

Or should I use high precision floats like torch.float32 and then round in the end, by binning real numbers to the nearest integer?  I think this second strategy is the way to go, but maybe I'm missing out on something that would work nicely.


title: How can I have a PyTorch Conv1d work over a vector?

body:
I understand Conv1d strides in one dimension. But my input is of shape [64, 20, 161], where 64 is the batches, 20 is the sequence length and 161 is the dimension of my vector.

I'm not sure how to set up my Conv1d to stride over the vector.

I'm trying:

self.conv1 = torch.nn.Conv1d(batch_size, 20, 161, stride=1)

but getting:

RuntimeError: Given groups=1, weight of size 20 64 161, expected input[64, 20, 161] to have 64 channels, but got 20 channels instead


title: How to define ration of summary with hugging face transformers pipeline?

body:
I am using the following code to summarize an article from using huggingface-transformer's pipeline. Using this code:

from transformers import pipeline
summarizer = pipeline(task="summarization" )
summary = summarizer(text)
print(summary[0]['summary_text'])

How can I define a ratio between the summary and the original article? For example, 20% of the original article?

EDIT 1: I implemented the solution you suggested, but got the following error. This is the code I used:

summarizer(text, min_length = int(0.1 * len(text)), max_length = int(0.2 * len(text)))
print(summary[0]['summary_text'])

The error I got:

RuntimeError                              Traceback (most recent call last)
<ipython-input-9-bc11c5d8eb66> in <module>()
----> 1 summarizer(text, min_length = int(0.1 * len(text)), max_length = int(0.2 * len(text)))
      2 print(summary[0]['summary_text'])

13 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
   1482         # remove once script supports set_grad_enabled
   1483         _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-> 1484     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
   1485
   1486

RuntimeError: index out of range: Tried to access index 1026 out of table with 1025 rows. at /pytorch/aten/src/TH/generic/THTensorEvenMoreMath.cpp:418


title: How to use PyTorch’s DataLoader together with skorch’s GridSearchCV

body:
I am running a PyTorch ANN model (for a classification task) and I am using skorch’s GridSearchCV to search for the optimal hyperparameters.
When I ran GridSearchCV using n_jobs=1 (ie. doing one hyperparameter combination at a time), it runs really slowly.
When I set n_jobs to greater than 1, I get a memory blow-out error. So I am now trying to see if I could use PyTorch’s DataLoader to split up the dataset into batches to avoid the memory blow-out issue. According to this other PyTorch Forum question (https://discuss.pytorch.org/t/how-to-use-skorch-for-data-that-does-not-fit-into-memory/70081/2), it appears we could use SliceDataset. My code for this is as below:
# Setting up artifical neural net model
class TabularModel(nn.Module):
    # Initialize parameters embeds, emb_drop, bn_cont and layers
    def __init__(self, emb_szs, n_cont, out_sz, layers, p=0.5):
        super().__init__()
        self.embeds = nn.ModuleList([nn.Embedding(ni, nf) for ni, nf in emb_szs])
        self.emb_drop = nn.Dropout(p)
        self.bn_cont = nn.BatchNorm1d(n_cont)

        # Create empty list for each layer in the neural net
        layerlist = []
        # Number of all embedded columns for categorical features
        n_emb = sum((nf for ni, nf in emb_szs))
        # Number of inputs for each layer
        n_in = n_emb + n_cont

        for i in layers:
            # Set the linear function for the weights and biases, wX + b
            layerlist.append(nn.Linear(n_in, i))
            # Using ReLu activation function
            layerlist.append(nn.ReLU(inplace=True))
            # Normalised all the activation function output values
            layerlist.append(nn.BatchNorm1d(i))
            # Set some of the normalised activation function output values to zero
            layerlist.append(nn.Dropout(p))
            # Reassign number of inputs for the next layer
            n_in = i
        # Append last layer
        layerlist.append(nn.Linear(layers[-1], out_sz))
        # Create sequential layers
        self.layers = nn.Sequential(*layerlist)

    # Function for feedforward
    def forward(self, x_cat_cont):
        x_cat = x_cat_cont[:,0:cat_train.shape[1]].type(torch.int64)
        x_cont = x_cat_cont[:,cat_train.shape[1]:].type(torch.float32)

        # Create empty list for embedded categorical features
        embeddings = []
        # Embed categorical features
        for i, e in enumerate(self.embeds):
            embeddings.append(e(x_cat[:,i]))
        # Concatenate embedded categorical features
        x = torch.cat(embeddings, 1)
        # Apply dropout rates to categorical features
        x = self.emb_drop(x)

        # Batch normalize continuous features
        x_cont = self.bn_cont(x_cont)

        # Concatenate categorical and continuous features
        x = torch.cat([x, x_cont], 1)

        # Feed categorical and continuous features into neural net layers
        x = self.layers(x)
        return x

# Use cross entropy loss function since this is a classification problem
# Assign class weights to the loss function
criterion_skorch = nn.CrossEntropyLoss
# Use Adam solver with learning rate 0.001
optimizer_skorch = torch.optim.Adam

from skorch import NeuralNetClassifier

# Random seed chosen to ensure results are reproducible by using the same initial random weights and biases,
# and applying dropout rates to the same random embedded categorical features and neurons in the hidden layers
torch.manual_seed(0)

net = NeuralNetClassifier(module=TabularModel,
                          module__emb_szs=emb_szs,
                          module__n_cont=con_train.shape[1],
                          module__out_sz=2,
                          module__layers=[30],
                          module__p=0.0,
                          criterion=criterion_skorch,
                          criterion__weight=cls_wgt,
                          optimizer=optimizer_skorch,
                          optimizer__lr=0.001,
                          max_epochs=150,
                          device='cuda'
                          )

from sklearn.model_selection import GridSearchCV

param_grid = {'module__layers': [[30], [50,20]],
              'module__p': [0.0],
              'max_epochs': [150, 175]
             }

from torch.utils.data import TensorDataset, DataLoader
from skorch.helper import SliceDataset

# cat_con_train and y_train is a PyTorch tensor
tsr_ds = TensorDataset(cat_con_train.cpu(), y_train.cpu())

torch.manual_seed(0)   # Set random seed for shuffling results to be reproducible
d_loader = DataLoader(tsr_ds, batch_size=100000, shuffle=True)

d_loader_slice_X = SliceDataset(d_loader, idx=0)
d_loader_slice_y = SliceDataset(d_loader, idx=1)

models = GridSearchCV(net, param_grid, scoring='roc_auc', n_jobs=2).fit(d_loader_slice_X, d_loader_slice_y)

However, when I ran this code, I get the following error message:
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-47-df3fc792ad5e> in <module>()
    104
--> 105 models = GridSearchCV(net, param_grid, scoring='roc_auc', n_jobs=2).fit(d_loader_slice_X, d_loader_slice_y)
    106

6 frames
/usr/local/lib/python3.6/dist-packages/skorch/helper.py in __getitem__(self, i)
    230     def __getitem__(self, i):
    231         if isinstance(i, (int, np.integer)):
--> 232             Xn = self.dataset[self.indices_[i]]
    233             Xi = self._select_item(Xn)
    234             return self.transform(Xi)

TypeError: 'DataLoader' object does not support indexing

How do I fix this? Is there a way to use PyTorch’s DataLoader together with skorch’s GridSearchCV (ie. is there a way to load data in batches into skorch’s GridSearchCV, to avoid memory blow-out issues when I set n_jobs to greater than 1 in GridSearchCV)?
Many many thanks in advance!


title: Pytorch - Purpose of images preprocessing in the transfer learning tutorial

body:
In the Pytorch transfer learning tutorial, the images in both the training and the test sets are being pre-processed using the following code:

data_transforms = {
'train': transforms.Compose([
    transforms.RandomResizedCrop(224),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(),
    transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
]),
'val': transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
]),
}

My question is - what is the intuition behind this choice of transforms? In particular, what is the intuition behind choosing RandomResizedCrop(224) and RandomHorizontalFlip()? Wouldn't it be better to just let the neural network train on the entire image? (or at least, augment the dataset using these transformation)? I understand why it is reasonable to insert only the portion of the image that contains the ant/bees to the neural network but can't understand why it is reasonable to insert a random crop...

Hope I managed to make all my questions clear

Thanks!


title: Getting distributed package doesn't have mpi built in error

body:
I have been trying to write a distributed application using pytorch. I have been following tutorial here. Over there, I am using the "MPI Backend" option. According to that, I need to follow the basic steps to install pytorch and then install openmpi as conda install -c conda-forge openmpi

Unfortunately, whenever I try to run a script using mpirun mpiexec -n 2 python ptdist.py, I get the following error RuntimeError: Distributed package doesn't have MPI built in. I believe this is happening because of error in import ProcessGroupMPI code here in python.

I have tried to install openmpi from their source code as well as sudo apt-get install python-mpi4py, but am still facing the same error.

I also tried pip install mpi4py but that also does not help

Does anyone know what is the problem?


title: Clip or threshold a tensor using condition and zero pad the result in PyTorch

body:
let's say I have a tensor like this

w = [[0.1, 0.7, 0.7, 0.8, 0.3],
    [0.3, 0.2, 0.9, 0.1, 0.5],
    [0.1, 0.4, 0.8, 0.3, 0.4]]

Now I want to eliminate certain values base on some condition (for example greater than 0.5 or not)

w = [[0.1, 0.3],
     [0.3, 0.2, 0.1],
     [0.1, 0.4, 0.3, 0.4]]

Then pad it to equal length:

w = [[0.1, 0.3, 0, 0],
     [0.3, 0.2, 0.1, 0],
     [0.1, 0.4, 0.3, 0.4]]

and this is how I implemented it in pytorch:

w = torch.rand(3, 5)
condition = w <= 0.5
w = [w[i][condition[i]] for i in range(3)]
w = torch.nn.utils.rnn.pad_sequence(w)

But apparently this is going to be extremely slow, mainly because of the list comprehension.
is there any better way to do it?


title: How to compile torch 1.5.0 without GPU support?

body:
I want to install pytorch 1.5.0 on AWS lambda. Since the torch library is very large, I need to make it as small as possible to fit within the size limits. My script looks like this so far:

    mkdir python
    docker run \
        --rm \
        -v $(pwd):/build \
        python:3.8 \
        sh -c "
            cd /build;
            pip3 install torch==1.5.0 -t python/torch --no-cache-dir;
            find . -type d -name '__pycache__' | xargs rm -rf;
            find . -type d -name 'tests' | xargs rm -rf;
            find . -type f -name '*.py[co]' | xargs rm -rf;
        ";

    zip -r9 torch.zip python;

But the resulting zip file is very large (500+ MB). However, one of the largest files in the installation package is libtorch_cuda.so. Removing that file makes the zip file less than half the size. I know that cuda is a library for GPUs, and since AWS lambda doesn't have a GPU, i don't need this support. But when I remove that file torch will not import correctly.

torch 1.4.0, by comparison, is much smaller because it did not by default include the cuda libraries.

I want torch 1.5.0 without the GPU support.

Is there a way to pip install torch==1.5.0 without gpu support?


title: PyTorch Pretrained VGG19 KeyError

body:
I'm working on fine tuning the first 10 layers of VGG19 net to extract features from images. But I'm getting the below error which I couldn't find a get around:

Traceback (most recent call last):
  File "TODO-train_from_scratch.py", line 390, in <module>
    main()
  File "TODO-train_from_scratch.py", line 199, in main
    model.load_state_dict(weights_load)
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py", line 339, in load_state_dict
    raise KeyError('missing keys in state_dict: "{}"'.format(missing))

Corresponding snippet from training code is:

# create model
vgg19 = models.vgg19(pretrained = True)

vgg19_state_dict = vgg19.state_dict()

vgg19_keys = vgg19_state_dict.keys()

model = get_model()

weights_load = {}

for i in range(20):
    weights_load[model.state_dict().keys()[i]] = vgg19_state_dict[vgg19_keys[i]]

model.load_state_dict(weights_load)
model = torch.nn.DataParallel(model).cuda()


title: Adding parameters to a pretrained model

body:
In Pytorch, we load the pretrained model as follows:

net.load_state_dict(torch.load(path)['model_state_dict'])

Then the network structure and the loaded model have to be exactly the same. However, is it possible to load the weights but then modify the network/add an extra parameter?

Note:
If we add an extra parameter to the model earlier before loading the weights, e.g.

self.parameter = Parameter(torch.ones(5),requires_grad=True)

we will get Missing key(s) in state_dict: error when loading the weights.


title: How to use the lbfgs optimizer with pytorch-lightning?

body:
I have a problem in using the LBFGS optimizer from pytorch with lightning.
I use the template from here to start a new project and here is the code that I tried (only the training portion):

def training_step(self, batch, batch_nb):
    x, y = batch
    x = x.float()
    y = y.float()
    y_hat = self.forward(x)
    return {'loss': F.mse_loss(y_hat, y)}

def configure_optimizers(self):
    optimizer = torch.optim.LBFGS(self.parameters())
    return optimizer

def optimizer_step(self, epoch_nb, batch_nb, optimizer, optimizer_i):
    def closure():
        optimizer.zero_grad()
        l = self.training_step(batch, batch_nb)
        loss = l['loss']
        loss.backward()
        return loss

    optimizer.step(closure)

The LBFGS optimizer from pytorch requires a closure function (see here and here), but I don't know how to define it inside the template, specially I don't know how the batch data is passed to the optimizer. I tried to define a custom optimizer_step function but I have some problems to passing the batch inside the closure function.

I will be very thankful of any advise that helps me to solve this problem or points me to the right direction.

Environment:

PyTorch version:  1.2.0+cpu
Lightning version: 0.4.9
Test-tube version: 0.7.1


title: Extracting attention weights of each token at each layer of transformer in python

body:
I am doing some NLP and I am interested in extracting attention weights of individual test token at each layer of transformer via Python (PyTorch, TensorFlow, etc.)

Is coding up a Transformer (any transformers like Transformer-XL, OpenAL-GPT, GPT2 ,etc.) from the scratch the only way to get attention weights of individual test token at each transformer layer? Is there easier way to perform this task in Python? More specifically, can Keras-transformer be used for this purpose? If someone can provide me with some example code, it will be great!

Thank you,


title: How to use custom torch.autograd.Function in nn.Sequential model

body:
Is there any way that I can use custom torch.autograd.Function in a nn.Sequential object or should I use explicitly an nn.Module object with forward function. Specifically I am trying to implement a sparse autoencoder and I need to add L1 distance of the code(hidden representation) to the loss.
I have defined custom torch.autograd.Function L1Penalty below then tried to use it inside a nn.Sequential object as below. However when I run I got the error TypeError: __main__.L1Penalty is not a Module subclass How can I solve this issue?

class L1Penalty(torch.autograd.Function):
    @staticmethod
    def forward(ctx, input, l1weight = 0.1):
        ctx.save_for_backward(input)
        ctx.l1weight = l1weight
        return input, None

    @staticmethod
    def backward(ctx, grad_output):
        input, = ctx.saved_variables
        grad_input = input.clone().sign().mul(ctx.l1weight)
        grad_input+=grad_output
        return grad_input

model = nn.Sequential(
    nn.Linear(10, 10),
    nn.ReLU(),
    nn.Linear(10, 6),
    nn.ReLU(),
    # sparsity
    L1Penalty(),
    nn.Linear(6, 10),
    nn.ReLU(),
    nn.Linear(10, 10),
    nn.ReLU()
).to(device)