I'm trying to import this :

import torchvision.models.video as video

But it throws:

  ModuleNotFoundError: No module named 'torchvision.models.video'

I can import other models without any problems like :

import torchvision.models as models

resnet = models.resnet18()

Any idea how this can happen?


title: Calculate covariance of torch tensor (2d feature map)

body:
I have a torch tensor with shape (batch_size, number_maps, x_val, y_val). The tensor is normalized with a sigmoid function, so within range [0, 1]. I want to find the covariance for each map, so I want to have a tensor with shape (batch_size, number_maps, 2, 2). As far as I know, there is no torch.cov() function as in numpy. How can I efficiently calculate the covariance without converting it to numpy?
Edit:
def get_covariance(tensor):
bn, nk, w, h = tensor.shape
tensor_reshape = tensor.reshape(bn, nk, 2, -1)
x = tensor_reshape[:, :, 0, :]
y = tensor_reshape[:, :, 1, :]
mean_x = torch.mean(x, dim=2).unsqueeze(-1)
mean_y = torch.mean(y, dim=2).unsqueeze(-1)

xx = torch.sum((x - mean_x) * (x - mean_x), dim=2).unsqueeze(-1) / (h*w - 1)
xy = torch.sum((x - mean_x) * (y - mean_y), dim=2).unsqueeze(-1) / (h*w - 1)
yx = xy
yy = torch.sum((y - mean_y) * (y - mean_y), dim=2).unsqueeze(-1) / (h*w - 1)

cov = torch.cat((xx, xy, yx, yy), dim=2)
cov = cov.reshape(bn, nk, 2, 2)

return cov

I tried the following now, but I m pretty sure it's not correct.


title: How to use a different test batch size for RNN in PyTorch?

body:
I want to train an RNN over 5 training points where each sequence also has a size of 5. At test time, I want to send in a single data point and compute the output.

The task is to predict the next character in a sequence of five characters (all encoded as 1-hot vectors). I have tried duplicating the test data point five times. However, I am sure that this is not the right way to solve this problem.

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F

# Define the parameters
H = [ 1, 0, 0, 0 ]
E = [ 0, 1, 0, 0 ]
L = [ 0, 0, 1, 0 ]
O = [ 0, 0, 0, 1 ]

# Define the model
net = nn.RNN(input_size=4, hidden_size=4, batch_first=True)

# Generate data
data = [[H,E,L,L,O],
        [E,L,L,O,H],
        [L,L,O,H,E],
        [L,O,H,E,L],
        [O,H,E,L,L]]
inputs = torch.tensor(data).float()
hidden = torch.randn(1,5,4) # Random initialization
correct_outputs = torch.tensor(np.array(data[1:]+[data[0]]).astype(float).tolist(), requires_grad=True)

# Set the loss function
criterion = torch.nn.MSELoss()

# Set the optimizer
optimizer = torch.optim.SGD(net.parameters(), lr=0.1)

# Perform gradient descent until convergence
for epoch in range(1000):
    # Forward Propagation
    outputs, hidden = net(inputs, hidden)
    # Compute and print loss
    loss = criterion(nn.functional.softmax(outputs,2), correct_outputs)
    print('epoch: ', epoch,' loss: ', loss.item())
    # Zero the gradients
    optimizer.zero_grad()
    # Backpropagation
    loss.backward(retain_graph=True)
    # Parameter update
    optimizer.step()

# Predict
net(torch.tensor([[H,E,L,L,O]]).float(),hidden)

I get the following error:

RuntimeError: Expected hidden size (1, 1, 4), got (1, 5, 4)

I understand that torch wants a tensor of size (1,1,4) but I am not sure how I can convert the initial hidden state from (1, 5, 4) to (1, 1, 4). Any help would be highly appreciated!


title: PyTorch LSTM: RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 0. Got 1219 and 440 in dimension 1

body:
I have a basic PyTorch LSTM:

import torch.nn as nn
import torch.nn.functional as F

class BaselineLSTM(nn.Module):
    def __init__(self):
        super(BaselineLSTM, self).__init__()

        self.lstm = nn.LSTM(input_size=13, hidden_size=13)

    def forward(self, x):
        x = self.lstm(x)

        return x

For my data, I have:

train_set = CorruptedAudioDataset(corrupted_path, train_set=True)
train_loader = torch.utils.data.DataLoader(train_set, batch_size=128, shuffle=True, **kwargs)

My CorruptedAudioDataset has:

    def __getitem__(self, index):
        corrupted_sound_file = SoundFile(self.file_paths[index])
        corrupted_samplerate = corrupted_sound_file.samplerate
        corrupted_signal_audio_array = corrupted_sound_file.read()

        clean_path = self.file_paths[index].split('/')
        # print(self.file_paths[index], clean_path)
        clean_sound_file = SoundFile(self.file_paths[index])
        clean_samplerate = clean_sound_file.samplerate
        clean_signal_audio_array = clean_sound_file.read()

        corrupted_mfcc = mfcc(corrupted_signal_audio_array, samplerate=corrupted_samplerate)
        clean_mfcc = mfcc(clean_signal_audio_array, samplerate=clean_samplerate)

        print('return', corrupted_mfcc.shape, clean_mfcc.shape)
        return corrupted_mfcc, clean_mfcc

My training loop looks like:

    model = BaselineLSTM()
    for epoch in range(300):
        for inputs, outputs in train_loader:
            print('inputs', inputs)

And that's the line that I get the error on:

  File "train_lstm_baseline.py", line 47, in train
    for inputs, outputs in train_loader:
...
RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 0. Got 1219 and 440 in dimension 1 at ../aten/src/TH/generic/THTensor.cpp:612

Not sure what I'm doing wrong. Any help would be appreciated. Thanks!


i am using this code earlier in Jupiter notebook it was not showing error but accuracy was veryless then i have tried the same code in google colab there it is showing error, please suggest some way to increase accuracy . i am trying to perform multilevel CNN for detecting leaf with downsampling of image

import torch
import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1,32,2)
        self.conv2 = nn.Conv2d(32,64,2)
        self.conv2_bn = nn.BatchNorm2d(64)
        x= torch.randn(256,256).view(-1,1,256,256)
        self._to_linear = None
        self.convs(x)

        self.fc1= nn.Linear(self._to_linear, 512)
        self.fc2 = nn.Linear(512,6)

    def convs(self,x):
        y=torch.nn.functional.interpolate(x, size=([128,128]), scale_factor=None, mode='nearest', align_corners=None)
        z=torch.nn.functional.interpolate(x, size=([64,64]), scale_factor=None, mode='nearest', align_corners=None)
        w=torch.nn.functional.interpolate(x, size=([32,32]), scale_factor=None, mode='nearest', align_corners=None)

        # print(x[0].shape)

        x= F.relu(self.conv1(x))
        m = nn.ConstantPad2d(1,0)
        x=m(x)
        x = F.relu(F.max_pool2d(self.conv2_bn(self.conv2(x)), 2))
#         print(x[0].shape)

        y= F.relu(self.conv1(y))
        m = nn.ConstantPad2d(1,0)
        y=m(y)
        y = F.relu(self.conv2_bn(self.conv2(y)), 2)
#         print(y[0].shape)

        CAT_1=torch.cat((x,y),1)
        CAT_1=F.max_pool2d(CAT_1,(2,2))
#         print(CAT_1[0].shape)

        z= F.relu(self.conv1(z))
        m = nn.ConstantPad2d(1,0)
        z=m(z)
        z= F.relu(self.conv2_bn(self.conv2(z)))
#         print(z[0].shape)

        CAT_2=torch.cat((CAT_1,z),1)
        CAT_2=F.max_pool2d(CAT_2,(2,2))
#         print(CAT_2[0].shape)

        w= F.relu(self.conv1(w))
        m = nn.ConstantPad2d(1,0)
        w=m(w)
        w = F.relu((self.conv2_bn(self.conv2(w))))
#         print(w[0].shape)

        x=torch.cat((CAT_2,w),1)
        x=F.max_pool2d(x,(2,2))
#         print("i lov pp")
#         print(x[0].shape)
        x=torch.nn.functional.avg_pool2d(x, (2,2))
#         print("i lov pp")
#         print(x[0].shape)

        if self._to_linear is None:
            self._to_linear = x[0].shape[0]*x[0].shape[1]*x[0].shape[2]
        return x

    def forward(self, x):
#         print("i lov pp")
        x=self.convs(x)
        x=x.view(-1, self._to_linear)

        x= F.relu(self.fc1(x))
        x= self.fc2(x)
        return F.softmax(x, dim=1)
#         print(x[0].shape)
net=Net()

import torch.optim as optim

optimizer = optim.Adam(net.parameters(), lr=0.001)
loss_function = nn.MSELoss()

X = torch.Tensor([i[0] for i in training_data]).view(-1,256,256)
X=X/255.0
y = torch.Tensor([i[1] for i in training_data])

VAL_PCT = 0.1
val_size=int (len(X)*VAL_PCT)
print(val_size)

train_X= X[:-val_size]
train_y= y[:-val_size]

test_X=X[-val_size:]
test_y = y[-val_size:]

print(len(train_X))
print(len(test_X))

BATCH_SIZE =10
EPOCHS = 1

for epoch in range(EPOCHS):
    for i in  (range(0, len(train_X), BATCH_SIZE)):
        #print(i, i+BATCH_SIZE)
        batch_X = train_X[i:i+BATCH_SIZE].view(-1,1,256,256)
        # print(batch_X.shape)

        batch_y = train_y[i:i+BATCH_SIZE]
        #print(batch_y.shape)
        net.zero_grad()

        outputs = net(batch_X)
        #print (outputs.shape)
        loss = loss_function(outputs, batch_y)
        loss.backward()
        optimizer.step()
        #print(loss)
        #print(f"Epoch: {epoch}. Loss: {loss}")

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/loss.py:432: UserWarning: Using a target size (torch.Size([10, 256, 256, 3])) that is different to the input size (torch.Size([10, 6])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
  return F.mse_loss(input, target, reduction=self.reduction)
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-38-a154b102127f> in <module>()
     15         outputs = net(batch_X)
     16         #print (outputs.shape)
---> 17         loss = loss_function(outputs, batch_y)
     18         loss.backward()
     19         optimizer.step()

3 frames
/usr/local/lib/python3.6/dist-packages/torch/functional.py in broadcast_tensors(*tensors)
     60         if any(type(t) is not Tensor for t in tensors) and has_torch_function(tensors):
     61             return handle_torch_function(broadcast_tensors, tensors, *tensors)
---> 62     return _VF.broadcast_tensors(tensors)
     63
     64

RuntimeError: The size of tensor a (10) must match the size of tensor b (9) at non-singleton dimension 3


title: Gradients are zero in manual weights update in PyTorch

body:
I'm trying to implement a simple neural net with manual update of weights for MNIST with AUTOGRAD, similar to AUTOGRAD example given here. This is my code:

import os
import sys

import torch
import torchvision
class Datasets:
    """Helper for extracting datasets."""

    def __init__(self, root='data/', batch_size=25):
        if not os.path.exists(root):
            os.mkdir(root)
        self.root = root
        self.batch_size = batch_size

    def get_mnist_loaders(self):
        train_data = torchvision.datasets.MNIST(
                root=self.root, train=True, download=True)
        test_data = torchvision.datasets.MNIST(
                root=self.root, train=False, download=True)

        train_loader = torch.utils.data.DataLoader(
                dataset=train_data, batch_size=self.batch_size, shuffle=True)
        test_loader = torch.utils.data.DataLoader(
                dataset=test_data, batch_size=self.batch_size, shuffle=False)

        return train_loader, test_loader

    def create_batches(self, data, labels, batch_size):
        return [(data[i:i+batch_size], labels[i:i+batch_size])
            for i in range(0, len(data), max(1, batch_size))]

def train1():
    dtype = torch.float
    n_inputs = 28*28
    n_hidden1 = 300
    n_hidden2 = 100
    n_outputs = 10
    batch_size = 200
    n_epochs = 25
    learning_rate = 0.01
    test_step = 100
    device = torch.device("cpu")

    datasets = Datasets(batch_size=batch_size)
    train_loader, test_loader = datasets.get_mnist_loaders()

    def feed_forward(X):
        x_shape = list(X.size())
        X = X.view(x_shape[0], x_shape[1]*x_shape[2])
        hidden1 = torch.mm(X, w1)
        hidden1 += b1
        hidden1 = hidden1.clamp(min=0)
        hidden2 = torch.mm(hidden1, w2) + b2
        hidden2 = hidden2.clamp(min=0)
        logits = torch.mm(hidden2, w3) + b3
        softmax = pytorch_softmax(logits)
        return softmax

    def accuracy(y_pred, y):

        if list(y_pred.size()) != list(y.size()):
            raise ValueError('Inputs have different shapes.')

        total_correct = 0
        total = 0
        for i, (y1, y2) in enumerate(zip(y_pred, y)):
            if y1 == y2:
                total_correct += 1
            total += 1

        return total_correct / total

    w1 = torch.randn(n_inputs, n_hidden1, device=device, dtype=dtype, requires_grad=True)
    b1 = torch.nn.Parameter(torch.zeros(n_hidden1), requires_grad=True)

    w2 = torch.randn(n_hidden1, n_hidden2, requires_grad=True)
    b2 = torch.nn.Parameter(torch.zeros(n_hidden2), requires_grad=True)

    w3 = torch.randn(n_hidden2, n_outputs, dtype=dtype, requires_grad=True)
    b3 = torch.nn.Parameter(torch.zeros(n_outputs), requires_grad=True)

    pytorch_softmax = torch.nn.Softmax(0)
    pytorch_cross_entropy = torch.nn.CrossEntropyLoss(reduction='elementwise_mean')

    step = 0
    for epoch in range(n_epochs):
        batches = datasets.create_batches(train_loader.dataset.train_data,
                                          train_loader.dataset.train_labels,
                                          batch_size)
        for x, y in batches:
            step += 1

            softmax = feed_forward(x.float())
            vals, y_pred = torch.max(softmax, 1)
            accuracy_ = accuracy(y_pred, y)
            cross_entropy = pytorch_cross_entropy(softmax, y)

            print(epoch, step, cross_entropy.item(), accuracy_)

            cross_entropy.backward()

            with torch.no_grad():
                w1 -= learning_rate * w1.grad
                w2 -= learning_rate * w2.grad
                w3 -= learning_rate * w3.grad

                b1 -= learning_rate * b1.grad
                b2 -= learning_rate * b2.grad
                b3 -= learning_rate * b3.grad

                w1.grad.zero_()
                w2.grad.zero_()
                w3.grad.zero_()

                b1.grad.zero_()
                b2.grad.zero_()
                b3.grad.zero_()

if __name__ == '__main__':
    train1()

However the network doesn't seem to train. When I print parts of gradients (e.g. w1.grad.data[:10, :10]) they consist of zeros. I've tried to use weight.data and weight.grad.data to update the weights and tried to remove the w.grad.zero_() part (even though it is in the example) but it doesn't help. What is the problem here?


title: How to convert tensorflow model (InceptionResnetV2 pb file) to pytorch model using mmconvert?

body:
Platform : ubuntu 16.04
Python version: 3.5.2
mmdnn version : 0.2.5
Source framework with version : Tensorflow 1.14.0 with GPU
Destination framework with version : Pytorch 1.1.0

Pre-trained model path :

inception resnet v2 model(tensorflow)
(from https://github.com/tensorflow/models/tree/master/research/slim)

Running scripts:

mmconvert -sf tensorflow -iw inception_resnet_v2_jsy.pb --inNodeName input --inputShape 299,299,3 --dstNodeName InceptionResnetV2/Logits/Logits/BiasAdd -df pytorch -om tf_to_pytorch_inception_resnet_v2.pth

Hello,

I'm tried to convert tensorflow model (pb file of inception resnet v2 ) to pytorch model for using mmdnn.

I got successful results for 2 models with pb files (resnet_v1_50, inception_v3) , but when I tried to convert inception_resnet_v2, I got below errors.

Is there anyone who have some ideas to solve them or to explain those problems?

Error logs.

IR network structure is saved as [09db48a5839944eeb4492ee2a0959097.json].
IR network structure is saved as [09db48a5839944eeb4492ee2a0959097.pb].
IR weights are saved as [09db48a5839944eeb4492ee2a0959097.npy].
Parse file [09db48a5839944eeb4492ee2a0959097.pb] with binary format successfully.
Target network code snippet is saved as [tf_to_pytorch_inception_resnet_v2.py].
Target weights are saved as [09db48a5839944eeb4492ee2a0959097.npy].
Traceback (most recent call last):
File "/usr/local/bin/mmconvert", line 11, in
sys.exit(_main())
File "/usr/local/lib/python3.5/dist-packages/mmdnn/conversion/_script/convert.py", line 112, in _main
dump_code(args.dstFramework, network_filename + '.py', temp_filename + '.npy', args.outputModel, args.dump_tag)
File "/usr/local/lib/python3.5/dist-packages/mmdnn/conversion/_script/dump_code.py", line 32, in dump_code
save_model(MainModel, network_filepath, weight_filepath, dump_filepath)
File "/usr/local/lib/python3.5/dist-packages/mmdnn/conversion/pytorch/saver.py", line 5, in save_model
model = MainModel.KitModel(weight_filepath)
File "tf_to_pytorch_inception_resnet_v2.py", line 476, in init
self.InceptionResnetV2_Logits_Logits_MatMul = self.__dense(name = 'InceptionResnetV2/Logits/Logits/MatMul', in_features = -1, out_features = 1001, bias = True)
File "tf_to_pytorch_inception_resnet_v2.py", line 1444, in __dense
layer = nn.Linear(**kwargs)
File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/linear.py", line 76, in init
self.weight = Parameter(torch.Tensor(out_features, in_features))

RuntimeError: Trying to create tensor with negative dimension -1: [1001, -1]


title: What does Pytorch's default padding in Conv2D signify?

body:
In the Pytorch documentation for the MaxPool2D states:

  padding (int or tuple, optional) – Zero-padding added to both sides of the input. Default: 0

Does it mean that the default values for padding would be 0 or that there will be no padding?

In my code: nn.Conv2d(input_channels, output_channels, kernel_size, stride), I didn't pass any padding parameter and when I printed it, it shows the following

  (0): Conv2d(4, 8, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2))

So, I'm assuming it by default adds padding to match the input size with values 0.

Now, if that's the case, then how can I turn it off and/or add a different padding value?


title: How to do numerical integration with pytorch similar to numpy's trapz function?

body: Title says it all. Is there a convenient function in pytorch that can do something like np.trapz(y, x) (integrating over the the points in x and y via trapezoidal rule)?


title: how to import coco_utils and coco_eval?

body:
To get a PyTorch script to work, I need to be able to do:

import coco_utils
import coco_eval

I'm using Ubuntu 18.04 with Python 3.

Based on this post:

How to install COCO PythonAPI in python3

I've done the following so far:

cd ~

git clone https://github.com/cocodataset/cocoapi.git

cd cocoapi/PythonAPI

# open Makefile in gedit, change the two instances of "python" to "python3"

python3 setup.py build_ext --inplace

sudo python3 setup.py install

now this works:

import pycocotools

but these still don't:

import coco_utils
import coco_eval

How can I import coco_utils and coco_eval ??


title: CUDA error 59: Device-side assert triggered

body:
I get the above error with Pytorch, with the following assertion:

/opt/conda/conda-bld/pytorch_1565272269120/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda [](int)->auto::operator()(int)->auto: block: [1,0,0], thread: [127,0,0]

assertion `index >= -size[i] && index < size at] && "index out of bounds"` failed

I have seen the other solutions to this problem that describe how it might due to the labels being not from (0, num_classes-1), etc. However, I have ensured that in my case and the error comes while computing Hinge Loss as follows:

diff_hinge_loss+=  F.hinge_embedding_loss( neg_dist - pos_dist, torch.tensor(-1).to(cuda), args.diff_margin, reduction='sum').to(cuda)

Everything is normal while training initially, however after training for certain epochs, I get CUDA runtime error with the computation of Hinge Loss.

Full Error Trace:

/opt/conda/conda-bld/pytorch_1565272269120/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda [](int)->auto::operator()(int)->auto: block: [1,0,0], thread: [127,0,0]
Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.

Traceback (most recent call last):

  File "ours-vision.py", line 1106, in <module>
    penalty_erm, penalty_irm, penalty_ws, penalty_same_ctr, penalty_diff_ctr = train( train_dataset, data_match_tensor, label_match_tensor, phi, opt, opt_ws, scheduler, epoch, base_domain_idx, bool_erm, bool_ws, bool_ctr )

  File "ours-vision.py", line 688, in train
diff_hinge_loss+=  F.hinge_embedding_loss( neg_dist - pos_dist, torch.tensor(-1).to(cuda), args.diff_margin, reduction='sum').to(cuda)

RuntimeError: CUDA error: device-side assert triggered


title: How to obtain input data from ONNX model?

body:
I have exported my PyTorch model to ONNX. Now, is there a way for me to obtain the input layer from that ONNX model?

Exporting PyTorch model to ONNX

import torch.onnx
checkpoint = torch.load("./saved_pytorch_model.pth")
model.load_state_dict(checkpoint['state_dict'])
input = torch.tensor(df_X.values).float()
torch.onnx.export(model, input, "onnx_model.onnx")

Loading ONNX model

onnx_model = onnx.load('onnx_model.onnx')

I want to be able to somehow obtain the input layer from onnx_model. Is this possible?


title: How to get rid of every column that are filled with zero from a Pytorch tensor?

body:
I have a pytorch tensor A like below:

A =
tensor([[  4,   3,   3,  ...,   0,   0,   0],
        [ 13,   4,  13,  ...,   0,   0,   0],
        [707, 707,   4,  ...,   0,   0,   0],
        ...,
        [  7,   7,   7,  ...,   0,   0,   0],
        [  0,   0,   0,  ...,   0,   0,   0],
        [195, 195, 195,  ...,   0,   0,   0]], dtype=torch.int32)

I would like to:

identify all the columns whose all of its entries are equal to 0
delete only those columns that has all of their entries equal to 0

I can imagine doing:

zero_list = []
for j in range(A.size()[1]):
    if torch.sum(A[:,j]) == 0:
         zero_list = zero_list.append(j)

to identify the columns that only has 0 for its elements
but I am not sure how to delete such columns filled with 0 from the original tensor.

How can I delete the columns with zero from a pytorch tensor based on the index number?

Thank you,


title: Huggingface Transformers ByteLevelBPETokenizer tokenizer not found

body:
I'm trying to run through the (new) tutorial here: https://huggingface.co/blog/how-to-train, but hit an error trying to load the ByteLevelBPETokenizer. I started from an existing conda env and also tried with a totally fresh env, but both give the same error:

Exception has occurred: ImportError
cannot import name 'ByteLevelBPETokenizer' from 'tokenizers' (/home/james/anaconda3/envs/torch/lib/python3.7/site-packages/tokenizers/__init__.py)

Any thoughts as to what might be wrong?

I'm on Ubuntu 18.04, Python 3.7


title: How to fix the error "TypeError: forward()... - CUDA

body:
For my bachelor thesis I need to train a network with some music similarity data using GPU and CUDA.

Tried to fix the problem several times with different approaches, but none of them worked.

use_cuda = torch.cuda.is_available()

BSG_model = bayesian_skipgram(V, EMBEDDING_DIM)

if use_cuda:
    BSG_model.cuda()
optimizer = torch.optim.Adam(BSG_model.parameters(), lr=0.005)
BSG_model.train()
loss_progress = []
iter_time = time.time()

dataloader = DataLoader(data, batch_size=16, shuffle=True)

print("N_batches", len(dataloader))

for epoch in range(1):
    for i, batch in enumerate(dataloader):
        batch_start = time.time()
        main_word = batch[:,0]
        context_word = batch[:,1:]

    #print("Main word:,", main_word.shape, context_word.shape)

    optimizer.zero_grad()

    if use_cuda:
        loss = BSG_model.forward(main_word.cuda(), context_word.cuda(), use_cuda=True)
    else:
        loss = BSG_model.forward(main_word, context_word)

    loss.backward()
    optimizer.step()
    batch_end = time.time() - batch_start
    if i % 10 == 0:
        print("epoch, batch ", epoch, i)
        loss_progress.append(loss.item())
        print(time.time()-iter_time)
        print(loss)
        iter_time = time.time()

"The expected result should be that the model starts to train the embeddings..."
"The output is the following:"

TypeError                                 Traceback (most recent call last)
<ipython-input-36-c69aba816e22> in <module>()
     34
     35         if use_cuda:
---> 36             loss = BSG_model.forward(main_word.cuda(), context_word.cuda(), use_cuda=True)
     37         else:
     38             loss = BSG_model.forward(main_word, context_word)

 TypeError: forward() got an unexpected keyword argument 'use_cuda'


title: How to implement Batchnorm2d in Pytorch myself?

body:
I'm trying to implement Batchnorm2d() layer with:
class BatchNorm2d(nn.Module):

    def __init__(self, num_features):
        super(BatchNorm2d, self).__init__()
        self.num_features = num_features
        device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
        self.eps = 1e-5
        self.momentum = 0.1
        self.first_run = True

    def forward(self, input):
        # input: [batch_size, num_feature_map, height, width]
        device = input.device
        if self.training:
            mean = torch.mean(input, dim=0, keepdim=True).to(device)  # [1, num_feature, height, width]
            var = torch.var(input, dim=0, unbiased=False, keepdim=True).to(device)  # [1, num_feature, height, width]
            if self.first_run:
                self.weight = Parameter(torch.randn(input.shape, dtype=torch.float32, device=device), requires_grad=True)
                self.bias = Parameter(torch.randn(input.shape, dtype=torch.float32, device=device), requires_grad=True)
                self.register_buffer('running_mean', torch.zeros(input.shape).to(input.device))
                self.register_buffer('running_var', torch.ones(input.shape).to(input.device))
                self.first_run = False
            self.running_mean = (1 - self.momentum) * self.running_mean + self.momentum * mean
            self.running_var = (1 - self.momentum) * self.running_var + self.momentum * var
            bn_init = (input - mean) / torch.sqrt(var + self.eps)
        else:
            bn_init = (input - self.running_mean) / torch.sqrt(self.running_var + self.eps)
        return self.weight * bn_init + self.bias

But after training & testing I found that the results using my layer is incomparable with the results using nn.Batchnorm2d(). There must be something wrong with it, and I guess the problem relates to initializing parameters in forward()? I did that because I don't know how to know the shape of input in __init__(), maybe there is a better way.  I don't know how to fix it, please help. Thanks!!


title: Pytorch Tensor help in LongTensor

body:
a=[1,2,3];
context_var = autograd.Variable(torch.LongTensor(a))

This is giving an error

RuntimeError: tried to construct a tensor from a int sequence, but found an item of type numpy.int32 at index

I am not able to figure out how to get over this.


title: how to see where exactly torch is installed pip vs conda torch installation

body:
On my machine i can't "pip install torch" - i get infamous "single source externally managed error" - i could not fix it and used "conda install torch" from anaconda.
Still, checking version is easy - torch.__version__
But how to see where is it installed -the home dir of torch?
Suppose if I had had both torches installed via pip and conda - how to know which one is used in a project?
import torch
print(torch__version__)


title: PyTorch: Calculating the Hessian vector product with nn.parameters()

body:
Using PyTorch, I would like to calculate the Hessian vector product, where the Hessian is the second-derivative matrix of the loss function of some neural net, and the vector will be the vector of gradients of that loss function.

I know how to calculate the Hessian vector product for a regular function thanks to this post. However, I am running into trouble when the function is the loss function of a neural network. This is because the parameters are packaged into a module, accessible via nn.parameters(), and not a torch tensor.

I want to do something like this (doesn't work):

### a simple neural network
linear = nn.Linear(10, 20)

x = torch.randn(1, 10)

y = linear(x).sum()
### compute the gradient and make a copy that is detached from the graph
grad = torch.autograd.grad(y, linear.parameters(),create_graph=True)

v = grad.clone().detach()
### compute the Hessian vector product
z = grad @ v
z.backward()

In analogy this this (does work):

x = Variable(torch.Tensor([1, 1]), requires_grad=True)

f = 3*x[0]**2 + 4*x[0]*x[1] + x[1]**2

grad, = torch.autograd.grad(f, x, create_graph=True)

v = grad.clone().detach()

z = grad @ v

z.backward()

This post addresses a similar (possibly the same?) issue, but I don't understand the solution.


title: Bitwise operations in Pytorch

body:
Could someone help me how to perform bitwise  AND operations on two tensors in Pytorch 1.4?

Apparently I could only find NOT and XOR operations in official document


title: Get Euclidian and infinite distance in Pytorch

body:
I'm trying to get the Euclidian Distance in Pytorch, using torch.dist, as shown below:

torch.dist(vector1, vector2, 1)

If I use "1" as the third Parameter, I'm getting the Manhattan distance, and the result is correct, but I'm trying to get the Euclidian and Infinite distances and the result is not right. I tried with a lot of different numbers on the third parameter, but I'm not able to get the desired distances.

How can I get the Euclidian and Infinite distances using Pytorch?


title: How do I pass a keyword argument to the forward used by a pre-forward hook?

body:
Given a torch's nn.Module with a pre-forward hook, e.g.

import torch
import torch.nn as nn

class NeoEmbeddings(nn.Embedding):
    def __init__(self, num_embeddings:int, embedding_dim:int, padding_idx=-1):
        super().__init__(num_embeddings, embedding_dim, padding_idx)
        self.register_forward_pre_hook(self.neo_genesis)

    @staticmethod
    def neo_genesis(self, input, higgs_bosson=0):
        if higgs_bosson:
            input = input + higgs_bosson
        return input

It's possible to let an input tensor go through some manipulation before going to the actual forward() function, e.g.

>>> x = NeoEmbeddings(10, 5, 1)
>>> x.forward(torch.tensor([0,2,5,8]))
tensor([[-1.6449,  0.5832, -0.0165, -1.3329,  0.6878],
        [-0.3262,  0.5844,  0.6917,  0.1268,  2.1363],
        [ 1.0772,  0.1748, -0.7131,  0.7405,  1.5733],
        [ 0.7651,  0.4619,  0.4388, -0.2752, -0.3018]],
       grad_fn=<EmbeddingBackward>)

>>> print(x._forward_pre_hooks)
OrderedDict([(25, <function NeoEmbeddings.neo_genesis at 0x1208d10d0>)])

How could we pass the arguments (*args or **kwargs) that the pre-forward hook needs but not accepted by the default forward() function?

Without modification/overriding the forward() function, this is not possible:

>>> x = NeoEmbeddings(10, 5, 1)
>>> x.forward(torch.tensor([0,2,5,8]), higgs_bosson=2)

----------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-102-8705a40a3cc2> in <module>
      1 x = NeoEmbeddings(10, 5, 1)
----> 2 x.forward(torch.tensor([0,2,5,8]), higgs_bosson=2)

TypeError: forward() got an unexpected keyword argument 'higgs_bosson'


title: How to convert caffe model and weight to pytorch

body:
Hello I need help converting weight and model of caffe to pytorch. I have tried using the github that most other post suggest to use this github but when I used it, I encounter a lot of problem since it used python 2 and currently I am using python 3. I have already tried to remove some layer that the github doesn't cover, manually change old syntax to the new syntax but the last error reference to nn module from pytorch and I have no idea to fix that.

Traceback (most recent call last):
  File "caffe2pytorch.py", line 30, in <module>
    pytorch_blobs, pytorch_models = forward_pytorch(protofile, weightfile)
  File "caffe2pytorch.py", line 17, in forward_pytorch
    net = caffenet.CaffeNet(protofile)
  File "/home/cgal/reference/SfSNet/caffe2pytorch/caffenet.py", line 384, in __init__
    self.add_module(name, model)
  File "/home/cgal/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 186, in add_module
    raise KeyError("module name can't contain \".\"")
KeyError: 'module name can\'t contain "."'

So is there any suggestion on how to convert caffe weight and model to pytorch?

This is the caffe model that I want to convert download here


title: PyTorch DLL load failed

body:
I am using Anaconda 3 2018 12 version and python 3.6.8 version. I am try to install pytorch it shows following error message all the time.

>>> import torch
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\Office\Anaconda3\envs\mainenv\lib\site-packages\torch\__init__.py", line 76, in <module>
    from torch._C import *
ImportError: DLL load failed: The specified module could not be found.

I tried the following command in anaconda prompt.

conda install pytorch-cpu torchvision-cpu -c pytorch

This recommends an official pytorch website. It also gives this error.

Then I uninstalled it and tried the following code.

conda install -c peterjc123 pytorch-cpu

It is also given that error message.

Again I create another new environment for test the use peterjc123 command it is also given above error message.

Now I am really fed up. Please help me solve this error.

This is init.py file code

################################################################################
# Load the extension module
################################################################################

# Loading the extension with RTLD_GLOBAL option allows to not link extension
# modules against the _C shared object. Their missing THP symbols will be
# automatically filled by the dynamic loader.
import os as _dl_flags

# if we have numpy, it *must* be imported before the call to setdlopenflags()
# or there is risk that later c modules will segfault when importing numpy
try:
    import numpy as np
except ImportError:
    pass

if platform.system() == 'Windows':
    # first get nvToolsExt PATH
    def get_nvToolsExt_path():
        NVTOOLEXT_HOME = _dl_flags.getenv('NVTOOLSEXT_PATH', 'C:\\Program Files\\NVIDIA Corporation\\NvToolsExt')

        if _dl_flags.path.exists(NVTOOLEXT_HOME):
            return NVTOOLEXT_HOME + '\\bin\\x64\\'
        else:
            return ''

    # then add the path to env
    _dl_flags.environ['PATH'] = _dl_flags.path.dirname(
        __file__) + '\\lib\\;' + get_nvToolsExt_path() + ';' + _dl_flags.environ['PATH']

else:
    # first check if the os package has the required flags
    if not hasattr(_dl_flags, 'RTLD_GLOBAL') or not hasattr(_dl_flags, 'RTLD_LAZY'):
        try:
            # next try if DLFCN exists
            import DLFCN as _dl_flags
        except ImportError:
            # as a last attempt, use compile-time constants
            import torch._dl as _dl_flags

    old_flags = sys.getdlopenflags()
    sys.setdlopenflags(_dl_flags.RTLD_GLOBAL | _dl_flags.RTLD_LAZY)

del _dl_flags

try:
    import torch._nvrtc
except ImportError:
    pass

from torch._C import *

__all__ += [name for name in dir(_C)
            if name[0] != '_' and
            not name.endswith('Base')]

if platform.system() != 'Windows':
    sys.setdlopenflags(old_flags)
    del old_flags


title: Converting Keras (Tensorflow) convolutional neural networks to PyTorch convolutional networks?

body:
Keras and PyTorch use different arguments for padding: Keras requires a string to be input, while PyTorch works with numbers. What is the difference, and how can one be translated into another (what code gets the equivalent result in either framework)?

PyTorch also takes the args in_channels, out_chanels while keras only takes an argument called filters. What does 'filters' mean?


title: How to fine tune BERT on unlabeled data?

body:
I want to fine tune BERT on a specific domain. I have texts of that domain in text files. How can I use these to fine tune BERT?
I am looking here currently.

My main objective is to get sentence embeddings using BERT.


I'm working with the project 'lda2vec-pytorch' on Google CoLab,
runnin pytorch 1.1.0

https://github.com/TropComplique/lda2vec-pytorch

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
cuda:0

I'm getting an exception in the forward method adding 'noise' in my
class negative_sampling_loss(nn.Module):

        noise = self.multinomial.draw(batch_size*window_size*self.num_sampled)
        noise = Variable(noise).view(batch_size, window_size*self.num_sampled)

        device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
        self.embedding = self.embedding.to(device)
        #print("negative_sampling_loss::forward() self.embedding", self.embedding.is_cuda)  This line get's an error.

        # shape: [batch_size, window_size*num_sampled, embedding_dim]
        noise = self.embedding(noise)  # Exception HERE

Here's the stack trace:

Traceback (most recent call last):
  File "train.py", line 36, in <module>
    main()
  File "train.py", line 32, in main
    save_every=20, grad_clip=5.0
  File "../utils/training.py", line 138, in train
    neg_loss, dirichlet_loss = model(doc_indices, pivot_words, target_words)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "../utils/lda2vec_loss.py", line 82, in forward
    neg_loss = self.neg(pivot_words, target_words, doc_vectors, w)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "../utils/lda2vec_loss.py", line 167, in forward
    noise = self.embedding(noise)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/sparse.py", line 117, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py", line 1506, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: Expected object of backend CUDA but got backend CPU for argument #3 'index'

Any ideas?


title: "Installing From Source" Within Anaconda Environment

body:
What I would like to do:

I am using macOS and Anaconda 2.
I would like to install a Python package (specifically PyTorch) from source.
I would like to install all the dependencies and the package itself within an Anaconda environment.
I don't want this Anaconda environment to be the default/ root Anaconda environment, but an environment I particularly created for installing this package and its dependencies from source.

What I have done:

First, I created the environment as follows

conda create --name my_env python=3.5

Now, the instructions for installing PyTorch from source are as follows:

export CMAKE_PREFIX_PATH=[anaconda root directory]
conda install numpy pyyaml setuptools cmake cffi
git clone --recursive https://github.com/pytorch/pytorch
MACOSX_DEPLOYMENT_TARGET=10.9 CC=clang CXX=clang++ python setup.py install

Now, my questions are:

Following this instructions, requires me to specify anaconda root directory for the CMAKE_PREFIX_PATH. What should that directory be given that I want everything set-up in my_env?
Is it reasonable to create an extra environment for a package installed from source and its dependencies? Why would one do or not do it? My motivation is mainly fear that one day I  may screw my system up big time and hence want things to be cleanly separated.

If you can only answer one of the two questions, that is already greatly appreciated. Thanks!


title: How to know the input shape of a pytorch model?

body:
I have a list of 100 matrix with the shape (20,48) and I want to pass this matrix in pytorch.

This is the sample code

import torch.nn.functional as F
import torch.nn as nn
import torch

sample = torch.randn(100,20,48)

class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv = nn.Conv1d(20, 48, kernel_size=2)
    def forward(self, x):
        return self.conv(x)

net = Net()

for i in net.state_dict().keys():
    print(i)

for i in list(net.parameters()):
    print(i.shape)

#output

conv.weight
conv.bias

torch.Size([48, 20, 2])
torch.Size([48])

How to check that my the model takes an input of particular shape? In my case how can I confirm if my model input conv layer takes a matrix of the size (bs,20,48)?


title: Deploying a pytorch model in java

body: I have a pytorch model trained and saved and now I want to use it in a java (not android) environment in windows os (since I'm using some library only available in java), Is it possible? I couldn't find a straight answer in the pytorch docs, and when clicking java api docs the link is broken.


title: Pytorch - Concatenating Datasets before using Dataloader

body:
I am trying to load two datasets and use them both for training.

Package versions: python 3.7;
pytorch 1.3.1

It is possible to create data_loaders seperately and train on them sequentially:

from torch.utils.data import DataLoader, ConcatDataset

train_loader_modelnet = DataLoader(ModelNet(args.modelnet_root, categories=args.modelnet_categories,split='train', transform=transform_modelnet, device=args.device),batch_size=args.batch_size, shuffle=True)

train_loader_mydata = DataLoader(MyDataset(args.customdata_root, categories=args.mydata_categories, split='train', device=args.device),batch_size=args.batch_size, shuffle=True)

for e in range(args.epochs):
    for idx, batch in enumerate(tqdm(train_loader_modelnet)):
        # training on dataset1
    for idx, batch in enumerate(tqdm(train_loader_custom)):
        # training on dataset2

Note: MyDataset is a custom dataset class which has def __len__(self): def __getitem__(self, index): implemented. As the above configuration works it seems that this is implementation is OK.

But I would ideally like to combine them into a single dataloader object. I attempted this as per the pytorch documentation:

train_modelnet = ModelNet(args.modelnet_root, categories=args.modelnet_categories,
                          split='train', transform=transform_modelnet, device=args.device)
train_mydata = CloudDataset(args.customdata_root, categories=args.mydata_categories,
                             split='train', device=args.device)
train_loader = torch.utils.data.ConcatDataset(train_modelnet, train_customdata)

for e in range(args.epochs):
    for idx, batch in enumerate(tqdm(train_loader)):
        # training on combined

However, on random batches I get the following 'expected a tensor as element X in argument 0, but got a tuple instead' type of error. Any help would be much appreciated!

>   40%|████      | 53/131 [01:03<02:00,  1.55s/it]
>  Traceback (mostrecent call last):   File
> "/home/chris/Programs/pycharm-anaconda-2019.3.4/plugins/python/helpers/pydev/pydevd.py",
> line 1434, in _exec
>     pydev_imports.execfile(file, globals, locals)  # execute the script   File
> "/home/chris/Programs/pycharm-anaconda-2019.3.4/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
>     exec(compile(contents+"\n", file, 'exec'), glob, loc)   File "/home/chris/Documents/4yp/Data/my_kaolin/Classification/pointcloud_classification_combinedset.py",
> line 83, in <module>
>     for idx, batch in enumerate(tqdm(train_loader)):   File "/home/chris/anaconda3/envs/4YP/lib/python3.7/site-packages/tqdm/std.py",
> line 1107, in __iter__
>     for obj in iterable:   File "/home/chris/anaconda3/envs/4YP/lib/python3.7/site-packages/torch/utils/data/dataloader.py",
> line 346, in __next__
>     data = self._dataset_fetcher.fetch(index)  # may raise StopIteration   File
> "/home/chris/anaconda3/envs/4YP/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py",
> line 47, in fetch
>     return self.collate_fn(data)   File "/home/chris/anaconda3/envs/4YP/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py",
> line 79, in default_collate
>     return [default_collate(samples) for samples in transposed]   File "/home/chris/anaconda3/envs/4YP/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py",
> line 79, in <listcomp>
>     return [default_collate(samples) for samples in transposed]   File "/home/chris/anaconda3/envs/4YP/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py",
> line 55, in default_collate
>     return torch.stack(batch, 0, out=out) TypeError: expected Tensor as element 3 in argument 0, but got tuple


title: Error when running Grad-CAM visualization

body:
I am using the open source Pytorch Grad-CAM code in order to visualize my model, which looks like this:
classifier = cnn.CNN(in_shape=(1,28,28), n_classes=10).cuda()

The Grad-CAM implementation is from here: https://github.com/jacobgil/pytorch-grad-cam/
However, when I run
def preprocess_image(img):
    means = [0.485, 0.456, 0.406]
    stds = [0.229, 0.224, 0.225]

    preprocessed_img = img.copy()[:, :, ::-1]
    for i in range(3):
        preprocessed_img[:, :, i] = preprocessed_img[:, :, i] - means[i]
        preprocessed_img[:, :, i] = preprocessed_img[:, :, i] / stds[i]
    preprocessed_img = \
        np.ascontiguousarray(np.transpose(preprocessed_img, (2, 0, 1)))
    preprocessed_img = torch.from_numpy(preprocessed_img)
    preprocessed_img.unsqueeze_(0)
    input = preprocessed_img.requires_grad_(True)
    return input

model = classifier
grad_cam = GradCam(model=model, feature_module=model.features, target_layer_names=["2"], use_cuda=True)

img = cv2.imread('/content/drive/My Drive/pic.png', 1)
img = np.float32(cv2.resize(img, (224, 224))) / 255
input = preprocess_image(img)
input = input[:, 0:1, :, :]

target_index = None
mask = grad_cam(input, target_index)
show_cam_on_image(img, mask)

I get this error:
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-30-fe930a4e6b09> in <module>()
    155 # Otherwise, targets the requested index.
    156 target_index = None
--> 157 mask = grad_cam(input, target_index)
    158 show_cam_on_image(img, mask)

6 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in linear(input, weight, bias)
   1674         ret = torch.addmm(bias, input, weight.t())
   1675     else:
-> 1676         output = input.matmul(weight.t())
   1677         if bias is not None:
   1678             output += bias

RuntimeError: mat1 dim 1 must match mat2 dim 0

In case this is helpful, here is the output of print(model.features):
Sequential(
  (0): Conv2d(1, 16, kernel_size=(1, 1), stride=(1, 1), bias=False)
  (1): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (2): ReLU(inplace=True)
  (3): Conv2d(16, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
  (4): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (5): ReLU(inplace=True)
  (6): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  (7): Conv2d(32, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
  (8): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (9): ReLU(inplace=True)
  (10): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  (11): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
  (12): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (13): ReLU(inplace=True)
  (14): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
)

How can I fix this error and make the grad-CAM run as expected?


title: IOError: [Errno 28] No space left on device while installing pytorch

body:
l got the following error when installing pytorch

 pip install http://download.pytorch.org/whl/cu80/torch-0.1.11.post5-cp27-none-linux_x86_64.whl
Collecting torch==0.1.11.post5 from http://download.pytorch.org/whl/cu80/torch-0.1.11.post5-cp27-none-linux_x86_64.whl
  Downloading http://download.pytorch.org/whl/cu80/torch-0.1.11.post5-cp27-none-linux_x86_64.whl (475.7MB)
100% |████████████████████████████████| 475.7MB 3.5MB/s

the following error

Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/usr/local/lib/python2.7/dist-packages/pip/commands/install.py", line 335, in run
    wb.build(autobuilding=True)
  File "/usr/local/lib/python2.7/dist-packages/pip/wheel.py", line 749, in build
    self.requirement_set.prepare_files(self.finder)
  File "/usr/local/lib/python2.7/dist-packages/pip/req/req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "/usr/local/lib/python2.7/dist-packages/pip/req/req_set.py", line 620, in _prepare_file
    session=self.session, hashes=hashes)
  File "/usr/local/lib/python2.7/dist-packages/pip/download.py", line 821, in unpack_url
    hashes=hashes
  File "/usr/local/lib/python2.7/dist-packages/pip/download.py", line 663, in unpack_http_url
    unpack_file(from_path, location, content_type, link)
  File "/usr/local/lib/python2.7/dist-packages/pip/utils/__init__.py", line 599, in unpack_file
    flatten=not filename.endswith('.whl')
  File "/usr/local/lib/python2.7/dist-packages/pip/utils/__init__.py", line 501, in unzip_file
    fp.write(data)
IOError: [Errno 28] No space left on device

What's wrong ?  l don't understand the error coming from /python2.7/dist-packages/pip/


title: Runtime error while executing code from google colab document for creating Deepfakes image animation

body:
enter image description hereI'm getting runtime error while executing code from google colab document for creating Deepfakes image animation.
RuntimeError                              Traceback (most recent call last)

<ipython-input-5-dbd18151b569> in <module>()
  1 from demo import load_checkpoints
  2 generator, kp_detector = load_checkpoints(config_path='config/vox-256.yaml',

----> 3                             checkpoint_path='/content/gdrive/My Drive/first-order-motion-model/vox-cpk.pth.tar')
10 frames
/usr/local/lib/python3.6/dist-packages/torch/cuda/__init__.py in _lazy_init()
188             raise AssertionError(

189                 "libcudart functions unavailable. It looks like you have a broken build?")

--> 190         torch._C._cuda_init()
191         # Some of the queued calls may reentrantly call _lazy_init();

192         # we need to just return without initializing in that case.

RuntimeError: cuda runtime error (100) : no CUDA-capable device is detected at /pytorch/aten/src/THC/THCGeneral.cpp:47


title: Pytorch torchvision MNIST download

body:
I'm new to Pytorch and torchvision. I followed a tutorial that is roughly a year old where he tried to download mnist via python and torchvision.

This is how:

import torch
from torchvision import datasets, transforms

kwargs = {'num_workers': 1, 'pin_memory': True}
train = torch.utils.data.DataLoader(
    datasets.MNIST('data', train=True, download=True,
                   transform=transforms.Compose([transforms.ToTensor(),
                   transforms.Normalize((0.1307,), (0.3081,))])),
    batch_size=64, shuffle=True, **kwargs)
test = torch.utils.data.DataLoader(
    datasets.MNIST('data', train=False,
                   transform=transforms.Compose([transforms.ToTensor(),
                   transforms.Normalize((0.1307,), (0.3081,))])),
    batch_size=64, shuffle=True, **kwargs)

Now my problem is that I get this error:

  Traceback (most recent call last):

  Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz to data\MNIST\raw\train-images-idx3-ubyte.gz

  File "C:/Users/Nico/PycharmProjects/PyTorch/mnist.py", line 13, in
      transforms.Normalize((0.1307,), (0.3081,))])),

  File "C:\Users\Nico\AppData\Local\Programs\Python\Python37\lib\site-packages\torchvision\datasets\mnist.py", line 68, in init
      self.download()

  File "C:\Users\Nico\AppData\Local\Programs\Python\Python37\lib\site-packages\torchvision\datasets\mnist.py", line 143, in download
      download_url(url, root=self.raw_folder, filename=filename, md5=None)

  File "C:\Users\Nico\AppData\Local\Programs\Python\Python37\lib\site-packages\torchvision\datasets\utils.py", line 73, in download_url
      reporthook=gen_bar_updater(tqdm())

  TypeError: init() missing 1 required positional argument: 'total'

Do any of you guys know what I have to change, or how I can download/use them? As I said earlier I'm new to it and I don't have any clue.

I hope you guys can help me, thanks in advance.

Greetings Nico aka. Myridor


title: How do I compile a pytorch script into an exe file with small size?

body:
I have a semantic segmentation model using PyTorch. In order to participate in a competition, I am compiling the test.py to an exe file with PyInstaller and UPX. Although the resulting executable file runs correctly, its size is nearly 800MB. How do I make it smaller?

This is my test.py:

from torch import nn
from torch.autograd import Variable as V
from torch import Tensor
from torch import cuda
from torch import load

import cv2
import os
import numpy as np

from time import time

from networks.unet import Unet
# from networks.dunet import Dunet
# from networks.dinknet import LinkNet34, DinkNet34, DinkNet50, DinkNet101, DinkNet34_less_pool
# from networks.dinkbranch import DinkBranch50, DinkBranch34

BATCHSIZE_PER_CARD = 2

class TTAFrame():
    def __init__(self, net):
        self.net = net().cuda()
        self.net = nn.DataParallel(self.net, device_ids=range(cuda.device_count()))

    def test_one_img_from_path(self, path, evalmode = True):
        if evalmode:
            self.net.eval()
        batchsize = cuda.device_count() * BATCHSIZE_PER_CARD
        if batchsize >= 8:
            return self.test_one_img_from_path_1(path)
        elif batchsize >= 4:
            return self.test_one_img_from_path_2(path)
        elif batchsize >= 2:
            return self.test_one_img_from_path_4(path)

    def test_one_img_from_path_8(self, path):
        img = cv2.imread(path)#.transpose(2,0,1)[None]
        img90 = np.array(np.rot90(img))
        img1 = np.concatenate([img[None],img90[None]])
        img2 = np.array(img1)[:,::-1]
        img3 = np.array(img1)[:,:,::-1]
        img4 = np.array(img2)[:,:,::-1]

        img1 = img1.transpose(0,3,1,2)
        img2 = img2.transpose(0,3,1,2)
        img3 = img3.transpose(0,3,1,2)
        img4 = img4.transpose(0,3,1,2)

        img1 = V(Tensor(np.array(img1, np.float32)/255.0 * 3.2 -1.6).cuda())
        img2 = V(Tensor(np.array(img2, np.float32)/255.0 * 3.2 -1.6).cuda())
        img3 = V(Tensor(np.array(img3, np.float32)/255.0 * 3.2 -1.6).cuda())
        img4 = V(Tensor(np.array(img4, np.float32)/255.0 * 3.2 -1.6).cuda())

        maska = self.net.forward(img1).squeeze().cpu().data.numpy()
        maskb = self.net.forward(img2).squeeze().cpu().data.numpy()
        maskc = self.net.forward(img3).squeeze().cpu().data.numpy()
        maskd = self.net.forward(img4).squeeze().cpu().data.numpy()

        mask1 = maska + maskb[:,::-1] + maskc[:,:,::-1] + maskd[:,::-1,::-1]
        mask2 = mask1[0] + np.rot90(mask1[1])[::-1,::-1]

        return mask2

    def test_one_img_from_path_4(self, path):
        img = cv2.imread(path)#.transpose(2,0,1)[None]
        img90 = np.array(np.rot90(img))
        img1 = np.concatenate([img[None],img90[None]])
        img2 = np.array(img1)[:,::-1]
        img3 = np.array(img1)[:,:,::-1]
        img4 = np.array(img2)[:,:,::-1]

        img1 = img1.transpose(0,3,1,2)
        img2 = img2.transpose(0,3,1,2)
        img3 = img3.transpose(0,3,1,2)
        img4 = img4.transpose(0,3,1,2)

        img1 = V(Tensor(np.array(img1, np.float32)/255.0 * 3.2 -1.6).cuda())
        img2 = V(Tensor(np.array(img2, np.float32)/255.0 * 3.2 -1.6).cuda())
        img3 = V(Tensor(np.array(img3, np.float32)/255.0 * 3.2 -1.6).cuda())
        img4 = V(Tensor(np.array(img4, np.float32)/255.0 * 3.2 -1.6).cuda())

        maska = self.net.forward(img1).squeeze().cpu().data.numpy()
        maskb = self.net.forward(img2).squeeze().cpu().data.numpy()
        maskc = self.net.forward(img3).squeeze().cpu().data.numpy()
        maskd = self.net.forward(img4).squeeze().cpu().data.numpy()

        mask1 = maska + maskb[:,::-1] + maskc[:,:,::-1] + maskd[:,::-1,::-1]
        mask2 = mask1[0] + np.rot90(mask1[1])[::-1,::-1]

        return mask2

    def test_one_img_from_path_2(self, path):
        img = cv2.imread(path)#.transpose(2,0,1)[None]
        img90 = np.array(np.rot90(img))
        img1 = np.concatenate([img[None],img90[None]])
        img2 = np.array(img1)[:,::-1]
        img3 = np.concatenate([img1,img2])
        img4 = np.array(img3)[:,:,::-1]
        img5 = img3.transpose(0,3,1,2)
        img5 = np.array(img5, np.float32)/255.0 * 3.2 -1.6
        img5 = V(Tensor(img5).cuda())
        img6 = img4.transpose(0,3,1,2)
        img6 = np.array(img6, np.float32)/255.0 * 3.2 -1.6
        img6 = V(Tensor(img6).cuda())

        maska = self.net.forward(img5).squeeze().cpu().data.numpy()#.squeeze(1)
        maskb = self.net.forward(img6).squeeze().cpu().data.numpy()

        mask1 = maska + maskb[:,:,::-1]
        mask2 = mask1[:2] + mask1[2:,::-1]
        mask3 = mask2[0] + np.rot90(mask2[1])[::-1,::-1]

        return mask3

    def test_one_img_from_path_1(self, path):
        img = cv2.imread(path)#.transpose(2,0,1)[None]

        img90 = np.array(np.rot90(img))
        img1 = np.concatenate([img[None],img90[None]])
        img2 = np.array(img1)[:,::-1]
        img3 = np.concatenate([img1,img2])
        img4 = np.array(img3)[:,:,::-1]
        img5 = np.concatenate([img3,img4]).transpose(0,3,1,2)
        img5 = np.array(img5, np.float32)/255.0 * 3.2 -1.6
        img5 = V(Tensor(img5).cuda())

        mask = self.net.forward(img5).squeeze().cpu().data.numpy()#.squeeze(1)
        mask1 = mask[:4] + mask[4:,:,::-1]
        mask2 = mask1[:2] + mask1[2:,::-1]
        mask3 = mask2[0] + np.rot90(mask2[1])[::-1,::-1]

        return mask3

    def load(self, path):
        self.net.load_state_dict(load(path))

#source = 'dataset/test/'
import sys
if len(sys.argv) < 2:
    arg1 = r'dataset/504/original'
else:
    arg1 = sys.argv[1]

# source = r'dataset/504/original'
source = arg1
source_path = os.path.join(os.getcwd(), source)

val = os.listdir(source_path)
solver = TTAFrame(Unet)
model_path = '/'
model_path = r'weights/log02_Unet.th'
solver.load(os.path.join(os.getcwd(), model_path))

tic = time()
target = r'submits/log02_baseline504'

target_path = os.path.join(os.getcwd(), target)
if os.path.exists(target_path):
    pass
else:
    os.makedirs(target_path)
for i,name in enumerate(val):
    if i%10 == 0:
        print(i/10, '    ','%.2f'%(time()-tic))
    mask = solver.test_one_img_from_path(os.path.join(source_path, name))
    mask[mask>4.0] = 255
    mask[mask<=4.0] = 0
    mask = np.concatenate([mask[:,:,None],mask[:,:,None],mask[:,:,None]],axis=2)
    cv2.imwrite(target_path+r'/'+name[:-7]+'mask.png', mask.astype(np.uint8))

This is the 'Unet' file:

from torch import autograd, cat
from torch import nn

class Unet(nn.Module):
    def __init__(self):
        super(Unet, self).__init__()

        self.down1 = self.conv_stage(3, 8)
        self.down2 = self.conv_stage(8, 16)
        self.down3 = self.conv_stage(16, 32)
        self.down4 = self.conv_stage(32, 64)
        self.down5 = self.conv_stage(64, 128)
        self.down6 = self.conv_stage(128, 256)
        self.down7 = self.conv_stage(256, 512)

        self.center = self.conv_stage(512, 1024)
        #self.center_res = self.resblock(1024)

        self.up7 = self.conv_stage(1024, 512)
        self.up6 = self.conv_stage(512, 256)
        self.up5 = self.conv_stage(256, 128)
        self.up4 = self.conv_stage(128, 64)
        self.up3 = self.conv_stage(64, 32)
        self.up2 = self.conv_stage(32, 16)
        self.up1 = self.conv_stage(16, 8)

        self.trans7 = self.upsample(1024, 512)
        self.trans6 = self.upsample(512, 256)
        self.trans5 = self.upsample(256, 128)
        self.trans4 = self.upsample(128, 64)
        self.trans3 = self.upsample(64, 32)
        self.trans2 = self.upsample(32, 16)
        self.trans1 = self.upsample(16, 8)

        self.conv_last = nn.Sequential(
            nn.Conv2d(8, 1, 3, 1, 1),
            nn.Sigmoid()
        )

        self.max_pool = nn.MaxPool2d(2)

        for m in self.modules():
            if isinstance(m, nn.Conv2d) or isinstance(m, nn.ConvTranspose2d):
                if m.bias is not None:
                    m.bias.data.zero_()

    def conv_stage(self, dim_in, dim_out, kernel_size=3, stride=1, padding=1, bias=True, useBN=False):
        if useBN:
            return nn.Sequential(
              nn.Conv2d(dim_in, dim_out, kernel_size=kernel_size, stride=stride, padding=padding, bias=bias),
              nn.BatchNorm2d(dim_out),
              #nn.LeakyReLU(0.1),
              nn.ReLU(),
              nn.Conv2d(dim_out, dim_out, kernel_size=kernel_size, stride=stride, padding=padding, bias=bias),
              nn.BatchNorm2d(dim_out),
              #nn.LeakyReLU(0.1),
              nn.ReLU(),
            )
        else:
            return nn.Sequential(
              nn.Conv2d(dim_in, dim_out, kernel_size=kernel_size, stride=stride, padding=padding, bias=bias),
              nn.ReLU(),
              nn.Conv2d(dim_out, dim_out, kernel_size=kernel_size, stride=stride, padding=padding, bias=bias),
              nn.ReLU()
            )

    def upsample(self, ch_coarse, ch_fine):
        return nn.Sequential(
            nn.ConvTranspose2d(ch_coarse, ch_fine, 4, 2, 1, bias=False),
            nn.ReLU()
        )

    def forward(self, x):
        conv1_out = self.down1(x)
        conv2_out = self.down2(self.max_pool(conv1_out))
        conv3_out = self.down3(self.max_pool(conv2_out))
        conv4_out = self.down4(self.max_pool(conv3_out))
        conv5_out = self.down5(self.max_pool(conv4_out))
        conv6_out = self.down6(self.max_pool(conv5_out))
        conv7_out = self.down7(self.max_pool(conv6_out))

        out = self.center(self.max_pool(conv7_out))
        #out = self.center_res(out)

        out = self.up7(cat((self.trans7(out), conv7_out), 1))
        out = self.up6(cat((self.trans6(out), conv6_out), 1))
        out = self.up5(cat((self.trans5(out), conv5_out), 1))
        out = self.up4(cat((self.trans4(out), conv4_out), 1))
        out = self.up3(cat((self.trans3(out), conv3_out), 1))
        out = self.up2(cat((self.trans2(out), conv2_out), 1))
        out = self.up1(cat((self.trans1(out), conv1_out), 1))

        out = self.conv_last(out)

        return out


title: KeyError when enumerating over dataloader - why?

body:
I am writing a binary classification model that consists of audio files of 40 participants and classifies them according to whether they have a speech disorder or not. The audio files have been divided into 5 second segments and to avoid subject bias, I have split the training/testing/validation sets such that a subject only appears in one set (i.e. participant ID02 does not appear in both the training and testing sets). The following error appears when I attempt to enumerate over the DataLoader validLoader in the code below and I'm not entirely sure why this error is occurring. Does anyone have any advice?
KeyError                                  Traceback (most recent call last)
<ipython-input-69-55be99283cf7> in <module>()
----> 1 for i, data in enumerate(valid_loader, 0):
      2   images, labels = data
      3   print("Batch", i, "size:", len(images))

3 frames
/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py in __next__(self)
    361
    362     def __next__(self):
--> 363         data = self._next_data()
    364         self._num_yielded += 1
    365         if self._dataset_kind == _DatasetKind.Iterable and \

/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py in _next_data(self)
    987             else:
    988                 del self._task_info[idx]
--> 989                 return self._process_data(data)
    990
    991     def _try_put_index(self):

/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py in _process_data(self, data)
   1012         self._try_put_index()
   1013         if isinstance(data, ExceptionWrapper):
-> 1014             data.reraise()
   1015         return data
   1016

/usr/local/lib/python3.6/dist-packages/torch/_utils.py in reraise(self)
    393             # (https://bugs.python.org/issue2651), so we work around it.
    394             msg = KeyErrorMessage(msg)
--> 395         raise self.exc_type(msg)

KeyError: Caught KeyError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/worker.py", line 185, in _worker_loop
    data = fetcher.fetch(index)
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "<ipython-input-44-245be0a1e978>", line 19, in __getitem__
    x = Image.open(self.df['path'][index])
  File "/usr/local/lib/python3.6/dist-packages/pandas/core/series.py", line 871, in __getitem__
    result = self.index.get_value(self, key)
  File "/usr/local/lib/python3.6/dist-packages/pandas/core/indexes/base.py", line 4405, in get_value
    return self._engine.get_value(s, k, tz=getattr(series.dtype, "tz", None))
  File "pandas/_libs/index.pyx", line 80, in pandas._libs.index.IndexEngine.get_value
  File "pandas/_libs/index.pyx", line 90, in pandas._libs.index.IndexEngine.get_value
  File "pandas/_libs/index.pyx", line 138, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 998, in pandas._libs.hashtable.Int64HashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 1005, in pandas._libs.hashtable.Int64HashTable.get_item
KeyError: 36

Can anyone advise why this is happening?
from google.colab import drive
drive.mount('/content/drive')

import torch
import torchvision
import torch.optim as optim
import torch.nn as nn
import torchvision.transforms as transforms
from torchvision import utils
from  torch.utils.data import Dataset

from sklearn.metrics import confusion_matrix
from skimage import io, transform, data
from skimage.color import rgb2gray

import matplotlib.pyplot as plt
from tqdm import tqdm
from PIL import Image
import pandas as pd
import numpy as np
import csv
import os
import math
import cv2

root_dir = "/content/drive/My Drive/Read_Text/5_Second_Segments/"
class_names = [
  "Parkinsons_Disease",
  "Healthy_Control"
]

def get_meta(root_dir, dirs):
    """ Fetches the meta data for all the images and assigns labels.
    """
    paths, classes = [], []
    for i, dir_ in enumerate(dirs):
        for entry in os.scandir(root_dir + dir_):
            if (entry.is_file()):
                paths.append(entry.path)
                classes.append(i)

    return paths, classes

paths, classes = get_meta(root_dir, class_names)

data = {
    'path': paths,
    'class': classes
}

data_df = pd.DataFrame(data, columns=['path', 'class'])
data_df = data_df.sample(frac=1).reset_index(drop=True) # Shuffles the data

from pandas import option_context

print("Found", len(data_df), "images.")

with option_context('display.max_colwidth', 400):
    display(data_df.head(100))

class Audio(Dataset):

    def __init__(self, df, transform=None):
        """
        Args:
            image_dir (string): Directory with all the images
            df (DataFrame object): Dataframe containing the images, paths and classes
            transform (callable, optional): Optional transform to be applied
                on a sample.
        """
        self.df = df
        self.transform = transform

    def __len__(self):
        return len(self.df)

    def __getitem__(self, index):
        # Load image from path and get label
        x = Image.open(self.df['path'][index])
        try:
          x = x.convert('RGB') # To deal with some grayscale images in the data
        except:
          pass
        y = torch.tensor(int(self.df['class'][index]))

        if self.transform:
            x = self.transform(x)

        return x, y

def compute_img_mean_std(image_paths):
    """
        Author: @xinruizhuang. Computing the mean and std of three channel on the whole dataset,
        first we should normalize the image from 0-255 to 0-1
    """

    img_h, img_w = 224, 224
    imgs = []
    means, stdevs = [], []

    for i in tqdm(range(len(image_paths))):
        img = cv2.imread(image_paths[i])
        img = cv2.resize(img, (img_h, img_w))
        imgs.append(img)

    imgs = np.stack(imgs, axis=3)
    print(imgs.shape)

    imgs = imgs.astype(np.float32) / 255.

    for i in range(3):
        pixels = imgs[:, :, i, :].ravel()  # resize to one row
        means.append(np.mean(pixels))
        stdevs.append(np.std(pixels))

    means.reverse()  # BGR --> RGB
    stdevs.reverse()

    print("normMean = {}".format(means))
    print("normStd = {}".format(stdevs))
    return means, stdevs

norm_mean, norm_std = compute_img_mean_std(paths)

data_transform = transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(256),
        transforms.ToTensor(),
        transforms.Normalize(norm_mean, norm_std),
    ])

unique_users = data_df['path'].str[-20:-16].unique()
train_users, test_users = np.split(np.random.permutation(unique_users), [int(0.8*len(unique_users))])
df_train = data_df[data_df['path'].str[-20:-16].isin(train_users)]
test_data_df = data_df[data_df['path'].str[-20:-16].isin(test_users)]

train_unique_users = df_train['path'].str[-20:-16].unique()
train_users, validate_users = np.split(np.random.permutation(train_unique_users), [int(0.875*len(train_unique_users))])
train_data_df = df_train[df_train['path'].str[-20:-16].isin(train_users)]
valid_data_df = df_train[df_train['path'].str[-20:-16].isin(validate_users)]

ins_dataset_train = Audio(
    df=train_data_df,
    transform=data_transform,
)

ins_dataset_valid = Audio(
    df=valid_data_df,
    transform=data_transform,
)

ins_dataset_test = Audio(
    df=test_data_df,
    transform=data_transform,
)

train_loader = torch.utils.data.DataLoader(
    ins_dataset_train,
    batch_size=8,
    shuffle=True,
    num_workers=2
)

test_loader = torch.utils.data.DataLoader(
    ins_dataset_test,
    batch_size=16,
    shuffle=True,
    num_workers=2
)

valid_loader = torch.utils.data.DataLoader(
    ins_dataset_valid,
    batch_size=16,
    shuffle=True,
    num_workers=2
)

//(This is where the error is occurring.)
for i, data in enumerate(valid_loader, 0):
  images, labels = data
  print("Batch", i, "size:", len(images))


title: RuntimeError: CUDA error: out of memory - DCGAN with image_size = 256

body:
I am trying to implement a DCGAN with image_size = 256 (using PyTorch). I already implemented the generator and discriminator codes, in the following:

class Generator(nn.Module):
def init(self, ngpu):
super(Generator, self).init()
self.ngpu = ngpu
self.main = nn.Sequential(
# input is Z, going into a convolution
nn.ConvTranspose2d( nz, ngf * 32, 4, 1, 0, bias=False),
nn.BatchNorm2d(ngf * 32),
nn.ReLU(True),
# state size. (ngf32) x 4 x 4
nn.ConvTranspose2d( ngf * 32, ngf * 16, 4, 2, 0, bias=False),
nn.BatchNorm2d(ngf * 16),
nn.ReLU(True),
# state size. (ngf16) x 8 x 8
nn.ConvTranspose2d(ngf * 16, ngf * 8, 4, 2, 1, bias=False),
nn.BatchNorm2d(ngf * 8),
nn.ReLU(True),
# state size. (ngf8) x 16 x 16
nn.ConvTranspose2d(ngf * 8, ngf * 4, 4, 2, 1, bias=False),
nn.BatchNorm2d(ngf * 4),
nn.ReLU(True),
# state size. (ngf4) x 32 x 32
nn.ConvTranspose2d(ngf * 4, ngf * 2, 4, 2, 1, bias=False),
nn.BatchNorm2d(ngf * 2),
nn.ReLU(True),
# state size. (ngf*2) x 64 x 64
nn.ConvTranspose2d(ngf * 2, ngf, 4, 2, 1, bias=False),
nn.BatchNorm2d(ngf),
nn.ReLU(True),
# state size. (ngf) x 128 x 128
nn.ConvTranspose2d( ngf, nc, 4, 2, 1, bias=False),
nn.Tanh()
# state size. (nc) x 256 x 256
)

class Discriminator(nn.Module):
def init(self, ngpu):
super(Discriminator, self).init()
self.ngpu = ngpu
self.main = nn.Sequential(
# input is (nc) x 256 x 256
nn.Conv2d(nc, ndf, 4, stride=2, padding=1, bias=False),
nn.LeakyReLU(0.2, inplace=True),
# state size. (ndf) x 128 x 128
nn.Conv2d(ndf, ndf * 2, 4, stride=2, padding=1, bias=False),
nn.BatchNorm2d(ndf * 2),
nn.LeakyReLU(0.2, inplace=True),
# state size. (ndf2) x 64 x 64
nn.Conv2d(ndf * 2, ndf * 4, 4, stride=2, padding=1, bias=False),
nn.BatchNorm2d(ndf * 4),
nn.LeakyReLU(0.2, inplace=True),
# state size. (ndf4) x 32 x 32
nn.Conv2d(ndf * 4, ndf * 8, 4, stride=2, padding=1, bias=False),
nn.BatchNorm2d(ndf * 8),
nn.LeakyReLU(0.2, inplace=True),
# state size. (ndf8) x 16 x 16
nn.Conv2d(ndf * 8, ndf * 16, 4, stride=2, padding=1, bias=False),
nn.BatchNorm2d(ndf * 16),
nn.LeakyReLU(0.2, inplace=True),
# state size. (ndf6) x 8 x 8
nn.Conv2d(ndf * 16, ndf * 32, 4, stride=2, padding=1, bias=False),
nn.BatchNorm2d(ndf * 32),
nn.LeakyReLU(0.2, inplace=True),
# state size. (ndf*32) x 4 x 4
nn.Conv2d(ndf * 32, 1, 4, stride=1, padding=0, bias=False),
nn.Sigmoid()
# state size. 1
)

After reading something at the GAN forums, I found that the batch_size must be low, considering I am using a GTX 1050 Ti with 4GB of memory (actually, my batch_size variable is set to 5).

I am finding the error:

RuntimeError: CUDA error: out of memory

Can somebody help me with this?

Update 2: Resizing code

# Create the dataset
dataset = dset.ImageFolder(root=dataroot,
                       transform=transforms.Compose([
                           transforms.Resize(image_size),
                           transforms.CenterCrop(image_size),
                           transforms.ToTensor(),
                           transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),
                       ]))
# Create the dataloader
dataloader = torch.utils.data.DataLoader(dataset, batch_size=batch_size,
                                     shuffle=True, num_workers=workers)


title: How to return back to cpu from gpu in pytorch?

body:
I have a text classifier in pytorch and I want to use GPUs to increase running speed.
I have used this part of code to check CUDA and use it:

if torch.cuda.device_count() > 1:
    print("Let's use", torch.cuda.device_count(), "GPUs!")
    my_rnn_model = nn.DataParallel(my_rnn_model)
if torch.cuda.is_available():
    my_rnn_model.cuda()

Now I want to return back to use cpu (instead of gpu). So I cleared this part of code. But it does’nt work and I receive this error:

RuntimeError: cuda runtime error (8) : invalid device function at /opt/conda/conda-bld/pytorch_1503963423183/work/torch/lib/THC/THCTensorCopy.cu:204

Would you please guide me how can I return back to cpu running?


title: requires_grad of params is True even with torch.no_grad()

body:
I am experiencing a strange problem with PyTorch today.

When checking network parameters in the with scope, I am expecting requires_grad to be False, but apparently this is not the case unless I explicitly set all params myself.

Code

Link to Net -> Gist

net = InceptionResnetV2()

with torch.no_grad():

    for name, param in net.named_parameters():
        print("{} {}".format(name, param.requires_grad))

The above code will tell me all the params are still requiring grad, unless I explicitly specify param.requires_grad = False.

My torch version: 1.0.1.post2


title: ImportError: cannot import name 'parameter_parser' from 'parser' (unknown location)

body:
I am trying to Import Parameter_parser from Parser. but it is showing the error below:

ImportError: cannot import name 'parameter_parser' from 'parser'

In the line below that I also get:

ModuleNotFoundError: No module named 'load_data'

This is my code:

    import matplotlib
    matplotlib.use('agg')
    import numpy as np
    import time
    import os
    import torch.utils.data
    import torch.nn.functional as F
    import torch.optim as optim
    import torch.optim.lr_scheduler as lr_scheduler
    from torch.utils.data import DataLoader
    from os.path import join as pjoin
    from parser import parameter_parser
    from load_data import split_ids, GraphData, collate_batch
    from models.gcn_modify import GCN_MODIFY
    from models.gcn_origin import GCN_ORIGIN
    from models.gat import GAT
    from models.mgcn import MGCN
    from sklearn import metrics`


title: How do I pass a color image through a Pytorch convolutional layer with custom filter?

body:
I'm trying to visualize what happens when a color image passes through a convolutional layer. For that, I'm setting custom weights with zeroes and ones. The problem I'm facing is that I'm losing the 3D channels, and get a 1D channel after passing the data through the layer.

import requests
from io import BytesIO
from PIL import Image
import torch
import torch.nn as nn
import matplotlib.pyplot as plt
import numpy as np

link = 'https://audimediacenter-a.akamaihd.net/system/production/media/85094/images' \
       '/2a4e98976b1f9088fe6ae883f2f29e4d8f3ed473/A1912967_x500.jpg?1575885688'

r = requests.get(link, timeout=10)
im = Image.open(BytesIO(r.content))

pic = np.array(im)

horizontal_filter = torch.zeros(5, 5)
horizontal_filter[2, :] = 1

print(horizontal_filter)

This is my custom filter:

tensor([[0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0.],
        [1., 1., 1., 1., 1.],
        [0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0.]])

Now I'm using the custom filter and repeat it to fit 3 channels.

hz = nn.Conv2d(in_channels=3,
               out_channels=3,
               kernel_size=5,
               stride=1,
               bias=None)

hz.weight.data =  horizontal_filter.type('torch.FloatTensor').repeat(1, 3, 1, 1)

print(hz.weight.data.shape)

This is the shape of the filter:

torch.Size([1, 3, 5, 5])

I pass it through the convolutional filter and I'm losing the 3 channels:

zz = hz(torch.tensor(pic[None, ...]).permute(0, 3, 1, 2).type('torch.FloatTensor'))

print(np.transpose(zz.detach().numpy(), (0, 2, 3, 1)).shape)

(1, 329, 496, 1)

If I plot it, I don't have the colors anymore.

z = np.transpose(zz.detach().numpy(), (0, 2, 3, 1))[0, :, :, 0]

f, axarr = plt.subplots()
axarr.imshow(z)
plt.show()

tl;dr: How do I pass a 3D picture through a convolutional layer and return an image with 3 channels?


title: Hugging face: tokenizer for masked lm question

body:
I am using transformer version 3.0.0 for my project and have some questions.
I want to use a bert model with masked lm pretraining for protein sequences.
To get a character level tokenizer I derived from the BertTokenizer
from transformers import BertTokenizer
class DerivedBertTok(BertTokenizer):
    def __init__(self, **kwargs):
        super().__init__(**kwargs)
    def tokenize(self, text):
        if isinstance(text, np.ndarray):
            assert len(text) == 1
            text = text[0]
        return [x if x in self.vocab else self.unk_token for x in text]

my vocab looks like this
[PAD]
[CLS]
[SEP]
[UNK]
[MASK]
A
R
N
D
B
C
E
Q
Z
G
H
I
L
K
M
F
P
S
T
W
Y
V

The usage seems quite similar to what i have seen in the docs:
d_tokenizer = DerivedBertTok(
    vocab_file=vocab_path,
    do_lower_case=False,
    do_basic_tokenize=False,
    tokenize_chinese_chars=False
)
d_tokenizer.encode_plus(np.array(["AXEF"])[0],
                      max_length=20,
                      pad_to_max_length=True,
                      add_special_tokens=True,
                      truncation=True,
                      return_tensors='pt')

From this I was building a pytorch Dataset with a custom collate function.
all the collate function does is taking all input tensors and stacking them
from transformers import BatchEncoding
    def collate_fn(self, batch):
        # this function will not work for higher dimension inputs
        elem = batch[0]
        elem_type = type(elem)
        if isinstance(elem, BatchEncoding):
            new_shapes = {key: (len(batch), value.shape[1]) for key, value in elem.items()}
            outs = {key: value.new_empty(new_shapes[key]) for key, value in elem.items()}
            if torch.utils.data.get_worker_info() is not None:
                [v.share_memory_() for v in outs.values()]
            return {key: torch.stack(tuple((d[key].view(-1) for d in batch)), 0, out=outs[key]) for key in elem.keys()}
        else:
            raise ValueError(f"type: {elem_type} not understood")

Question 1:
So I was wondering if the BatchEncoding or another class is already capable of doing this (and doing it possibly better?). Or using a different Dataset/ DataLoader class altogether.
Question 2:
Additionally, I want to mask some of the Inputs as required for the masked LM, however I did not manage find any implementation in the transformer library. Are there any recommendations for doing this?


title: PyTorch out of GPU memory just after initialization

body:
I built my model in PyTorch. The code works well on CPU. However, it seems to be running out of GPU memory just after initializing the network and switching it to cuda.

The failed code is:

model = init_from_scratch(args, train_exs, dev_exs)
model.init_optimizer()

nParams= sum([np.prod(list(p.size())) for p in model.network.parameters()])
print('* total number of parameters:',nParams)

device = torch.device("cuda:"+str(args.gpu) if args.cuda else "cpu")
model.set_device(device) #call last!!

# DATA ITERATORS  (not run to here)
train_dataset = data.ReaderDataset()
train_sampler = torch.utils.data.sampler.RandomSampler()
train_loader = torch.utils.data.DataLoader()
...

# TRAIN (not run to here)
for epoch in range(start_epoch, args.num_epochs):
....

def set_device(self,device):
    print("device",device)
    self.use_cuda = False if str(device) == "cpu" else True
    self.network = self.network.to(device)

total number of parameters: 30708906
Torch version is '0.4.0'
TITAN Xp with 12G GPU memory and nvidia-smi shows it free

So, what's wrong with my code?

The error output is:

Traceback (most recent call last):
File "script/train2.0.py", line 683, in <module>
  main(args)
File "script/train2.0.py", line 542, in main
  model.set_device(device)
File "./model.py", line 588, in set_device
  self.network = self.network.to(device)
File "/home/username/.local/lib/python3.5/site-packages/torch/nn/modules/module.py", line 393, in to
  return self._apply(lambda t: t.to(device))
File "/home/username/.local/lib/python3.5/site-packages/torch/nn/modules/module.py", line 176, in _apply
  module._apply(fn)
File "/home/username/.local/lib/python3.5/site-packages/torch/nn/modules/module.py", line 182, in _apply
  param.data = fn(param.data)
File "/home/username/.local/lib/python3.5/site-packages/torch/nn/modules/module.py", line 393, in <lambda>
  return self._apply(lambda t: t.to(device))
File "/home/username/.local/lib/python3.5/site-packages/torch/cuda/__init__.py", line 161, in _lazy_init
  torch._C._cuda_init()

RuntimeError: cuda runtime error (2) : out of memory at /pytorch/aten/src/THC/THCTensorRandom.cu:25


title: How to display more than 10 images in Tensorboard?

body:
I noticed that it doesn't matter how many image I save to the tensorboard log file, tensorboard will only ever show 10 of them (per tag).

  How can we increase the number of images or at least select which ones are displayed?

To reproduce what I mean run following MCVE:

import torch
from torch.utils.tensorboard import SummaryWriter
tb = SummaryWriter(comment="test")

for k in range(100):
    # create an image with some funny pattern
    b = [n for (n, c) in enumerate(bin(k)) if c == '1']
    img = torch.zeros((1,10,10))
    img[0, b, :] = 0.5
    img =img + img.permute([0, 2, 1])

    # add the image to the tensorboard file
    tb.add_image(tag="test", img_tensor=img, global_step=k)

This creates a folder runs in which the data is saved. From the same folder execute tensorboard --logdir runs, open the browser and go to localhost:6006 (or replace 6006 with whatever port tensorboard happens to display after starting it). Then go to the tab called "images" and move the slider above the grayscale image.

In my case it only displayed the images from steps

k = 3, 20, 24, 32, 37, 49, 52, 53, 67, 78

which isn't even an nice even spacing, but looks pretty random. I'd prefer to have

see more than just 10 of the images I saved, and
have a more even spacing of number of steps between each image displayed.

How can I achieve this?

EDIT: I just found the option --samples_per_plugin and tried tensorboard --logdir runs --samples_per_plugin "images=100". This indeed increased the number of images, but it only showed the images from steps k = 0,1,2,3....,78, but none from above 78.


title: Expected device cuda:0 but got device cpu in PyTorch when I have already assigned the device to be cuda

body:
I have a following neural network code, and I get "Expected device cuda:0 but got device cpu in PyTorch" error and I can't figure out why. I assign the device to be cuda and the print line returns cuda. I've tried assign the device as device = cuda:0 as well just in case but that had no effect. Here's the code:
def run():
    torch.multiprocessing.freeze_support()
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    print(device)

    metabolites = pd.read_excel("testmetabolitedata.xlsx")
    subject_metadata = pd.read_excel("testsubj.xlsx")
    metabolitesdf = pd.DataFrame(data=metabolites)
    metabolitesdf = metabolitesdf.iloc[:, 1:9153]
    subjectsdf = pd.DataFrame(data=subject_metadata)

    n_samples, n_metabolites = metabolitesdf.shape
    print(n_samples)

    #genotypes of the target gene
    print(subjectsdf['SLCO1B1_rs4149056'])
    genotypes = subjectsdf['SLCO1B1_rs4149056']
    print(genotypes)
    # print('{} unique genotypes'.format(len(set(genotypes))))

    labels = [1 if g == 1 else 0 for g in genotypes]
    print('{} samples with genotype 1 out of {}  samples ({:.1%})'.format(sum(labels), len(labels),
                                                    sum(labels) / len(labels)))

    #Insert 0 into index 0 (first) into the list for the first row with column names
    labels.insert(0, 0)
    #log transform
    log_metabol = np.log10(metabolitesdf + 1)

    #Split data into training and validation 70% / 30%
    data = torch.utils.data.TensorDataset(torch.Tensor(np.array(log_metabol)),
                                                    torch.Tensor(labels))
    train, val = torch.utils.data.random_split(data, [int(0.7 * len(data)),
                                                                len(data) - int(0.7 * len(data))])

    print('{:.0f}/{} training/total ({:.1%}) in training set, {:.0f}/{} val/total ({:.1%}) in validation set'.format(\
        train[:][1].sum(), len(train), train[:][1].sum() / len(train),
        val[:][1].sum(), len(val), val[:][1].sum() / len(val)))

    class MultiLayerPredictor(torch.nn.Module):
        def __init__(self, input_shape, output_shape=1, hidden_dim=1024, **kwargs):
            super().__init__()
            self.fc1 = torch.nn.Linear(in_features=input_shape, out_features=hidden_dim)
            self.bn1 = torch.nn.BatchNorm1d(hidden_dim)
            self.fc2 = torch.nn.Linear(in_features=hidden_dim, out_features=hidden_dim)
            self.bn2 = torch.nn.BatchNorm1d(hidden_dim)
            self.fc3 = torch.nn.Linear(in_features=hidden_dim, out_features=output_shape)

        def forward(self, x):
            l1 = torch.relu(self.bn1(self.fc1(x)))
            l2 = torch.relu(self.bn2(self.fc2(l1)))
            return torch.sigmoid(self.fc3(l2)).reshape(-1)

    #load the training and validation sets
    print("Load training and validation data ")
    train_loader = torch.utils.data.DataLoader(train, batch_size=128,
                                                shuffle=True, num_workers=10, pin_memory=True)
    val_loader = torch.utils.data.DataLoader(val, batch_size=128,
                                                shuffle=False, num_workers=10, pin_memory=True)

    print("Loading complete, create model")
    model3 = MultiLayerPredictor(input_shape=n_metabolites).to(device)
    print("Model created! Moving to optimizer")
    optimizer3 = torch.optim.SGD(model3.parameters(), lr=1e-2)
    print("Optimizer done")
    objective3 = torch.nn.BCELoss()
    epochs = 30

    print_stats_interval = 10
    log3 = []
    print("Moving to training loop")

    for epoch in range(epochs):
        loss = n_correct = 0
        model3.train()
        for batch, target in train_loader:
            batch = batch.view(-1, n_metabolites).to(device)
            optimizer3.zero_grad()
            outputs = model3(batch) # stack trace shows the issue being either on this line
            train_loss = objective3(outputs, target) # or this line
            loss += train_loss.item()
            n_correct += (target == (outputs.reshape(-1) > 0.5).float()).sum()
            train_loss.backward()
            optimizer3.step()

        loss = loss / len(train_loader)
        acc = (n_correct.float() / len(train)).numpy()
        epoch += 1

        model3.eval();
        val_loss = val_n_correct = 0
        with torch.no_grad():
            for batch, target in val_loader:
                batch = batch.view(-1, n_metabolites).to(device)
                outputs = model3(batch)
                val_loss += objective3(outputs, target)
                val_n_correct += (target == (outputs.reshape(-1) > 0.5).float()).sum()
        val_loss = (val_loss / len(val_loader)).numpy()
        val_acc = (val_n_correct.float() / len(val)).numpy()

        if (epoch % print_stats_interval) == 0 or epoch == epochs:
            print(f'epoch={epoch:.0f}, loss={loss:.5f}, val_loss={np.round(val_loss,5):.5f}, acc={np.round(acc,5):.5f}, val_acc={np.round(val_acc,5):.5f}')
        log3.append((epoch, loss, val_loss, acc, val_acc))
    log3 = pd.DataFrame(log3, columns=['epoch', 'loss', 'val_loss', 'acc', 'val_acc'])

    plt.figure(figsize=(6, 3))
    plt.plot(log3['epoch'], log3['loss'], label='Training');
    plt.plot(log3['epoch'], log3['val_loss'], label='Validation');
    plt.xlabel('Epoch'); plt.ylabel('Loss')
    plt.legend();

    val_log_mutations = val_hcc[:][0].numpy().reshape(-1)
    val_true_labels = val_hcc[:][1].numpy() + 0

    res = model3(val_hcc[:][0])
    predictions = (res.detach().numpy().reshape(-1) > 0.5) + 0
    correct = (val_true_labels == predictions) + 0
    n_correct = correct.sum()
    print('{}/{} ({:.1%}) in the validation set'.format(n_correct, len(correct), n_correct / len(correct)))
    print('Majority classifier accuracy: {:.1%}'.format((len(correct) - val_true_labels.sum()) / len(correct)))

if __name__ == '__main__':
    run()

What's going on here? The stack trace here:
Traceback (most recent call last):
  File "//ad..fi/home/h/h/Desktop/neuralnet/neuralnet_train.py", line 142, in <module>
    run()
  File "//ad..fi/home/h/h/Desktop/neuralnet/neuralnet_train.py", line 99, in run
    train_loss = objective3(outputs, target)
  File "C:\Users\h\AppData\Roaming\Python\Python38\site-packages\torch\nn\modules\module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\Users\h\AppData\Roaming\Python\Python38\site-packages\torch\nn\modules\loss.py", line 516, in forward
    return F.binary_cross_entropy(input, target, weight=self.weight, reduction=self.reduction)
  File "C:\Users\h\AppData\Roaming\Python\Python38\site-packages\torch\nn\functional.py", line 2378, in binary_cross_entropy
    return torch._C._nn.binary_cross_entropy(
RuntimeError: expected device cuda:0 but got device cpu
PS Microsoft.PowerShell.Core\FileSystem::\\ad..fi\home\h\h\Desktop\neuralnet>


title: Whats the equivalent of tf.nn.softmax_cross_entropy_with_logits in pytorch?

body:
I was trying to replicate a code ,which was written in tensorflow ,with pytorch. I came across a loss function in tensorflow, softmax_cross_entropy_with_logits.I was looking for an equivalent of it in pytorch and i found torch.nn.MultiLabelSoftMarginLoss,though im not quite sure it is the right function.Also i dont know how to measure the accuracy of my model when i use this loss function and no relu layer at the end of the network here is my code :

# GRADED FUNCTION: compute_cost

def compute_cost(Z3, Y):

    loss = torch.nn.MultiLabelSoftMarginLoss()
    return loss(Z3,Y)

def model(net,X_train, y_train, X_test, y_test, learning_rate = 0.009,
          num_epochs = 100, minibatch_size = 64, print_cost = True):

    optimizer = torch.optim.Adam(net.parameters(), lr=learning_rate)
    optimizer.zero_grad()

    total_train_acc=0

    for epoch in range(num_epochs):
        for i, data in enumerate(train_loader, 0):
            running_loss = 0.0

            inputs, labels = data

            inputs, labels = Variable(inputs), Variable(labels)

            Z3 = net(inputs)

            # Cost function
            cost = compute_cost(Z3, labels)

            # Backpropagation: Define the optimizer.
            # Use an AdamOptimizer that minimizes the cost.

            cost.backward()
            optimizer.step()

            running_loss += cost.item()

            # Measuring the accuracy of minibatch
            acc = (labels==Z3).sum()
            total_train_acc += acc.item()
            #Print every 10th batch of an epoch
            if epoch%1 == 0:
            print("Cost after epoch {} :
            {:.3f}".format(epoch,running_loss/len(train_loader)))


title: Reshaping Image for PyTorch

body: I used to use keras and the image format it followed is [Height x Width x Channels x Samples]. i decided to switch to PyTorch. But i didn’t switch out my data loading schemes. So now i have numpy arrays of shape HxWxCxS, instead of SxCxHxW which is required for PyTorch. Does anyone have any idea to convert this ?


title: Is it possible to use different L1 / L2 regularization parameters for different sets of weights in chainer or pytorch?

body:
(As an example) When implementing a simple linear model for noutput target values as a neural network in pytorch:

l1=L.Linear(ninput, noutput)

(call)

y = self.l1(x)
return y

Adding this hook will do L2 regularization on all weights, imposing the same alpha=0.01 everywhere:

optimizer.add_hook(optimizer.WeightDecay(rate=0.01))

Is it possible to use a different alpha for each set of weights leading from all ninput input units to one of the noutput output units?


title: What is the proper way to load a transfer learning model for inference in PyTorch?

body:
I am training a model using transfer learning based on Resnet152. Based on PyTorch tutorial, I have no problem in saving a trained model, and loading it for inference. However, the time needed to load the model is slow. I don't know if I did it correct, here is my code:
To save the trained model as state dict:
torch.save(model.state_dict(), 'model.pkl')

To load it for inference:
model = models.resnet152()
num_ftrs = model.fc.in_features
model.fc = nn.Linear(num_ftrs, len(classes))
st = torch.load('model.pkl', map_location='cuda:0' if torch.cuda.is_available() else 'cpu')
model.load_state_dict(st)
model.eval()

I timed the code and found that the first line model = models.resnet152() takes the longest time to load. On CPU, it takes 10 seconds to test one image. So my thinking is that this might not be the proper way to load it?
If I save the entire model instead of the state.dict like this:
torch.save(model, 'model_entire.pkl')

and test it like this:
model = torch.load('model_entire.pkl')
model.eval()

on the same machine it takes only 5 seconds to test one image.
So my question is: is it the proper way to load the state_dict for inference?


title: I meet a problem which is "RuntimeError: Input and hidden tensors are not at the same device, found input tensor at cuda:0 and hidden tensor at cpu"

body:
I guess the hidden tensor is the tenor which I need to initialize in the beginning of RNN. So I set the h0 and c0 by .cuda(), but that is not useful. Following is my code.Please, who can give me a hand?

class LSTM_net(nn.Module):
def __init__(self, Embedding, vocab, label, batch):
    super(LSTM_net, self).__init__()
    self.Hidden_dim = Embedding
    self.Embedding = nn.Embedding(vocab, Embedding)
    self.lstm = nn.LSTM(Embedding, self.Hidden_dim)  # 32 * 32
    self.hidden2label = nn.Linear(Embedding, label)
    self.hidden = self.init_hidden(batch)

def init_hidden(self, batch):
    # the first is the hidden h
    # the second is the cell c

    if torch.cuda.is_available():
        return (autograd.Variable(torch.zeros(1, batch, self.Hidden_dim)),
                autograd.Variable(torch.zeros(1, batch, self.Hidden_dim)))
    else:
        return (autograd.Variable(torch.zeros(1, batch, self.Hidden_dim).cuda()),
                autograd.Variable(torch.zeros(1, batch, self.Hidden_dim).cuda()))

def forward(self, sentence):
    # 64 * 52
    x = self.Embedding(sentence)
    # x = 64 * 52 * 32
    x = x.permute(1, 0, 2)
    # x = 52 * 64 * 32
    lstm_y, lstm_hidden = self.lstm(x, self.hidden)
    # lstm_y = batch * 52 * 32
    # input: lstm_y[-1] = batch * 32
    y = self.hidden2label(lstm_y[-1])
    # y = batch * 5
    log_probs = F.log_softmax(y)
    return log_probs