title: Multilingual Bert sentence vector captures language used more than meaning - working as interned?

body:
Playing around with BERT, I downloaded the Huggingface Multilingual Bert and entered three sentences, saving their sentence vectors (the embedding of [CLS]), then translated them via Google Translate, passed them through the model and saved their sentence vectors.

I then compared the results using cosine similarity.

I was surprised to see that each sentence vector was pretty far from the one generated from the sentence translated from it (0.15-0.27 cosine distance) while different sentences from the same language were quite close indeed (0.02-0.04 cosine distance).

So instead of having sentences of similar meaning (but different languages) grouped together (in 768 dimensional space ;) ), dissimilar sentences of the same language are closer.

To my understanding the whole point of Multilingual Bert is inter-language transfer learning - for example training a model (say, and FC net) on representations in one language and having that model be readily used in other languages.

How can that work if sentences (of different languages) of the exact meaning are mapped to be more apart than dissimilar sentences of the same language?

My code:

import torch

import transformers
from transformers import AutoModel,AutoTokenizer

bert_name="bert-base-multilingual-cased"
tokenizer = AutoTokenizer.from_pretrained(bert_name)
MBERT = AutoModel.from_pretrained(bert_name)

#Some silly sentences
eng1='A cat jumped from the trees and startled the tourists'
e=tokenizer.encode(eng1, add_special_tokens=True)
ans_eng1=MBERT(torch.tensor([e]))

eng2='A small snake whispered secrets to large cats'
t=tokenizer.tokenize(eng2)
e=tokenizer.encode(eng2, add_special_tokens=True)
ans_eng2=MBERT(torch.tensor([e]))

eng3='A tiger sprinted from the bushes and frightened the guests'
e=tokenizer.encode(eng3, add_special_tokens=True)
ans_eng3=MBERT(torch.tensor([e]))

# Translated to Hebrew with Google Translate
heb1='חתול קפץ מהעץ והבהיל את התיירים'
e=tokenizer.encode(heb1, add_special_tokens=True)
ans_heb1=MBERT(torch.tensor([e]))

heb2='נחש קטן לחש סודות לחתולים גדולים'
e=tokenizer.encode(heb2, add_special_tokens=True)
ans_heb2=MBERT(torch.tensor([e]))

heb3='נמר רץ מהשיחים והפחיד את האורחים'
e=tokenizer.encode(heb3, add_special_tokens=True)
ans_heb3=MBERT(torch.tensor([e]))

from scipy import spatial
import numpy as np

# Compare Sentence Embeddings

result = spatial.distance.cosine(ans_eng1[1].data.numpy(), ans_heb1[1].data.numpy())

print ('Eng1-Heb1 - Translated sentences',result)

result = spatial.distance.cosine(ans_eng2[1].data.numpy(), ans_heb2[1].data.numpy())

print ('Eng2-Heb2 - Translated sentences',result)

result = spatial.distance.cosine(ans_eng3[1].data.numpy(), ans_heb3[1].data.numpy())

print ('Eng3-Heb3 - Translated sentences',result)

print ("\n---\n")

result = spatial.distance.cosine(ans_heb1[1].data.numpy(), ans_heb2[1].data.numpy())

print ('Heb1-Heb2 - Different sentences',result)

result = spatial.distance.cosine(ans_eng1[1].data.numpy(), ans_eng2[1].data.numpy())

print ('Heb1-Heb3 - Similiar sentences',result)

print ("\n---\n")

result = spatial.distance.cosine(ans_eng1[1].data.numpy(), ans_eng2[1].data.numpy())

print ('Eng1-Eng2 - Different sentences',result)

result = spatial.distance.cosine(ans_eng1[1].data.numpy(), ans_eng3[1].data.numpy())

print ('Eng1-Eng3 - Similiar sentences',result)

#Output:
"""
Eng1-Heb1 - Translated sentences 0.2074061632156372
Eng2-Heb2 - Translated sentences 0.15557605028152466
Eng3-Heb3 - Translated sentences 0.275478720664978

---

Heb1-Heb2 - Different sentences 0.044616520404815674
Heb1-Heb3 - Similar sentences 0.027982771396636963

---

Eng1-Eng2 - Different sentences 0.027982771396636963
Eng1-Eng3 - Similar sentences 0.024596810340881348
"""

P.S.

At least the Heb1 was closer to Heb3 than to Heb2.
This was also observed for the English equivalents, but less so.


title: Validate on entire validation set when using ddp backend with PyTorch Lightning

body:
I'm training an image classification model with PyTorch Lightning and running on a machine with more than one GPU, so I use the recommended distributed backend for best performance ddp (DataDistributedParallel). This naturally splits up the dataset, so each GPU will only ever see one part of the data.
However, for validation, I would like to compute metrics like accuracy on the entire validation set and not just on a part. How would I do that? I found some hints in the official documentation, but they do not work as expected or are confusing to me. What's happening is that validation_epoch_end is called num_gpus times with 1/num_gpus of the validation data each. I would like to aggregate all results and only run the validation_epoch_end once.
In this section they state that when using dp/ddp2 you can add an additional function called like this
def validation_step(self, batch, batch_idx):
    loss, x, y, y_hat = self.step(batch)
    return {"val_loss": loss, 'y': y, 'y_hat': y_hat}

def validation_step_end(self, self, *args, **kwargs):
    # do something here, I'm not sure what,
    # as it gets called in ddp directly after validation_step with the exact same values
    return args[0]

However, the results are not being aggregated and validation_epoch_end is still called num_gpu times. Is this kind of behavior not available for ddp? Is there some other way how achieve this aggregation behavior?


title: return next(self._sampler_iter) # may raise StopIteration

body:
Instead of using enumerate(data loader) for some reason, I am creating iterator for the data loader. In the while loop shown below, it gives me StopIteration error.
Minimalistic code that depicts the cause:
loader = DataLoader(dataset, batch_size=args.batch_size)
dataloader_iter = iter(loader)
while(dataloader_iter):
    X, y = next(dataloader_iter)
    ...

What would be the correct condition (to specify within the while loop) to check if the iterator is empty?


title: Kaggle Pytorch run length encoding

body:
Im working on the DSB problem in pytorch,I have my predictions but im not sure how to get those into the run length encoding format that is required for the submission,In short this is what it is

===================================================================

In order to reduce the submission file size, our metric uses run-length encoding on the pixel values. Instead of submitting an exhaustive list of indices for your segmentation, you will submit pairs of values that contain a start position and a run length. E.g. '1 3' implies starting at pixel 1 and running a total of 3 pixels (1,2,3).

The competition format requires a space delimited list of pairs. For example, '1 3 10 5' implies pixels 1,2,3,10,11,12,13,14 are to be included in the mask. The pixels are one-indexed and numbered from top to bottom, then left to right: 1 is pixel (1,1), 2 is pixel (2,1), etc.

===================================================================

I get the predictions like this

model = model.eval()
for data in testdataloader:
    data = t.autograd.Variable(data.cuda(device = 1))
    pred = model(data)

but now that i have my prediction im not sure how to move forward,
I found this script online,but im not sure how to modify this for my use case

def rle_encoding(x):
    dots = np.where(x.T.flatten() == 1)[0]
    run_lengths = []
    prev = -2
    for b in dots:
        if (b>prev+1): run_lengths.extend((b + 1, 0))
        run_lengths[-1] += 1
        prev = b
    return run_lengths

def prob_to_rles(x, cutoff=0.5):
    lab_img = label(x > cutoff)
    for i in range(1, lab_img.max() + 1):
        yield rle_encoding(lab_img == i)

Any suggestions on how may i get started or how do i modify this would be really helpfull!


title: Fastai - CUDA error: device-side assert triggered

body:
I am using Fastai for Image Segmentation on the Kaggle Dataset - TGS Salt Identification Challenge. I have tried the code on both CPU and GPU using colab.

Link to notebook file using CPU: bit.ly/2I2ZOMX_TGS_CPU
Link to notebook file using GPU: bit.ly/2Vq0ROZ_TGS_GPU

In CPU notebook, I am getting the following error:
Target 255 is out of bounds

I have set div = True in open_mask and have also set num_workers = 0

In GPU notebook I am getting the following error:
CUDA error: device-side assert triggered

I have taken the custom metric of IOU.
Fastai version: 1.0.60

Kindly have a look and suggest me how to fix the issue.


title: Asking gpt-2 to finish sentence with huggingface transformers

body:
I am currently generating text from left context using the example script run_generation.py of the huggingface transformers library with gpt-2:

$ python transformers/examples/run_generation.py \
  --model_type gpt2 \
  --model_name_or_path gpt2 \
  --prompt "Hi, " --length 5

=== GENERATED SEQUENCE 1 ===
Hi,  could anyone please inform me

I would like to generate short complete sentences. Is there any way to tell the model to finish a sentence before length words?

Note: I don't mind changing model, but would prefer an auto-regressive one.


title: The grad of F.normalize in pytorch is strange

body:
The grad of F.normalize in pytorch is nearly zero.
Codes in python3 & pytorch==1.3.0:
import torch
import torch.nn.functional as F

x = torch.autograd.Variable(torch.Tensor([[1, 0]]), requires_grad=True)
x_norm = F.normalize(x, dim=-1)
z = x_norm.mm(x_norm.t())
print('x_norm', x_norm)
print('z', z)

x.register_hook(lambda g: print(g))
x_norm.register_hook(lambda g: print(g))
z.register_hook(lambda g: print(g))
z.backward()

Output:
x_norm tensor([[1., 0.]], grad_fn=<DivBackward0>)
z tensor([[1.]], grad_fn=<MmBackward>)
tensor([[1.]])                      # z grad
tensor([[2., 0.]])                  # x_norm grad
tensor([[0., 0.]])                  # x grad

Why is the grad of x zero? 'x.grad' gives the same result.
According to the chain rule, I think the result should be [-1, 1]*[2, 0]=[-2, 0].
Actually, when I use the codes above in a whole network, the grad works fine.
Is there anything wrong with my codes?
Tried
x_norm = x / torch.sqrt((x[0, 0]**2 + x[0, 1]**2))

The same result.
Tried CUDA, the same.
Tried the following codes, replacing x_norm with y_norm:
import torch
import torch.nn.functional as F

x = torch.autograd.Variable(torch.Tensor([[1, 0]]), requires_grad=True)
y = torch.autograd.Variable(torch.Tensor([[2, 0]]), requires_grad=True)
x_norm = F.normalize(x, dim=-1)
y_norm = F.normalize(y, dim=-1)
z = x_norm.mm(y_norm.t())

y.register_hook(lambda g: print(g))
x.register_hook(lambda g: print(g))
x_norm.register_hook(lambda g: print(g))
z.register_hook(lambda g: print(g))
z.backward()

Output:
tensor([[1.]])                      # z grad
tensor([[2., 0.]])                  # x_norm grad
tensor([[0., 0.]])                  # y grad
tensor([[0., 0.]])                  # x grad

However, if I replace z = x_norm.mm(y_norm.t()) with z = x_norm.mm(y.t()), y has its non-zero value while x doesn't.
Also tried to put F.normalize in the middle of the codes:
import torch
import torch.nn.functional as F

#x = torch.autograd.Variable(torch.randn(1, 2), requires_grad=True)
x = torch.autograd.Variable(torch.Tensor([[1,0]]), requires_grad=True).cuda()                                                                                y = torch.autograd.Variable(torch.Tensor([[2,0]]), requires_grad=True).cuda()
x1 = x * 2
x1_norm = F.normalize(x1, dim=-1)

print('x1', x1)
z = x1_norm.mm(y.t())
print('x1_norm', x1_norm)
print('z', z)
y.register_hook(lambda g: print('y', g))
x.register_hook(lambda g: print('x', g))
x1.register_hook(lambda g: print('x1', g))
x1_norm.register_hook(lambda g: print('x1_norm', g))
z.register_hook(lambda g: print('z', g))
z.backward()

Output
x1 tensor([[2., 0.]], device='cuda:0', grad_fn=<MulBackward0>)
x1_norm tensor([[1., 0.]], device='cuda:0', grad_fn=<DivBackward0>)
z tensor([[2.]], device='cuda:0', grad_fn=<MmBackward>)
z grad tensor([[1.]], device='cuda:0')            # z grad
x1_norm grad tensor([[2., 0.]], device='cuda:0')  # (x*2)_norm grad
x1 grad tensor([[0., 0.]], device='cuda:0')       # x*2 grad
y grad tensor([[1., 0.]], device='cuda:0')        # y grad
x grad tensor([[0., 0.]], device='cuda:0')        # x grad


title: how to implement ResNet50 in PyTorch?

body:
I learn NN in Coursera course, by deeplearning.ai and for one of my homework was an assignment for ResNet50 implementation by using Keras, but I see Keras is too high-level language) and decided to implement it in the more sophisticated library - PyTorch. I recorded it, but something went wrong. May someone, please, say to me what's going on and why there is appeared error with parameters when I cause method ResNet.parameters() when putting it in Adam optimization.
class implementation:
class ResNet50(torch.nn.Module):
    def __init__(self, input_shape = (3, 96, 96), classes = 10):
        super(ResNet50, self).__init__()
        """
        Implementation of the popular ResNet50 the following architecture:
        Conv2d -> BatchNorm -> ReLU -> MaxPool -> ConvBlock - > IdBlock*2 - > convBlock -> IdBlock*3 -> ConvBlock -> IdBlock*5 -> ConvBlock -> IdBlock*2 -> AvgPool -> FCLayer

        Arguments:
        input_shape -- shape of the image of the dataset
        classes -- integer, number of classes
        """

        self.input_shape = input_shape
        self.classes = classes
        self.relu = torch.nn.ReLU()

    def identity_block(self, X, f, filters):
        # Notice that there is no any kind of Pooling.
        """
        Implementation of the identity block.

        Arguments:
        X -- input tensor of shape(m , n_H_prev, n_W_prev, n_C_prev)
        f -- integer, specifying the shape of the middle CONV's window for the main path
        filters -- python list of integers, defining the number of filters in the CONV layers of the main path

        Returns:
        X -- output of the identity block, tensor of shape (n_H, n_W, n_C)
        """

        # Retrieve Filters
        F1, F2, F3 = filters

        # Save the input value. It will be needed later to be added back to the main path.
        X_shortcut = X

        # First component of the main path
        X = torch.nn.Conv2d(in_channels=X.shape[0], out_channels=F1, kernel_size=1, stride=1, padding=0)(X)
        X = torch.nn.BatchNorm2d(num_features=F1)(X)
        X = self.relu(X)

        # Second component of the main path
        X = torch.nn.Conv2d(in_channels=F1, out_channels=F2, kernel_size=f, stride=1, padding=f//2)(X)
        X = torch.nn.BatchNorm2d(num_features=F2)(X)
        X = self.relu(X)

        # Third component of the main path
        X = torch.nn.Conv2d(in_channels=F2, out_channels=F3, kernel_size=1, stride=1, padding=0)(X)
        X = torch.nn.BatchNorm2d(num_features=F3)(X)
        # X = self.relu(X) - NO RELU, notice this!

        # Final step: Add shortcut value to main path, and pass it through a ReLU
        X = X_shortcut + X
        X = self.relu(X)

        return X
    def convolution_block(self, X, f, filters, s = 2):
        # Notice that here is no any kind of Pooling.
        """
        Implementation of the convolutional block.

        Arguments:
        X -- input tensor of shape (m, n_H_prev, n_W_prev, n_C_prev)
        f -- integer, specifying the shape of middle CONV's window for main path
        filters -- python list of integers, defining the number of filters in the CONV layers of the main path
        s -- integer, specifying the stride to be used

        Returns:
        X -- output of the convolution block, tensor of shape (n_H, n_W, n_C)
        """

        # Retrieve Filters
        F1, F2, F3 = filters

        # Save the input value
        X_shortcut = X

        # First component of the main path
        X = torch.nn.Conv2d(in_channels=X.shape[0], out_channels=F1, kernel_size=1, stride=s, padding=0)(X)
        X = torch.nn.BatchNorm2d(num_features=F1)(X)
        X = self.relu(X)

        # Second component of the main path
        X = torch.nn.Conv2d(in_channels=F1, out_channels=F2, kernel_size=f, stride=1, padding=f//2)(X)
        X = torch.nn.BatchNorm2d(num_features=F2)(X)
        X = self.relu(X)

        # Third component of the main path
        X = torch.nn.Conv2d(in_channels=F2, out_channels=F3, kernel_size=1, stride=1, padding=0)(X)
        X = torch.nn.BatchNorm2d(num_features=F2)(X)
        # X = self.relu(X) - NO RELU, notice this!

        # Shortcut path
        X_shortcut = torch.nn.Conv2d(in_channels=X_shortcut.shape[0], out_channels=F3, kernel_size=1, stride=s, padding=0)(X)
        X_shortcut = torch.nn.BatchNorm2d(num_features=F3)(X)
        # X = self.relu(X) - NO RELU, notice this!

        # Final step: Add shortcut value to main path, and pass it through a ReLU
        X = X_shortcut + X
        X = self.relu(X)

        return X
    def forward(self, X):
        """
        Forward propogation by the following architecture:
        Conv2d -> BatchNorm -> ReLU -> MaxPool -> ConvBlock - > IdBlock*2 - > convBlock -> IdBlock*3 -> ConvBlock -> IdBlock*5 -> ConvBlock -> IdBlock*2 -> AvgPool -> FCLayer

        Arguments:
        X -- input data for Network that needed to be propagated

        Returns:
        X -- output of the ResNet50, that propagated through it
        """

        # # Define the input as a tensor with shape self.input_shape
        # X = torch.zeros_like(self.input_shape)

        # Stage 1
        X = torch.nn.Conv2d(in_channels=3, out_channels=64, kernel_size=7, stride=2, padding=3)(X) # 96x96x3 -> 48x48x64
        X = torch.nn.BatchNorm2d(num_features=64)(X)
        X = self.relu(X)
        X = torch.nn.MaxPool2d(kernel_size=3, stride=2, padding=0)(X) # 48x48x64 -> 23x23x64

        # Stage 2
        X = self.convolution_block(X, f=3, filters=[64, 64, 256], s=1) # 23x23x64 -> 23x23x256
        X = self.identity_block(X, 3, [64, 64, 256]) # same
        X = self.identity_block(X, 3, [64, 64, 256]) # same

        # Stage 3
        X = self.convolution_block(X, f=3, filters=[128, 128, 512], s=2) # 23x23x256 -> 12x12x512
        X = self.identity_block(X, 3, [128, 128, 512]) # same
        X = self.identity_block(X, 3, [128, 128, 512]) # same
        X = self.identity_block(X, 3, [128, 128, 512]) # same

        # Stage 4
        X = self.convolution_block(X, f=3, filters=[256, 256, 1024], s=2) # 12x12x512 -> 6x6x1024
        X = self.identity_block(X, 3, [256, 256, 1024]) # same
        X = self.identity_block(X, 3, [256, 256, 1024]) # same
        X = self.identity_block(X, 3, [256, 256, 1024]) # same
        X = self.identity_block(X, 3, [256, 256, 1024]) # same
        X = self.identity_block(X, 3, [256, 256, 1024]) # same

        # Stage 5
        X = self.convolution_block(X, f=3, filters=[512, 512, 2048], s=2) # 6x6x1024 -> 3x3x2048
        X = self.identity_block(X, 3, [512, 512, 2048]) # same
        X = self.identity_block(X, 3, [512, 512, 2048]) # same

        # AvgPool
        X = torch.nn.AvgPool2d(kernel_size=2)(X) # 3x3x2048 -> 2x2x2048

        # Output layer
        X = X.reshape(X.shape[0], -1)
        X = torch.nn.Linear(in_features=X.shape[1], out_features=self.classes)
        X = torch.nn.Softmax(X)

        return X

next script:
NNet = ResNet50()

device = torch.device('cuda:0' if torch.cuda.is_available else 'cpu')
NNet = NNet.to(device)

loss = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(NNet.parameters(), lr = 0.001)

random.seed(k)
np.random.seed(k)
torch.manual_seed(k)
torch.cuda.manual_seed(k)
torch.backends.cudnn.deterministic = True

Returned error:
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-18-71ee8a51c6b2> in <module>()
      5
      6 loss = torch.nn.CrossEntropyLoss()
----> 7 optimizer = torch.optim.Adam(NNet.parameters(), lr = 0.001)
      8
      9 random.seed(k)

1 frames
/usr/local/lib/python3.6/dist-packages/torch/optim/optimizer.py in __init__(self, params, defaults)
     44         param_groups = list(params)
     45         if len(param_groups) == 0:
---> 46             raise ValueError("optimizer got an empty parameter list")
     47         if not isinstance(param_groups[0], dict):
     48             param_groups = [{'params': param_groups}]

ValueError: optimizer got an empty parameter list


title: ValueError: too many dimensions 'str' python in pytorch

body:
I keep getting the error:
ValueError: too many dimensions 'str'
I attach my colab notebook to have a look at it. Have not found anything online yet that helps me solve the problem.
link:
https://colab.research.google.com/drive/1ikol2D8mmiIPKhNHbcFlTfVpuU_Gf9BZ?usp=sharing


title: PyTorch: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True

body:
This is an error message I get as I work with some synthetic data. I am a bit puzzled because the error persists although I do what I am advised to do.  Could it somehow be related to the fact that I do not specify a batch?  Would the use of PyTorch Dataset relieve the problem?

Here is my code (I am new in PyTorch, just learning it now) -- it should be reproducible:

Data Creation:

x, y = np.meshgrid(np.random.randn(100) , np.random.randn(100))
z = 2 * x + 3 * y + 1.5 * x * y - x ** 2 - y**2
X = x.ravel().reshape(-1, 1)
Y = y.ravel().reshape(-1, 1)
Z = z.ravel().reshape(-1, 1)
U = np.concatenate([X, Y], axis = 1)
U = torch.tensor(U, requires_grad=True)
Z = torch.tensor(Z, requires_grad=True)
V = []

for i in range(U.shape[0]):
    u = U[i, :]
    u1 = u.view(-1, 1) @ u.view(1, -1)
    u1 = u1.triu()
    ones = torch.ones_like(u1)
    mask = ones.triu()
    mask = (mask == 1)
    u2 = torch.masked_select(u1, mask)
    u3 = torch.cat([u, u2])
    u3 = u3.view(1, -1)
    V.append(u3)

V = torch.cat(V, dim = 0)

Training a Model

from torch import nn
from torch import optim
net = nn.Sequential(nn.Linear(V.shape[1], 1))
criterion = nn.MSELoss()
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)

for epoch in range(50):  # loop over the dataset multiple times
    running_loss = 0.0
    i = 0
    for inputs , labels in zip(V, Z):
        # get the inputs; data is a list of [inputs, labels]

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = net(inputs)
        loss = criterion(outputs, labels)

        loss.backward(retain_graph = True)

        optimizer.step()

        # print statistics
        running_loss += loss.item()

        i += 1

        if i % 2000 == 1999:    # print every 2000 mini-batches
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 2000))
            running_loss = 0.0

print('Finished Training')

Error Message:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-143-2454f4bb70a5> in <module>
     25
     26
---> 27         loss.backward(retain_graph = True)
     28
     29         optimizer.step()

~\Anaconda3\envs\torch\lib\site-packages\torch\tensor.py in backward(self, gradient, retain_graph, create_graph)
    193                 products. Defaults to ``False``.
    194         """
--> 195         torch.autograd.backward(self, gradient, retain_graph, create_graph)
    196
    197     def register_hook(self, hook):

~\Anaconda3\envs\torch\lib\site-packages\torch\autograd\__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
     97     Variable._execution_engine.run_backward(
     98         tensors, grad_tensors, retain_graph, create_graph,
---> 99         allow_unreachable=True)  # allow_unreachable flag
    100
    101

RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.

Could you explain the error and fix the code?


1:
issues with pytorch nn. made an auto encoder out of linear layers for a tensorflow conversion project (to use pysyft at a future point)
2:
have made sure that the forward method does return a value, does work in other situations.
3:
the full error:
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "B:\tools and software\PyCharm 2020.1\plugins\python\helpers\pydev\_pydev_bundle\pydev_umd.py", line 197, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
  File "B:\tools and software\PyCharm 2020.1\plugins\python\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "B:/projects/openProjects/githubprojects/BotnetTrafficAnalysisFederaedLearning/anomaly-detection/pytorch_conversion.py", line 157, in <module>
    torch.from_numpy(x_test).float(), tr=1)
  File "B:/projects/openProjects/githubprojects/BotnetTrafficAnalysisFederaedLearning/anomaly-detection/pytorch_conversion.py", line 88, in test
    x_test_predictions = net(x_test.detach().numpy())
  File "B:\tools and software\Anaconda\envs\pysyft-pytorch\lib\site-packages\torch\nn\modules\module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "B:/projects/openProjects/githubprojects/BotnetTrafficAnalysisFederaedLearning/anomaly-detection/pytorch_conversion.py", line 121, in forward
    x = torch.tanh(self.fc1(x))
  File "B:\tools and software\Anaconda\envs\pysyft-pytorch\lib\site-packages\torch\nn\modules\module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "B:\tools and software\Anaconda\envs\pysyft-pytorch\lib\site-packages\torch\nn\modules\linear.py", line 87, in forward
    return F.linear(input, self.weight, self.bias)
  File "B:\tools and software\Anaconda\envs\pysyft-pytorch\lib\site-packages\torch\nn\functional.py", line 1368, in linear
    if input.dim() == 2 and bias is not None:
AttributeError: 'numpy.ndarray' object has no attribute 'dim'

the method:
def test(net, x_test, tr):
    correct = 0
    total = 0
    x_test_predictions = net(x_test.detach().numpy())
    print("Calculating MSE on test set...")
    mse_test = np.mean(np.power(x_test - x_test_predictions, 2), axis=1)
    over_tr = mse_test > tr
    false_positives = sum(over_tr)
    test_size = mse_test.shape[0]
    print(f"{false_positives} false positives on dataset without attacks with size {test_size}")

    with torch.no_grad():
        for i in tqdm(range(len(x_test))):
            real_class = torch.argmax(x_test[i])
            net_out = net(x_test[i])
            predicted_class = torch.argmax(net_out)
            if predicted_class == real_class:
                correct += 1
            total += 1
    print("Accuracy: ", round(correct / total, 3))
enter code here

The NN:
class Net(nn.Module):
    def __init__(self, input_dim=10):
        super().__init__()
        self.fc1 = nn.Linear(input_dim, int(0.75 * input_dim))
        self.fc2 = nn.Linear(int(0.75 * input_dim), int(0.5 * input_dim))
        self.fc3 = nn.Linear(int(0.5 * input_dim), int(0.33 * input_dim))
        self.fc4 = nn.Linear(int(0.33 * input_dim), int(0.25 * input_dim))
        self.fc5 = nn.Linear(int(0.25 * input_dim), int(0.33 * input_dim))
        self.fc6 = nn.Linear(int(0.33 * input_dim), int(0.5 * input_dim))
        self.fc7 = nn.Linear(int(0.5 * input_dim), int(0.75 * input_dim))
        self.fc8 = nn.Linear(int(0.75 * input_dim), input_dim)

    def forward(self, x):
        x = torch.tanh(self.fc1(x))
        x = torch.tanh(self.fc2(x))
        x = torch.tanh(self.fc3(x))
        x = torch.tanh(self.fc4(x))
        x = torch.tanh(self.fc5(x))
        x = torch.tanh(self.fc6(x))
        x = torch.tanh(self.fc7(x))
        x = self.fc8(x)
        return torch.softmax(x, dim=1)


I have a Bayesian algorithm program to be executed, I am using python 3
import numpy as np
import csv
import pandas as pd
from pgmpy.models import BayesianModel
from pgmpy.estimators import MaximumLikelihoodEstimator
from pgmpy.inference import VariableElimination

heartDisease = pd.read_csv('heart.csv')
heartDisease = heartDisease.replace('?',np.nan)

print('Few examples from the dataset are given below')
print(heartDisease.head())

model = BayesianModel([('age','trestbps'),('age','fbs'),('sex','trestbps'),('exang','trestbps'),('trestbps','heartdisease'),('fbs','heartdisease'),('heartdisease','restecg'),('heartdisease','thalach'),('heartdisease','chol')])

print('\nLearning CPD using Maximum likelihood estimators')
model.fit(heartDisease,estimator=MaximumLikelihoodEstimator)

print('\n Inferencing with Bayesian Network:')
HeartDisease_infer = VariableElimination(model)

print('\n 1. Probability of HeartDisease given Age=28')
q=HeartDisease_infer.query(variables=['heartdisease'],evidence={'age':28})
print(q['heartdisease'])

print('\n 2. Probability of HeartDisease given cholesterol=100')
q=HeartDisease_infer.query(variables=['heartdisease'],evidence={'chol':100})
print(q['heartdisease'])

the error that i have recieved when i run my Bayesian network program is:
TypeError                                 Traceback (most recent call last)
<ipython-input-7-84a6b48627b2> in <module>
     23 print('\n 1. Probability of HeartDisease given Age=28')
     24 q=HeartDisease_infer.query(variables=['heartdisease'],evidence={'age':28})
---> 25 print(q['heartdisease'])
     26
     27 print('\n 2. Probability of HeartDisease given cholesterol=100')

TypeError: 'DiscreteFactor' object is not subscriptable

So far I haven't seen this exact error here on stackoverflow. Can anyone explain why I am getting this error?


title: How does one padd a tensor of 3 dimensions in Pytorch?

body:
I was trying to use the built in padding function but it wasn't padding things for me for some reason. This is my reproducible code:

import torch

def padding_batched_embedding_seq():
    ## 3 sequences with embedding of size 300
    a = torch.ones(1, 4, 5) # 25 seq len (so 25 tokens)
    b = torch.ones(1, 3, 5) # 22 seq len (so 22 tokens)
    c = torch.ones(1, 2, 5) # 15 seq len (so 15 tokens)
    ##
    sequences = [a, b, c]
    batch = torch.nn.utils.rnn.pad_sequence(sequences)

if __name__ == '__main__':
    padding_batched_embedding_seq()

error message:

Traceback (most recent call last):
  File "padding.py", line 51, in <module>
    padding_batched_embedding_seq()
  File "padding.py", line 40, in padding_batched_embedding_seq
    batch = torch.nn.utils.rnn.pad_sequence(sequences)
  File "/Users/rene/miniconda3/envs/automl/lib/python3.7/site-packages/torch/nn/utils/rnn.py", line 376, in pad_sequence
    out_tensor[:length, i, ...] = tensor
RuntimeError: The expanded size of the tensor (4) must match the existing size (3) at non-singleton dimension 1.  Target sizes: [1, 4, 5].  Tensor sizes: [3, 5]

any idea?

cross posted: https://discuss.pytorch.org/t/how-does-one-padd-a-tensor-of-3-dimensions/51097


title: Different results after converting pytorch to torchscript? Converting NSnumber to Float cause any loss?

body:
I converted the pytorch pretrained-model(.pt) to torchscript model(.pt) for using it in Swift 5(ios-iphone6s, xcode 11). In Swift, the “predict” function of the model gave me its embedding values(Tensor). Since it returned the NSNumber array as a result of prediction, I used type casting [NSNumber] to both [Double] or [Float] to calculate the distance between two embedding values. L2 normalization, dot product, etc.

However, while the pytorch version got the correct answers, the torchscript model got so many wrong answers. Not only the answers are different, the distance calculations of the two embedding pairs in torchscript are also different from the results of the pytorch model on the PC(CPU, Pycharm). In fact, before using type casting for distance calculations, the embedding values in NSNumber(Swift) are so different from the values in float32(pytorch). I used same input images.

I tried to find the reason.. Once, I copied the embedding values( [NSNumber] ) from swift-torchscript and calculated the distance between two embeddings in pytorch, to check if there was a problem with my distance calculation implementation in Swift. I used torch.FloatTensor to use type casting [NSNumber] -> [Float]. I also tried [Double]. As a result of this, I found many infinite numbers. Is this infinite numbers related to the wrong answer?

What does this “inf” mean? Is it a calculation or type casting error? Did I lost information while casting from NSNumber to Float or Double? How could I get the correct value from the torchscript model in swift? What should I check?

I used the following codes to convert. pytorch -> torchscript.

import torch

from models.inception_resnet_v1 import InceptionResnetV1

device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')

resnet = InceptionResnetV1(pretrained='vggface2').eval().to(device)

example = torch.rand(1, 3, 160, 160)
traced_script_module = torch.jit.trace(resnet, example)
traced_script_module.save("mobile_model.pt")


title: Why do I get NONE gradient of parameters in a loaded model in Pytorch, even after backword?

body:
I have a pretrained model which was saved by

torch.save(net, 'lenet5_mnist_model')

And now I am loading it back and trying to calculate fisher information matrix like this:

precision_matrices = {}
batch_size = 32
my_model = torch.load('lenet5_mnist_model')
my_model.eval()              #  I tried to comment this off, but still no luck
for n, p in deepcopy({n: p for n, p in my_model.named_parameters()}).items()
   p = torch.tensor(p, requires_grad = True)
   p.data.zero_()
   precision_matrices[n] = variable(p.data)
for idx in range(int(images.shape[0]/batch_size)):
   x = images[idx*batch_size : (idx+1)*batch_size]
   my_model.zero_grad()
   x = Variable(x.cuda(), requires_grad = True)
   output = my_model(x).view(1,-1)
   label = output.max(1)[1].view(-1)
   loss = F.nll_loss(F.log_softmax(output, dim=1), label)
   loss = Variable(loss, requires_grad = True)
   loss.backward()
   for n, p in my_model.named_parameters():
       precision_matrices[n].data += p.grad.data**2

Finally, the above code will crash at the last line, because p.grad is NoneType. So the error is:

AttributeError: 'NoneType' object has no attribute 'data'.

Could someone provide some guidance on what caused the NoneType grad for the parameters? How should I fix this?


title: Instance Normalization with batch size 1

body:
I am really confused with the meaning of Instance Norm and whether I can use it with a batch size of 1. I am using PyTorch and nothing in the documentation says that batch size should be greater than 1.

I know that for BatchNorm the performance is adversely affected when batch size is less than 8 and hence it puts a sort of soft bound on the batch size. However, I did not see any such analysis on Instance Norm and am a bit confused now. Should I remove the norm layer if my batch size is 1 then?


title: Pytorch: How to implement nested transformers: a character-level transformer for words and a word-level transformer for sentences?

body:
I have a model in mind, but I'm having a hard time figuring out how to actually implement it in Pytorch, especially when it comes to training the model (e.g. how to define mini-batches, etc.). First of all let me quickly introduce the context:

I'm working on VQA (visual question answering), in which the task is to answer questions about images, for example:

So, letting aside many details, I just want to focus here on the NLP aspect/branch of the model. In order to process the natural language question, I want to  use character-level embeddings (instead of traditional word-level embeddings) because they are more robust in the sense that they can easily accommodate for morphological variations in words (e.g. prefixes, suffixes, plurals, verb conjugations, hyphens, etc.). But at the same time I don't want to lose the inductive bias of reasoning at the word level. Therefore, I came up with the following design:

As you can see in the picture above, I want to use transformers (or even better, universal transformers), but with a little twist. I want to use 2 transformers: the first one will process each word characters in isolation (character-level transformer) to produce an initial word-level embedding for each word in the question. Once we have all these initial word-level embeddings, a second word-level transformer will refine these embeddings to enrich their representation with context, thus obtaining context-aware word-level embeddings.

The full model for the whole VQA task obviously is more complex, but I just want to focus here on this NLP part. So my question is basically about which Pytorch functions should I pay attention to when implementing this. For example, since I'll be using character-level embeddings I have to define a character-level embedding matrix, but then I have to perform lookups on this matrix to generate the inputs for the character-level transformer, repeat this for each word in the question and then feed all these vectors into the word-level transformer. Moreover, words in a single question can have different lengths, and questions within a single mini-batch can have different lengths too. So in my code I have to somehow account for different lengths at the word and the question level simultaneously in a single mini-batch (during training), and I've got no idea how to do that in Pytorch or whether it's even possible at all.

Any tips on how to go about implementing this in Pytorch that could lead me in the right direction will be deeply appreciated.


title: Using Pytorch's dataloaders & transforms with sklearn

body:
I have been using pytorch a lot and got used to their dataloaders and transforms, in particular when it comes to data augmentation, as they're very user-friendly and easy to understand.

However, I need to run some ML models from sklearn.

Is there a way to use pytorch's dataloaders for sklearn ?


title: Tensorflow and PyTorch hang on initializing with CUDA

body:
When I try to run a very minimal Tensorflow example:

import tensorflow as tf

c = tf.constant([1,2,3])

The system hangs forever (at least for ten minutes) with no sign of what it is doing. It uses 100% of one virtual CPU core when in this state. When run in a Juypter notebook the kernel outputs this to the console:

2020-03-31 11:12:04.840507: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer.so.6'; dlerror: libnvinfer.so.6: cannot open shared object file: No such file or directory
2020-03-31 11:12:04.840576: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer_plugin.so.6'; dlerror: libnvinfer_plugin.so.6: cannot open shared object file: No such file or directory
2020-03-31 11:12:04.840589: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:30] Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
2020-03-31 11:12:05.521172: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-03-31 11:12:05.539193: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-03-31 11:12:05.539639: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce GTX 1070 computeCapability: 6.1
coreClock: 1.7845GHz coreCount: 15 deviceMemorySize: 7.93GiB deviceMemoryBandwidth: 238.66GiB/s
2020-03-31 11:12:05.539841: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-03-31 11:12:05.541113: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-03-31 11:12:05.542119: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-03-31 11:12:05.542324: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-03-31 11:12:05.543632: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-03-31 11:12:05.544401: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-03-31 11:12:05.547212: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-03-31 11:12:05.547337: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-03-31 11:12:05.548015: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-03-31 11:12:05.548512: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
2020-03-31 11:12:05.567845: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3393550000 Hz
2020-03-31 11:12:05.568364: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x564107e16440 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-03-31 11:12:05.568395: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version

I did have Tensorflow working previously on this system, so I think this might be some sort of library issue that got caused by a system update.

The GPU is an Nvidia GTX 1070. Tensorflow version is 2.1.0, and hasn't changed since when it was working. Running Arch Linux, if that matters.

I tried downgrading from CUDA 10.2 to 10.1, but the issue still occurs.

I can also reproduce this with PyTorch:

import torch
import transformers

t = torch.tensor([1,2,3])
t.cuda()

(import transformers prevents a "CUDA: Out of memory" issue - there must be something it does that initializes PyTorch that I don't know how to do.)

This has the same issue, where it freezes pegging one CPU core, though it produces less output:

020-03-31 11:13:41.428483: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer.so.6'; dlerror: libnvinfer.so.6: cannot open shared object file: No such file or directory
2020-03-31 11:13:41.428571: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer_plugin.so.6'; dlerror: libnvinfer_plugin.so.6: cannot open shared object file: No such file or directory
2020-03-31 11:13:41.428587: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:30] Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.

I'm pretty sure the complaints about TensorRT are not relevant, because when I had this working previously it would output those as well.

How can I resolve this issue? Or at least, what else can I do to determine what it is doing while frozen?


title: Custom submodules in pytorch / libtorch C++

body:
Full disclosure, I asked this same question on the PyTorch forums about a few days ago and got no reply, so this is technically a repost, but I believe it's still a good question, because I've been unable to find an answer anywhere online. Here goes:

Can you show an example of using register_module with a custom module?
The only examples I’ve found online are registering linear layers or convolutional layers as the submodules.

I tried to write my own module and register it with another module and I couldn’t get it to work.
My IDE is telling me no instance of overloaded function "MyModel::register_module" matches the argument list -- argument types are: (const char [14], TreeEmbedding)

(TreeEmbedding is the name of another struct I made which extends torch::nn::Module.)

Am I missing something? An example of this would be very helpful.

Edit: Additional context follows below.

I have a header file "model.h" which contains the following:

struct TreeEmbedding : torch::nn::Module {
    TreeEmbedding();
    torch::Tensor forward(Graph tree);
};

struct MyModel : torch::nn::Module{
    size_t embeddingSize;
    TreeEmbedding treeEmbedding;

    MyModel(size_t embeddingSize=10);
    torch::Tensor forward(std::vector<Graph> clauses, std::vector<Graph> contexts);
};

I also have a cpp file "model.cpp" which contains the following:

MyModel::MyModel(size_t embeddingSize) :
    embeddingSize(embeddingSize)
{
    treeEmbedding = register_module("treeEmbedding", TreeEmbedding{});
}

This setup still has the same error as above. The code in the documentation does work (using built-in components like linear layers), but using a custom module does not. After tracking down torch::nn::Linear, it looks as though that is a ModuleHolder (Whatever that is...)

Thanks,
Jack


title: Error installing torchvision (python 3.6)

body:
I am trying to install torchvision by writing
pip3 install torchvision
but I get the following message
Requirement already satisfied: torchvision in ./venv/lib/python3.6/site-packages (0.7.0)
Requirement already satisfied: numpy in ./venv/lib/python3.6/site-packages (from torchvision) (1.19.2)
Requirement already satisfied: pillow>=4.1.1 in ./venv/lib/python3.6/site-packages (from torchvision) (7.2.0)
Collecting torch==1.6.0 (from torchvision)
  Could not find a version that satisfies the requirement torch==1.6.0 (from torchvision) (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2, 0.3.0.post4, 0.3.1, 0.4.0, 0.4.1, 1.0.0, 1.0.1, 1.0.1.post2, 1.1.0, 1.1.0.post2, 1.2.0, 1.3.0, 1.3.0.post2, 1.3.1, 1.4.0)
No matching distribution found for torch==1.6.0 (from torchvision)

I currently have torch==1.4.0 installed but it is apparently not sufficient to install torchvision. When I tried the code pip3 install torch==1.6.0
I got the following error
Collecting torch==1.6.0
  Could not find a version that satisfies the requirement torch==1.6.0 (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2, 0.3.0.post4, 0.3.1, 0.4.0, 0.4.1, 1.0.0, 1.0.1, 1.0.1.post2, 1.1.0, 1.1.0.post2, 1.2.0, 1.3.0, 1.3.0.post2, 1.3.1, 1.4.0)
No matching distribution found for torch==1.6.0

Does someone know how I can fix this?


title: Invalid combination of arguments - eq()

body:
I'm using a code shared here to test a CNN image classifier. When I call the test function, I got this error on line 155:

test_acc += torch.sum(prediction == labels.data)
TypeError: eq() received an invalid combination of arguments - got (numpy.ndarray), but expected one of:
 * (Tensor other)
      didn't match because some of the arguments have invalid types: ([31;1mnumpy.ndarray[0m)
 * (Number other)
      didn't match because some of the arguments have invalid types: ([31;1mnumpy.ndarray[0m)

Fragment of the test function:

def test():
    model.eval()
    test_acc = 0.0
    for i, (images, labels) in enumerate(test_loader):

        if cuda_avail:
                images = Variable(images.cuda())
                labels = Variable(labels.cuda())

        #Predict classes using images from the test set
        outputs = model(images)
        _,prediction = torch.max(outputs.data, 1)
        prediction = prediction.cpu().numpy()
        test_acc += torch.sum(prediction == labels.data) #line 155

    #Compute the average acc and loss over all 10000 test images
    test_acc = test_acc / 10000

return test_acc

After a quick search I see that the error is probably related to the comparison between the prediction and labels, as seem in this SO question.

Any idea on how to fix this?


title: slow Inference time for Neural Net

body:
I have written a simple Fully connected Neural Network in Pytorch. I saved the model and loaded it in C++ using LibTorch but my inference time is pretty slow for my application field. Inference time right now is about 10 ms. Is it normal or am I doing something wrong?

I measured the inference time on python only first. then to maybe make it faster I loaded the network on C++ but it didn't help.

Here is the code for network

class network(nn.Module):
    def __init__(self):
        super(network,self).__init__()
        input_nodes = 362
        hidden_nodes1 = 50
        hidden_nodes2 = 30
        output_nodes = 1

    self.fc1 = nn.Linear(input_nodes,hidden_nodes1)
    nn.init.xavier_uniform_(self.fc1.weight)
    self.bn1 = nn.BatchNorm1d(num_features=hidden_nodes1)

    self.fc2 = nn.Linear(hidden_nodes1,hidden_nodes2)
    nn.init.xavier_uniform_(self.fc2.weight)
    self.bn2 = nn.BatchNorm1d(num_features = hidden_nodes2)

    self.fc3 = nn.Linear(hidden_nodes2,output_nodes)
    nn.init.xavier_uniform_(self.fc3.weight)

    self.out_act = nn.Sigmoid();

def forward(self,X):
    X = F.relu(self.bn1(self.fc1(X)))
    X = self.fc2(X)
    X = F.dropout2d(X,p=0.3)
    X = F.relu(X)
    X = self.fc3(X)
    out = self.out_act(X)
    return out

I want inference to somewhat take around 0.01 milliseconds.


title: torchtext field with values converted to ids got error integer is required

body:
I followed this tutorial
http://www.programmersought.com/article/2609385756/

to create a TabularDataset with data that already tokenized and converted to ids and I do not want to use vocab or build vocab because the data is numerical

so I defined my field variable as:

myField = Field(tokenize= x_tokenize, use_vocab=False, sequential=True)
train,val, test = data.TabularDataset.splits(path='./', train=train_path, validation=valid_path, test=test_path ,format='csv', fields=data_fields, skip_header=True)

train output:

print(vars(train[0])['src'])
#output this [101, 3177, 3702, 11293, 1116, 102]

and I used a BucketIterator:

train_iter= BucketIterator(train,
                       batch_size=BATCH_SIZE,
                       device = DEVICE,
                       sort_key=lambda x: (len(x.src), len(x.trg)),
                       train=True,
                       batch_size_fn=batch_size_fn,
                       repeat=False)

when I run this code:

batch = next(iter(train_iter))

I got
TypeError: an integer is required (got type list)

  TypeError                                 Traceback (most recent call
  last)  in ()
  ----> 1 batch = next(iter(train_iter))

  3 frames
  /usr/local/lib/python3.6/dist-packages/torchtext/data/iterator.py in
  iter(self)
      155                     else:
      156                         minibatch.sort(key=self.sort_key, reverse=True)
  --> 157                 yield Batch(minibatch, self.dataset, self.device)
      158             if not self.repeat:
      159                 return

  /usr/local/lib/python3.6/dist-packages/torchtext/data/batch.py in
  init(self, data, dataset, device)
       32                 if field is not None:
       33                     batch = [getattr(x, name) for x in data]
  ---> 34                     setattr(self, name, field.process(batch, device=device))
       35
       36     @classmethod

  /usr/local/lib/python3.6/dist-packages/torchtext/data/field.py in
  process(self, batch, device)
      199         """
      200         padded = self.pad(batch)
  --> 201         tensor = self.numericalize(padded, device=device)
      202         return tensor
      203

  /usr/local/lib/python3.6/dist-packages/torchtext/data/field.py in
  numericalize(self, arr, device)
      321                 arr = self.postprocessing(arr, None)
      322
  --> 323         var = torch.tensor(arr, dtype=self.dtype, device=device)
      324
      325         if self.sequential and not self.batch_first:

  TypeError: an integer is required (got type list)


title: How does one vectorize reinforcement learning environments?

body: I have a Python class that conforms to OpenAI's environment API, but it's written in non-vectorized form i.e. it receives one input action per step and returns one reward per step. How do I vectorize the environment? I haven't been able to find any clear explanation on GitHub.


title: fastai ImportError: file too short

body:
I installed fastai as explained here in my ubuntu machine https://forums.fast.ai/t/wiki-thread-lesson-1/6825/63

However, when i import libraries, i am getting below error message.

from fastai import *
from fastai.text import *

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-9acdcc7330cd> in <module>
----> 1 from fastai import *        # Quick access to most common functionality
      2 from fastai.text import *   # Quick access to NLP functionality

~/.local/lib/python3.6/site-packages/fastai/__init__.py in <module>
----> 1 from .basic_train import *
      2 from .callback import *
      3 #from .callbacks import *
      4 from .core import *
      5 from .basic_data import *

~/.local/lib/python3.6/site-packages/fastai/basic_train.py in <module>
      1 "Provides basic training and validation with `Learner`"
----> 2 from .torch_core import *
      3 from .basic_data import *
      4 from .callback import *
      5

~/.local/lib/python3.6/site-packages/fastai/torch_core.py in <module>
      1 "Utility functions to help deal with tensors"
----> 2 from .imports.torch import *
      3 from .core import *
      4
      5 AffineMatrix = Tensor

~/.local/lib/python3.6/site-packages/fastai/imports/__init__.py in <module>
----> 1 from .core import *
      2 from .torch import *

~/.local/lib/python3.6/site-packages/fastai/imports/core.py in <module>
      4 import abc, collections, hashlib, itertools, json, operator, pathlib
      5 import mimetypes, inspect, typing, functools, importlib
----> 6 import html, re, spacy, requests, tarfile, numbers
      7
      8 from abc import abstractmethod, abstractproperty

~/.local/lib/python3.6/site-packages/spacy/__init__.py in <module>
      6
      7 # These are imported as part of the API
----> 8 from thinc.neural.util import prefer_gpu, require_gpu
      9
     10 from .cli.info import info as cli_info

~/.local/lib/python3.6/site-packages/thinc/neural/__init__.py in <module>
----> 1 from ._classes.model import Model

~/.local/lib/python3.6/site-packages/thinc/neural/_classes/model.py in <module>
     10
     11 from .. import util
---> 12 from ..train import Trainer
     13 from ..ops import NumpyOps, CupyOps
     14 from ..mem import Memory

~/.local/lib/python3.6/site-packages/thinc/neural/train.py in <module>
      1 from __future__ import unicode_literals, print_function
      2
----> 3 from .optimizers import Adam, SGD, linear_decay
      4 from .util import minibatch
      5

optimizers.pyx in init thinc.neural.optimizers()

ops.pyx in init thinc.neural.ops()

ImportError: /home/user/.local/lib/python3.6/site-packages/murmurhash/mrmr.cpython-36m-x86_64-linux-gnu.so: file too short

Can anyone help me on this please? I searched on the internet for hours and didn't find any information. Thanks


title: nibabel.filebasedimages.ImageFileError: Cannot work out file type

body:
i am getting this error when i am applying medical torch code , i know my data is tif and gif format , and i think this part " mt_datasets.SegmentationPair2D" deals just with “nii” format what i can change to make it work with other fromat than “nii”

ROOT_DIR_GMCHALLENGE = r'C:\Users~~~\eyedata\train'
mri_input_filename = os.path.join(ROOT_DIR_GMCHALLENGE,
                                          'img')
mri_gt_filename = os.path.join(ROOT_DIR_GMCHALLENGE,
                                       'label')

pair = mt_datasets.SegmentationPair2D(mri_input_filename, mri_gt_filename)
slice_pair = pair.get_pair_slice(0)
input_slice = slice_pair["input"]
gt_slice = slice_pair["gt"]


title: Why cant I use ONNX Runtime training with pytorch?

body:
When I run
from onnxruntime.capi.ort_trainer import ORTTrainer
as stated at https://github.com/microsoft/onnxruntime/#training-start, I get this error:
ModuleNotFoundError: No module named 'onnxruntime.capi.ort_trainer'
What can I do to fix this?
I have onnxruntime installed via pip but I couldn't even find "ort_trainer.py" in [python path]/site_packages/onnx-runtime/capi


title: How to solve "NotImplementedError"

body:
I defined a three layer convolution layer(self.convs) ,the input tensor has the shape([100,10,24])
x_convs = self.convs(Variable(torch.from_numpy(X).type(torch.FloatTensor)))

>>Variable(torch.from_numpy(X).type(torch.FloatTensor)).shape

torch.Size([100, 10, 24])

>>self.convs

ModuleList(
  (0): ConvBlock(
    (conv): Conv1d(24, 8, kernel_size=(5,), stride=(1,), padding=(2,))
    (relu): ReLU()
    (maxpool): AdaptiveMaxPool1d(output_size=10)
    (zp): ConstantPad1d(padding=(1, 0), value=0)
  )
  (1): ConvBlock(
    (conv): Conv1d(8, 8, kernel_size=(5,), stride=(1,), padding=(2,))
    (relu): ReLU()
    (maxpool): AdaptiveMaxPool1d(output_size=10)
    (zp): ConstantPad1d(padding=(1, 0), value=0)
  )
  (2): ConvBlock(
    (conv): Conv1d(8, 8, kernel_size=(5,), stride=(1,), padding=(2,))
    (relu): ReLU()
    (maxpool): AdaptiveMaxPool1d(output_size=10)
    (zp): ConstantPad1d(padding=(1, 0), value=0)
  )
)

When I excuate x_convs = self.convs(Variable(torch.from_numpy(X).type(torch.FloatTensor))), it gives me the error
    `94             registered hooks while the latter silently ignores them.

     95         """

---> 96         raise NotImplementedError`

The ConvBlock is defined as below
class ConvBlock(nn.Module):
    def __init__(self, T, in_channels, out_channels, filter_size):
        super(ConvBlock, self).__init__()
        padding = self._calc_padding(T, filter_size)
        self.conv=nn.Conv1d(in_channels, out_channels, filter_size, padding=padding)
        self.relu=nn.ReLU()
        self.maxpool=nn.AdaptiveMaxPool1d(T)
        self.zp=nn.ConstantPad1d((1, 0), 0)

    def _calc_padding(self, Lin, kernel_size, stride=1, dilation=1):
        p = int(((Lin-1)*stride + 1 + dilation*(kernel_size - 1) - Lin)/2)
        return p

    def forward(self, x):
        x = x.permute(0,2,1)
        x = self.conv(x)
        x = self.relu(x)
        x = self.maxpool(x)
        x = x.permute(0,2,1)
        return x

The "forward" function has correct indent, so I cannot figure it out what is going on.


title: TypeError: reshape(): argument 'input' (position 1) must be Tensor, not numpy.ndarray

body:
I am a high school student who doesn't having much experience in using PyTorch and LIME. I'm having a lot of trouble with my image shape. Initially my image shape was (3,224,224), however the LIME algorithm only works with images that are in this shape(...,...,3). As a result, I tried transposing the image earlier. It seemed that I made some more progress by doing that, however, now I am getting a different error. Here is some of my code to understand what I have been doing before the error came up.
def get_preprocess_transform():
transf = transforms.Compose([
  #  transforms.ToPILImage(), #had to convert image to PIL as error was showing up two cells below about needing it in pil
    transforms.Resize(input_size),
    transforms.CenterCrop(input_size),
    transforms.ToTensor(),
    transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),

])

return transf

preprocess_transform = get_preprocess_transform() ## use your data_transform but in a method version

def batch_predict(image):
    model_ft.eval()
    batch = torch.reshape(image,(1,3,224,224))
    print(type(batch))

    logits = model_ft(batch)
    probs = F.softmax(logits, dim=1)
    return probs.detach().cpu().numpy()

print(img_t.shape)
img_t = torch.reshape(img_t,(1,3,224,224))
test_pred = batch_predict(img_t)
test_pred.squeeze().argmax()

img_t = np.ones((3, 224, 224))
np.transpose(img_t, (2,1,0)).shape

img_x = np.transpose(img_t, (2, 1, 0))
print(img_x.shape)

from lime import lime_image

explainer = lime_image.LimeImageExplainer()
explanation = explainer.explain_instance(img_x, ## pass your image, do not transform
                                        batch_predict, # classification function
                                        top_labels=5,
                                        hide_color=0,
                                        num_samples=1000)

Here is the error message that comes from the "explainer cell"


title: How to handle imbalanced dataset in PyTorch?

body:
all.
I have some problems when I handled an imbalanced dataset.
I tried to use WeightRandomSampler to over-sampling/under-sampling, but the result is worse than I directly train on an imbalanced dataset.
Here, I list the detail in the imbalanced dataset and how to calculate my weight for WeightRandomSampler.
The number of each class:
Class 1: 1865
Class 1: 2677
Class 2: 3602
Class 3: 916
Class 4: 4354
Class 5: 4061
Class 6: 892
Class 7: 1718
Class 8: 3417
Class 9: 152
My code block:
weights = list(1.0 / np.array([1865, 2677, 3602, 916, 4354, 4061, 892, 1718, 3417, 152]))
sample_weights = [weights[t] for t in train_ds.target]
sampler = WeightedRandomSampler(sample_weights, len(train_ds), replacement=True)
train_dl = torch.utils.data.DataLoader(dataset=train_ds, batch_size=train_bs, sampler=sampler)

Also, I tried to use weight for nn.CrossEntropy(), but have some errors.
Thanks


title: Sequence generation using pytorch RNN

body:
I asked this over at the pytorch discussion forum as well (link) but I figured I'd ask here in case it gets more traction.

I trained a model using pytorch's nn.RNN() module by feeding it batches of sequences (of varying seq_len) and properly padding when sequences in the batch were of different lengths.

I'm now trying to generate sequences using the trained RNN model.  I make a sequence of length one containing the index of my GO_TOKEN and feed that to my trained RNN. Unfortunately what I'm getting back are all 0s.  Has anyone done this before?

Some code in case that helps:

class Model(nn.Module):

    def __init__(self,
                 inputs_vocab_size,
                 embedding_dim,
                 out_units=128,
                 hidden_size=128,
                 nlayers=1,
                 num_directions=1,
                 dropout=0.1):

        super(Model, self).__init__()

        ...

        self.encoder = nn.Embedding(inputs_vocab_size, embedding_dim).cuda()
        self.rnn = nn.RNN(embedding_dim, hidden_size, nlayers, dropout=0.5)
        self.decoder = nn.Linear(hidden_size * num_directions, inputs_vocab_size)

        ...

Any ideas?


title: PyTorch how to implement disconnection (connections and corresponding gradients are masked)?

body:
I try to implement the following graph. As you can see, the neurons are not fully connected, i.e., the weights are masked and so are their corresponding gradients.

import torch
import numpy as np

x = torch.rand((3, 1))
# tensor([[ 0.8525],
#         [ 0.1509],
#         [ 0.9724]])

weights = torch.rand((2, 3), requires_grad=True)
# tensor([[ 0.3240,  0.0792,  0.6858],
#         [ 0.5248,  0.4565,  0.3625]])

mask = torch.Tensor([[0,1,0],[1,0,1]])
# tensor([[ 0.,  1.,  0.],
#         [ 1.,  0.,  1.]])

mask_weights = weights * mask
# tensor([[ 0.0000,  0.0792,  0.0000],
#         [ 0.5248,  0.0000,  0.3625]])

y = torch.mm(mask_weights, x)
# tensor([[ 0.0120],
#         [ 0.7999]])

This question is originally posted at Pytorch Forum. Note the above way

  mask_weights = weights * mask

is NOT suitable since corresponding gradients are not 0.

Is there an elegant way to do that please?

Thank you in advance.


title: Assign Torch and Tensorflow models two separate GPUs

body:
I am comparing two pre-trained models, one is in Tensorflow and one is in Pytorch, on a machine that has multiple GPUs. Each model fits on one GPU. They are both loaded in the same Python script. How can I assign one GPU to the Tensorflow model and another GPU to the Pytorch model?

Setting CUDA_VISIBLE_DEVICES=0,1 only tells both models that these GPUs are available - how can I (within Python I guess), make sure that Tensorflow takes GPU 0 and Pytorch takes GPU 1?


title: How can I use a PyTorch DataLoader for Reinforcement Learning?

body:
I'm trying to set up a generalized Reinforcement Learning framework in PyTorch to take advantage of all the high-level utilities out there which leverage PyTorch DataSet and DataLoader, like Ignite or FastAI, but I've hit a blocker with the dynamic nature of Reinforcement Learning data:

Data Items are generated from code, not read from a file, and they are dependent on previous actions and model results, therefore each nextItem call needs access to the model state.
Training episodes are not fixed length so I need a dynamic batch size as well as a dynamic total data set size. My preference would be to use a terminating condition function instead of a number. I could "possibly" do this with padding, as in NLP sentence processing, but that's a real hack.

My Google and StackOverflow searches so far have yielded zilch. Anyone here know of existing solutions or workarounds to using DataLoader or DataSet with Reinforcement Learning? I hate to loose access to all the existing libraries out there which depend on those.


title: Best way to run a trained PyTorch LSTM/GRU model fully in the browser

body:
I'm looking into running a trained PyTorch model (containing LSTM/GRU layers) fully in the browser (no backend) as part of an interactive blog post. I've looked at ONNX.js, and that works great, but not for a model containing a GRU layer. I saw someone comment on the ONNX.js github that Gated RNN's are not supported yet, but that was over half a year ago and I can't find any other information about this.

Other than that, it seems like the best option would be to just rewrite the model in Tensorflow and export to Tensorflow.js.

Is there an easier and more direct solution?


title: How to do hyperparameter tuning in Detectron2

body:
Detectron2
COCO Person Keypoint Detection Baselines with Keypoint R-CNN R50-FPN
How do I do hyperparameter tuning with the model above? Which files do I have to open?
Thanks


title: PyTorch dataloader shows odd behavior with string dataset

body:
I'm working on an NLP problem and am using PyTorch. For some reason, my dataloader is returning malformed batches. I have input data that comprises sentences and integer labels.
The sentences can either a list of sentences or a list of list of tokens. I will later convert the tokens to integers in a downstream component.
list_labels = [ 0, 1, 0]

# List of sentences.
list_sentences = [ 'the movie is terrible',
                   'The Film was great.',
                   'It was just awful.']

# Or list of list of tokens.
list_sentences = [['the', 'movie', 'is', 'terrible'],
                  ['The', 'Film', 'was', 'great.'],
                  ['It', 'was', 'just', 'awful.']]

I created the following custom dataset:
import torch
from torch.utils.data import DataLoader, Dataset

class MyDataset(torch.utils.data.Dataset):

    def __init__(self, sentences, labels):

        self.sentences = sentences
        self.labels = labels

    def __getitem__(self, i):
        result = {}
        result['sentences'] = self.sentences[i]
        result['label'] = self.labels[i]
        return result

    def __len__(self):
        return len(self.labels)

When I provide input in the form of a list of sentences, the dataloader correctly returns batches of complete sentences. Note that batch_size=2:
list_sentences = [ 'the movie is terrible', 'The Film was great.', 'It was just awful.']
list_labels = [ 0, 1, 0]

dataset = MyDataset(list_sentences, list_labels)
dataloader = DataLoader(dataset, batch_size=2)

batch = next(iter(dataloader))
print(batch)
# {'sentences': ['the movie is terrible', 'The Film was great.'], <-- Great! 2 sentences in batch!
#  'label': tensor([0, 1])}

The batch correctly contains two sentences and two labels because batch_size=2.
However, when I instead enter the sentences as pre-tokenized list of list of token, I get weird results:
list_sentences = [['the', 'movie', 'is', 'terrible'], ['The', 'Film', 'was', 'great.'], ['It', 'was', 'just', 'awful.']]
list_labels = [ 0, 1, 0]

dataset = MyDataset(list_sentences, list_labels)
dataloader = DataLoader(dataset, batch_size=2)

batch = next(iter(dataloader))
print(batch)
# {'sentences': [('the', 'The'), ('movie', 'Film'), ('is', 'was'), ('terrible', 'great.')], <-- WHAT?
#  'label': tensor([0, 1])}

Note that this batch's sentences is one single list with tuples of word pairs. I was expecting sentences to be a list of two lists, like this:
{'sentences': [['the', 'movie', 'is', 'terrible'], ['The', 'Film', 'was', 'great.']

What is going on?


title: How to replicate PyTorch normalization in OpenCV or NumPy?

body:
I need to replicate PyTorch image normalization in OpenCV or NumPy.
Quick backstory: I'm doing a project where I'm training in PyTorch but will have to inference in OpenCV due to deploying to an embedded device where I won't have the storage space to install PyTorch.  After training in PyTorch and saving a PyTorch graph I'm then converting to an ONNX graph.  For inferencing in OpenCV I'm opening the image as an OpenCV image (i.e. NumPy array), then resizing, then successively calling cv2.normalize, cv2.dnn.blobFromImage, net.setInput, and net.forward.
I'm getting slightly different accuracy results when test inferencing in PyTorch vs inferencing in OpenCV, and I suspect the difference is due to the normalization process producing a slightly different result between the two.
Here is a quick script I put together to show the difference on a single image.  Note that I'm using grayscale (single-channel) and I'm normalizing into the -1.0 to +1.0 range:
# scratchpad.py

import torch
import torchvision

import cv2
import numpy as np
import PIL
from PIL import Image

TRANSFORM = torchvision.transforms.Compose([
    torchvision.transforms.Resize((224, 224)),
    torchvision.transforms.ToTensor(),
    torchvision.transforms.Normalize([0.5], [0.5])
])

def main():
    # 1st show PyTorch normalization

    # open the image as an OpenCV image
    openCvImage = cv2.imread('image.jpg', cv2.IMREAD_GRAYSCALE)
    # convert OpenCV image to PIL image
    pilImage = PIL.Image.fromarray(openCvImage)
    # convert PIL image to a PyTorch tensor
    ptImage = TRANSFORM(pilImage).unsqueeze(0)
    # show the PyTorch tensor info
    print('\nptImage.shape = ' + str(ptImage.shape))
    print('ptImage max = ' + str(torch.max(ptImage)))
    print('ptImage min = ' + str(torch.min(ptImage)))
    print('ptImage avg = ' + str(torch.mean(ptImage)))
    print('ptImage: ')
    print(str(ptImage))

    # 2nd show OpenCV normalization

    # resize the image
    openCvImage = cv2.resize(openCvImage, (224, 224))
    # convert to float 32 (necessary for passing into cv2.dnn.blobFromImage which is not show here)
    openCvImage = openCvImage.astype('float32')
    # use OpenCV version of normalization, could also do this with numpy
    cv2.normalize(openCvImage, openCvImage, 1.0, -1.0, cv2.NORM_MINMAX)
    # show results
    print('\nopenCvImage.shape = ' + str(openCvImage.shape))
    print('openCvImage max = ' + str(np.max(openCvImage)))
    print('openCvImage min = ' + str(np.min(openCvImage)))
    print('openCvImage avg = ' + str(np.mean(openCvImage)))
    print('openCvImage: ')
    print(str(openCvImage))

    print('\ndone !!\n')
# end function

if __name__ == '__main__':
    main()

Here is the test image that I'm using:

and here are the results I'm getting currently:
$ python3 scratchpad.py

ptImage.shape = torch.Size([1, 1, 224, 224])
ptImage max = tensor(0.9608)
ptImage min = tensor(-0.9686)
ptImage avg = tensor(0.1096)
ptImage:
tensor([[[[ 0.0431, -0.0431,  0.1294,  ...,  0.8510,  0.8588,  0.8588],
          [ 0.0510, -0.0510,  0.0980,  ...,  0.8353,  0.8510,  0.8431],
          [ 0.0588, -0.0431,  0.0745,  ...,  0.8510,  0.8588,  0.8588],
          ...,
          [ 0.6157,  0.6471,  0.5608,  ...,  0.6941,  0.6627,  0.6392],
          [ 0.4902,  0.3961,  0.3882,  ...,  0.6627,  0.6471,  0.6706],
          [ 0.3725,  0.4039,  0.5451,  ...,  0.6549,  0.6863,  0.6549]]]])

openCvImage.shape = (224, 224)
openCvImage max = 1.0000001
openCvImage min = -1.0
openCvImage avg = 0.108263366
openCvImage:
[[ 0.13725497 -0.06666661  0.20000008 ...  0.8509805   0.8666668
   0.8509805 ]
 [ 0.15294124 -0.06666661  0.09019614 ...  0.8274511   0.8431374
   0.8274511 ]
 [ 0.12156869 -0.06666661  0.0196079  ...  0.8509805   0.85882366
   0.85882366]
 ...
 [ 0.5843138   0.74117655  0.5450981  ...  0.83529425  0.59215695
   0.5764707 ]
 [ 0.6862746   0.34117654  0.39607853 ...  0.67843145  0.6705883
   0.6470589 ]
 [ 0.34117654  0.4117648   0.5215687  ...  0.5607844   0.74117655
   0.59215695]]

done !!

As you can see the results are similar but definitely not exactly the same.
How can I do the normalization in OpenCV and have it come out exactly or almost exactly the same as the PyTorch normalization?  I've tried various options in both OpenCV and with NumPy but could not get it any closer than the above results, which are substantially different.
-- Edit ---------------------------
In response to Ivan, I also tried this:
# resize the image
openCvImage = cv2.resize(openCvImage, (224, 224))
# convert to float 32 (necessary for passing into cv2.dnn.blobFromImage which is not show here)
openCvImage = openCvImage.astype('float32')
mean = np.mean(openCvImage)
stdDev = np.std(openCvImage)
openCvImage = (openCvImage - mean) / stdDev
# show results
print('\nopenCvImage.shape = ' + str(openCvImage.shape))
print('openCvImage max = ' + str(np.max(openCvImage)))
print('openCvImage min = ' + str(np.min(openCvImage)))
print('openCvImage avg = ' + str(np.mean(openCvImage)))
print('openCvImage: ')
print(str(openCvImage))

Which results in:
openCvImage.shape = (224, 224)
openCvImage max = 2.1724665
openCvImage min = -2.6999729
openCvImage avg = 7.298528e-09
openCvImage:
[[ 0.07062991 -0.42616782  0.22349077 ...  1.809422    1.8476373
   1.809422  ]
 [ 0.10884511 -0.42616782 -0.04401573 ...  1.7520993   1.7903144
   1.7520993 ]
 [ 0.0324147  -0.42616782 -0.21598418 ...  1.809422    1.8285296
   1.8285296 ]
 ...
 [ 1.1597633   1.5419154   1.0642253  ...  1.7712069   1.178871
   1.1406558 ]
 [ 1.4081622   0.56742764  0.70118093 ...  1.3890547   1.3699471
   1.3126242 ]
 [ 0.56742764  0.7393961   1.0069026  ...  1.1024406   1.5419154
   1.178871  ]]

Which is similar to the PyTorch normalization but clearly not the same.
I'm attempting to achieve normalization in OpenCV that produces the same result as the PyTorch normalization.
I realize that due to slight differences in the resizing operation (and possibly very small rounding differences) I'll probably never get exactly the same normalized result but I'd like to get as close as possible to the PyTorch result.


Today I have started to learn Pytorch and I stuck here. The code piece in the comment raises this error:
TypeError: Cannot interpret 'torch.uint8' as a data type
For changing the data type of the tensor I used: quzu_torch = quzu_torch.type(torch.float) But this time I got this error:
TypeError: Cannot interpret 'torch.float32' as a data type
import numpy as np
import torch
from matplotlib import pyplot as plt

quzu = np.array(Image.open('wool.jpg').resize((224,224)))

quzu_torch = torch.from_numpy(quzu)

plt.imsave('quzucuq.jpg', quzu)

# quzu_torch = quzu_torch.type(torch.bool)
plt.imsave('quzucuq_tensor.jpg', quzu_torch)```


title: Machine Translation using Google T5

body:
I want to use the Google T5 model for my machine translation task. I wish to finetune the pretrained T5 model using my own dataset. I have been searching the internet to do this task but have been unsuccessful in finding a pretrained model which I could finetune further. I understand this is a new model and hence probably lacks tutorials on Medium. If some documentation or tutorial is available, could you please point me there, or if you have done this before could you please share your code?
I have searched through for documentation of T5. HuggingFace claims that T5 is a work in progress and/or refers to PyTorch where I couldn't get any documentation.


title: Bert pre-trained model giving random output each time

body:
I was trying to add an additional layer after huggingface bert transformer, so I used BertForSequenceClassification inside my nn.Module Network. But, I see the model is giving me random outputs when compared to loading the model directly.

Model 1:

from transformers import BertForSequenceClassification

model = BertForSequenceClassification.from_pretrained('bert-base-uncased', num_labels = 5) # as we have 5 classes

import torch
from transformers import BertTokenizer
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

input_ids = torch.tensor(tokenizer.encode(texts[0], add_special_tokens=True, max_length = 512)).unsqueeze(0)  # Batch size 1

print(model(input_ids))

Out:

(tensor([[ 0.3610, -0.0193, -0.1881, -0.1375, -0.3208]],
        grad_fn=<AddmmBackward>),)

Model 2:

import torch
from torch import nn

class BertClassifier(nn.Module):
    def __init__(self):
        super(BertClassifier, self).__init__()
        self.bert = BertForSequenceClassification.from_pretrained('bert-base-uncased', num_labels = 5)
        # as we have 5 classes

        # we want our output as probability so, in the evaluation mode, we'll pass the logits to a softmax layer
        self.softmax = torch.nn.Softmax(dim = 1) # last dimension
    def forward(self, x):
        print(x.shape)
        x = self.bert(x)

        if self.training == False: # in evaluation mode
            pass
            #x = self.softmax(x)

        return x

# create our model

bertclassifier = BertClassifier()

print(bertclassifier(input_ids))

torch.Size([1, 512])
torch.Size([1, 5])
(tensor([[-0.3729, -0.2192,  0.1183,  0.0778, -0.2820]],
        grad_fn=<AddmmBackward>),)

They should be the same model, right. I found a similar issue here but no reasonable explanation https://github.com/huggingface/transformers/issues/2770

Does Bert has some ranomized parameter if so how to get reproducible output?
Why the two models give me different outputs? Is there something I'm doing wrong?


I'm new in pytorch and just try to write a network. The data.shape is (204,6170) and the last 5 cols are some labels. The number in data is float number like 0.030822.
#%%
from sklearn.feature_selection import RFE
import numpy as np
import pandas as pd
import torch
import torch.nn as nn
from sklearn.model_selection import train_test_split
import torch.functional as F

#%%
data = pd.read_table("table.log")
data_x = data.iloc[:, 0:(data.shape[1]-5)]
data_y = data.loc[:, 'target']
X_train, X_test, y_train, y_test = train_test_split(data_x,data_y,test_size=0.2,random_state=0)

#%%
from sklearn.linear_model import LinearRegression
lr = LinearRegression(normalize=True)
lr.fit(X_train,y_train)
rfe1 = RFE(estimator=lr,n_features_to_select=2000)
rfe1 = rfe1.fit(X_train,y_train)

#%%
x_train_rfe1 = X_train[X_train.columns[rfe1.support_]]
print(x_train_rfe1.head())
class testmodel(nn.Module):
    def __init__(self):
        super(testmodel,self).__init__()
        self.conv = nn.Sequential(
            nn.Conv1d(1500, 500, 1500, 0, 0),
            nn.ReLU(),
            nn.Conv1d(500, 100, 500, 0),
            nn.ReLU(),
            nn.Conv1d(100, 20, 100, 0),
            nn.Sigmoid()
        )
    def forward(self,x):
        x = self.conv
        return x
#%%
x_train_rfe1 = torch.Tensor(x_train_rfe1.values)
y_train = torch.Tensor(y_train.values.astype(np.int64))
model = testmodel()
y = model(x_train_rfe1)

criterion = nn.MSELoss()
loss = criterion(y, y_train)
print(loss)

error message:
Traceback (most recent call last):
  File "<input>", line 7, in <module>
  File "/miniconda3/envs/ml/lib/python3.8/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/miniconda3/envs/ml/lib/python3.8/site-packages/torch/nn/modules/loss.py", line 431, in forward
    return F.mse_loss(input, target, reduction=self.reduction)
  File "/miniconda3/envs/ml/lib/python3.8/site-packages/torch/nn/functional.py", line 2203, in mse_loss
    if not (target.size() == input.size()):
  File "/miniconda3/envs/ml/lib/python3.8/site-packages/torch/nn/modules/module.py", line 575, in __getattr__
    raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'Sequential' object has no attribute 'size'

Where can the error was?
Is the internet usually written like this?
How could I improve it?


title: How to transfer the follow Embedding code in tensorflow to pytorch?

body:
I have an Embedding code in Tensorflow as follow

self.input_u = tf.placeholder(tf.int32, [None, user_length], name="input_u")
with tf.name_scope("user_embedding"):
        self.W1 = tf.Variable(
            tf.random_uniform([user_vocab_size, embedding_size], -1.0, 1.0),
            name="W")
        self.embedded_user = tf.nn.embedding_lookup(self.W1, self.input_u)
        self.embedded_users = tf.expand_dims(self.embedded_user, -1)

And I want to re-write in pytorch, How to do that?


title: Difference between model.train(False) and required_grad = False

body:
I use the Pytorch library and i'm looking for a way to make the weights and biases in my model to freeze.
I saw these 2 options:

model.train(False)

for param in model.parameters(): param.requires_grad = False

What is the difference (if there is any) and which one should i use to freeze the current state of my model?


title: Uneven usage of GPU memory by DataParallel causes out-of-memory error

body:
I am running my NN model using DataParallel on 3 GPUs. In one GPU, the memory usage rises to 12gb and as a result, the program stops after giving an out-of-memory error. While one GPU uses a large amount of memory (~12gb), the other two GPU memory usage is quite low (~2-3gb).

Is there any way I can make sure that the GPU memory usage is balanced when DataParallel is used in PyTorch?

Edit: I am working on Neural Machine Translation (NMT) and I am sharing part of my code where I am using DataParallel.

class NMT(nn.Module):
    """A sequence-to-sequence model for machine translation."""

    def __init__(self, dictionary, embedding_index, args):
        super(NMT, self).__init__()

        self.config = args
        self.src_embedding = EmbeddingLayer(len(dictionary[0]), False, self.config)
        self.tgt_embedding = EmbeddingLayer(len(dictionary[1]), True, self.config)
        if embedding_index is not None:
            if isinstance(embedding_index, tuple):
                self.src_embedding.init_embedding_weights(dictionary[0], embedding_index[0], self.config.emsize)
                self.tgt_embedding.init_embedding_weights(dictionary[1], embedding_index[1], self.config.emsize)
            else:
                self.src_embedding.init_embedding_weights(dictionary[0], embedding_index, self.config.emsize)

        self.encoder_decoder = Encoder_Decoder(args)
        if torch.cuda.device_count() > 1:
            self.encoder_decoder = torch.nn.DataParallel(self.encoder_decoder)

        # word decoding layer
        self.out = nn.Linear(self.config.emsize, len(dictionary[1]))
        # tie target embedding weights with decoder prediction layer weights
        self.tgt_embedding.embedding.weight = self.out.weight

    def forward(self, s1, s1_len, s2, s2_len):
        """
        Forward computational step of sequence-to-sequence to machine translation.
        :param s1: source sentences [batch_size x max_s1_length]
        :param s1_len: source sentences' length [batch_size]
        :param s2: target sentences [batch_size x max_s2_length]
        :param s2_len: target sentences' length [batch_size]
        :return: decoding loss [batch_size]
        """
        # embedded_s1 = batch_size x max_s1_length x em_size
        embedded_s1 = self.src_embedding(s1)
        # embedded_s2 = batch_size x max_s2_length x em_size
        embedded_s2 = self.tgt_embedding(s2)

        # decoder_out: batch_size x max_s2_length x em_size
        decoder_out = self.encoder_decoder(embedded_s1, s1_len, embedded_s2)
        predictions = f.log_softmax(self.out(decoder_out.view(-1, decoder_out.size(2))), 1)
        predictions = predictions.view(*decoder_out.size()[:-1], -1)

        decoding_loss, total_local_decoding_loss_element = 0, 0
        for idx in range(s2.size(1) - 1):
            local_loss, num_local_loss = self.compute_decoding_loss(predictions[:, idx, :], s2[:, idx + 1], idx, s2_len)
            decoding_loss += local_loss
            total_local_decoding_loss_element += num_local_loss

        if total_local_decoding_loss_element > 0:
            decoding_loss = decoding_loss / total_local_decoding_loss_element

        return decoding_loss

Here, I am using specifically DataParallel.

if torch.cuda.device_count() > 1:
    self.encoder_decoder = torch.nn.DataParallel(self.encoder_decoder)

  A high-level overview of the model is depicted in the following figure.

I have particularly kept the target_embedding layer and the next_word_prediction layer outside DataParallel because the layer is associated with a large number of parameters (24M) and they are tied.
Also, I am not training the source_embedding layer, the weights are frozen. Following this setting, I have observed improved runtime but with a small sized data. When I try large dataset, it gives me an out-of-memory error.


title: Dimensions Not Matching In PyTorch Linear Layer

body:
Following the training lesson in PyTorch on this page:  https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#sphx-glr-beginner-blitz-cifar10-tutorial-py

It's basically their 'Hello World!' version of an image classifier.

What I'm trying to do is manually code the training steps in the network to make sure I understand each one, but I am currently getting a dimension mismatch in one of my linear layers, which has me stumped.  Especially since (AFAIK) I'm recreating the steps in the tutorial exactly.

Anyways........

MY NETWORK:

class net(nn.Module):
def __init__(self):
    super(net, self).__init__()
    self.conv1 = nn.Conv2d(3, 6, 5)
    self.pool  = nn.MaxPool2d(2, 2)
    self.conv2 = nn.Conv2d(6, 16, 5)
    self.fc1   = nn.Linear(16*5*5, 120)
    self.fc2   = nn.Linear(120, 84)
    self.fc2   = nn.Linear(84, 10)

def forward(self, x):
    x = self.pool(F.relu(self.conv1(x)))
    x = self.pool(F.relu(self.conv2(x)))
    x = x.view(-1, 16 * 5 * 5)
    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(x))
    x = self.fc3(x)

    return x

net = net()

I believe this is exactly as they have it on their own page.

I'm trying to calculate the following step without a loop:

for epoch in range(2):  # loop over the dataset multiple times

  running_loss = 0.0
  for i, data in enumerate(trainloader, 0):
    # get the inputs; data is a list of [inputs, labels]
    inputs, labels = data

    # zero the parameter gradients
    optimizer.zero_grad()

    # forward + backward + optimize
    outputs = net(inputs)
    loss = criterion(outputs, labels)
    loss.backward()
    optimizer.step()

    # print statistics
    running_loss += loss.item()
    if i % 2000 == 1999:    # print every 2000 mini-batches
        print('[%d, %5d] loss: %.3f' %
              (epoch + 1, i + 1, running_loss / 2000))
        running_loss = 0.0

  print('Finished Training')

What I'm doing is this:

data = enumerate(trainloader)
inputs, labels = next(data)[1]
outputs = net(inputs)

And the last line gives me the following traceback:

RuntimeError                              Traceback (most recent call last)
<ipython-input-285-d4be5abf5bb1> in <module>
----> 1 outputs = net(inputs)

~\Anaconda\lib\site-packages\torch\nn\modules\module.py in __call__(self,
*input, **kwargs)
    487             result = self._slow_forward(*input, **kwargs)
    488         else:
--> 489             result = self.forward(*input, **kwargs)
    490         for hook in self._forward_hooks.values():
    491             hook_result = hook(self, input, result)

<ipython-input-282-a6eca2e3e9db> in forward(self, x)
    14         x = x.view(-1, 16 * 5 * 5)
    15         x = F.relu(self.fc1(x))
---> 16         x = F.relu(self.fc2(x))
    17         x = self.fc3(x)

Which closes out with:

RuntimeError: size mismatch, m1: [4 x 120], m2: [84 x 10] at
c:\a\w\1\s\tmp_conda_3.7_110206\conda\conda-
bld\pytorch_1550401474361\work\aten\src\th\generic/THTensorMath.cpp:940

I know this means my dimension values don't match, and I suspect it has to do with the line x = x.view(-1, 16 * 5 * 5) where I go from the convolutional to linear layer, but I have two confusions:

As far as I can tell my network matches exactly what's on the PyTorch page
My error occurs on the second linear layer, not the first, and columns of the preceeding layer match the rows of the current one, so I find it confusing why this error is happening.


title: SDKManager Install - No ARM files

body:
The l4t-base:r32.2.1 base image (jetson) provided by NVIDIA does not have cuda binaries
Pytorch is looking for. I am unable to download source files from Jetpack 4.2.2 (specifically cuda-l4t-repo-10.x.x-arm64.deb) to include within docker build.

I am building a docker container for jetsons. I used the l4t image from NVIDIA, but it is missing some binaries pytorch is looking for (I understand you're supposed to mount some files for CUDA, but I don't want to). I therefore was going to install CUDA 10 directly into the container.

I am trying to download the source files for Jetpack 4.2.2. From what I have read, there should be a file, "cuda-repo-l4t-10-0-local-_arm64.deb". However, I only see
cuda-repo-cross-aarch64-10-0-local-10.0.326_1.0-1_all.deb and cuda-repo-ubuntu1804-10-0-local-10.0.326-410.108_1.0-1_amd64.deb.

I created an Ubuntu 18 container to run the SDK in. I run the below command and get the following files...
2019_06_23_2352-26660552-NVIDIA_Nomad_2019.2.19174.2352_Release_External-L4T.linux.deb,
libopencv-dev_3.3.1-2-gb3f86dcd5_amd64.deb,
Jetson_Linux_R32.2.1_aarch64.tbz2,
libopencv-python_3.3.1-2-gb3f86dcd5_amd64.deb
NVIDIA_VisionWorks_References.zip,
libopencv-samples_3.3.1-2-gb3f86dcd5_amd64.deb
NsightSystems-linux-public-2019.4.1.10-a76094a.deb
libopencv_3.3.1-2-gb3f86dcd5_amd64.deb
Tegra_Linux_Sample-Root-Filesystem_R32.2.1_aarch64.tbz2
libvisionworks-repo_1.6.0.500n_amd64.deb
cuda-repo-cross-aarch64-10-0-local-10.0.326_1.0-1_all.deb
libvisionworks-sfm-repo_0.90.4_amd64.deb
cuda-repo-ubuntu1804-10-0-local-10.0.326-410.108_1.0-1_amd64.deb
libvisionworks-tracking-repo_0.88.2_amd64.deb
devtools_docs.zip
sdkml3_jetpack_l4t_422_rev1_b30.json

sdkmanager --cli downloadonly --user $NV_USER \
--view live --host --logintype devzone --product Jetson --version 4.2 \
--targetos Linux --target $DEVICE_ID --flash all --license accept \
--downloadfolder /tmp/cuda_pkg

Desired results:

- download jetson jetpack ARM specific cuda packages during docker build
- all cuda binaries are available in container

instead of:

root@24081bcd996b:/usr/local/cuda-10.0/lib64# ls
libcudadevrt.a  libcudart_static.a  stubs

should look more like:

libOpenCL.so            libcufft.so.10.0         libnppc.so             libnppif_static.a     libnpps.so.10.0.176
libOpenCL.so.1          libcufft.so.10.0.176     libnppc.so.10.0         libnppig.so           libnpps_static.a
libOpenCL.so.1.0        libcufft_static.a       libnppc.so.10.0.176     libnppig.so.10.0       libnvToolsExt.so
libOpenCL.so.1.0.0      libcufftw.so            libnppc_static.a       libnppig.so.10.0.176   libnvToolsExt.so.1
libaccinj64.so          libcufftw.so.10.0        libnppial.so           libnppig_static.a     libnvToolsExt.so.1.0.0
libaccinj64.so.10.0      libcufftw.so.10.0.176    libnppial.so.10.0       libnppim.so           libnvblas.so
libaccinj64.so.10.0.176  libcufftw_static.a      libnppial.so.10.0.176   libnppim.so.10.0       libnvblas.so.10.0
libcublas.so            libcuinj64.so           libnppial_static.a     libnppim.so.10.0.176   libnvblas.so.10.0.176
libcublas.so.10.0        libcuinj64.so.10.0       libnppicc.so           libnppim_static.a     libnvblas.so.10.0.480
libcublas.so.10.0.176    libcuinj64.so.10.0.176   libnppicc.so.10.0       libnppist.so          libnvgraph.so
libcublas.so.10.0.480    libculibos.a            libnppicc.so.10.0.176   libnppist.so.10.0      libnvgraph.so.10.0
libcublas_device.a      libcurand.so            libnppicc_static.a     libnppist.so.10.0.176  libnvgraph.so.10.0.176
libcublas_static.a      libcurand.so.10.0        libnppicom.so          libnppist_static.a    libnvgraph_static.a
libcudadevrt.a          libcurand.so.10.0.176    libnppicom.so.10.0      libnppisu.so          libnvrtc-builtins.so
libcudart.so            libcurand_static.a      libnppicom.so.10.0.176  libnppisu.so.10.0      libnvrtc-builtins.so.10.0
libcudart.so.10.0        libcusolver.so          libnppicom_static.a    libnppisu.so.10.0.176  libnvrtc-builtins.so.10.0.176
libcudart.so.10.0.176    libcusolver.so.10.0      libnppidei.so          libnppisu_static.a    libnvrtc.so
libcudart_static.a      libcusolver.so.10.0.176  libnppidei.so.10.0      libnppitc.so          libnvrtc.so.10.0
libcudnn.so             libcusolver_static.a    libnppidei.so.10.0.176  libnppitc.so.10.0      libnvrtc.so.10.0.176
libcudnn.so.7           libcusparse.so          libnppidei_static.a    libnppitc.so.10.0.176  stubs
libcudnn.so.7.6.3       libcusparse.so.10.0      libnppif.so            libnppitc_static.a
libcudnn_static.a       libcusparse.so.10.0.176  libnppif.so.10.0        libnpps.so
libcufft.so             libcusparse_static.a    libnppif.so.10.0.176    libnpps.so.10.0


title: pytorch Pretrained Resnet is not same accuracy Imagenet

body:
CPU : intel Zeon
GPU : NVIDIA RTX2080Ti
python : 3.6
torch version : 1.4
torchvision version : 0.5.0

I Download ILSVRC2012(Imagenet Dataset) torrent, and do evaluation validation set.

choosed model is torchvision.datasets.Resnet50(pretrained=True) and i expected to validation top 1 accuracy over 60%.

I find corrects label of validation set in the below link https://github.com/fh295/semanticCNN/blob/master/imagenet_labels/ILSVRC2012_validation_ground_truth.txt

And cross verified with the link below.

https://github.com/tensorflow/models/blob/master/research/inception/inception/data/imagenet_2012_validation_synset_labels.txt

i made custom dataset for validation set, and evaluation.

But contrary to my expectation, the evaluation accuracy came out very low as if it had not been learned.
(accuracy is 0.1% ~ 0.05%)

this is my testing code and uploaded github https://github.com/pdh930105/pytorch_imagenet_test

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.autograd import Variable
import PIL.Image as Image
import numpy as np
from torch.utils.data import Dataset, DataLoader
import os
from torchvision import transforms, datasets
import matplotlib.pyplot as plt
import torchvision
from tqdm import tqdm
import time

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

transform = transforms.Compose([transforms.Resize((256, 256)),
                                transforms.CenterCrop((224, 224)),
                              transforms.ToTensor(),
                            transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                 std=[0.229, 0.224, 0.225])])

class val_imagenet_dataset(Dataset):
    def __init__(self, root="../imagenet/", transform=None):
        self.transform = transform
        self.root = root
        self.data = []
        self.target = []
        f = open(self.root+"/ILSVRC2012_validation_ground_truth.txt")
        label_data = f.readlines()
        file_list = os.listdir(self.root)
        image_list = [image for image in file_list if image.endswith(".JPEG")]
        image_list.sort()
        for idx, label in enumerate(tqdm(label_data)):
            self.data.append(image_list[idx])
            self.target.append(int(label)-1)

    def __len__(self):
        return len(self.data)

    def __getitem__(self, idx):
        path, target = self.data[idx], self.target[idx]

        with open(self.root+path, 'rb') as f:
            img = Image.open(f)
            sample = img.convert('RGB')

        if self.transform is not None:
            sample = self.transform(sample)

        return sample, target

def val_model(model, dataloader, device, criterion =None):
    since = time.time()
    acc = 0
    loss = 0
    data_size = dataloader.dataset.__len__()
    for inputs, labels in tqdm(dataloader):
        model.eval()
        model.to(device)
        inputs = inputs.to(device)
        labels = labels.to(device)
        outputs = model(inputs)
        _, preds = torch.max(outputs, 1)
        acc += torch.sum(preds==labels.data)

        if criterion is not None:
            _loss = criterion(outputs, labels)
            loss +=_loss.item() * inputs.size(0)
            print("local loss : {:4f}".format(loss))

    acc = acc.double()/data_size
    loss = loss / data_size

    print("Acc : {:4f}".format(acc))
    if criterion is not None:
        print("Loss : {:4f}".format(loss))

val_imagenet = val_imagenet_dataset(transform=transform)
val_dataloader = DataLoader(val_imagenet, batch_size=1, shuffle=False, num_workers=2)

resnet_50_origin = torchvision.models.resnet50(pretrained=True)

val_model(resnet_50_origin, val_dataloader, device=device)

It is my code but run accracy is 0.01%

Please let me know the problem of my code.


title: Pytorch dynamic amount of Layers?

body:
I am trying to specify a dynamic amount of layers, which I seem to be doing wrong.
My issue is that when I define the 100 layers here, I will get an error in the forward step.
But when I define the layer properly it works?
Below simplified example
class PredictFromEmbeddParaSmall(LightningModule):
def __init__(self, hyperparams={'lr': 0.0001}):
    super(PredictFromEmbeddParaSmall, self).__init__()
    #Input is something like tensor.size=[768*100]
    self.TO_ILLUSTRATE = nn.Linear(768, 5)
    self.enc_ref=[]
    for i in range(100):
        self.enc_red.append(nn.Linear(768, 5))
    # gather the layers output sth
    self.dense_simple1 = nn.Linear(5*100, 2)
    self.output = nn.Sigmoid()
def forward(self, x):
    # first input to enc_red
    x_vecs = []
    for i in range(self.para_count):
        layer = self.enc_red[i]
        # The first dim is the batch size here, output is correct
        processed_slice = x[:, i * 768:(i + 1) * 768]
        # This works and give the out of size 5
        rand = self.TO_ILLUSTRATE(processed_slice)
        #This will fail? Error below
        ret = layer(processed_slice)
        #more things happening we can ignore right now since we fail earlier

I get this error when executing "ret = layer.forward(processed_slice)"

RuntimeError: Expected object of device type cuda but got device type
cpu for argument #1 'self' in call to _th_addmm

Is there a smarter way to program this? OR solve the error?