title: Use skimage feature extractors in PyTorch Transforms

body:
I want to apply skimage’s Local Binary Pattern feature extraction on my data, and was wondering if there was any possibility of doing this inside my torch’s Transforms, which right now is the following:

data_transforms = {
    'train': transforms.Compose([
        transforms.CenterCrop(178),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize([0.5, 0.5, 0.5], [0.5, 0.5, 0.5])
    ]),
    'val': transforms.Compose([
        transforms.CenterCrop(178),
        transforms.ToTensor(),
        transforms.Normalize([0.5, 0.5, 0.5], [0.5, 0.5, 0.5])
    ]),
}

If not, how would I implement it? Would I have to do it when importing the data?


title: Pytorch does not backpropagate through a iterative tensor construction

body:
I am currently trying to build a tensor iteratively in Pytorch.
Sadly the backprop does not work with the inplace operation in the loop. I already tried equivalent programs with stack for example. Does somebody know how I could build the tensor with a working backprop?

This is a minimal example which produces the error:

import torch

k=2
a =torch.Tensor([10,20])
a.requires_grad_(True)
b = torch.Tensor([10,20])
b.requires_grad_(True)

batch_size = a.size()[0]
uniform_samples = Uniform(torch.tensor([0.0]), torch.tensor([1.0])).rsample(torch.tensor([batch_size,k])).view(-1,k)
exp_a = 1/a
exp_b = 1/b
km = (1- uniform_samples.pow(exp_b)).pow(exp_a)

sticks = torch.zeros(batch_size,k)
remaining_sticks = torch.ones_like(km[:,0])
for i in range(0,k-1):
    sticks[:,i] = remaining_sticks * km[:,i]
    remaining_sticks *= (1-km[:,i])
sticks[:,k-1] = remaining_sticks
latent_variables = sticks

latent_variables.sum().backward()

The stack trace:

/opt/conda/conda-bld/pytorch_1570910687230/work/torch/csrc/autograd/python_anomaly_mode.cpp:57: UserWarning: Traceback of forward call that caused the error:
  File "/opt/conda/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/opt/conda/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/opt/conda/lib/python3.6/site-packages/ipykernel_launcher.py", line 16, in <module>
    app.launch_new_instance()
  File "/opt/conda/lib/python3.6/site-packages/traitlets/config/application.py", line 664, in launch_instance
    app.start()
  File "/opt/conda/lib/python3.6/site-packages/ipykernel/kernelapp.py", line 563, in start
    self.io_loop.start()
  File "/opt/conda/lib/python3.6/site-packages/tornado/platform/asyncio.py", line 148, in start
    self.asyncio_loop.run_forever()
  File "/opt/conda/lib/python3.6/asyncio/base_events.py", line 438, in run_forever
    self._run_once()
  File "/opt/conda/lib/python3.6/asyncio/base_events.py", line 1451, in _run_once
    handle._run()
  File "/opt/conda/lib/python3.6/asyncio/events.py", line 145, in _run
    self._callback(*self._args)
  File "/opt/conda/lib/python3.6/site-packages/tornado/ioloop.py", line 690, in <lambda>
    lambda f: self._run_callback(functools.partial(callback, future))
  File "/opt/conda/lib/python3.6/site-packages/tornado/ioloop.py", line 743, in _run_callback
    ret = callback()
  File "/opt/conda/lib/python3.6/site-packages/tornado/gen.py", line 787, in inner
    self.run()
  File "/opt/conda/lib/python3.6/site-packages/tornado/gen.py", line 748, in run
    yielded = self.gen.send(value)
  File "/opt/conda/lib/python3.6/site-packages/ipykernel/kernelbase.py", line 361, in process_one
    yield gen.maybe_future(dispatch(*args))
  File "/opt/conda/lib/python3.6/site-packages/tornado/gen.py", line 209, in wrapper
    yielded = next(result)
  File "/opt/conda/lib/python3.6/site-packages/ipykernel/kernelbase.py", line 268, in dispatch_shell
    yield gen.maybe_future(handler(stream, idents, msg))
  File "/opt/conda/lib/python3.6/site-packages/tornado/gen.py", line 209, in wrapper
    yielded = next(result)
  File "/opt/conda/lib/python3.6/site-packages/ipykernel/kernelbase.py", line 541, in execute_request
    user_expressions, allow_stdin,
  File "/opt/conda/lib/python3.6/site-packages/tornado/gen.py", line 209, in wrapper
    yielded = next(result)
  File "/opt/conda/lib/python3.6/site-packages/ipykernel/ipkernel.py", line 300, in do_execute
    res = shell.run_cell(code, store_history=store_history, silent=silent)
  File "/opt/conda/lib/python3.6/site-packages/ipykernel/zmqshell.py", line 536, in run_cell
    return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2855, in run_cell
    raw_cell, store_history, silent, shell_futures)
  File "/opt/conda/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2881, in _run_cell
    return runner(coro)
  File "/opt/conda/lib/python3.6/site-packages/IPython/core/async_helpers.py", line 68, in _pseudo_sync_runner
    coro.send(None)
  File "/opt/conda/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3058, in run_cell_async
    interactivity=interactivity, compiler=compiler, result=result)
  File "/opt/conda/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3249, in run_ast_nodes
    if (await self.run_code(code, result,  async_=asy)):
  File "/opt/conda/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3326, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-124-2bbdbc3af797>", line 16, in <module>
    sticks[:,i] = remaining_sticks * km[:,i]

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-124-2bbdbc3af797> in <module>
     19 latent_variables = sticks
     20
---> 21 latent_variables.sum().backward()

/opt/conda/lib/python3.6/site-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph)
    148                 products. Defaults to ``False``.
    149         """
--> 150         torch.autograd.backward(self, gradient, retain_graph, create_graph)
    151
    152     def register_hook(self, hook):

/opt/conda/lib/python3.6/site-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
     97     Variable._execution_engine.run_backward(
     98         tensors, grad_tensors, retain_graph, create_graph,
---> 99         allow_unreachable=True)  # allow_unreachable flag
    100
    101

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [2]] is at version 1; expected version 0 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!


title: neighbours of a cell in matrix pytorch

body:
I am trying to get neighbours of a cell of matrix in pytorch using below part of code.
it works correctly but it is very time consumming.
Have you any suggestion to to get it faster

def neighbour(x):
    result=F.pad(input=x, pad=(1, 1, 1, 1), mode='constant', value=0)
    for m in range(1,x.size(0)+1):
        for n in range(1,x.size(1)+1):
                y=torch.Tensor([result[m][n],result[m-1][n-1],result[m-1][n],result[m-1]
           [n+1],result[m][n-1],result[m][n+1],result[m+1][n-1],result[m+1][n],result[m+1][n+1]])
                x[m-1][n-1]=y.mean()

    return x


title: Is there a way to convert a pytorch_model.bin to pytorch_model.pt and vice versa ---- BERT vs SBERT

body:
I have been working on BERT for a while. I used Nvidia BERT which uses a model.pt checkpoint whereas while using SBERT/sentence_BERT, it uses model.bin with a bunch of files (i.e. model.bin, vocab.txt, config.json). What is the difference between these two. And, is there a script that can be used to convert one into another (and vice versa), so both can be used in place of each other. i.e. Re-using pretrained model from SBERT into Nvidia_BERT scripts and the reverse.

Thank You!


title: LayerNorm inside nn.Sequential in torch

body:
I am trying to use LayerNorm inside nn.Sequential in torch. This is what I am looking for-
import torch.nn as nn

class LayerNormCnn(nn.Module):
    def __init__(self):
        super(LayerNormCnn, self).__init__()
        self.net = nn.Sequential(
                nn.Conv2d(3, 32, kernel_size=3, stride=2, padding=1),
                nn.LayerNorm(),
                nn.ReLU(),
                nn.Conv2d(32, 64, kernel_size=3, stride=2, padding=1),
                nn.LayerNorm(),
                nn.ReLU(),
            )

    def forward(self, x):
        x = self.net(x)
        return x

Unfortunately, it doesn't work because LayerNorm requires normalized_shape as input. The code above throws following exception-
    nn.LayerNorm(),
TypeError: __init__() missing 1 required positional argument: 'normalized_shape'

Right now, this is how I have implemented it-
import torch.nn as nn
import torch.nn.functional as F

class LayerNormCnn(nn.Module):
    def __init__(self, state_shape):
        super(LayerNormCnn, self).__init__()
        self.conv1 = nn.Conv2d(state_shape[0], 32, kernel_size=3, stride=2, padding=1)
        self.conv2 = nn.Conv2d(32, 64, kernel_size=3, stride=2, padding=1)

        # compute shape by doing a forward pass
        with torch.no_grad():
            fake_input = torch.randn(1, *state_shape)
            out        = self.conv1(fake_input)
            bn1_size   = out.size()[1:]
            out        = self.conv2(out)
            bn2_size   = out.size()[1:]

        self.bn1 = nn.LayerNorm(bn1_size)
        self.bn2 = nn.LayerNorm(bn2_size)

    def forward(self, x):
        x = F.relu(self.bn1(self.conv1(x)))
        x = F.relu(self.bn2(self.conv2(x)))
        return x

if __name__ == '__main__':
    in_shape   = (3, 128, 128)
    batch_size = 32

    model = LayerNormCnn(in_shape)
    x = torch.randn((batch_size,) + in_shape)
    out = model(x)
    print(out.shape)

Is it possible to use LayerNorm inside nn.Sequential?


title: Use *.pth model in C++

body:
I want to run inference in C++ using a yolo3 model I trained with pytorch. I am unable to make the conversions using tracing and scripting provided by pytorch. I have this error during conversion
First diverging operator:
    Node diff:
        - %2 : __torch__.torch.nn.modules.container.ModuleList = prim::GetAttr[name="module_list"](%self.1)
        + %2 : __torch__.torch.nn.modules.container.___torch_mangle_139.ModuleList = prim::GetAttr[name="module_list"](%self.1)
        ?                                           ++++++++++++++++++++
ERROR: Tensor-valued Constant nodes differed in value across invocations. This often indicates that the tracer has encountered untraceable code.
    Node:
        %358 : Tensor = prim::Constant[value=<Tensor>](), scope: __module.module_list.16.yolo_16


title: How do deep learning frameworks such as PyTorch handle memory when using multiple GPUs?

body: I have recently run into a situation where I am running out of memory on a single Nvidia V100. I have limited experience using multiple GPUs to train networks so I'm a little unsure on how the data parallelization process works. Lets say I'm using a model and batch size that requires something like 20-25GB of memory. Is there any way to take advantage of the full 32GB of memory I have between two 16GB V100s? Would PyTorch's DataParallel functionality achieve this? I suppose there is also the possibility of breaking the model up and using model parallelism as well. Please excuse my lack of knowledge on this subject. Thanks in advance for any help or clarification!


title: Rotation matrix for 4D Tensor

body:
I have a tensor with shape 8,24,24,24 which represents a 3D-voxelized picture with 8 channels.
Now I have to rotate this, to get all 24 permutations. Because the data are very big, I want to rotate on the fly. For this, I think the most efficient way is the multiplication with a rotation matrix.

I have to say, that i absolutely have no idea, who to calculate a tensor, which I can use to rotate the 4D-tensor or better the picture of every channel without a for-loop.

I only found rotation of a vector in 3D-space or using a predefined command. But who I rotate a whole tensor?

x = np.arange(27).reshape(3,3,3)
np.rot90(x)

array([[[ 6,  7,  8],
       [15, 16, 17],
       [24, 25, 26]],

      [[ 3,  4,  5],
       [12, 13, 14],
       [21, 22, 23]],

      [[ 0,  1,  2],
       [ 9, 10, 11],
       [18, 19, 20]]])

I found, that a rotation in 3D is nothing else than a rotation in 2D with specified axis. For this one can use for example this:

r = [[1,0,0],[0,0,-1],[0,1,0]]
np.matmul(x,r)

array([[[  0,   2,  -1],
       [  3,   5,  -4],
       [  6,   8,  -7]],

      [[  9,  11, -10],
       [ 12,  14, -13],
       [ 15,  17, -16]],

      [[ 18,  20, -19],
       [ 21,  23, -22],
       [ 24,  26, -25]]])

But the result is completely different. Did I misunderstood something?

In addition: actually I use the following:

def calc_rot(data):
        t1 = []
        n1 = data
        for i in range(4):
            n1 = np.rot90(n1, axes=(2, 1))
            t1.append(n1)
            n2 = np.copy(n1)
            for j in range(3):
                n2 = np.rot90(n2, axes=(3, 2))
                t1.append(n2)
        for k in [0, 2, 4, 6, 8, 10, 12, 14]:
            t1.append(np.rot90(t1[k], axes=(3, 1)))
        return t1

def calc_rot2(data):
        t1 = []
        n1 = data
        for i in range(4):
            n1 = n1.swapaxes(2,1)[:,:,::-1,:]
            t1.append(n1)
            n2 = np.copy(n1)
            for j in range(3):
                n2 = n2.swapaxes(3,2)[:,:,::-1,:]
                t1.append(n2)
        for k in [0, 2, 4, 6, 8, 10, 12, 14]:
            t1.append(t1[k].swapaxes(3,1)[:,:,::-1,:])
        return t1

Both do the same. Swapaxes seems a little bit faster. But I think in general a tensor/matrix-multiplication should be faster. Right?


title: DataLoader messing up transformed data

body:
I am testing the MNIST dataset in Pytorch, and after I apply a transformation to the X data, it seems the DataLoader puts all values out of the original order, potentially messing up the training step.

My transformation is to divide all values by 255. One should notice that the transformation itself does not change positions, as shown by the first scatterplots. But after the data is passed to the DataLoader and I retrieve it back, they are out of order. If I make no transformation, everything is fine (not shown). The distribution of the values is the same among before, after1 (divided by 255/before DataLoader) and after2 (divided by 255/after DataLoader) (also not shown), only the order seems to be affected.

import torch
from torchvision import datasets
import torchvision.transforms as transforms
import matplotlib.pyplot as plt

transform = transforms.ToTensor()

train = datasets.MNIST(root = '.', train = True, download = True, transform = transform)
test = datasets.MNIST(root = '.', train = False, download = True, transform = transform)

before = train.data[0]

train.data = train.data.float()/255
after1 = train.data[0]

train_loader = torch.utils.data.DataLoader(train, batch_size = 128)
test_loader = torch.utils.data.DataLoader(test, batch_size = 128)

fig, ax = plt.subplots(1, 2)
ax[0].scatter(range(len(before.view(-1))), before.view(-1))
ax[0].set_title('Before')
ax[1].scatter(range(len(after1.view(-1))), after1.view(-1))
ax[1].set_title('After1')

after2 = next(iter(train_loader))[0][0]

fig, ax = plt.subplots(1, 2)
ax[0].scatter(range(len(before.view(-1))), before.view(-1))
ax[0].set_title('Before')
ax[1].scatter(range(len(after2.view(-1))), after2.view(-1))
ax[1].set_title('After2')

fig, ax = plt.subplots(1, 3)
ax[0].imshow(before, cmap = 'gray')
ax[1].imshow(after1, cmap = 'gray')
ax[2].imshow(after2.view(28, 28), cmap = 'gray')

I know that this might not be the best way to deal with this data (transforms.Normalize should solve it), but I would really like to understand what is happening.

Thank you!


title: Cross-entropy rises while AUC rises (pytorch)

body:
I have a feed-forward neural network and a binary classification problem.

Defining the loss function as

def cross_entropy(ys,ts):
    cross_entropy = -torch.sum(ts * torch.log(ys+0.00001) + (1-ts)*torch.log(1-ys+0.00001))
    return cross_entropy

and the AUC as

def auc(ys, ts):
    ts = ts.detach().numpy()
    ys = ys.detach().numpy()
    return roc_auc_score(ts,ys)

where ts and ys is target/net-output (for class 1) respectively.
For some reason, when I train, the cross-entropy rises and the AUC rises. I would think either should fall when the other one grows.


title: why my dataset.data's shape is different with each image?

body:
I download the cifar10 dataset with torchvision

for idx,(img,target) in enumerate(trainloader):
    print(trainloader.dataset.data.shape)
    print(img.shape)
    img = img.squeeze()
    img = torch.tensor(img)
    img=img.squeeze()
    print(img.shape)
    img = (img).permute(1, 2, 0)
    plt.imshow(img)
    plt.show()
    if idx==0:break

But the print result is strange

  (50000, 32, 32, 3)
  torch.Size([1, 3, 224, 224]) torch.Size([3, 224,
  224])

each image is size with (3,224,224) but the dataset.data's shape is (32,32,3)<br/>
I want to make the dataset's shape with (500000,224,224,3)

def load_data_cifar10(batch_size=128,test=False):
    if not test:
        train_dset = torchvision.datasets.CIFAR10(root='/mnt/3CE35B99003D727B/input/pytorch/data', train=True,
                                                download=True, transform=transform)
    else:
        train_dset = torchvision.datasets.CIFAR10(root='/mnt/3CE35B99003D727B/input/pytorch/data', train=False,
                                               download=True, transform=transform)
    train_loader = torch.utils.data.DataLoader(train_dset, batch_size=batch_size, shuffle=False)

transform = transforms.Compose([
    transforms.Resize(224),
    transforms.ToTensor(),
])


title: How can I trim / remove part of a Tensor to match the shape of another Tensor with PyTorch?

body:
I have 2 tensors:

outputs: torch.Size([4, 27, 161])       pred: torch.Size([4, 30, 161])

I want to cut pred (from the end) so that it'll have the same dimensions as outputs.

What's the best way to do it with PyTorch?


title: Passing embedded sequence to LSTM and getting TypeError: 'int' object is not subscriptable

body:
I have some really basic pytorch code here where I'm trying to test running an input tensor through what will eventually become my forward function.

Goal: Treat the a sentence as a single input sequence after embedding each word number.

Embed a tensor
Convert that embedding back to a float32 tensor
Reshape embedding to shape (batch_size, seq_len, input_size)
Pass through lstm.

I've converted back to a float32 tensor after embedding so idk why I'm getting this error.

hidden_size=10
embedding = nn.Embedding(VOC.n_words, hidden_size)
lstm = nn.LSTM(hidden_size, hidden_size, # Will output 2x hidden size
               num_layers=1, dropout=0.5,
               bidirectional=True, batch_first=True)

print("Input tensor",idx_sentence)
# Forward test
embedded = embedding(idx_sentence.long())
embedded = torch.tensor(embedded, dtype=torch.float32)
print(f"embedding: {embedded.size()}")

# reshape to (batch_size, seq_len, input_size)
sequence = embedded.view(1,-1,hidden_size)
print(f"sequence shape: {sequence.size()}")

output, hidden = lstm(sequence, hidden_size)
print(f"output shape: {output.size()}")

Input tensor tensor([ 3., 20., 21., 90.,  9.])
embedding: torch.Size([5, 10])
sequence shape: torch.Size([1, 5, 10])
/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:10: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  # Remove the CWD from sys.path while we load stuff.
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-116-ab3d6ed0e51c> in <module>()
     16
     17 # Input have shape (seq_len, batch, input_size)
---> 18 output, hidden = lstm(sequence, hidden_size)
     19 print(f"output shape: {output.size()}")

2 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/rnn.py in check_forward_args(self, input, hidden, batch_sizes)
    520         expected_hidden_size = self.get_expected_hidden_size(input, batch_sizes)
    521
--> 522         self.check_hidden_size(hidden[0], expected_hidden_size,
    523                                'Expected hidden[0] size {}, got {}')
    524         self.check_hidden_size(hidden[1], expected_hidden_size,

TypeError: 'int' object is not subscriptable


title: Transformer Image captioning model produces just padding rather than a caption

body:
I am trying to produce a model that will produce a caption for an image using resnet as the encoder, transformer as the decoder and COCO as the database.
After training my model for 10 epochs, my model failed to produce anything other than the word <pad> which implies that the only result after going through the model only produced tokens of 0 which corresponds to <pad>.
After using the debugger it seems that the error occurs at the argmax, where the output just becomes zero rather than anything else, but I don't know how to fix it, is it an issue with my model, or the way it is trained?
I based my model off of this github if it helps.
The script to download the COCO model is here:
Download.sh
mkdir data
wget http://msvocds.blob.core.windows.net/annotations-1-0-3/captions_train-val2014.zip -P ./data/
wget http://images.cocodataset.org/zips/train2014.zip -P ./data/
wget http://images.cocodataset.org/zips/val2014.zip -P ./data/

unzip ./data/captions_train-val2014.zip -d ./data/
rm ./data/captions_train-val2014.zip
unzip ./data/train2014.zip -d ./data/
rm ./data/train2014.zip
unzip ./data/val2014.zip -d ./data/
rm ./data/val2014.zip

Any help is much appreciated.
Here is my code:
model.py
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
import math, copy, time
import torchvision.models as models
from torch.nn import TransformerDecoderLayer, TransformerDecoder
from torch.nn.utils.rnn import pack_padded_sequence
from torch.autograd import Variable

class EncoderCNN(nn.Module):
    def __init__(self, embed_size):
        super(EncoderCNN, self).__init__()
        resnet = models.resnet152(pretrained=True)
        self.resnet = nn.Sequential(*list(resnet.children())[:-2])
        self.conv1 = nn.Conv2d(2048, embed_size, 1)
        self.embed_size = embed_size

        self.fine_tune()

    def forward(self, images):
        features = self.resnet(images)
        batch_size, _,_,_ = features.shape
        features = self.conv1(features)
        features = features.view(batch_size, self.embed_size, -1)
        features = features.permute(2, 0, 1)

        return features

    def fine_tune(self, fine_tune=True):
        for p in self.resnet.parameters():
            p.requires_grad = False
        # If fine-tuning, only fine-tune convolutional blocks 2 through 4
        for c in list(self.resnet.children())[5:]:
            for p in c.parameters():
                p.requires_grad = fine_tune

class PositionEncoder(nn.Module):
    def __init__(self, d_model, dropout, max_len=5000):
        super(PositionEncoder, self).__init__()
        self.dropout = nn.Dropout(p=dropout)

        pe = torch.zeros(max_len, d_model)
        position = torch.arange(0, max_len, dtype=torch.float).unsqueeze(1)
        div_term = torch.exp(torch.arange(0, d_model, 2).float() * (-math.log(10000.0) / d_model))
        pe[:, 0::2] = torch.sin(position * div_term)
        pe[:, 1::2] = torch.cos(position * div_term)
        pe = pe.unsqueeze(0).transpose(0, 1)
        self.register_buffer('pe', pe)

    def forward(self, x):
        x = x + self.pe[:x.size(0), :]
        return self.dropout(x)

class Embedder(nn.Module):
    def __init__(self, vocab_size, d_model):
        super().__init__()
        self.embed = nn.Embedding(vocab_size, d_model)
    def forward(self, x):
        return self.embed(x)

class Transformer(nn.Module):
    def __init__(self, vocab_size, d_model, h, num_hidden, N, device, dropout_dec=0.1, dropout_pos=0.1):
        super(Transformer, self).__init__()
        decoder_layers = TransformerDecoderLayer(d_model, h, num_hidden, dropout_dec)
        self.source_mask = None
        self.device = device
        self.d_model = d_model
        self.pos_decoder = PositionalEncoder(d_model, dropout_pos)
        self.decoder = TransformerDecoder(decoder_layers, N)
        self.embed = Embedder(vocab_size, d_model)
        self.linear = nn.Linear(d_model, vocab_size)

        self.init_weights()

    def forward(self, source, mem):
        source = source.permute(1,0)
        if self.source_mask is None or self.source_mask.size(0) != len(source):
            self.source_mask = nn.Transformer.generate_square_subsequent_mask(self=self, sz=len(source)).to(self.device)

        source = self.embed(source)
        source = source*math.sqrt(self.d_model)
        source = self.pos_decoder(source)
        output = self.decoder(source, mem, self.source_mask)
        output = self.linear(output)
        return output

    def init_weights(self):
        initrange = 0.1
        self.linear.bias.data.zero_()
        self.linear.weight.data.uniform_(-initrange, initrange)

    def pred(self, memory, pred_len):
        batch_size = memory.size(1)
        src = torch.ones((pred_len, batch_size), dtype=int) * 2
        if self.source_mask is None or self.source_mask.size(0) != len(src):
            self.source_mask = nn.Transformer.generate_square_subsequent_mask(self=self, sz=len(src)).to(self.device)
        output = torch.ones((pred_len, batch_size), dtype=int)
        src, output = src.cuda(), output.cuda()
        for i in range(pred_len):
            src_emb = self.embed(src) # src_len * batch size * embed size
            src_emb = src_emb*math.sqrt(self.d_model)
            src_emb = self.pos_decoder(src_emb)
            out = self.decoder(src_emb, memory, self.source_mask)
            out = out[i]
            out = self.linear(out) # batch_size * vocab_size
            out = out.argmax(dim=1)
            if i < pred_len-1:
                src[i+1] = out
            output[i] = out
        return output

Data_Loader.py
import torch
import torchvision.transforms as transforms
import torch.utils.data as data
import os
import pickle
import numpy as np
import nltk
from PIL import Image
from build_vocab import Vocabulary
from pycocotools.coco import COCO

class CocoDataset(data.Dataset):
    """COCO Custom Dataset compatible with torch.utils.data.DataLoader."""
    def __init__(self, root, json, vocab, transform=None):
        """Set the path for images, captions and vocabulary wrapper.

        Args:
            root: image directory.
            json: coco annotation file path.
            vocab: vocabulary wrapper.
            transform: image transformer.
        """
        self.root = root
        self.coco = COCO(json)
        self.ids = list(self.coco.anns.keys())
        self.vocab = vocab
        self.transform = transform

    def __getitem__(self, index):
        """Returns one data pair (image and caption)."""
        coco = self.coco
        vocab = self.vocab
        ann_id = self.ids[index]
        caption = coco.anns[ann_id]['caption']
        img_id = coco.anns[ann_id]['image_id']
        path = coco.loadImgs(img_id)[0]['file_name']

        image = Image.open(os.path.join(self.root, path)).convert('RGB')
        if self.transform is not None:
            image = self.transform(image)

        # Convert caption (string) to word ids.
        tokens = nltk.tokenize.word_tokenize(str(caption).lower())
        caption = []
        caption.append(vocab('<start>'))
        caption.extend([vocab(token) for token in tokens])
        caption.append(vocab('<end>'))
        target = torch.Tensor(caption)
        return image, target

    def __len__(self):
        return len(self.ids)

def collate_fn(data):
    """Creates mini-batch tensors from the list of tuples (image, caption).

    We should build custom collate_fn rather than using default collate_fn,
    because merging caption (including padding) is not supported in default.
    Args:
        data: list of tuple (image, caption).
            - image: torch tensor of shape (3, 256, 256).
            - caption: torch tensor of shape (?); variable length.
    Returns:
        images: torch tensor of shape (batch_size, 3, 256, 256).
        targets: torch tensor of shape (batch_size, padded_length).
        lengths: list; valid length for each padded caption.
    """
    # Sort a data list by caption length (descending order).
    data.sort(key=lambda x: len(x[1]), reverse=True)
    images, captions = zip(*data)

    # Merge images (from tuple of 3D tensor to 4D tensor).
    images = torch.stack(images, 0)

    # Merge captions (from tuple of 1D tensor to 2D tensor).
    lengths = [len(cap) for cap in captions]
    targets = torch.zeros(len(captions), max(lengths)).long()
    for i, cap in enumerate(captions):
        end = lengths[i]
        targets[i, :end] = cap[:end]
    return images, targets, lengths

def get_loader(root, json, vocab, transform, batch_size, shuffle, num_workers):
    """Returns torch.utils.data.DataLoader for custom coco dataset."""
    # COCO caption dataset
    coco = CocoDataset(root=root,
                       json=json,
                       vocab=vocab,
                       transform=transform)

    # Data loader for COCO dataset
    # This will return (images, captions, lengths) for each iteration.
    # images: a tensor of shape (batch_size, 3, 224, 224).
    # captions: a tensor of shape (batch_size, padded_length).
    # lengths: a list indicating valid length for each caption. length is (batch_size).
    data_loader = torch.utils.data.DataLoader(dataset=coco,
                                              batch_size=batch_size,
                                              shuffle=shuffle,
                                              num_workers=num_workers,
                                              collate_fn=collate_fn)
    return data_loader

Build_vocab.py
import nltk
import pickle
import argparse
from collections import Counter
from pycocotools.coco import COCO

class Vocabulary(object):
    def __init__(self):
        self.word2idx = {}
        self.idx2word = {}
        self.idx = 0

    def add_word(self, word):
        if not word in self.word2idx:
            self.word2idx[word] = self.idx
            self.idx2word[self.idx] = word
            self.idx += 1

    def __call__(self, word):
        if not word in self.word2idx:
            return self.word2idx['<unk>']
        return self.word2idx[word]

    def __len__(self):
        return len(self.word2idx)

def build_vocab(json, threshold):
    coco = COCO(json)
    counter = Counter()
    ids = coco.anns.keys()
    for i, id in enumerate(ids):
        caption = str(coco.anns[id]['caption'])
        tokens = nltk.tokenize.word_tokenize(caption.lower())
        counter.update(tokens)

        if (i+1) % 1000 == 0:
            print("[{}/{}] Tokenized the captions.".format(i+1, len(ids)))

    # If the word frequency is less than 'threshold', then the word is discarded.
    words = [word for word, cnt in counter.items() if cnt >= threshold]

    # Create a vocab wrapper and add some special tokens.
    vocab = Vocabulary()
    vocab.add_word('<pad>')
    vocab.add_word('<start>')
    vocab.add_word('<end>')
    vocab.add_word('<unk>')

    # Add the words to the vocabulary.
    for i, word in enumerate(words):
        vocab.add_word(word)
    return vocab

def main(args):
    vocab = build_vocab(json=args.caption_path, threshold=args.threshold)
    vocab_path = args.vocab_path
    with open(vocab_path, 'wb') as f:
        pickle.dump(vocab, f)
    print("Total vocabulary size: {}".format(len(vocab)))
    print("Saved the vocabulary wrapper to '{}'".format(vocab_path))

if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('--caption_path', type=str,
                        default='./data/annotations/captions_train2014.json',
                        help='path for train annotation file')
    parser.add_argument('--vocab_path', type=str, default='./data/vocab.pkl',
                        help='path for saving vocabulary wrapper')
    parser.add_argument('--threshold', type=int, default=4,
                        help='minimum word count threshold')
    args = parser.parse_args()
    main(args)

train.py
import argparse
import torch
import torch.nn as nn
import numpy as np
import os
import pickle
import math
from tqdm import tqdm
from data_loader import get_loader
from build_vocab import Vocabulary
from model import EncoderCNN, Decoder
from torch.nn.utils.rnn import pack_padded_sequence
from torchvision import transforms

# Device configuration
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

def main(args):
    batch_size = 64
    embed_size = 512
    num_heads = 8
    num_layers = 6
    num_workers = 2
    num_epoch = 5
    lr = 1e-3
    load = False
    # Create model directory
    if not os.path.exists('models/'):
        os.makedirs('models/')

    # Image preprocessing, normalization for the pretrained resnet
    transform = transforms.Compose([
        transforms.RandomCrop(224),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize((0.485, 0.456, 0.406),
                             (0.229, 0.224, 0.225))])

    # Load vocabulary wrapper
    with open('data/vocab.pkl', 'rb') as f:
        vocab = pickle.load(f)

    # Build data loader
    data_loader = get_loader('data/resized2014', 'data/annotations/captions_train2014.json', vocab,
                             transform, batch_size,
                             shuffle=True, num_workers=num_workers)

    encoder = EncoderCNN(embed_size).to(device)
    encoder.fine_tune(False)
    decoder = Decoder(len(vocab), embed_size, num_heads, embed_size, num_layers).to(device)

    if(load):
        encoder.load_state_dict(torch.load(os.path.join('models/', 'encoder-{}-{}.ckpt'.format(5, 5000))))
        decoder.load_state_dict(torch.load(os.path.join('models/', 'decoder-{}-{}.ckpt'.format(5, 5000))))
        print("Load Successful")

    # Loss and optimizer
    criterion = nn.CrossEntropyLoss()
    encoder_optim = torch.optim.Adam(encoder.parameters(), lr=lr)
    decoder_optim = torch.optim.Adam(decoder.parameters(), lr=lr)

    # Train the models
    for epoch in range(num_epoch):
        encoder.train()
        decoder.train()
        for i, (images, captions, lengths) in tqdm(enumerate(data_loader), total=len(data_loader), leave=False):

            # Set mini-batch dataset
            images = images.to(device)
            captions = captions.to(device)

            # Forward, backward and optimize
            features = encoder(images)
            cap_input = captions[:, :-1]
            cap_target = captions[:, 1:]
            outputs = decoder(cap_input, features)
            outputs = outputs.permute(1,0,2)
            outputs_shape = outputs.reshape(-1, len(vocab))
            loss = criterion(outputs_shape, cap_target.reshape(-1))
            decoder.zero_grad()
            encoder.zero_grad()
            loss.backward()
            encoder_optim.step()
            decoder_optim.step()

            # Save the model checkpoints
            if (i+1) % args.save_step == 0:
                torch.save(decoder.state_dict(), os.path.join(
                    'models/', 'decoder-{}-{}.ckpt'.format(epoch+1, i+1)))
                torch.save(encoder.state_dict(), os.path.join(
                    'models/', 'encoder-{}-{}.ckpt'.format(epoch+1, i+1)))

if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('--log_step', type=int , default=10, help='step size for prining log info')
    parser.add_argument('--save_step', type=int , default=1000, help='step size for saving trained models')

    args = parser.parse_args()
    print(args)
    main(args)

sample.py

import torch
import matplotlib.pyplot as plt
import numpy as np
import argparse
import pickle
import os
from torchvision import transforms
from build_vocab import Vocabulary
from data_loader import get_loader
from model import EncoderCNN, Decoder
from PIL import Image

# Device configuration
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
#

def token_sentence(decoder_out, itos):
    tokens = decoder_out
    tokens = tokens.transpose(1, 0)
    tokens = tokens.cpu().numpy()
    results = []
    for instance in tokens:
        result = ' '.join([itos[x] for x in instance])
        results.append(''.join(result.partition('<eos>')[0])) # Cut before '<eos>'
    return results

def load_image(image_path, transform=None):
    image = Image.open(image_path).convert('RGB')
    image = image.resize([224, 224], Image.LANCZOS)

    if transform is not None:
        image = transform(image).unsqueeze(0)

    return image

def main(args):
    batch_size = 64
    embed_size = 512
    num_heads = 8
    num_layers = 6
    num_workers = 2

    # Image preprocessing
    transform = transforms.Compose([
        transforms.ToTensor(),
        transforms.Normalize((0.485, 0.456, 0.406),
                             (0.229, 0.224, 0.225))])

    # Load vocabulary wrapper
    with open(args.vocab_path, 'rb') as f:
        vocab = pickle.load(f)

    data_loader = get_loader('data/resized2014', 'data/annotations/captions_train2014.json', vocab,
                             transform, batch_size,
                             shuffle=True, num_workers=num_workers)

    # Build models
    encoder = EncoderCNN(embed_size).to(device)
    encoder.fine_tune(False)
    decoder = Decoder(len(vocab), embed_size, num_heads, embed_size, num_layers).to(device)

    # Load trained models
    encoder.load_state_dict(torch.load(os.path.join('models/', 'encoder-{}-{}.ckpt'.format(1, 4000))))
    decoder.load_state_dict(torch.load(os.path.join('models/', 'decoder-{}-{}.ckpt'.format(1, 4000))))
    encoder.eval()
    decoder.eval()

    itos = vocab.idx2word
    pred_len = 100
    result_collection = []

    # Decode with greedy
    # with torch.no_grad():
    #     for i, (images, captions, lengths) in enumerate(data_loader):
    #         images = images.to(device)
    #         features = encoder(images)
    #         output = decoder.generator(features, pred_len)
    #         result_caption = token_sentence(output, itos)
    #         result_collection.extend(result_caption)

# Decode with greedy
    with torch.no_grad():
        for batch_index, (inputs, captions, caplens) in enumerate(data_loader):
            inputs, captions = inputs.cuda(), captions.cuda()
            enc_out = encoder(inputs)
            captions_input = captions[:, :-1]
            captions_target = captions[:, 1:]
            output = decoder.pred(enc_out, pred_len)
            result_caption = token_sentence(output, itos)
            result_collection.extend(result_caption)

    print("Prediction-greedy:", result_collection[1])
    print("Prediction-greedy:", result_collection[2])
    print("Prediction-greedy:", result_collection[3])
    print("Prediction-greedy:", result_collection[4])
    print("Prediction-greedy:", result_collection[5])
    print("Prediction-greedy:", result_collection[6])
    print("Prediction-greedy:", result_collection[7])
    print("Prediction-greedy:", result_collection[8])
    print("Prediction-greedy:", result_collection[9])
    print("Prediction-greedy:", result_collection[10])
    print("Prediction-greedy:", result_collection[11])

    # # Prepare an image
    # image = load_image(args.image, transform)
    # image_tensor = image.to(device)

    # # Generate an caption from the image
    # feature = encoder(image_tensor)
    # sampled_ids = decoder.generator(feature, pred_len)
    # sampled_ids = sampled_ids[0].cpu().numpy()          # (1, max_seq_length) -> (max_seq_length)

    # # Convert word_ids to words
    # sampled_caption = []
    # for word_id in sampled_ids:
    #     word = vocab.idx2word[word_id]
    #     sampled_caption.append(word)
    #     if word == '<end>':
    #         break
    # sentence = ' '.join(sampled_caption)

    # # Print out the image and the generated caption
    # print (sentence)
    # image = Image.open(args.image)
    # plt.imshow(np.asarray(image))

if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('--image', type=str, required=False, help='input image for generating caption')
    parser.add_argument('--vocab_path', type=str, default='data/vocab.pkl', help='path for vocabulary wrapper')
    args = parser.parse_args()
    main(args)

resize.py

import argparse
import os
from PIL import Image

def resize_image(image, size):
    """Resize an image to the given size."""
    return image.resize(size, Image.ANTIALIAS)

def resize_images(image_dir, output_dir, size):
    """Resize the images in 'image_dir' and save into 'output_dir'."""
    if not os.path.exists(output_dir):
        os.makedirs(output_dir)

    images = os.listdir(image_dir)
    num_images = len(images)
    for i, image in enumerate(images):
        with open(os.path.join(image_dir, image), 'r+b') as f:
            with Image.open(f) as img:
                img = resize_image(img, size)
                img.save(os.path.join(output_dir, image), img.format)
        if (i+1) % 100 == 0:
            print ("[{}/{}] Resized the images and saved into '{}'."
                   .format(i+1, num_images, output_dir))

def main(args):
    image_dir = args.image_dir
    output_dir = args.output_dir
    image_size = [args.image_size, args.image_size]
    resize_images(image_dir, output_dir, image_size)

if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('--image_dir', type=str, default='./data/train2014/',
                        help='directory for train images')
    parser.add_argument('--output_dir', type=str, default='./data/resized2014/',
                        help='directory for saving resized images')
    parser.add_argument('--image_size', type=int, default=256,
                        help='size for image after processing')
    args = parser.parse_args()
    main(args)


While running a UNet traning code I found DLL load failed error. Here is the code:
'''
     import torch
     import scipy
   import albumentations as A
  from ._nnls import nnls
  from albumentations.pytorch import ToTensorV2
  from tqdm import tqdm
  import torch.nn as nn
 import torch.optim as optim
 from unet_model import UNet
 from utilscar import (
load_checkpoint,
save_checkpoint,
get_loaders,
check_accuracy,
save_predictions_as_imgs,
)

  # Hyperparameters etc.
LEARNING_RATE = 1e-4
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
BATCH_SIZE = 16
NUM_EPOCHS = 3
NUM_WORKERS = 2
IMAGE_HEIGHT = 160  # 1280 originally
IMAGE_WIDTH = 240  # 1918 originally
PIN_MEMORY = True
LOAD_MODEL = False
TRAIN_IMG_DIR = "Dataset/train_images/"
TRAIN_MASK_DIR = "Dataset/train_masks/"
VAL_IMG_DIR = "Dataset/val_images/"
VAL_MASK_DIR = "Dataset/val_masks/"

def train_fn(loader, model, optimizer, loss_fn, scaler):
loop = tqdm(loader)

for batch_idx, (data, targets) in enumerate(loop):
    data = data.to(device=DEVICE)
    targets = targets.float().unsqueeze(1).to(device=DEVICE)

    # forward
    with torch.cuda.amp.autocast():
        predictions = model(data)
        loss = loss_fn(predictions, targets)

    # backward
    optimizer.zero_grad()
    scaler.scale(loss).backward()
    scaler.step(optimizer)
    scaler.update()

    # update tqdm loop
    loop.set_postfix(loss=loss.item())

def main():
   train_transform = A.Compose(
    [
        A.Resize(height=IMAGE_HEIGHT, width=IMAGE_WIDTH),
        A.Rotate(limit=35, p=1.0),
        A.HorizontalFlip(p=0.5),
        A.VerticalFlip(p=0.1),
        A.Normalize(
            mean=[0.0, 0.0, 0.0],
            std=[1.0, 1.0, 1.0],
            max_pixel_value=255.0,
        ),
        ToTensorV2(),
    ],
)

val_transform = A.Compose(
    [
        A.Resize(height=IMAGE_HEIGHT, width=IMAGE_WIDTH),
        A.Normalize(
            mean=[0.0, 0.0, 0.0],
            std=[1.0, 1.0, 1.0],
            max_pixel_value=255.0,
        ),
        ToTensorV2(),
    ],
)

model = UNet(in_channels=3, out_channels=1).to(DEVICE)
loss_fn = nn.BCEWithLogitsLoss()
optimizer = optim.Adam(model.parameters(), lr=LEARNING_RATE)

train_loader, val_loader = get_loaders(
    TRAIN_IMG_DIR,
    TRAIN_MASK_DIR,
    VAL_IMG_DIR,
    VAL_MASK_DIR,
    BATCH_SIZE,
    train_transform,
    val_transform,
    NUM_WORKERS,
    PIN_MEMORY,
)

if LOAD_MODEL:
    load_checkpoint(torch.load("my_checkpoint.pth.tar"), model)

check_accuracy(val_loader, model, device=DEVICE)
scaler = torch.cuda.amp.GradScaler()

for epoch in range(NUM_EPOCHS):
    train_fn(train_loader, model, optimizer, loss_fn, scaler)

    # save model
    checkpoint = {
        "state_dict": model.state_dict(),
        "optimizer":optimizer.state_dict(),
    }
    save_checkpoint(checkpoint)

    # check accuracy
    check_accuracy(val_loader, model, device=DEVICE)

    # print some examples to a folder
    save_predictions_as_imgs(
        val_loader, model, folder="saved_images/", device=DEVICE
    )

   if __name__ == "__main__":
   main()

'''
I found this error:
from ._nnls import nnls
ImportError: DLL load failed: The specified module could not be found.
Here I want to mention that, my python is 64 bit and all the libraries are also in 64 bit. I updated everything using conda update.


title: Pytorch 0.4: Should the requires_grad flag of the network input be set to True during training?

body: The requires_grad flag of input is default False during my last training. I was wondering if I should set it to True.


title: Model learns with SGD but not Adam

body:
I was going through a basic PyTorch MNIST example here and noticed that when I changed the optimizer from SGD to Adam the model did not converge. Specifically, I changed line 106 from

optimizer = optim.SGD(model.parameters(), lr=args.lr, momentum=args.momentum)

to

optimizer = optim.Adam(model.parameters(), lr=args.lr)

I thought this would have no effect on the model. With SGD the loss quickly dropped to low values after about a quarter of an epoch. However with Adam, the loss did not drop at all even after 10 epochs. I'm curious to why this is happening; it seems to me these should have nearly identical performance.

I ran this on Win10/Py3.6/PyTorch1.01/CUDA9

And to save you a tiny bit of code digging, here are the hyperparams:

lr=0.01
momentum=0.5
batch_size=64


title: How to convert embedding matrix to torch.Tensor

body:
I am new to pytorch and not sure how to convert an embedding matrix to a torch.Tensor type

I have 240 rows of input text data that I convert to embedding using Sentence Transformer library like below

embedding_model = SentenceTransformer('bert-base-nli-mean-tokens')
features = embedding_model.encode(df.features.values)

Now this features is a numpy.ndarray of shape (240, 768)

I have defined the model as

class NClassifier(nn.Module):

    def __init__(self, input_dim, embedding_dim, hidden_dim, tagset_size):
        super(NClassifier, self).__init__()
        self.hidden_dim = hidden_dim

        self.word_embeddings = nn.Embedding(input_dim, embedding_dim)

        # The LSTM takes word embeddings as inputs, and outputs hidden states
        # with dimensionality hidden_dim.
        self.lstm = nn.LSTM(embedding_dim, hidden_dim)

        # The linear layer that maps from hidden state space to code space (output clases)
        self.hidden2code = nn.Linear(hidden_dim, tagset_size)

    def forward(self, features):
        embeds = self.word_embeddings(features)
        lstm_out, _ = self.lstm(embeds.view(len(features), 1, -1))
        code_space = self.hidden2code(lstm_out.view(len(features), -1))
        code_scores = F.log_softmax(code_space, dim=1)
        return code_scores

INPUT_DIM = 240
EMBEDDING_DIM = 768
HIDDEN_DIM = 256
OUTPUT_DIM = 34

model = NClassifier(INPUT_DIM, EMBEDDING_DIM, HIDDEN_DIM, OUTPUT_DIM)

Now when I do scores = model(features) I get error as features is NOT a tensor. I see the example of converting the input to tensor here but it is not clear to me.

Can anyone please help?


title: pytorch sparse (sparse_coo_tensor) broadcast dense shape in multiplication

body:
Using broadcasting in normal tensors, the following works

torch.ones(3, 1)*torch.ones(3, 10)

I need to extend this behavior to sparse vectors, but I can't:

i = torch.tensor([[0, 1, 1],
                  [2, 0, 2]])
a = torch.sparse_coo_tensor(i, torch.ones(3, 1), [2, 4, 1])
b = torch.sparse_coo_tensor(i, torch.ones(3, 10), [2, 4, 10])
a*b
# gives RuntimeError: mul operands have incompatible sizes

Why shouldn't this work?
Is there an pytorch function to do this?
If not what is the best alternative algorithm?


title: How to solve error: no match between expected input batch size and target batch size in PyTorch?

body:
I attempting to create a logistic model on CIFAR10 dataset by PyTorch. However I am getting an error:

  ValueError: Expected input batch_size (900) to match target batch_size (300).

What I think is happening is that 3*100 is 300. So may be the 3 axis of the RGB image is doing that but I cant figure how to solve.

These are my hyperparameters.

batch_size = 100
learning_rate = 0.001

# Other constants
input_size = 32*32
num_classes = 10

Here I divide my data into train, validation and test data.

transform_train = transforms.Compose([transforms.Resize((32,32)),
                                      transforms.RandomHorizontalFlip(),
                                      transforms.RandomRotation(10),
                                      transforms.RandomAffine(0, shear=10, scale=(0.8,1.2)),
                                      transforms.ColorJitter(brightness=0.2, contrast=0.2, saturation=0.2),
                                      transforms.ToTensor(),
                                      transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
                               ])

transform = transforms.Compose([transforms.Resize((32,32)),
                               transforms.ToTensor(),
                               transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
                               ])
training_dataset = CIFAR10(root='D:\PyTorch\cifar-10-python', train=True, download=True, transform=transform_train)
train_ds, val_ds = random_split(training_dataset, [40000, 10000])
test_ds = CIFAR10(root='D:\PyTorch\cifar-10-python', train=False, download=True, transform=transform)

train_loader = DataLoader(train_ds, batch_size=100, shuffle=True)
val_loader = DataLoader(val_ds, batch_size = 100, shuffle = False)
test_loader = DataLoader(test_ds, batch_size = 100, shuffle=False)

This is the model.

class CifarModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(input_size,  num_classes)
    def forward(self, xb):
        xb = xb.view(-1, 32*32)
        #xb = xb.reshape(-1, 784)
        print(xb.shape)
        out = self.linear(xb)
        return out

    def training_step(self, batch):
        images, labels = batch
        out = self(images)                  # Generate predictions
        loss = F.cross_entropy(out, labels) # Calculate loss
        return loss

    def validation_step(self, batch):
        images, labels = batch
        out = self(images)                    # Generate predictions
        loss = F.cross_entropy(out, labels)   # Calculate loss
        acc = accuracy(out, labels)           # Calculate accuracy
        return {'val_loss': loss.detach(), 'val_acc': acc.detach()}

    def validation_epoch_end(self, outputs):
        batch_losses = [x['val_loss'] for x in outputs]
        epoch_loss = torch.stack(batch_losses).mean()   # Combine losses
        batch_accs = [x['val_acc'] for x in outputs]
        epoch_acc = torch.stack(batch_accs).mean()      # Combine accuracies
        return {'val_loss': epoch_loss.item(), 'val_acc': epoch_acc.item()}

    def epoch_end(self, epoch, result):
        print("Epoch [{}], val_loss: {:.4f}, val_acc: {:.4f}".format(epoch, result['val_loss'], result['val_acc']))

model = CifarModel()

def accuracy(outputs, labels):
    _, preds = torch.max(outputs, dim=1)
    return torch.tensor(torch.sum(preds == labels).item() / len(preds))
def evaluate(model, val_loader):
    outputs = [model.validation_step(batch) for batch in val_loader]
    return model.validation_epoch_end(outputs)

def fit(epochs, lr, model, train_loader, val_loader, opt_func=torch.optim.SGD):
    history = []
    optimizer = opt_func(model.parameters(), lr)
    for epoch in range(epochs):
        # Training Phase
        for batch in train_loader:
            loss = model.training_step(batch)
            loss.backward()
            optimizer.step()
            optimizer.zero_grad()
        # Validation phase
        result = evaluate(model, val_loader)
        model.epoch_end(epoch, result)
        history.append(result)
    return history
evaluate(model, val_loader)

Here's the error I encounter when I run evaluate function:

torch.Size([900, 1024])

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-23-3621eab8de1a> in <module>
     21         history.append(result)
     22     return history
---> 23 evaluate(model, val_loader)

<ipython-input-23-3621eab8de1a> in evaluate(model, val_loader)
      3     return torch.tensor(torch.sum(preds == labels).item() / len(preds))
      4 def evaluate(model, val_loader):
----> 5     outputs = [model.validation_step(batch) for batch in val_loader]
      6     return model.validation_epoch_end(outputs)
      7

<ipython-input-23-3621eab8de1a> in <listcomp>(.0)
      3     return torch.tensor(torch.sum(preds == labels).item() / len(preds))
      4 def evaluate(model, val_loader):
----> 5     outputs = [model.validation_step(batch) for batch in val_loader]
      6     return model.validation_epoch_end(outputs)
      7

<ipython-input-22-c9e17d21eaff> in validation_step(self, batch)
     19         images, labels = batch
     20         out = self(images)                    # Generate predictions
---> 21         loss = F.cross_entropy(out, labels)   # Calculate loss
     22         acc = accuracy(out, labels)           # Calculate accuracy
     23         return {'val_loss': loss.detach(), 'val_acc': acc.detach()}

~\Anaconda3\lib\site-packages\torch\nn\functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction)
   2019     if size_average is not None or reduce is not None:
   2020         reduction = _Reduction.legacy_get_string(size_average, reduce)
-> 2021     return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
   2022
   2023

~\Anaconda3\lib\site-packages\torch\nn\functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)
   1834     if input.size(0) != target.size(0):
   1835         raise ValueError('Expected input batch_size ({}) to match target batch_size ({}).'
-> 1836                          .format(input.size(0), target.size(0)))
   1837     if dim == 2:
   1838         ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)

ValueError: Expected input batch_size (900) to match target batch_size (300).


title: Scatter tensor in pytorch along the rows

body:
I want to scatter tensors in granularities of rows.

For example consider,

Input = torch.tensor([[2, 3], [3, 4], [4, 5]])

I want to scatter

S = torch.tensor([[1,2],[1,2]])

to indices

I = torch.tensor([0,2])

I expect the output to be torch.tensor([[1, 2], [3, 4], [1, 2]]).

Here S[0] is scattered to Input[I[0]], similarly S[1] is scattered to Input[I[1]]

How can I achieve this? Instead of looping over the row in S, I am looking for a more efficient way.


title: How can I get a view of input as a complex tensor? RuntimeError: Tensor must have a last dimension with stride 1

body:
I have a tensor with 64 elements in pytorch and I want to convert it to a complex tensor with 32 elements. Order is important for me and everything should be in PyTorch so I can use it in my customized loss function:
the first half in my primary tensor (W) are my real numbers and the second half are my imaginary ones. so my final tensor should be like:
W_final = tensor(W[0]+jW[32], W[1]+jW[33], W[2]+jW[34], W[3]+jW[35], ... , W[31]+jW[63])
I tried this approach:
import torch
W_1 = = torch.reshape(W,(2,32)) #reshape W with shape (64) to W_1 with shape (2,32)
W_2 = torch.transpose(W_1,0,1) #transpose W_1 to W_2 with shape (32,2), so I can use view_as_complex
W_final = torch.view_as_complex(W_2)

The problem is that with transpose, the stride also changes and I get this error:
RuntimeError: Tensor must have a last dimension with stride 1
Do know how can I deal with stride? or is there any way to reshape with different orders same as numpy?
or any other way to convert to complex?


title: How to randomly set a fixed number of elements in each row of a tensor in PyTorch

body:
I was wondering if there is any more efficient alternative for the below code, without using the "for" loop in the 4th line?
import torch
n, d = 37700, 7842
k = 4
sample = torch.cat([torch.randperm(d)[:k] for _ in range(n)]).view(n, k)
mask = torch.zeros(n, d, dtype=torch.bool)
mask.scatter_(dim=1, index=sample, value=True)

Basically, what I am trying to do is to create an n by d mask tensor, such that in each row exactly k random elements are True.


title: Customizing the batch with specific elements

body:
I am a fresh starter with pytorch. Strangley I cannot find anything related to this, although it seems rather simple.
I want to structure my batch with specific examples, like all examples per batch having the same label or just fill the batch with examples of just 2 classes.
How would I do that? For me it seems the right place within the dataloader and not in the dataset? As the dataloader is responsible for the batches and not the dataset?
Is there simple minimal example?


title: How to train neural network over days?

body:
I need to train a CNN that will take 1-2 days to train on a remotely accessed GPU server.

Will I simply need to leave my laptop on overnight for the training to be complete or is there a way to save the state of the training and resume from there the next day?

(Implementation in pytorch)


title: Access the output of several layers of pretrained DistilBERT model

body:
I am trying to access the output embeddings from several different layers of the pretrained "DistilBERT" model. ("distilbert-base-uncased")

bert_output = model(input_ids, attention_mask=attention_mask)

The bert_output seems to return only the embedding values of the last layer for the input tokens.


title: How can we provide the output of a Linear layer to a Conv2D in PyTorch?

body:
I am building an Autoencoder where I need to encode an image into a latent representation of length 100. I am using the following architecture for my model.

        self.conv1 = nn.Conv2d(in_channels = 3, out_channels = 32, kernel_size=3)
        self.conv2 = nn.Conv2d(in_channels=32,out_channels=64,kernel_size=3,stride=2)
        self.conv3 = nn.Conv2d(in_channels=64,out_channels=128,kernel_size=3,stride=2)

        self.linear = nn.Linear(in_features=128*30*30,out_features=100)

        self.conv1_transpose = nn.ConvTranspose2d(in_channels=128,out_channels=64,kernel_size=3,stride=2,output_padding=1)
        self.conv2_transpose = nn.ConvTranspose2d(in_channels=64,out_channels=32,kernel_size=3,stride=2,output_padding=1)
        self.conv3_transpose = nn.ConvTranspose2d(in_channels=32,out_channels=3,kernel_size=3,stride=1)

Is there any way I could give my Linear layer's output to a Conv2D or a ConvTranspose2D layer so that I can reconstruct my image? The output is restored if I remove the Linear layer. I want to know how I can reconstruct my image keeping the Linear layer

Any help would be appreciated. Thanks!


title: Masking only specific words with Huggingface

body:
What would be the best strategy to mask only specific words during the LM training?
My aim is to dynamically mask at batch-time only words of interest which I have previously collected in a list.
I have already had a look at the mask_tokens() function into the DataCollatorForLanguageModeling class, which is the function actually masking the tokens during each batch, but I cannot find any efficient and smart way to mask only specific words and their corresponding IDs.
I tried one naive approach consisting of matching all the IDs of each batch with a list of word's IDs to mask. However, a for-loop approach has a negative on the performance.
.

Side issue about word prefixed space - Already fixed
Thanks to @amdex1 and @cronoik for helping with a side issue.
This problem arose since the tokenizer, not only splits a single word in multiple tokens, but it also adds special characters if the word does not occur at the begging of a sentence.
E.g.:

The word "Valkyria":

at the beginning of a sentences gets split as ['V', 'alky', 'ria'] with corresponding IDs: [846, 44068, 6374].
while in the middle of a sentence as ['ĠV', 'alky', 'ria'] with corresponding IDs: [468, 44068, 6374],

It is solved by setting add_prefix_space=True in the tokenizer.


title: pytorch operations: union and inversion of values in torch tensor

body:
I have set of masks on cuda/GPU corresponding to different objects in an image (shapes and sizes below)
masks.shape:            torch.Size([10, 240, 320, 1])
masks[0].shape:         torch.Size([240, 320, 1])
masks[0][:,:,0].shape:  torch.Size([240, 320])

1: Can I produce union of the these masks using torch.tensor operation? so that I can apply all of those once on the image?
2: How do I invert the values in torch tensor? I mean for 1, turn into 0 and vice versa. I have tried to ~mytensor but it says the operator is only applicable to integer or bool values. I have got float values in my tensors i.e. [1.] etc.
I intent to do all these operations on GPU without moving data back on CPU.
Thank you.


title: Use pytorch on Alpine

body:
I have a docker image in alpine, and need to use pytorch on it. However, as stated in the pytorch website, it requires glibc to run, otherwise it will give error below
import torch
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.7/site-packages/torch/__init__.py", line 189, in <module>
    _load_global_deps()
  File "/usr/lib/python3.7/site-packages/torch/__init__.py", line 142, in _load_global_deps
    ctypes.CDLL(lib_path, mode=ctypes.RTLD_GLOBAL)
  File "/usr/lib/python3.7/ctypes/__init__.py", line 364, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: Error relocating /usr/lib/python3.7/site-packages/torch/lib/libgomp-7c85b1e2.so.1: pthread_attr_setaffinity_np: symbol not found

I've tried to install glibc from https://github.com/sgerrand/alpine-pkg-glibc/releases/download/2.29-r0/glibc-2.29-r0.apk, but error info stay the same.


title: If I Trace a PyTorch Network on Cuda, can I use it on CPU?

body:
I traced my Neural Network using torch.jit.trace on a CUDA-compatible GPU server. When I reloaded that Trace on the same server, I could reload it and use it fine. Now, when I downloaded it onto my laptop (for quick testing), when I try to load the trace I get:
RuntimeError: Could not run 'aten::empty_strided' with arguments from the 'CUDA' backend. 'aten::empty_strided' is only available for these backends: [CPU, BackendSelect, Named, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, AutogradPrivateUse1, AutogradPrivateUse2, AutogradPrivateUse3, Tracer, Autocast, Batched, VmapMode].

Can I not switch between GPU and CPU on a trace? Or is there something else going on?


title: AttributeError: module 'torch' has no attribute 'swapaxes'

body: For some reason when using PyTorch on Google Colab, I'm unable to use torch.swapaxes and returns this error. Am I missing some additional import or something?


title: How to tell PyTorch which CUDA version to take?

body:
I have two version of CUDA installed on my Ubuntu 16.04 machine: 9.0 and 10.1.
They are located in /usr/local/cuda-9.0 and /usr/local/10.1 respectively.
If I install PyTorch 1.6.0 (which needs CUDA 10.1) via pip (pip install torch==1.6.0), it uses version 9.0 and thus detects no GPUs. I already changed my LD_LIBRARY_PATH to "/usr/local/cuda-10.1/lib64:/usr/local/cuda-10.1/cuda/extras/CUPTI/lib64" but PyTorch is still using CUDA 9.0.
How do I tell PyTorch to use CUDA 10.1?


title: necessity of transposed convolution when feature maps are not downsampled

body:
I was reading a paper here. The authors in the paper have proposed a symmetric generator network which contains a stack of convolution layers followed by a stack of de-convolution (transposed convolution) layers. It is also mentioned that a stride of 1 with appropriate padding is used to ensure that feature map size is same as input image size.

My question is if there is no downsampling, then why transposed conolution layers are used? Can't the generator be constructed only with convolution layers? Am I missing something about transposed convolution layers here (is it being used for some other purpose)? Please help.

Update: I am re-opening this question, as I came across this paper which states in section 2.1.1 that "deconvolution is used to compensate the details". However, I am not able to appreciate this because there is no downsampling of feature maps in the proposed model. Can somebody explain why deconvolution is preferred over convolution here? What makes deconvolution layer perform better than convolution in this case?


title: Word2vec CBOW model implementations, deviations from the original algorithm

body:
I am trying to implement CBOW model by pytorch.
What I understood from the explanation of word2vec is that word2vec has 2 layers (and therefore 2 matrices), the first matrix contains low dimensional word vectors, which is actually a lookup table and the vector representation of the word projected on the projection layer (no non-linearity, therefore not a hidden layer). The word vectors then multiplied by the 2nd matrix and the that goes to the output through a softmax function. After training, the first matrix can be used as a word embedding.

I see many implementation use 3 layers (1 embedding layer plus 2 more layers.), which is contradictory to my understanding above. Some example implementations here, here and here.
The following three lines of codes have commonly used as a model to implment 3 layer:

self.embeddings = nn.Embedding(vocab_size, embedding_dim)
self.linear1 = nn.Linear(context_size * embedding_dim, 128)
self.linear2 = nn.Linear(128, vocab_size)

My questions are, if my understanding is okay, then why they are using 3 layers? Are there any advantages?

One obvious disadvantage, I think is, it will be computationally expensive.

Word2vec resembles the idea of autoencoder (which also have two-layer), deviating from this proven idea might harm the embedding quality. Am I right?

Another important thing is that, according to the paper that I mentioned above, for multiple context words the average of the vectors will be projected on the protection layer. But instead of averaging, they are concatenating the vectors. Why is that? is there any advantages?

Also, they are using non-linearity at the hidden layer which I think will create a serious performance issue in training with a huge amount of data. Right?


title: "Can't get attribute 'ImageList'" using FastAI and PyTorch

body:
Version Numbers:

torchvision: 0.3.0
torch: 1.1.0
fastai: 1.0.57

For no reason that comes to mind, I always run into this error: Can't get attribute 'ImageList' when loading my model. I use load_learner(".", fname="model.pkl") to load my model, and it has the relevant files within the folder (final.pth,model.pkl,classes.txt) but it doesn't seem to work at all. I've checked the internet, but I've found nothing with ImageList.

The full error: AttributeError: Can't get attribute 'ImageList' on <module 'fastai.vision.data' from '/python3.6/site-packages/fastai/vision/data.py'>


title: pytorch: sum of cross entropy over all classes

body:
I want to compute sum of cross entropy over all classes for each prediction, where the input is batch (size n), and the output is batch (size n).

The simplest way is for loop (for 1000 classes):

def sum_of_CE_lost(input):
    L = 0
    for c in range(1000):
        L = L + torch.nn.CrossEntropyLoss(input, c)
    return L

However, it is very slow. What is a better way? How can we parallelized it for GPU (CUDA)?


title: OCR in pytorch/keras with LPRNet and CTCLoss doesn't converge

body:
I implemented LPRNet from this paper in keras and then in pytorch. For pytorch was used this code as example. It didn't converge in any version. Loss starts at 30 and goes to 27 even after 150000 epochs. In pytorch version I am not using validation, for faster training.

what I tried:

initialized conv2d weights with xavier_uniform_, xavier_normal and all the others I found in torch.nn.init
used another dataset with 50 clear images used by other guys for their rcnn ocr model
tried sgd instead of adam

full colab with pytorch is here and my dataset is here

In keras I once succeded to overfit the model over training set. My training set has only 50 images but I apply random effect for not overfitting. In their paper, authors didn't specify how large dataset they used. May be the problem is in the model implementation ? Also, original paper contained spatial transformer network which I didn't use because they told it's optional, but can it's absence be an issue ?

class Softmax(nn.Module):
    def __init__(self):
        super().__init__()

    def forward(self, x):
        #se aplica pt torch.Size([1, 30, 1, 74])
        return torch.nn.functional.log_softmax(x, 3)

class Reshape(nn.Module):
    def __init__(self):
        super().__init__()

    def forward(self, x):
        # se aplica pt torch.Size([1, 30, 1, 74])
        #pentru CTCloss
        #trebuie sa fie (T,N,C) , where T=input length, N=batch size, C=number of classes
        x = x.permute(3, 0, 1, 2)
        return x.view(x.shape[0], x.shape[1], x.shape[2])

def init_weights(m):
    if type(m) == nn.Conv2d:
        torch.nn.init.xavier_uniform_(m.weight)
        #m.bias.data.fill_(0.01)

class small_basic_block(nn.Module):
    def __init__(self, ch_in, ch_out):
        super().__init__()
        self.block = nn.Sequential(
            nn.Conv2d(ch_in, ch_out//4, kernel_size=1),
            nn.ReLU(),
            nn.Conv2d(ch_out//4, ch_out//4, kernel_size=(3,1), padding=(1,0)),
            nn.ReLU(),
            nn.Conv2d(ch_out//4, ch_out//4, kernel_size=(1,3), padding=(0,1)),
            nn.Conv2d(ch_out//4, ch_out, kernel_size=1)
        )
        self.block.apply(init_weights)

    def forward(self, x):
        return self.block(x)

class LPRNet(nn.Module):
    def __init__(self, class_num):
        super().__init__()
        self.lprnet = nn.Sequential(
            nn.Conv2d(3, 64, kernel_size=3, stride=1),
            nn.BatchNorm2d(num_features=64),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=3, stride=1),
            small_basic_block(ch_in=64, ch_out=128),
            nn.BatchNorm2d(num_features=128),
            nn.ReLU(),
            nn.MaxPool3d(kernel_size=(1, 3, 3), stride=(2, 2, 1)),
            small_basic_block(ch_in=64, ch_out=256),
            nn.BatchNorm2d(num_features=256),
            nn.ReLU(),
            small_basic_block(ch_in=256, ch_out=256),
            nn.BatchNorm2d(num_features=256),
            nn.ReLU(),
            nn.MaxPool3d(kernel_size=(1,3,3), stride=(4,2,1)),
            nn.Dropout(0.5),
            nn.Conv2d(64, 256, kernel_size=(4,1), stride=1),
            nn.BatchNorm2d(num_features=256),
            nn.ReLU(),
            nn.Dropout(0.5),
            nn.Conv2d(256, class_num, kernel_size=(1,13), stride=1),
            nn.BatchNorm2d(num_features=class_num),
            nn.ReLU(), # torch.Size([1, 30, 1, 74])
            Softmax(),
            Reshape()
        )
        self.lprnet.apply(init_weights)

    def forward(self, x):
        return self.lprnet(x)

ctc_loss = nn.CTCLoss(blank=alphabet.index('$'))

model = LPRNet(len(alphabet)+1)
model.to(dev)
opt = optim.Adam(model.parameters())
scheduler = optim.lr_scheduler.StepLR(opt, step_size=100000, gamma=0.1)

def train(epochs):
    for i in range(epochs):
        X_data, Y_data, X_data_len, Y_data_len = train_set.next_batch()
        X_data = model(X_data)
        loss = ctc_loss(X_data, Y_data, X_data_len, Y_data_len)
        loss.backward()
        opt.step()
        opt.zero_grad()
        scheduler.step()


title: masked_scatter but rowwise?

body:
Assuming a mask as follows:

mask = torch.tensor([
  [True,  True,  False, True,  False],
  [True,  False, True,  True,  True ],
])

I would like to number the True values with sequential values in each row separately. I don't care what's in the False spots, so 0 for simplicity. Thus the desired result is

tensor([[0, 1, 0, 2, 0],    # 0 1 _ 2 _
        [0, 0, 1, 2, 3]])   # 0 _ 1 2 3

I hoped this would work:

replacements = torch.arange(mask.size(1)).expand(mask.size())
target = torch.zeros(mask.size(), dtype=int)
target.masked_scatter(mask, replacements)

Unfortunately, masked_scatter ignores the shape of replacements, so this code results in:

tensor([[0, 1, 0, 2, 0],    # 0 1 _ 2 _
        [3, 0, 4, 0, 1]])   # 3 _ 4 0 1

What would I need to do instead?


title: flip with any degree of angle?

body:
This is the one of python example of flip the image with the corresponding box
is it possible to write this as flip 20 degree angle or 20,25 or 45 degree angle?
how to change this code to make any degree of angles???
import torchvision.transforms.functional as FT

def flip(image, boxes):
    """
    Flip image horizontally.

    :param image: image, a PIL Image
    :param boxes: bounding boxes in boundary coordinates, a tensor of dimensions (n_objects, 4)
    :return: flipped image, updated bounding box coordinates
    """
    # Flip image
    new_image = FT.hflip(image)
    #print('new_image->',new_image)
    # Flip boxes
    new_boxes = boxes
    new_boxes[:, 0] = image.width - boxes[:, 0] - 1
    new_boxes[:, 2] = image.width - boxes[:, 2] - 1
    new_boxes = new_boxes[:, [2, 1, 0, 3]]

    return new_image, new_boxes


title: What does it mean for the shape of an image tensor to be (64, 64)? Does that mean there are no channels?

body:
I was exploring the images of the dataset : Tiny Imagenet and I found that most of the image tensors have shapes (64, 64, 3) i.e. images of height and width 64 and 64 respectively and three channels for Red, Green and Blue. But some of the image tensors in the dataset have shape (64, 64).

Does it mean that there are no channels? How is it possible?
EDIT - I downloaded this image with tensor shape (64, 64):

EDIT - Is it possible that it has one channel of some other colour space like Lab? (here b refers to the colours from blue to yellow).


title: Why does the evaluation loss increases when training a huggingface transformers NER model?

body:
Training a huggingface transformers NER model according to the documentation, the evaluation loss increases after a few epochs, but the other scores (accuracy, precision, recall, f1) continuously getting better. The behaviour seems unexpected, is there a simple explanation for this effect? Can this depend on the given data?

model = TokenClassificationModel.from_pretrained('roberta-base', num_labels=len(tag_values))

model.train()
model.zero_grad()

   for epoch in range(epochs):
      for batch in range(batches):
         -- train --
         ...

      train_loss = model.evaluate(train_data)
      validation_loss = model.evaluate(validation_data)


title: Forward method calling in Linear class in Pytorch

body:
I am trying to use Linear class of Pytorch module.In this automatically forward method is called once I create an instance of the class.Can anyone explain me from basics (If I can get code of Linear class of pytorch that would be great)

from torch.nn mport Linear
model=Linear(in_features=1,out_features=1)
x=torch.tensor([1])
y=model(x)

I am not able to understand how model(x) displays the result.As per my understanding model.forward() should display the actual result
It would be great if someone explain me from basics (Would be great if someone can share the link of code of Linear class in pytorch)


title: SampleRNN - Pytorch implementation beginner

body:
I'm trying to start work with this: https://github.com/deepsound-project/samplernn-pytorch
I've installed all the library dependencies through Anaconda console but I'm then not sure how I'm to run the python training scripts.

I guess I just need general help with getting a git RNN in python working? I've found a lot of tutorials that show working from notebooks in Jupyter or even from scratch but can't find ones working from python code files?
I'm sorry if my terminology is backward, I'm an architect who is attempting coding, note a software engineer.


title: Custom LSTM cell implementation

body:
Shoutout to all Pytorch core developers!

I would like to implement a custom version of the typical LSTM cell as it is implemented in Pytorch, say, change one of the activation functions at a gate. For this, I would like to see how the LSTM is implemented in Pytorch at the moment. I can find some code here, but unfortunately, I cannot find the exact LSTM computations there, including the implementation of the gate formulas as they are also specified in the docstrings.

Related posts can for example be found here, but all they delivered me is that nobody has found the LSTM cell code in github.

Where is the LSTM cell implemented in the Pytorch Github exactly?


title: Pytorch inconsistent size with pad_packed_sequence, seq2seq

body:
I'm having some inconsistencies with the output of a encoder I got from this github .

The encoder looks as follows:

class Encoder(nn.Module):
    r"""Applies a multi-layer LSTM to an variable length input sequence.
    """

    def __init__(self, input_size, hidden_size, num_layers,
                 dropout=0.0, bidirectional=True, rnn_type='lstm'):
        super(Encoder, self).__init__()
        self.input_size = 40
        self.hidden_size = 512
        self.num_layers = 8
        self.bidirectional = True
        self.rnn_type = 'lstm'
        self.dropout = 0.0
        if self.rnn_type == 'lstm':
            self.rnn = nn.LSTM(input_size, hidden_size, num_layers,
                               batch_first=True,
                               dropout=dropout,
                               bidirectional=bidirectional)

    def forward(self, padded_input, input_lengths):
        """
        Args:
            padded_input: N x T x D
            input_lengths: N
        Returns: output, hidden
            - **output**: N x T x H
            - **hidden**: (num_layers * num_directions) x N x H
        """
        total_length = padded_input.size(1)  # get the max sequence length
        packed_input = pack_padded_sequence(padded_input, input_lengths,
                                            batch_first=True,enforce_sorted=False)
        packed_output, hidden = self.rnn(packed_input)
        pdb.set_trace()
        output, _ = pad_packed_sequence(packed_output, batch_first=True, total_length=total_length)
        return output, hidden

So it only consists of a rnn lstm cell, if I print the encoder this is the output:

LSTM(40, 512, num_layers=8, batch_first=True, bidirectional=True)

So it should have a 512 sized output right? But when I feed a tensor with size torch.Size([16, 1025, 40]) 16 samples of 1025 vectors with size 40 (that gets packed to fit the RNN) the output that I get from the RNN has a new encoded size of 1024 torch.Size([16, 1025, 1024]) when it should have been encoded to 512 right?

Is there something Im missing?


I am using Anaconda base environment for second.pytorch of ubuntu 16.04
after executing the following:

nvidia-docker run -it --rm -v /media/yy/960evo/datasets/:/root/data -v $HOME/pretrained_models:/root/model --ipc=host second-pytorch:latest

the next thing I excuted is stated below:

python ./pytorch/train.py evaluate --config_path=./configs/car.config --model_dir=/root/model/car

I get the following error:

Traceback (most recent call last):   File "/root/second.pytorch/second/core/non_max_suppression/nms_cpu.py", line 10, in <module>
    from second.core.non_max_suppression.nms import ( ModuleNotFoundError: No module named 'second.core.non_max_suppression.nms'

I also tried to go through the solutions provided and ran the following piece of code:

nvcc -std=c++11 -c -o ../cc/nms/nms_kernel.cu.o ../cc/nms/nms_kernel.cu.cc -I/usr/local/cuda/include -x cu -Xcompiler -fPIC -arch=sm_61 --expt-relaxed-constexpr

I got the following error

gcc: error: ../cc/nms/nms_kernel.cu.cc: No such file or directory
gcc: warning: '-x c++' after last input file has no effect
gcc: fatal error: no input files
compilation terminated.

any help would be appreciated


title: PyTorch gradients have different shape for CUDA and CPU

body:
I’m dealing with a strange issue where the gradients after backward pass have different shapes depending on whether CUDA or CPU is used. The model used is relatively simple:
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool1 = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.pool2 = nn.MaxPool2d(2, 2)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)
        self.relu1 = nn.ReLU()
        self.relu2 = nn.ReLU()
        self.relu3 = nn.ReLU()
        self.relu4 = nn.ReLU()

    def forward(self, x):
        x = self.pool1(self.relu1(self.conv1(x)))
        x = self.pool2(self.relu2(self.conv2(x)))
        x = x.view(-1, 16 * 5 * 5)
        x = self.relu3(self.fc1(x))
        x = self.relu4(self.fc2(x))
        x = self.fc3(x)
        return x

The input tensor has shape (1, 3, 32, 32), and the relevant section of code is as follows, with the method generate_gradients being of particular importance:
class VanillaBackprop():
    """
        Produces gradients generated with vanilla back propagation from the image
    """
    def __init__(self, model):
        self.model = model
        self.gradients = None
        # Put model in evaluation mode
        self.model.eval()
        # Hook the first layer to get the gradient
        self.hook_layers()
        self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
        self.model.to(self.device)

    def hook_layers(self):
        def hook_function(module, grad_in, grad_out):
            self.gradients = grad_in[0]

        # Register hook to the first layer
        try:
            first_layer = list(self.model.features._modules.items())[0][1]
        except:
            first_layer = list(self.model._modules.items())[0][1]
        first_layer.register_backward_hook(hook_function)

    def generate_gradients(self, input_image, target_class):
        # Forward
        model_output = self.model(input_image.to(self.device))
        # Zero grads
        self.model.zero_grad()
        # Target for backprop
        one_hot_output = torch.FloatTensor(1, model_output.size()[-1]).zero_()
        one_hot_output[0][target_class] = 1
        # Backward pass
        model_output.backward(gradient=one_hot_output.to(self.device))
        # Convert Pytorch variable to numpy array
        gradients_as_arr = self.gradients.data.cpu().numpy()[0]
        return gradients_as_arr

When on CPU, self.gradients has shape (1, 3, 32, 32), while on CUDA it has shape (1, 6, 28, 28). How is that possible, and how do I fix this? Any help is much appreciated.


title: HParams in Tensorboard, Run IDs and naming

body:
I'm using SummaryWriter.add_hparams(params, values) to log hyperparameters during training of my Seq2Seq model. My runs are named with a timestamp like 2020-09-10 14-50-27. In the HParams tab in Tensorboard, everything looks fine, but the HParam Trial IDs are different; they have another string of numbers attached like this: 2020-09-10 14-50-27/1599742915.9712806. These also appear in the Scalar tab as different runs, which is quite inconvenient. Is there a way to turn of this extra naming or to stop them of appearing in the Scalars tab? I use pytorch and its summarywriter like this:
params = {
    'max_epochs' : max_epochs,
    'learning_rate': learning_rate,
    'batch_size': batch_size,
    'optimizer_name': optimizer_name,
    'dropout_fc': dropout_fc
}
values = {
    'hparam/hp_total_time': t1_stop - t0_start,
    'hparam/score' : best_score
}

tb.add_hparams(params, values)


title: Python Error RuntimeError: expected scalar type Long but found Double

body:
Firstly, I am fairly new to python/ML in general. I am attempting to utilize the model depicted at stackabuse over my own data set.
Everything flows smoothly until I get ready to run the epochs.
In debugging I see that it is failing on CrossEntropyLoss function and I get the error expected long found double. The data set it appears to fail on is the my tdiff column that I calculated but I can't seem to figure out how to convert it to a long.
Is there something that I'm missing in trying to figure this out?
To be clear, this is what I THINK it is based on my extremely limited knowledge on the function:
import pandas as pd
import sqlite3 as sq
import numpy as np
import torch
import torch.nn as nn
import matplotlib.pyplot as plt
import seaborn as sns

con = sq.connect("filepath")
df = pd.read_sql_query("SELECT b.actcod," +
         "a.trnqty, " +
          "b.dlytrn_id, " +
              "a.usr_id, " +
 " b.oprcod, " +
 "b.usr_id as usrid," +
 " b.tostol as frstol," +
 " a.tostol, " +
 " b.trndte as bdte," +
 " a.trndte as adte," +
 " '' as tdiff" +
 " FROM" +
 " (SELECT  row_number() over(PARTITION by usr_id order by trndte) AS rown," +
 " trndte," +
 " dlytrn_id," +
 " trnqty, " +
 " oprcod," +
 " actcod," +
 " frstol," +
 " tostol," +
 " usr_id" +
 " FROM Master_File" +
 " WHERE actcod = 'PALPCK') a " +
 " JOIN "+
 " (SELECT  row_number() over(PARTITION by usr_id order by trndte) AS rown," +
 " trndte,dlytrn_id, trnqty, oprcod,actcod, frstol, tostol, usr_id" +
 " FROM Master_File" +
 " WHERE actcod = 'PALPCK' ) b" +
 " ON b.rown = (a.rown -1)" +
 " and b.usr_id = a.usr_id" #+
 #" LIMIT 10"
,con)

# Create record Count
record_count = df['dlytrn_id'].count()
test_records = int(record_count * .2)
print(record_count)
print(test_records)

#Calculate time delta between scans

df['tdiff'] = pd.TimedeltaIndex.total_seconds(pd.to_datetime(df['adte'].values) - pd.to_datetime(df['bdte'].values))

# Establish column types

cat_column = ['usrid','actcod','tostol','frstol']
num_column = ['trnqty']
dte_column = ['bdte','adte']
op_column = ['tdiff']

#convert column datatypes
for category in cat_column:
    df[category] = df[category].astype('category')
for category in num_column:
    df[category] = df[category].astype('float64')
for category in dte_column:
    df[category] = df[category].astype('datetime64')

#create categorical tensor

usr = df['usrid'].cat.codes.values
act = df['actcod'].cat.codes.values
to = df['tostol'].cat.codes.values
fr = df['frstol'].cat.codes.values

cat_data = np.stack([usr,act,to,fr], 1)
cat_data = torch.tensor(cat_data, dtype=torch.int64)

#Create Numerical Tensor

num_data = np.stack([df[col].values for col in num_column], 1)
num_data = torch.tensor(num_data, dtype=torch.float)

#convert output array

op_data = torch.tensor(df[op_column].values).flatten()

print(df.shape)
print(cat_data.shape)
print(num_data.shape)
print(op_data.shape)

#convert categorical data into an N-Dimensional vector,
#this will allow better ML analysis on relationships

cat_col_sizes = [len(df[column].cat.categories) for column in cat_column]
cat_embedding_sizes = [(col_size, min(50, (col_size+1)//2)) for col_size in cat_col_sizes]
print(cat_embedding_sizes)

#create test batches

cat_train_data = cat_data[:record_count-test_records]
cat_test_data = cat_data[record_count-test_records:record_count]
num_train_data = num_data[:record_count-test_records]
num_test_data = num_data[record_count-test_records:record_count]
train_op = op_data[:record_count-test_records]
test_op = op_data[record_count-test_records:record_count]

print(len(cat_train_data))
print(len(num_train_data))
print(len(train_op))

print(len(cat_test_data))
print(len(num_test_data))
print(len(test_op))

class Model(nn.Module):

    def __init__(self, embedding_size, num_numerical_cols, output_size, layers, p=0.4):
        super().__init__()
        self.all_embeddings = nn.ModuleList([nn.Embedding(ni, nf) for ni, nf in embedding_size])
        self.embedding_dropout = nn.Dropout(p)
        self.batch_norm_num = nn.BatchNorm1d(num_numerical_cols)

        all_layers = []
        num_categorical_cols = sum((nf for ni, nf in embedding_size))
        input_size = num_categorical_cols + num_numerical_cols

        for i in layers:
            all_layers.append(nn.Linear(input_size, i))
            all_layers.append(nn.ReLU(inplace=True))
            all_layers.append(nn.BatchNorm1d(i))
            all_layers.append(nn.Dropout(p))
            input_size = i

        all_layers.append(nn.Linear(layers[-1], output_size))

        self.layers = nn.Sequential(*all_layers)

    def forward(self, x_categorical, x_numerical):
        embeddings = []
        for i,e in enumerate(self.all_embeddings):
            embeddings.append(e(x_categorical[:,i]))
        x = torch.cat(embeddings, 1)
        x = self.embedding_dropout(x)

        x_numerical = self.batch_norm_num(x_numerical)
        x = torch.cat([x, x_numerical], 1)
        x = self.layers(x)
        return x

model = Model(cat_embedding_sizes, num_data.shape[1], 1, [200,100,50], p=0.4)
print(model)

loss_function = nn.CrossEntropyLoss()

optimizer = torch.optim.Adam(model.parameters(), lr=0.001)

epochs = 300
aggregated_losses = []

for i in range(epochs):
    i += 1
    y_pred = model(cat_train_data, num_train_data)
    single_loss = loss_function(y_pred, train_op)
    aggregated_losses.append(single_loss)

    if i%25 == 1:
        print(f'epoch: {i:3} loss: {single_loss.item():10.8f}')

    optimizer.zero_grad()
    single_loss.backward()
    optimizer.step()

print(f'epoch: {i:3} loss: {single_loss.item():10.10f}')

print(train_op)
print(type(train_op))

print(op_column)
print(type(op_column))

print(op_data)
print(type(op_data))

con.close()

output:
51779
10355
(51779, 11)
torch.Size([51779, 4])
torch.Size([51779, 1])
torch.Size([51779])
[(185, 50), (1, 1), (302, 50), (303, 50)]
41424
41424
41424
10355
10355
10355
Model(
  (all_embeddings): ModuleList(
    (0): Embedding(185, 50)
    (1): Embedding(1, 1)
    (2): Embedding(302, 50)
    (3): Embedding(303, 50)
  )
  (embedding_dropout): Dropout(p=0.4, inplace=False)
  (batch_norm_num): BatchNorm1d(1, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (layers): Sequential(
    (0): Linear(in_features=152, out_features=200, bias=True)
    (1): ReLU(inplace=True)
    (2): BatchNorm1d(200, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (3): Dropout(p=0.4, inplace=False)
    (4): Linear(in_features=200, out_features=100, bias=True)
    (5): ReLU(inplace=True)
    (6): BatchNorm1d(100, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (7): Dropout(p=0.4, inplace=False)
    (8): Linear(in_features=100, out_features=50, bias=True)
    (9): ReLU(inplace=True)
    (10): BatchNorm1d(50, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (11): Dropout(p=0.4, inplace=False)
    (12): Linear(in_features=50, out_features=1, bias=True)
  )
)
Traceback (most recent call last):

  File  line 193, in <module>
    single_loss = loss_function(y_pred, train_op)

  File anaconda3\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)

  File anaconda3\lib\site-packages\torch\nn\modules\loss.py", line 961, in forward
    return F.cross_entropy(input, target, weight=self.weight,

  File anaconda3\lib\site-packages\torch\nn\functional.py", line 2468, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)

  File anaconda3\lib\site-packages\torch\nn\functional.py", line 2264, in nll_loss
    ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)

RuntimeError: expected scalar type Long but found Double


title: PyTorch's DataParallel is only using one GPU

body:
I'm trying to use two GPU's for training a model in PyTorch. I'm using torch.nn.DataParallel but for some reason nvidia-smi is saying that I'm only using one GPU.
The code is something along the lines of:
>>> import torch.nn as nn
>>> model = SomeModel()
>>> model = nn.DataParallel(model)
>>> model.to('cuda')

When I run the program and observe the output of nvidia-smi, I only see GPU 0 running. Would anybody know what the problem is?


title: Where should we put attention in an autoencoder?

body:
In this tutorial in tensorflow site we can see a code for the implementation of an autoencoder which it's Decoder is as follows:
class Decoder(tf.keras.Model):
  def __init__(self, vocab_size, embedding_dim, dec_units, batch_sz):
    super(Decoder, self).__init__()
    self.batch_sz = batch_sz
    self.dec_units = dec_units
    self.embedding = tf.keras.layers.Embedding(vocab_size, embedding_dim)
    self.gru = tf.keras.layers.GRU(self.dec_units,
                                   return_sequences=True,
                                   return_state=True,
                                   recurrent_initializer='glorot_uniform')
    self.fc = tf.keras.layers.Dense(vocab_size)

    # used for attention
    self.attention = BahdanauAttention(self.dec_units)

  def call(self, x, hidden, enc_output):
    # enc_output shape == (batch_size, max_length, hidden_size)
    context_vector, attention_weights = self.attention(hidden, enc_output)

    # x shape after passing through embedding == (batch_size, 1, embedding_dim)
    x = self.embedding(x)

    # x shape after concatenation == (batch_size, 1, embedding_dim + hidden_size)
    x = tf.concat([tf.expand_dims(context_vector, 1), x], axis=-1)

    # passing the concatenated vector to the GRU
    output, state = self.gru(x)

    # output shape == (batch_size * 1, hidden_size)
    output = tf.reshape(output, (-1, output.shape[2]))

    # output shape == (batch_size, vocab)
    x = self.fc(output)

    return x, state, attention_weights

the BahdanauAttention is applied to the output of the encoder and previous hidden state then it is concated with the lookup of the input and then is fed to GRU.
And yet in another code from this github repository (which is implemented using pytorch) the attention is applied to the output of the GRU:
class DecoderAttn(nn.Module):
    def __init__(self, input_size, hidden_size, output_size, out_bias):
        super(DecoderAttn, self).__init__()
        self.hidden_size = hidden_size
        self.input_size = input_size

        self.embedding = nn.Embedding(input_size, hidden_size)
        self.emb_drop = nn.Dropout(0.2)
        self.gru = nn.GRU(hidden_size, hidden_size, batch_first=True)
        self.gru_drop = nn.Dropout(0.2)
        self.mlp = nn.Linear(hidden_size*2, output_size)
        if out_bias is not None:
            out_bias_tensor = torch.tensor(out_bias, requires_grad=False)
            self.mlp.bias.data[:] = out_bias_tensor
        self.logsoftmax = nn.LogSoftmax(dim=2)

        self.att_mlp = nn.Linear(hidden_size, hidden_size, bias=False)
        self.attn_softmax = nn.Softmax(dim=2)

    def forward(self, input, hidden, encoder_outs):
        emb = self.embedding(input)
        out, hidden = self.gru(self.emb_drop(emb), hidden)

        out_proj = self.att_mlp(out)
        enc_out_perm = encoder_outs.permute(0, 2, 1)
        e_exp = torch.bmm(out_proj, enc_out_perm)
        attn = self.attn_softmax(e_exp)

        ctx = torch.bmm(attn, encoder_outs)

        full_ctx = torch.cat([self.gru_drop(out), ctx], dim=2)

        out = self.mlp(full_ctx)
        out = self.logsoftmax(out)
        return out, hidden, attn

I wonder if the second case is a mistake? If it is not a mistake what is the difference between it and the first decoder? How changing the attention place affect the output?


title: PyTorch CNN linear layer shape after conv2d

body:
I was trying to learn PyTorch and came across a tutorial where a CNN is defined like below,
class Net(Module):
    def __init__(self):
        super(Net, self).__init__()

        self.cnn_layers = Sequential(
            # Defining a 2D convolution layer
            Conv2d(1, 4, kernel_size=3, stride=1, padding=1),
            BatchNorm2d(4),
            ReLU(inplace=True),
            MaxPool2d(kernel_size=2, stride=2),
            # Defining another 2D convolution layer
            Conv2d(4, 4, kernel_size=3, stride=1, padding=1),
            BatchNorm2d(4),
            ReLU(inplace=True),
            MaxPool2d(kernel_size=2, stride=2),
        )

        self.linear_layers = Sequential(
            Linear(4 * 7 * 7, 10)
        )

    # Defining the forward pass
    def forward(self, x):
        x = self.cnn_layers(x)
        x = x.view(x.size(0), -1)
        x = self.linear_layers(x)
        return x

I understood how the cnn_layers are made. After the cnn_layers, the data should be flattened and given to linear_layers.
I don't understand how the number of features to Linear is 4*7*7. I understand that 4 is the output dimension from the last Conv2d layer.
How is 7*7 coming in to picture? Does stride or padding got any role in that?
Input image shape is [1, 28, 28]


title: Implementing a loss function (MSVE) in Reinforcement learning

body:
I am trying to build a temporal difference learning agent for Othello. While the rest of my implementation seems to run as intended I am wondering about the loss function used to train my network. In Sutton's book "Reinforcement learning: An Introduction", the Mean Squared Value Error (MSVE is presented as the standard loss function. It is basically a Mean Square Error multiplied with the on policy distribution. (Sum over all states s ( onPolicyDistribution(s) * [V(s) - V'(s,w)]² ) )

My question is now: How do I obtain this on policy distribution when my policy is an e-greedy function of a learned value function? Is it even necessary and what's the issue if I just use an MSELoss instead?

I'm implementing all of this in pytorch, so bonus points for an easy implementation there :)