title: import torch error in google Colaboratory

body:
I get the following error on google colaboratory.

  ImportErrorTraceback (most recent call last)
   in ()
  ----> 1 import torch

  /usr/local/lib/python3.6/site-packages/torch/init.py in ()
       54 except ImportError:
       55     pass
  ---> 56 from torch._C import *
       57
       58 all += [name for name in dir(_C)

  ImportError: No module named _C

I tried importing from a different directory. But, still get the same error. Any help?


I can't use torch.Tensor() with requires_grad parameter (torch version : 0.4.1)

without requires_grad :

x = torch.Tensor([[.5, .3, 2.1]])
print(x)
> tensor([[0.5000, 0.3000, 2.1000]])

with requires_grad=True or requires_grad=False :

x = torch.Tensor([[.5, .3, 2.1]], requires_grad=False)
print(x)
Traceback (most recent call last):
  File "D:/_P/dev/ai/pytorch/notes/tensor01.py", line 4, in <module>
    x = torch.Tensor([[.5, .3, 2.1]], requires_grad=False)
TypeError: new() received an invalid combination of arguments - got (list, requires_grad=bool), but expected one of:
 * (torch.device device)
 * (torch.Storage storage)
 * (Tensor other)
 * (tuple of ints size, torch.device device)
      didn't match because some of the keywords were incorrect: requires_grad
 * (object data, torch.device device)
      didn't match because some of the keywords were incorrect: requires_grad


title: Pytorch is not found & cannot be installed in pycharm

body:
I need pytorch module to run my project. But When I tried to install it via command prompt via two different ways, it shows an error - .

C:\Users\Toothless>pip install torchvision --user
Collecting torchvision
Using cached https://files.pythonhosted.org/packages/b7/ff /091b4503d5f228bd1120db784e2c071617211b965a8a78018e75750c7199/torchvision-0.3.0-cp37-cp37m-win_amd64.whl
Requirement already satisfied: six in c:\users\toothless\appdata\local\programs\python\python37\lib\site-packages (from torchvision) (1.12.0)
Collecting pillow>=4.1.1 (from torchvision)
Downloading https://files.pythonhosted.org/packages/ae/96/6f83deebfcd20a5d4ad35e4e989814a16559d8715741457e670aae1a5a09/Pillow-6.1.0-cp37-cp37m-win_amd64.whl (2.0MB)
 |████████████████████████████████| 2.0MB 27kB/s
Requirement already satisfied: numpy in c:\users\toothless\appdata\local\programs\python\python37\lib\site-packages (from torchvision) (1.17.0)
 Collecting torch>=1.1.0 (from torchvision)
 ERROR: Could not find a version that satisfies the requirement torch>=1.1.0 (from torchvision) (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)
 ERROR: No matching distribution found for torch>=1.1.0 (from torchvision)

 C:\Users\Toothless>pip3 install torchvision
 Collecting torchvision
 Using cached https://files.pythonhosted.org/packages/b7/ff/091b4503d5f228bd1120db784e2c071617211b965a8a78018e75750c7199/torchvision-0.3.0-cp37-cp37m-win_amd64.whl
 Requirement already satisfied: six in c:\users\toothless\appdata\local\programs\python\python37\lib\site-packages (from torchvision) (1.12.0)
 Collecting torch>=1.1.0 (from torchvision)
 ERROR: Could not find a version that satisfies the requirement torch>=1.1.0 (from torchvision) (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)
 ERROR: No matching distribution found for torch>=1.1.0 (from torchvision)

Again I've tried to installed through pycharm .

I've been following this question but conda`` is not listed in my envairoment variable.

Edited: When I tried pip install torch torchvision --user this command it gives an error - ERROR: torchvision 0.3.0 has requirement torch>=1.1.0, but you'll have torch 0.1.2.post2 which is incompatible.

Edited: I've also tried pip install torchvision but it shows with error -

C:\Users\Toothless>pip install torchvision
Collecting torchvision
Using cached https://files.pythonhosted.org/packages/b7/ff/091b4503d5f228bd1120db784e2c071617211b965a8a78018e75750c7199/torchvision-0.3.0-cp37-cp37m-win_amd64.whl
Collecting torch>=1.1.0 (from torchvision)
ERROR: Could not find a version that satisfies the requirement torch>=1.1.0 (from torchvision) (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)
ERROR: No matching distribution found for torch>=1.1.0 (from torchvision)

What should i do now to make a workable pytorch?


title: Pytorch DataLoader memory is not released

body:
I'd like to implement SRGAN on pythorch on google collaboratory, but memory of DataLoader seems to be released, so if you turn epoch, memory error will occur.
It would be greatly appreciated if you tell me how to do it in order to free up memory per batch.
This is the github link of the code
https://github.com/pacifinapacific/Hello-World/blob/master/Untitled0.ipynb

It turned 48 and a memory error occurred on 1 echoch,
If you set the batch size to 1/6 of 8, you will get an error at about 6 epoch.

I am reading high resolution and low resolution images with the following code. Extend ImageFolder

but For example, even if an error occurs when learning is executed, the memory of the GPU is not released

class DownSizePairImageFolder(ImageFolder):
def __init__(self, root, transform=None, large_size=256, small_size=64, **kwds):
    super().__init__(root, transform=transform, **kwds)
    self.large_resizer = transforms.Scale(large_size)
    self.small_resizer = transforms.Scale(small_size)

def __getitem__(self, index):
    path, _ = self.imgs[index]
    img = self.loader(path)
    large_img = self.large_resizer(img)
    small_img = self.small_resizer(img)
    if self.transform is not None:
        large_img = self.transform(large_img)
        small_img = self.transform(small_img)
    return small_img, large_img

train_data = DownSizePairImageFolder('./lfw-deepfunneled/train',   transform=transforms.ToTensor())
test_data = DownSizePairImageFolder('./lfw-deepfunneled/test',    transform=transforms.ToTensor())
batch_size = 8
train_loader = DataLoader(train_data, batch_size, shuffle=True)
test_loader = DataLoader(test_data, batch_size, shuffle=False)


title: Trying to understand cross_entropy loss in PyTorch

body:
This is a very newbie question but I'm trying to wrap my head around cross_entropy loss in Torch so I created the following code:

x = torch.FloatTensor([
                        [1.,0.,0.]
                       ,[0.,1.,0.]
                       ,[0.,0.,1.]
                       ])

print(x.argmax(dim=1))

y = torch.LongTensor([0,1,2])
loss = torch.nn.functional.cross_entropy(x, y)

print(loss)

which outputs the following:

tensor([0, 1, 2])
tensor(0.5514)

What I don't understand is given my input matches the expected output why is the loss not 0?


title: How to save and load random number generator state in Pytorch?

body:
I am training a DL model in Pytorch, and want to train my model in a deterministic way.
As written in this official guide, I set random seeds like this:

np.random.seed(0)
torch.manual_seed(0)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False

Now, my training is long and i want to save, then later load everything, including the RNGs. I use torch.save and torch.load_state_dict for the model and the optimizer.

How can the random number generators be saved & loaded?


title: What is meant by the_model = TheModelClass(*args, **kwargs)?

body:
I am using PyTorch to do image classification. After training the most, I want to save the trained model.

I don't understand what is meant by

the_model = TheModelClass(*args, **kwargs)

This line of code is given by PyTorch website (https://pytorch.org/docs/master/notes/serialization.html).


title: How can I slice a PyTorch tensor with another tensor?

body:
I have:

inp =  torch.randn(4, 1040, 161)

and I have another tensor called indices with values:

tensor([[124, 583, 158, 529],
        [172, 631, 206, 577]], device='cuda:0')

I want the equivalent of:

inp0 = inp[:,124:172,:]
inp1 = inp[:,583:631,:]
inp2 = inp[:,158:206,:]
inp3 = inp[:,529:577,:]

Except all added together, to have a .size of [4, 48, 161]. How can I accomplish this?

Currently, my solution is a for loop:

            left_indices = torch.empty(inp.size(0), self.side_length, inp.size(2))
            for batch_index in range(len(inp)):
                print(left_indices_start[batch_index].item())
                left_indices[batch_index] = inp[batch_index, left_indices_start[batch_index].item():left_indices_end[batch_index].item()]


title: Tracing back deprecated warning in pytorch

body:
I am training yolov3 on my data using this code here :
https://github.com/cfotache/pytorch_custom_yolo_training/

But I am getting this annoying deprecation warnings

Warning: indexing with dtype torch.uint8 is now deprecated, please use a dtype torch.bool instead. (expandTensors at /pytorch/aten/src/ATen/native/IndexingUtils.h:20)

I tried using python3 -W ignore train.py
I tried adding :

import warnings
warnings.filterwarnings('ignore')

but the warning is still persistent.

I found this piece of code on here on stackoverflow that prints that stack on warnings ,

import traceback
import warnings
import sys

def warn_with_traceback(message, category, filename, lineno, file=None, line=None):

    log = file if hasattr(file,'write') else sys.stderr
    traceback.print_stack(file=log)
    log.write(warnings.formatwarning(message, category, filename, lineno, line))

warnings.showwarning = warn_with_traceback

and here's what I get :

  File "/content/pytorch_custom_yolo_training/train.py", line 102, in <module>
   loss = model(imgs, targets)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/content/pytorch_custom_yolo_training/models.py", line 267, in forward
    x, *losses = module[0](x, targets)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/content/pytorch_custom_yolo_training/models.py", line 203, in forward
    loss_x = self.mse_loss(x[mask], tx[mask])
  File "/usr/lib/python3.6/warnings.py", line 99, in _showwarnmsg
    msg.file, msg.line)
  File "/content/pytorch_custom_yolo_training/train.py", line 29, in warn_with_traceback
    traceback.print_stack(file=log)
  /pytorch/aten/src/ATen/native/IndexingUtils.h:20: UserWarning: indexing with dtype torch.uint8 is now deprecated, please use a dtype torch.bool instead.

Going to the files and functions mentioned in the stack , I don't find any uint8 .
What can I to solve the problem or even stop getting these warnings ?


title: Need help understand this implementation of ConvLSTM code in pytorch

body:
I am having issue understand the following implementation of ConvLSTM. I don't really understand what input_size + hidden_size is? Also the 4 * hidden_size value for output? model = ConvLSTMCell(c, d) tells us that c and d are input_size and hidden_size which are 3 and 5 respectively. I assume c is channel and d stands for output dimension? Also, why are we multiplying hidden_size by 4 for output? Is 4 the default 4 gates of the LSTM cell? Can some one more experience explain to me what is going on within the convolution? Thank you.

self.Gates = nn.Conv2d(input_size + hidden_size, 4 * hidden_size,
KERNEL_SIZE, padding=PADDING)

import torch
from torch import nn
import torch.nn.functional as f
from torch.autograd import Variable

# Define some constants
KERNEL_SIZE = 3
PADDING = KERNEL_SIZE // 2

class ConvLSTMCell(nn.Module):
    """
    Generate a convolutional LSTM cell
    """

    def __init__(self, input_size, hidden_size):
        super().__init__()
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.Gates = nn.Conv2d(input_size + hidden_size, 4 * hidden_size, KERNEL_SIZE, padding=PADDING)

    def forward(self, input_, prev_state):

        # get batch and spatial sizes
        batch_size = input_.data.size()[0]
        spatial_size = input_.data.size()[2:]

        # generate empty prev_state, if None is provided
        if prev_state is None:
            state_size = [batch_size, self.hidden_size] + list(spatial_size)
            prev_state = (
                Variable(torch.zeros(state_size)),
                Variable(torch.zeros(state_size))
            )

        prev_hidden, prev_cell = prev_state

        # data size is [batch, channel, height, width]
        stacked_inputs = torch.cat((input_, prev_hidden), 1)
        gates = self.Gates(stacked_inputs)

        # chunk across channel dimension
        in_gate, remember_gate, out_gate, cell_gate = gates.chunk(4, 1)

        # apply sigmoid non linearity
        in_gate = f.sigmoid(in_gate)
        remember_gate = f.sigmoid(remember_gate)
        out_gate = f.sigmoid(out_gate)

        # apply tanh non linearity
        cell_gate = f.tanh(cell_gate)

        # compute current cell and hidden state
        cell = (remember_gate * prev_cell) + (in_gate * cell_gate)
        hidden = out_gate * f.tanh(cell)

        return hidden, cell

def _main():
    """
    Run some basic tests on the API
    """

    # define batch_size, channels, height, width
    b, c, h, w = 1, 3, 4, 8
    d = 5           # hidden state size
    lr = 1e-1       # learning rate
    T = 6           # sequence length
    max_epoch = 20  # number of epochs

    # set manual seed
    torch.manual_seed(0)

    print('Instantiate model')
    model = ConvLSTMCell(c, d)
    print(repr(model))

    print('Create input and target Variables')
    x = Variable(torch.rand(T, b, c, h, w))
    y = Variable(torch.randn(T, b, d, h, w))

    print('Create a MSE criterion')
    loss_fn = nn.MSELoss()

    print('Run for', max_epoch, 'iterations')
    for epoch in range(0, max_epoch):
        state = None
        loss = 0
        for t in range(0, T):
            state = model(x[t], state)
            loss += loss_fn(state[0], y[t])

        print(' > Epoch {:2d} loss: {:.3f}'.format((epoch+1), loss.data[0]))

        # zero grad parameters
        model.zero_grad()

        # compute new grad parameters through time!
        loss.backward()

        # learning_rate step against the gradient
        for p in model.parameters():
            p.data.sub_(p.grad.data * lr)

    print('Input size:', list(x.data.size()))
    print('Target size:', list(y.data.size()))
    print('Last hidden state size:', list(state[0].size()))

if __name__ == '__main__':
    _main()

__author__ = "Alfredo Canziani"
__credits__ = ["Alfredo Canziani"]
__maintainer__ = "Alfredo Canziani"
__email__ = "alfredo.canziani@gmail.com"
__status__ = "Prototype"  # "Prototype", "Development", or "Production"
__date__ = "Jan 17"


title: TensorboardX: tensorboard: command not found

body:
so I am attempting to access the tensorboard visualizations from this project. I am not to familiar with the library, but when I try to access the tensorboard during training using the command

tensorboard --dir data

I get returned with the tensor board command not found. (Running on Ubuntu 18.04). pip list displays that I have tensorboardX 1.9 downloaded, so I am not sure why the command is not being recognized. I have pytorch installed, and I am not aware of any other dependencies that are needed.


title: PyTorch: How to get around the RuntimeError: in-place operations can be only used on variables that don't share storage with any other variables

body:
With PyTorch I'm having a problem doing an operation with two Variables:

sub_patch  : [torch.FloatTensor of size 9x9x32]

pred_patch : [torch.FloatTensor of size 5x5x32]

sub_patch is a Variable made by torch.zeros
pred_patch is a Variable of which I index each of the 25 nodes with a nested for-loop, and that I multiply with its corresponding unique filter (sub_filt_patch) of size [5,5,32]. The result is added to its respective place in sub_patch.

This is a piece of my code:

for i in range(filter_sz):
    for j in range(filter_sz):

        # index correct filter from filter tensor
        sub_filt_col = (patch_col + j) * filter_sz
        sub_filt_row = (patch_row + i) * filter_sz

        sub_filt_patch = sub_filt[sub_filt_row:(sub_filt_row + filter_sz), sub_filt_col:(sub_filt_col+filter_sz), :]

        # multiply filter and pred_patch and sum onto sub patch
        sub_patch[i:(i + filter_sz), j:(j + filter_sz), :] += (sub_filt_patch * pred_patch[i,j]).sum(dim=3)

The error I get from the bottom line of the piece of code here is

RuntimeError: in-place operations can be only used on variables that don't share storage with any other variables, but detected that there are 2 objects sharing it

I get why it happens, since sub_patch is a Variable, and pred_patch is a Variable too, but how can I get around this error? Any help would be greatly appreciated!

Thank you!


title: Pytorch - how to undersample using weightedrandomsampler

body:
I have an unbalanced dataset and would like to undersample the class that is overrepresented.How do I go about it. I would like to use to weightedrandomsampler but I am also open to other suggestions.

So far I am assuming that my code will have to be structured kind of like the following. But I dont know how to exaclty do it.

trainset = datasets.ImageFolder(path_train,transform=transform)
...
sampler = data.WeightedRandomSampler(weights=..., num_samples=..., replacement=...)
...
trainloader = data.DataLoader(trainset, batchsize = batchsize, sampler=sampler)

I hope someone can help. Thanks a lot


title: Unable to save Pytorch model to Google Drive in Google Colab?

body:
I am trying to save my model to my drive on google colab. I have used the following code to mount my Google Drive-

from google.colab import drive
drive.mount('/content/gdrive')

After all the preprocessing, model definition and training, I want to save my model to the drive because training it will take a long time. So, I will save it to drive at regular intervals and reload from that point to continue.
The code to save my model is:

def save_model(model, model_name, iter):
  path = f'content/gdrive/My Drive/Machine Learning Models/kaggle_jigsaw_{model_name}_iter_{iter}.pth'
  print(f'Saving {model_name} model...')
  torch.save(model.state_dict(), path)
  print(f'{model_name} saved successfully.')

EMBEDDING_DIMS = 128
HIDDEN_SIZE = 256

gru = GRU(vocab.n_words, EMBEDDING_DIMS, HIDDEN_SIZE, 2).to(device)
save_model(gru, 'gru', 0)

I am getting the following error:

Saving gru model...
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-27-d2510611a9d4> in <module>()
      9
     10 gru = GRU(vocab.n_words, EMBEDDING_DIMS, HIDDEN_SIZE, 2).to(device)
---> 11 save_model(gru, 'gru', 0)

<ipython-input-27-d2510611a9d4> in save_model(model, model_name, iter)
      2   path = f'content/gdrive/My Drive/Machine Learning Models/kaggle_jigsaw_{model_name}_iter_{iter}.pth'
      3   print(f'Saving {model_name} model...')
----> 4   torch.save(model.state_dict(), path)
      5   print(f'{model_name} saved successfully.')
      6

/usr/local/lib/python3.6/dist-packages/torch/serialization.py in save(obj, f, pickle_module, pickle_protocol)
    217         >>> torch.save(x, buffer)
    218     """
--> 219     return _with_file_like(f, "wb", lambda f: _save(obj, f, pickle_module, pickle_protocol))
    220
    221

/usr/local/lib/python3.6/dist-packages/torch/serialization.py in _with_file_like(f, mode, body)
    140             (sys.version_info[0] == 3 and isinstance(f, pathlib.Path)):
    141         new_fd = True
--> 142         f = open(f, mode)
    143     try:
    144         return body(f)

FileNotFoundError: [Errno 2] No such file or directory: 'content/gdrive/My Drive/Machine Learning Models/kaggle_jigsaw_gru_iter_0.pth'

I have manually created the folder in my drive and only the file needs to be created. Still, the error persists. Though, I am sure that manually creating the folder was not required. The problem is something else.
Where am I going wrong?


title: Calculating Variance in Pytorch on Spatial axis

body:
I am trying to calculate variance in Pytorch but unable to do on multiple axis.

I have similar thing done in Tensorflow but unable to do it on Pytorch as torch.var function takes int as dimension instead of axes.
Below code is channel last code, I expect axes=[2,3]

Lambda(lambda x: tf.nn.moments(x, axes=[1, 2]))

For example, if input_dims = (5, 10, 25, 25) then output_dims should be (5,10, 1, 1).


title: LSTM in Pytorch: how to add/change sequence length dimension?

body:
I am running LSTM in pytorch but as I understand, it is only taking sequence length = 1. When I reshape to have sequence length to 4 or other number, then I get an error of mismatching length in input and target. If I reshape both input and target, then the model complains that it does not accept multi-target labels.

My train dataset has 66512 rows and 16839 columns, 3 categories/classes in the target. I would like to use a batch size 200 and a sequence length of 4, i.e. use 4 rows of data in a sequence.

Please advise how to adjust my model and/or data to be able to run model for various sequence lengths (e.g., 4).

batch_size=200
import torch
from torch.utils.data import TensorDataset
from torch.utils.data import DataLoader
train_target = torch.tensor(train_data[['Label1','Label2','Label3']].values.astype(np.float32))
train_target = np.argmax(train_target, axis=1)
train = torch.tensor(train_data.drop(['Label1','Label2','Label3'], axis = 1).values.astype(np.float32))
train_tensor = TensorDataset(train.unsqueeze(1), train_target)
train_loader = DataLoader(dataset = train_tensor, batch_size = batch_size, shuffle = True)

print(train.shape)
print(train_target.shape)

torch.Size([66512, 16839])
torch.Size([66512])

import torch.nn as nn

class LSTMModel(nn.Module):
    def __init__(self, input_dim, hidden_dim, layer_dim, output_dim):
        super(LSTMModel, self).__init__()
        # Hidden dimensions
        self.hidden_dim = hidden_dim

        # Number of hidden layers
        self.layer_dim = layer_dim

        # Building LSTM
        self.lstm = nn.LSTM(input_dim, hidden_dim, layer_dim, batch_first=True)

        # Readout layer
        self.fc = nn.Linear(hidden_dim, output_dim)

    def forward(self, x):

        # Initialize hidden state with zeros
        h0 = torch.zeros(self.layer_dim, x.size(0), self.hidden_dim).requires_grad_().to(device)

        # Initialize cell state
        c0 = torch.zeros(self.layer_dim, x.size(0), self.hidden_dim).requires_grad_().to(device)

        out, (hn, cn) = self.lstm(x, (h0,c0))

        # Index hidden state of last time step
        out = self.fc(out[:, -1, :])

        return out

input_dim = 16839
hidden_dim = 100
output_dim = 3
layer_dim = 1

batch_size = batch_size
num_epochs = 1

model = LSTMModel(input_dim, hidden_dim, layer_dim, output_dim)

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
model.to(device)

criterion = nn.CrossEntropyLoss()
learning_rate = 0.1

optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

print(len(list(model.parameters())))
for i in range(len(list(model.parameters()))):
    print(list(model.parameters())[i].size())

6
torch.Size([400, 16839])
torch.Size([400, 100])
torch.Size([400])
torch.Size([400])
torch.Size([3, 100])
torch.Size([3])

for epoch in range(num_epochs):
    for i, (train, train_target) in enumerate(train_loader):
        # Load data as a torch tensor with gradient accumulation abilities
        train = train.requires_grad_().to(device)
        train_target = train_target.to(device)

        # Clear gradients w.r.t. parameters
        optimizer.zero_grad()

        # Forward pass to get output/logits
        outputs = model(train)

        # Calculate Loss: softmax --> cross entropy loss
        loss = criterion(outputs, train_target)

        # Getting gradients w.r.t. parameters
        loss.backward()

        # Updating parameters
        optimizer.step()
print('Epoch: {}. Loss: {}. Accuracy: {}'.format(epoch, np.around(loss.item(), 4), np.around(accuracy,4)))


title: How to use GPU in pytorch?

body:
I tried following steps at: https://pytorch.org/get-started/locally/

First I created a conda environment as:

conda create -n facenet37_2 python=3.7

Then on above site I selected:

PyTorch Build: Stable (1.4)
OS: Linux (I am using Ubuntu 18.04)
Package: conda
Language: python
CUDA: 10.1

and it asked me to run following command:

conda install pytorch torchvision cudatoolkit=10.1 -c pytorch

But after that when I opened python and typed:

import torch
torch.cuda.is_available()

I get False

I have GeForce GT 630M (computeCapability: 2.1). But it is not getting detected. Why? Is it too old and no longer supported? How can I fix the issue?

Edit: Why did I get a negative vote?


title: Faster way to do multiple embeddings in PyTorch?

body:
I'm working on a torch-based library for building autoencoders with tabular datasets.

One big feature is learning embeddings for categorical features.

In practice, however, training many embedding layers simultaneously is creating some slowdowns. I am using for-loops to do this and running the for-loop on each iteration is (I think) what's causing the slowdowns.

When building the model, I associate embedding layers with each categorical feature in the user's dataset:

        for ft in self.categorical_fts:
            feature = self.categorical_fts[ft]
            n_cats = len(feature['cats']) + 1
            embed_dim = compute_embedding_size(n_cats)
            embed_layer = torch.nn.Embedding(n_cats, embed_dim)
            feature['embedding'] = embed_layer

Then, with a call to .forward():

        embeddings = []
        for i, ft in enumerate(self.categorical_fts):
            feature = self.categorical_fts[ft]
            emb = feature['embedding'](codes[i])
            embeddings.append(emb)

        #num and bin are numeric and binary features
        x = torch.cat(num + bin + embeddings, dim=1)

Then x goes into dense layers.

This gets the job done but running this for loop during each forward pass really slows down training, especially when a dataset has tens or hundreds of categorical columns.

Does anybody know of a way of vectorizing something like this? Thanks!

UPDATE:
For more clarity, I made this sketch of how I'm feeding categorical features into the network. You can see that each categorical column has its own embedding matrix, while numeric features are concatenated directly to their output before being passed into the feed-forward network.

Can we do this without iterating through each embedding matrix?


title: pytorch restore model with different batch size

body:
I have a problem about how to reload pytorch model with different batch size. In training, my batch size is 64, but in inference, I would like the batch size is 1(feed data one by one). This is the code I used to save and restore model:

torch.save(agent.qnetwork_local.state_dict(), './ckpt/checkpoint.pth')
saved_model = QNetwork(state_size=37, action_size=4, seed=0)
saved_model.load_state_dict(torch.load('./ckpt/checkpoint.pth'))

And I got this error when running the inference model:

RuntimeError: size mismatch, m1: [37 x 1], m2: [37 x 64] at /Users/soumith/code/builder/wheel/pytorch-src/aten/src/TH/generic/THTensorMath.cpp:2070

This error means the model's input must be 37x64, where 37 is the data dimension and 64 is the training batch size. But testing input is 37x1 which means data dimension is 37 and batch size is 1.

Is there any solution to different batch size in reload pytorch model?
Thank you very much.


title: Resize Vs CenterCrop Vs RandomResizedCrop Vs RandomCrop

body:
Can anyone tell me in which situations the above functions are used and how they affect the image size?
I want to resize the Cat V Dogs images and i am a bit confuse about how to use them.


title: Pytorch: How to access tensor(values) by tensor(keys) in python dictionary

body:
I have a dictionary with tensor keys and tensor values. I want to access the values by the keys.

from torch import tensor
x = {tensor(0): [tensor(1)], tensor(1): [tensor(0)]}
for i in x.keys():
  print(i, x[i])

Returns:

tensor(0) [tensor(1)]
tensor(1) [tensor(0)]

But when i try to access the values without looping through the keys,

try:
    print(x[tensor(0)])

except:
    print(Exception)
    print(x[0])

Throws Exception:

 KeyError                                   Traceback (most recent call last)
 <ipython-input-34-746d28dcd450> in <module>()
  6 try:
  ----> 7   print(x[tensor(0)])
  8

  KeyError: tensor(0)

  During handling of the above exception, another exception occurred:

  KeyError                                  Traceback (most recent call last)
  <ipython-input-34-746d28dcd450> in <module>()
  9 except:
  10   print(Exception)
  ---> 11   print(x[0])
  12 continue

  KeyError: 0


title: Unet input and output

body:
I am totally new to computer vision, and am trying to train a model that takes an normal 3 channel image (RGB) and gives an output, which is just a simple binary mask. From my best knowledge, architecture called Unet does something like this. For example, TernausnetV2 worked quite well in a kaggle competition. In this kind of model architecture, is the input and output I'm thinking of correct ?

I have 3 channel image (RGB) and the binary masked image (which is same size as the 3 channel image, and all the values are 1 or 0).
Thanks for the help in prior.


title: RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False, Dataloader Error, and setting pin_memory=False

body:
I am a beginner trying to evaluate this video object segmentation network paper.

When following the instructions on https://github.com/seoungwugoh/STM

It says the requirements are as follows:-

python 3.6
pytorch 1.0.1.post2
numpy, opencv, pillow

I couldn't get this pytorch version to install, so I installed the conda-forge pytorch version 1.5.

and I run this command in either Windows 10 or Ubuntu 16.04 using Anaconda

(STMVOS) oneworld@oneworld:~/Documents/VideoObjectSegmentation/STMVOS$ python eval_DAVIS.py -g '1' -s val -y 16 -D ../DAVISSemiSupervisedTrainVal480

after doing pip install matplotlib, and pip install tqdm ...

I get the following error message:-

Space-time Memory Networks: initialized.
STM : Testing on DAVIS
Loading weights: STM_weights.pth
Traceback (most recent call last):

File "eval_DAVIS.py", line 111, in
  model.load_state_dict(torch.load(pth_path))

File "/home/oneworld/anaconda3/envs/STMVOS/lib/python3.8/site-packages/torch/serialization.py", line 593, in load
  return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)

File "/home/oneworld/anaconda3/envs/STMVOS/lib/python3.8/site-packages/torch/serialization.py", line 773, in _legacy_load
  result = unpickler.load()

File "/home/oneworld/anaconda3/envs/STMVOS/lib/python3.8/site-packages/torch/serialization.py", line 729, in persistent_load

deserialized_objects[root_key] = restore_location(obj, location)

File "/home/oneworld/anaconda3/envs/STMVOS/lib/python3.8/site-packages/torch/serialization.py", line 178, in default_restore_location
  result = fn(storage, location)

File "/home/oneworld/anaconda3/envs/STMVOS/lib/python3.8/site-packages/torch/serialization.py", line 154, in _cuda_deserialize
    device = validate_cuda_device(location)

File "/home/oneworld/anaconda3/envs/STMVOS/lib/python3.8/site-packages/torch/serialization.py", line 138, in validate_cuda_device raise RuntimeError('Attempting to deserialize object on a CUDA '

RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU

My Graphics Card Driver, and System and Packages are as follows:-

(STMVOS) oneworld@oneworld:~/Documents/VideoObjectSegmentation/STMVOS$ nvidia-smi

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.64.00    Driver Version: 440.64.00    CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 1070    Off  | 00000000:01:00.0  On |                  N/A |
| 26%   34C    P8    10W / 151W |    392MiB /  8118MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      1247      G   /usr/lib/xorg/Xorg                           229MiB |
|    0      2239      G   compiz                                       126MiB |
|    0      9385      G   /usr/lib/firefox/firefox                       2MiB |
|    0     11686      G   /proc/self/exe                                30MiB |
+-----------------------------------------------------------------------------+

I also tried this

(STMVOS) oneworld@oneworld:~/Documents/VideoObjectSegmentation/STMVOS$ python -c 'import torch; print(torch.rand(2,3).cuda())'

tensor([[0.9178, 0.8239, 0.4761],
        [0.9429, 0.8877, 0.0097]], device='cuda:0')

Which shows that cuda is working here

(STMVOS) oneworld@oneworld:~/Documents/VideoObjectSegmentation/STMVOS$ conda info
    active environment : STMVOS
    active env location : /home/oneworld/anaconda3/envs/STMVOS
            shell level : 1
       user config file : /home/oneworld/.condarc
 populated config files :
          conda version : 4.8.2
    conda-build version : 3.18.11
         python version : 3.7.6.final.0
       virtual packages : __cuda=10.2
                          __glibc=2.23
       base environment : /home/oneworld/anaconda3  (writable)
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /home/oneworld/anaconda3/pkgs
                          /home/oneworld/.conda/pkgs
       envs directories : /home/oneworld/anaconda3/envs
                          /home/oneworld/.conda/envs
               platform : linux-64
             user-agent : conda/4.8.2 requests/2.22.0 CPython/3.7.6 Linux/4.4.0-179-generic ubuntu/16.04.6 glibc/2.23
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False

(STMVOS) oneworld@oneworld:~/Documents/VideoObjectSegmentation/STMVOS$ conda list

packages in environment at /home/oneworld/anaconda3/envs/STMVOS:

Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main
blas                      1.0                         mkl
bzip2                     1.0.8                h516909a_2    conda-forge
ca-certificates           2020.4.5.1           hecc5488_0    conda-forge
cairo                     1.16.0            hcf35c78_1003    conda-forge
certifi                   2020.4.5.1               py38_0
cudatoolkit               10.2.89              hfd86e86_1
cycler                    0.10.0                   pypi_0    pypi
dbus                      1.13.6               he372182_0    conda-forge
expat                     2.2.9                he1b5a44_2    conda-forge
ffmpeg                    4.2.3                h167e202_0    conda-forge
fontconfig                2.13.1            h86ecdb6_1001    conda-forge
freetype                  2.9.1                h8a8886c_1
gettext                   0.19.8.1          hc5be6a0_1002    conda-forge
giflib                    5.2.1                h516909a_2    conda-forge
glib                      2.64.3               h6f030ca_0    conda-forge
gmp                       6.2.0                he1b5a44_2    conda-forge
gnutls                    3.6.5             hd3a4fd2_1002    conda-forge
graphite2                 1.3.13            he1b5a44_1001    conda-forge
gst-plugins-base          1.14.5               h0935bb2_2    conda-forge
gstreamer                 1.14.5               h36ae1b5_2    conda-forge
harfbuzz                  2.4.0                h9f30f68_3    conda-forge
hdf5                      1.10.6          nompi_h3c11f04_100    conda-forge
icu                       64.2                 he1b5a44_1    conda-forge
intel-openmp              2020.1                      217
jasper                    1.900.1           h07fcdf6_1006    conda-forge
jpeg                      9c                h14c3975_1001    conda-forge
kiwisolver                1.2.0                    pypi_0    pypi
lame                      3.100             h14c3975_1001    conda-forge
ld_impl_linux-64          2.33.1               h53a641e_7
libblas                   3.8.0                    15_mkl    conda-forge
libcblas                  3.8.0                    15_mkl    conda-forge
libclang                  9.0.1           default_hde54327_0    conda-forge
libedit                   3.1.20181209         hc058e9b_0
libffi                    3.2.1             he1b5a44_1007    conda-forge
libgcc-ng                 9.1.0                hdf63c60_0
libgfortran-ng            7.3.0                hdf63c60_0
libiconv                  1.15              h516909a_1006    conda-forge
liblapack                 3.8.0                    15_mkl    conda-forge
liblapacke                3.8.0                    15_mkl    conda-forge
libllvm9                  9.0.1                he513fc3_1    conda-forge
libopencv                 4.2.0                    py38_6    conda-forge
libpng                    1.6.37               hbc83047_0
libstdcxx-ng              9.1.0                hdf63c60_0
libtiff                   4.1.0                h2733197_0
libuuid                   2.32.1            h14c3975_1000    conda-forge
libwebp                   1.0.2                h56121f0_5    conda-forge
libxcb                    1.13              h14c3975_1002    conda-forge
libxkbcommon              0.10.0               he1b5a44_0    conda-forge
libxml2                   2.9.10               hee79883_0    conda-forge
matplotlib                3.2.1                    pypi_0    pypi
mkl                       2020.1                      217
mkl-service               2.3.0            py38he904b0f_0
mkl_fft                   1.0.15           py38ha843d7b_0
mkl_random                1.1.1            py38h0573a6f_0
ncurses                   6.2                  he6710b0_1
nettle                    3.4.1             h1bed415_1002    conda-forge
ninja                     1.9.0            py38hfd86e86_0
nspr                      4.25                 he1b5a44_0    conda-forge
nss                       3.47                 he751ad9_0    conda-forge
numpy                     1.18.1           py38h4f9e942_0
numpy-base                1.18.1           py38hde5b4d6_1
olefile                   0.46                       py_0
opencv                    4.2.0                    py38_6    conda-forge
openh264                  2.1.1                h8b12597_0    conda-forge
openssl                   1.1.1g               h516909a_0    conda-forge
pcre                      8.44                 he1b5a44_0    conda-forge
pillow                    7.1.2            py38hb39fc2d_0
pip                       20.0.2                   py38_3
pixman                    0.38.0            h516909a_1003    conda-forge
pthread-stubs             0.4               h14c3975_1001    conda-forge
py-opencv                 4.2.0            py38h23f93f0_6    conda-forge
pyparsing                 2.4.7                    pypi_0    pypi
python                    3.8.1                h0371630_1
python-dateutil           2.8.1                    pypi_0    pypi
python_abi                3.8                      1_cp38    conda-forge
pytorch                   1.5.0           py3.8_cuda10.2.89_cudnn7.6.5_0    pytorch
qt                        5.12.5               hd8c4c69_1    conda-forge
readline                  7.0                  h7b6447c_5
setuptools                46.4.0                   py38_0
six                       1.14.0                   py38_0
sqlite                    3.31.1               h62c20be_1
tk                        8.6.8                hbc83047_0
torchvision               0.6.0                py38_cu102    pytorch
tqdm                      4.46.0                   pypi_0    pypi
wheel                     0.34.2                   py38_0
x264                      1!152.20180806       h14c3975_0    conda-forge
xorg-kbproto              1.0.7             h14c3975_1002    conda-forge
xorg-libice               1.0.10               h516909a_0    conda-forge
xorg-libsm                1.2.3             h84519dc_1000    conda-forge
xorg-libx11               1.6.9                h516909a_0    conda-forge
xorg-libxau               1.0.9                h14c3975_0    conda-forge
xorg-libxdmcp             1.1.3                h516909a_0    conda-forge
xorg-libxext              1.3.4                h516909a_0    conda-forge
xorg-libxrender           0.9.10            h516909a_1002    conda-forge
xorg-renderproto          0.11.1            h14c3975_1002    conda-forge
xorg-xextproto            7.3.0             h14c3975_1002    conda-forge
xorg-xproto               7.0.31            h14c3975_1007    conda-forge
xz                        5.2.5                h7b6447c_0
zlib                      1.2.11               h7b6447c_3
zstd                      1.3.7                h0b5b093_0

The code it gets stuck on in eval_DAVIS.py is as follows:-

print('Loading weights:', pth_path)
model.load_state_dict(torch.load(pth_path))

I am using Ubuntu 16.04, however I tried a similar setup in windows 10 and received the same error messages.

Any help much appreciated.

Kind regards

OneWorld


title: Save only best weights with huggingface transformers

body: Currently, I'm building a new transformer-based model with huggingface-transformers, where attention layer is different from the original one. I used run_glue.py to check performance of my model on GLUE benchmark. However, I found that Trainer class of huggingface-transformers saves all the checkpoints that I set, where I can set the maximum number of checkpoints to save. However, I want to save only the weight (or other stuff like optimizers) with best performance on validation dataset, and current Trainer class doesn't seem to provide such thing. (If we set the maximum number of checkpoints, then it removes older checkpoints, not ones with worse performances). Someone already asked about same question on Github, but I can't figure out how to modify the script and do what I want. Currently, I'm thinking about making a custom Trainer class that inherits original one and change the train() method, and it would be great if there's an easy and simple way to do this. Thanks in advance.


title: How to fix 'The kernel appears to have died. It will restart automatically" caused by pytorch

body:
I have a strange problem with Pytorch. When i use something torch functions with tensors like tensor.rehsape  or torch.transpose, i don't have problem and all ok, even when i've created network all ok, but when i wanna train network my jupyter crashed.

.

i find where error,but i don't know why it is there and how it fix.

I installed pytorch using conda. I have Ubuntu 18.04.
I don't have cuda.


title: AttributeError: module 'enum' has no attribute 'IntFlag' - fastai install

body:
When I try install fastai (pip install fastai) i get following error:

AttributeError: module 'enum' has no attribute 'IntFlag'

Full traceback:

 Installing build dependencies ... error
  ERROR: Complete output from command /usr/bin/python3 /usr/local/lib/python3.6/dist-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-b8vreosn/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools wheel 'numpy==1.13.3; python_version=='"'"'2.7'"'"' and platform_system!='"'"'AIX'"'"'' 'numpy==1.13.3; python_version=='"'"'3.5'"'"' and platform_system!='"'"'AIX'"'"'' 'numpy==1.13.3; python_version=='"'"'3.6'"'"' and platform_system!='"'"'AIX'"'"'' 'numpy==1.14.5; python_version>='"'"'3.7'"'"' and platform_system!='"'"'AIX'"'"'' 'numpy==1.16.0; python_version=='"'"'2.7'"'"' and platform_system=='"'"'AIX'"'"'' 'numpy==1.16.0; python_version=='"'"'3.5'"'"' and platform_system=='"'"'AIX'"'"'' 'numpy==1.16.0; python_version=='"'"'3.6'"'"' and platform_system=='"'"'AIX'"'"'' 'numpy==1.16.0; python_version>='"'"'3.7'"'"' and platform_system=='"'"'AIX'"'"'':
  ERROR: Traceback (most recent call last):
    File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
      "__main__", mod_spec)
    File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
      exec(code, run_globals)
    File "/usr/local/lib/python3.6/dist-packages/pip/__main__.py", line 16, in <module>
      from pip._internal import main as _main  # isort:skip # noqa
    File "/usr/local/lib/python3.6/dist-packages/pip/_internal/__init__.py", line 4, in <module>
      import locale
    File "/usr/lib/python3.6/locale.py", line 16, in <module>
      import re
    File "/usr/lib/python3.6/re.py", line 142, in <module>
      class RegexFlag(enum.IntFlag):
  AttributeError: module 'enum' has no attribute 'IntFlag'
  Error in sys.excepthook:
  Traceback (most recent call last):
    File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 53, in apport_excepthook
      if not enabled():
    File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 24, in enabled
      import re
    File "/usr/lib/python3.6/re.py", line 142, in <module>
      class RegexFlag(enum.IntFlag):
  AttributeError: module 'enum' has no attribute 'IntFlag'

  Original exception was:
  Traceback (most recent call last):
    File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
      "__main__", mod_spec)
    File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
      exec(code, run_globals)
    File "/usr/local/lib/python3.6/dist-packages/pip/__main__.py", line 16, in <module>
      from pip._internal import main as _main  # isort:skip # noqa
    File "/usr/local/lib/python3.6/dist-packages/pip/_internal/__init__.py", line 4, in <module>
      import locale
    File "/usr/lib/python3.6/locale.py", line 16, in <module>
      import re
    File "/usr/lib/python3.6/re.py", line 142, in <module>
      class RegexFlag(enum.IntFlag):
  AttributeError: module 'enum' has no attribute 'IntFlag'
  ----------------------------------------
ERROR: Command "/usr/bin/python3 /usr/local/lib/python3.6/dist-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-b8vreosn/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools wheel 'numpy==1.13.3; python_version=='"'"'2.7'"'"' and platform_system!='"'"'AIX'"'"'' 'numpy==1.13.3; python_version=='"'"'3.5'"'"' and platform_system!='"'"'AIX'"'"'' 'numpy==1.13.3; python_version=='"'"'3.6'"'"' and platform_system!='"'"'AIX'"'"'' 'numpy==1.14.5; python_version>='"'"'3.7'"'"' and platform_system!='"'"'AIX'"'"'' 'numpy==1.16.0; python_version=='"'"'2.7'"'"' and platform_system=='"'"'AIX'"'"'' 'numpy==1.16.0; python_version=='"'"'3.5'"'"' and platform_system=='"'"'AIX'"'"'' 'numpy==1.16.0; python_version=='"'"'3.6'"'"' and platform_system=='"'"'AIX'"'"'' 'numpy==1.16.0; python_version>='"'"'3.7'"'"' and platform_system=='"'"'AIX'"'"''" failed with error code 1 in None

Enum34 cause issues in many packages.I tried to solve this by using:

pip freeze | grep enum34

and

pip uninstall -y enum34

also i tried unset PYTHONPATH and I still get the same error.

pip show enum34 <-- this command give: WARNING: Package(s) not found: enum34


title: Trouble installing PyTorch for CUDA 9.0

body:
I am trying to install PyTorch for CUDA 9.0 (Ubuntu 18.04 LTS). I followed instructions as per official documentation, which says

Download the whl file with the desired version from the following html page:
https://download.pytorch.org/whl/cu90/torch_stable.html

$ pip install [downloaded file]

But above link is broken, I am unable to open it.


title: Change the image size and range

body:
I want to ask for data transforms if I have an image of size 28 * 28 and I want to resize it to be 32 *32, I know that this could be done with transforms.Resize() but I'm sure how.
Also for the normalization, if I want it to be within the range of [-1,1], I did it previously if I want it to be within [0,1] using transforms.Normalize((0.485,0.456,0.406),(0.229,0.224,0.225))


title: torch.cuda.is_available() retuns FALSE after installing appropriate versions of CUDA and CuDNN on Ubuntu 20.04

body:
I should say installations on Z490 motherboard with Ubuntu 20.04 are quite tricky.

Above are the screenshots from nvidia --version and nvidia-smi
I was able to complete all CUDA and CuDNN installations.
I used all of these. But GPU is not detected with pytorch.
pip3 install torch==1.5.1+cu101 torchvision==0.6.1+cu101 -f https://download.pytorch.org/whl/torch_stable.html

pip install torch torchvision

pip install torch==1.6.0+cu101 torchvision==0.7.0+cu101 -f https://download.pytorch.org/whl/torch_stable.html

A very huge thanks in advance. Someone please help me get out of this.
whereis cuda
cuda: /usr/lib/cuda /usr/include/cuda.h

whereis nvcc
nvcc: /usr/bin/nvcc /etc/nvcc.profile /usr/share/man/man1/nvcc.1.gz

But in some forums i have seen in some forums the path is /usr/local/cuda
In addition to that, my ~/.bashrc contains these:
export LD_LIBRARY_PATH=/usr/lib/cuda/lib64:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/usr/lib/cuda/include:$LD_LIBRARY_PATH


title: Pytorch - how do I index a 2-D matrix row wise?

body:
I would like to index a 2-D matrix row-wise and re-assign values.

For example, first consider a 1-D vector case where we have three 1-D tensors t1, indexes, t2 with the same shape. We can do this indexing and re-assignment as follows:

indexes = torch.tensor([0, 2, 1, 3])
t1 = torch.tensor([0.0, 0.0, 0.0, 0.0])
t2 = torch.tensor([0.1, 0.2, 0.3, 0.4])

t1[indexes] = t2

Now, say that t1, indexes, t2 are 2-D matrices instead of 1-D vectors and have the same shape (R X C). I would like to do similar indexing as above for every row in these matrices where:

for i in range(R):
    t1[i][indexes[i]] = t2[i]

I would like to vectorize this operation instead of using a for loop. How do I do this?


title: Pytorch: Training loss not decreasing in VAE

body:
I have implemented a Variational Autoencoder model in Pytorch that is trained on SMILES strings (String representations of molecular structures).
While training the autoencoder to output the same string as the input, the Loss function does not decrease between epochs.

I have tried the following with no success:
1) Adding 3 more GRU layers to the decoder to increase learning capability of the model.
2) Increasing the latent vector size from 292 to 350.
3) Increasing and decreasing the learning rate.
4) Changing the optimizer from Adam to SGD.
5) Trained the model on upto 50 epochs.
6) Increasing and decreasing the batch size.

The following is the link to my code.
https://colab.research.google.com/drive/1LctSm_Emnn5sHpw_Hon8xL5fF4bmKRw5

The following is an equivalent keras model(Same architecture) that is able to train successfully.
https://colab.research.google.com/drive/170Peseik03CFYpWPNyD8B8mxUGxTQx67


title: How to calculate Batch Pairwise Distance in PyTorch efficiently

body:
I have tensors X of shape BxNxD and Y of shape BxNxD.

I want to compute the pairwise distances for each element in the batch, i.e. I a BxMxN tensor.

How do I do this?

There is some discussion on this topic here: https://github.com/pytorch/pytorch/issues/9406, but I don't understand it as there are many implementation details while no actual solution is highlighted.

A naive approach would be to use the answer for non-batched pairwise distances as discussed here: https://discuss.pytorch.org/t/efficient-distance-matrix-computation/9065, i.e.

import torch
import numpy as np

B = 32
N = 128
M = 256
D = 3

X = torch.from_numpy(np.random.normal(size=(B, N, D)))
Y = torch.from_numpy(np.random.normal(size=(B, M, D)))

def pairwise_distances(x, y=None):
    x_norm = (x**2).sum(1).view(-1, 1)
    if y is not None:
        y_t = torch.transpose(y, 0, 1)
        y_norm = (y**2).sum(1).view(1, -1)
    else:
        y_t = torch.transpose(x, 0, 1)
        y_norm = x_norm.view(1, -1)

    dist = x_norm + y_norm - 2.0 * torch.mm(x, y_t)
    return torch.clamp(dist, 0.0, np.inf)

out = []
for b in range(B):
    out.append(pairwise_distances(X[b], Y[b]))
print(torch.stack(out).shape)

How can I do this without looping over B?
Thanks


title: Embedding 3D data in Pytorch

body:
I want to implement character-level embedding.

This is usual word embedding.

Word Embedding

Input: [ [‘who’, ‘is’, ‘this’] ]
-> [ [3, 8, 2] ]     # (batch_size, sentence_len)
-> // Embedding(Input)
 # (batch_size, seq_len, embedding_dim)

This is what i want to do.

Character Embedding

Input: [ [ [‘w’, ‘h’, ‘o’, 0], [‘i’, ‘s’, 0, 0], [‘t’, ‘h’, ‘i’, ‘s’] ] ]
-> [ [ [2, 3, 9, 0], [ 11, 4, 0, 0], [21, 10, 8, 9] ] ]      # (batch_size, sentence_len, word_len)
-> // Embedding(Input) # (batch_size, sentence_len, word_len, embedding_dim)
-> // sum each character embeddings  # (batch_size, sentence_len, embedding_dim)
The final output shape is same as Word embedding. Because I want to concat them later.

Although I tried it, I am not sure how to implement 3-D embedding. Do you know how to implement such a data?

def forward(self, x):
    print('x', x.size()) # (N, seq_len, word_len)
    bs = x.size(0)
    seq_len = x.size(1)
    word_len = x.size(2)
    embd_list = []
    for i, elm in enumerate(x):
        tmp = torch.zeros(1, word_len, self.embd_size)
        for chars in elm:
            tmp = torch.add(tmp, 1.0, self.embedding(chars.unsqueeze(0)))

Above code got an error because output of self.embedding is Variable.

TypeError: torch.add received an invalid combination of arguments - got (torch.FloatTensor, float, Variable), but expected one of:
 * (torch.FloatTensor source, float value)
 * (torch.FloatTensor source, torch.FloatTensor other)
 * (torch.FloatTensor source, torch.SparseFloatTensor other)
 * (torch.FloatTensor source, float value, torch.FloatTensor other)
      didn't match because some of the arguments have invalid types: (torch.FloatTensor, float, Variable)
 * (torch.FloatTensor source, float value, torch.SparseFloatTensor other)
      didn't match because some of the arguments have invalid types: (torch.FloatTensor, float, Variable)

Update

I could do this. But for is not effective for batch. Do you guys know more efficient way?

def forward(self, x):
    print('x', x.size()) # (N, seq_len, word_len)
    bs = x.size(0)
    seq_len = x.size(1)
    word_len = x.size(2)
    embd = Variable(torch.zeros(bs, seq_len, self.embd_size))
    for i, elm in enumerate(x): # every sample
        for j, chars in enumerate(elm): # every sentence. [ [‘w’, ‘h’, ‘o’, 0], [‘i’, ‘s’, 0, 0], [‘t’, ‘h’, ‘i’, ‘s’] ]
            chars_embd = self.embedding(chars.unsqueeze(0)) # (N, word_len, embd_size) [‘w’,‘h’,‘o’,0]
            chars_embd = torch.sum(chars_embd, 1) # (N, embd_size). sum each char's embedding
            embd[i,j] = chars_embd[0] # set char_embd as word-like embedding

    x = embd # (N, seq_len, embd_dim)

Update2

This is my final code. Thank you, Wasi Ahmad!

def forward(self, x):
    # x: (N, seq_len, word_len)
    input_shape = x.size()
    bs = x.size(0)
    seq_len = x.size(1)
    word_len = x.size(2)
    x = x.view(-1, word_len) # (N*seq_len, word_len)
    x = self.embedding(x) # (N*seq_len, word_len, embd_size)
    x = x.view(*input_shape, -1) # (N, seq_len, word_len, embd_size)
    x = x.sum(2) # (N, seq_len, embd_size)

    return x


title: Tensorflow : What is actually tf.nn.dropout output_keep_prob?

body:
I am trying to understand concept of output_keep_prob:

So if my example is simple RNN :

    with tf.variable_scope('encoder') as scope:
        cells = rnn.LSTMCell(num_units=500)
        cell = rnn.DropoutWrapper(cell=cells, output_keep_prob=0.5)

        model = tf.nn.bidirectional_dynamic_rnn(cell, cell, inputs=embedding_lookup, sequence_length=sequence_le,
                                                dtype=tf.float32)

My confusion is if I am giving output_keep_prob=0.5 what actually it means? I know that it makes less prone to overfitting (called regularizing) by adding a dropout. It  randomly turns off activations of neurons during training, ok I got this point but I am confused when I give

output_keep_prob=0.5 and my no_of_nodes = 500 then 0.5 means it will randomly turn of 50% nodes at each iteration or it means it will keep only those connections which have more or equal probability of 0.5

keep_layers whose probability =>0.5

or

turn off 50% randomly nodes unit at each iteration ??

I tried to understand the concept by this stackoverflow answer but there is also same confusion that what actually 0.5 means? it should drop 50% nodes at each iteration or keep only those nodes which have probability more or equal to 0.5

if the answer is second keep only those nodes which have probability more or equal to 0.5 :

then it means suppose I have given 500 nodes units and only 30 nodes have 0.5 probability so it will turn of rest 470 nodes and will use only 30 nodes for incoming and outgoing connections?

Because  this answer says :

  Suppose you have 10 units in the layer and set the keep_prob to 0.1,
  Then the activation of 9 randomly chosen units out of 10 will be set
  to 0, and the remaining one will be scaled by a factor of 10. I think
  a more precise description is that you only keep the activation of 10
  percent of the nodes.

While other side this answer by @mrry says :

  it means that each connection between layers (in this case between the
  last densely connected layer and the readout layer) will only be used
  with probability 0.5 when training.

can anyone give a clear explanation which one is correct and what actually this value represent in keep_prob?


title: MemoryError when attempting to create a docker image with Torch/PyTorch

body:
I am following this tutorial to create a Docker image for a flask application. The application depends on Torch/PyTorch. As such, my requirements.txt file looks like the following.

flask
flask-cors
pytorch
torchvision
pandas

My Dockerfile then looks like the following.

FROM ubuntu:latest
LABEL My Company "info@mycompany.com"
RUN apt-get update -y
RUN apt-get install -y python-pip python-dev build-essential
COPY . /app
WORKDIR /app
RUN pip install -vvv --no-cache-dir -r requirements.txt
ENTRYPOINT ["python3"]
CMD ["app.py"]

When I type in the command docker build -t flask-sample-one:latest . I get the following error message (which happens when pip is trying to install torch) on my Mac.

Exception:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/usr/lib/python2.7/dist-packages/pip/commands/install.py", line 353, in run
    wb.build(autobuilding=True)
  File "/usr/lib/python2.7/dist-packages/pip/wheel.py", line 749, in build
    self.requirement_set.prepare_files(self.finder)
  File "/usr/lib/python2.7/dist-packages/pip/req/req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "/usr/lib/python2.7/dist-packages/pip/req/req_set.py", line 620, in _prepare_file
    session=self.session, hashes=hashes)
  File "/usr/lib/python2.7/dist-packages/pip/download.py", line 821, in unpack_url
    hashes=hashes
  File "/usr/lib/python2.7/dist-packages/pip/download.py", line 659, in unpack_http_url
    hashes)
  File "/usr/lib/python2.7/dist-packages/pip/download.py", line 882, in _download_http_url
    _download_url(resp, link, content_file, hashes)
  File "/usr/lib/python2.7/dist-packages/pip/download.py", line 603, in _download_url
    hashes.check_against_chunks(downloaded_chunks)
  File "/usr/lib/python2.7/dist-packages/pip/utils/hashes.py", line 46, in check_against_chunks
    for chunk in chunks:
  File "/usr/lib/python2.7/dist-packages/pip/download.py", line 571, in written_chunks
    for chunk in chunks:
  File "/usr/lib/python2.7/dist-packages/pip/utils/ui.py", line 139, in iter
    for x in it:
  File "/usr/lib/python2.7/dist-packages/pip/download.py", line 560, in resp_read
    decode_content=False):
  File "/usr/share/python-wheels/urllib3-1.22-py2.py3-none-any.whl/urllib3/response.py", line 436, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "/usr/share/python-wheels/urllib3-1.22-py2.py3-none-any.whl/urllib3/response.py", line 384, in read
    data = self._fp.read(amt)
  File "/usr/share/python-wheels/CacheControl-0.11.7-py2.py3-none-any.whl/cachecontrol/filewrapper.py", line 63, in read
    self._close()
  File "/usr/share/python-wheels/CacheControl-0.11.7-py2.py3-none-any.whl/cachecontrol/filewrapper.py", line 50, in _close
    self.__callback(self.__buf.getvalue())
  File "/usr/share/python-wheels/CacheControl-0.11.7-py2.py3-none-any.whl/cachecontrol/controller.py", line 275, in cache_response
    self.serializer.dumps(request, response, body=body),
  File "/usr/share/python-wheels/CacheControl-0.11.7-py2.py3-none-any.whl/cachecontrol/serialize.py", line 55, in dumps
    "body": _b64_encode_bytes(body),
  File "/usr/share/python-wheels/CacheControl-0.11.7-py2.py3-none-any.whl/cachecontrol/serialize.py", line 12, in _b64_encode_bytes
    return base64.b64encode(b).decode("ascii")
MemoryError
The command '/bin/sh -c pip install -vvv -r requirements.txt' returned a non-zero code: 2

I am not sure what the MemoryError is coming from. I tried to build the docker image on a Macbook Pro with 16 GB of RAM. Any ideas on what's going on or how to create a docker image with Python?

On an AWS EC2 instance, I get the following error.

Running setup.py install for pytorch: started
    Running command /usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-SeBh33/pytorch/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-5jxyuS-record/install-record.txt --single-version-externally-managed --compile
    Traceback (most recent call last):
      File "", line 1, in
      File "/tmp/pip-build-SeBh33/pytorch/setup.py", line 13, in
        raise Exception(message)
    Exception: You should install pytorch from http://pytorch.org
    Running setup.py install for pytorch: finished with status 'error'
Cleaning up...
  Removing source in /tmp/pip-build-SeBh33/pytorch
Command "/usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-SeBh33/pytorch/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-5jxyuS-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-SeBh33/pytorch/
Exception information:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/usr/lib/python2.7/dist-packages/pip/commands/install.py", line 360, in run
    prefix=options.prefix_path,
  File "/usr/lib/python2.7/dist-packages/pip/req/req_set.py", line 784, in install
    **kwargs
  File "/usr/lib/python2.7/dist-packages/pip/req/req_install.py", line 878, in install
    spinner=spinner,
  File "/usr/lib/python2.7/dist-packages/pip/utils/__init__.py", line 725, in call_subprocess
    % (command_desc, proc.returncode, cwd))
InstallationError: Command "/usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-SeBh33/pytorch/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-5jxyuS-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-SeBh33/pytorch/
The command '/bin/sh -c pip install -vvv --no-cache-dir -r requirements.txt' returned a non-zero code: 1

Any ideas on what's going on? I am simply trying to build a docker image with those few dependencies so I can serve a PyTorch model from a flask application. I'm not sure if there's a tutorial out there on doing this (couldn't find one yet).


title: FastAi What does the slice(lr) do in fit_one_cycle()

body:
In Lesson 3 - planet, I saw these 2 lines of code:

lr = 0.01
learn.fit_one_cycle(5, slice(lr))

if the slice(min_lr, max_lr) then I understand the fit_one_cycle() will use the spread-out Learning Rates from slice(min_lr, max_lr). (Hopefully, my understanding to this is correct)

But in this case slice(lr) only has one parameter,

What are the differences between fit_one_cycle(5, lr) and fit_one_cycle(5, slice(lr))  ?
And what are the benefits of using slice(lr) instead of lr directly?


Using the macOS terminal, I'm trying to run ./autoner_train.sh by following this guide on GitHub.

I have activated my Conda environment and check my PyTorch version

(pytorch_env) myname (master) AutoNER $ python -c "import torch; print(torch.__version__)"
1.3.1

After that, when running, I get the following error

  ModuleNotFoundError: No module named 'torch_scope'

I don't know where's the problem. I have installed everything and I tried googling the problem, all I found is that I need PyTorch installed, which I already have.


title: Pytorch: Batch size is missing in data after torch.utils.random_split() is used on dataloader.dataset

body:
I used random_split() to divide my data into train and test and I observed that if random split is done after the dataloader is created, batch size is missing when getting a batch of data from the dataloader.

import torch
from torchvision import transforms, datasets
from torch.utils.data import random_split

# Normalize the data
transform_image = transforms.Compose([
  transforms.Resize((240, 320)),
  transforms.ToTensor(),
  transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
])

data = '/data/imgs/train'

def load_dataset():
  data_path = data
  main_dataset = datasets.ImageFolder(
    root = data_path,
    transform = transform_image
  )

  loader = torch.utils.data.DataLoader(
    dataset = main_dataset,
    batch_size= 64,
    num_workers = 0,
    shuffle= True
  )

  # Dataset has 22424 data points
  trainloader, testloader = random_split(loader.dataset, [21000, 1424])

  return trainloader, testloader

trainloader, testloader = load_dataset()

Now to get a single batch of images from the train and test loaders:

images, labels = next(iter(trainloader))
images.shape
# %%
len(trainloader)

# %%
images_test, labels_test = next(iter(testloader))
images_test.shape

# %%
len(testloader)

The output that I get is does not have the batch size for train or test batches. Teh output dims should be [batch x channel x H x W] but I get [channel x H x W].

Output:

But if I create the split from the dataset and then make two data loaders using the splits, I get the batchsize in the output.

def load_dataset():
    data_path = data
    main_dataset = datasets.ImageFolder(
      root = data_path,
      transform = transform_image
    )
    # Dataset has 22424 data points
    train_data, test_data = random_split(main_dataset, [21000, 1424])

    trainloader = torch.utils.data.DataLoader(
      dataset = train_data,
      batch_size= 64,
      num_workers = 0,
      shuffle= True
    )

    testloader = torch.utils.data.DataLoader(
      dataset = test_data,
      batch_size= 64,
      num_workers= 0,
      shuffle= True
    )

    return trainloader, testloader

trainloader, testloader = load_dataset()

On running the same 4 commands to get a single train and test batch:

Is the first approach wrong? Although the length shows that the data has been split. So why do I not see the batch size?


title: GPU program failed to execute : cublas runtime error

body:
I am trying to train a network via pytorch on CUDA enabled GeForce GTX 1070 gpu. I don't understand the error nor have I found any similar problem anywhere. I don't know if its cuda's issue or something in my code.

Traceback (most recent call last):
  File "main.py", line 497, in <module>
    main()
  File "main.py", line 167, in main
    train(train_loader, model, criterion, optimizer, epoch, normalizer)
  File "main.py", line 244, in train
    output = model(*input_var)
  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 357, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\Users\1546544\Desktop\ML\model.py", line 147, in forward
    atom_fea = conv_func(atom_fea, nbr_fea, nbr_fea_idx)
  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 357, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\Users\1546544\Desktop\ML\model.py", line 66, in forward
    total_gated_fea = self.fc_full(total_nbr_fea)
  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 357, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\linear.py", line 55, in forward
    return F.linear(input, self.weight, self.bias)
  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\functional.py", line 837, in linear
    output = input.matmul(weight.t())
  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\autograd\variable.py", line 386, in matmul
    return torch.matmul(self, other)
  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\functional.py", line 192, in matmul
    output = torch.mm(tensor1, tensor2)
RuntimeError: cublas runtime error : the GPU program failed to execute at C:/Anaconda2/conda-bld/pytorch_1519496000060/work/torch/lib/THC/THCBlas.cu:247


title: PyTorch - BCELoss: ValueError: Target and input must have the same number of elements

body:
When I use BCELoss as the loss function of my neural network, getting the ValueError: Target and input must have the same number of elements.

Here is my code for test phase (which is a quite typical test phase code):

network.eval()
test_loss = 0
correct = 0
with torch.no_grad():
    for data, target in test_loader:
        data, target = data.to(device), target.to(device)

    output = network(data)
    output = output.to(device)
    test_loss += loss_function(output, target).item() # error happens here
    _, predicted = torch.max(output.data, 1)
    correct += (predicted == target).sum().item()

The shape of the variable output is [1000, 10] as there are 10 target classes (in MNIST dataset), and the shape of the variable target is [1000] as it contains the target classes of the tested batch (the batch size for test is set to 10). So, the question is how can I apply BCELoss as the loss function of a CNN network?

p.s. The dataset I use is the MNIST dataset which is provided by the torchvision library.

p.s. The answer provided to a similar question here does not propose a solution for my case.


title: Ground Truth Image to One Hot Encoded Array (Semantic Segmentation)

body:
I'm creating my own Dataset for People and Street Segmentation. Below, you see a labled Ground Truth (GT) Image.

In the past I did a simple Regression between the model output and the GT Image (in the past I only used Streets). Now I read, Cross Entropy Loss is more common in that case. Since, my GT and also the model output Image has the same width w and height h as the input image, I have to create an array of size h x w x c, where c is the number of classes (in my case 3, background, street, people). I think, this is called One-Hot-Encoded Array.

I solved this as follows:

        for height in range(len(img_as_np_array)):
            for width in range(len(img_as_np_array[0])):
                temp = np.zeros(classes)
                if get_class(img_as_np_array[height,width]) == 1:
                    temp[1] = 1
                    one_hot_label[height,width] = temp
                if get_class(img_as_np_array[height,width]) == 2:
                    temp[2] = 1
                    one_hot_label[height,width] = temp

where the method get_class(channels) decides the pixel class by the color of the pixel.

def get_class(channels):
    threshold = 40
    # Class 1 corresponds to streets, roads
    if channels[0] in np.arange(243-threshold,243+threshold,1) and \
        channels[1] in np.arange(169-threshold,169+threshold,1) and \
        channels[2] in np.arange(0,threshold,1):
        return 1

    # Class 2 corresponds to people
    if channels[0] in np.arange(0,threshold,1) and \
        channels[1] in np.arange(163-threshold,163+threshold,1) and \
        channels[2] in np.arange(232-threshold,232+threshold,1):
        return 2

    # Class 0 corresponds to background respectively other things
    return 0

I have two questions:

My approach is very slow (about 3 minutes for a Full HD Image), is there a way to speed this up?
I noticed that the colors differ in the sense of channel values. For example, orange should be [243,169,0] (RGB), but I found entries like this [206,172,8] or even this [207,176,24] could that happen, because I store my labels as jpg? Is there a better way to find the orange and blue pixels than my idea above with the threshold?

EDIT:

I solved the first question by my self. This takes 2 or 3 seconds for an Full HD Image:

threshold = 40
class_1_shape_cond_1 = (img_as_array[:, :, 0] >= 243 - threshold) * (img_as_array[:, :, 0] <= 243 + threshold)
class_1_shape_cond_2 = (img_as_array[:, :, 1] >= 171 - threshold) * (img_as_array[:, :, 1] <= 171 + threshold)
class_1_shape_cond_3 = (img_as_array[:, :, 2] >= 0) * (img_as_array[:, :, 2] <= threshold)
class_1_shape = (class_1_shape_cond_1 * class_1_shape_cond_2 * class_1_shape_cond_3)

Then I do the same for class 2 and for class 3 (everything else) I can do:

class_3_shape = 1 - (class_1_shape + class_2_shape)

After that I have to adjust the type with:

class_1_shape = class_1_shape.astype(np.uint8)
class_2_shape = class_2_shape.astype(np.uint8)
class_3_shape = class_3_shape.astype(np.uint8)

Question 2 is still an open.


title: Is there an function in PyTorch for converting convolutions to fully-connected networks form?

body:
I'm trying to convert a convolution layer to a fully-connected layer.

For example, there is an example of 3×3 input and 2x2 kernel:

which is equivalent to a vector-matrix multiplication，

Is there a function in PyTorch to get the matrix B?


title: PyTorch LogSoftmax vs Softmax for CrossEntropyLoss

body:
I understand that PyTorch's LogSoftmax function is basically just a more numerically stable way to compute Log(Softmax(x)). Softmax lets you convert the output from a Linear layer into a categorical probability distribution.
The pytorch documentation says that CrossEntropyLoss combines nn.LogSoftmax() and nn.NLLLoss() in one single class.
Looking at NLLLoss, I'm still confused...Are there 2 logs being used? I think of negative log as information content of an event. (As in entropy)
After a bit more looking, I think that NLLLoss assumes that you're actually passing in log probabilities instead of just probabilities. Is this correct? It's kind of weird if so...


title: Pytorch why is .float() needed here for RuntimeError: expected scalar type Float but found Double

body:
Simple question, i wanted to experiment with the simplest possible network, but i kept running into RuntimeError: expected scalar type Float but found Double unless i casted data into .float() (see below code with comment)
What i dont understand is, why is this casting needed? data is already a torch.float64 type. Whys the explicit re-casting in the output = model(data.float()) line needed?
Code
from __future__ import print_function
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torchvision import datasets, transforms
from torch.optim.lr_scheduler import StepLR
from sklearn.datasets import make_classification
from torch.utils.data import TensorDataset, DataLoader

# =============================================================================
# Simplest Example
# =============================================================================
X, y = make_classification()
X, y = torch.tensor(X), torch.tensor(y)
print("X Shape :{}".format(X.shape))
print("y Shape :{}".format(y.shape))

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(X.shape[1], 128)
        self.fc2 = nn.Linear(128, 10)
        self.fc3 = nn.Linear(10, 2)

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

device = torch.device("cuda")
lr = 1
batch_size = 32
gamma = 0.7
epochs = 14
args = {'log_interval': 10, 'dry_run':False}

kwargs = {'batch_size': batch_size}
kwargs.update({'num_workers': 1,
               'pin_memory': True,
               'shuffle': True},
                 )

model = Net().to(device)
optimizer = optim.Adam(model.parameters(), lr=lr)
scheduler = StepLR(optimizer, step_size=1, gamma=gamma)

my_dataset = TensorDataset(X,y) # create dataset
train_loader = DataLoader(my_dataset,**kwargs) #generate dataloader

cross_entropy_loss = torch.nn.CrossEntropyLoss()

for epoch in range(1, epochs + 1):
    ## Train step ##
    model.train()

    for batch_idx, (data, target) in enumerate(train_loader):
        data, target = data.to(device), target.to(device)
        optimizer.zero_grad()
        output = model(data.float()) #HERE: why is .float() needed here?
        loss = cross_entropy_loss(output, target)
        loss.backward()
        optimizer.step()
        if batch_idx % args['log_interval'] == 0:
            print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                epoch, batch_idx * len(data), len(train_loader.dataset),
                100. * batch_idx / len(train_loader), loss.item()))
            if args['dry_run']:
                break

    scheduler.step()


title: Pytorch - handling picutres and .jpeg files (beginner's questions)

body:
I am new at Pytorch, and have a couple of questions regarding the way pictures are being handled:

1) In the "training a classifier" tutorial, the pictures are PIL files, and are being handled via the following commands (where "transform" also turns the PIL format into a tensor format):

trainset = torchvision.datasets.CIFAR10(root='./data', train=True,
                                    download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,
                                          shuffle=True, num_workers=2)

It seems like trainset[1] (and also for the other indices) consists of a tensor, and a number.  I want to define a new variable "image" that will consist of the tensor part of trainset[ 1 ] and then print it - how can I do it?

2) Assume that I have a different dataset that I want to classify. It consists of .jpeg images that are located in the folder "C:/temp/dataset". How can I define the variable "trainset" to consist of these images?

Thanks a lot in advance!


title: Problem importing module: ModuleNotFoundError: No module named 'torch.utils.checkpoint'

body:
I am following this example of ULMFiT. Here is the github.

I am running the model_forward_pass.ipynb in Google Colab. I am running the first code block which is this:

import pandas as pd
import numpy as np
import torch
!pip install fastai==0.7.0
!pip install torch
import fastai
from fastai.imports import *
from fastai.torch_imports import *
from fastai.core import *
from fastai.model import fit
from fastai.dataset import *

import torchtext
from torchtext import vocab, data
from torchtext.datasets import language_modeling

from fastai.rnn_reg import *
from fastai.rnn_train import *
from fastai.nlp import *
from fastai.lm_rnn import *
from fastai.text import *

import dill as pickle
!pip install spacy
!python -m spacy download en
import spacy
import html

import pickle
import collections
#!pip install nbimporter
import nbimporter

I am getting the following error:

ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-1-bd3be060f151> in <module>()
      6 import fastai
      7 from fastai.imports import *
----> 8 from fastai.torch_imports import *
      9 from fastai.core import *
     10 from fastai.model import fit

3 frames
/usr/local/lib/python3.6/dist-packages/fastai/torch_imports.py in <module>()
      1 import os
----> 2 import torch, torchvision, torchtext
      3 from torch import nn, cuda, backends, FloatTensor, LongTensor, optim
      4 import torch.nn.functional as F
      5 from torch.autograd import Variable

/usr/local/lib/python3.6/dist-packages/torchvision/__init__.py in <module>()
      1 import warnings
      2
----> 3 from torchvision import models
      4 from torchvision import datasets
      5 from torchvision import ops

/usr/local/lib/python3.6/dist-packages/torchvision/models/__init__.py in <module>()
      4 from .squeezenet import *
      5 from .inception import *
----> 6 from .densenet import *
      7 from .googlenet import *
      8 from .mobilenet import *

/usr/local/lib/python3.6/dist-packages/torchvision/models/densenet.py in <module>()
      3 import torch.nn as nn
      4 import torch.nn.functional as F
----> 5 import torch.utils.checkpoint as cp
      6 from collections import OrderedDict
      7 from .utils import load_state_dict_from_url

ModuleNotFoundError: No module named 'torch.utils.checkpoint'

I found a similar question here, but I am still not sure how to fix my problem.
What should I do to fix this?


title: Setting the same seed for torch, random number and numpy throughout all the modules

body:
I am trying to set the same seed throughout all the project.
Below are the parameters I am setting in my main file, in which all other modules will be imported -
seed = 42
os.environ['PYTHONHASHSEED'] = str(seed)
# Torch RNG
torch.manual_seed(seed)
torch.cuda.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
# Python RNG
np.random.seed(seed)
random.seed(seed)

My project directory looks like below -
├── Combined_Files_without_label5.csv
├── __pycache__
│   ├── dataset.cpython-37.pyc
│   ├── datasets.cpython-37.pyc
│   └── testing.cpython-37.pyc
├── datasets.py
├── import_packages
│   ├── __init__.py
│   ├── __pycache__
│   │   ├── __init__.cpython-37.pyc
│   │   ├── dataset.cpython-37.pyc
│   │   ├── dataset_class.cpython-37.pyc
│   │   ├── dataset_partition.cpython-37.pyc
│   │   └── visualising.cpython-37.pyc
│   ├── dataset_class.py
│   ├── dataset_partition.py
│   └── visualising.py
├── main.py

Now, the problem is I am importing the module from dataset_partition.py and the function  needs a seed value there. e.g -
    df_train, df_temp, y_train, y_temp = train_test_split(X,
                                                      y,
                                                      stratify=y,
                                                      test_size=(1.0 - frac_train), # noqa
                                                      random_state=seed)

Now, my questions are, 1)If I just remove the random_state parameter from the above statement so will it take the seed from my main file? If not, then how to set it?
2)Does all the other function which requires seed like torch.manual.seed, torch.cuda.manual_seed(seed) will behave in the same way?(If not, then how to resolve it)


title: pytorch running: RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu

body:
When I running the python code, there is a runtimeError in line 44:RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
    42  feats = self.node_features[self.train_mask]
    43  labels = self.node_labels[train_mask]

    44  A = torch.mm(feats.t(), feats) + 1e-05 * torch.eye(feats.size(1))

    45  labels_one_hot = torch.zeros((feats.size(0), self.n_classes))

Can anyone who knows the reason and help me fix it!  Thanks!


title: Unable to figure out inplace operation in the pytorch code?

body:
I have the following implementation in PyTorch for learning using LSTM:

https://gist.github.com/rahulbhadani/f1d64042cc5a80280755cac262aa48aa

However, the code is experiencing in-place operation error

where my error output is:

/home/ivory/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:10: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  # Remove the CWD from sys.path while we load stuff.
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-86-560ec78f2b64> in <module>
     27 linear = torch.nn.Linear(hidden_nums, output_dim)
     28
---> 29 global_loss_list = global_training(lstm2)

<ipython-input-84-152890a3028c> in global_training(optimizee)
      3     adam_global_optimizer = torch.optim.Adam([{'params': optimizee.parameters()},
      4                                      {'params':linear.parameters()}], lr = 0.0001)
----> 5     _, global_loss_1 = learn2(LSTM_Optimizee, training_steps, retain_graph_flag=True, reset_theta=True)
      6
      7     print(global_loss_1)

<ipython-input-83-0357a528b94d> in learn2(optimizee, unroll_train_steps, retain_graph_flag, reset_theta)
     43             # requires_grad=True. These are accumulated into x.grad for every
     44             # parameter x. In pseudo-code: x.grad += dloss/dx
---> 45             loss.backward(retain_graph = retain_graph_flag) #The default is False, when the optimized LSTM is set to True
     46
     47             print('x.grad: {}'.format(x.grad))

~/anaconda3/lib/python3.7/site-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph)
    116                 products. Defaults to ``False``.
    117         """
--> 118         torch.autograd.backward(self, gradient, retain_graph, create_graph)
    119
    120     def register_hook(self, hook):

~/anaconda3/lib/python3.7/site-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
     91     Variable._execution_engine.run_backward(
     92         tensors, grad_tensors, retain_graph, create_graph,
---> 93         allow_unreachable=True)  # allow_unreachable flag
     94
     95

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [1, 10]] is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

I have tried to trace the error but didn't succeed. Any help in this regard will be appreciated.

Thanks.


title: Making a Prediction Sagemaker Pytorch

body:
I have trained and deployed a model in Pytorch with Sagemaker. I am able to call the endpoint and get a prediction. I am using the default input_fn() function (i.e. not defined in my serve.py).

model = PyTorchModel(model_data=trained_model_location,
                     role=role,
                     framework_version='1.0.0',
                     entry_point='serve.py',
                     source_dir='source')
predictor = model.deploy(initial_instance_count=1, instance_type='ml.m4.xlarge')

A prediction can be made as follows:

input ="0.12787057,  1.0612601,  -1.1081504"
predictor.predict(np.genfromtxt(StringIO(input), delimiter=",").reshape(1,3) )

I want to be able to serve the model with REST API and am HTTP POST using lambda and API gateway. I was able to use invoke_endpoint() for this with an XGBOOST model in Sagemaker this way. I am not sure what to send into the body for Pytorch.

client = boto3.client('sagemaker-runtime')
response = client.invoke_endpoint(EndpointName=ENDPOINT  ,
ContentType='text/csv',
Body=???)

I believe I need to understand how to write the customer input_fn to accept and process the type of data I am able to send through invoke_client. Am I on the right track and if so, how could the input_fn be written to accept a csv from invoke_endpoint?


title: Create a torch::Tensor in C++ to change the shape

body:
I have a tensor array, and I want to change the shape of tensor. I tried to use torch.view, but it raise an exception that "shape[1] is invalid for input of size 10000". Anyone can give me a tips for the error information?

int shape[] = {1,1,100,100};
torch::Tensor img = torch::zeros((100,100),torch::KF32);
torch::Tensor tmg = img.view(*shape);


title: Pytorch on Windows gives ImportError

body:
I installed Pytorch for PYthon 3.6 using pip as instructed on https://pytorch.org/.

Pytorch is installed succesfully, but when I run code, I get this:

  File "C:\Users\\PycharmProjects\chatbot-light\pytorch\rnn_attention\seq2seq_translation_tutorial.py", line 93, in <module>
    import torch
  File "C:\Users\\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\__init__.py", line 78, in <module>
    from torch._C import *
ImportError: DLL load failed: The specified module could not be found


title: How to `dot` weights to batch data in PyTorch?

body:
I have batch data and want to dot() to the data. W is trainable parameters.
How to dot between batch data and weights?

hid_dim = 32
data = torch.randn(10, 2, 3, hid_dim)
data = data.view(10, 2*3, hid_dim)
W = torch.randn(hid_dim) # assume trainable parameters via nn.Parameter
result = torch.bmm(data, W).squeeze() # error, want (N, 6)
result = result.view(10, 2, 3)

Update

How about this one?

hid_dim = 32
data = torch.randn(10, 2, 3, hid_dim)
data = tdata.view(10, 2*3, hid_dim)
W = torch.randn(hid_dim, 1) # assume trainable parameters via nn.Parameter
W = W.unsqueeze(0).expand(10, hid_dim, 1)
result = torch.bmm(data, W).squeeze() # error, want (N, 6)
result = result.view(10, 2, 3)


title: PyTorch: Variable data has to be a tensor -- data is already as tenors

body:
I am trying to load data using pytorch's Dataset and DataLoader classes. I use torch.from_numpyto convert each array to a tensor in the torch Dataset and from looking into the data, each X and y is indeed a tensor

# At this point dataset is {'X': numpy array of arrays, 'y': numpy array of arrays  }

class TorchDataset(torch.utils.data.Dataset):
    def __init__(self, dataset):
        self.X_train = torch.from_numpy(dataset['X'])
        self.y_train = torch.from_numpy(dataset['y'])

    def __len__(self):
        return len(self.X_train)

    def __getitem__(self, index):
        return {'X': self.X_train[index], 'y': self.y_train[index]}

torch_dataset = TorchDataset(dataset)
dataloader = DataLoader(torch_dataset, batch_size=4, shuffle=True, num_workers=4)

for epoch in range(num_epochs):
    for X, y in enumerate(dataloader):
        features = Variable(X)
        labels = Variable(y)
        ....

However on features = Variable(X) i get:

RuntimeError: Variable data has to be a tensor, but got int

An example of an X and y in the dataset are:

In [1]: torch_dataset[1]
Out[1]:
{'X':
 -2.5908 -3.1123 -2.9460  ...  -3.9898 -4.0000 -3.9975
 -3.0867 -2.9992 -2.5254  ...  -4.0000 -4.0000 -4.0000
 -2.7665 -2.5318 -2.7035  ...  -4.0000 -4.0000 -4.0000
       ...             ⋱             ...
 -2.4784 -2.6061 -1.6280  ...  -4.0000 -4.0000 -4.0000
 -2.2046 -2.1778 -1.5626  ...  -3.9597 -3.9366 -3.9497
 -1.9623 -1.9468 -1.5352  ...  -3.8485 -3.8474 -3.8474
 [torch.DoubleTensor of size 1024x1024], 'y':
  107
 [torch.LongTensor of size 1]}

which is why it is very confusing for me that torch thinks X is an int. Any help would be much appreciated - thanks!


(Pdb) import os
(Pdb) import numpy as np
(Pdb) from tempfile import mkdtemp
(Pdb) import os.path as path

(Pdb) tensor
tensor([     0,      0,  20480,  ...,      0,      0, -16400], dtype=torch.int16)
(Pdb) tensor.dtype
torch.int16

(Pdb) filename = path.join(mkdtemp(), f)
*** TypeError: join() argument must be str or bytes, not 'BufferedReader'

(Pdb) import torch

(Pdb) fp = np.memmp(filename, dtype='int16')
(Pdb) fp
memmap([16, 12, 12, ..., 44, 44, 44], dtype=int16)
(Pdb)

I am trying to create a memory-map to an array stored in a binary file on disk. But I am getting *** TypeError: join() argument must be str or bytes, not 'BufferedReader'  error. Even after the error the memory map is working but I want it free from error. Please help me out. Thanks


title: How to ensure that a batch contains samples from all workers with PyTorch's DataLoader?

body:
I want to know how to use torch.utils.data.DataLoader in PyTorch, especially in a multi-worker case.

I found that one batch output from DataLoader always comes from a single worker.
I expected that there is a queue in the DataLoader which stores data from all of the workers and DataLoader shuffles them in the queue to output the random batch data. I think this is the way in tf.data.Dataset in Tensorflow.
Can we implement a similar function in PyTorch? I want to load a dataset from big serialized files (like Tfrecord) by using multi workers. In this case, mixing the source file in one batch, which means mixing the source of the worker, is important.

Please refer to following code:

import random
import time

import torch

class MyDataset(torch.utils.data.Dataset):
    def __len__(self):
        return 50

    def __getitem__(self, idx):
        info = torch.utils.data.get_worker_info()

        time.sleep(random.uniform(0, 1))
        print("[{}]:{}".format(info.id, idx))
        return idx, info.id

if __name__ == '__main__':
    dataset = MyDataset()
    dataloader = torch.utils.data.DataLoader(dataset, batch_size=5, shuffle=False, num_workers=2)
    for batch in dataloader:
        print(batch)

Output:

[0]:0
[1]:5
[0]:1
[1]:6
[0]:2
[0]:3
[1]:7
[0]:4
[tensor([0, 1, 2, 3, 4]), tensor([0, 0, 0, 0, 0])]
[1]:8
[1]:9
[tensor([5, 6, 7, 8, 9]), tensor([1, 1, 1, 1, 1])]
[0]:10
[0]:11
[1]:15
[1]:16
[0]:12
[1]:17
...

Here, [0, 1, 2, 3, 4] and [0, 0, 0, 0, 0] in [tensor([0, 1, 2, 3, 4]), tensor([0, 0, 0, 0, 0])] mean that this batch includes index 0-th to 4-th data came from worker id 0.
Note that shuffle=True does not solve this problem which only change the indices of data.

In this case, I want to get a batch like: [tensor([0, 5, 1, 6, 2]), tensor([0, 1, 0, 1, 0])].


title: How does one make sure that everything is running on GPU automatically in Pytorch?

body:
I wanted a way with minimal amount of code such that everything in my script runs automatically in GPU (or the standard way pytorch did it). Something like:

torch.everything_to_gpu()

and then it "just works". I don't care about manually putting things in GPU etc. I just want it to do its stuff automatically (sort of the way tensorflow does it?). I did see a related question in the pytorch forum but it doesn't seem that they address my issue directly.

Right now it seems to me (from the examples I've been through) that one can do something like what I want by specifying a simple type to every torch Variable/tensor as follows:

dtype = torch.FloatTensor
# dtype = torch.cuda.FloatTensor # Uncomment this to run on GPU

so as long as every variable/tensor is takes dtype somehow e.g.

Variable(torch.FloatTensor(x).type(dtype), requires_grad=False)

then we can use that single variable to control what is in GPU and not. The issue that I am encountering that makes things ambiguous to me if such a single command exists is when using torch.nn.Module package. For example when using

l = torch.nn.Linear(D_in,D_out)

or costume NN classes (that inherit from it). Such cases seems that the best way to deal with it is to use the:

torch.nn.Module.cuda(device_id=device_id) # device_id = None is the default

function/method. However this seems to suggest to me that there might be other hidden functions that I might not be aware of to make sure that everything does indeed run in GPU.

Thus: Is there a centralized way to make sure everything runs in some (ideally automatically) assigned GPU?

In reflection I think one thing that is confusing me is that I don't understand the model of how pytorch carriers on computations on GPU. For example, I am fairly certain tht the way MATLAB works is that if at least one thing is on GPU then all further computations will be on GPU. So I guess, I am wondering, is this how pytorch works? If possible, how does it compare to TensorFlow?


title: Convert a list of tensors to tensors of tensors pytorch

body:
I have this code:

import torch

list_of_tensors = [ torch.randn(3), torch.randn(3), torch.randn(3)]
tensor_of_tensors = torch.tensor(list_of_tensors)

I am getting the error:

  ValueError: only one element tensors can be converted to Python scalars

How can I convert the list of tensors to a tensor of tensors in pytorch?


title: PyTorch: Different Forward Methods for Train and Test/Validation

body:
I'm currently trying to extend a model that is based on FairSeq/PyTorch. During training I need to train two encoders: one with the target sample, and the original one with the source sample.

So the current forward function looks like this:

def forward(self, src_tokens=None, src_lengths=None, prev_output_tokens=None, **kwargs):
    encoder_out = self.encoder(src_tokens, src_lengths=src_lengths, **kwargs)
    decoder_out = self.decoder(prev_output_tokens, encoder_out=encoder_out, **kwargs)
    return decoder_out

And based on this this idea i want something like this:

def forward_test(self, src_tokens=None, src_lengths=None, prev_output_tokens=None, **kwargs):
    encoder_out = self.encoder(src_tokens, src_lengths=src_lengths, **kwargs)
    decoder_out = self.decoder(prev_output_tokens, encoder_out=encoder_out, **kwargs)
    return decoder_out

def forward_train(self, src_tokens=None, src_lengths=None, prev_output_tokens=None, **kwargs):
    encoder_out = self.encoder(src_tokens, src_lengths=src_lengths, **kwargs)
    autoencoder_out = self.encoder(tgt_tokens, src_lengths=src_lengths, **kwargs)
    concat = some_concatination_func(encoder_out, autoencoder_out)
    decoder_out = self.decoder(prev_output_tokens, encoder_out=concat, **kwargs)
    return decoder_out

Is there any way to do this?

Edit:
These are the constraints that I have, since I need to extend FairseqEncoderDecoderModel:

@register_model('transformer_mass')
class TransformerMASSModel(FairseqEncoderDecoderModel):
    def __init__(self, encoder, decoder):
        super().__init__(encoder, decoder)

Edit 2:
The parameters passed to the forward function in Fairseq can be altered by implementing your own Criterion, see for example CrossEntropyCriterion, where sample['net_input'] is passed to the __call__ function of the model, which invokes the forward method.