title: pytorch Network.parameters() missing 1 required positional argument: 'self'

body:
There's a problem when I call Network.parameters() in pytorch in this line in my main function:
optimizer = optim.SGD(Network.parameters(), lr=0.001, momentum=0.9)

I get the error code:

TypeError: parameters() missing 1 required positional argument: 'self'

My network is defined in this class

class Network(nn.Module):
def __init__(self):
    super(Network, self).__init__()
    self.conv1 = nn.Conv2d(1, 32, 5)
    self.pool = nn.MaxPool2d(2, 2)
    self.conv2 = nn.Conv2d(32, 64, 5)
    self.pool2 = nn.MaxPool2d(2, 2)
    self.conv3 = nn.Conv2d(64, 64, 5)
    self.pool2 = nn.MaxPool2d(2, 2)
    self.fc1 = nn.Linear(64 * 5 * 5, 512)
    self.fc2 = nn.Linear(512, 640)
    self.fc3 = nn.Linear(640, 3756)

def forward(self, x):
    x = self.pool(F.relu(self.conv(x)))
    x = self.pool(F.relu(self.conv2(x)))
    x = self.pool(F.relu(self.conv3(x)))
    x = x.view(-1, 64 * 5 * 5)
    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(x))
    x = self.fc3(x)
    return x

Pretty sure that I imported all torch modules correctly.
Any ideas of what I'm doing wrong here?

Thank you!


title: How to clear GPU memory after PyTorch model training without restarting kernel

body:
I am training PyTorch deep learning models on a Jupyter-Lab notebook, using CUDA on a Tesla K80 GPU to train. While doing training iterations, the 12 GB of GPU memory are used. I finish training by saving the model checkpoint, but want to continue using the notebook for further analysis (analyze intermediate results, etc.).

However, these 12 GB continue being occupied (as seen from nvtop) after finishing training. I would like to free up this memory so that I can use it for other notebooks.

My solution so far is to restart this notebook's kernel, but that is not solving my issue because I can't continue using the same notebook and its respective output computed so far.


title: Nvcc missing when installing cudatoolkit?

body:
I have installed cuda along pytorch with

conda install pytorch torchvision cudatoolkit=10.0 -c pytorch

However, it seems like nvcc was not installed along with it. If I want to use for example nvcc -V, I get the error that nvcc was not found, and that I should install it with sudo apt install nvidia-cuda-toolkit.
Can I do this (I dont want to just try and then find out that it is not working/messes up the whole cuda setup).
And is this a bug or expected behavior?

I am using Ubuntu 18.04 and have cuda 10.2


I am making a DataLoader from DataSet in PyTorch.

Start from loading the DataFrame with all dtype as an np.float64

result = pd.read_csv('dummy.csv', header=0, dtype=DTYPE_CLEANED_DF)

Here is my dataset classes.

from torch.utils.data import Dataset, DataLoader
class MyDataset(Dataset):
    def __init__(self, result):
        headers = list(result)
        headers.remove('classes')

        self.x_data = result[headers]
        self.y_data = result['classes']
        self.len = self.x_data.shape[0]

    def __getitem__(self, index):
        x = torch.tensor(self.x_data.iloc[index].values, dtype=torch.float)
        y = torch.tensor(self.y_data.iloc[index], dtype=torch.float)
        return (x, y)

    def __len__(self):
        return self.len

Prepare the train_loader and test_loader

train_size = int(0.5 * len(full_dataset))
test_size = len(full_dataset) - train_size
train_dataset, test_dataset = torch.utils.data.random_split(full_dataset, [train_size, test_size])

train_loader = DataLoader(dataset=train_dataset, batch_size=16, shuffle=True, num_workers=1)
test_loader = DataLoader(dataset=train_dataset)

Here is my csv file

When I try to iterate over the train_loader. It raises the error

for i , (data, target) in enumerate(train_loader):
    print(i)

TypeError                                 Traceback (most recent call last)
<ipython-input-32-0b4921c3fe8c> in <module>
----> 1 for i , (data, target) in enumerate(train_loader):
      2     print(i)

/opt/conda/lib/python3.6/site-packages/torch/utils/data/dataloader.py in __next__(self)
    635                 self.reorder_dict[idx] = batch
    636                 continue
--> 637             return self._process_next_batch(batch)
    638
    639     next = __next__  # Python 2 compatibility

/opt/conda/lib/python3.6/site-packages/torch/utils/data/dataloader.py in _process_next_batch(self, batch)
    656         self._put_indices()
    657         if isinstance(batch, ExceptionWrapper):
--> 658             raise batch.exc_type(batch.exc_msg)
    659         return batch
    660

TypeError: Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 138, in _worker_loop
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "/opt/conda/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 138, in <listcomp>
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "/opt/conda/lib/python3.6/site-packages/torch/utils/data/dataset.py", line 103, in __getitem__
    return self.dataset[self.indices[idx]]
  File "<ipython-input-27-107e03bc3c6a>", line 12, in __getitem__
    x = torch.tensor(self.x_data.iloc[index].values, dtype=torch.float)
  File "/opt/conda/lib/python3.6/site-packages/pandas/core/indexing.py", line 1478, in __getitem__
    return self._getitem_axis(maybe_callable, axis=axis)
  File "/opt/conda/lib/python3.6/site-packages/pandas/core/indexing.py", line 2091, in _getitem_axis
    return self._get_list_axis(key, axis=axis)
  File "/opt/conda/lib/python3.6/site-packages/pandas/core/indexing.py", line 2070, in _get_list_axis
    return self.obj._take(key, axis=axis)
  File "/opt/conda/lib/python3.6/site-packages/pandas/core/generic.py", line 2789, in _take
    verify=True)
  File "/opt/conda/lib/python3.6/site-packages/pandas/core/internals.py", line 4537, in take
    new_labels = self.axes[axis].take(indexer)
  File "/opt/conda/lib/python3.6/site-packages/pandas/core/indexes/base.py", line 2195, in take
    return self._shallow_copy(taken)
  File "/opt/conda/lib/python3.6/site-packages/pandas/core/indexes/range.py", line 267, in _shallow_copy
    return self._int64index._shallow_copy(values, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/pandas/core/indexes/numeric.py", line 68, in _shallow_copy
    return self._shallow_copy_with_infer(values=values, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/pandas/core/indexes/base.py", line 538, in _shallow_copy_with_infer
    if not len(values) and 'dtype' not in kwargs:
TypeError: object of type 'numpy.int64' has no len()

Related issues:
https://github.com/pytorch/pytorch/issues/10165
https://github.com/pytorch/pytorch/pull/9237
https://github.com/pandas-dev/pandas/issues/21946

Questions:
How to workaround pandas issue here?


title: PyTorch - Element-wise multiplication between a variable and a tensor?

body:
As of PyTorch 0.4 this question is no longer valid. In 0.4 Tensors and Variables were merged.

How can I perform element-wise multiplication with a variable and a tensor in PyTorch? With two tensors works fine. With a variable and a scalar works fine. But when attempting to perform element-wise multiplication with a variable and tensor I get:

XXXXXXXXXXX in mul
    assert not torch.is_tensor(other)
AssertionError

For example, when running the following:

import torch

x_tensor = torch.Tensor([[1, 2], [3, 4]])
y_tensor = torch.Tensor([[5, 6], [7, 8]])

x_variable = torch.autograd.Variable(x_tensor)

print(x_tensor * y_tensor)
print(x_variable * 2)
print(x_variable * y_tensor)

I would expect the first and last print statements to show similar results. The first two multiplications work as expected, with the error coming up in the third. I have attempted the aliases of * in PyTorch (i.e. x_variable.mul(y_tensor), torch.mul(y_tensor, x_variable), etc.).

It seems that element-wise multiplication between a tensor and a variable is not supported given the error and the code which produces it. Is this correct? Or is there something I'm missing? Thank you!


title: Pytorch: How to compute IoU (Jaccard Index) for semantic segmentation

body: Can someone provide a toy example of how to compute IoU (intersection over union) for semantic segmentation in pytorch?


title: How can I make a FloatTensor with requires_grad=True from a numpy array using PyTorch 0.4.0?

body:
Pytorch 0.4.0 introduced the merging on the Tensor and Variable classes.

Before this version, when I wanted to create a Variable with autograd from a numpy array I would do the following (where x is a numpy array):

x = Variable(torch.from_numpy(x).float(), requires_grad=True)

With PyTorch version 0.4.0, the migration guide shows how we can create Tensors with autograd enabled, examples show you can do things such as

x = torch.ones(3, 4, requires_grad=True)

and also set requires_grad to existing tensors

existing_tensor.requires_grad_()

I have tried the following three things to try and create a Tensor with requires_grad=True which give errors (where x is a numpy array):

The first is

x = FloatTensor(x, requires_grad=True)

which gives the error

TypeError: new() received an invalid combination of arguments - got
(numpy.ndarray, requires_grad=bool), but expected one of:
 * (torch.device device)
 * (tuple of ints size, torch.device device)
      didn't match because some of the keywords were incorrect:
requires_grad
 * (torch.Storage storage)
 * (Tensor other)
 * (object data, torch.device device)
      didn't match because some of the keywords were incorrect:
requires_grad

The second is to do

x = FloatTensor(x)
x.requires_grad()

And the third is

x = torch.from_numpy(x).single()
x.requires_grad()

Which both throw the following error on the second line:

TypeError: 'bool' object is not callable

These errors give me little hint at what I'm doing wrong, and since the latest version is so new its hard to find content online to help. How can I make a FloatTensor with requires_grad=True from a numpy array using PyTorch 0.4.0, preferably in a single line?


From https://pytorch.org/

to install pytorch on MacOS the following is stated :

conda install pytorch torchvision -c pytorch
# MacOS Binaries dont support CUDA, install from source if CUDA is needed

Why would want to install pytorch without cuda enabled ?

Reason I ask is I receive error :

  --------------------------------------------------------------------------- AssertionError                            Traceback (most recent call
  last)  in ()
       78 #     predicted = outputs.data.max(1)[1]
       79
  ---> 80     output = model(torch.tensor([[1,1]]).float().cuda())
       81     predicted = output.data.max(1)[1]
       82

  ~/anaconda3/lib/python3.6/site-packages/torch/cuda/init.py in
  _lazy_init()
      159         raise RuntimeError(
      160             "Cannot re-initialize CUDA in forked subprocess. " + msg)
  --> 161     _check_driver()
      162     torch._C._cuda_init()
      163     _cudart = _load_cudart()

  ~/anaconda3/lib/python3.6/site-packages/torch/cuda/init.py in
  _check_driver()
       73 def _check_driver():
       74     if not hasattr(torch._C, '_cuda_isDriverSufficient'):
  ---> 75         raise AssertionError("Torch not compiled with CUDA enabled")
       76     if not torch._C._cuda_isDriverSufficient():
       77         if torch._C._cuda_getDriverVersion() == 0:

  AssertionError: Torch not compiled with CUDA enabled

when attempting to execute code :

x = torch.tensor([[0,0] , [0,1] , [1,0]]).float()
print(x)

y = torch.tensor([0,1,1]).long()
print(y)

my_train = data_utils.TensorDataset(x, y)
my_train_loader = data_utils.DataLoader(my_train, batch_size=2, shuffle=True)

# Device configuration
device = 'cpu'
print(device)

# Hyper-parameters
input_size = 2
hidden_size = 100
num_classes = 2

learning_rate = 0.001

train_dataset = my_train

train_loader = my_train_loader

pred = []

for i in range(0 , model_iters) :
    # Fully connected neural network with one hidden layer
    class NeuralNet(nn.Module):
        def __init__(self, input_size, hidden_size, num_classes):
            super(NeuralNet, self).__init__()
            self.fc1 = nn.Linear(input_size, hidden_size)
            self.relu = nn.ReLU()
            self.fc2 = nn.Linear(hidden_size, num_classes)

        def forward(self, x):
            out = self.fc1(x)
            out = self.relu(out)
            out = self.fc2(out)
            return out

    model = NeuralNet(input_size, hidden_size, num_classes).to(device)

    # Loss and optimizer
    criterion = nn.CrossEntropyLoss()
    optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

    # Train the model
    total_step = len(train_loader)
    for epoch in range(num_epochs):
        for i, (images, labels) in enumerate(train_loader):
            # Move tensors to the configured device
            images = images.reshape(-1, 2).to(device)
            labels = labels.to(device)

            # Forward pass
            outputs = model(images)
            loss = criterion(outputs, labels)

            # Backward and optimize
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
{:.4f}'.format(epoch+1, num_epochs, i+1, total_step, loss.item()))

    output = model(torch.tensor([[1,1]]).float().cuda())

To fix this error I need to install pytorch from source with cuda already installed ?


title: Module not found in pycharm (Windows)

body:
I wanted to install Pytorch via anaconda and it worked but PyCharm can't find the module (

  ModuleNotFoundError: No module named 'torch'

I also have CUDA installed but when I looked up to add a the package with pycharm it also gives an error. When I added the anaconda interpreter I can't run the code. I use Python 3.6


title: Print exact value of PyTorch tensor (floating point precision)

body:
I'm trying to print torch.FloatTensor like:
a = torch.FloatTensor(3,3)
print(a)

This way I can get a value like:
0.0000e+00  0.0000e+00  3.2286e-41
1.2412e-40  1.2313e+00  1.6751e-37
2.6801e-36  3.5873e-41  9.4463e+21

But I want to get more accurate value, like 10 decimal point:
0.1234567891+01

With other python numerical objects, I could get it with:
print('{:.10f}'.format(a))

but in the case of a tensor, I get this error:
TypeError: unsupported format string passed to torch.FloatTensor.__format__

How can I print more precise values of tensors?


title: How does Pytorch's "Fold" and "Unfold" work?

body:
I've gone through the official doc. I'm having a hard time understanding what this function is used for and how it works. Can someone explain this in Layman terms?

I get an error for the example they provide, although the Pytorch version I'm using matches the documentation. Perhaps fixing the error, which I did, is supposed to teach me something? The snippet given in the documentation is:

   fold = nn.Fold(output_size=(4, 5), kernel_size=(2, 2))
   input = torch.randn(1, 3 * 2 * 2, 1)
   output = fold(input)
   output.size()

and the fixed snippet is:

   fold = nn.Fold(output_size=(4, 5), kernel_size=(2, 2))
   input = torch.randn(1, 3 * 2 * 2, 3 * 2 * 2)
   output = fold(input)
   output.size()

Thanks!


title: Pytorch. How does pin_memory work in Dataloader?

body:
I want to understand how pin_memory in Dataloader works.
According to the documentation:
pin_memory (bool, optional) – If True, the data loader will copy tensors into CUDA pinned memory before returning them.

Below is a self-contained code example.
import torchvision
import torch

print('torch.cuda.is_available()', torch.cuda.is_available())
train_dataset = torchvision.datasets.CIFAR10(root='cifar10_pytorch', download=True, transform=torchvision.transforms.ToTensor())
train_dataloader = torch.utils.data.DataLoader(train_dataset, batch_size=64, pin_memory=True)
x, y = next(iter(train_dataloader))
print('x.device', x.device)
print('y.device', y.device)

Producing the following output:
torch.cuda.is_available() True
x.device cpu
y.device cpu

But I was expecting something like this, because I specified flag pin_memory=True in Dataloader.
torch.cuda.is_available() True
x.device cuda:0
y.device cuda:0

Also I run some benchmark:
import torchvision
import torch
import time
import numpy as np

pin_memory=True
train_dataset =torchvision.datasets.CIFAR10(root='cifar10_pytorch', download=True, transform=torchvision.transforms.ToTensor())
train_dataloader = torch.utils.data.DataLoader(train_dataset, batch_size=64, pin_memory=pin_memory)
print('pin_memory:', pin_memory)
times = []
n_runs = 10
for i in range(n_runs):
    st = time.time()
    for bx, by in train_dataloader:
        bx, by = bx.cuda(), by.cuda()
    times.append(time.time() - st)
print('average time:', np.mean(times))

I got the following results.
pin_memory: False
average time: 6.5701503753662

pin_memory: True
average time: 7.0254474401474

So pin_memory=True only makes things slower.
Can someone explain me this behaviour?


title: How do I load up an image and convert it to a proper tensor for PyTorch?

body:
I'm trying to custom load some image files (JPG files) with some labels and feed them into a convolutional neural network (CNN) in PyTorch following the example here. However, there still seem to be no decent end-to-end tutorials. The problem that I am seeing is the following.

RuntimeError: thnn_conv2d_forward is not implemented for type
torch.ByteTensor

My Dataset looks like the following.

class ImageData(Dataset):
    def __init__(self, width=256, height=256, transform=None):
        self.width = width
        self.height = height
        self.transform = transform
        y, x = get_images() #y is a list of labels, x is a list of file paths
        self.y = y
        self.x = x

    def __getitem__(self, index):
        img = Image.open(self.x[index]) # use pillow to open a file
        img = img.resize((self.width, self.height)) # resize the file to 256x256
        img = img.convert('RGB') #convert image to RGB channel
        if self.transform is not None:
            img = self.transform(img)

        img = np.asarray(img).transpose(-1, 0, 1) # we have to change the dimensions from width x height x channel (WHC) to channel x width x height (CWH)
        img = torch.from_numpy(np.asarray(img)) # create the image tensor
        label = torch.from_numpy(np.asarray(self.y[index]).reshape([1, 1])) # create the label tensor
        return img, label

    def __len__(self):
        return len(self.x)

The CNN is taken from here and is modified to handle NCWH (batch x channel x width x height) as follows.

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 256, 256)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 5 * 5)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

The learning loop is also taken from the same tutorial and looks like the following.

for epoch in range(2):  # loop over the dataset multiple times

    running_loss = 0.0
    for i, data in enumerate(dataloader, 0):
        # get the inputs
        inputs, labels = data

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = net(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()
        if i % 2000 == 1999:    # print every 2000 mini-batches
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 2000))
            running_loss = 0.0

print('Finished Training')

However, the RuntimeError mentioned above is thrown. Any ideas on what I am doing wrong?

Additionally, I know that without transposing the image data, it is in the shape of WHC, but the NN model requires it as CWH. The problem is this, if we change from WHC to CWH, then we can no longer simply plot the images if we iterate over the DataLoader.

data = ImageData()
dataloader = DataLoader(data, batch_size=10, shuffle=True, num_workers=1)
imgs, labels = next(iter(dataloader))
plt.imshow(imgs.numpy()[0,:,:,:])
plt.show()

Attempting to do will throw the following error.

TypeError: Invalid dimensions for image data

To me, that Pillow gives you WHC and you can use that to plot, but the PyTorch CNN wants CWH to process, is a nuisance. Any idea on how to consistently or easily not do so many transforms but be able to plot and feed the data into the CNN? Or is this mismatch of WHC vs CWH just something we have to live with?

Without transposing the image, when feeding it to the CNN, the following error is thrown.

RuntimeError: Given groups=1, weight[256, 3, 256, 256], so expected
input[10, 256, 256, 3] to have 3 channels, but got 256 channels

instead.


title: How do I install PyTorch v1.0.0+ on Google Colab?

body:
PyTorch v1.0.0 stable was released on 8 December 2018 after being announced 7 months earlier.

I want get a version optimised for the hardware that my IPython kernel is running on.

How do I get this version on Google Colab?


I am trying to train a U-Net which looks like this

`class UNet(nn.Module):
def __init__(self, imsize):
    super(UNet, self).__init__()
    self.imsize = imsize

    self.activation = F.relu
    self.pool1 = nn.MaxPool2d(2)
    self.pool2 = nn.MaxPool2d(2)
    self.pool3 = nn.MaxPool2d(2)
    self.pool4 = nn.MaxPool2d(2)
    self.conv_block1_64 = UNetConvBlock(4, 64)
    self.conv_block64_128 = UNetConvBlock(64, 128)
    self.conv_block128_256 = UNetConvBlock(128, 256)
    self.conv_block256_512 = UNetConvBlock(256, 512)
    self.conv_block512_1024 = UNetConvBlock(512, 1024)

    self.up_block1024_512 = UNetUpBlock(1024, 512)
    self.up_block512_256 = UNetUpBlock(512, 256)
    self.up_block256_128 = UNetUpBlock(256, 128)
    self.up_block128_64 = UNetUpBlock(128, 64)

    self.last = nn.Conv2d(64, 1, 1)`

The loss function i am using is

`class BCELoss2d(nn.Module):

def __init__(self, weight=None, size_average=True):
    super(BCELoss2d, self).__init__()
    self.bce_loss = nn.BCELoss(weight, size_average)

def forward(self, logits, targets):
    probs = F.sigmoid(logits)
    probs_flat = probs.view(-1)
    targets_flat = targets.view(-1)
    return self.bce_loss(probs_flat, targets_flat)`

The input image tensor is [1,1,68,68] and labels are also of the same shape

I get this error:

<ipython-input-72-270210759010> in forward(self, x)
 75
 76         block4 = self.conv_block256_512(pool3)
---> 77         pool4 = self.pool4(block4)
     78
  79         block5 = self.conv_block512_1024(pool4)

/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py in _    _call__(self, *input, **kwargs)
    323         for hook in self._forward_pre_hooks.values():
    324             hook(self, input)
 325         result = self.forward(*input, **kwargs)
    326         for hook in self._forward_hooks.values():
    327             hook_result = hook(self, input, result)

/usr/local/lib/python3.5/dist-packages/torch/nn/modules/pooling.py in forward(self, input)
    141         return F.max_pool2d(input, self.kernel_size, self.stride,
    142                             self.padding, self.dilation, self.ceil_mode,
--> 143                             self.return_indices)
    144
    145     def __repr__(self):

/usr/local/lib/python3.5/dist-packages/torch/nn/functional.py in max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode, return_indices)
    332     See :class:`~torch.nn.MaxPool2d` for details.
    333     """
--> 334     ret = torch._C._nn.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)
    335     return ret if return_indices else ret[0]
    336

RuntimeError: Given input size: (128x1x1). Calculated output size: (128x0x0). Output size is too small at /pytorch/torch/lib/THCUNN/generic/SpatialDilatedMaxPooling.cu:69

I'm guessing I'm making a mistake in my channel size or pooling size but i'm not sure where exactly is the mistake.


title: img should be PIL Image. Got

body:
I'm trying to iterate through a loader to check if it's working, however the below error is given:

TypeError: img should be PIL Image. Got <class 'torch.Tensor'>

I've tried adding both transforms.ToTensor() and transforms.ToPILImage() and it gives me an error asking for the opposite. i.e, with ToPILImage(), it will ask for tensor, and vice versa.

# Imports here
%matplotlib inline
import matplotlib.pyplot as plt
from torch import nn, optim
import torch.nn.functional as F
import torch
from torchvision import transforms, datasets, models
import seaborn as sns
import pandas as pd
import numpy as np

data_dir = 'flowers'
train_dir = data_dir + '/train'
valid_dir = data_dir + '/valid'
test_dir = data_dir + '/test'

#Creating transform for training set
train_transforms = transforms.Compose(
[transforms.Resize(255),
transforms.CenterCrop(224),
transforms.ToTensor(),
transforms.RandomHorizontalFlip(),
transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])])

#Creating transform for test set
test_transforms = transforms.Compose(
[transforms.Resize(255),
transforms.CenterCrop(224),
transforms.ToTensor(),
transforms.Normalize([0.485, 0.456, 0.406],[0.229, 0.224, 0.225])])

#transforming for all data
train_data = datasets.ImageFolder(train_dir, transform=train_transforms)
test_data = datasets.ImageFolder(test_dir, transform = test_transforms)
valid_data = datasets.ImageFolder(valid_dir, transform = test_transforms)

#Creating data loaders for test and training sets
trainloader = torch.utils.data.DataLoader(train_data, batch_size = 32,
shuffle = True)
testloader = torch.utils.data.DataLoader(test_data, batch_size=32)
images, labels = next(iter(trainloader))

It should allow me to simply see the image once I run plt.imshow(images[0]), if its working correctly.


title: Loss Function & Its Inputs For Binary Classification PyTorch

body:
I'm trying to write a neural Network for binary classification in PyTorch and I'm confused about the loss function.

I see that BCELoss is a common function specifically geared for binary classification. I also see that an output layer of N outputs for N possible classes is standard for general classification. However, for binary classification it seems like it could be either 1 or 2 outputs.

So, should I have 2 outputs (1 for each label) and then convert my 0/1 training labels into [1,0] and [0,1] arrays, or use something like a sigmoid for a single-variable output?

Here are the relevant snippets of code so you can see:

self.outputs = nn.Linear(NETWORK_WIDTH, 2) # 1 or 2 dimensions?

def forward(self, x):
  # other layers omitted
  x = self.outputs(x)
  return F.log_softmax(x)  # <<< softmax over multiple vars, sigmoid over one, or other?

criterion = nn.BCELoss() # <<< Is this the right function?

net_out = net(data)
loss = criterion(net_out, target) # <<< Should target be an integer label or 1-hot vector?

Thanks in advance.


title: How to run matlab .m files in google colab

body:
I am currently trying to run this repo
https://github.com/Fanziapril/mvfnet
which requires a step:

"Run the Matlab/ModelGeneration/ModelGenerate.m to generate the shape
model "Model_Shape.mat" and copy it to the Matlab/"

Is it possible to run a .m file in colab to do this?

Also, I have looked into oct2py library
https://blink1073.github.io/oct2py/,
but was not able to successfully run the file.
I followed this How to run a MATLAB code on Python


title: No N-dimensional tranpose in PyTorch

body:
PyTorch's torch.transpose function only transposes 2D inputs. Documentation is here.

On the other hand, Tensorflow's tf.transpose function allows you to transpose a tensor of N arbitrary dimensions.

Can someone please explain why PyTorch does not/cannot have N-dimension transpose functionality? Is this due to the dynamic nature of the computation graph construction in PyTorch versus Tensorflow's Define-then-Run paradigm?


title: Why Pytorch officially use mean=[0.485, 0.456, 0.406] and std=[0.229, 0.224, 0.225] to normalize images?

body:
In this page (https://pytorch.org/docs/stable/torchvision/models.html), it says that "All pre-trained models expect input images normalized in the same way, i.e. mini-batches of 3-channel RGB images of shape (3 x H x W), where H and W are expected to be at least 224. The images have to be loaded in to a range of [0, 1] and then normalized using mean = [0.485, 0.456, 0.406] and std = [0.229, 0.224, 0.225]".

Shouldn't the usual mean and std of normalization be [0.5, 0.5, 0.5] and [0.5, 0.5, 0.5]? Why is it setting such strange values?


title: How to import the tensorflow lite interpreter in Python?

body:
I'm developing a Tensorflow embedded application using TF lite on the Raspberry Pi 3b, running Raspbian Stretch.  I've converted the graph to a flatbuffer (lite) format and have built the TFLite static library natively on the Pi.  So far so good.  But the application is Python and there seems to be no Python binding available.  The Tensorflow Lite development guide (https://www.tensorflow.org/mobile/tflite/devguide) states "There are plans for Python bindings and a demo app."  Yet there is wrapper code in /tensorflow/contrib/lite/python/interpreter_wrapper that has all the needed interpreter methods.  Yet calling this from Python eludes me.

I have generated a SWIG wrapper but the build step fails with many errors.  There is no readme.md describing the state of the interpreter_wrapper.  So, I wonder if the wrapper has worked for others and I should persist or is it fundamentally broken and I should look elsewhere (PyTorch)?  Has anyone found a path to the TFLite Python bindings for the Pi3?


title: Why do we call .detach() before calling .numpy() on a Pytorch Tensor?

body:
It has been firmly established that my_tensor.detach().numpy() is the correct way to get a numpy array from a torch tensor.
I'm trying to get a better understanding of why.
In the accepted answer to the question just linked, Blupon states that:

You need to convert your tensor to another tensor that isn't requiring a gradient in addition to its actual value definition.

In the first discussion he links to, albanD states:

This is expected behavior because moving to numpy will break the graph and so no gradient will be computed.
If you don’t actually need gradients, then you can explicitly .detach() the Tensor that requires grad to get a tensor with the same content that does not require grad. This other Tensor can then be converted to a numpy array.

In the second discussion he links to, apaszke writes:

Variable's can’t be transformed to numpy, because they’re wrappers around tensors that save the operation history, and numpy doesn’t have such objects. You can retrieve a tensor held by the Variable, using the .data attribute. Then, this should work: var.data.numpy().

I have studied the internal workings of PyTorch's autodifferentiation library, and I'm still confused by these answers.  Why does it break the graph to to move to numpy? Is it because any operations on the numpy array will not be tracked in the autodiff graph?
What is a Variable? How does it relate to a tensor?
I feel that a thorough high-quality Stack-Overflow answer that explains the reason for this to new users of PyTorch who don't yet understand autodifferentiation is called for here.
In particular, I think it would be helpful to illustrate the graph through a figure and show how the disconnection occurs in this example:

import torch

tensor1 = torch.tensor([1.0,2.0],requires_grad=True)

print(tensor1)
print(type(tensor1))

tensor1 = tensor1.numpy()

print(tensor1)
print(type(tensor1))


title: Creating one hot vector from indices given as a tensor

body:
I have a tensor of size 4 x 6 where 4 is batch size and 6 is sequence length. Every element of the sequence vectors are some index (0 to n). I want to create a 4 x 6 x n tensor where the vectors in 3rd dimension will be one hot encoding of the index which means I want to put 1 in the specified index and rest of the values will be zero.

For example, I have the following tensor:

[[5, 3, 2, 11, 15, 15],
[1, 4, 6, 7, 3, 3],
[2, 4, 7, 8, 9, 10],
[11, 12, 15, 2, 5, 7]]

Here, all the values are in between (0 to n) where n = 15. So, I want to convert the tensor to a 4 X 6 X 16 tensor where the third dimension will represent one hot encoding vector.

How can I do that using PyTorch functionalities? Right now, I am doing this with loop but I want to avoid looping!


title: What's the meaning of function eval() in torch.nn module

body: Official comment shows that "This has any effect only on modules such as Dropout or BatchNorm." But I don't understand its implementation.


title: Printing all the contents of a tensor

body:
I came across this PyTorch tutorial (in neural_networks_tutorial.py) where they construct a simple neural network and run an inference. I would like to print the contents of the entire input tensor for debugging purposes. What I get when I try to print the tensor is something like this and not the entire tensor:

I saw a similar link for numpy but was not sure about what would work for PyTorch. I can convert it to numpy and may be view it, but would like to avoid the extra overhead. Is there a way for me to print the entire tensor?


title: Cross validation for MNIST dataset with pytorch and sklearn

body:
I am new to pytorch and are trying to implement a feed forward neural network to classify the mnist data set. I have some problems when trying to use cross-validation. My data has the following shapes:
 x_train:
torch.Size([45000, 784]) and
y_train: torch.Size([45000])

I tried to use KFold from sklearn.

kfold =KFold(n_splits=10)

Here is the first part of my train method where I'm dividing the data into folds:

for  train_index, test_index in kfold.split(x_train, y_train):
        x_train_fold = x_train[train_index]
        x_test_fold = x_test[test_index]
        y_train_fold = y_train[train_index]
        y_test_fold = y_test[test_index]
        print(x_train_fold.shape)
        for epoch in range(epochs):
         ...

The indices for the y_train_fold variable is right, it's simply:
[   0    1    2 ... 4497 4498 4499], but it's not for x_train_fold, which is [ 4500  4501  4502 ... 44997 44998 44999]. And the same goes for the test folds.

For the first iteration I want the varibale x_train_fold to be the first 4500 pictures, in other words to have the shape torch.Size([4500, 784]), but it has the shape torch.Size([40500, 784])

Any tips on how to get this right?


title: What is the difference between view() and unsqueeze() in Torch?

body:
Use of unsqueeze():
input = torch.Tensor(2, 4, 3) # input: 2 x 4 x 3
print(input.unsqueeze(0).size()) # prints - torch.size([1, 2, 4, 3])

Use of view():
input = torch.Tensor(2, 4, 3) # input: 2 x 4 x 3
print(input.view(1, -1, -1, -1).size()) # prints - torch.size([1, 2, 4, 3])

According to documentation, unsqueeze() inserts singleton dim at position given as parameter and view() creates a view with different dimensions of the storage associated with tensor.
What view() does is clear to me, but I am unable to distinguish it from unsqueeze(). Moreover, I don't understand when to use view() and when to use unsqueeze()?
Any help with good explanation would be appreciated!


title: How can I use LSTM in pytorch for classification?

body:
My code is as below:

class Mymodel(nn.Module):
    def __init__(self, input_size, hidden_size, output_size, num_layers, batch_size):
        super(Discriminator, self).__init__()
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.output_size = output_size
        self.num_layers = num_layers
        self.batch_size = batch_size

        self.lstm = nn.LSTM(input_size, hidden_size)
        self.proj = nn.Linear(hidden_size, output_size)
        self.hidden = self.init_hidden()

    def init_hidden(self):
        return (Variable(torch.zeros(self.num_layers, self.batch_size, self.hidden_size)),
                Variable(torch.zeros(self.num_layers, self.batch_size, self.hidden_size)))

    def forward(self, x):
        lstm_out, self.hidden = self.lstm(x, self.hidden)
        output = self.proj(lstm_out)
        result = F.sigmoid(output)
        return result

I want to use LSTM to classify a sentence to good (1) or bad (0). Using this code, I get the result which is time_step * batch_size * 1 but not 0 or 1. How to edit the code in order to get the classification result?


title: Understanding PyTorch einsum

body: I'm familiar with how einsum works in NumPy. A similar functionality is also offered by PyTorch: torch.einsum(). What are the similarities and differences, either in terms of functionality or performance? The information available at PyTorch documentation is rather scanty and doesn't provide any insights regarding this.


title: pytorch delete model from gpu

body:
I want to make a cross validation in my project based on Pytorch.
And I didn't find any method that pytorch provided to delete the current model and empty the memory of GPU. Could you tell that how can I do it?


title: load a pretrained model pytorch - dict object has no attribute eval

body:
def save_checkpoint(state, is_best, filename='checkpoint.pth.tar'):
    torch.save(state, filename)
    if is_best:
        shutil.copyfile(filename, 'model_best.pth.tar')

save_checkpoint({
                'epoch': epoch + 1,
                'arch': args.arch,
                'state_dict': model.state_dict(),
                'best_prec1': best_prec1,
                'optimizer': optimizer.state_dict()
            }, is_best)

I am saving my model like this. How can I load back the model so that I can use it in other places, like cnn visualization?

This is how I am loading the model now:

torch.load('model_best.pth.tar')

But when I do this, I get this error:

  AttributeError: 'dict' object has no attribute 'eval'

What am I missing here???

EDIT: I want to use the model that I trained to visualize the filters and grads. I am using this repo to make the vis. I replaced line 179 with torch.load('model_best.pth.tar')


title: How to Fix "RuntimeError: CUDA error: device-side assert triggered" in Pytorch

body:
I am trying to train the yolo-v3 model from this repo https://github.com/eriklindernoren/PyTorch-YOLOv3
on my custom dataset of shapes, but I keep getting the error "RuntimeError: CUDA error: device-side assert triggered"

I have tried to lookup the solution and tried several things suggested in different answers (like fixing the indexing of the classes in the annotations) but the error persists.

I am following the description in the readme of the repo to train on a custom dataset, and have adjusted custom.data and the data/custom/ accordingly.

I keep receiving this output.

C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [32,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [33,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [34,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [35,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [36,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [37,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [38,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [39,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [40,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [41,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [42,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [43,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [44,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [45,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [0,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [1,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [2,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [3,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [4,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [5,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [6,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [7,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [12,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [13,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [14,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [15,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [20,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [21,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [22,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [23,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [24,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [25,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [26,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [27,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [28,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [29,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [31,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
Traceback (most recent call last):
  File "train.py", line 105, in <module>
    loss, outputs = model(imgs, targets)
  File "C:\Users\user\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\nn\modules\module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "D:\Documents\GP\Code\TorchYolo\PyTorch-YOLOv3\models.py", line 259, in forward
    x, layer_loss = module[0](x, targets, img_dim)
  File "C:\Users\user\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\nn\modules\module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "D:\Documents\GP\Code\TorchYolo\PyTorch-YOLOv3\models.py", line 188, in forward
    ignore_thres=self.ignore_thres,
  File "D:\Documents\GP\Code\TorchYolo\PyTorch-YOLOv3\utils\utils.py", line 318, in build_targets
    iou_scores[b, best_n, gj, gi] = bbox_iou(pred_boxes[b, best_n, gj, gi], target_boxes, x1y1x2y2=False)
  File "D:\Documents\GP\Code\TorchYolo\PyTorch-YOLOv3\utils\utils.py", line 199, in bbox_iou
    b1_x1, b1_x2 = box1[:, 0] - box1[:, 2] / 2, box1[:, 0] + box1[:, 2] / 2
RuntimeError: CUDA error: device-side assert triggered

with the only thing changing being the "2" in the array index when messing around with the train.jpg class label index

b1_x1, b1_x2 = box1[:, 0] - box1[:, 2] / 2, box1[:, 0] + box1[:, 2] / 2


title: pytorch RuntimeError: Expected object of scalar type Double but got scalar type Float

body:
I am trying to implement a custom dataset for my neural network. But got this error when running the forward function. The code is as follows.

import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.utils.data import Dataset, DataLoader
import numpy as np

class ParamData(Dataset):
    def __init__(self,file_name):
        self.data = torch.Tensor(np.loadtxt(file_name,delimiter = ','))    #first place
    def __len__(self):
        return self.data.size()[0]
    def __getitem__(self,i):
        return self.data[i]

class Net(nn.Module):
    def __init__(self,in_size,out_size,layer_size=200):
        super(Net,self).__init__()
        self.layer = nn.Linear(in_size,layer_size)
        self.out_layer = nn.Linear(layer_size,out_size)

    def forward(self,x):
        x = F.relu(self.layer(x))
        x = self.out_layer(x)
        return x

datafile = 'data1.txt'

net = Net(100,1)
dataset = ParamData(datafile)
n_samples = len(dataset)

#dataset = torch.Tensor(dataset,dtype=torch.double)   #second place
#net.float()                                          #thrid place

net.forward(dataset[0])         #fourth place

In the file data1.txt is a csv formatted text file containing certain numbers, and each dataset[i] is a size 100 by 1 torch.Tensor object of dtype torch.float64. The error message is as follows:

Traceback (most recent call last):
  File "Z:\Wrong.py", line 33, in <module>
    net.forward(dataset[0])
  File "Z:\Wrong.py", line 23, in forward
    x = F.relu(self.layer(x))
  File "E:\Python38\lib\site-packages\torch\nn\modules\module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "E:\Python38\lib\site-packages\torch\nn\modules\linear.py", line 87, in forward
    return F.linear(input, self.weight, self.bias)
  File "E:\Python38\lib\site-packages\torch\nn\functional.py", line 1372, in linear
    output = input.matmul(weight.t())
RuntimeError: Expected object of scalar type Double but got scalar type Float for argument #2 'mat2' in call to _th_mm

It seems that I should change the dtype of the numbers in dataset to torch.double. I tried things like

changing the line at the first place to self.data = torch.tensor(np.loadtxt(file_name,delimiter = ','),dtype=torch.double)
changing the line at the fourth place to net.forward(dataset[0].double())
uncommenting one of the two lines at the second or the thrid place

I think these are the solutions I have seen from similar questions, but they either give new errors or don't do anything. What should I do?

Update: So I got it working by changing the first place to

self.data = torch.from_numpy(np.loadtxt(file_name,delimiter = ',')).float()

which is weird because it is exactly the opposite of the error message. Is this a bug? I'd still like some explaining.


title: Difference between 1 LSTM with num_layers = 2 and 2 LSTMs in pytorch

body:
I am new to deep learning and currently working on using LSTMs for language modeling. I was looking at the pytorch documentation and was confused by it.

If I create a

nn.LSTM(input_size, hidden_size, num_layers)

where hidden_size = 4 and num_layers = 2, I think I will have an architecture something like:

op0    op1 ....
LSTM -> LSTM -> h3
LSTM -> LSTM -> h2
LSTM -> LSTM -> h1
LSTM -> LSTM -> h0
x0     x1 .....

If I do something like

nn.LSTM(input_size, hidden_size, 1)
nn.LSTM(input_size, hidden_size, 1)

I think the network architecture will look exactly like above. Am I wrong? And if yes, what is the difference between these two?


title: Pytorch: Image label

body:
I am working on an image classifier with 31 classes(Office dataset). There is one folder for each of the classes.  I have a python script written using PyTorch that loads the dataset using datasets.ImageFolder and assigns a label to each image and then trains. Here is my code snippet for loading data:

from torchvision import datasets, transforms
import torch

def load_training(root_path, dir, batch_size, kwargs):
    transform = transforms.Compose(
        [transforms.Resize([256, 256]),
         transforms.RandomCrop(224),
         transforms.RandomHorizontalFlip(),
         transforms.ToTensor()])
    data = datasets.ImageFolder(root=root_path + dir, transform=transform)
    train_loader = torch.utils.data.DataLoader(data, batch_size=batch_size, shuffle=True, drop_last=True, **kwargs)
    return train_loader

The code takes each folder, assigns the same label to all images in that folder. Is there any way to find which label is assigned to which image/image folder?


title: PyTorch Dimension out of range (expected to be in range of [-1, 0], but got 1)

body:
I have the following PyTorch tensors:

predicted = torch.tensor([4, 4, 4, 1, 1, 1, 1, 1, 1, 4, 4, 1, 1, 1, 4, 1, 1, 4, 0, 4, 4, 1, 4, 1])

target    = torch.tensor([3, 0, 0, 1, 1, 0, 1, 1, 1, 3, 2, 4, 1, 1, 1, 0, 1, 1, 2, 1, 1, 1, 1, 1,])

I want to compute the Cross Entropy Loss (as part of an Logistic Regression implementation) between them with the following lines:

loss = nn.CrossEntropyLoss()
computed_loss = loss(predicted, target)

However, when my code runs, I get the following IndexError:

IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

Any suggestions on what I'm doing wrong?

/ ##################################################################### /

Here is the full TraceBack:

-----------------------------------------------------------
IndexError                Traceback (most recent call last)
<ipython-input-208-3cdb253d6620> in <module>
      1 batch_size = 1000
      2 train_class = Train((training_set.shape[1]-1), number_of_target_labels, 0.01, 1000)
----> 3 train_class.train_model(training_set, batch_size)

<ipython-input-207-f3e2c7f7979a> in train_model(self, training_data, n_iters)
     42                 out = self.model(x)
     43                 _, predicted = torch.max(out.data, 1)
---> 44                 loss = self.criterion(predicted, y)
     45                 self.optimizer.zero_grad()
     46                 loss.backward()

/anaconda3/envs/malicious_ml/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    491             result = self._slow_forward(*input, **kwargs)
    492         else:
--> 493             result = self.forward(*input, **kwargs)
    494         for hook in self._forward_hooks.values():
    495             hook_result = hook(self, input, result)

/anaconda3/envs/malicious_ml/lib/python3.6/site-packages/torch/nn/modules/loss.py in forward(self, input, target)
    940     def forward(self, input, target):
    941         return F.cross_entropy(input, target, weight=self.weight,
--> 942                                ignore_index=self.ignore_index, reduction=self.reduction)
    943
    944

/anaconda3/envs/malicious_ml/lib/python3.6/site-packages/torch/nn/functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction)
   2054     if size_average is not None or reduce is not None:
   2055         reduction = _Reduction.legacy_get_string(size_average, reduce)
-> 2056     return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
   2057
   2058

/anaconda3/envs/malicious_ml/lib/python3.6/site-packages/torch/nn/functional.py in log_softmax(input, dim, _stacklevel, dtype)
   1348         dim = _get_softmax_dim('log_softmax', input.dim(), _stacklevel)
   1349     if dtype is None:
-> 1350         ret = input.log_softmax(dim)
   1351     else:
   1352         ret = input.log_softmax(dim, dtype=dtype)

/ ##################################################################### /

If you are interested in seeing the rest of my code, here it is:

import torch
import torch.nn as nn
from torch.autograd import Variable

class LogisticRegressionModel(nn.Module):

    def __init__(self, in_dim, num_classes):
        super().__init__()
        self.linear = nn.Linear(in_dim, num_classes)

    def forward(self, x):
        return self.linear(x)

class Train(LogisticRegressionModel):

    def __init__(self, in_dim, num_classes, lr, batch_size):
        super().__init__(in_dim, num_classes)
        self.batch_size = batch_size
        self.learning_rate = lr
        self.input_layer_dim = in_dim
        self.output_layer_dim = num_classes
        self.criterion = nn.CrossEntropyLoss()
        self.model = LogisticRegressionModel(self.input_layer_dim, self.output_layer_dim)
        self.device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
        self.model = self.model.to(self.device)
        self.optimizer = torch.optim.SGD(self.model.parameters(), lr = self.learning_rate)

    def epochs(self, iterations, train_dataset, batch_size):
        epochs = int(iterations/(len(train_dataset)/batch_size))
        return epochs

    def train_model(self, training_data, n_iters):
        batch = self.batch_size
        epochs = self.epochs(n_iters, training_data, batch)
        training_data = torch.utils.data.DataLoader(dataset = training_data, batch_size = batch, shuffle = True)

        for epoch in range(epochs):

            for i, data in enumerate(training_data):

                X_train = data[:, :-1]
                Y_train = data[:, -1]

                if torch.cuda.is_available():
                    x = Variable(torch.Tensor(X_train).cuda())
                    y = Variable(torch.Tensor(Y_train).cuda())

                else:
                    x = Variable(torch.Tensor(X_train.float()))
                    y = Variable(torch.Tensor(Y_train.float()))

                out = self.model(x)
                _, predicted = torch.max(out.data, 1)
                loss = self.criterion(predicted, y)
                self.optimizer.zero_grad()
                loss.backward()
                self.optimizer.step()

                if i % 100 == 0:
                    print('[{}/{}] Loss: {:.6f}'.format(epoch + 1, epochs, loss))


title: How can i process multi loss in pytorch?

body:
Such as this, I want to using some auxiliary loss to promoting my model performance.
Which type code can implement it in pytorch?

#one
loss1.backward()
loss2.backward()
loss3.backward()
optimizer.step()
#two
loss1.backward()
optimizer.step()
loss2.backward()
optimizer.step()
loss3.backward()
optimizer.step()
#three
loss = loss1+loss2+loss3
loss.backward()
optimizer.step()

Thanks for your answer!


title: How to compute the cosine_similarity in pytorch for all rows in a matrix with respect to all rows in another matrix

body:
In pytorch, given that I have 2 matrixes how would I compute cosine similarity of all rows in each with all rows in the other.

For example

Given the input =

matrix_1 = [a b]
           [c d]
matrix_2 = [e f]
           [g h]

I would like the output to be

output =

 [cosine_sim([a b] [e f])  cosine_sim([a b] [g h])]
 [cosine_sim([c d] [e f])  cosine_sim([c d] [g h])]

At the moment I am using torch.nn.functional.cosine_similarity(matrix_1, matrix_2) which returns the cosine of the row with only that corresponding row in the other matrix.

In my example I have only 2 rows, but I would like a solution which works for many rows. I would even like to handle the case where the number of rows in the each matrix is different.

I realize that I could use the expand, however I want to do it without using such a large memory footprint.


title: Resolved package not found

body:
When I try to execute conda env create -f virtual_platform_windows.yml
It shows

ResolvePackageNotFound:
  - pytorch==0.1.12=py35_0.1.12cu80

I tried installing pytorch for windows and error still comes.How to solve this??


title: How to install older version of pytorch

body:
Following this https://pytorch.org/get-started/previous-versions/#via-pip

pip install torch==0.2.0_4 -f https://download.pytorch.org/whl/cpu/stable
Collecting torch==0.2.0_4
  Could not find a version that satisfies the requirement torch==0.2.0_4 (from versions: 0.1.2, 0.1.2.post1, 0.3.1, 0.4.0, 0.4.1, 1.0.0, 1.0.1, 1.0.1.post2, 1.1.0)
No matching distribution found for torch==0.2.0_4

How to install older version of pytorch?


First of all i tried command from their main page, that they gave me:

pip3 install torch==1.3.1+cpu torchvision==0.4.2+cpu -f https://download.pytorch.org/whl/torch_stable.html

Could not find a version that satisfies the requirement torch==1.3.1+cpu (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)
ERROR: No matching distribution found for torch==1.3.1.+cpu

After this i decided to take available from this list: https://download.pytorch.org/whl/cpu/stable

So at the end i tried something like this

pip3 install torch-1.1.0-cp37-cp37m-win_amd64.whl -f https://download.pytorch.org/whl/torch_stable.html

And now they write that this is not supported wheel on my platform. Wtf?
( I use windows 7, python64, have amd)
( location of python: C:\Python38, location of pip C:\Python38\Scripts )


title: PyTorch RuntimeError: DataLoader worker (pid(s) 15332) exited unexpectedly

body:
I am a beginner at PyTorch and I am just trying out some examples on this webpage. But I can't seem to get the 'super_resolution' program running due to this error:

RuntimeError: DataLoader worker (pid(s) 15332) exited unexpectedly

I searched the Internet and found that some people suggest setting num_workers to 0. But if I do that, the program tells me that I am running out of memory (either with CPU or GPU):

RuntimeError: [enforce fail at ..\c10\core\CPUAllocator.cpp:72] data. DefaultCPUAllocator: not enough memory: you tried to allocate 9663676416 bytes. Buy new RAM!

or

RuntimeError: CUDA out of memory. Tried to allocate 1024.00 MiB (GPU 0; 4.00 GiB total capacity; 2.03 GiB already allocated; 0 bytes free; 2.03 GiB reserved in total by PyTorch)

How do I fix this?

I am using python 3.8 on Win10(64bit) and pytorch 1.4.0.

More complete error messages (--cuda means using GPU, --threads x means passing x to the num_worker parameter):

with command line arguments --upscale_factor 1 --cuda

  File "E:\Python38\lib\site-packages\torch\utils\data\dataloader.py", line 761, in _try_get_data
    data = self._data_queue.get(timeout=timeout)
  File "E:\Python38\lib\multiprocessing\queues.py", line 108, in get
    raise Empty
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "Z:\super_resolution\main.py", line 81, in <module>
    train(epoch)
  File "Z:\super_resolution\main.py", line 48, in train
    for iteration, batch in enumerate(training_data_loader, 1):
  File "E:\Python38\lib\site-packages\torch\utils\data\dataloader.py", line 345, in __next__
    data = self._next_data()
  File "E:\Python38\lib\site-packages\torch\utils\data\dataloader.py", line 841, in _next_data
    idx, data = self._get_data()
  File "E:\Python38\lib\site-packages\torch\utils\data\dataloader.py", line 808, in _get_data
    success, data = self._try_get_data()
  File "E:\Python38\lib\site-packages\torch\utils\data\dataloader.py", line 774, in _try_get_data
    raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 16596, 9376, 12756, 9844) exited unexpectedly

with command line arguments --upscale_factor 1 --cuda --threads 0

  File "Z:\super_resolution\main.py", line 81, in <module>
    train(epoch)
  File "Z:\super_resolution\main.py", line 52, in train
    loss = criterion(model(input), target)
  File "E:\Python38\lib\site-packages\torch\nn\modules\module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "Z:\super_resolution\model.py", line 21, in forward
    x = self.relu(self.conv2(x))
  File "E:\Python38\lib\site-packages\torch\nn\modules\module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "E:\Python38\lib\site-packages\torch\nn\modules\conv.py", line 345, in forward
    return self.conv2d_forward(input, self.weight)
  File "E:\Python38\lib\site-packages\torch\nn\modules\conv.py", line 341, in conv2d_forward
    return F.conv2d(input, weight, self.bias, self.stride,
RuntimeError: CUDA out of memory. Tried to allocate 1024.00 MiB (GPU 0; 4.00 GiB total capacity; 2.03 GiB already allocated; 954.35 MiB free; 2.03 GiB reserved in total by PyTorch)


title: Calling cuda() with async results in SyntaxError

body:
I'm trying to run this PyTorch code:

for i, (input, target) in enumerate(train_loader):

    input = input.float().cuda(async=True)
    target = target.cuda(async=True)
    input_var = torch.autograd.Variable(input)
    target_var = torch.autograd.Variable(target)

    output = model(input_var)

But when I try I am getting this error message:

input = input.float().cuda(async=True)
                               ^
SyntaxError: invalid syntax
Process finished with exit code 1

What am I doing wrong? I already installed cuda.


title: PyTorch softmax with dim

body:
Which dimension should softmax be applied to ?

This code :

%reset -f

import torch.nn as nn
import numpy as np
import torch

my_softmax = nn.Softmax(dim=-1)

mu, sigma = 0, 0.1 # mean and standard deviation

train_dataset = []
image = []
image_x = np.random.normal(mu, sigma, 24).reshape((3 , 4, 2))
train_dataset.append(image_x)

x = torch.tensor(train_dataset).float()

print(x)
print(my_softmax(x))
my_softmax = nn.Softmax(dim=1)
print(my_softmax(x))

prints following :

tensor([[[[-0.1500,  0.0243],
          [ 0.0226,  0.0772],
          [-0.0180, -0.0278],
          [ 0.0782, -0.0853]],

         [[-0.0134, -0.1139],
          [ 0.0385, -0.1367],
          [-0.0447,  0.1493],
          [-0.0633, -0.2964]],

         [[ 0.0123,  0.0061],
          [ 0.1086, -0.0049],
          [-0.0918, -0.1308],
          [-0.0100,  0.1730]]]])
tensor([[[[ 0.4565,  0.5435],
          [ 0.4864,  0.5136],
          [ 0.5025,  0.4975],
          [ 0.5408,  0.4592]],

         [[ 0.5251,  0.4749],
          [ 0.5437,  0.4563],
          [ 0.4517,  0.5483],
          [ 0.5580,  0.4420]],

         [[ 0.5016,  0.4984],
          [ 0.5284,  0.4716],
          [ 0.5098,  0.4902],
          [ 0.4544,  0.5456]]]])
tensor([[[[ 0.3010,  0.3505],
          [ 0.3220,  0.3665],
          [ 0.3445,  0.3230],
          [ 0.3592,  0.3221]],

         [[ 0.3450,  0.3053],
          [ 0.3271,  0.2959],
          [ 0.3355,  0.3856],
          [ 0.3118,  0.2608]],

         [[ 0.3540,  0.3442],
          [ 0.3509,  0.3376],
          [ 0.3200,  0.2914],
          [ 0.3289,  0.4171]]]])

So first tensor is prior to softmax being applied, second tensor is result of softmax applied to tensor with dim=-1 and third tensor is result of softmax applied to tensor with dim=1 .

For result of first softmax can see corresponding elements sum to 1, for example [ 0.4565,  0.5435] -> 0.4565 + 0.5435 == 1.

What is summing to 1 as result of of second softmax ?

Which dim value should I choose ?

Update : The dimension (3 , 4, 2) corresponds to image dimension where 3 is the RGB value , 4 is the number of horizontal pixels (width) , 2 is the number of vertical pixels (height). This is an image classification problem. I'm using cross entropy loss function. Also, I'm using softmax in final layer in order to back-propagate probabilities.


title: cuDNN error: CUDNN_STATUS_BAD_PARAM.Can someone explain why i am getting this error and how can i correct it?

body:
I am trying to implement a Character LSTM using Pytorch.But I am getting cudnn_status_bad_params errors.This is the training loop.I getting error on line output = model(input_seq).

for epoch in tqdm(range(epochs)):
  for i in range(len(seq)//batch_size):
   sidx = i*batch_size
   eidx = sidx + batch_size
   x = seq[sidx:eidx]
   x = torch.tensor(x).cuda()
   input_seq =torch.nn.utils.rnn.pack_padded_sequence(x,seq_lengths,batch_first = True)
   y = out_seq[sidx:eidx]
   output = model(input_seq)
   loss = criterion(output,y)
   loss.backward()
   optimizer.step()
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    487             result = self._slow_forward(*input, **kwargs)
    488         else:
--> 489             result = self.forward(*input, **kwargs)
    490         for hook in self._forward_hooks.values():
    491             hook_result = hook(self, input, result)
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/rnn.py in forward(self, input, hx)
    180         else:
    181             result = _impl(input, batch_sizes, hx, self._flat_weights, self.bias,
--> 182                            self.num_layers, self.dropout, self.training, self.bidirectional)
    183         output = result[0]
    184         hidden = result[1:] if self.mode == 'LSTM' else result[1]

 RuntimeError: cuDNN error: CUDNN_STATUS_BAD_PARAM


title: No module named 'torch' or 'torch.C'

body:
Would appreciate an explanation like I'm 5 simply because I have checked all relevant answers and none have helped.

I have installed Python.
I have installed Pycharm.
I have installed Anaconda.
I have installed Microsoft Visual Studio.
I have not installed the CUDA toolkit.

In Anaconda, I used the commands mentioned on Pytorch.org (06/05/18)

conda install pytorch -c pytorch
pip3 install torchvision

Both have downloaded and installed properly, and I can find them in my Users/Anaconda3/pkgs folder, which I have added to the Python path.

Trying to enter

import torch

in the Python console proved unfruitful - always giving me the same error,

  No module named 'torch'

I have also tried using the Project Interpreter to download the Pytorch package. It worked for numpy (sanity check, I suppose) but told me to go to Pytorch.org when I tried to install the "pytorch" or "torch" packages.

When trying to use the console in PyCharm, pip3 install codes (thinking maybe I need to save the packages into my current project, rather than in the Anaconda folder) return me an error message saying

  torch-0.4.0-cp35-cp35m-win_amd64.whl is not a supported wheel on this
  platform.

The same message shows no matter if I try downloading the CUDA version or not, or if I choose to use the 3.5 or 3.6 Python link (I have Python 3.7)

Currently the closest I have gotten to a solution, is manually copying the "torch" and "torch-0.4.0-py3.6.egg-info" folders into my current Project's lib folder.
However, when I do that and then run "import torch" I received the following error:

  Traceback (most recent call last):

  File "", line 1, in

  File "C:\Program Files\JetBrains\PyCharm Community Edition 2018.1.2\helpers\pydev_pydev_bundle\pydev_import_hook.py", line 19, in do_import

  module = self._system_import(name, *args, **kwargs)   File "C:\Users\Michael\PycharmProjects\Pytorch_2\venv\lib\site-packages\torch__init__.py",

  line 78, in

  from torch._C import *

  File "C:\Program Files\JetBrains\PyCharm Community Edition 2018.1.2\helpers\pydev_pydev_bundle\pydev_import_hook.py", line 19, in do_import

  module = self._system_import(name, *args, **kwargs) ModuleNotFoundError: No module named 'torch._C'


title: PyTorch - "Attribute Error: module 'torch' has no attribute 'float'

body:
I'm trying to implement the Spatial Transformer Network from here and I am running into this issue:

class STNLayer(torch.nn.Module):
    def __init__(self, input_size):
        super(STNLayer, self).__init__()
        self.input_size = input_size
        self.localization = nn.Sequential(
            nn.Conv2d(self.input_size, 8, kernel_size = 7),
            nn.MaxPool2d(2, stride = 2),
            nn.ReLU(True),
            nn.Conv2d(8, 10, kernel_size = 5),
            nn.MaxPool2d(2, stride = 2),
            nn.ReLU(True)
        )

        self.fc_loc = nn.Sequential(
            nn.Linear(10 * 12 * 12, 32),
            nn.ReLU(True),
            #nn.BatchNorm1d(32),
            nn.Linear(32, 3*2)
        )

        # Initialize weights to identity transformation
        self.fc_loc[2].weight.data.zero_()
        self.fc_loc[2].bias.data = torch.cuda.FloatTensor([1,0,0,0,1,0])

The line

self.fc_loc[2].bias.data = torch.cuda.FloatTensor([1,0,0,0,1,0])

is giving the error:

*** AttributeError: module 'torch' has no attribute 'float'

How do I fix this?


title: Pytorch loss inf nan

body:
I'm trying to do simple linear regression with 1 feature. It's a simple 'predict salary given years experience' problem.
The NN trains on years experience (X) and a salary (Y).
For some reason the loss is exploding and ultimately returns inf or nan

This is the code I have:

import torch
import torch.nn as nn
import pandas as pd
import numpy as np

dataset = pd.read_csv('./salaries.csv')

x_temp = dataset.iloc[:, :-1].values
y_temp = dataset.iloc[:, 1:].values

X_train = torch.FloatTensor(x_temp)
Y_train = torch.FloatTensor(y_temp)

class Model(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = torch.nn.Linear(1,1)

    def forward(self, x):
        y_pred = self.linear(x)
        return y_pred

model = Model()

loss_func = torch.nn.MSELoss(size_average=False)
optim = torch.optim.SGD(model.parameters(), lr=0.01)

#training
for epoch in range(200):
    #calculate y_pred
    y_pred = model(X_train)

    #calculate loss
    loss = loss_func(y_pred, Y_train)
    print(epoch, "{:.2f}".format(loss.data))

    #backward pass + update weights
    optim.zero_grad()
    loss.backward()
    optim.step()

test_exp = torch.FloatTensor([[8.0]])
print("8 years experience --> ", model(test_exp).data[0][0].item())

As I mentioned, once it starts training the loss gets super big and ends up showing inf after like the 10th epoch.

I suspect it may have something to do with how i'm loading the data? This is what is in salaries.csv file:

Years Salary
1.1 39343
1.3 46205
1.5 37731
2   43525
2.2 39891
2.9 56642
3   60150
3.2 54445
3.2 64445
3.7 57189
3.9 63218
4   55794
4   56957
4.1 57081
4.5 61111
4.9 67938
5.1 66029
5.3 83088

Thank you for your help


title: How can I enable pytorch GPU support in Google Colab?

body:
How can I enable pytorch to work on GPU?

I've installed pytorch successfully in google colab notebook:

Tensorflow reports GPU to be in place:

But torch.device function fails somehow:

How can I fix this?


title: Google Colab: "Unable to connect to the runtime" after uploading Pytorch model from local

body:
I am using a simple (not necessarily efficient) method for Pytorch model saving.

import torch
from google.colab import files

torch.save(model, filename) # save a trained model on the VM
files.download(filename) # download the model to local

best_model = files.upload() # select the model just downloaded
best_model[filename] # access the model

Colab disconnects during execution of the last line, and hitting RECONNECT tab always shows ALLOCATING -> CONNECTING (fails, with "unable to connect to the runtime" message in the left bottom corner) -> RECONNECT. At the same time, executing any one of the cells gives Error message "Failed to execute cell, Could not send execute message to runtime: [object CloseEvent]"

I know it is related to the last line, because I can successfully connect with my other google accounts which doesn't execute that.

Why does it happen? It seems the google accounts which have executed the last line can no longer connect to the runtime.

Edit:

One night later, I can reconnect with the google account after session expiration. I just attempted the approach in the comment, and found that just files.upload() the Pytorch model would lead to the problem. Once the upload completes, Colab disconnects.


title: PyTorch: access weights of a specific module in nn.Sequential()

body:
When I use a pre-defined module in PyTorch, I can typically access its weights fairly easily. However, how do I access them if I wrapped the module in nn.Sequential() first? r.g:
class My_Model_1(nn.Module):
    def __init__(self,D_in,D_out):
        super(My_Model_1, self).__init__()
        self.layer = nn.Linear(D_in,D_out)
    def forward(self,x):
        out = self.layer(x)
        return out

class My_Model_2(nn.Module):
    def __init__(self,D_in,D_out):
        super(My_Model_2, self).__init__()
        self.layer = nn.Sequential(nn.Linear(D_in,D_out))
    def forward(self,x):
        out = self.layer(x)
        return out

model_1 = My_Model_1(10,10)
print(model_1.layer.weight)
model_2 = My_Model_2(10,10)

How do I print the weights now?
model_2.layer.0.weight doesn't work.


title: Add / substract between matrix and vector in pytorch

body:
I want to do + / - / *  between matrix and vector in pytorch.
How can I do with good performance?
I tried to use expand, but it's really slow (I am using big matrix with small vector).

a = torch.rand(2,3)
print(a)
 0.7420  0.2990  0.3896
 0.0715  0.6719  0.0602
[torch.FloatTensor of size 2x3]
b = torch.rand(2)
print(b)
 0.3773
 0.6757
[torch.FloatTensor of size 2]
a.add(b)
Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 3066, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-17-a1cb1b03d031>", line 1, in <module>
    a.add(b)
RuntimeError: inconsistent tensor size, expected r_ [2 x 3], t [2 x 3] and src [2] to have the same number of elements, but got 6, 6 and 2 elements respectively at c:\miniconda2\conda-bld\pytorch-cpu_1519449358620\work\torch\lib\th\generic/THTensorMath.c:1021

Expected result:

 0.7420-0.3773  0.2990-0.3773  0.3896-0.3773
 0.0715-0.6757  0.6719-0.6757  0.0602-0.6757


title: Cross entropy loss in pytorch nn.CrossEntropyLoss()

body:
maybe someone is able to help me here. I am trying to compute the cross entropy loss of a given output of my network

print output
Variable containing:
1.00000e-02 *
-2.2739  2.9964 -7.8353  7.4667  4.6921  0.1391  0.6118  5.2227  6.2540
-7.3584
[torch.FloatTensor of size 1x10]

and the desired label, which is of the form

print lab
Variable containing:
x
[torch.FloatTensor of size 1]

where x is an integer between 0 and 9.
According to the pytorch documentation (http://pytorch.org/docs/master/nn.html)

criterion = nn.CrossEntropyLoss()
loss = criterion(output, lab)

this should work, but unfortunately I get a weird error

TypeError: FloatClassNLLCriterion_updateOutput received an invalid combination of arguments - got (int, torch.FloatTensor, !torch.FloatTensor!, torch.FloatTensor, bool, NoneType, torch.FloatTensor, int), but expected (int state, torch.FloatTensor input, torch.LongTensor target, torch.FloatTensor output, bool sizeAverage, [torch.FloatTensor weights or None], torch.FloatTensor total_weight, int ignore_index)

Can anyone help me? I am really confused and tried almost everything I could imagined to be helpful.

Best


title: What is the difference between model.to(device) and model=model.to(device)?

body:
Suppose the model is originally stored on CPU, and then I want to move it to GPU0, then I can do:

device = torch.device('cuda:0')
model = model.to(device)
# or
model.to(device)

What is the difference between those two lines?


title: Get the data type of a PyTorch tensor

body:
I understand that PyTorch tensors are homogenous, ie, each of the elements are of the same type.

How do I find out the type of the elements in a PyTorch tensor?


title: Autograd.grad() for Tensor in pytorch

body:
I want to compute the gradient between two tensors in a net. The input X tensor (batch size x m) is sent through a set of convolutional layers which give me back and output Y tensor(batch size x n).
I’m creating a new loss and I would like to know the gradient of Y w.r.t. X. Something that in tensorflow would be like:
tf.gradients(ys=Y, xs=X)
Unfortunately, I’ve been making tests with torch.autograd.grad(), but I could not figure out how to do it. I get errors like: “RunTimeerror: grad can be implicitly created only for scalar outputs”.
What should be the inputs in torch.autograd.grad() if I want to know the gradient of Y w.r.t. X?


title: Bidirectional LSTM output question in PyTorch

body:
Hi I have a question about how to collect the correct result from a BI-LSTM module’s output.

Suppose I have a 10-length sequence feeding into a single-layer LSTM module with 100 hidden units:

lstm = nn.LSTM(5, 100, 1, bidirectional=True)

output will be of shape:

[10 (seq_length), 1 (batch),  200 (num_directions * hidden_size)]
# or according to the doc, can be viewed as
[10 (seq_length), 1 (batch),  2 (num_directions), 100 (hidden_size)]

If I want to get the 3rd (1-index) input’s output at both directions (two 100-dim vectors), how can I do it correctly?

I know output[2, 0] will give me a 200-dim vector. Does this 200 dim vector represent the output of 3rd input at both directions?

A thing bothering me is that when do reverse feeding, the 3rd (1-index) output vector is calculated from the 8th(1-index) input, right?

Will pytorch automatically take care of this and group output considering direction?

Thanks!


title: PyTorch NotImplementedError in forward

body:
import torch
import torch.nn as nn

device = torch.device('cuda' if torch.cuda.is_available() else
'cpu')

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.layer = nn.Sequential(
            nn.Conv2d(1, 16, kernel_size=3, stride=1, padding=1),
            nn.BatchNorm2d(16),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2), # 16x16x650
            nn.Conv2d(16, 32, kernel_size=3, stride=1, padding=1), # 32x16x650
            nn.ReLU(),
            nn.Dropout2d(0.5),
            nn.Conv2d(32, 64, kernel_size=3, stride=1, padding=1), # 64x16x650
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2), # 64x8x325
            nn.Conv2d(64, 128, kernel_size=3, stride=1, padding=1),
            nn.ReLU()) # 64x8x325

        self.fc = nn.Sequential(
            nn.Linear(64*8*325, 128),
            nn.ReLU(),
            nn.Linear(128, 256),
            nn.ReLU(),
            nn.Linear(256, 1),
        )

        def forward(self, x):
            out = self.layer1(x)
            out = self.layer2(out)
            out = out.reshape(out.size(0), -1)
            out = self.fc(out)
            return out

# HYPERPARAMETER
learning_rate = 0.0001
num_epochs = 15

import data

def main():
    model = Model().to(device)

    criterion = nn.CrossEntropyLoss()
    optimizer = torch.optim.Adam(model.parameters(),
lr=learning_rate)

    total_step = len(data.train_loader)
    for epoch in range(num_epochs):
        for i, (images, labels) in enumerate(data.train_loader):
            images = images.to(device)
            labels = labels.to(device)

            outputs = model(images)
            loss = criterion(outputs, labels)

            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

        if (i + 1) % 100 == 0:
            print('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}'
                  .format(epoch + 1, num_epochs, i + 1, total_step, loss.item()))

    model.eval()
    with torch.no_grad():
        correct = 0
        total = 0
        for images, labels in data.test_loader:
            images = images.to(device)
            labels = labels.to(device)
            outputs = model(images)
            _, predicted = torch.max(outputs.data, 1)
            total += labels.size(0)
            correct += (predicted == labels).sum().item()

        print('Test Accuracy of the model on the 10000 test images: {} %'.format(100 * correct / total))

if __name__ == '__main__':
    main()

Error:

File "/home/rladhkstn8/Desktop/SWID/tmp/pycharm_project_853/model.py", line 82, in <module>
    main()
  File "/home/rladhkstn8/Desktop/SWID/tmp/pycharm_project_853/model.py", line 56, in main
    outputs = model(images)
  File "/home/rladhkstn8/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/rladhkstn8/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 83, in forward
    raise NotImplementedError
NotImplementedError

I do not know where the problem is. I know that NotImplementedError should be implemented, but it happens when there is unimplemented code.


title: How Pytorch do row normalization for each matrix in a 3D Tensor(Variable)?

body:
If I have a 3D Tensor (Variable) with size [a,b,c].
consider it as a b*c matrix, and I hope that all these a matrix got row normalized.


title: How to fix 'Input and hidden tensors are not at the same device' in pytorch

body:
When I want to put the model on the GPU, I get the following error:

"RuntimeError: Input and hidden tensors are not at the same device, found input tensor at cuda:0 and hidden tensor at cpu"

However, all of the above had been put on the GPU:
for m in model.parameters():
    print(m.device) #return cuda:0

if torch.cuda.is_available():
    model = model.cuda()
    test = test.cuda() # test is the Input

Windows 10 server
Pytorch 1.2.0 + cuda 9.2
cuda 9.2
cudnn 7.6.3 for cuda 9.2


title: How to get entire dataset from dataloader in PyTorch

body:
How to load entire dataset from the DataLoader? I am getting only one batch of dataset.

This is my code

dataloader = torch.utils.data.DataLoader(dataset=dataset, batch_size=64)
images, labels = next(iter(dataloader))


title: Error in pip install torchvision on Windows 10

body:
on pytorch, installing on Windows 10, conda and Cuda 9.0.

cmd did not complain when i ran conda install pytorch cuda90 -c pytorch, then when I ran pip3 install torchvision I get this error message.

Requirement already satisfied: torchvision in PATHTOFILE\python35\lib\site-packages (0.2.1)
Requirement already satisfied: numpy in PATHTOFILE\python35\lib\site-packages (from torchvision) (1.12.0+mkl)
Requirement already satisfied: six in PATHTOFILE\python35\lib\site-packages (from torchvision) (1.10.0)
Collecting pillow>=4.1.1 (from torchvision)
  Using cached https://files.pythonhosted.org/packages/ab/d2/d27a21bd3e64db1ca1dc7dc16026a16d77f5c3ffca9ec619eddeea7c47ce/Pillow-5.1.0-cp35-cp35m-win_amd64.whl
Collecting torch (from torchvision)
  Using cached https://files.pythonhosted.org/packages/5f/e9/bac4204fe9cb1a002ec6140b47f51affda1655379fe302a1caef421f9846/torch-0.1.2.post1.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\Users\USERNAME~1\AppData\Local\Temp\pip-install-a70g611u\torch\setup.py", line 11, in <module>
        raise RuntimeError(README)
    RuntimeError: PyTorch does not currently provide packages for PyPI (see status at https://github.com/pytorch/pytorch/issues/566).

    Please follow the instructions at http://pytorch.org/ to install with miniconda instead.

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in C:\Users\USERNAME~1\AppData\Local\Temp\pip-install-a70g611u\torch\

Anyone got this error?


title: PyTorch autograd -- grad can be implicitly created only for scalar outputs

body:
I am using the autograd tool in PyTorch, and have found myself in a situation where I need to access the values in a 1D tensor by means of an integer index. Something like this:

def basic_fun(x_cloned):
    res = []
    for i in range(len(x)):
        res.append(x_cloned[i] * x_cloned[i])
    print(res)
    return Variable(torch.FloatTensor(res))

def get_grad(inp, grad_var):
    A = basic_fun(inp)
    A.backward()
    return grad_var.grad

x = Variable(torch.FloatTensor([1, 2, 3, 4, 5]), requires_grad=True)
x_cloned = x.clone()
print(get_grad(x_cloned, x))

I am getting the following error message:

[tensor(1., grad_fn=<ThMulBackward>), tensor(4., grad_fn=<ThMulBackward>), tensor(9., grad_fn=<ThMulBackward>), tensor(16., grad_fn=<ThMulBackward>), tensor(25., grad_fn=<ThMulBackward>)]
Traceback (most recent call last):
  File "/home/mhy/projects/pytorch-optim/predict.py", line 74, in <module>
    print(get_grad(x_cloned, x))
  File "/home/mhy/projects/pytorch-optim/predict.py", line 68, in get_grad
    A.backward()
  File "/home/mhy/.local/lib/python3.5/site-packages/torch/tensor.py", line 93, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/mhy/.local/lib/python3.5/site-packages/torch/autograd/__init__.py", line 90, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

I am in general, a bit skeptical about how using the cloned version of a variable is supposed to keep that variable in gradient computation. The variable itself is effectively not used in the computation of A, and so when you call A.backward(), it should not be part of that operation.

I appreciate your help with this approach or if there is a better way to avoid losing the gradient history and still index through a 1D tensor with requires_grad=True!

**Edit (September 15):**

res is a list of zero-dimensional tensors containing squared values of 1 to 5. To concatenate in a single tensor containing [1.0, 4.0, ..., 25.0], I changed return Variable(torch.FloatTensor(res)) to torch.stack(res, dim=0), which produces tensor([ 1.,  4.,  9., 16., 25.], grad_fn=<StackBackward>).

However, I am getting this new error, caused by the A.backward() line.

Traceback (most recent call last):
  File "<project_path>/playground.py", line 22, in <module>
    print(get_grad(x_cloned, x))
  File "<project_path>/playground.py", line 16, in get_grad
    A.backward()
  File "/home/mhy/.local/lib/python3.5/site-packages/torch/tensor.py", line 93, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/mhy/.local/lib/python3.5/site-packages/torch/autograd/__init__.py", line 84, in backward
    grad_tensors = _make_grads(tensors, grad_tensors)
  File "/home/mhy/.local/lib/python3.5/site-packages/torch/autograd/__init__.py", line 28, in _make_grads
    raise RuntimeError("grad can be implicitly created only for scalar outputs")
RuntimeError: grad can be implicitly created only for scalar outputs


title: pytorch embedding index out of range

body:
I'm following this tutorial here https://cs230-stanford.github.io/pytorch-nlp.html. In there a neural model is created, using nn.Module, with an embedding layer, which is initialized here

self.embedding = nn.Embedding(params['vocab_size'], params['embedding_dim'])

vocab_size is the total number of training samples, which is 4000.  embedding_dim is 50.  The relevant piece of the forward method is below

def forward(self, s):
        # apply the embedding layer that maps each token to its embedding
        s = self.embedding(s)   # dim: batch_size x batch_max_len x embedding_dim

I get this exception when passing a batch to the model like so
model(train_batch)
train_batch is a numpy array of dimension batch_sizexbatch_max_len.  Each sample is a sentence, and each sentence is padded so that it has the length of the longest sentence in the batch.

  File
  "/Users/liam_adams/Documents/cs512/research_project/custom/model.py",
  line 34, in forward
      s = self.embedding(s)   # dim: batch_size x batch_max_len x embedding_dim   File
  "/Users/liam_adams/Documents/cs512/venv_research/lib/python3.7/site-packages/torch/nn/modules/module.py",
  line 493, in call
      result = self.forward(*input, **kwargs)   File "/Users/liam_adams/Documents/cs512/venv_research/lib/python3.7/site-packages/torch/nn/modules/sparse.py",
  line 117, in forward
      self.norm_type, self.scale_grad_by_freq, self.sparse)   File "/Users/liam_adams/Documents/cs512/venv_research/lib/python3.7/site-packages/torch/nn/functional.py",
  line 1506, in embedding
      return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse) RuntimeError: index out of range at
  ../aten/src/TH/generic/THTensorEvenMoreMath.cpp:193

Is the problem here that the embedding is initialized with different dimensions than those of my batch array?  My batch_size will be constant but batch_max_len will change with every batch. This is how its done in the tutorial.


I am training a CNN model. I am facing issue while doing the training iteration for my model. The code is as below:

class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()

        #convo layers
        self.conv1 = nn.Conv2d(3,32,3)
        self.conv2 = nn.Conv2d(32,64,3)
        self.conv3 = nn.Conv2d(64,128,3)
        self.conv4 = nn.Conv2d(128,256,3)
        self.conv5 = nn.Conv2d(256,512,3)

        #pooling layer
        self.pool = nn.MaxPool2d(2,2)

        #linear layers
        self.fc1 = nn.Linear(512*5*5,2048)
        self.fc2 = nn.Linear(2048,1024)
        self.fc3 = nn.Linear(1024,133)

        #dropout layer
        self.dropout = nn.Dropout(0.3)
        def forward(self, x):
        #first layer
        x = self.conv1(x)
        x = F.relu(x)
        x = self.pool(x)
        #x = self.dropout(x)
        #second layer
        x = self.conv2(x)
        x = F.relu(x)
        x = self.pool(x)
        #x = self.dropout(x)
        #third layer
        x = self.conv3(x)
        x = F.relu(x)
        x = self.pool(x)
        #x = self.dropout(x)
        #fourth layer
        x = self.conv4(x)
        x = F.relu(x)
        x = self.pool(x)
        #fifth layer
        x = self.conv5(x)
        x = F.relu(x)
        x = self.pool(x)
        #x = self.dropout(x)

        #reshape tensor
        x = x.view(-1,512*5*5)
        #last layer
        x = self.dropout(x)
        x = self.fc1(x)
        x = F.relu(x)
        x = self.dropout(x)
        x = self.fc2(x)
        x = F.relu(x)
        x = self.fc3(x)

        return x

        #loss func
        criterion = nn.MSELoss()
        optimizer = optim.Adam(net.parameters(), lr = 0.0001)
        #criterion = nn.CrossEntropyLoss()
        #optimizer = optim.SGD(net.parameters(), lr = 0.05)

        def train(n_epochs,model,loader,optimizer,criterion,save_path):
           for epoch in range(n_epochs):
              train_loss = 0
              valid_loss = 0
              #training
              net.train()
              for batch, (data,target) in enumerate(loaders['train']):
                   optimizer.zero_grad()
                   outputs = net(data)
                   #print(outputs.shape)
                   loss = criterion(outputs,target)
                   loss.backward()
                   optimizer.step()

When I use the CrossEntropy Loss function and SGD optimizer, I able able to train the model with no error.
When I use MSE loss function and Adam optimizer, I am facing the following error:

RuntimeError Traceback (most recent call last) <ipython-input-20-2223dd9058dd> in <module>
      1 #train the model
      2 n_epochs = 2
----> 3 train(n_epochs,net,loaders,optimizer,criterion,'saved_model/dog_model.pt')

<ipython-input-19-a93d145ef9f7> in train(n_epochs, model, loader, optimizer, criterion, save_path)
     22
     23             #calculate loss
---> 24             loss = criterion(outputs,target)
     25
     26             #backward prop

RuntimeError: The size of tensor a (133) must match the size of tensor b (10) at non-singleton dimension 1.

Does the selected loss function and optimizer effect the training of the model? Can anyone please help on this?


title: Is there really no padding=same option for PyTorch's Conv2d?

body:
I'm currently working on building a convolutional neural network (CNN) that will work on time series data.
More specifically, the data is financial data of shape (100, 40), with 100 meaning 100 time stamps and 40 meaning 40 features.
The CNN that I'm using uses asymmetric kernel sizes (i.e. 1 x 2 and 4 x 1) and also asymmetric strides (i.e. 1 x 2 for the 1 x 2 layers and 1 x 1 for the 4 x 1 layers).
In order to maintain the height dimension to stay 100, I needed to apply some padding to the data. I was looking into ways to do this and noticed that people who use TensorFlow or Keras simply do padding='same', but this option is apparently unavailable in PyTorch according to many resources I've found including this thread on Github.
What I've found is that according to some answers in this question and also this answer on the PyTorch discussion forum, I can manually calculate how I need to pad my data and can use torch.nn.ZeroPad2d to solve my problem, as it seems that normal torch.nn.Conv2d layers don't support asymmetric padding (I believe that the total padding I need is 3 in height and 0 in width).
The experimental code that I've written to test this out is like this:
import torch
import torch.nn as nn

conv = nn.Conv2d(1, 1, kernel_size=(4, 1))
pad = nn.ZeroPad2d((0, 0, 2, 1)) # Add 2 to top and 1 to bottom.

x = torch.randint(low=0, high=9, size=(100, 40))
x = x.unsqueeze(0).unsqueeze(0)

y = pad(x)

x.shape # (1, 1, 100, 40)
y.shape # (1, 1, 103, 40)

print(conv(x.float()).shape)
print(conv(y.float()).shape)

# Output
# x -> (1, 1, 97, 40)
# y -> (1, 1, 100, 40)

As you can see, it does work in the sense that the size of dimensions is remaining the same. However, I've been wondering is there really no padding='same' option out there? Also, how do we know whether to apply the padding 2 to the top or to the bottom?
Thank you.

Edit
This is a little late, but in case anyone's curious how I solved this issue I basically manually added padding in order to emulate the padding=same option.


title: PyTorch BERT TypeError: forward() got an unexpected keyword argument 'labels'

body:
Training a BERT model using PyTorch transformers (following the tutorial here).

Following statement in the tutorial

loss = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask, labels=b_labels)

leads to

TypeError: forward() got an unexpected keyword argument 'labels'

Here is the full error,

TypeError                                 Traceback (most recent call last)
<ipython-input-53-56aa2f57dcaf> in <module>
     26         optimizer.zero_grad()
     27         # Forward pass
---> 28         loss = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask, labels=b_labels)
     29         train_loss_set.append(loss.item())
     30         # Backward pass

~/anaconda3/envs/systreviewclassifi/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    539             result = self._slow_forward(*input, **kwargs)
    540         else:
--> 541             result = self.forward(*input, **kwargs)
    542         for hook in self._forward_hooks.values():
    543             hook_result = hook(self, input, result)

TypeError: forward() got an unexpected keyword argument 'labels'

I cant seem to figure out what kind of argument the forward() function expects.

There is a similar problem here, but I still do not get what the solution is.

System information:

OS: Ubuntu 16.04 LTS
Python version: 3.6.x
Torch version: 1.3.0
Torch Vision version: 0.4.1
PyTorch transformers version: 1.2.0


title: Random Choice with Pytorch?

body:
I have a tensor of pictures, and would like to randomly select from it. I'm looking for the equivalent of np.random.choice().

import torch

pictures = torch.randint(0, 256, (1000, 28, 28, 3))

Let's say I want 10 of these pictures.