title: Pytorch, pipenv and numpy support?

body:
I am using pytorch and installed it follow these instructions:

pipenv install git+https://github.com/pytorch/pytorch#egg=pytorch
pipenv install git+https://github.com/pytorch/vision#egg=torchvision

in order to install pytorch with pipenv. This runs some of the examples, but for instance, this MNIST code:

from __future__ import print_function
import argparse
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torchvision import datasets, transforms
from torch.autograd import Variable

# Training settings
parser = argparse.ArgumentParser(description='PyTorch MNIST Example')
parser.add_argument('--batch-size', type=int, default=64, metavar='N',
                    help='input batch size for training (default: 64)')
parser.add_argument('--test-batch-size', type=int, default=1000, metavar='N',
                    help='input batch size for testing (default: 1000)')
parser.add_argument('--epochs', type=int, default=10, metavar='N',
                    help='number of epochs to train (default: 10)')
parser.add_argument('--lr', type=float, default=0.01, metavar='LR',
                    help='learning rate (default: 0.01)')
parser.add_argument('--momentum', type=float, default=0.5, metavar='M',
                    help='SGD momentum (default: 0.5)')
parser.add_argument('--no-cuda', action='store_true', default=False,
                    help='disables CUDA training')
parser.add_argument('--seed', type=int, default=1, metavar='S',
                    help='random seed (default: 1)')
parser.add_argument('--log-interval', type=int, default=10, metavar='N',
                    help='how many batches to wait before logging training status')
args = parser.parse_args()
args.cuda = not args.no_cuda and torch.cuda.is_available()

torch.manual_seed(args.seed)
if args.cuda:
    torch.cuda.manual_seed(args.seed)

kwargs = {'num_workers': 1, 'pin_memory': True} if args.cuda else {}
train_loader = torch.utils.data.DataLoader(
    datasets.MNIST('../data', train=True, download=True,
                   transform=transforms.Compose([
                       transforms.ToTensor(),
                       transforms.Normalize((0.1307,), (0.3081,))
                   ])),
    batch_size=args.batch_size, shuffle=True, **kwargs)
test_loader = torch.utils.data.DataLoader(
    datasets.MNIST('../data', train=False, transform=transforms.Compose([
                       transforms.ToTensor(),
                       transforms.Normalize((0.1307,), (0.3081,))
                   ])),
    batch_size=args.test_batch_size, shuffle=True, **kwargs)

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 10, kernel_size=5)
        self.conv2 = nn.Conv2d(10, 20, kernel_size=5)
        self.conv2_drop = nn.Dropout2d()
        self.fc1 = nn.Linear(320, 50)
        self.fc2 = nn.Linear(50, 10)

    def forward(self, x):
        x = F.relu(F.max_pool2d(self.conv1(x), 2))
        x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))
        x = x.view(-1, 320)
        x = F.relu(self.fc1(x))
        x = F.dropout(x, training=self.training)
        x = self.fc2(x)
        return F.log_softmax(x, dim=1)

model = Net()
if args.cuda:
    model.cuda()

optimizer = optim.SGD(model.parameters(), lr=args.lr, momentum=args.momentum)

def train(epoch):
    model.train()
    for batch_idx, (data, target) in enumerate(train_loader):
        if args.cuda:
            data, target = data.cuda(), target.cuda()
        data, target = Variable(data), Variable(target)
        optimizer.zero_grad()
        output = model(data)
        loss = F.nll_loss(output, target)
        loss.backward()
        optimizer.step()
        if batch_idx % args.log_interval == 0:
            print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                epoch, batch_idx * len(data), len(train_loader.dataset),
                100. * batch_idx / len(train_loader), loss.data[0]))

def test():
    model.eval()
    test_loss = 0
    correct = 0
    for data, target in test_loader:
        if args.cuda:
            data, target = data.cuda(), target.cuda()
        data, target = Variable(data, volatile=True), Variable(target)
        output = model(data)
        test_loss += F.nll_loss(output, target, size_average=False).data[0] # sum up batch loss
        pred = output.data.max(1, keepdim=True)[1] # get the index of the max log-probability
        correct += pred.eq(target.data.view_as(pred)).long().cpu().sum()

    test_loss /= len(test_loader.dataset)
    print('\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\n'.format(
        test_loss, correct, len(test_loader.dataset),
        100. * correct / len(test_loader.dataset)))

for epoch in range(1, args.epochs + 1):
    train(epoch)
    test()

Does not run giving the error:

RuntimeError: PyTorch was compiled without NumPy support

Running:

pipenv install numpy

does not resolve the error. Is there a way to use pipenv and install pytorch with numpy support?


title: Pytorch Runtime Error - The size of tensor a (5) must match the size of tensor b (3) at non-singleton dimension

body:
I am trying to train a Faster RCNN Network on a custom dataset consisting of images for object detection. However, I don't want to directly give an RGB image as input, I actually need to pass it through another network (a feature extractor) along with the corresponding thermal image and give the extracted features as the input to the FRCNN Network. The feature extractor combines these two images into a 4 channel tensor and the output is a 5 channel tensor. It is this 5 channel tensor that I wish to give as input to the Faster RCNN Network.

I followed the PyTorch docs for Object Detection Finetuning (link here) and came up with the following code to suit my dataset.

class CustomDataset(torch.utils.data.Dataset):

    def __getitem__(self, idx):
        self.num_classes = 5
        img_rgb_path = os.path.join(self.root, "rgb/", self.rgb_imgs[idx])
        img_thermal_path = os.path.join(self.root, "thermal/", self.thermal_imgs[idx])

        img_rgb = Image.open(img_rgb_path)
        img_rgb = np.array(img_rgb)
        x_rgb = TF.to_tensor(img_rgb)
        x_rgb.unsqueeze_(0)

        img_thermal = Image.open(img_thermal_path)
        img_thermal = np.array(img_thermal)
        img_thermal = np.expand_dims(img_thermal,-1)
        x_th = TF.to_tensor(img_thermal)
        x_th.unsqueeze_(0)

        print(x_rgb.shape)  # shape of [3,640,512]
        print(x_th.shape) # shape of [1,640,512]

        input = torch.cat((x_rgb,x_th),dim=1) # shape of [4,640,512]

        img = self.feature_extractor(input) #  My custom feature extractor which returns a 5 dimensional tensor

        print(img.shape) # shape of [5,640,512]

        filename = os.path.join(self.root,'annotations',self.annotations[idx])
        tree = ET.parse(filename)
        objs = tree.findall('object')

        num_objs = len(objs)
        boxes = np.zeros((num_objs, 4), dtype=np.uint16)
        labels = np.zeros((num_objs), dtype=np.float32)
        seg_areas = np.zeros((num_objs), dtype=np.float32)

        boxes = []
        for ix, obj in enumerate(objs):
            bbox = obj.find('bndbox')
            x1 = float(bbox.find('xmin').text)
            y1 = float(bbox.find('ymin').text)
            x2 = float(bbox.find('xmax').text)
            y2 = float(bbox.find('ymax').text)

            cls = self._class_to_ind[obj.find('name').text.lower().strip()]
            boxes.append([x1, y1, x2, y2])
            labels[ix] = cls
            seg_areas[ix] = (x2 - x1 + 1) * (y2 - y1 + 1)

        boxes = torch.as_tensor(boxes, dtype=torch.float32)
        seg_areas = torch.as_tensor(seg_areas, dtype=torch.float32)
        labels = torch.as_tensor(labels, dtype=torch.float32)

        target =  {'boxes': boxes,
                'labels': labels,
                'seg_areas': seg_areas,
                }

        return img,target

My main function code is as follows

import utils

def train_model(model, criterion,dataloader,num_epochs):
    since = time.time()

    best_model = model
    best_acc = 0.0

    for epoch in range(num_epochs):
        print('Epoch {}/{}'.format(epoch, num_epochs - 1))
        print('-' * 10)

        optimizer = torch.optim.SGD(params, lr=0.005,
                            momentum=0.9, weight_decay=0.0005)

        lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer,
                                               step_size=3,
                                               gamma=0.1)

        # optimizer = lr_scheduler(optimizer, epoch)
        model.train()  # Set model to training mode

        running_loss = 0.0
        running_corrects = 0

        for data in dataloader:
            inputs, labels = data[0][0], data[1]

            inputs = inputs.to(device)
            # zero the parameter gradients

            optimizer.zero_grad()

            # forward
            outputs = model(inputs, labels)
            _, preds = torch.max(outputs.data, 1)
            loss = criterion(outputs, labels)

            loss.backward()
            optimizer.step()

            running_loss += loss.item()
            running_corrects += torch.sum(preds == labels).item()

        epoch_loss = running_loss / len(dataloader)
        epoch_acc = running_corrects / len(dataloader)

        print('{} Loss: {:.4f} Acc: {:.4f}'.format(
            phase, epoch_loss, epoch_acc))

backbone = torchvision.models.mobilenet_v2(pretrained=True).features
backbone.out_channels = 1280

anchor_generator = AnchorGenerator(sizes=((32, 64, 128, 256, 512),),
                                   aspect_ratios=((0.5, 1.0, 2.0),))

roi_pooler = torchvision.ops.MultiScaleRoIAlign(featmap_names=[0],
                                                output_size=7,
                                                sampling_ratio=2)
num_classes = 5

model = FasterRCNN(backbone = backbone,num_classes=5,rpn_anchor_generator=anchor_generator,box_roi_pool=roi_pooler)

dataset = CustomDataset('train_folder/')
data_loader_train = torch.utils.data.DataLoader(dataset, batch_size=1, shuffle=True,collate_fn=utils.collate_fn)

train_model(model, criterion, data_loader_train, num_epochs=10)

The collate_fn defined in the utils.py file is the following

def collate_fn(batch):
    return tuple(zip(*batch))

I, however, get the following error while training

Traceback (most recent call last):
  File "train.py", line 147, in <module>
    train_model(model, criterion, data_loader_train, num_epochs)
  File "train.py", line 58, in train_model
    outputs = model(inputs, labels)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torchvision/models/detection/generalized_rcnn.py", line 66, in forward
    images, targets = self.transform(images, targets)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torchvision/models/detection/transform.py", line 46, in forward
    image = self.normalize(image)
  File "/usr/local/lib/python3.6/dist-packages/torchvision/models/detection/transform.py", line 66, in normalize
    return (image - mean[:, None, None]) / std[:, None, None]
RuntimeError: The size of tensor a (5) must match the size of tensor b (3) at non-singleton dimension 0

I am a newbie in Pytorch.


title: BCELoss for binary pixel-wise segmentation pytorch

body:
I'm implementing a UNet for binary segmentation while using Sigmoid and BCELoss. The problem is that after several iterations the network tries to predict very small values per pixel  while for some regions it should predict values close to one (for ground truth mask region). Does it give any intuition about the wrong behavior?

Besides, there exist NLLLoss2d which is used for pixel-wise loss. Currently, I'm simply ignoring this and I'm using MSELoss() directly. Should I use NLLLoss2d with Sigmoid activation layer?

Thanks


title: How to change PyTorch tensor into a half size and/or double size with different dimension?

body:
I'm new to PyTorch and tensor data thing. I have a problem about switching shape of tensors.

I have two questions.

First, what should I do if I have a tensor with torch.Size([8, 512, 16, 16]) and I want to change it into torch.Size([8, 256, 32, 32]) which is double size of the original tensor.

Second, what should I do if I have a tensor with torch.Size([8, 256, 32, 32]) and I want to change it into torch.Size([8, 512, 16, 16]) which is half size of the original tensor.

In the first question, I've tried on ZeroPadding2D(8) function to reshape it into torch.Size([8, 512, 32, 32]) but I don't know how to change the 2nd dimension which is 512 into 256.

The actual usage in the first question is something like this.

x = input # torch.Size([8, 512, 16, 16])
x = layer(x) # torch.Size([8, 256, 32, 32]
x = x + input # what I want to do is adding tensor values before and after passing the layer together (like skip connection)

I expect the output of adding two tensors to be success but the actual output is an error about unequal size in dimensions


title: Modifying a pytorch tensor and then getting the gradient lets the gradient not work

body:
I am a beginner in pytorch and I face the following issue:

When I get the gradient of the below tensor (note that I use some variable x in some way as you can see below), I get the gradient:

import torch
myTensor = torch.randn(2, 2,requires_grad=True)
with torch.enable_grad():
    x=myTensor.sum() *10
x.backward()
print(myTensor.grad)

Now, if I try to modify an element of myTensor, I get the error of leaf variable has been moved into the graph interior. See this code:

import torch
myTensor = torch.randn(2, 2,requires_grad=True)
myTensor[0,0]*=5
with torch.enable_grad():
    x=myTensor.sum() *10
x.backward()
print(myTensor.grad)

What is wrong with my latter code? And how do I correct it?

Any help would be highly appreciated. Thanks a lot!


title: how to convert series numpy array into tensors using pytorch

body:
I am trying to convert image labels convert into tensor, but  I got some error please help me to convert to tensor:
Here My code:

features_train, features_test, targets_train, targets_test = train_test_split(X,Y,test_size=0.2,
                                                                              random_state=42)
X_train = torch.from_numpy(features_train)
X_test = torch.from_numpy(features_test)

Y_train =torch.from_numpy(targets_train).type(torch.IntTensor)
Y_test = torch.from_numpy(targets_test).type(torch.IntTensor)
train = torch.utils.data.TensorDataset(X_train,Y_train)
test = torch.utils.data.TensorDataset(X_test,Y_test)

train_loader = torch.utils.data.DataLoader(train, batch_size = train_batch_size, shuffle = False)
test_loader = torch.utils.data.DataLoader(test, batch_size = test_batch_size, shuffle = False)

Here my error:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-32-f1578581ff5c> in <module>()
      5 X_test = torch.from_numpy(features_test)
      6
----> 7 Y_train =torch.from_numpy(targets_train).type(torch.IntTensor)
      8 Y_test = torch.from_numpy(targets_test).type(torch.IntTensor)
      9 train = torch.utils.data.TensorDataset(X_train,Y_train)

TypeError: expected np.ndarray (got Series)

Here my array values:

targets_train
478     1
5099    3
1203    2
5674    2
142     1
4836    2
4031    1
1553    3
4416    1
605     5
1194    3
4319    4
1498    5


I was running a deep learning program on my Linux server and I suddenly got this error.
UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 804: forward compatibility was attempted on non supported HW (Triggered internally at  /opt/conda/conda-bld/pytorch_1603729096996/work/c10/cuda/CUDAFunctions.cpp:100.)
Earlier when I just created this conda environment, torch.cuda.is_available() returned true and I could use CUDA & GPU. But all of a sudden I could not use CUDA and torch.cuda.is_available()returned false. What should I do?
ps. I use GeForce RTX 3080 and cuda 11.0 + pytorch 1.7.0. It worked before but now it doesn't.


title: Error with _DataLoaderIter in torch.utils.data.dataloader

body:
I want to run a code which needs to import _DataLoaderIter from torch.utils.data.dataloader. By checking the source code for dataloader class, that method exist. However, I get the error:

  Traceback (most recent call last):

  File "main.py", line 4, in

  import data

  File "D:\Hyperspectral Data\RCAN\RCAN_TrainCode\code\data\__init__.py", line 3, in module

  from dataloader import MSDataLoader

  File "D:\Hyperspectral Data\RCAN\RCAN_TrainCode\code\dataloader.py", line 14, in module

  from torch.utils.data.dataloader import _DataLoaderIter

  ImportError: cannot import name '_DataLoaderIter'

Why is this happening?


title: How to add parameters in module class in pytorch custom model?

body:
I tried to find the answer but I can't.

I make a custom deep learning model using pytorch. For example,

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()

        self.nn_layers = nn.ModuleList()
        self.layer = nn.Linear(2,3).double()
        torch.nn.init.xavier_normal_(self.layer.weight)

        self.bias = torch.nn.Parameter(torch.randn(3))

        self.nn_layers.append(self.layer)

    def forward(self, x):
        activation = torch.tanh
        output = activation(self.layer(x)) + self.bias

        return output

If I print

model = Net()
print(list(model.parameters()))

it does not contains model.bias, so
optimizer = optimizer.Adam(model.parameters()) does not update model.bias.
How can I go through this?
Thanks!


title: Pytorch batch matrix vector outer product

body:
I am trying to generate a vector-matrix outer product (tensor) using PyTorch. Assuming the vector v has size p and the matrix M has size qXr, the result of the product should be pXqXr.

Example:

#size: 2
v = [0, 1]
#size: 2X3
M = [[0, 1, 2],
     [3, 4, 5]]
#size: 2X2X3
v*M = [[[0, 0, 0],
        [0, 0, 0]],
       [[0, 1, 2],
        [3, 4, 5]]]

For two vectors v1 and v2, I can use torch.bmm(v1.view(1, -1, 1), v2.view(1, 1, -1)). This can be easily extended for a batch of vectors. However, I am not able to find a solution for vector-matrix case. Also, I need to do this operation for batches of vectors and matrices.


title: PyTorch: CrossEntropyLoss, changing class weight does not change the computed loss

body:
According to Doc for cross entropy loss, the weighted loss is calculated by multiplying the weight for each class and the original loss.

However, in the pytorch implementation, the class weight seems to have no effect on the final loss value unless it is set to zero. Following is the code:

from torch import nn
import torch

logits = torch.FloatTensor([
    [0.1, 0.9],
])
label = torch.LongTensor([0])

criterion = nn.CrossEntropyLoss(weight=torch.FloatTensor([1, 1]))
loss = criterion(logits, label)
print(loss.item()) # result: 1.1711

# Change class weight for the first class to 0.1
criterion = nn.CrossEntropyLoss(weight=torch.FloatTensor([0.1, 1]))
loss = criterion(logits, label)
print(loss.item()) # result: 1.1711, should be 0.11711

# Change weight for first class to 0
criterion = nn.CrossEntropyLoss(weight=torch.FloatTensor([0, 1]))
loss = criterion(logits, label)
print(loss.item()) # result: 0

As illustrated in the code, the class weight seems to have no effect unless it is set to 0, this behavior contradicts to the documentation.

Updates
I implemented a version of weighted cross entropy which is in my eyes the "correct" way to do it.

import torch
from torch import nn

def weighted_cross_entropy(logits, label, weight=None):
    assert len(logits.size()) == 2
    batch_size, label_num = logits.size()
    assert (batch_size == label.size(0))

    if weight is None:
        weight = torch.ones(label_num).float()

    assert (label_num == weight.size(0))

    x_terms = -torch.gather(logits, 1, label.unsqueeze(1)).squeeze()
    log_terms = torch.log(torch.sum(torch.exp(logits), dim=1))

    weights = torch.gather(weight, 0, label).float()

    return torch.mean((x_terms+log_terms)*weights)

logits = torch.FloatTensor([
    [0.1, 0.9],
    [0.0, 0.1],

])

label = torch.LongTensor([0, 1])

neg_weight = 0.1

weight = torch.FloatTensor([neg_weight, 1])

criterion = nn.CrossEntropyLoss(weight=weight)
loss = criterion(logits, label)

print(loss.item()) # results: 0.69227
print(weighted_cross_entropy(logits, label, weight).item()) # results: 0.38075

What I did is to multiply each instance in the batch with its associated class weight. The result is still different from the original pytorch implementation, which makes me wonder how pytorch actually implement this.


title: Why do I get CUDA out of memory when running PyTorch model [with enough GPU memory]?

body:
I am asking this question because I am successfully training a segmentation network on my GTX 2070 on laptop with 8GB VRAM and I use exactly the same code and exactly the same software libraries installed on my desktop PC with a GTX 1080TI and it still throws out of memory.
Why does this happen, considering that:

The same Windows 10 + CUDA 10.1 + CUDNN 7.6.5.32 + Nvidia Driver 418.96 (comes along with CUDA 10.1) are both on laptop and on PC.

The fact that training with TensorFlow 2.3 runs smoothly on the GPU on my PC, yet it fails allocating memory for training only with PyTorch.

PyTorch recognises the GPU (prints GTX 1080 TI) via the command : print(torch.cuda.get_device_name(0))

PyTorch allocates memory when running this command: torch.rand(20000, 20000).cuda() #allocated 1.5GB of VRAM.

What is the solution to this?


title: While converting a PIL image into a tensor why the pixels are changing?

body:
transform = transforms.Compose([transforms.ToPILImage(), transforms.ToTensor()])

Before applying the transformation

After applying the transformation

Q.1 Why the pixel values are changed?
Q.2 How to correct this?


Traning the image classification with pytorch  got following
  error messageK

  RuntimeError                              Traceback (most recent call
  last)  in
       29         print(len(train_loader.dataset),len(valid_loader.dataset))
       30         #break
  ---> 31         train_loss, train_acc ,model= train(model, device, train_loader, optimizer, criterion)
       32         valid_loss, valid_acc,model = evaluate(model, device, valid_loader, criterion)
       33

   in train(model, device, iterator,
  optimizer, criterion)
       21         acc = calculate_accuracy(fx, y)
       22         #print("5.")
  ---> 23         loss.backward()
       24
       25         optimizer.step()

  ~/venv/lib/python3.7/site-packages/torch/tensor.py in backward(self,
  gradient, retain_graph, create_graph)
      164                 products. Defaults to False.
      165         """
  --> 166         torch.autograd.backward(self, gradient, retain_graph, create_graph)
      167
      168     def register_hook(self, hook):

  ~/venv/lib/python3.7/site-packages/torch/autograd/init.py in
  backward(tensors, grad_tensors, retain_graph, create_graph,
  grad_variables)
       97     Variable._execution_engine.run_backward(
       98         tensors, grad_tensors, retain_graph, create_graph,
  ---> 99         allow_unreachable=True)  # allow_unreachable flag
      100
      101

  RuntimeError: cuda runtime error (710) : device-side assert triggered
  at /pytorch/aten/src/THC/generic/THCTensorMath.cu:26

Related code block is here

def train(model, device, iterator, optimizer, criterion):

print('train')
epoch_loss = 0
epoch_acc = 0

model.train()

for (x, y) in iterator:
    #print(x,y)
    x,y = x.cuda(), y.cuda()
    #x = x.to(device)
    #y = y.to(device)
    #print('1')
    optimizer.zero_grad()
    #print('2')
    fx = model(x)
    #print('3')
    loss = criterion(fx, y)
    #print("4.loss->",loss)
    acc = calculate_accuracy(fx, y)
    #print("5.")
    loss.backward()

    optimizer.step()

    epoch_loss += loss.item()
    epoch_acc += acc.item()

return epoch_loss / len(iterator), epoch_acc / len(iterator),model

    EPOCHS = 5
    SAVE_DIR = 'models'
    MODEL_SAVE_PATH = os.path.join(SAVE_DIR, 'please.pt')
    from torch.utils.data import DataLoader
    best_valid_loss = float('inf')

    if not os.path.isdir(f'{SAVE_DIR}'):
        os.makedirs(f'{SAVE_DIR}')
    print("start")
    for epoch in range(EPOCHS):
        print('================================',epoch ,'================================')
        for i , (train_idx, valid_idx) in enumerate(zip(train_indexes, valid_indexes)):
            print(i,train_idx,valid_idx,len(train_idx),len(valid_idx))

            traindf = df_train.iloc[train_index, :].reset_index()
            validdf = df_train.iloc[valid_index, :].reset_index()

            #traindf = df_train
            #validdf = df_train

            train_dataset = TrainDataset(traindf, mode='train', transforms=data_transforms)
            valid_dataset = TrainDataset(validdf, mode='valid', transforms=data_transforms)

            train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
            valid_loader = DataLoader(valid_dataset, batch_size=batch_size, shuffle=False)

            print(len(train_loader.dataset),len(valid_loader.dataset))
            #break
            train_loss, train_acc ,model= train(model, device, train_loader, optimizer, criterion)
            valid_loss, valid_acc,model = evaluate(model, device, valid_loader, criterion)

            if valid_loss < best_valid_loss:
                best_valid_loss = valid_loss
                torch.save(model,MODEL_SAVE_PATH)

            print(f'| Epoch: {epoch+1:02} | Train Loss: {train_loss:.3f} | Train Acc: {train_acc*100:05.2f}% | Val. Loss: {valid_loss:.3f} | Val. Acc: {valid_acc*100:05.2f}% |')

  splits = zip(train_indexes, valid_indexes)
  [ 3692  3696  3703 ... 30733 30734 30735] [   0    1    2 ... 4028
  4041 4046] [    0     1     2 ... 30733 30734 30735] [3692 3696 3703
  ... 7986 7991 8005] [    0     1     2 ... 30733 30734 30735] [ 7499
  7500  7502 ... 11856 11858 11860] [    0     1     2 ... 30733 30734
  30735] [11239 11274 11280 ... 15711 15716 15720] [    0     1     2
  ... 30733 30734 30735] [15045 15051 15053 ... 19448 19460 19474] [
  0     1     2 ... 30733 30734 30735] [18919 18920 18926 ... 23392
  23400 23402] [    0     1     2 ... 30733 30734 30735] [22831 22835
  22846 ... 27118 27120 27124] [    0     1     2 ... 27118 27120 27124]
  [26718 26721 26728 ... 30733 30734 30735]


title: How to set python environment variables in VS Code?

body:
I know how to add arguments for the Python script I want to run. For example, if test.py is my script file and it has one argument like '--batch_size' then I can edit launch.json in VS Code and set "args": ["--batch_size", "32"]

But I don't know how to add environmental arguments for Python itself. For example, Python has -m environmental variable, which runs library module as a script. If I want to run python -m torch.distributed.launch test.py --batch_size 32, what should I edit in VS Code to run the debugger?

UPDATE:
Here is my launch.json


title: Moving a tensor to cuda device cause illegal memory access in Pytorch

body:
I am trying the following snippet in Colab but causes the following error.
Is it wrong to move a tensor object to Cuda device?.
import torch
a = torch.Tensor(torch.randn(5,5,5))
# a.device("cuda")
device = torch.device("cuda")
class abc(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.w1 = torch.nn.Linear(5,5)

    def forward(self,x):
        return self.w1(x)
mod = abc()
a.cuda()
mod.to(device)
mod(a)

Output:
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-10-5372fb4d5512> in <module>()
     11         return self.w1(x)
     12 mod = abc()
---> 13 a.cuda()
     14 mod.to(device)
     15 mod(a)

RuntimeError: CUDA error: an illegal memory access was encountered


Why does the following error popup? What should be in this range and why? What does -2 dimension mean?

RuntimeError: dimension out of range (expected to be in range of [-2, 1], but got 2)

This code will produce the error

import torch

torch.bmm(torch.randn(1000, 784) , torch.randn(784, 10))


title: Is hidden and output the same for a GRU unit in Pytorch?

body:
I do understand conceptually what an LSTM or GRU should (thanks to this question What's the difference between "hidden" and "output" in PyTorch LSTM?) BUT when I inspect the output of the GRU h_n and output are NOT the same while they should be...

(Pdb) rnn_output
tensor([[[ 0.2663,  0.3429, -0.0415,  ...,  0.1275,  0.0719,  0.1011],
         [-0.1272,  0.3096, -0.0403,  ...,  0.0589, -0.0556, -0.3039],
         [ 0.1064,  0.2810, -0.1858,  ...,  0.3308,  0.1150, -0.3348],
         ...,
         [-0.0929,  0.2826, -0.0554,  ...,  0.0176, -0.1552, -0.0427],
         [-0.0849,  0.3395, -0.0477,  ...,  0.0172, -0.1429,  0.0153],
         [-0.0212,  0.1257, -0.2670,  ..., -0.0432,  0.2122, -0.1797]]],
       grad_fn=<StackBackward>)
(Pdb) hidden
tensor([[[ 0.1700,  0.2388, -0.4159,  ..., -0.1949,  0.0692, -0.0630],
         [ 0.1304,  0.0426, -0.2874,  ...,  0.0882,  0.1394, -0.1899],
         [-0.0071,  0.1512, -0.1558,  ..., -0.1578,  0.1990, -0.2468],
         ...,
         [ 0.0856,  0.0962, -0.0985,  ...,  0.0081,  0.0906, -0.1234],
         [ 0.1773,  0.2808, -0.0300,  ..., -0.0415, -0.0650, -0.0010],
         [ 0.2207,  0.3573, -0.2493,  ..., -0.2371,  0.1349, -0.2982]],

        [[ 0.2663,  0.3429, -0.0415,  ...,  0.1275,  0.0719,  0.1011],
         [-0.1272,  0.3096, -0.0403,  ...,  0.0589, -0.0556, -0.3039],
         [ 0.1064,  0.2810, -0.1858,  ...,  0.3308,  0.1150, -0.3348],
         ...,
         [-0.0929,  0.2826, -0.0554,  ...,  0.0176, -0.1552, -0.0427],
         [-0.0849,  0.3395, -0.0477,  ...,  0.0172, -0.1429,  0.0153],
         [-0.0212,  0.1257, -0.2670,  ..., -0.0432,  0.2122, -0.1797]]],
       grad_fn=<StackBackward>)

they are some transpose of each other...why?


title: Understanding PyTorch Bernoulli distribution from the documention

body:
So I was reading the pytorch document trying to learn and understand somethings(because I'm new to the machine learning ), I found the torch.bernoulli() and I understood (I miss understood it) that it approximates the tensors that the have the values between 1 and 0 to 1 or 0 depends on the value (like classic school less than 0.5 = 0 , more the than or equals 0.5 = 1)

After some experimentations on my own that yeah it works as expected

 >>>y = torch.Tensor([0.500])
 >>>x
 >>>  0.5000
     [torch.FloatTensor of size 1]
 >>> torch.bernoulli(x)
 >>> 1
     [torch.FloatTensor of size 1]

But when I looked at the document something a bit weird

>>> a = torch.Tensor(3, 3).uniform_(0, 1) # generate a uniform random matrix with range [0, 1]
>>> a

 0.7544  0.8140  0.9842
**0.5282** 0.0595  0.6445
 0.1925  0.9553  0.9732
[torch.FloatTensor of size 3x3]

>>> torch.bernoulli(a)

 1  1  1
 **0**  0  1
 0  1  1
[torch.FloatTensor of size 3x3]

in the example the 0.5282 got approximated to 0 ,
how did that happen ? or it's a fault in the document because I tried it and the 0.5282 got approximated as expected to 1.


title: Cuda and pytorch memory usage

body:
I am using Cuda and Pytorch:1.4.0.

When I try to increase batch_size, I've got the following error:

CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 4.00 GiB total capacity;  2.74 GiB already allocated; 7.80 MiB free; 2.96 GiB reserved in total by PyTorch)

I haven't found anything about Pytorch memory usage.

Also, I don't understand why I have only 7.80 mib available?

Should I just use a videocard with better perfomance, or can I free some memory? FYI, I have a GTX 1050 TI, python 3,7 and torch==1.4.0 and my os is Windows 10.


title: PyTorch Softmax Dimensions error

body:
I'm attempting to write a simple NN module, with 2 layers, first layer ReLU activation, output softmax with 3 classes (one-hot encoded). It seems theres something wrong with the way I'm using the softmax function, but I'm not sure what's going on.

X is 178x13
Y is 178x3

Dataset I'm using is fairly simple, and can be found here.

I keep getting the error:

RuntimeError: dimension out of range (expected to be in range of [-2, 1], but got 3) .

.

import pandas as pd
import numpy as np
import torch
from torch.autograd import Variable
from sklearn.preprocessing import LabelBinarizer

# Read in dataset, specifying that this set didn't come with column headers
x = pd.read_csv('Datasets/wine.data', header=None)

# Rename columns
x.columns = ['Class', 'A1', 'A2', 'A3', 'A4', 'A5', 'A6', 'A7', 'A8', 'A9', 'A10', 'A11', 'A12', 'A13']

y = x[['Class']].values

#turn class labels into one-hot encoding
one_hot = LabelBinarizer()
y = Variable(torch.from_numpy(one_hot.fit_transform(y)), )

x = Variable(torch.from_numpy(x.iloc[:, 1:14].values).float())

N, D_in, H, D_out = y.shape[0], x.shape[1], 20, 3

# Implement neural net with nn module

model = torch.nn.Sequential(
    torch.nn.Linear(D_in, H),
    torch.nn.ReLU(),
    torch.nn.Linear(H, D_out),
    torch.nn.LogSoftmax(dim=3)
)

loss_fn = torch.nn.NLLLoss

learning_rate = 1e-4
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

for t in range(500):
    y_pred = model(x)

    loss = loss_fn(y_pred, y)
    print("Iteration: %d | Loss: %.3f" % (t, loss))

    optimizer.zero_grad()

    loss.backward()

    optimizer.step()


I set my model and data to the same device, but always raise the error like this:

  RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the same

The following is my training code, I hope you can answer it.Thanks!

def train(train_img_path, train_label_path, pths_path, interval, log_file):

    file_num = len(os.listdir(train_img_path))
    device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
    net = EAST(extractor=extractor, geometry_mode=geometry_mode, pretrained=True)
    net = net.to(device)
    trainset = custom_dataset(train_img_path, train_label_path)
    train_loader = data.DataLoader(trainset, batch_size=batch_size,
                                   shuffle=True, num_workers=num_workers, drop_last=True)

    optimizer = optim.SGD(net.parameters(), lr=initial_lr, momentum=momentum, weight_decay=weight_decay_sgd)
    criterion = Loss(weight_geo, weight_angle, geometry_mode="RBOX")

    net.train()
    epoch_loss = 0.
    for epoch in range(max_epoch):
        epoch_time = time.time()
        for i, (img, score_gt, geo_gt, ignored_map) in enumerate(train_loader):
            start_time = time.time()
            img, score_gt, geo_gt, ignored_map = img.to(device), score_gt.to(device),\
                                                 geo_gt.to(device), ignored_map.to(device)

            score_pred, geo_pred = net(img)
            total_loss, score_loss, loss_AABB, loss_angle = criterion(score_pred, geo_pred, score_gt, geo_gt, ignored_map)

            epoch_loss += total_loss.item()

            optimizer.zero_grad()
            total_loss.backward()

            optimizer.step()


title: How to use autograd.gradcheck in PyTorch?

body: The documentation does not include any example use case of gradcheck, where would it be useful?


title: torch transform.resize() vs cv2.resize()

body:
The CNN model takes an image tensor of size (112x112) as input and gives  (1x512) size tensor as output.
Using Opencv function cv2.resize() or using Transform.resize in pytorch to resize the input to (112x112) gives different outputs.
What's the reason for this? (I understand that the difference in the underlying implementation of opencv resizing vs torch resizing might be a cause for this, But I'd like to have a detailed understanding of it)
import cv2
import numpy as np
from PIL import image
import torch
import torchvision
from torchvision import transforms as trans

# device for pytorch
device = torch.device('cuda:0')

torch.set_default_tensor_type('torch.cuda.FloatTensor')

model = torch.jit.load("traced_facelearner_model_new.pt")
model.eval()

# read the example image used for tracing
image=cv2.imread("videos/example.jpg")

test_transform = trans.Compose([
            trans.ToTensor(),
            trans.Normalize([0.5, 0.5, 0.5], [0.5, 0.5, 0.5])
        ])
test_transform2 = trans.Compose([
            trans.Resize([int(112), int(112)]),
            trans.ToTensor(),
            trans.Normalize([0.5, 0.5, 0.5], [0.5, 0.5, 0.5])
        ])

resized_image = cv2.resize(image, (112, 112))

tensor1 = test_transform(resized_image).to(device).unsqueeze(0)
tensor2 = test_transform2(Image.fromarray(image)).to(device).unsqueeze(0)
output1 = model(tensor1)
output2 = model(tensor2)

The output1 and output2 tensors have different values.


title: Problem with building PyTorch from source on Linux

body:
❓ Problem with building PyTorch from source

Hello everyone,

I have problem with building PyTorch from source. I followed the official build instructions. I use Anaconda Python 3.7.1 (version 2018.12, build py37_0). I installed all neccessary dependencies using conda and issued  python setup.py install command to build it. It builds all files successfully but then it fails at the installation step saying:

build/temp.linux-x86_64-3.7/torch/csrc/stub.o: file not recognized: file format not recognized

I tried building with gcc/g++ versions 5,6,7 but it did not help.  Can you help me resolve this problem?

Build output in terminal

This is the output of python setup.py install with the error I am getting:

[manjaro-pc pytorch]# python setup.py install
Building wheel torch-1.1.0a0+44809fd
-- Building version 1.1.0a0+44809fd
[0/1] Install the project...
-- Install configuration: "Release"
running install
running build
running build_py
copying torch/version.py -> build/lib.linux-x86_64-3.7/torch
copying caffe2/proto/caffe2_pb2.py -> build/lib.linux-x86_64-3.7/caffe2/proto
copying caffe2/proto/caffe2_legacy_pb2.py -> build/lib.linux-x86_64-3.7/caffe2/proto
copying caffe2/proto/predictor_consts_pb2.py -> build/lib.linux-x86_64-3.7/caffe2/proto
copying caffe2/proto/metanet_pb2.py -> build/lib.linux-x86_64-3.7/caffe2/proto
copying caffe2/proto/torch_pb2.py -> build/lib.linux-x86_64-3.7/caffe2/proto
copying caffe2/proto/prof_dag_pb2.py -> build/lib.linux-x86_64-3.7/caffe2/proto
copying caffe2/proto/hsm_pb2.py -> build/lib.linux-x86_64-3.7/caffe2/proto
running build_ext
-- Building with NumPy bindings
-- Not using cuDNN
-- Not using MIOpen
-- Not using CUDA
-- Not using MKLDNN
-- Not using NCCL
-- Building without distributed package

Copying extension caffe2.python.caffe2_pybind11_state
Copying caffe2.python.caffe2_pybind11_state from torch/lib/python3.7/site-packages/caffe2/python/caffe2_pybind11_state.cpython-37m-x86_64-linux-gnu.so to /home/manjaro/Downloads/pytorch/build/lib.linux-x86_64-3.7/caffe2/python/caffe2_pybind11_state.cpython-37m-x86_64-linux-gnu.so
building 'torch._C' extension
gcc -pthread -B /opt/anaconda/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/opt/anaconda/include/python3.7m -c torch/csrc/stub.cpp -o build/temp.linux-x86_64-3.7/torch/csrc/stub.o -std=c++11 -Wall -Wextra -Wno-strict-overflow -Wno-unused-parameter -Wno-missing-field-initializers -Wno-write-strings -Wno-unknown-pragmas -Wno-deprecated-declarations -fno-strict-aliasing -Wno-missing-braces
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
g++ -pthread -shared -B /opt/anaconda/compiler_compat -L/opt/anaconda/lib -Wl,-rpath=/opt/anaconda/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.7/torch/csrc/stub.o -L/home/manjaro/Downloads/pytorch/torch/lib -lshm -ltorch_python -o build/lib.linux-x86_64-3.7/torch/_C.cpython-37m-x86_64-linux-gnu.so -Wl,-rpath,$ORIGIN/lib
/opt/anaconda/compiler_compat/ld: build/temp.linux-x86_64-3.7/torch/csrc/stub.o: unable to initialize decompress status for section .debug_info
/opt/anaconda/compiler_compat/ld: build/temp.linux-x86_64-3.7/torch/csrc/stub.o: unable to initialize decompress status for section .debug_info
/opt/anaconda/compiler_compat/ld: build/temp.linux-x86_64-3.7/torch/csrc/stub.o: unable to initialize decompress status for section .debug_info
/opt/anaconda/compiler_compat/ld: build/temp.linux-x86_64-3.7/torch/csrc/stub.o: unable to initialize decompress status for section .debug_info
build/temp.linux-x86_64-3.7/torch/csrc/stub.o: file not recognized: file format not recognized
collect2: error: ld returned 1 exit status
error: command 'g++' failed with exit status 1

From this output, I assume the error was caused by this line:

g++ -pthread -shared -B /opt/anaconda/compiler_compat -L/opt/anaconda/lib -Wl,-rpath=/opt/anaconda/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.7/torch/csrc/stub.o -L/home/manjaro/Downloads/pytorch/torch/lib -lshm -ltorch_python -o build/lib.linux-x86_64-3.7/torch/_C.cpython-37m-x86_64-linux-gnu.so -Wl,-rpath,$ORIGIN/lib

My environment

PyTorch version: 1.0
Is debug build: N/A
CUDA used to build PyTorch: 10

OS: Manjaro Linux
GCC version: (GCC) 6.4.1 20171002
CMake version: version 3.12.2

Python version: 3.7
Is CUDA available: N/A
CUDA runtime version: 10.0.130
GPU models and configuration: GPU 0: GeForce GTX 650
Nvidia driver version: 415.27
cuDNN version: Probably one of the following:
/usr/lib/libcudnn.so.7.4.1
/usr/lib/libcudnn_static_v7.a

Versions of relevant libraries:
[pip] Could not collect
[conda] blas                      1.0                         mkl
[conda] magma-cuda100             2.4.0                         1    pytorch
[conda] mkl                       2019.1                      144
[conda] mkl-include               2019.1                      144
[conda] mkl-service               1.1.2            py37he904b0f_5
[conda] mkl_fft                   1.0.6            py37hd81dba3_0
[conda] mkl_random                1.0.2            py37hd81dba3_0
[conda] mkldnn                    0.16.1                        0    mingfeima

My CMake configuration printed before building PyTorch

Building wheel torch-1.1.0a0+44809fd
-- Building version 1.1.0a0+44809fd
['cmake',
 '/home/manjaro/Downloads/pytorch',
 '-GNinja',
 '-DBLAS=MKL',
 '-DBUILDING_WITH_TORCH_LIBS=ON',
 '-DBUILD_BINARY=False',
 '-DBUILD_CAFFE2_OPS=False',
 '-DBUILD_PYTHON=True',
 '-DBUILD_SHARED_LIBS=ON',
 '-DBUILD_TEST=False',
 '-DBUILD_TORCH=ON',
 '-DCAFFE2_STATIC_LINK_CUDA=False',
 '-DCMAKE_BUILD_TYPE=Release',
 '-DCMAKE_CXX_FLAGS= ',
 '-DCMAKE_C_FLAGS= ',
 '-DCMAKE_EXE_LINKER_FLAGS=',
 '-DCMAKE_INSTALL_PREFIX=/home/manjaro/Downloads/pytorch/torch',
 '-DCMAKE_PREFIX_PATH=/opt/anaconda/bin/../',
 '-DCMAKE_SHARED_LINKER_FLAGS=',
 '-DINSTALL_TEST=False',
 '-DNCCL_EXTERNAL=False',
 '-DNUMPY_INCLUDE_DIR=/opt/anaconda/lib/python3.7/site-packages/numpy/core/include',
 '-DONNX_NAMESPACE=onnx_torch',
 '-DPYTHON_EXECUTABLE=/opt/anaconda/bin/python',
 '-DPYTHON_INCLUDE_DIR=/opt/anaconda/include/python3.7m',
 '-DPYTHON_LIBRARY=/opt/anaconda/lib/libpython3.7m.so.1.0',
 '-DTHD_SO_VERSION=1',
 '-DTORCH_BUILD_VERSION=1.1.0a0+44809fd',
 '-DUSE_CUDA=False',
 '-DUSE_DISTRIBUTED=False',
 '-DUSE_FBGEMM=False',
 '-DUSE_FFMPEG=False',
 '-DUSE_LEVELDB=False',
 '-DUSE_LMDB=False',
 '-DUSE_MKLDNN=False',
 '-DUSE_NNPACK=False',
 '-DUSE_NUMPY=True',
 '-DUSE_OPENCV=False',
 '-DUSE_QNNPACK=False',
 '-DUSE_ROCM=False',
 '-DUSE_SYSTEM_EIGEN_INSTALL=OFF',
 '-DUSE_SYSTEM_NCCL=False',
 '-DUSE_TENSORRT=False']
{'BLAS': 'MKL',
 'BUILD_CAFFE2_OPS': '0',
 'BUILD_TEST': '0',
 'CMAKE_PREFIX_PATH': '/opt/anaconda/bin/../',
 'DBUS_SESSION_BUS_ADDRESS': 'unix:path=/run/user/1000/bus',
 'HG': '/usr/bin/hg',
 'HOME': '/root',
 'LANG': 'en_US.UTF-8',
 'LC_ADDRESS': 'sk_SK.UTF-8',
 'LC_IDENTIFICATION': 'sk_SK.UTF-8',
 'LC_MEASUREMENT': 'sk_SK.UTF-8',
 'LC_MONETARY': 'sk_SK.UTF-8',
 'LC_NAME': 'sk_SK.UTF-8',
 'LC_NUMERIC': 'sk_SK.UTF-8',
 'LC_PAPER': 'sk_SK.UTF-8',
 'LC_TELEPHONE': 'sk_SK.UTF-8',
 'LC_TIME': 'sk_SK.UTF-8',
 'LD_LIBRARY_PATH': ':/usr/local/lib64:/usr/local/lib:/opt/cuda/lib64',
 'LOGNAME': 'manjaro',
 'LS_COLORS': 'rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.pdf=00;32:*.ps=00;32:*.txt=00;32:*.patch=00;32:*.diff=00;32:*.log=00;32:*.tex=00;32:*.doc=00;32:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.axa=00;36:*.oga=00;36:*.spx=00;36:*.xspf=00;36:',
 'MAIL': '/var/spool/mail/manjaro',
 'MAX_JOBS': '4',
 'MOZ_PLUGIN_PATH': '/usr/lib/mozilla/plugins',
 'NO_CAFFE2_OPS': '1',
 'NO_CUDA': '1',
 'NO_CUDNN': '1',
 'NO_DISTRIBUTED': '1',
 'NO_FBGEMM': '1',
 'NO_MIOPEN': '1',
 'NO_MKLDNN': '1',
 'NO_NNPACK': '1',
 'NO_QNNPACK': '1',
 'NO_TEST': '1',
 'OLDPWD': '/home/manjaro/Downloads',
 'PATH': '/opt/anaconda/bin:/opt/anaconda/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/opt/cuda/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/usr/local/bin:/opt/cuda/bin:/usr/local/bin:/opt/cuda/bin',
 'PWD': '/home/manjaro/Downloads/pytorch',
 'PYTHONPATH': ':/usr/local/lib/python3.7/site-packages/:/usr/local/lib/python3.7/site-packages/',
 'QT_LINUX_ACCESSIBILITY_ALWAYS_ON': '1',
 'SHELL': '/bin/bash',
 'SHLVL': '2',
 'SSH_CLIENT': '62.197.243.30 45360 44444',
 'SSH_CONNECTION': '62.197.243.30 45360 192.168.1.147 44444',
 'SSH_TTY': '/dev/pts/4',
 'TERM': 'xterm-256color',
 'TORCH_CUDA_ARCH_LIST': '3.0',
 'USER': 'manjaro',
 'USE_CUDA': '0',
 'USE_CUDNN': '0',
 'USE_DISTRIBUTED': '0',
 'USE_FBGEMM': '0',
 'USE_MIOPEN': '0',
 'USE_MKLDNN': '0',
 'USE_NNPACK': '0',
 'USE_QNNPACK': '0',
 'XDG_DATA_DIRS': '/home/manjaro/.local/share/flatpak/exports/share:/var/lib/flatpak/exports/share:/usr/local/share:/usr/share',
 'XDG_RUNTIME_DIR': '/run/user/1000',
 'XDG_SESSION_ID': 'c8',
 'YAOURT_COLORS': 'nb=1:pkg=1:ver=1;32:lver=1;45:installed=1;42:grp=1;34:od=1;41;5:votes=1;44:dsc=0:other=1;35',
 '_': '/opt/anaconda/bin/python'}
-- The CXX compiler identification is GNU 5.4.1
-- The C compiler identification is GNU 5.4.1
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Performing Test COMPILER_WORKS
-- Performing Test COMPILER_WORKS - Success
-- Performing Test SUPPORT_GLIBCXX_USE_C99
-- Performing Test SUPPORT_GLIBCXX_USE_C99 - Success
-- Performing Test CAFFE2_EXCEPTION_PTR_SUPPORTED
-- Performing Test CAFFE2_EXCEPTION_PTR_SUPPORTED - Success
-- std::exception_ptr is supported.
-- Performing Test CAFFE2_IS_NUMA_AVAILABLE
-- Performing Test CAFFE2_IS_NUMA_AVAILABLE - Success
-- NUMA is available
-- Performing Test CAFFE2_NEED_TO_TURN_OFF_DEPRECATION_WARNING
-- Performing Test CAFFE2_NEED_TO_TURN_OFF_DEPRECATION_WARNING - Success
-- Performing Test CAFFE2_COMPILER_SUPPORTS_AVX2_EXTENSIONS
-- Performing Test CAFFE2_COMPILER_SUPPORTS_AVX2_EXTENSIONS - Success
-- Current compiler supports avx2 extension. Will build perfkernels.
-- Performing Test CAFFE2_COMPILER_SUPPORTS_AVX512_EXTENSIONS
-- Performing Test CAFFE2_COMPILER_SUPPORTS_AVX512_EXTENSIONS - Success
-- Current compiler supports avx512f extension. Will build fbgemm.
-- Performing Test COMPILER_SUPPORTS_HIDDEN_VISIBILITY
-- Performing Test COMPILER_SUPPORTS_HIDDEN_VISIBILITY - Success
-- Performing Test COMPILER_SUPPORTS_HIDDEN_INLINE_VISIBILITY
-- Performing Test COMPILER_SUPPORTS_HIDDEN_INLINE_VISIBILITY - Success
-- Performing Test COMPILER_SUPPORTS_RDYNAMIC
-- Performing Test COMPILER_SUPPORTS_RDYNAMIC - Success
-- Building using own protobuf under third_party per request.
-- Use custom protobuf build.
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Caffe2 protobuf include directory: $<BUILD_INTERFACE:/home/manjaro/Downloads/pytorch/third_party/protobuf/src>$<INSTALL_INTERFACE:include>
-- Trying to find preferred BLAS backend of choice: MKL
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Looking for stdint.h
-- Looking for stdint.h - found
-- Looking for stddef.h
-- Looking for stddef.h - found
-- Check size of void*
-- Check size of void* - done
-- Checking for [mkl_intel_lp64 - mkl_gnu_thread - mkl_core - gomp - pthread - m - dl]
--   Library mkl_intel_lp64: /opt/anaconda/lib/libmkl_intel_lp64.so
--   Library mkl_gnu_thread: /opt/anaconda/lib/libmkl_gnu_thread.so
--   Library mkl_core: /opt/anaconda/lib/libmkl_core.so
-- Found OpenMP_C: -fopenmp (found version "4.0")
-- Found OpenMP_CXX: -fopenmp (found version "4.0")
-- Found OpenMP: TRUE (found version "4.0")
--   Library gomp: -fopenmp
--   Library pthread: /usr/lib/libpthread.so
--   Library m: /usr/lib/libm.so
--   Library dl: /usr/lib/libdl.so
-- Looking for cblas_sgemm
-- Looking for cblas_sgemm - found
-- MKL library found
-- Check if compiler accepts -pthread
-- Check if compiler accepts -pthread - yes
-- Caffe2: Found gflags with new-style gflags target.
-- Caffe2: Cannot find glog automatically. Using legacy find.
-- Found glog: /usr/include
-- Caffe2: Found glog (include: /usr/include, library: /usr/lib/libglog.so)
-- Found Numa: /usr/include
-- Found Numa  (include: /usr/include, library: /usr/lib/libnuma.so)
-- Downloading PSimd to /home/manjaro/Downloads/pytorch/build/confu-srcs/psimd (define PSIMD_SOURCE_DIR to avoid it)
-- Configuring done
-- Generating done
-- Build files have been written to: /home/manjaro/Downloads/pytorch/build/confu-deps/psimd-download
[1/9] Creating directories for 'psimd'
[2/9] Performing download step (git clone) for 'psimd'
Cloning into 'psimd'...
Already on 'master'
Your branch is up to date with 'origin/master'.
[3/9] No patch step for 'psimd'
[4/9] Performing update step for 'psimd'
Current branch master is up to date.
[5/9] No configure step for 'psimd'
[6/9] No build step for 'psimd'
[7/9] No install step for 'psimd'
[8/9] No test step for 'psimd'
[9/9] Completed 'psimd'
-- Using third party subdirectory Eigen.
Python 3.7.1
-- Found PythonInterp: /opt/anaconda/bin/python (found suitable version "3.7.1", minimum required is "2.7")
-- Found PythonLibs: /opt/anaconda/lib/libpython3.7m.so.1.0 (found suitable version "3.7.1", minimum required is "2.7")
-- Found PythonInterp: /opt/anaconda/bin/python (found version "3.7.1")
-- Found PythonLibs: /opt/anaconda/lib/libpython3.7m.so.1.0
-- System pybind11 found
-- pybind11 include dirs: /usr/include;/opt/anaconda/include/python3.7m
CMake Warning at cmake/Dependencies.cmake:805 (message):
  Not using CUDA, so disabling NCCL.  Suppress this warning with
  -DUSE_NCCL=OFF.
Call Stack (most recent call first):
  CMakeLists.txt:219 (include)

CMake Warning at cmake/Dependencies.cmake:950 (message):
  Metal is only used in ios builds.
Call Stack (most recent call first):
  CMakeLists.txt:219 (include)

--
-- ******** Summary ********
--   CMake version         : 3.12.2
--   CMake command         : /opt/anaconda/bin/cmake
--   System                : Linux
--   C++ compiler          : /usr/bin/c++
--   C++ compiler version  : 5.4.1
--   CXX flags             :   -fvisibility-inlines-hidden -Wnon-virtual-dtor
--   Build type            : Release
--   Compile definitions   : TH_BLAS_MKL
--   CMAKE_PREFIX_PATH     : /opt/anaconda/bin/../
--   CMAKE_INSTALL_PREFIX  : /home/manjaro/Downloads/pytorch/torch
--   CMAKE_MODULE_PATH     : /home/manjaro/Downloads/pytorch/cmake/Modules;/usr/share/cmake/pybind11
--
--   ONNX version          : 1.4.1
--   ONNX NAMESPACE        : onnx_torch
--   ONNX_BUILD_TESTS      : OFF
--   ONNX_BUILD_BENCHMARKS : OFF
--   ONNX_USE_LITE_PROTO   : OFF
--   ONNXIFI_DUMMY_BACKEND : OFF
--
--   Protobuf compiler     :
--   Protobuf includes     :
--   Protobuf libraries    :
--   BUILD_ONNX_PYTHON     : OFF
-- Found gcc >=5 and CUDA <= 7.5, adding workaround C++ flags
-- Could not find CUDA with FP16 support, compiling without torch.CudaHalfTensor
-- Removing -DNDEBUG from compile flags
-- Found OpenMP_C: -fopenmp (found version "4.0")
-- Found OpenMP_CXX: -fopenmp (found version "4.0")
-- Compiling with OpenMP support
-- MAGMA not found. Compiling without MAGMA support
-- Could not find hardware support for NEON on this machine.
-- No OMAP3 processor on this machine.
-- No OMAP4 processor on this machine.
-- Looking for cpuid.h
-- Looking for cpuid.h - found
-- Performing Test HAVE_GCC_GET_CPUID
-- Performing Test HAVE_GCC_GET_CPUID - Success
-- Performing Test NO_GCC_EBX_FPIC_BUG
-- Performing Test NO_GCC_EBX_FPIC_BUG - Success
-- Performing Test C_HAS_AVX_1
-- Performing Test C_HAS_AVX_1 - Failed
-- Performing Test C_HAS_AVX_2
-- Performing Test C_HAS_AVX_2 - Success
-- Performing Test C_HAS_AVX2_1
-- Performing Test C_HAS_AVX2_1 - Failed
-- Performing Test C_HAS_AVX2_2
-- Performing Test C_HAS_AVX2_2 - Success
-- Performing Test CXX_HAS_AVX_1
-- Performing Test CXX_HAS_AVX_1 - Failed
-- Performing Test CXX_HAS_AVX_2
-- Performing Test CXX_HAS_AVX_2 - Success
-- Performing Test CXX_HAS_AVX2_1
-- Performing Test CXX_HAS_AVX2_1 - Failed
-- Performing Test CXX_HAS_AVX2_2
-- Performing Test CXX_HAS_AVX2_2 - Success
-- AVX compiler support found
-- AVX2 compiler support found
-- Performing Test HAS_C11_ATOMICS
-- Performing Test HAS_C11_ATOMICS - Failed
-- Performing Test HAS_MSC_ATOMICS
-- Performing Test HAS_MSC_ATOMICS - Failed
-- Performing Test HAS_GCC_ATOMICS
-- Performing Test HAS_GCC_ATOMICS - Success
-- Atomics: using GCC intrinsics
-- Performing Test BLAS_F2C_DOUBLE_WORKS
-- Performing Test BLAS_F2C_DOUBLE_WORKS - Failed
-- Performing Test BLAS_F2C_FLOAT_WORKS
-- Performing Test BLAS_F2C_FLOAT_WORKS - Success
-- Performing Test BLAS_USE_CBLAS_DOT
-- Performing Test BLAS_USE_CBLAS_DOT - Success
-- Found a library with BLAS API (mkl).
-- Found a library with LAPACK API (mkl).
disabling CUDA because NOT USE_CUDA is set
-- CuDNN not found. Compiling without CuDNN support
disabling ROCM because NOT USE_ROCM is set
-- MIOpen not found. Compiling without MIOpen support
disabling MKLDNN because USE_MKLDNN is not set
-- Looking for clock_gettime in rt
-- Looking for clock_gettime in rt - found
-- Looking for mmap
-- Looking for mmap - found
-- Looking for shm_open
-- Looking for shm_open - found
-- Looking for shm_unlink
-- Looking for shm_unlink - found
-- Looking for malloc_usable_size
-- Looking for malloc_usable_size - found
-- Performing Test C_HAS_THREAD
-- Performing Test C_HAS_THREAD - Success
-- GCC 5.4.1: Adding gcc and gcc_s libs to link line
-- NUMA paths:
-- /usr/include
-- /usr/lib/libnuma.so
disabling CUDA because USE_CUDA is set false
-- Check size of long double
-- Check size of long double - done
-- Performing Test COMPILER_SUPPORTS_LONG_DOUBLE
-- Performing Test COMPILER_SUPPORTS_LONG_DOUBLE - Success
-- Performing Test COMPILER_SUPPORTS_FLOAT128
-- Performing Test COMPILER_SUPPORTS_FLOAT128 - Success
-- Performing Test COMPILER_SUPPORTS_SSE2
-- Performing Test COMPILER_SUPPORTS_SSE2 - Success
-- Performing Test COMPILER_SUPPORTS_SSE4
-- Performing Test COMPILER_SUPPORTS_SSE4 - Success
-- Performing Test COMPILER_SUPPORTS_AVX
-- Performing Test COMPILER_SUPPORTS_AVX - Success
-- Performing Test COMPILER_SUPPORTS_FMA4
-- Performing Test COMPILER_SUPPORTS_FMA4 - Success
-- Performing Test COMPILER_SUPPORTS_AVX2
-- Performing Test COMPILER_SUPPORTS_AVX2 - Success
-- Performing Test COMPILER_SUPPORTS_SVE
-- Performing Test COMPILER_SUPPORTS_SVE - Failed
-- Performing Test COMPILER_SUPPORTS_AVX512F
-- Performing Test COMPILER_SUPPORTS_AVX512F - Failed
-- Found OpenMP_C: -fopenmp (found version "4.0")
-- Found OpenMP_CXX: -fopenmp (found version "4.0")
-- Performing Test COMPILER_SUPPORTS_OPENMP
-- Performing Test COMPILER_SUPPORTS_OPENMP - Success
-- Performing Test COMPILER_SUPPORTS_WEAK_ALIASES
-- Performing Test COMPILER_SUPPORTS_WEAK_ALIASES - Success
-- Performing Test COMPILER_SUPPORTS_BUILTIN_MATH
-- Performing Test COMPILER_SUPPORTS_BUILTIN_MATH - Success
-- Configuring build for SLEEF-v3.2
   Target system: Linux-4.14.94-1-MANJARO
   Target processor: x86_64
   Host system: Linux-4.14.94-1-MANJARO
   Host processor: x86_64
   Detected C compiler: GNU @ /usr/bin/cc
-- Using option `-Wall -Wno-unused -Wno-attributes -Wno-unused-result -Wno-psabi -ffp-contract=off -fno-math-errno -fno-trapping-math` to compile libsleef
-- Building shared libs : OFF
-- MPFR : /opt/anaconda/lib/libmpfr.so
-- MPFR header file in /opt/anaconda/include
-- GMP : /opt/anaconda/lib/libgmp.so
-- RUNNING_ON_TRAVIS : 0
-- COMPILER_SUPPORTS_OPENMP : 1
-- /usr/bin/c++ /home/manjaro/Downloads/pytorch/torch/abi-check.cpp -o /home/manjaro/Downloads/pytorch/build/abi-check
-- Determined _GLIBCXX_USE_CXX11_ABI=1
-- NCCL operators skipped due to no CUDA support
-- Excluding ideep operators as we are not using ideep
-- Excluding image processing operators due to no opencv
-- Excluding video processing operators due to no opencv
-- MPI operators skipped due to no MPI support
-- Include Observer library
-- Using lib/python3.7/site-packages as python relative installation path
CMake Warning at CMakeLists.txt:416 (message):
  Generated cmake files are only fully tested if one builds with system glog,
  gflags, and protobuf.  Other settings may generate files that are not well
  tested.

--
-- ******** Summary ********
-- General:
--   CMake version         : 3.12.2
--   CMake command         : /opt/anaconda/bin/cmake
--   System                : Linux
--   C++ compiler          : /usr/bin/c++
--   C++ compiler version  : 5.4.1
--   BLAS                  : MKL
--   CXX flags             :   -fvisibility-inlines-hidden -D_FORCE_INLINES -D_MWAITXINTRIN_H_INCLUDED -D__STRICT_ANSI__ -fopenmp -O2 -fPIC -Wno-narrowing -Wall -Wextra -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -Wno-unused-but-set-variable -Wno-maybe-uninitialized
--   Build type            : Release
--   Compile definitions   : TH_BLAS_MKL;ONNX_NAMESPACE=onnx_torch;USE_GCC_ATOMICS=1;HAVE_MMAP=1;_FILE_OFFSET_BITS=64;HAVE_SHM_OPEN=1;HAVE_SHM_UNLINK=1;HAVE_MALLOC_USABLE_SIZE=1
--   CMAKE_PREFIX_PATH     : /opt/anaconda/bin/../
--   CMAKE_INSTALL_PREFIX  : /home/manjaro/Downloads/pytorch/torch
--
--   TORCH_VERSION         : 1.1.0
--   CAFFE2_VERSION        : 1.1.0
--   BUILD_ATEN_MOBILE     : OFF
--   BUILD_ATEN_ONLY       : OFF
--   BUILD_BINARY          : False
--   BUILD_CUSTOM_PROTOBUF : ON
--     Link local protobuf : ON
--   BUILD_DOCS            : OFF
--   BUILD_PYTHON          : True
--     Python version      : 3.7.1
--     Python executable   : /opt/anaconda/bin/python
--     Pythonlibs version  : 3.7.1
--     Python library      : /opt/anaconda/lib/libpython3.7m.so.1.0
--     Python includes     : /opt/anaconda/include/python3.7m
--     Python site-packages: lib/python3.7/site-packages
--   BUILD_CAFFE2_OPS      : False
--   BUILD_SHARED_LIBS     : ON
--   BUILD_TEST            : False
--   USE_ASAN              : OFF
--   USE_CUDA              : False
--   USE_ROCM              : False
--   USE_EIGEN_FOR_BLAS    :
--   USE_FBGEMM            : OFF
--   USE_FFMPEG            : False
--   USE_GFLAGS            : ON
--   USE_GLOG              : ON
--   USE_LEVELDB           : False
--   USE_LITE_PROTO        : OFF
--   USE_LMDB              : False
--   USE_METAL             : OFF
--   USE_MKL               : ON
--   USE_MKLDNN            : OFF
--   USE_NCCL              : OFF
--   USE_NNPACK            : False
--   USE_NUMPY             : ON
--   USE_OBSERVERS         : ON
--   USE_OPENCL            : OFF
--   USE_OPENCV            : False
--   USE_OPENMP            : OFF
--   USE_PROF              : OFF
--   USE_QNNPACK           : False
--   USE_REDIS             : OFF
--   USE_ROCKSDB           : OFF
--   USE_ZMQ               : OFF
--   USE_DISTRIBUTED       : False
--   Public Dependencies  : Threads::Threads;caffe2::mkl;glog::glog
--   Private Dependencies : cpuinfo;/usr/lib/libnuma.so;fp16;onnxifi_loader;rt;gcc_s;gcc;dl
-- Configuring done
-- Generating done
CMake Warning:
  Manually-specified variables were not used by the project:

    NCCL_EXTERNAL
    THD_SO_VERSION


title: What is the difference between sample() and rsample()?

body:
When I sample from a distribution in PyTorch, both sample and rsample appear to give similar results:
import torch, seaborn as sns

x = torch.distributions.Normal(torch.tensor([0.0]), torch.tensor([1.0]))

sns.distplot(x.sample((100000,)))
sns.distplot(x.rsample((100000,)))

When should I use sample(), and when should I use rsample()?


CUDA error with pytorch

Installed pytorch with conda and CUDA support

The error occurs when I am trying to run a tensor on the cuda gpu

torch.rand(10).to(torch.device('cuda'))

RuntimeError Traceback (most recent call last)
~/anaconda3/envs/deep/lib/python3.7/site-packages/IPython/core/formatters.py in call(self, obj)
700 type_pprinters=self.type_printers,
701 deferred_pprinters=self.deferred_printers)
--> 702 printer.pretty(obj)
703 printer.flush()
704 return stream.getvalue()

~/anaconda3/envs/deep/lib/python3.7/site-packages/IPython/lib/pretty.py in pretty(self, obj)
397 if cls is not object
398 and callable(cls.dict.get('repr')):
--> 399 return _repr_pprint(obj, self, cycle)
400
401 return _default_pprint(obj, self, cycle)

~/anaconda3/envs/deep/lib/python3.7/site-packages/IPython/lib/pretty.py in repr_pprint(obj, p, cycle)
687 """A pprint that just redirects to the normal repr function."""
688 # Find newlines and replace them with p.break()
--> 689 output = repr(obj)
690 for idx,output_line in enumerate(output.splitlines()):
691 if idx:

~/anaconda3/envs/deep/lib/python3.7/site-packages/torch/tensor.py in repr(self)
128 # characters to replace unicode characters with.
129 if sys.version_info > (3,):
--> 130 return torch._tensor_str._str(self)
131 else:
132 if hasattr(sys.stdout, 'encoding'):

~/anaconda3/envs/deep/lib/python3.7/site-packages/torch/_tensor_str.py in _str(self)
309 tensor_str = _tensor_str(self.to_dense(), indent)
310 else:
--> 311 tensor_str = _tensor_str(self, indent)
312
313 if self.layout != torch.strided:

~/anaconda3/envs/deep/lib/python3.7/site-packages/torch/_tensor_str.py in _tensor_str(self, indent)
207 if self.dtype is torch.float16 or self.dtype is torch.bfloat16:
208 self = self.float()
--> 209 formatter = _Formatter(get_summarized_data(self) if summarize else self)
210 return _tensor_str_with_formatter(self, indent, formatter, summarize)
211

~/anaconda3/envs/deep/lib/python3.7/site-packages/torch/_tensor_str.py in init(self, tensor)
85
86 else:
---> 87 nonzero_finite_vals = torch.masked_select(tensor_view, torch.isfinite(tensor_view) & tensor_view.ne(0))
88
89 if nonzero_finite_vals.numel() == 0:

~/anaconda3/envs/deep/lib/python3.7/site-packages/torch/functional.py in isfinite(tensor)
225 if not tensor.is_floating_point():
226 return torch.ones_like(tensor, dtype=torch.bool)
--> 227 return (tensor == tensor) & (tensor.abs() != inf)
228
229

RuntimeError: CUDA error: no kernel image is available for execution on the device

This is the collect_env.py output:

Collecting environment information...

PyTorch version: 1.3.1

Is debug build: No

CUDA used to build PyTorch: 10.0.130

OS: Ubuntu 16.04.6 LTS

GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.12) 5.4.0 20160609

CMake version: version 3.5.1

Python version: 3.7

Is CUDA available: Yes

CUDA runtime version: 10.0.130

GPU models and configuration: GPU 0: GeForce 920M

Nvidia driver version: 415.27

cuDNN version: Could not collect

Versions of relevant libraries:

[pip] numpy==1.17.4

[pip] numpy-stl==2.10.1

[pip] torch==1.3.1

[pip] torchvision==0.4.2

[conda] blas 1.0 mkl

[conda] mkl 2019.4 243

[conda] mkl-service 2.3.0 py37he904b0f_0

[conda] mkl_fft 1.0.15 py37ha843d7b_0

[conda] mkl_random 1.1.0 py37hd6b4f25_0

[conda] pytorch 1.3.1 py3.7_cuda10.0.130_cudnn7.6.3_0 pytorch

[conda] torchvision 0.4.2 py37_cu100 pytorch

Some have stated that the TORCH_CUDA_ARCH_LIST needs to be manually set to ='3.5', but I can't figure out how


title: How to load images with multiple JSON annotation in PyTorch

body:
I would like to know how I can use the data loader in PyTorch for the custom file structure of mine. I have gone through PyTorch documentation, but all those are with separate folders with class.

My folder structure consists of 2 folders(called training and validation), each with 2 subfolders(called images and json_annotations). Each image in the "images" folder has multiple objects(like cars, cycles, man etc) and each is annotated and have separate JSON files. Standard coco annotation is followed. My intention is to make a neural network which can do real-time classification from videos.

Edit 1:
I have done the coding as suggested by Fábio Perez.

class lDataSet(data.Dataset):
    def __init__(self, path_to_imgs, path_to_json):
        self.path_to_imgs = path_to_imgs
        self.path_to_json = path_to_json
        self.img_ids = os.listdir(path_to_imgs)

    def __getitem__(self, idx):
        img_id = self.img_ids[idx]
        img_id = os.path.splitext(img_id)[0]
        img = cv2.imread(os.path.join(self.path_to_imgs, img_id + ".jpg"))
        load_json = json.load(open(os.path.join(self.path_to_json, img_id + ".json")))
        #n = len(load_json)
        #bboxes = load_json['annotation'][n]['segmentation']
        return img, load_json

    def __len__(self):
        return len(self.image_ids)

When I try this

l_data = lDataSet(path_to_imgs = '/home/training/images', path_to_json = '/home/training/json_annotations')

I'm getting l_data with l_data[][0] - images and l_data with json. Now I'm confused. How will I use it with finetuning example availalbe in PyTorch? In that example, dataset and dataloader is done as shown below.
https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html

# Create training and validation datasets
image_datasets = {x: datasets.ImageFolder(os.path.join(data_dir, x), data_transforms[x]) for x in ['train', 'val']}
# Create training and validation dataloaders
dataloaders_dict = {x: torch.utils.data.DataLoader(image_datasets[x], batch_size=batch_size, shuffle=True, num_workers=4) for x in ['train', 'val']}


title: torch.cat but create a new dimension

body:
I would like to concatenate tensors, not along a dimension, but by creating a new dimension.

For example:

x = torch.randn(2, 3)
x.shape # (2, 3)

torch.cat([x,x,x,x], 0).shape # (8, 3)
# This concats along dim 0, not what I want

torch.cat([x,x,x,x], -1).shape # (2, 10)
# This concats along dim 1, not what I want

torch.cat([x[None, :, :],x[None, :, :],x[None, :, :],x[None, :, :]], 0).shape
# => (4, 2, 3)
# This is what I want, but unwieldy

Is there a simpler way?


title: Pytorch: Why is the memory occupied by the `tensor` variable so small?

body:
In Pytorch 1.0.0, I found that a tensor variable occupies very small memory. I wonder how it stores so much data.
Here's the code.

a = np.random.randn(1, 1, 128, 256)
b = torch.tensor(a, device=torch.device('cpu'))

a_size = sys.getsizeof(a)
b_size = sys.getsizeof(b)

a_size is 262288. b_size is 72.


I am initializing a Convolutional DQN with the following code:

class ConvDQN(nn.Module):

    def __init__(self, input_dim, output_dim):
        super(ConvDQN, self).__init__()
        self.input_dim = input_dim
        self.output_dim = output_dim
        self.conv = nn.Sequential(
            nn.Conv2d(self.input_dim, 32, kernel_size=8, stride=4),
            nn.ReLU(),
            nn.Conv2d(32, 64, kernel_size=4, stride=2),
            nn.ReLU(),
            nn.Conv2d(64, 64, kernel_size=3, stride=1),
            nn.ReLU()
        )

        self.fc_input_dim = self.feature_size()

        self.fc = nn.Sequential(
            nn.Linear(self.fc_input_dim, 128),
            nn.ReLU(),
            nn.Linear(128, 256),
            nn.ReLU(),
            nn.Linear(256, self.output_dim)
        )

    def forward(self, state):
        features = self.conv(state)
        features = features.view(features.size(0), -1)
        qvals = self.fc(features)
        return qvals

    def feature_size(self):
        return self.conv(autograd.Variable(torch.zeros(1, *self.input_dim))).view(1, -1).size(1)

And it gives me the error:

  File "dqn.py", line 86, in __init__
    self.fc_input_dim = self.feature_size()
  File "dqn.py", line 105, in feature_size
    return self.conv(autograd.Variable(torch.zeros(32, *self.input_dim))).view(1, -1).size(1)
  File "C:\Users\ariji\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\nn\modules\module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\Users\ariji\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\nn\modules\container.py", line 92, in forward
    input = module(input)
  File "C:\Users\ariji\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\nn\modules\module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\Users\ariji\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\nn\modules\conv.py", line 320, in forward
    self.padding, self.dilation, self.groups)
RuntimeError: Expected 4-dimensional input for 4-dimensional weight [32, 4, 8, 8], but got 2-dimensional input of size [1, 4] instead

So I get the fact that the input that I am passing to the convolutional network is of incorrect dimensions. What I do not understand is how I am supposed to add the required dimensions to my input? Or should I change something in my convolutional network?


title: How do I calculate the confusion matrix in PyTorch efficiently?

body: I have a tensor that contains my predictions and a tensor that contains the actual labels for my binary classification problem. How can I calculate the confusion matrix efficiently?


title: How to dynamically index the tensor in pytorch?

body:
For example, I got a tensor:

tensor = torch.rand(12, 512, 768)

And I got an index list, say it is:

[0,2,3,400,5,32,7,8,321,107,100,511]

I wish to select 1 element out of 512 elements on dimension 2 given the index list. And then the tensor's size would become (12, 1, 768).

Is there a way to do it?


title: Pytorch "NCCL error": unhandled system error, NCCL version 2.4.8"

body:
I use pytorch to distributed training my model.I have two nodes and two gpu for each node, and I run the code for one node:
python train_net.py  --config-file configs/InstanceSegmentation/pointrend_rcnn_R_50_FPN_1x_coco.yaml  --num-gpu 2  --num-machines 2 --machine-rank 0 --dist-url tcp://192.168.**.***:8000

and the other:
python train_net.py  --config-file configs/InstanceSegmentation/pointrend_rcnn_R_50_FPN_1x_coco.yaml  --num-gpu 2  --num-machines 2 --machine-rank 1 --dist-url tcp://192.168.**.***:8000

However the other has RuntimeError problem
global_rank 3 machine_rank 1 num_gpus_per_machine 2 local_rank 1
global_rank 2 machine_rank 1 num_gpus_per_machine 2 local_rank 0
Traceback (most recent call last):
  File "train_net.py", line 109, in <module>
    args=(args,),
  File "/root/detectron2_repo/detectron2/engine/launch.py", line 49, in launch
    daemon=False,
  File "/root/anaconda3/envs/PointRend/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 171, in spawn
    while not spawn_context.join():
  File "/root/anaconda3/envs/PointRend/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 118, in join
    raise Exception(msg)
Exception:

-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/root/anaconda3/envs/PointRend/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
    fn(i, *args)
  File "/root/detectron2_repo/detectron2/engine/launch.py", line 72, in _distributed_worker
    comm.synchronize()
  File "/root/detectron2_repo/detectron2/utils/comm.py", line 79, in synchronize
    dist.barrier()
  File "/root/anaconda3/envs/PointRend/lib/python3.6/site-packages/torch/distributed/distributed_c10d.py", line 1489, in barrier
    work = _default_pg.barrier()
RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:410, unhandled system error, NCCL version 2.4.8

IF I change mask-rank = 1 to mask-rank = 0, then no error will be reported, but can't distributed training,Does anyone know why this error may occur?


title: Pytorch modify dataset label

body:
This is a code snippet for loading images as dataset from pytorch transfer learning tutorial:

data_transforms = {
    'train': transforms.Compose([
        transforms.RandomResizedCrop(224),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
    'val': transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
}

data_dir = 'data'
image_datasets = {x: datasets.ImageFolder(os.path.join(data_dir, x),
                                          data_transforms[x])
                  for x in ['train', 'val']}

dataloaders = {x: torch.utils.data.DataLoader(image_datasets[x], batch_size=4,
                                             shuffle=True, num_workers=4)
              for x in ['train', 'val']}
dataset_sizes = {x: len(image_datasets[x]) for x in ['train', 'val']}

And this is one of the examples in dataset:

image_datasets['val'][0]:

(tensor([[[ 2.2489,  2.2489,  2.2489,  ...,  2.2489,  2.2489,  2.2489],
          [ 2.2489,  2.2489,  2.2489,  ...,  2.2489,  2.2489,  2.2489],
          [ 2.2489,  2.2489,  2.2489,  ...,  2.2489,  2.2489,  2.2489],
          ...,
          [ 2.2489,  2.2489,  2.2489,  ...,  2.2489,  2.2489,  2.2489],
          [ 2.2489,  2.2489,  2.2489,  ...,  2.2489,  2.2489,  2.2489],
          [ 2.2489,  2.2489,  2.2489,  ...,  2.2489,  2.2489,  2.2489]],

         [[ 2.4286,  2.4286,  2.4286,  ...,  2.4286,  2.4286,  2.4286],
          [ 2.4286,  2.4286,  2.4286,  ...,  2.4286,  2.4286,  2.4286],
          [ 2.4286,  2.4286,  2.4286,  ...,  2.4286,  2.4286,  2.4286],
          ...,
          [ 2.4286,  2.4286,  2.4286,  ...,  2.4286,  2.4286,  2.4286],
          [ 2.4286,  2.4286,  2.4286,  ...,  2.4286,  2.4286,  2.4286],
          [ 2.4286,  2.4286,  2.4286,  ...,  2.4286,  2.4286,  2.4286]],

         [[ 2.6400,  2.6400,  2.6400,  ...,  2.6400,  2.6400,  2.6400],
          [ 2.6400,  2.6400,  2.6400,  ...,  2.6400,  2.6400,  2.6400],
          [ 2.6400,  2.6400,  2.6400,  ...,  2.6400,  2.6400,  2.6400],
          ...,
          [ 2.6400,  2.6400,  2.6400,  ...,  2.6400,  2.6400,  2.6400],
          [ 2.6400,  2.6400,  2.6400,  ...,  2.6400,  2.6400,  2.6400],
          [ 2.6400,  2.6400,  2.6400,  ...,  2.6400,  2.6400,  2.6400]]]), 0)

Is there any method (best practices) to change the example data in dataset, for example change label 0 to label 1. The following does not work:

image_datasets['val'][0] = (image_datasets['val'][0][0], 1)


title: Pytorch squeeze and unsqueeze

body:
I don't understand what squeeze and unsqueeze do to a tensor, even after looking at the docs and related questions.
I tried to understand it by exploring it myself in python. I first created a random tensor with
x = torch.rand(3,2,dtype=torch.float)
>>> x
tensor([[0.3703, 0.9588],
        [0.8064, 0.9716],
        [0.9585, 0.7860]])

But regardless of how I squeeze it, I end up with the same results:
torch.equal(x.squeeze(0), x.squeeze(1))
>>> True

If I now try to unsqueeze I get the following,
>>> x.unsqueeze(1)
tensor([[[0.3703, 0.9588]],
        [[0.8064, 0.9716]],
        [[0.9585, 0.7860]]])
>>> x.unsqueeze(0)
tensor([[[0.3703, 0.9588],
         [0.8064, 0.9716],
         [0.9585, 0.7860]]])
>>> x.unsqueeze(-1)
tensor([[[0.3703],
         [0.9588]],
        [[0.8064],
         [0.9716]],
        [[0.9585],
         [0.7860]]])

However if I now create a tensor  x = torch.tensor([1,2,3,4]), and I try to unsqueeze it then it appears that 1 and -1 makes it a column where as 0 remains the same.
x.unsqueeze(0)
tensor([[1, 2, 3, 4]])
>>> x.unsqueeze(1)
tensor([[1],
        [2],
        [3],
        [4]])
>>> x.unsqueeze(-1)
tensor([[1],
        [2],
        [3],
        [4]])

Can someone provide an explanation of what squeeze and unsqueeze are doing to a tensor? And what's the difference between providing the arguements 0, 1 and -1?


title: How to get Docker to recognize NVIDIA drivers?

body:
I have a container that loads a Pytorch model. Every time I try to start it up, I get this error:

Traceback (most recent call last):
  File "server/start.py", line 166, in <module>
    start()
  File "server/start.py", line 94, in start
    app.register_blueprint(create_api(), url_prefix="/api/1")
  File "/usr/local/src/skiff/app/server/server/api.py", line 30, in create_api
    atomic_demo_model = DemoModel(model_filepath, comet_dir)
  File "/usr/local/src/comet/comet/comet/interactive/atomic_demo.py", line 69, in __init__
    model = interactive.make_model(opt, n_vocab, n_ctx, state_dict)
  File "/usr/local/src/comet/comet/comet/interactive/functions.py", line 98, in make_model
    model.to(cfg.device)
  File "/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 381, in to
    return self._apply(convert)
  File "/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 187, in _apply
    module._apply(fn)
  File "/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 187, in _apply
    module._apply(fn)
  File "/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 193, in _apply
    param.data = fn(param.data)
  File "/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 379, in convert
    return t.to(device, dtype if t.is_floating_point() else None, non_blocking)
  File "/usr/local/lib/python3.7/site-packages/torch/cuda/__init__.py", line 161, in _lazy_init
    _check_driver()
  File "/usr/local/lib/python3.7/site-packages/torch/cuda/__init__.py", line 82, in _check_driver
    http://www.nvidia.com/Download/index.aspx""")
AssertionError:
Found no NVIDIA driver on your system. Please check that you
have an NVIDIA GPU and installed a driver from
http://www.nvidia.com/Download/index.aspx

I know that nvidia-docker2 is working.

$ docker run --runtime=nvidia --rm nvidia/cuda:9.0-base nvidia-smi
Tue Jul 16 22:09:40 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.39       Driver Version: 418.39       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce RTX 208...  Off  | 00000000:1A:00.0 Off |                  N/A |
|  0%   44C    P0    72W / 260W |      0MiB / 10989MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce RTX 208...  Off  | 00000000:1B:00.0 Off |                  N/A |
|  0%   44C    P0    66W / 260W |      0MiB / 10989MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  GeForce RTX 208...  Off  | 00000000:1E:00.0 Off |                  N/A |
|  0%   44C    P0    48W / 260W |      0MiB / 10989MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  GeForce RTX 208...  Off  | 00000000:3E:00.0 Off |                  N/A |
|  0%   41C    P0    54W / 260W |      0MiB / 10989MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   4  GeForce RTX 208...  Off  | 00000000:3F:00.0 Off |                  N/A |
|  0%   42C    P0    48W / 260W |      0MiB / 10989MiB |      1%      Default |
+-------------------------------+----------------------+----------------------+
|   5  GeForce RTX 208...  Off  | 00000000:41:00.0 Off |                  N/A |
|  0%   42C    P0     1W / 260W |      0MiB / 10989MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

However, I keep getting the error above.

I've tried the following:

Setting "default-runtime": nvidia in /etc/docker/daemon.json
Using docker run --runtime=nvidia <IMAGE_ID>
Adding the variables below to my Dockerfile:

ENV NVIDIA_VISIBLE_DEVICES all
ENV NVIDIA_DRIVER_CAPABILITIES compute,utility
LABEL com.nvidia.volumes.needed="nvidia_driver"

I expect this container to run - we have a working version in production without these issues. And I know that Docker can find the drivers, as the output above shows. Any ideas?


title: Reproducable Pytorch Results & Random Seeds

body:
I have a simple toy NN with Pytorch. I am setting all the seeds I can find in the docs as well as numpy random.

If I run the code below from top to bottom, the results appear to be reproducible.

BUT, if I run block 1 only once and then each time run block 2, the result changes (sometimes dramatically). I am unsure why this happens since the network is being re-initialized and optimizer reset each time.

I am using version 0.4.0

BLOCK #1

from __future__ import division
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

import torch
import torch.utils.data as utils_data
from torch.autograd import Variable
from torch import optim, nn
from torch.utils.data import Dataset
import torch.nn.functional as F
from torch.nn.init import xavier_uniform_, xavier_normal_,uniform_

torch.manual_seed(123)

import random
random.seed(123)

from sklearn.datasets import load_boston
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import train_test_split

%matplotlib inline

cuda=True #set to true uses GPU

if cuda:
    torch.cuda.manual_seed(123)

#load boston data from scikit
boston = load_boston()
x=boston.data
y=boston.target
y=y.reshape(y.shape[0],1)

#train and test
x_train, x_test, y_train, y_test = train_test_split(x,y,test_size=0.3, random_state=123, shuffle=False)

#change to tensors
x_train = torch.from_numpy(x_train)
y_train = torch.from_numpy(y_train)

#create dataset and use data loader
training_samples = utils_data.TensorDataset(x_train, y_train)
data_loader_trn = utils_data.DataLoader(training_samples, batch_size=64,drop_last=False)

#change to tensors
x_test = torch.from_numpy(x_test)
y_test = torch.from_numpy(y_test)

#create dataset and use data loader
testing_samples = utils_data.TensorDataset(x_test, y_test)
data_loader_test = utils_data.DataLoader(testing_samples, batch_size=64,drop_last=False)

#simple model
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()

        #all the layers
        self.fc1   = nn.Linear(x.shape[1], 20)
        xavier_uniform_(self.fc1.weight.data) #this is how you can change the weight init
        self.drop = nn.Dropout(p=0.5)
        self.fc2   = nn.Linear(20, 1)

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x=  self.drop(x)
        x = self.fc2(x)
        return x

BLOCK #2

net=Net()

if cuda:
    net.cuda()

# create a stochastic gradient descent optimizer
optimizer = optim.Adam(net.parameters())
# create a loss function (mse)
loss = nn.MSELoss(size_average=False)

# run the main training loop
epochs =20
hold_loss=[]

for epoch in range(epochs):
    cum_loss=0.
    cum_records_epoch =0

    for batch_idx, (data, target) in enumerate(data_loader_trn):
        tr_x, tr_y = data.float(), target.float()
        if cuda:
            tr_x, tr_y = tr_x.cuda(), tr_y.cuda()

        # Reset gradient
        optimizer.zero_grad()

        # Forward pass
        fx = net(tr_x)
        output = loss(fx, tr_y) #loss for this batch

        cum_loss += output.item() #accumulate the loss

        # Backward
        output.backward()

        # Update parameters based on backprop
        optimizer.step()

        cum_records_epoch +=len(tr_x)
        if batch_idx % 1 == 0:
            print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
            epoch, cum_records_epoch, len(data_loader_trn.dataset),
            100. * (batch_idx+1) / len(data_loader_trn), output.item()))
    print('Epoch average loss: {:.6f}'.format(cum_loss/cum_records_epoch))

    hold_loss.append(cum_loss/cum_records_epoch)

#training loss
plt.plot(np.array(hold_loss))
plt.show()


title: torchtext ImportError in colab

body:
I am trying to run this tutorial in colab.
However, when I try to import a bunch of modules:
import io
import torch
from torchtext.utils import download_from_url
from torchtext.data.utils import get_tokenizer
from torchtext.vocab import build_vocab_from_iterator

It gives me the errors for extract_archive and build_vocab_from_iterator:
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-5-a24e72502dbc> in <module>()
      1 import io
      2 import torch
----> 3 from torchtext.utils import download_from_url, extract_archive
      4 from torchtext.data.utils import get_tokenizer
      5 from torchtext.vocab import build_vocab_from_iterator

ImportError: cannot import name 'extract_archive'

ImportError                               Traceback (most recent call last)
<ipython-input-4-02a401fd241b> in <module>()
      3 from torchtext.utils import download_from_url
      4 from torchtext.data.utils import get_tokenizer
----> 5 from torchtext.vocab import build_vocab_from_iterator
      6
      7 url = 'https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-2-v1.zip'

ImportError: cannot import name 'build_vocab_from_iterator'

Please help me with this one.


title: How padded sequences given as packed sequences are dealt by RNN in pytorch?

body:
In pytorch, we can give a packed sequence as an input to the RNN. From official doc, input of an RNN can be as follows.

  input (seq_len, batch, input_size): tensor containing the features of the input sequence. The input can also be a packed variable length sequence.

Example

packed = torch.nn.utils.rnn.pack_padded_sequence(embedded, input_lengths)
outputs, hidden = self.rnn(packed, hidden)
outputs, output_lengths = torch.nn.utils.rnn.pad_packed_sequence(outputs)

Here, embedded is the embedded representation of a batch input.

My question is, how the computation is carried out for packed sequences in RNN? How the hidden states are computed for padded sequences in a batch through packed representation?


I just got this message when trying to run a feed forward torch.nn.Conv2d, getting the following stacktrace:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-26-04bd4a00565d> in <module>
      3
      4 # call training function
----> 5 losses = train(D, G, n_epochs=n_epochs)

<ipython-input-24-b539315e0aa0> in train(D, G, n_epochs, print_every)
     46                 real_images = real_images.cuda()
     47
---> 48             D_real = D(real_images)
     49             d_real_loss = real_loss(D_real, True) # smoothing label 1 => 0.9
     50

~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--> 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

<ipython-input-14-bf68e57c25ff> in forward(self, x)
     48         """
     49
---> 50         x = self.leaky_relu(self.conv1(x))
     51         x = self.leaky_relu(self.conv2(x))
     52         x = self.leaky_relu(self.conv3(x))

~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--> 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/container.py in forward(self, input)
     98     def forward(self, input):
     99         for module in self:
--> 100             input = module(input)
    101         return input
    102

~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--> 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/conv.py in forward(self, input)
    347
    348     def forward(self, input):
--> 349         return self._conv_forward(input, self.weight)
    350
    351 class Conv3d(_ConvNd):

~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/conv.py in _conv_forward(self, input, weight)
    344                             _pair(0), self.dilation, self.groups)
    345         return F.conv2d(input, weight, self.bias, self.stride,
--> 346                         self.padding, self.dilation, self.groups)
    347
    348     def forward(self, input):

RuntimeError: Unable to find a valid cuDNN algorithm to run convolution

Running nvidia-smi shows:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.33.01    Driver Version: 440.33.01    CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 770     On   | 00000000:01:00.0 N/A |                  N/A |
| 38%   50C    P8    N/A /  N/A |    624MiB /  4034MiB |     N/A      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0                    Not Supported                                       |
+-----------------------------------------------------------------------------+

I'm using Python 3.7, Pytorch 1.5, and GPU is Nvidia GeForce GTX 770, running on Ubuntu 18.04.2. I haven't found that error message anywhere. Does it ring any bell?.

Thanks a lot in advance.


title: Can´t install Pytorch on PyCharm: No matching distribution found for torch==1.7.0+cpu

body:
I tried multiple times installing Pytorch on Pycharm. I used the code that the pytorch web site give you for a specific configuration. I use this one:
enter image description here
Then I copied this information on Pycharm Terminal and I get this message:
(venv) D:\Usuarios\AuCap\Documents\mnist>pip install torch==1.7.0+cpu torchvision==0.8.1+cpu torchaudio===0.7.0 -f https://download.pytorch.org/w
hl/torch_stable.html
Looking in links: https://download.pytorch.org/whl/torch_stable.html
ERROR: Could not find a version that satisfies the requirement torch==1.7.0+cpu (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)
ERROR: No matching distribution found for torch==1.7.0+cpu
I also tried installing it using Python interpreter in the Pycharm Settings and also didnd´t work.
Thanks for your help


title: Implementing an “infinite loop” Dataset & DataLoader in PyTorch

body:
I’d like to implement an infinite loop Dataset & DataLoader. Here’s what I tried:

class Infinite(Dataset):
    def __len__(self):
        return HPARAMS.batch_size
#         return 1<<30 # This causes huge memory usage.
    def __getitem__(self, idx):
        """Randomly generates one new example."""
        return sample_func_to_be_parallelized()

infinite_loader = DataLoader(
    dataset=Infinite(),
    batch_size=HPARAMS.batch_size,
    num_workers=16,
    worker_init_fn=lambda worker_id: np.random.seed(worker_id),
)

while True:
    for idx, data in enumerate(infinite_loader):
        # forward + backward on "data"

As you can see, the main challenge here is the __len()__ method. If I put a large enough number there, like 1<<30, the symptom is memory usage will JUMP TO 10+GB on the first iteration of train loop. After a while the workers are killed presumably due to OOM.

If I put a small number there, like 1 or BATCH_SIZE, the sampled “data” in the train loop will be periodically duplicated. This is not what I want as I’d like new data to be generated & trained on at every iteration.

I’m guessing the culprit of the excessive memory usage is somewhere in the stack, a bunch of things are cached. Upon a casual look at the Python side of things I can’t pinpoint where.

Can someone advise what’s the best way to have what I want implemented? (Use DataLoader’s parallel loading, while simultaneously guaranteeing every batch loaded is entirely new.)


title: Does it differ to use optimizer.step or model.step in pytorch?

body:
In pytorch, to update the model, should I use optimizer.step() or model.step() ?

Here is a example snippet:

import torch
import torch nn
class SomeNeuralNet(nn.Module):
    def __init__(self,hs,es,dropout):
          SomeNeuralNet(ClaimRecognizer, self).__init__()
          # Some initialization here
    def forward(x):
          # forward propagation here

model = SomeNeuralNet(es,hs,dp)
optimizer = optim.Adam(model.parameters())
loss_function = nn.NLLLoss()
for epoch in N:
   for x in data:
      logp = model(x)
      loss = loss_function(logp,gold_outs)
      loss.backward()
       # Which one I should call ? Optimizer.step() or model.step() or both ?
       optimizer.step()
       model.step()


title: Using torch.cat on list of tensors

body:
I have a list of torch tensors
list_tensor = [tensor([[1, 2, 3],
                       [3, 4, 5]]),
               tensor([[4, 5, 6],
                       [6, 4, 3]]),
               tensor([[4, 2, 1],
                       [3, 3, 1]]),
               tensor([[1, 4, 5],
                       [3, 1, 0]]),
               tensor([[1, 3, 3],
                       [2, 2, 2]])]

I want to do a cross validation on this set, so I want to consider four tensors as training, and keep 1 for testing - and I want to do this for len(list_tensor) times.
So I thought of doing,
for num in range(1, len(list_tensor) + 1):
       train_x = torch.cat((list_tensor[:num], list_tensor[num:]))

The problem is I cannot use lists for a torch.cat operation because both  list_tensor[:num] and  list_tensor[num:] return lists. For example, for num = 1,
list_tensor[:num] = [tensor([[1, 2, 3], [3, 4, 5]])]
list_tensor[num:] = [tensor([[4, 5, 6], [6, 4, 3]]), tensor([[4, 2, 1],[3, 3, 1]]), tensor([[1, 4, 5],
                      [3, 1, 0]]), tensor([[1, 3, 3], [2, 2, 2]])]

How do I perform torch.cat on this?