title: Efficient PyTorch DataLoader collate_fn function for inputs of various dimensions

body:
I'm having trouble writing a custom collate_fn function for the PyTorch DataLoader class. I need the custom function because my inputs have different dimensions.

I'm currently trying to write the baseline implementation of the Stanford MURA paper. The dataset has a set of labeled studies. A study may contain more than one image. I created a custom Datasetclass that stacks these multiple images using torch.stack.

The stacked tensor is then provided as input to the model and the list of outputs is averaged to obtain a single output. This implementation works fine with DataLoader when batch_size=1. However, when I try to set the batch_size to 8, as is the case in the original paper, the DataLoader fails since it uses torch.stack to stack the batch and the inputs in my batch have variable dimensions (since each study can have multiple number of images).

In order to fix this, I tried to implement my custom collate_fn function.

def collate_fn(batch):
    imgs = [item['images'] for item in batch]
    targets = [item['label'] for item in batch]
    targets = torch.LongTensor(targets)
    return imgs, targets

Then in my training epoch loop, I loop through each batch like this:

for image, label in zip(*batch):
    label = label.type(torch.FloatTensor)
    # wrap them in Variable
    image = Variable(image).cuda()
    label = Variable(label).cuda()
    # forward
    output = model(image)
    output = torch.mean(output)
    loss = criterion(output, label, phase)

However, this does not give me any improved timings on the epoch and still takes as long as it did with a batch size of only 1. I've also tried setting the batch size to 32 and that does not improve the timings either.

Am I doing something wrong?
Is there a better approach to this?


title: Column/row slicing a torch sparse tensor

body:
I have a pytorch sparse tensor that I need sliced row/column wise using this slice [idx][:,idx] where idx is a list of indexes, using the mentioned slice yields my desired result on an ordinary float tensor. Is it possible applying the same slicing on a sparse tensor? Example here:

#constructing sparse matrix
i = np.array([[0,1,2,2],[0,1,2,1]])
v = np.ones(4)
i = torch.from_numpy(i.astype("int64"))
v = torch.from_numpy(v.astype("float32"))
test1 = torch.sparse.FloatTensor(i, v)

#constructing float tensor
test2 = np.array([[1,0,0],[0,1,0],[0,1,1]])
test2 = autograd.Variable(torch.cuda.FloatTensor(test2), requires_grad=False)

#slicing
idx = [1,2]
print(test2[idx][:,idx])

output:

Variable containing:
 1  0
 1  1
[torch.cuda.FloatTensor of size 2x2 (GPU 0)]

I am holding a 250.000 x 250.000 adjacency matrix, where I need to slice n rows and n columns, using the random idx, by simply sampling n random idx's. Since the dataset is so large it is not realistic to convert to a more convenient datatype.

can I achieve the same slicing result on test1? Is it even possible? If not, are there any work-arounds?

Right now I am running my model with the following "hack" of a solution:

idx = sorted(random.sample(range(0, np.shape(test1)[0]), 9000))
test1 = test1AsCsr[idx][:,idx].todense().astype("int32")
test1 = autograd.Variable(torch.cuda.FloatTensor(test1), requires_grad=False)

Where test1AsCsr is my test1 converted to a numpy CSR matrix. This solution works, it is however very slow, and makes my GPU utilization very low, since it needs to read/write from CPU memory, constantly.

Edit: Its fine with a non-sparse tensor as result


title: what does dim=-1 or -2 mean in torch.sum()?

body:
let me take a 2D matrix as example:

mat = torch.arange(9).view(3, -1)

tensor([[0, 1, 2],
        [3, 4, 5],
        [6, 7, 8]])

torch.sum(mat, dim=-2)

tensor([ 9, 12, 15])

I find the result of torch.sum(mat, dim=-2) is equal to torch.sum(mat, dim=0) and dim=-1 equal to dim=1. My question is how to understand the negative dimension here. What if the input matrix has 3 or more dimensions?


I am running a nn on colab and came across this error which was not there when i ran the same code on my local system. I have tried with reduced batch size too but the error still persists.

Loading dataset
Start training
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-4-37432f9d142a> in <module>()
     70           start_epoch=start_epoch, log=log_interval,
     71           checkpoint_path=os.path.join(dataset_dir, "cnn_block_frame_flow"),
---> 72           validate=True, resume=False, flow=True, use_cuda=cuda)
     73
     74     #model = models.model()

/content/KTH-Action-Recognition/main/train_helper.py in train(model, num_epochs, train_set, dev_set, lr, batch_size, start_epoch, log, checkpoint_path, validate, resume, flow, use_cuda)
    107             outputs = get_outputs(model, samples["instance"], flow=flow,
    108                                   use_cuda=use_cuda)
--> 109             loss = criterion(outputs, labels)
    110             loss.backward()
    111             optimizer.step()

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    487             result = self._slow_forward(*input, **kwargs)
    488         else:
--> 489             result = self.forward(*input, **kwargs)
    490         for hook in self._forward_hooks.values():
    491             hook_result = hook(self, input, result)

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/loss.py in forward(self, input, target)
    902     def forward(self, input, target):
    903         return F.cross_entropy(input, target, weight=self.weight,
--> 904                                ignore_index=self.ignore_index, reduction=self.reduction)
    905
    906

/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction)
   1968     if size_average is not None or reduce is not None:
   1969         reduction = _Reduction.legacy_get_string(size_average, reduce)
-> 1970     return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
   1971
   1972

/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)
   1788                          .format(input.size(0), target.size(0)))
   1789     if dim == 2:
-> 1790         ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
   1791     elif dim == 4:
   1792         ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)

RuntimeError: Expected object of scalar type Long but got scalar type Byte for argument #2 'target'

Can someone tell me what is causing this error? thank you


title: PyTorch: passing numpy array for weight initialization

body:
I'd like to initialize the parameters of RNN with np arrays.

In the following example, I want to pass w to the parameters of rnn. I know pytorch provides many initialization methods like Xavier, uniform, etc., but is there way to initialize the parameters by passing numpy arrays?

import numpy as np
import torch as nn
rng = np.random.RandomState(313)
w = rng.randn(input_size, hidden_size).astype(np.float32)

rnn = nn.RNN(input_size, hidden_size, num_layers)


title: tf.cast equivalent in pytorch?

body:
I am new to PyTorch. TensorFlow has an API tf.cast() and tf.shape(). the tf.cast has specific purpose in TensorFlow, is there anything equivalent in torch?
i have tensor x= tensor(shape(128,64,32,32)): tf.shape(x) create tensor of dimension 1 x.shape create the true dimension. i need to use tf.shape(x) in torch.

tf.cast has a different role than just changing tensor dtype in torch.

did anyone have equivalent API in torch/PyTorch.


I try to run the following programe for images classification problem in Pytorch:

import torch
import torch.nn as nn
import torchvision
import torchvision.transforms as transforms
import torch.utils.data as data

# Device configuration
device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')

# Hyper parameters
num_epochs = 5
num_classes = 10
batch_size = 100
learning_rate = 0.001

TRAIN_DATA_PATH = "train/"
TEST_DATA_PATH = "test/"
TRANSFORM_IMG = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(256),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406],
                         std=[0.229, 0.224, 0.225] )
    ])

train_dataset = torchvision.datasets.ImageFolder(root=TRAIN_DATA_PATH, transform=TRANSFORM_IMG)
train_loader = data.DataLoader(train_dataset, batch_size=batch_size, shuffle=True,  num_workers=4)
test_dataset = torchvision.datasets.ImageFolder(root=TEST_DATA_PATH, transform=TRANSFORM_IMG)
test_loader  = data.DataLoader(test_dataset, batch_size=batch_size, shuffle=True, num_workers=4)

# Convolutional neural network (two convolutional layers)
class ConvNet(nn.Module):
    def __init__(self, num_classes=10):
        super(ConvNet, self).__init__()
        self.layer1 = nn.Sequential(
            nn.Conv2d(1, 16, kernel_size=5, stride=1, padding=2),
            nn.BatchNorm2d(16),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2))
        self.layer2 = nn.Sequential(
            nn.Conv2d(16, 32, kernel_size=5, stride=1, padding=2),
            nn.BatchNorm2d(32),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2))
        self.fc = nn.Linear(7 * 7 * 32, num_classes)

    def forward(self, x):
        out = self.layer1(x)
        out = self.layer2(out)
        out = out.reshape(out.size(0), -1)
        out = self.fc(out)
        return out

model = ConvNet(num_classes).to(device)

# Loss and optimizer
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

# Train the model
total_step = len(train_loader)
for epoch in range(num_epochs):
    for i, (images, labels) in enumerate(train_loader):
        images = images.to(device)
        labels = labels.to(device)

        # Forward pass
        outputs = model(images)
        loss = criterion(outputs, labels)

        # Backward and optimize
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if (i + 1) % 100 == 0:
            print('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}'
                  .format(epoch + 1, num_epochs, i + 1, total_step, loss.item()))

# Test the model
model.eval()  # eval mode (batchnorm uses moving mean/variance instead of mini-batch mean/variance)
with torch.no_grad():
    correct = 0
    total = 0
    for images, labels in test_loader:
        images = images.to(device)
        labels = labels.to(device)
        outputs = model(images)
        _, predicted = torch.max(outputs.data, 1)
        total += labels.size(0)
        correct += (predicted == labels).sum().item()

    print('Test Accuracy of the model on the 10000 test images: {} %'.format(100 * correct / total))

# Save the model checkpoint
torch.save(model.state_dict(), 'model/model.ckpt')

But I get a RuntimeError:

Traceback (most recent call last):

RuntimeError: Given groups=1, weight of size 16 1 5 5, expected input[100, 3, 256, 256] to have 1 channels, but got 3 channels instead

Someone could help to fix the bug? Thanks a lot.

Reference related:

https://discuss.pytorch.org/t/given-groups-1-weight-16-1-5-5-so-expected-input-100-3-64-64-to-have-1-channels-but-got-3-channels-instead/28831/17

RuntimeError: Given groups=1, weight of size [64, 3, 7, 7], expected input[3, 1, 224, 224] to have 3 channels, but got 1 channels instead


title: How to use if statement PyTorch using torch.FloatTensor

body:
I am trying to use the if statement in my PyTorch code using torch.FloatTensor as data type, to speed it up into the GPU.

This is my code:

import torch
import time
def fitness(x):
     return torch.pow(x, 2)

def velocity(v, gxbest, pxbest, pybest, x, pop):
     return torch.rand(pop).type(dtype)*v + \
            torch.rand(pop).type(dtype)*(pxbest - x) + \
            torch.rand(pop).type(dtype)*(gxbest.expand(x.size(0)) - x)

dtype = torch.cuda.FloatTensor
def main():

    pop, xmax, xmin, niter = 300000, 50, -50, 100
    v                      = torch.rand(pop).type(dtype)
    x                      = (xmax-xmin)*torch.rand(pop).type(dtype)+xmin
    y                      = fitness(x)
    [miny, indexminy]      = y.min(0)
    gxbest                 = x[indexminy]
    pxbest                 = x
    pybest                 = y

    for K in range(niter):

        vnext = velocity(v, gxbest, pxbest, pybest, x, pop)

        xnext = x + vnext
        ynext = fitness(x)
        [minynext, indexminynext]  = ynext.min(0)

        if (minynext < miny):
            miny   = minynext
            gxbest = xnext[indexminynext]

        indexpbest         = (ynext < pybest)
        pxbest[indexpbest] = xnext[indexpbest]
        pybest[indexpbest] = ynext[indexpbest]
        x                  = xnext
        v                  = vnext
main()

Unfortanally it is not working. It is giving me a error message and I can not figure it out what is the problem.

  RuntimeError: bool value of non-empty torch.cuda.ByteTensor objects is ambiguous

How can I use the if in PyTorch? I tried to convert the cuda.Tensor into a numpy array but it did not work also.

  minynext = minynext.cpu().numpy()
  miny = miny.cpu().numpy()

PS: Am I doing the code the efficient/faster way possible ? Or should I change something to achieve faster results?


title: PyCharm won't install PyTorch - Building wheel failed

body:
I have tried for some time to install PyTorch and TorchVision to aid in some neural networks / ML projects I'm doing.

My issue is that I can' seem to get the installation to work, neither in PyCharm pip, Anaconda prompt or the cmd/with the pip3 command.

The output I get when trying to run the following command through PyCharm:

pip install --user pytorch==0.1.2

is this:

Collecting pytorch
  Using cached https://files.pythonhosted.org/packages/a9/41/4487bc23e3ac4d674943176f5aa309427b011e00607eb98899e9d951f67b/pytorch-0.1.2.tar.gz
Building wheels for collected packages: pytorch
  Running setup.py bdist_wheel for pytorch: started
  Running setup.py bdist_wheel for pytorch: finished with status 'error'
  Complete output from command C:\Users\Even\Anaconda3\envs\python.exe -u -c "import setuptools, tokenize;__file__='C:\\Users\\Even\\AppData\\Local\\Temp\\pycharm-packaging\\pytorch\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d C:\Users\Even\AppData\Local\Temp\pip-wheel-2sfgqmm7 --python-tag cp36:
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "C:\Users\Even\AppData\Local\Temp\pycharm-packaging\pytorch\setup.py", line 17, in <module>
      raise Exception(message)
  Exception: You should install pytorch from http://pytorch.org

  ----------------------------------------
  Running setup.py clean for pytorch
Failed to build pytorch
Installing collected packages: pytorch
  Running setup.py install for pytorch: started
    Running setup.py install for pytorch: finished with status 'error'
    Complete output from command C:\Users\Even\Anaconda3\envs\python.exe -u -c "import setuptools, tokenize;__file__='C:\\Users\\Even\\AppData\\Local\\Temp\\pycharm-packaging\\pytorch\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record C:\Users\Even\AppData\Local\Temp\pip-record-1jx5e1ke\install-record.txt --single-version-externally-managed --compile --user --prefix=:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\Users\Even\AppData\Local\Temp\pycharm-packaging\pytorch\setup.py", line 13, in <module>
        raise Exception(message)
    Exception: You should install pytorch from http://pytorch.org

    ----------------------------------------

  Failed building wheel for pytorch
Command "C:\Users\Even\Anaconda3\envs\python.exe -u -c "import setuptools, tokenize;__file__='C:\\Users\\Even\\AppData\\Local\\Temp\\pycharm-packaging\\pytorch\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record C:\Users\Even\AppData\Local\Temp\pip-record-1jx5e1ke\install-record.txt --single-version-externally-managed --compile --user --prefix=" failed with error code 1 in C:\Users\Even\AppData\Local\Temp\pycharm-packaging\pytorch\

I've also tried this command (CMD):

pip3 install http://download.pytorch.org/whl/cpu/torch-0.4.1-cp36-cp36m-
win_amd64.whl

Which returned in this output:

torch-0.4.1-cp36-cp36m-win_amd64.whl is not a supported wheel on this
platform.

I might be missing some crucial information, but I've been googling, searching forums, and asked people I know about this. No one has an answer. I hope some of you may have encountered the same issue.

Thanks.


title: Problem with Dataloader object not subscriptable

body:
I am now running a Python program using Pytorch. I use my own dataset, not torch.data.dataset. I download data from a pickle file extracted from feature extraction. But the following errors appear:

Traceback (most recent call last):
  File "C:\Users\hp\Downloads\efficient_densenet_pytorch-master\demo-emotion.py", line 326, in <module>
    fire.Fire(demo)
  File "C:\Users\hp\Anaconda3\envs\tf-gpu\lib\site-packages\fire\core.py", line 138, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "C:\Users\hp\Anaconda3\envs\tf-gpu\lib\site-packages\fire\core.py", line 468, in _Fire
    target=component.__name__)
  File "C:\Users\hp\Anaconda3\envs\tf-gpu\lib\site-packages\fire\core.py", line 672, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "C:\Users\hp\Downloads\efficient_densenet_pytorch-master\demo-emotion.py", line 304, in demo
    train(model,train_set1, valid_set=valid_set, test_set=test1, save=save, n_epochs=n_epochs,batch_size=batch_size,seed=seed)
  File "C:\Users\hp\Downloads\efficient_densenet_pytorch-master\demo-emotion.py", line 172, in train
    n_epochs=n_epochs,
  File "C:\Users\hp\Downloads\efficient_densenet_pytorch-master\demo-emotion.py", line 37, in train_epoch
    loader=np.asarray(list(loader))
  File "C:\Users\hp\Anaconda3\envs\tf-gpu\lib\site-packages\torch\utils\data\dataloader.py", line 345, in __next__
    data = self._next_data()
  File "C:\Users\hp\Anaconda3\envs\tf-gpu\lib\site-packages\torch\utils\data\dataloader.py", line 385, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "C:\Users\hp\Anaconda3\envs\tf-gpu\lib\site-packages\torch\utils\data\_utils\fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "C:\Users\hp\Anaconda3\envs\tf-gpu\lib\site-packages\torch\utils\data\_utils\fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "C:\Users\hp\Anaconda3\envs\tf-gpu\lib\site-packages\torch\utils\data\dataset.py", line 257, in __getitem__
    return self.dataset[self.indices[idx]]
TypeError: 'DataLoader' object is not subscriptable

The code is:

train_set1 = Owndata()

train1, test1 = train_set1 .get_splits()
# prepare data loaders
train_dl = torch.utils.data.DataLoader(train1, batch_size=32, shuffle=True)
test_dl =torch.utils.data.DataLoader(test1, batch_size=1024, shuffle=False)
test_set1 = Owndata()
'''print('test_set# ',test_set)'''
if valid_size:
    valid_set = Owndata()
    indices = torch.randperm(len(train_set1))
    train_indices = indices[:len(indices) - valid_size]
    valid_indices = indices[len(indices) - valid_size:]
    train_set1 = torch.utils.data.Subset(train_dl, train_indices)
    valid_set = torch.utils.data.Subset(valid_set, valid_indices)
else:
    valid_set = None
model = DenseNet(
    growth_rate=growth_rate,
    block_config=block_config,
    num_classes=10,
    small_inputs=True,
    efficient=efficient,
)
train(model,train_set1, valid_set=valid_set, test_set=test1, save=save, n_epochs=n_epochs, batch_size=batch_size, seed=seed)

Any help is appreciated! Thanks a lot in advance!!


title: Tensorflow 2.0 dataset and dataloader

body:
I am a pytorch user, and I am used to the data.dataset and data.dataloader api in pytorch. I am trying to build a same model with tensorflow 2.0, and I wonder whether there is an api that works similarly with these api in pytorch.

If there is no such api, can any of you tell me how people usually do to implement the data loading part in tensorflow ? I've used tensorflow 1, but never had an experience with dataset api. I've hard coded before. I hope there is something like overriding getitem with only index as an input.

Thanks much in advance.


title: BertForSequenceClassification vs. BertForMultipleChoice for sentence multi-class classification

body:
I'm working on a text classification problem (e.g. sentiment analysis), where I need to classify a text string into one of five classes.
I just started using the Huggingface Transformer package and BERT with PyTorch. What I need is a classifier with a softmax layer on top so that I can do 5-way classification. Confusingly, there seem to be two relevant options in the Transformer package: BertForSequenceClassification and BertForMultipleChoice.
Which one should I use for my 5-way classification task? What are the appropriate use cases for them?
The documentation for BertForSequenceClassification doesn't mention softmax at all, although it does mention cross-entropy. I am not sure if this class is only for 2-class classification (i.e. logistic regression).

Bert Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks.

labels (torch.LongTensor of shape (batch_size,), optional, defaults to None) – Labels for computing the sequence classification/regression loss. Indices should be in [0, ..., config.num_labels - 1]. If config.num_labels == 1 a regression loss is computed (Mean-Square loss), If config.num_labels > 1 a classification loss is computed (Cross-Entropy).

The documentation for BertForMultipleChoice mentions softmax, but the way the labels are described, it sound like this class is for multi-label classification (that is, a binary classification for multiple labels).

Bert Model with a multiple choice classification head on top (a linear layer on top of the pooled output and a softmax) e.g. for RocStories/SWAG tasks.

labels (torch.LongTensor of shape (batch_size,), optional, defaults to None) – Labels for computing the multiple choice classification loss. Indices should be in [0, ..., num_choices] where num_choices is the size of the second dimension of the input tensors.

Thank you for any help.


title: What does the -1 mean in tensor.size(-1)?

body:
I have seen something like this in the Pytorch documentation,

import torch

a = torch.tensor([1, 2])
a.size() # torch.Size([2])
a.size(-1) # 2

How does this work? I didn't find a description. Thanks,


I'm using PyTorch,but I get a error!
My error code as following:

for train_data in trainloader:
    example_count += 1
    if example_count == 100:
        break
    optimer.zero_grad()
    image, label = train_data
    image = image.cuda()
    label = label.cuda()
    out = model(image)
    _, out = torch.max(out, 1)
    # print(out.cpu().data.numpy())
    # print(label.cpu().data.numpy())
    # out = torch.zeros(4, 10).scatter_(1, out.cpu(), 1).cuda()
    # label= torch.zeros(4, 10).scatter_(1, label.cpu(), 1).cuda()
    l = loss(out, label)
    l.bakeward()
    optimer.setp()
    j += 1
    count += label.size(0)
    acc += (out == label).sum().item()
    if j % 1000 == 0:
        print(j + ' step:curent accurity is %f' % (acc / count))

the traceback:

    Traceback (most recent call last):
  File "VGG实现.py", line 178, in <module>
    utils.train(testloader,model)
  File "VGG实现.py", line 153, in train
    l=loss(out,label)
  File "/home/tang/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/tang/anaconda3/lib/python3.7/site-packages/torch/nn/modules/loss.py", line 435, in forward
    return F.mse_loss(input, target, reduction=self.reduction)
  File "/home/tang/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py", line 2156, in mse_loss
    ret = torch._C._nn.mse_loss(expanded_input, expanded_target, _Reduction.get_enum(reduction))
RuntimeError: _thnn_mse_loss_forward is not implemented for type torch.cuda.LongTensor

I get a answer whose here
Pytorch RuntimeError: "host_softmax" not implemented for 'torch.cuda.LongTensor'

But I don't know how to solve this question.


title: Data loading with variable batch size?

body:
I am currently working on patch based super-resolution. Most of the papers divide an image into smaller patches and then use the patches as input to the models.I was able to create patches using custom dataloader. The code is given below:

import torch.utils.data as data
from torchvision.transforms import CenterCrop, ToTensor, Compose, ToPILImage, Resize, RandomHorizontalFlip, RandomVerticalFlip
from os import listdir
from os.path import join
from PIL import Image
import random
import os
import numpy as np
import torch

def is_image_file(filename):
    return any(filename.endswith(extension) for extension in [".png", ".jpg", ".jpeg", ".bmp"])

class TrainDatasetFromFolder(data.Dataset):
    def __init__(self, dataset_dir, patch_size, is_gray, stride):
        super(TrainDatasetFromFolder, self).__init__()
        self.imageHrfilenames = []
        self.imageHrfilenames.extend(join(dataset_dir, x)
                                     for x in sorted(listdir(dataset_dir)) if is_image_file(x))
        self.is_gray = is_gray
        self.patchSize = patch_size
        self.stride = stride

    def _load_file(self, index):
        filename = self.imageHrfilenames[index]
        hr = Image.open(self.imageHrfilenames[index])
        downsizes = (1, 0.7, 0.45)
        downsize = 2
        w_ = int(hr.width * downsizes[downsize])
        h_ = int(hr.height * downsizes[downsize])
        aug = Compose([Resize([h_, w_], interpolation=Image.BICUBIC),
                       RandomHorizontalFlip(),
                       RandomVerticalFlip()])

        hr = aug(hr)
        rv = random.randint(0, 4)
        hr = hr.rotate(90*rv, expand=1)
        filename = os.path.splitext(os.path.split(filename)[-1])[0]
        return hr, filename

    def _patching(self, img):

        img = ToTensor()(img)
        LR_ = Compose([ToPILImage(), Resize(self.patchSize//2, interpolation=Image.BICUBIC), ToTensor()])

        HR_p, LR_p = [], []
        for i in range(0, img.shape[1] - self.patchSize, self.stride):
            for j in range(0, img.shape[2] - self.patchSize, self.stride):
                temp = img[:, i:i + self.patchSize, j:j + self.patchSize]
                HR_p += [temp]
                LR_p += [LR_(temp)]

        return torch.stack(LR_p),torch.stack(HR_p)

    def __getitem__(self, index):
        HR_, filename = self._load_file(index)
        LR_p, HR_p = self._patching(HR_)
        return LR_p, HR_p

    def __len__(self):
        return len(self.imageHrfilenames)

Suppose the batch size is 1, it takes an image and gives an output of size [x,3,patchsize,patchsize]. When batch size is 2, I will have two different outputs of size [x,3,patchsize,patchsize] (for example image 1 may give[50,3,patchsize,patchsize], image 2 may give[75,3,patchsize,patchsize] ). To handle this a custom collate function was required that stacks these two outputs along dimension 0. The collate function is given below:

def my_collate(batch):
    data = torch.cat([item[0] for item in batch],dim = 0)
    target = torch.cat([item[1] for item in batch],dim = 0)

    return [data, target]

This collate function concatenates along x (From the above example, I finally get [125,3,patchsize,pathsize]. For training purposes, I need to train the model using a minibatch size of say 25. Is there any method or any functions which I can use to directly get an output of size [25 , 3, patchsize, pathsize] directly from the dataloader using the necessary number of images as input to the Dataloader?


title: How to convert 3D tensor to 2D tensor in pytorch?

body:
I am new to pytorch. I have 3D tensor (32,10,64) and I want a 2D tensor (32, 64).
I tried view() and used after passing to linear layer squeeze() which converted it to (32,10).


title: How can we convert a .pth model into .pb file?

body: I have already got the complete model by using pytorch, however I wanna convert the .pth file into .pb, which could be used in Tensorflow. Does anyone have some ideas?


title: Pytorch custom activation functions?

body: I'm having issues with implementing custom activation functions in Pytorch, such as Swish. How should I go about implementing and using custom activation functions in Pytorch?


title: Pytorch: define custom function

body:
I wanted to write my own activation function, but I got a problem. Saying the matrix multiplication will call .data. I searched but got little useful information. Any help will be appreciated. The error information is

 Traceback (most recent call last):
      File "defineAutogradFuncion.py", line 126, in <module>
        test = gradcheck(argmin, input, eps=1e-6, atol=1e-4)
      File "/home/zhaosl/.local/lib/python2.7/site-packages/torch/autograd/gradcheck.py", line 154, in gradcheck
        output = func(*inputs)
      File "defineAutogradFuncion.py", line 86, in forward
        output = output.mm(dismap).squeeze(-1)
      File "/home/zhaosl/.local/lib/python2.7/site-packages/torch/autograd/variable.py", line 578, in mm
        output = Variable(self.data.new(self.data.size(0), matrix.data.size(1)))
      File "/home/zhaosl/.local/lib/python2.7/site-packages/torch/tensor.py", line 374, in data
        raise RuntimeError('cannot call .data on a torch.Tensor: did you intend to use autograd.Variable?')
    RuntimeError: cannot call .data on a torch.Tensor: did you intend to use autograd.Variable?

    class Softargmin(torch.autograd.Function):
    """
    We can implement our own custom autograd Functions by subclassing
    torch.autograd.Function and implementing the forward and backward passes
    which operate on Tensors.
    """
    @staticmethod
    def forward(self, input):
        """
        In the forward pass we receive a Tensor containing the input and return a
        Tensor containing the output. You can cache arbitrary Tensors for use in the
        backward pass using the save_for_backward method.
        """
        #P = Fun.softmax(-input)
        inputSqueeze = input.squeeze(-1)
        P = Fun.softmax(-inputSqueeze)
        self.save_for_backward(P)

        output = P.permute(0,2,3,1)
        dismap = torch.arange(0,output.size(-1)+1).unsqueeze(1)
        output = output.mm(dismap).squeeze(-1)
       return output
    @staticmethod
    def backward(self, grad_output):
        """
        In the backward pass we receive a Tensor containing the gradient of the loss
        with respect to the output, and we need to compute the gradient of the loss
        with respect to the input.
        """
        P, = self.saved_tensors
        P = P.unsqueeze(-1)
        Pk = torch.squeeze(P,-1).permute(0,2,3,1)
        k = torch.arange(0,Pk.size(-1)+1).unsqueeze(1)
        sumkPk = Pk.mm(k)
        sumkPk = sumkPk.unsqueeze(1).expand(P.size())
        i = torch.arange(0,Pk.size(-1)+1).view(1,-1,1,1,1).expand(P.size())
        grad_output_expand =grad_output.unsqueeze(-1).unsqueeze(1).expand(P.size())
        grad_input = grad_output_expand*P*(sumkPk-i)
        return grad_input


title: torch.nn.sequential vs. combination of multiple torch.nn.linear

body:
I'm trying to create a multi layer neural net class in pytorch. I want to know if the following 2 pieces of code create the same network.

Model 1 with nn.Linear

class TestModel(nn.Module):
def __init__(self, input_dim, hidden_dim, output_dim):
    super(TestModel, self).__init__()
    self.fc1 = nn.Linear(input_dim,hidden_dim)
    self.fc2 = nn.Linear(hidden_dim,output_dim)

def forward(self, x):
    x = nn.functional.relu(self.fc1(x))
    x = nn.functional.softmax(self.fc2(x))
    return x

Model 2 with nn.Sequential

class TestModel2(nn.Module):
def __init__(self, input, hidden, output):
    super(TestModel2, self).__init__()
    self.seq = nn.Sequential(
               nn.Linear(input_dim,hidden_dim),
               nn.ReLU(),
               nn.Linear(hidden_dim,output_dim),
               nn.Softmax()
               )

def forward(self, x):
    return self.seq(x)


title: Pytorch CNN error: Expected input batch_size (4) to match target batch_size (64)

body:
I've been teaching myself this since November and any help on this would be really appreciated, thank you for looking, as I seem to be going round in circles. I am trying to use a Pytorch CNN example that was used with the Mnist dataset. Now I am trying to modify the CNN for facial key point recognition.  I am using the Kaggle dataset (CSV) of 7048 training images and key points (15 key points per face) and 1783 test images.  I split training dataset and converted the images to jpeg, made separate file for the key points (shape 15, 2). I have made dataset and data loader and can iterate through and display images and plot key points. When I run the CNN I am getting this error.

> Net(
  (conv1): Conv2d(1, 32, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2))
  (conv2): Conv2d(32, 64, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2))
  (conv2_drop): Dropout2d(p=0.5)
  (fc1): Linear(in_features=589824, out_features=100, bias=True)
  (fc2): Linear(in_features=100, out_features=30, bias=True)
)
Data and target shape:  torch.Size([64, 96, 96])   torch.Size([64, 15, 2])
Data and target shape:  torch.Size([64, 1, 96, 96])   torch.Size([64, 15, 2])

Traceback (most recent call last):
  File "/home/keith/PycharmProjects/FacialLandMarks/WorkOut.py", line 416, in <module>
    main()
  File "/home/keith/PycharmProjects/FacialLandMarks/WorkOut.py", line 412, in main
    train(args, model, device, train_loader, optimizer, epoch)
  File "/home/keith/PycharmProjects/FacialLandMarks/WorkOut.py", line 324, in train
    loss = F.nll_loss(output, target)
  File "/home/keith/Desktop/PycharmProjects/fkp/FacialLandMarks/lib/python3.6/site-packages/torch/nn/functional.py", line 1788, in nll_loss
    .format(input.size(0), target.size(0)))
ValueError: Expected input batch_size (4) to match target batch_size (64).

Process finished with exit code 1

Here are some links I have read, I could not figure out the problem
but may help some one else.

    https://github.com/pytorch/pytorch/issues/11762
        How do I modify this PyTorch convolutional neural network to accept a 64 x 64 image and properly output predictions?
            pytorch-convolutional-neural-network-to-accept-a-64-x-64-im
        Pytorch Validating Model Error: Expected input batch_size (3) to match target batch_size (4)
            model-error-expected-input-batch-size-3-to-match-target-ba

Here is my code:

    class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 32, kernel_size=5, stride=1, padding=(2, 2))
        self.conv2 = nn.Conv2d(32, 64, kernel_size=5, stride=1, padding=(2, 2))
        self.conv2_drop = nn.Dropout2d()
        self.fc1 = nn.Linear(64 * 96 * 96, 100)
        self.fc2 = nn.Linear(100, 30)  # 30 is x and y key points

    def forward(self, x):
        x = F.relu(F.max_pool2d(self.conv1(x), 2))
        x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))
        x = x.view(-1, 64 * 96 * 96)
        # x = x.view(x.size(0), -1)
        # x = x.view(x.size()[0], 30, -1)
        x = F.relu(self.fc1(x))
        x = F.dropout(x, training=self.training)
        x = self.fc2(x)
        return F.log_softmax(x, dim=1)

def train(args, model, device, train_loader, optimizer, epoch):
    model.train()
    for batch_idx, batch in enumerate(train_loader):
        data = batch['image']
        target = batch['key_points']
        print('Data and target shape: ', data.shape, ' ', target.shape)
        data, target = data.to(device), target.to(device)
        optimizer.zero_grad()
        data = data.unsqueeze(1).float()

        print('Data and target shape: ', data.shape, ' ', target.shape)

        output = model(data)
        loss = F.nll_loss(output, target)
        loss.backward()
        optimizer.step()
        if batch_idx % args.log_interval == 0:
            print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                epoch, batch_idx * len(data), len(train_loader.dataset),
                100. * batch_idx / len(train_loader), loss.item()))

# def test(args, model, device, test_loader):
#     model.eval()
#     test_loss = 0
#     correct = 0
#     with torch.no_grad():
#         for data, target in test_loader:
#             data, target = data.to(device), target.to(device)
#             output = model(data)
#             test_loss += F.nll_loss(output, target, reduction='sum').item() # sum up batch loss
#             pred = output.max(1, keepdim=True)[1] # get the index of the max log-probability
#             correct += pred.eq(target.view_as(pred)).sum().item()
#
#     test_loss /= len(test_loader.dataset)
#     print('\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\n'.format(
#         test_loss, correct, len(test_loader.dataset),
#         100. * correct / len(test_loader.dataset)))

def main():
    # Training settings
    parser = argparse.ArgumentParser(description='Project')
    parser.add_argument('--batch-size', type=int, default=64, metavar='N',
                        help='input batch size for training (default: 64)')
    parser.add_argument('--test-batch-size', type=int, default=1000, metavar='N',
                        help='input batch size for testing (default: 1000)')
    parser.add_argument('--epochs', type=int, default=10, metavar='N',   # ========  epoch
                        help='number of epochs to train (default: 10)')
    parser.add_argument('--lr', type=float, default=0.01, metavar='LR',
                        help='learning rate (default: 0.01)')
    parser.add_argument('--momentum', type=float, default=0.5, metavar='M',
                        help='SGD momentum (default: 0.5)')
    parser.add_argument('--no-cuda', action='store_true', default=False,
                        help='disables CUDA training')
    parser.add_argument('--seed', type=int, default=1, metavar='S',
                        help='random seed (default: 1)')
    parser.add_argument('--log-interval', type=int, default=10, metavar='N',
                        help='how many batches to wait before logging training status')
    args = parser.parse_args()
    use_cuda = not args.no_cuda and torch.cuda.is_available()

    torch.manual_seed(args.seed)

    device = torch.device("cuda" if use_cuda else "cpu")

    kwargs = {'num_workers': 1, 'pin_memory': True} if use_cuda else {}
    train_data_set = FaceKeyPointDataSet(csv_file='faces/Kep_points_and_id.csv',
                                         root_dir='faces/',
                                         transform=transforms.Compose([
                                             # Rescale(96),
                                             ToTensor()
                                         ]))

    train_loader = DataLoader(train_data_set, batch_size=args.batch_size,
                              shuffle=True)

    print('Number of samples: ', len(train_data_set))
    print('Number of train_loader: ', len(train_loader))

    model = Net().to(device)
    print(model)
    optimizer = optim.SGD(model.parameters(), lr=args.lr, momentum=args.momentum)

    for epoch in range(1, args.epochs + 1):
        train(args, model, device, train_loader, optimizer, epoch)
        # test(args, model, device, test_loader)

if __name__ == '__main__':
    main()


title: pytorch grad is None after .backward()

body:
I just installed torch-1.0.0 on Python 3.7.2 (macOS), and trying the tutorial, but the following code:

import torch
x = torch.ones(2, 2, requires_grad=True)
y = x + 2
z = y * y * 3
out = z.mean()
out.backward()
print(out.grad)

prints None which is not what's expected.

What's the problem?


title: Install PyTorch from requirements.txt

body:
Torch documentation says use
pip install torch==1.4.0+cpu torchvision==0.5.0+cpu -f https://download.pytorch.org/whl/torch_stable.html

to install the latest version of PyTorch. This works when I do it manually but when I add it to req.txt and do pip install -r req.txt, it fails and says  ERROR: No matching distribution.
Edit: adding the whole line from req.txt and error here.
torch==1.4.0+cpu -f https://download.pytorch.org/whl/torch_stable.html

torchvision==0.5.0+cpu -f https://download.pytorch.org/whl/torch_stable.htmltorch==1.4.0+cpu

ERROR: Could not find a version that satisfies the requirement torch==1.4.0+cpu (from -r requirements.txt (line 1)) (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2, 0.3.1, 0.4.0, 0.4.1, 1.0.0, 1.0.1, 1.0.1.post2, 1.1.0, 1.2.0, 1.3.0, 1.3.1, 1.4.0)
ERROR: No matching distribution found for torch==1.4.0+cpu (from -r requirements.txt (line 1))


title: How does `images, labels = dataiter.next() ` work in PyTorch Tutorial?

body:
From the tutorial cifar10_tutorial, how is images, labels assigned?

trainset = torchvision.datasets.CIFAR10(root='./data', train=True,
                                        download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,
                                          shuffle=True, num_workers=2)

def imshow(img):
    img = img / 2 + 0.5     # unnormalize
    npimg = img.numpy()
    plt.imshow(np.transpose(npimg, (1, 2, 0)))

# get some random training images
dataiter = iter(trainloader)

images, labels = dataiter.next()

How does the last line know how to automatically assign images, label in images, labels = dataiter.next()?

I checked the DataLoader class and the DataLoaderIter class, but think I need a bit more knowledge of iters in general.


title: PyTorch - RuntimeError: bool value of Tensor with more than one value is ambiguous

body:
I trained a GAN on the MNIST dataset, and I'm trying to make a very simple UI that has a button to generate and display new images. When I press a button I make a call to the generator and pass a new latent vector to the forward method and keep getting this error message.

def update_picture():
    print('press')
    _, img = netG.forward(create_noise(1))
    img = img.detach().cpu().numpy()[0][0]
    img = ((img - img.min()) * (1 / img.max() - img.min()) * 255)
    photo = ImageTk.PhotoImage(image=Image.fromarray(img))
    label = Label(image=photo).grid(row=0, column=0)

tk = Tk()

photo = ImageTk.PhotoImage(image=Image.fromarray(img))
label = Label(image=photo).grid(row=0, column=0)

create = Button(text="update", command=update_picture).grid(row=1, column=0)

tk.mainloop()

And when I press the button to generate a new picture I keep getting this error:

Traceback (most recent call last):
  File "C:\Users\daman\AppData\Local\Programs\Python\Python37\lib\tkinter\__init__.py", line 1705, in __call__
return self.func(*args)
  File "C:/Users/daman/PycharmProjects/untitled4/DCGAN_MNIST.py", line 243, in update_picture
_, img = netG.forward(create_noise(1))
  File "C:/Users/daman/PycharmProjects/untitled4/DCGAN_MNIST.py", line 104, in create_noise
return Variable(torch.zeros(b, feature_space, 1, 1).normal_(0, 1))
  File "C:\Users\daman\AppData\Local\Programs\Python\Python37\lib\tkinter\__init__.py", line 315, in __init__
if not master:
RuntimeError: bool value of Tensor with more than one value is ambiguous

The error traces back to my create noise function:

def create_noise(b):
    return Variable(torch.zeros(b, feature_space, 1, 1).normal_(0, 1))

Any ideas why this is happening and what does that error actually mean? I can post more code if needed.


title: PyTorch torch.max over multiple dimensions

body:
Have tensor like :x.shape = [3, 2, 2].

import torch

x = torch.tensor([
    [[-0.3000, -0.2926],[-0.2705, -0.2632]],
    [[-0.1821, -0.1747],[-0.1526, -0.1453]],
    [[-0.0642, -0.0568],[-0.0347, -0.0274]]
])

I need to take .max() over the 2nd and 3rd dimensions. I expect some like this [-0.2632, -0.1453, -0.0274] as output. I tried to use: x.max(dim=(1,2)), but this causes an error.


title: Pytorch Convolutional Autoencoders

body:
How one construct decoder part of convolutional autoencoder? Suppose I have this

(input -> conv2d -> maxpool2d -> maxunpool2d -> convTranspose2d -> output):

# CIFAR images shape = 3 x 32 x 32

class ConvDAE(nn.Module):
    def __init__(self):
        super().__init__()

        # input: batch x 3 x 32 x 32 -> output: batch x 16 x 16 x 16
        self.encoder = nn.Sequential(
            nn.Conv2d(3, 16, 3, stride=1, padding=1), # batch x 16 x 32 x 32
            nn.ReLU(),
            nn.BatchNorm2d(16),
            nn.MaxPool2d(2, stride=2) # batch x 16 x 16 x 16
        )

        # input: batch x 16 x 16 x 16 -> output: batch x 3 x 32 x 32
        self.decoder = nn.Sequential(
            # this line does not work
            # nn.MaxUnpool2d(2, stride=2, padding=0), # batch x 16 x 32 x 32
            nn.ConvTranspose2d(16, 16, 3, stride=2, padding=1, output_padding=1), # batch x 16 x 32 x 32
            nn.ReLU(),
            nn.BatchNorm2d(16),
            nn.ConvTranspose2d(16, 3, 3, stride=1, padding=1, output_padding=0), # batch x 3 x 32 x 32
            nn.ReLU()
        )

    def forward(self, x):
        print(x.size())
        out = self.encoder(x)
        print(out.size())
        out = self.decoder(out)
        print(out.size())
        return out

Pytorch specific question: why can't I use MaxUnpool2d in decoder part. This gives me the following error:

TypeError: forward() missing 1 required positional argument: 'indices'

And the conceptual question: Shouldn't we do in decoder inverse of whatever we did in encoder? I saw some implementations and it seems they only care about the dimensions of input and output of decoder. Here and  here are some examples.


title: How to free gpu memory by deleting tensors?

body:
Suppose I create a tensor and put it on the GPU and don't need it later and want to free the GPU memory allocated to it; How do I do it?

import torch
a=torch.randn(3,4).cuda() # nvidia-smi shows that some mem has been allocated.
# do something
# a does not exist and nvidia-smi shows that mem has been freed.

I have tried:

del a
del a; torch.cuda.empty_cache()

But none of them work.


title: any script to test the installation of Pytorch

body: I have installed the pytorch, and would like to check are there any script to test whether the installation is correct, e.g., whether it can enable CUDA or not, etc?


title: No module named 'torch.utils.data.distributed'

body:
I have installed anaconda and pytorch on my windows 10 and there was no errors when I installed it.

But I don't know why I don't have the modules or packages in pytorch.
Have you experienced the same problem?

This is the program I'm testing: https://github.com/pytorch/examples/blob/master/imagenet/main.py


title: AttributeError: module 'torch' has no attribute '_six'. Bert model in Pytorch

body:
I tried to load pre-trained model by using BertModel class in pytorch.

I have _six.py under torch, but it still shows module 'torch' has no attribute '_six'

import torch
from pytorch_pretrained_bert import BertTokenizer, BertModel, BertForMaskedLM
# Load pre-trained model (weights)
model = BertModel.from_pretrained('bert-base-uncased')
model.eval()

~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in __setattr__(self, name, value)
    551                                     .format(torch.typename(value), name))
    552                 modules[name] = value
--> 553             else:
    554                 buffers = self.__dict__.get('_buffers')
    555                 if buffers is not None and name in buffers:

~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in register_parameter(self, name, param)
    140             raise KeyError("parameter name can't be empty string \"\"")
    141         elif hasattr(self, name) and name not in self._parameters:
--> 142             raise KeyError("attribute '{}' already exists".format(name))
    143
    144         if param is None:

AttributeError: module 'torch' has no attribute '_six'


title: How to model Convolutional recurrent network ( CRNN ) in Keras

body:
I was trying to port CRNN model to Keras.

But, I got stuck while connecting output of Conv2D layer to LSTM layer.

Output from CNN layer will have a shape of ( batch_size, 512, 1, width_dash) where first one depends on batch_size, and last one depends on input width of input ( this model can accept variable width input )

For eg: an input with shape [2, 1, 32, 829] was resulting output with shape of (2, 512, 1, 208)

Now, as per Pytorch model, we have to do squeeze(2)  followed by permute(2, 0, 1)
it will result a tensor with shape [208, 2, 512 ]

I was trying to implement this is Keras, but I was not able to do that because, in Keras we can not alter batch_size dimension in a keras.models.Sequential model

Can someone please guide me how to port above part of this model to Keras?

Current state of ported CNN layer


title: How to get probabilities from Resnet using pytorch?

body:
I am finetuning resnet on my dataset which has multiple labels.

I would like to convert the 'scores' of the classification  layer to probabilities and use those probabilities to calculate the loss at the training.

Could you give an example code for this?
Can I use like this:

       P = net.forward(x)
       p = torch.nn.functional.softmax(P, dim=1)
       loss = torch.nn.functional.cross_entropy(P, y)

I am unclear whether this is the correct way or not as I am passing probabilities as the input to crossentropy loss.


title: How to find functions imported from torch._C in source code

body:
I'm trying to track down the implementation of torch.nn.NLLLoss in the source code. I got as far as a call to torch._C.nll_loss in the function nll_loss in the file torch.nn.functional. But I can't find a place where _C is created.

Anyone have any info on this?


title: torch.utils.data.dataloader outputs TypeError: 'module' object is not callable

body:
So im trying to learn pytorch and i got this code from a tutorial and its just there to import a mnist dataset but it outputs "TypeError: 'module' object is not callable"
In the tutorial     "dataloader" was written as     "Dataloader" but when i run it like that it outputs "AttributeError: module 'torch.utils.data' has no attribute 'Dataloader'"

The data downloaded inside a file mnist but i dont know if it is complete

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optom
from torchvision import datasets, transforms
from torch.autograd import Variable
kwargs={}
train=torch.utils.data.dataloader(datasets.MNIST("mnist",train=True,download=True,transform=transforms.Compose([transforms.ToTensor(),transforms.Normalize((0.1307),(0.3081,) )] ) ),batch_size=64, shuffle=True, **kwargs)


title: Delete an element from torch.Tensor

body:
I'm trying to delete an item from a tensor.

In the example below, How can I remove the third item from the tensor ?

tensor([[-5.1949, -6.2621, -6.2051, -5.8983, -6.3586, -6.2434, -5.8923, -6.1901,
         -6.5713, -6.2396, -6.1227, -6.4196, -3.4311, -6.8903, -6.1248, -6.3813,
         -6.0152, -6.7449, -6.0523, -6.4341, -6.8579, -6.1961, -6.5564, -6.6520,
         -5.9976, -6.3637, -5.7560, -6.7946, -5.4101, -6.1310, -3.3249, -6.4584,
         -6.2202, -6.3663, -6.9293, -6.9262]], grad_fn=<SqueezeBackward1>)


title: Adding custom labels to pytorch dataloader/dataset does not work for custom dataset

body:
I am working on the cactus image competition on Kaggle and I am trying to use the PyTorch dataloader for my CNN. However, I am running into an issue where I cannot set the labels for the training set. The training set images are given in a folder and the labels are in a csv file. This is my code.

 train = torchvision.datasets.ImageFolder(root='../input/train',
 transform=transform)

 train.targets = torch.from_numpy(df['has_cactus'].values)

 train_loader = torch.utils.data.DataLoader(train, batch_size=64, shuffle=True, num_workers=2)

 for i, data in enumerate(train_loader, 0):
     print(data[1])

This code outputs batch tensors of all zeros, which is clearly incorrect as the great majority of the labels(if you were to look at the dataframe) are ones. I believe that this is a problem with assigning the labels to "train.targets". If "train.targets" is printed prior to the assignment of the other labels, it returns a tensor of all zeros which is consistent with the incorrect results that I am getting. How do I fix this issue?


title: How is it that torch is not installed by torchvision?

body:
Somehow when I do the install it installs torchvision but not torch. Command I am running as dictated from the main website:

conda install pytorch torchvision cudatoolkit=10.0 -c pytorch

then I do conda list but look:

$ conda list
# packages in environment at /home/ubuntu/anaconda3/envs/pytorch_p36:
#
# Name                    Version                   Build  Channel
alabaster                 0.7.10           py36h306e16b_0
anaconda-client           1.6.14                   py36_0
anaconda-project          0.8.2            py36h44fb852_0
argparse                  1.4.0                     <pip>
asn1crypto                0.24.0                   py36_0
astroid                   1.6.3                    py36_0
astropy                   3.0.2            py36h3010b51_1
attrs                     18.1.0                   py36_0
autovizwidget             0.12.7                    <pip>
babel                     2.5.3                    py36_0
backcall                  0.1.0                    py36_0
backports                 1.0              py36hfa02d7e_1
backports.shutil_get_terminal_size 1.0.0            py36hfea85ff_2
bcrypt                    3.1.6                     <pip>
beautifulsoup4            4.6.0            py36h49b8c8c_1
bitarray                  0.8.1            py36h14c3975_1
bkcharts                  0.2              py36h735825a_0
blas                      1.0                         mkl
blaze                     0.11.3           py36h4e06776_0
bleach                    2.1.3                    py36_0
blosc                     1.14.3               hdbcaa40_0
bokeh                     1.0.4                    py36_0
boto                      2.48.0           py36h6e4cd66_1
boto3                     1.9.146                   <pip>
boto3                     1.9.134                    py_0
botocore                  1.12.146                  <pip>
botocore                  1.12.134                   py_0
bottleneck                1.2.1            py36haac1ea0_0
bzip2                     1.0.6                h14c3975_5
ca-certificates           2019.1.23                     0
cached-property           1.5.1                     <pip>
cairo                     1.14.12              h8948797_3
certifi                   2019.3.9                 py36_0
cffi                      1.11.5           py36h9745a5d_0
chardet                   3.0.4            py36h0f667ec_1
click                     6.7              py36h5253387_0
cloudpickle               0.5.3                    py36_0
clyent                    1.2.2            py36h7e57e65_1
colorama                  0.3.9            py36h489cec4_0
contextlib2               0.5.5            py36h6c84a62_0
cryptography              2.3.1            py36hc365091_0
cudatoolkit               10.0.130                      0
curl                      7.60.0               h84994c4_0
cycler                    0.10.0           py36h93f1223_0
cymem                     2.0.2            py36hfd86e86_0
cython                    0.28.2           py36h14c3975_0
cytoolz                   0.9.0.1          py36h14c3975_0
dask                      0.17.5                   py36_0
dask-core                 0.17.5                   py36_0
dataclasses               0.6                        py_0    fastai
datashape                 0.5.4            py36h3ad6b5c_0
dbus                      1.13.2               h714fa37_1
decorator                 4.3.0                    py36_0
defusedxml                0.6.0                      py_0
dill                      0.2.9                    py36_0
distributed               1.21.8                   py36_0
docker                    3.7.2                     <pip>
docker-compose            1.24.0                    <pip>
docker-pycreds            0.4.0                     <pip>
dockerpty                 0.4.1                     <pip>
docopt                    0.6.2                     <pip>
docutils                  0.14             py36hb0f60f5_0
entrypoints               0.2.3            py36h1aec115_2
environment-kernels       1.1.1                     <pip>
et_xmlfile                1.0.1            py36hd6bccc3_0
expat                     2.2.5                he0dffb1_0
fastai                    1.0.52                        1    fastai
fastcache                 1.0.2            py36h14c3975_2
fastprogress              0.1.21                     py_0    fastai
filelock                  3.0.4                    py36_0
flask                     1.0.2                    py36_1
flask-cors                3.0.4                    py36_0
fontconfig                2.13.0               h9420a91_0
freetype                  2.9.1                h8a8886c_1
fribidi                   1.0.5                h7b6447c_0
get_terminal_size         1.0.0                haa9412d_0
gevent                    1.3.0            py36h14c3975_0
glib                      2.56.1               h000015b_0
glob2                     0.6              py36he249c77_0
gmp                       6.1.2                h6c8ec71_1
gmpy2                     2.0.8            py36hc8893dd_2
graphite2                 1.3.11               h16798f4_2
graphviz                  2.40.1               h21bd128_2
greenlet                  0.4.13           py36h14c3975_0
gst-plugins-base          1.14.0               hbbd80ab_1
gstreamer                 1.14.0               hb453b48_1
h5py                      2.8.0            py36h989c5e5_3
harfbuzz                  1.8.4                hec2c2bc_0
hdf5                      1.10.2               hba1933b_1
hdijupyterutils           0.12.7                    <pip>
heapdict                  1.0.0                    py36_2
html5lib                  1.0.1            py36h2f9c1c0_0
icu                       58.2                 h9c2bf20_1
idna                      2.6              py36h82fb2a8_1
imageio                   2.3.0                    py36_0
imagesize                 1.0.0                    py36_0
intel-openmp              2018.0.0                      8
ipykernel                 4.8.2                    py36_0
ipyparallel               6.2.2                     <pip>
ipython                   6.4.0                    py36_0
ipython_genutils          0.2.0            py36hb52b0d5_0
ipywidgets                7.2.1                    py36_0
ipywidgets                7.4.0                     <pip>
isort                     4.3.4                    py36_0
itsdangerous              0.24             py36h93cc618_1
jbig                      2.1                  hdba287a_0
jdcal                     1.4                      py36_0
jedi                      0.12.0                   py36_1
jinja2                    2.10             py36ha16c418_0
jmespath                  0.9.4                      py_0
jpeg                      9b                   h024ee3a_2
jsonschema                2.6.0            py36h006f8b5_0
jupyter                   1.0.0                    py36_4
jupyter_client            5.2.3                    py36_0
jupyter_console           5.2.0            py36he59e554_1
jupyter_core              4.4.0            py36h7c827e3_0
jupyterlab                0.32.1                   py36_0
jupyterlab_launcher       0.10.5                   py36_0
kiwisolver                1.0.1            py36h764f252_0
krb5                      1.14.2               hcdc1b81_6
lazy-object-proxy         1.3.1            py36h10fcdad_0
libcurl                   7.60.0               h1ad7b7a_0
libedit                   3.1.20170329         h6b74fdf_2
libffi                    3.2.1                hd88cf55_4
libgcc-ng                 8.2.0                hdf63c60_1
libgfortran               3.0.0                         1    conda-forge
libgfortran-ng            7.2.0                hdf63c60_3
libpng                    1.6.37               hbc83047_0
libprotobuf               3.5.2                hd28b015_1    conda-forge
libsodium                 1.0.16               h1bed415_0
libssh2                   1.8.0                h9cfc8f7_4
libstdcxx-ng              8.2.0                hdf63c60_1
libtiff                   4.0.9                he85c1e1_1
libtool                   2.4.6                h544aabb_3
libuuid                   1.0.3                h1bed415_2
libxcb                    1.13                 h1bed415_1
libxml2                   2.9.8                h26e45fe_1
libxslt                   1.1.32               h1312cb7_0
llvmlite                  0.23.1           py36hdbcaa40_0
locket                    0.2.0            py36h787c0ad_1
lxml                      4.2.1            py36h23eabaa_0
lzo                       2.10                 h49e0be7_2
markupsafe                1.0              py36hd9260cd_1
matplotlib                2.2.2                     <pip>
matplotlib                3.0.3            py36h5429711_0
mccabe                    0.6.1            py36h5ad9710_1
mistune                   0.8.3            py36h14c3975_1
mkl                       2018.0.3                      1
mkl-service               1.1.2            py36h17a0993_4
mkl_fft                   1.0.6            py36h7dd41cf_0
mkl_random                1.0.1            py36h629b387_0
mock                      3.0.5                     <pip>
more-itertools            4.1.0                    py36_0
mpc                       1.0.3                hec55b23_5
mpfr                      3.1.5                h11a74b3_2
mpi                       1.0                     openmpi    conda-forge
mpmath                    1.0.0            py36hfeacd6b_2
msgpack                   0.6.0                     <pip>
msgpack-numpy             0.4.3.2                  py36_0
msgpack-python            0.5.6            py36h6bb024c_0
multipledispatch          0.5.0                    py36_0
murmurhash                1.0.2            py36he6710b0_0
nb_conda                  2.2.1                    py36_2    conda-forge
nb_conda_kernels          2.2.1                    py36_0    conda-forge
nbconvert                 5.4.1                    py36_3
nbformat                  4.4.0            py36h31c9010_0
ncurses                   6.1                  hf484d3e_0
networkx                  2.1                      py36_0
ninja                     1.8.2            py36h6bb024c_1
nltk                      3.3.0                    py36_0
nose                      1.3.7            py36hcdf7029_2
notebook                  5.5.0                    py36_0
numba                     0.38.0           py36h637b7d7_0
numexpr                   2.6.5            py36h7bf3b9c_0
numpy                     1.15.4           py36h1d66e8a_0
numpy                     1.15.4                    <pip>
numpy-base                1.15.4           py36h81de0dd_0
numpydoc                  0.8.0                    py36_0
nvidia-ml-py3             7.352.0                    py_0    fastai
odo                       0.5.1            py36h90ed295_0
olefile                   0.45.1                   py36_0
onnx                      1.4.1                     <pip>
openmpi                   3.1.0                h26a2512_3    conda-forge
openpyxl                  2.5.3                    py36_0
openssl                   1.0.2r               h7b6447c_0
packaging                 17.1                     py36_0
pandas                    0.24.2                    <pip>
pandas                    0.23.0           py36h637b7d7_0
pandoc                    1.19.2.1             hea2e7c5_1
pandocfilters             1.4.2            py36ha6701b7_1
pango                     1.42.3               h8589676_0
paramiko                  2.4.2                     <pip>
parso                     0.2.0                    py36_0
partd                     0.3.8            py36h36fd896_0
patchelf                  0.9                  hf79760b_2
path.py                   11.0.1                   py36_0
pathlib2                  2.3.2                    py36_0
patsy                     0.5.0                    py36_0
pcre                      8.42                 h439df22_0
pep8                      1.7.1                    py36_0
pexpect                   4.5.0                    py36_0
pickleshare               0.7.4            py36h63277f8_0
pillow                    5.2.0            py36heded4f4_0
pip                       10.0.1                   py36_0
pixman                    0.34.0               hceecf20_3
pkginfo                   1.4.2                    py36_1
plac                      0.9.6                    py36_0
plotly                    2.7.0                     <pip>
pluggy                    0.6.0            py36hb689045_0
ply                       3.11                     py36_0
preshed                   2.0.1            py36he6710b0_0
prompt_toolkit            1.0.15           py36h17d85b1_0
protobuf                  3.5.2            py36hd28b015_0    conda-forge
protobuf3-to-dict         0.1.5                     <pip>
psutil                    5.4.5            py36h14c3975_0
psycopg2                  2.7.5                     <pip>
ptyprocess                0.5.2            py36h69acd42_0
py                        1.5.3                    py36_0
py4j                      0.10.7                    <pip>
pyasn1                    0.4.5                     <pip>
pycodestyle               2.4.0                    py36_0
pycosat                   0.6.3            py36h0a5515d_0
pycparser                 2.18             py36hf9f622e_1
pycrypto                  2.6.1            py36h14c3975_8
pycurl                    7.43.0.1         py36hb7f436b_0
pyflakes                  1.6.0            py36h7bd6a15_0
pygal                     2.4.0                     <pip>
pygments                  2.2.0            py36h0d3125c_0
pykerberos                1.2.1            py36h14c3975_0
pylint                    1.8.4                    py36_0
PyNaCl                    1.3.0                     <pip>
pyodbc                    4.0.23           py36hf484d3e_0
pyopenssl                 18.0.0                   py36_0
pyparsing                 2.2.0            py36hee85983_1
pyqt                      5.9.2            py36h751905a_0
pysocks                   1.6.8                    py36_0
pyspark                   2.3.2                     <pip>
pytables                  3.4.3            py36h02b9ad4_2
pytest                    3.5.1                    py36_0
pytest-arraydiff          0.2                      py36_0
pytest-astropy            0.3.0                    py36_0
pytest-doctestplus        0.1.3                    py36_0
pytest-openfiles          0.3.0                    py36_0
pytest-remotedata         0.2.1                    py36_0
python                    3.6.5                hc3d631a_2
python-dateutil           2.7.3                    py36_0
pytorch                   1.1.0           py3.6_cuda10.0.130_cudnn7.5.1_0    pytorch
pytz                      2018.4                   py36_0
pywavelets                0.5.2            py36he602eb0_0
pyyaml                    3.12             py36hafb9ca4_1
pyzmq                     17.0.0           py36h14c3975_0
qt                        5.9.6                h52aff34_0
qtawesome                 0.4.4            py36h609ed8c_0
qtconsole                 4.3.1            py36h8f73b5b_0
qtpy                      1.4.1                    py36_0
readline                  7.0                  ha6073c6_4
regex                     2018.01.10      py36h14c3975_1000    fastai
requests                  2.20.0                py36_1000    conda-forge
requests-kerberos         0.12.0                    <pip>
rope                      0.10.7           py36h147e2ec_0
ruamel_yaml               0.15.35          py36h14c3975_1
s3fs                      0.1.5                    py36_0
s3transfer                0.2.0                     <pip>
s3transfer                0.2.0                    py36_0
sagemaker                 1.20.1                    <pip>
sagemaker-pyspark         1.2.4                     <pip>
scikit-image              0.13.1           py36h14c3975_1
scikit-learn              0.19.1           py36h7aa7ec6_0
scikit-learn              0.20.3                    <pip>
scipy                     1.1.0            py36hfc37229_0
seaborn                   0.8.1            py36hfad7ec4_0
send2trash                1.5.0                    py36_0
setuptools                39.1.0                   py36_0
simplegeneric             0.8.1                    py36_2
singledispatch            3.4.0.3          py36h7a266c3_0
sip                       4.19.8           py36hf484d3e_0
six                       1.11.0           py36h372c433_1
snappy                    1.1.7                hbae5bb6_3
snowballstemmer           1.2.1            py36h6febd40_0
sortedcollections         0.6.1                    py36_0
sortedcontainers          1.5.10                   py36_0
spacy                     2.0.18          py36hf484d3e_1000    fastai
sparkmagic                0.12.5                    <pip>
sphinx                    1.7.4                    py36_0
sphinxcontrib             1.0              py36h6d0f590_1
sphinxcontrib-websupport  1.0.1            py36hb5cb234_1
spyder                    3.2.8                    py36_0
SQLAlchemy                1.2.11                    <pip>
sqlalchemy                1.2.7            py36h6b74fdf_0
sqlite                    3.23.1               he433501_0
statsmodels               0.9.0            py36h3010b51_0
sympy                     1.1.1            py36hc6d1c1c_0
tblib                     1.3.2            py36h34cf8b6_0
terminado                 0.8.1                    py36_1
testpath                  0.3.1            py36h8cadb63_0
texttable                 0.9.1                     <pip>
thinc                     6.12.1          py36h637b7d7_1000    fastai
tk                        8.6.8                hbc83047_0
toolz                     0.9.0                    py36_0
torchvision               0.2.2                      py_3    pytorch
tornado                   5.0.2                    py36_0
tqdm                      4.31.1                   py36_1
traitlets                 4.3.2            py36h674d592_0
typing                    3.6.4                    py36_0
typing-extensions         3.7.2                     <pip>
ujson                     1.35             py36h14c3975_0
unicodecsv                0.14.1           py36ha668878_0
unixodbc                  2.3.6                h1bed415_0
urllib3                   1.23                     py36_0
wcwidth                   0.1.7            py36hdf4376a_0
webencodings              0.5.1            py36h800622e_1
websocket-client          0.56.0                    <pip>
werkzeug                  0.14.1                   py36_0
wheel                     0.31.1                   py36_0
widgetsnbextension        3.2.1                    py36_0
widgetsnbextension        3.4.2                     <pip>
wrapt                     1.10.11          py36h28b7045_0
xlrd                      1.1.0            py36h1db9f0c_1
xlsxwriter                1.0.4                    py36_0
xlwt                      1.3.0            py36h7b00a1f_0
xz                        5.2.4                h14c3975_4
yaml                      0.1.7                had09818_2
zeromq                    4.2.5                h439df22_0
zict                      0.1.3            py36h3a3bf81_0
zlib                      1.2.11               ha838bed_2

I was told I do have pytorch installed but my script keeps giving me this error:

$ cat nohup.out
Traceback (most recent call last):
  File "high_performing_data_point_models_cifar10.py", line 5, in <module>
    import torch
ModuleNotFoundError: No module named 'torch'

does that mean that I need to install it as pytroch and not torch? Is this not weird?

Note I am running this on an AWS instance p3.2xlarge. This keeps happening when I log out and then go back in that my torch package gets missing...?!?! :/

original post: https://discuss.pytorch.org/t/torchvision-installed-but-not-torch/51758

The issue persists even if I open just a python interactive and try to import it:

(pytorch_p36) ubuntu@ip-123-12-21-123:~$ python
Python 3.6.5 |Anaconda, Inc.| (default, Apr 29 2018, 16:14:56)
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'torch'
>>>

It also happens running the script directly:

(pytorch_p36) ubuntu@ip-123-12-21-123:~/project/folder$ python high_performing_data_point_models_cifar10.py
Traceback (most recent call last):
  File "high_performing_data_point_models_cifar10.py", line 5, in <module>
    import torch
ModuleNotFoundError: No module named 'torch'

I can't import torchvision either!

$ python
Python 3.6.5 |Anaconda, Inc.| (default, Apr 29 2018, 16:14:56)
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torchvision
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'torchvision'
>>>


I'm getting this error when training the MNIST data, the csvfiles is from Kaggle. Can someone show me where I went wrong? Here is my code. The version of PyTorch is 0.4.0.

import numpy as np
import pandas as pd
import torch
import torch.nn as nn
from torch.autograd import Variable
import torch.utils.data as data
import torchvision
import matplotlib.pyplot as plt

torch.manual_seed(1)

# Training Parameters
EPOCH = 20
BATCH_size = 15
LR = 0.001
img_row, img_col = 28, 28

# Networks structure
class CNN(nn.Module):
    def __init__(self):
        super(CNN, self).__init__()
        self.conv1 = nn.Sequential(
            nn.Conv2d(
                in_channels=1, out_channels=32,
                kernel_size=5, stride=1, padding=2
            ),
            nn.ReLU(),
            nn.Conv2d(32, 32, 5, 1, 2),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2),
            nn.Dropout(0.25)
        )
        self.conv2 = nn.Sequential(
            nn.Conv2d(32, 64, 3, 1, 1),
            nn.ReLU(),
            nn.Conv2d(64, 64, 3, 1, 1),
            nn.ReLU(),
            nn.MaxPool2d(2),
            nn.Dropout(0.25)
        )
        self.out = nn.Sequential(
            nn.Linear(64*7*7, 512),
            nn.ReLU(),
            nn.Dropout(0.5),
            nn.Linear(512, 10)
        )

    def forward(self, x):
        x = self.conv1(x)
        x = self.conv2(x)
        x = x.view(x.size(0), -1)
        output = self.out(x)
        return output

# Torch Dataset
class Torch_Dataset(data.Dataset):
    def __init__(self, root_dir, csvfile, img_rows, img_cols, train=True, transform=None):
        self.root_dir = root_dir
        self.transform = transform
        self.train = train
        if self.train:
            y_data0 = pd.read_csv(csvfile, header=0, usecols=['label'])
            y_data1 = np.array(y_data0)
            self.y_data = torch.from_numpy(y_data1)
            x_data0 = pd.read_csv(csvfile, header=0, usecols=[i for i in range(1, 785)])
            x_data1 = np.array(x_data0)
            x_data1 = x_data1.reshape(x_data1.shape[0], 1, img_rows, img_cols)
            x_data1 = x_data1.astype('float32')
            x_data1 /= 255
            self.x_data = torch.from_numpy(x_data1)
        else:
            x_data0 = pd.read_csv(csvfile, header=0)
            x_data1 = np.array(x_data0)
            x_data1 = x_data1.reshape(x_data1.shape[0], 1, img_rows, img_cols)
            x_data1 = x_data1.astype('float32')
            x_data1 /= 255
            self.x_data = torch.from_numpy(x_data1)

    def __len__(self):
        return len(self.x_data)

    def __getitem__(self, idx):
        if self.train:
            img, target = self.x_data[idx], self.y_data[idx]
        else:
            img = self.x_data[idx]
            target = None
        # sample = {'img': img, 'target': target}
        return img, target

train = Torch_Dataset(
    root_dir='./',                # root
    csvfile='train.csv',          # filename
    img_rows=img_row,             # image rows
    img_cols=img_col,             # image cols
    train=True                    # train or test
)
# DataLoader
loader = data.DataLoader(
    dataset=train,                # torch dataset format
    batch_size=BATCH_size,        # mini batch size
    shuffle=True,                 # shuffle the data
)
# train the data
cnn = CNN()
optimizer = torch.optim.Adam(cnn.parameters(), lr=LR)
loss_f = nn.CrossEntropyLoss()
for epoch in range(EPOCH):
    for step, (x, y) in enumerate(loader):
        b_x = Variable(x)
        b_y = Variable(y)
        b_y = b_y.squeeze

        output = cnn(b_x)
        loss = loss_f(output, b_y)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

  Traceback (most recent call last):

  File "C:/Users/Bryan Zoe/PycharmProjects/MNIST_TEST/PyTorch/test1.py", line 118, in
      loss = loss_f(output, b_y)

  File "C:\Users\Bryan Zoe\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 491, in __ call __
      result = self.forward(*input, **kwargs)

  File "C:\Users\Bryan Zoe\Anaconda3\lib\site-packages\torch\nn\modules\loss.py", line 757, in forward
      _assert_no_grad(target)

  File "C:\Users\Bryan Zoe\Anaconda3\lib\site-packages\torch\nn\modules\loss.py", line 11, in _assert_no_grad
      assert not tensor.requires_grad, \

  AttributeError: 'builtin_function_or_method' object has no attribute 'requires_grad'


title: AttributeError: 'Compose' object has no attribute 'Compose' (in Pytorch 0.2.1)

body:
This is the block of code where am getting this error:

train_transforms = transforms.Compose([transforms.RandomRotation(30),
                                       transforms.RandomResizedCrop(224),
                                       transforms.RandomHorizontalFlip(),
                                       transforms.ToTensor(),
                                       transforms.Normalize([0.5, 0.5, 0.5],
                                                            [0.5, 0.5, 0.5])])

I've tried updating my torchvision but had no luck!


title: Using GPU inside docker container - CUDA Version: N/A and torch.cuda.is_available returns False

body:
I'm trying to use GPU from inside my docker container. I'm using docker with version 19.03 on Ubuntu 18.04.
Outside the docker container if I run nvidia-smi I get the below output.
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.51.05    Driver Version: 450.51.05    CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla T4            On   | 00000000:00:1E.0 Off |                    0 |
| N/A   30C    P8     9W /  70W |      0MiB / 15109MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

If I run the samething inside the container created from nvidia/cuda docker image, I get the same output as above and everything is running smoothly. torch.cuda.is_available() returns True.
But If I run the same nvidia-smi command inside any other docker container, it gives the following output where you can see that the CUDA Version is coming as N/A. Inside the containers torch.cuda.is_available() also returns False.
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.51.05    Driver Version: 450.51.05    CUDA Version: N/A      |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla T4            On   | 00000000:00:1E.0 Off |                    0 |
| N/A   30C    P8     9W /  70W |      0MiB / 15109MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

I have installed nvidia-container-toolkit using the following commands.
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
curl -s -L https://nvidia.github.io/nvidia-docker/ubuntu18.04/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
sudo apt-get update
sudo apt-get install nvidia-container-toolkit
sudo systemctl restart docker

I started my containers using the following commands
sudo docker run --rm --gpus all nvidia/cuda nvidia-smi
sudo docker run -it --rm --gpus all ubuntu nvidia-smi


title: set `torch.backends.cudnn.benchmark = True` or not?

body: I am using pytorch and I wonder if I should use torch.backends.cudnn.benchmark = True. I find on google that I should use it when computation graph does not change. What is computation graph in pytorch?


title: GPU performing slower than CPU for Pytorch on Google Colaboratory

body:
The GPU trains this network in about 16 seconds. The CPU in about 13 seconds. (I am uncommenting/commenting appropriate lines to do the test). Can anyone see what's wrong with my code or pytorch installation? (I have already checked that the GPU is available, and that there is sufficient memory available on the GPU.

from os import path
from wheel.pep425tags import get_abbr_impl, get_impl_ver, get_abi_tag
platform = '{}{}-{}'.format(get_abbr_impl(), get_impl_ver(), get_abi_tag())

accelerator = 'cu80' if path.exists('/opt/bin/nvidia-smi') else 'cpu'
print(accelerator)
!pip install -q http://download.pytorch.org/whl/{accelerator}/torch-0.4.0-{platform}-linux_x86_64.whl torchvision
print("done")

#########################

import torch
from datetime import datetime

startTime = datetime.now()

dtype = torch.float
device = torch.device("cpu") # Comment this to run on GPU
# device = torch.device("cuda:0") # Uncomment this to run on GPU

# N is batch size; D_in is input dimension;
# H is hidden dimension; D_out is output dimension.
N, D_in, H, D_out = 64, 1024, 128, 8

# Create random Tensors to hold input and outputs.
x = torch.randn(N, D_in, device=device, dtype=dtype)
t = torch.randn(N, D_out, device=device, dtype=dtype)

# Create random Tensors for weights.
w1 = torch.randn(D_in, H, device=device, dtype=dtype, requires_grad=True)
w2 = torch.randn(H, D_out, device=device, dtype=dtype, requires_grad=True)
w3 = torch.randn(D_out, D_out, device=device, dtype=dtype, requires_grad=True)

learning_rate = 1e-9
for i in range(10000):
    y_pred = x.mm(w1).clamp(min=0).mm(w2).clamp(min=0).mm(w3)

    loss = (y_pred - t).pow(2).sum()

    if i % 1000 == 0:
        print(i, loss.item())

    loss.backward()

    # Manually update weights using gradient descent
    with torch.no_grad():
        w1 -= learning_rate * w1.grad
        w2 -= learning_rate * w2.grad

        # Manually zero the gradients after updating weights
        w1.grad.zero_()
        w2.grad.zero_()

print(datetime.now() - startTime)


title: PyTorch: How to convert pretrained FC layers in a CNN to Conv layers

body: I want to convert a pre-trained CNN (like VGG-16) to a fully convolutional network in Pytorch. How can I do so?


title: Why is the memory in GPU still in use after clearing the object?

body:
Starting with zero usage:

>>> import gc
>>> import GPUtil
>>> import torch
>>> GPUtil.showUtilization()
| ID | GPU | MEM |
------------------
|  0 |  0% |  0% |
|  1 |  0% |  0% |
|  2 |  0% |  0% |
|  3 |  0% |  0% |

Then I create a big enough tensor and hog the memory:

>>> x = torch.rand(10000,300,200).cuda()
>>> GPUtil.showUtilization()
| ID | GPU | MEM |
------------------
|  0 |  0% | 26% |
|  1 |  0% |  0% |
|  2 |  0% |  0% |
|  3 |  0% |  0% |

Then I tried several ways to see if the tensor disappears.

Attempt 1: Detach, send to CPU and overwrite the variable

No, doesn't work.

>>> x = x.detach().cpu()
>>> GPUtil.showUtilization()
| ID | GPU | MEM |
------------------
|  0 |  0% | 26% |
|  1 |  0% |  0% |
|  2 |  0% |  0% |
|  3 |  0% |  0% |

Attempt 2: Delete the variable

No, this doesn't work either

>>> del x
>>> GPUtil.showUtilization()
| ID | GPU | MEM |
------------------
|  0 |  0% | 26% |
|  1 |  0% |  0% |
|  2 |  0% |  0% |
|  3 |  0% |  0% |

Attempt 3: Use the torch.cuda.empty_cache() function

Seems to work, but it seems that there are some lingering overheads...

>>> torch.cuda.empty_cache()
>>> GPUtil.showUtilization()
| ID | GPU | MEM |
------------------
|  0 |  0% |  5% |
|  1 |  0% |  0% |
|  2 |  0% |  0% |
|  3 |  0% |  0% |

Attempt 4: Maybe clear the garbage collector.

No, 5% is still being hogged

>>> gc.collect()
0
>>> GPUtil.showUtilization()
| ID | GPU | MEM |
------------------
|  0 |  0% |  5% |
|  1 |  0% |  0% |
|  2 |  0% |  0% |
|  3 |  0% |  0% |

Attempt 5: Try deleting torch altogether (as if that would work when del x didn't work -_- )

No, it doesn't...*

>>> del torch
>>> GPUtil.showUtilization()
| ID | GPU | MEM |
------------------
|  0 |  0% |  5% |
|  1 |  0% |  0% |
|  2 |  0% |  0% |
|  3 |  0% |  0% |

And then I tried to check gc.get_objects() and it looks like there's still quite a lot of odd THCTensor stuff inside...

Any idea why is the memory still in use after clearing the cache?


title: Problems about torch.nn.DataParallel

body:
I am new in deep learning area. Now I am reproducing a paper’s codes. since they use several GPUs, there is a command torch.nn.DataParallel(model, device_ids= args.gpus).cuda() in codes. But I only have one GPU, what
should I change this code to match up my GPU?

Thank you!


title: Module Not Found Error when importing Pytorch_Transformers

body:
After downloading pytorch_transformers through Anaconda and executing the import command through the Jupyter Notebook, I am facing several errors related to missing modules.

I tried searching sacremoses to import the package via Anaconda, but it is only available for Linux machines. Has anyone else faced similar issues? Thanks in advance!

from pytorch_transformers import BertTokenizer, BertModel, BertForMaskedLM

This is the error:

<ipython-input-5-218d0858d00f> in <module>
----> 1 from pytorch_transformers import BertTokenizer, BertModel, BertForMaskedLM

~\Anaconda3\lib\site-packages\pytorch_transformers\__init__.py in <module>
      1 __version__ = "1.2.0"
----> 2 from .tokenization_auto import AutoTokenizer
      3 from .tokenization_bert import BertTokenizer, BasicTokenizer, WordpieceTokenizer
      4 from .tokenization_openai import OpenAIGPTTokenizer
      5 from .tokenization_transfo_xl import (TransfoXLTokenizer, TransfoXLCorpus)

~\Anaconda3\lib\site-packages\pytorch_transformers\tokenization_auto.py in <module>
     24 from .tokenization_transfo_xl import TransfoXLTokenizer
     25 from .tokenization_xlnet import XLNetTokenizer
---> 26 from .tokenization_xlm import XLMTokenizer
     27 from .tokenization_roberta import RobertaTokenizer
     28 from .tokenization_distilbert import DistilBertTokenizer

~\Anaconda3\lib\site-packages\pytorch_transformers\tokenization_xlm.py in <module>
     25 from io import open
     26
---> 27 import sacremoses as sm
     28
     29 from .tokenization_utils import PreTrainedTokenizer

ModuleNotFoundError: No module named 'sacremoses'```


title: axes don't match array in pytorch

body:
I am new to pytorch and i am stuck in this for more than a week now.
i am trying to use AlexNet to make a 'gta san Andreas' self driving car and i am having alot of problems with preparing the data.
for now i am getting this error.

Traceback (most recent call last):
  File "training_script.py", line 19, in <module>
    transformed_data = transform(all_data)
  File "C:\Users\Mukhtar\Anaconda3\lib\site-packages\torchvision\transforms\transforms.py", line 49, in __call__
    img = t(img)
  File "C:\Users\Mukhtar\Anaconda3\lib\site-packages\torchvision\transforms\transforms.py", line 76, in __call__
    return F.to_tensor(pic)
  File "C:\Users\Mukhtar\Anaconda3\lib\site-packages\torchvision\transforms\functional.py", line 48, in to_tensor
    img = torch.from_numpy(pic.transpose((2, 0, 1)))
ValueError: axes don't match array

this is the training script

from AlexNetPytorch import*
import torchvision
import torchvision.transforms as transforms
import torch.optim as optim
import torch.utils.data
import numpy as np
import torch

AlexNet = AlexNet()

criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(AlexNet.parameters(), lr=0.001, momentum=0.9)

all_data = np.load('training_data.npy')
transform = transforms.Compose([
    # you can add other transformations in this list
    transforms.ToTensor()
])
transformed_data = transform(all_data)
# # data_set = torchvision.datasets.ImageFolder('training_data.npy' ,transform = transforms.ToTensor() )
data_loader = torch.utils.data.DataLoader(training_data, batch_size=4,shuffle=True, num_workers=2)
# training_data = all_data[:-500]lk
# testing_data = all_data[-500:]

if __name__ == '__main__':
 for epoch in range(8):
  runing_loss = 0.0
  for i,data in enumerate(data_loader , 0):
     inputs= data[0]
     inputs = torch.FloatTensor(inputs)
     labels= data[1]
     labels = torch.FloatTensor(labels)
     optimizer.zero_grad()
     outputs = AlexNet(inputs)
     loss = criterion(outputs , labels)
     loss.backward()
     optimizer.step()

     runing_loss +=loss.item()
     if i % 2000 == 1999:    # print every 2000 mini-batches
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 2000))
            running_loss = 0.0
 print('finished')

and this is how i am preparing the data

import cv2
from PIL import ImageGrab
import numpy as np
import time
from directKeys import PressKey,W,A,S,D
from getKeys import key_check
import os

def keys_to_output(keys):
    output = [0,0,0]

    if 'A' in keys:
        output[0] = 1
    elif 'D' in keys:
        output[2] = 1
    else:
        output[1] = 1
    return output

file_name = "training_data.npy"
if os.path.isfile(file_name):
   print("file exists , loading previous data!")
   training_data = list(np.load(file_name))
else:
   print("file does not exist , starting fresh")
   training_data = []

last_time = time.time()

while True:

   kernel = np.ones((15 , 15) , np.float32)/225
   get_screen = ImageGrab.grab(bbox=(10,10,1280,720))
   screen_shot = np.array(get_screen)
   hsv = cv2.cvtColor(screen_shot , cv2.COLOR_BGR2HSV)
   lower_color = np.array([90 , 0 , 70])
   upper_color = np.array([100 , 100 ,  100])

   output = cv2.inRange(hsv , lower_color , upper_color)

   kernel = np.ones((1,20), np.uint8)  # note this is a horizontal kernel
   dilation = cv2.dilate(output, kernel, iterations=1)
   output = cv2.erode(dilation, kernel, iterations=1)
   # output = cv2.Canny(output , threshold1 = 50 , threshold2 = 300)
   # output = cv2.GaussianBlur(output , (15,15) , 0)

   resized = cv2.resize(output , (640 , 480))

   print('loop took {} seconds'.format(time.time()-last_time))
   last_time = time.time()

   cv2.imshow('manipulated' , resized)
   screen_output = cv2.resize(output , (32 ,32))
   keys = key_check()
   Keys_output = keys_to_output(keys)
   training_data.append([screen_output,Keys_output])

   if cv2.waitKey(1) & 0xFF == ord('q'):
        cv2.destroyAllWindows()
        break

   if len(training_data) % 500 == 0:
       print(len(training_data))
       np.save(file_name,training_data)

I tried a lot of solutions but none of them work, but I feel like I am missing something.
i am way over my head so please help


title: what is the default weight initializer for conv in pytorch?

body: The question How to initialize weights in PyTorch? shows how to initialize the weights in Pytorch.  However, what is the default weight initializer for Convand Dense in Pytorch? What distribution does Pytorch use?


title: Convert image to proper dimension PyTorch

body:
I have an input image, as numpy array of shape [H, W, C] where H - height, W - width and C - channels.
I want to convert it into [B, C, H, W] where B - batch size, which should be equal to 1 every time, and changing the place for C.

_image = np.array(_image)
h, w, c = _image.shape
image = torch.from_numpy(_image).unsqueeze_(0).view(1, c, h, w)

So, will this preserve the image properly i.e without displacing the original image pixel values?