title: CNN Pytorch Error : Input type (torch.cuda.ByteTensor) and weight type (torch.cuda.FloatTensor) should be the same

body:
I'm receiving the error,

  Input type (torch.cuda.ByteTensor) and weight type (torch.cuda.FloatTensor) should be the same

Following is my code,

device    = torch.device('cuda:0')

trainData = torchvision.datasets.FashionMNIST('/content/', train=True, transform=None, target_transform=None, download=True)
testData  = torchvision.datasets.FashionMNIST('/content/', train=False, transform=None, target_transform=None, download=True)

class Net(nn.Module):
  def __init__(self):
    super().__init__()

    '''
    Network Structure:

    input >
    (1)Conv2D > (2)MaxPool2D >
    (3)Conv2D > (4)MaxPool2D >
    (5)Conv2D > (6)MaxPool2D >
    (7)Linear > (8)LinearOut

    '''

    # Creating the convulutional Layers
    self.conv1 = nn.Conv2d(in_channels=CHANNELS, out_channels=32, kernel_size=3)
    self.conv2 = nn.Conv2d(in_channels=32, out_channels=64, kernel_size=3)
    self.conv3 = nn.Conv2d(in_channels=64, out_channels=128, kernel_size=3)

    self.flatten = None
    # Creating a Random dummy sample to get the Flattened Dimensions
    x = torch.randn(CHANNELS, DIM, DIM).view(-1, CHANNELS, DIM, DIM)
    x = self.convs(x)

    # Creating the Linear Layers
    self.fc1   = nn.Linear(self.flatten, 512)
    self.fc2   = nn.Linear(512, CLASSES)

  def convs(self, x):

    # Creating the MaxPooling Layers
    x = F.max_pool2d(F.relu(self.conv1(x)), kernel_size=(2, 2))
    x = F.max_pool2d(F.relu(self.conv2(x)), kernel_size=(2, 2))
    x = F.max_pool2d(F.relu(self.conv3(x)), kernel_size=(2, 2))

    if not self.flatten:
      self.flatten = x[0].shape[0] * x[0].shape[1] * x[0].shape[2]
    return x

  # FORWARD PASS
  def forward(self, x):
    x = self.convs(x)
    x = x.view(-1, self.flatten)
    sm = F.relu(self.fc1(x))
    x = F.softmax(self.fc2(sm), dim=1)
    return x, sm

  x_train, y_train = training_set
  x_train, y_train = x_train.to(device), y_train.to(device)
  optimizer = optim.Adam(net.parameters(), lr=LEARNING_RATE)
  loss_func = nn.MSELoss()
  loss_log  = []

  for epoch in range(EPOCHS):
    for i in tqdm(range(0, len(x_train), BATCH_SIZE)):
        x_batch = x_train[i:i+BATCH_SIZE].view(-1, CHANNELS, DIM, DIM).to(device)
        y_batch = y_train[i:i+BATCH_SIZE].to(device)

        net.zero_grad()
        output, sm = net(x_batch)
        loss = loss_func(output, y_batch.float())
        loss.backward()
        optimizer.step()
    loss_log.append(loss)
    # print(f"Epoch : {epoch} || Loss : {loss}")

  return loss_log

train_set = (trainData.train_data, trainData.train_labels)
test_set  = (testData.test_data, testData.test_labels)

EPOCHS        = 5
LEARNING_RATE = 0.001
BATCH_SIZE    = 32

net = Net().to(device)

loss_log = train(net, train_set, EPOCHS, LEARNING_RATE, BATCH_SIZE)

And this is the Error that I'm getting,

RuntimeError                              Traceback (most recent call last)
<ipython-input-8-0db1a1b4e37d> in <module>()
      5 net = Net().to(device)
      6
----> 7 loss_log = train(net, train_set, EPOCHS, LEARNING_RATE, BATCH_SIZE)

6 frames
<ipython-input-6-7de4a78e3736> in train(net, training_set, EPOCHS, LEARNING_RATE, BATCH_SIZE)
     13
     14         net.zero_grad()
---> 15         output, sm = net(x_batch)
     16         loss = loss_func(output, y_batch.float())
     17         loss.backward()

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    539             result = self._slow_forward(*input, **kwargs)
    540         else:
--> 541             result = self.forward(*input, **kwargs)
    542         for hook in self._forward_hooks.values():
    543             hook_result = hook(self, input, result)

<ipython-input-5-4fddc427892a> in forward(self, x)
     41   # FORWARD PASS
     42   def forward(self, x):
---> 43     x = self.convs(x)
     44     x = x.view(-1, self.flatten)
     45     sm = F.relu(self.fc1(x))

<ipython-input-5-4fddc427892a> in convs(self, x)
     31
     32     # Creating the MaxPooling Layers
---> 33     x = F.max_pool2d(F.relu(self.conv1(x)), kernel_size=(2, 2))
     34     x = F.max_pool2d(F.relu(self.conv2(x)), kernel_size=(2, 2))
     35     x = F.max_pool2d(F.relu(self.conv3(x)), kernel_size=(2, 2))

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    539             result = self._slow_forward(*input, **kwargs)
    540         else:
--> 541             result = self.forward(*input, **kwargs)
    542         for hook in self._forward_hooks.values():
    543             hook_result = hook(self, input, result)

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py in forward(self, input)
    343
    344     def forward(self, input):
--> 345         return self.conv2d_forward(input, self.weight)
    346
    347 class Conv3d(_ConvNd):

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py in conv2d_forward(self, input, weight)
    340                             _pair(0), self.dilation, self.groups)
    341         return F.conv2d(input, weight, self.bias, self.stride,
--> 342                         self.padding, self.dilation, self.groups)
    343
    344     def forward(self, input):

RuntimeError: Input type (torch.cuda.ByteTensor) and weight type (torch.cuda.FloatTensor) should be the same

I double-checked that my Neural Net and my Inputs both are in GPU. I'm still getting this error and I don't understand why!

Somebody, please help me to get out of this error.


title: pytorch code sudden fails on colab with NVIDIA driver on your system is too old

body:
I had some code which worked on colab (gpu runtime) just a short while ago. Suddenly I am getting
The NVIDIA driver on your system is too old (found version 10010).
nvcc shows
Cuda compilation tools, release 10.1, V10.1.243
I tried torch versions 1.5.1, then 1.13.0. Both keep getting this error.
There is a discussion showing other people having doubts. with no clear resolution.
https://github.com/pytorch/pytorch/issues/27738
Anyone having the same problem?


title: PyTorch: is there a definitive training loop similar to Keras' fit()?

body:
I'm coming over from Keras to PyTorch, and one of the surprising things I've found is that I'm supposed to implement my own training loop.

In Keras, there is a de facto fit() function that: (1) runs gradient descent and (2) collects a history of metrics for loss and accuracy over both the training set and validation set.

In PyTorch, it appears that the programmer needs to implement the training loop. Since I'm new to PyTorch, I don't know if my training loop implementation is correct. I just want to compare apples-to-apples loss and accuracy metrics with what I'm seeing in Keras.

I've already read through:

the official PyTorch 60-minute blitz, where they provide a sample training loop.
official PyTorch example code, where I've found the training loop placed in-line with other code.
the O'Reilly book Programming PyTorch for Deep Learning with its own training loop.
Stanford CS230 sample code.
various blog posts (e.g. here and here).

So I'm wondering: is there a definitive, universal training loop implementation that does the same thing and reports the same numbers as the Keras fit() function?

My points of frustration:

Pulling data out of the dataloader is not consistent between image data and NLP data.
Correctly computing loss and accuracy is not consistent in any sample code I've seen.
Some code examples use Variable, while others do not.
Unnecessarily detailed: moving data to/from the GPU; knowing when to call zero_grad().

For what it's worth, here is my current implementation. Are there any obvious bugs?

import time

def train(model, optimizer, loss_fn, train_dl, val_dl, epochs=20, device='cuda'):
    '''
    Runs training loop for classification problems. Returns Keras-style
    per-epoch history of loss and accuracy over training and validation data.

    Parameters
    ----------
    model : nn.Module
        Neural network model
    optimizer : torch.optim.Optimizer
        Search space optimizer (e.g. Adam)
    loss_fn :
        Loss function (e.g. nn.CrossEntropyLoss())
    train_dl :
        Iterable dataloader for training data.
    val_dl :
        Iterable dataloader for validation data.
    epochs : int
        Number of epochs to run
    device : string
        Specifies 'cuda' or 'cpu'

    Returns
    -------
    Dictionary
        Similar to Keras' fit(), the output dictionary contains per-epoch
        history of training loss, training accuracy, validation loss, and
        validation accuracy.
    '''

    print('train() called: model=%s, opt=%s(lr=%f), epochs=%d, device=%s\n' % \
          (type(model).__name__, type(optimizer).__name__,
           optimizer.param_groups[0]['lr'], epochs, device))

    history = {} # Collects per-epoch loss and acc like Keras' fit().
    history['loss'] = []
    history['val_loss'] = []
    history['acc'] = []
    history['val_acc'] = []

    start_time_sec = time.time()

    for epoch in range(epochs):

        # --- TRAIN AND EVALUATE ON TRAINING SET -----------------------------
        model.train()
        train_loss         = 0.0
        num_train_correct  = 0
        num_train_examples = 0

        for batch in train_dl:

            optimizer.zero_grad()

            x    = batch[0].to(device)
            y    = batch[1].to(device)
            yhat = model(x)
            loss = loss_fn(yhat, y)

            loss.backward()
            optimizer.step()

            train_loss         += loss.data.item() * x.size(0)
            num_train_correct  += (torch.max(yhat, 1)[1] == y).sum().item()
            num_train_examples += x.shape[0]

        train_acc   = num_train_correct / num_train_examples
        train_loss  = train_loss / len(train_dl.dataset)

        # --- EVALUATE ON VALIDATION SET -------------------------------------
        model.eval()
        val_loss       = 0.0
        num_val_correct  = 0
        num_val_examples = 0

        for batch in val_dl:

            x    = batch[0].to(device)
            y    = batch[1].to(device)
            yhat = model(x)
            loss = loss_fn(yhat, y)

            val_loss         += loss.data.item() * x.size(0)
            num_val_correct  += (torch.max(yhat, 1)[1] == y).sum().item()
            num_val_examples += y.shape[0]

        val_acc  = num_val_correct / num_val_examples
        val_loss = val_loss / len(val_dl.dataset)

        print('Epoch %3d/%3d, train loss: %5.2f, train acc: %5.2f, val loss: %5.2f, val acc: %5.2f' % \
              (epoch+1, epochs, train_loss, train_acc, val_loss, val_acc))

        history['loss'].append(train_loss)
        history['val_loss'].append(val_loss)
        history['acc'].append(train_acc)
        history['val_acc'].append(val_acc)

    # END OF TRAINING LOOP

    end_time_sec       = time.time()
    total_time_sec     = end_time_sec - start_time_sec
    time_per_epoch_sec = total_time_sec / epochs
    print()
    print('Time total:     %5.2f sec' % (total_time_sec))
    print('Time per epoch: %5.2f sec' % (time_per_epoch_sec))

    return history


title: Pytorch mask tensor with boolean numpy array

body:
I have a 84x84 pytorch tensor named target. I need to mask it with an 84x84 boolean numpy array which consists of True and False.

When I do target = target[mask], I get the error TypeError: can't convert np.ndarray of type numpy.bool_. The only supported types are: double, float, float16, int64, int32, and uint8.

Surprisingly, I get this error only when running on a GPU. When running on a CPU, everything works fine. How can I fix this?


title: Is there a bug with PyTorch training for large batch sizes, or with this script?

body:
I am following this PyTorch tutorial by Joshua L. Mitchell.  The grand finale of the tutorial is the following PyTorch training script.  One element, the batch size, I have parameterized in the first line of the script, which I run in a newly started Jupyter notebook.  The key parameter in question is BIGGER_BATCH, initially set to 4:

BIGGER_BATCH=4

import numpy as np
import torch # Tensor Package (for use on GPU)
import torch.nn as nn ## Neural Network package
import torch.optim as optim # Optimization package
import torchvision # for dealing with vision data
import torchvision.transforms as transforms # for modifying vision data to run it through models
from torch.autograd import Variable # for computational graph
import torch.nn.functional as F # Non-linearities package
import matplotlib.pyplot as plt # for plotting

def imshow(img):
    img = img / 2 + 0.5     # unnormalize
    npimg = img.numpy()
    plt.imshow(np.transpose(npimg, (1, 2, 0)))

transform = transforms.Compose( # we're going to use this to transform our data to make each sample more uniform
   [
    transforms.ToTensor(), # converts each sample from a (0-255, 0-255, 0-255) PIL Image format to a (0-1, 0-1, 0-1) FloatTensor format
    transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)) # for each of the 3 channels of the image, subtract mean 0.5 and divide by stdev 0.5
   ]) # the normalization makes each SGD iteration more stable and overall makes convergence easier

trainset = torchvision.datasets.CIFAR10(root='./data', train=True,
                                        download=True, transform=transform) # this is all we need to get/wrangle the dataset!

testset = torchvision.datasets.CIFAR10(root='./data', train=False,
                                       download=True, transform=transform)

trainloader = torch.utils.data.DataLoader(trainset, batch_size=BIGGER_BATCH,
                                          shuffle=False)
testloader = torch.utils.data.DataLoader(testset, batch_size=BIGGER_BATCH,
                                         shuffle=False)

classes = ('plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck') # each image can have 1 of 10 labels

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 10, 5) # Let's add more feature maps - that might help
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(10, 20, 5) # And another conv layer with even more feature maps
        self.fc1 = nn.Linear(20 * 5 * 5, 120) # and finally, adjusting our first linear layer's input to our previous output
        self.fc2 = nn.Linear(120, 10)

    def forward(self, x):
        x = self.conv1(x)
        x = F.relu(x) # we're changing our nonlinearity / activation function from sigmoid to ReLU for a slight speedup
        x = self.pool(x)
        x = self.conv2(x)
        x = F.relu(x)
        x = self.pool(x) # after this pooling layer, we're down to a torch.Size([4, 20, 5, 5]) tensor.
        x = x.view(-1, 20 * 5 * 5) # so let's adjust our tensor again.
        x = self.fc1(x)
        x = F.relu(x)
        x = self.fc2(x)
        x = F.relu(x)
        return x

net = Net().cuda()

NUMBER_OF_EPOCHS = 25
LEARNING_RATE = 1e-2
loss_function = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=LEARNING_RATE)

for epoch in range(NUMBER_OF_EPOCHS):
    train_loader_iter = iter(trainloader)
    for batch_idx, (inputs, labels) in enumerate(train_loader_iter):
        net.zero_grad()
        inputs, labels = Variable(inputs.float().cuda()), Variable(labels.cuda())
        output = net(inputs)
        loss = loss_function(output, labels)
        loss.backward()
        optimizer.step()
    if epoch % 5 is 0:
        print("Iteration: " + str(epoch + 1))

dataiter = iter(testloader)
images, labels = dataiter.next()

imshow(torchvision.utils.make_grid(images[0:4]))

outputs = net(Variable(images.cuda()))
_, predicted = torch.max(outputs.data, 1)
print('Predicted: ', ' '.join('%5s' % classes[predicted[j]]
                              for j in range(4)))
print('GroundTruth: ', ' '.join('%5s' % classes[labels[j]] for j in range(4)))

correct = 0
total = 0
for data in testloader:
    images, labels = data
    labels = labels.cuda()
    outputs = net(Variable(images.cuda()))
    _, predicted = torch.max(outputs.data, 1)
    total += labels.size(0)
    correct += (predicted == labels).sum()
print('Accuracy of the network on the 10000 test images: %d %%' % (
    100 * correct / total))

This gets a good, expected result of 58% accuracy:

Predicted:    cat  ship  ship plane
GroundTruth:    cat  ship  ship plane
Accuracy of the network on the 10000 test images: 58 %

Now if I change the first line of the above script to

BIGGER_BATCH=4096

and I restart the kernel and run the script I get 19% accuracy, consistently:

Predicted:    car  ship  ship  ship
GroundTruth:    cat  ship  ship plane
Accuracy of the network on the 10000 test images: 19 %

Note that I am not shuffling the inputs, so I can't attribute this change to the order of inputs in the training:

trainloader = torch.utils.data.DataLoader(trainset, batch_size=BIGGER_BATCH,
                                          shuffle=False)
testloader = torch.utils.data.DataLoader(testset, batch_size=BIGGER_BATCH,
                                         shuffle=False)

What accounts for the huge drop in accuracy when I increase the batch size?  Is there something wrong in the script, or something wrong in PyTorch, or something else I'm not thinking of?


title: How does python map works with torch.tensor?

body:
I am now in python so I am trying to understand this line from pytorch tutorial.

x_train, y_train, x_valid, y_valid = map(
    torch.tensor, (x_train, y_train, x_valid, y_valid)
)

I understand how map works on a single element

def sqr(a):
    return a * a

a = [1, 2, 3, 4]

a = map(sqr, a)
print(list(a))

And here I need to use list(a) to convert map object back to list.

But what I don't understand, is how does it work on multiple variables?

If I try to do this

def sqr(a):
    return a * a

a = [1, 2, 3, 4]
b = [1, 3, 5, 7]

a, b = map(sqr, (a, b))
print(list(a))
print(list(b))

I get an error: TypeError: can't multiply sequence by non-int of type 'list'

Please clarify this for me
Thank you


title: Unable to Normalize Tensor in PyTorch

body:
I am trying to normalize the tensor outputted by my network but am getting an error in doing so. The code is as follows:

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
model_load_path = r'path\to\saved\model\file'
model.load_state_dict(torch.load(model_load_path))
model.eval()
output = model(input).to(device).view(-1, 1, 150, 150)
inv_normalize = transforms.Compose(
    [
        transforms.Normalize(mean=[-0.5/0.5], std=[1/0.5])
    ]
)
print(output.size())  # The size printed is torch.Size([1, 1, 150, 150])
output = inv_normalize(output)

I am getting an error in the following line:

output = inv_normalize(output)

The error reads:

TypeError: tensor is not a torch image.

My output is a single image, with a single channel, and height and width = 150

Any help will be appreciated! Thanks!


I am applying transfer learning with the python framework (PyTorch). I am getting the below error, when loading a PyTorch pre-trained model in Google Colab. After changing the code 1 to be as code 2, I got the same error.

CODE 1:  BertModel.from_pretrained
CODE 2: TFBertModel.from_pretrained
Error: AttributeError: module 'transformers' has no attribute 'TFBertModel'

I tried to search the internet, but I didn't find any useful content.


title: Pytorch error: Could not run 'aten::slow_conv3d_forward' with arguments from the 'CUDATensorId' backend

body:
I am training a CNN on CUDA GPU which takes 3D medical images as input and outputs a classifier. I suspect there may be a bug in pytorch. I am running pytorch 1.4.0.  The GPU is 'Tesla P100-PCIE-16GB'. When I run the model on CUDA I get the error

Traceback (most recent call last):
  File "/home/ub/miniconda3/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3331, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-55-cc0dd3d9cbb7>", line 1, in <module>
    net(cc)
  File "/home/ub/miniconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "<ipython-input-2-19e11966d1cd>", line 181, in forward
    out = self.layer1(x)
  File "/home/ub/miniconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ub/miniconda3/lib/python3.7/site-packages/torch/nn/modules/container.py", line 100, in forward
    input = module(input)
  File "/home/ub/miniconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ub/miniconda3/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 480, in forward
    self.padding, self.dilation, self.groups)
RuntimeError: Could not run 'aten::slow_conv3d_forward' with arguments from the 'CUDATensorId' backend. 'aten::slow_conv3d_forward' is only available for these backends: [CPUTensorId, VariableTensorId].

To replicate the issue:

#input is a 64,64,64 3d image batch with 2 channels
class ConvNet(nn.Module):
    def __init__(self):
        super(ConvNet, self).__init__()
        self.layer1 = nn.Sequential(
            nn.Conv3d(2, 32, kernel_size=5, stride=1, padding=2),
            nn.ReLU(),
            nn.MaxPool3d(kernel_size=2, stride=2))
        self.layer2 = nn.Sequential(
            nn.Conv3d(32, 64, kernel_size=5, stride=1, padding=2),
            nn.ReLU(),
            nn.MaxPool3d(kernel_size=2, stride=2))
        self.drop_out = nn.Dropout()
        self.fc1 = nn.Linear(16 * 16*16 * 64, 1000)
        self.fc2 = nn.Linear(1000, 2)
        # self.softmax =  nn.LogSoftmax(dim=1)

    def forward(self, x):
        # print(out.shape)
        out = self.layer1(x)
        # print(out.shape)
        out = self.layer2(out)
        # print(out.shape)
        out = out.reshape(out.size(0), -1)
        # print(out.shape)
        out = self.drop_out(out)
        # print(out.shape)
        out = self.fc1(out)
        # print(out.shape)
        out = self.fc2(out)
        # out = self.softmax(out)
        # print(out.shape)
        return out

net = Convnet()
input = torch.randn(16, 2, 64, 64, 64)
net(input)


title: What is the equivalent of torch.nn.functional.grid_sample in Tensorflow / Numpy?

body: I am new to pytorch and have been trying to convert some code . Can't find this particular functionality . Does it exist in tensorflow ?


title: ‘DataParallel’ object has no attribute ‘init_hidden’

body:
What I want to do is using DataParallel in my custom RNN class.

It seems like I initialized hidden_0 in a wrong way...

class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size, n_layers=1):
    super(RNN, self).__init__()
    self.input_size = input_size
    self.hidden_size = hidden_size
    self.output_size = output_size
    self.n_layers = n_layers

    self.encoder = nn.Embedding(input_size, hidden_size)
    self.gru = nn.GRU(hidden_size, hidden_size, n_layers,batch_first = True)
    self.decoder = nn.Linear(hidden_size, output_size)
    self.init_hidden(batch_size)

def forward(self, input):
    input = self.encoder(input)
    output, self.hidden = self.gru(input,self.hidden)
    output = self.decoder(output.contiguous().view(-1,self.hidden_size))
    output = output.contiguous().view(batch_size,num_steps,N_CHARACTERS)
    #print (output.size())10,50,67

    return output

def init_hidden(self,batch_size):
    self.hidden = Variable(T.zeros(self.n_layers, batch_size, self.hidden_size).cuda())

And I call the network in this way:

decoder = T.nn.DataParallel(RNN(N_CHARACTERS, HIDDEN_SIZE, N_CHARACTERS), dim=1).cuda()

Then start training:

for epoch in range(EPOCH_):
    hidden = decoder.init_hidden()

But I get the error and I have no ideal how to fix it…

  'DataParallel' object has no attribute 'init_hidden'

Thanks for your help!


title: Install Detectron2 on Windows 10

body: I try to install Facebook's Detectron2 followed this official repo. Following that repo, detectron2 can only install on linux. However, I'm working on a server run on Windows operator. Anybody know how to install it on Windows?


title: IndexError when iterating my dataset using Dataloader in PyTorch

body:
I iterated my dataset using Dataloader in PyTorch 0.2 like these:

dataloader = torch.utils.data.DataLoader(...)
data_iter = iter(dataloader)
data = data_iter.next()

but IndexError was raised.

Traceback (most recent call last):
  File "main.py", line 193, in <module>
    data_target = data_target_iter.next()
  File "/usr/local/lib/python3.5/dist-packages/torch/utils/data/dataloader.py", line 201, in __next__
    return self._process_next_batch(batch)
  File "/usr/local/lib/python3.5/dist-packages/torch/utils/data/dataloader.py", line 221, in _process_next_batch
    raise batch.exc_type(batch.exc_msg)
IndexError: Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/torch/utils/data/dataloader.py", line 40, in _worker_loop
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "/usr/local/lib/python3.5/dist-packages/torch/utils/data/dataloader.py", line 40, in <listcomp>
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "/home/asr4/zhuminxian/adversarial/code/dataset/data_loader.py", line 33, in __getitem__
    return self.X_train[idx], self.y_train[idx]
IndexError: index 4196 is out of bounds for axis 0 with size 4135

I am wondering why the index was out of bounds. Is it the bug of Pytorch?

I tried to run my code again, the same error raised, but at different iteration and with different out-of-bound index.


title: How to use double as the default type for floating numbers in PyTorch

body: I want all the floating numbers in my PyTorch code double type by default, how can I do that?


title: Simple Pytorch Example - Loss on training doesnt decrease

body:
I am just starting to try and learn pytorch and am finding it frustrating regardless of how it is advertised :)

Here I am running a simple regression as an experiment but since the loss doesn't seem to be decreasing with each epoch (on the training) I must be doing something wrong -- either in training or how I am collecting the MSE?

from __future__ import division
import numpy as np
import matplotlib.pyplot as plt

import torch
import torch.utils.data as utils_data
from torch.autograd import Variable
from torch import optim, nn
from torch.utils.data import Dataset
import torch.nn.functional as F

from sklearn.datasets import load_boston

cuda=True

#regular old numpy
boston = load_boston()

x=boston.data
y=boston.target

x.shape

training_samples = utils_data.TensorDataset(x, y)
data_loader = utils_data.DataLoader(training_samples, batch_size=10)

len(data_loader) #number of batches in an epoch

#override this
class Net(nn.Module):
    def __init__(self):
         super(Net, self).__init__()

         #all the layers
         self.fc1   = nn.Linear(x.shape[1], 50)
         self.drop = nn.Dropout(p=0.2)
         self.fc2   = nn.Linear(50, 1)

    #
    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = self.drop(x)
        x = self.fc2(x)

        return x

net=Net()
if cuda:
    net.cuda()
print(net)

# create a stochastic gradient descent optimizer
optimizer = optim.Adam(net.parameters())
# create a loss function (mse)
loss = nn.MSELoss()

# create a stochastic gradient descent optimizer
optimizer = optim.Adam(net.parameters())
# create a loss function (mse)
loss = nn.MSELoss()

# run the main training loop
epochs =20
hold_loss=[]

for epoch in range(epochs):
    cum_loss=0.
    for batch_idx, (data, target) in enumerate(data_loader):
        tr_x, tr_y = Variable(data.float()), Variable(target.float())
        if cuda:
            tr_x, tr_y = tr_x.cuda(), tr_y.cuda()

        # Reset gradient
        optimizer.zero_grad()

        # Forward pass
        fx = net(tr_x)
        output = loss(fx, tr_y) #loss for this batch
        cum_loss += output.data[0]

        # Backward
        output.backward()

        # Update parameters based on backprop
        optimizer.step()
    hold_loss.append(cum_loss)
    #print(epoch+1, cum_loss) #

plt.plot(np.array(hold_loss))


title: Pytorch transform.ToTensor() changes image

body:
I want to convert images to tensor using torchvision.transforms.ToTensor(), after processing I printed the image but the image became so weird. Here is my code:
trans = transforms.Compose([
    transforms.ToTensor()])

demo = Image.open(img)
demo_img = trans(demo)
demo_array = demo_img.numpy()*255
print(Image.fromarray(demo_array.astype(np.uint8)))

The original image is this
But after processing it is showed like this
Did I write something wrong or miss something?


title: While using GPU for PyTorch models, getting the CUDA error: Unknown error?

body:
I am trying to use a pre-trained model using PyTorch. While loading the model to GPU, it is giving the following error:

Traceback (most recent call last):
  File "model\vgg_model.py", line 45, in <module>
    vgg_model1 = VGGFeatureExtractor(True).double().to(device)
  File "C:\Users\myidi\Anaconda3\envs\openpose\lib\site-packages\torch\nn\modules\module.py", line 386, in to
    return self._apply(convert)
  File "C:\Users\myidi\Anaconda3\envs\openpose\lib\site-packages\torch\nn\modules\module.py", line 193, in _apply
    module._apply(fn)
  File "C:\Users\myidi\Anaconda3\envs\openpose\lib\site-packages\torch\nn\modules\module.py", line 193, in _apply
    module._apply(fn)
  File "C:\Users\myidi\Anaconda3\envs\openpose\lib\site-packages\torch\nn\modules\module.py", line 199, in _apply
    param.data = fn(param.data)
  File "C:\Users\myidi\Anaconda3\envs\openpose\lib\site-packages\torch\nn\modules\module.py", line 384, in convert
    return t.to(device, dtype if t.is_floating_point() else None, non_blocking)
  File "C:\Users\myidi\Anaconda3\envs\openpose\lib\site-packages\torch\cuda\__init__.py", line 163, in _lazy_init
    torch._C._cuda_init()
RuntimeError: CUDA error: unknown error

I have a Windows 10 Laptop, Nvidia 940m GPU, Latest Pytorch and CUDA Toolkit 9.0 (Also tried on 10.0).

I have tried re-installing the GPU drivers, restarted my machine, re-installed PyTorch, Torchvision and CUDA Toolkit.

While using the following to see if PyTorch is detecting a GPU:

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

I am getting the following output: device(type='cuda').

What could be the possible issues? I have tried the solution mentioned here: https://github.com/pytorch/pytorch/issues/20990 and the issue still persists.

I simply put the torch.cuda.current_device() after import torch but the issue still persists.


title: CIFAR-10 Meaningless Normalization Values

body:
I tried to build a neural network for a CIFAR-10 database. I used Pytorch Framework.

I have a question about of data loading step.

transform_train = T.Compose([
    T.RandomCrop(32, padding=4),
    T.RandomHorizontalFlip(),
    T.ToTensor(),
    T.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010)),
])

trainset = tv.datasets.CIFAR10(root=root, train=True, download=True, transform=transform_train)

This is a normal step for data loading step. While loading data, I am normalizing values. At the beginning of the my project I found below row.

T.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))

After I searched better transform values, I found this values.

T.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010))

I did not find an explanation why these values were used. Do you have a description of these values?


title: padding zeroes to a tensor on both dimensions

body:
I have a tensor t

1 2
3 4
5 6
7 8

And I would like to make it

0 0 0 0
0 1 2 0
0 3 4 0
0 5 6 0
0 7 8 0
0 0 0 0

I tried stacking with new=torch.tensor([0. 0. 0. 0.]) tensor four times but that did not work.

t = torch.arange(8).reshape(1,4,2).float()
print(t)
new=torch.tensor([[0., 0., 0.,0.]])
print(new)
r = torch.stack([t,new])  # invalid argument 0: Tensors must have same number of dimensions: got 4 and 3
new=torch.tensor([[[0., 0., 0.,0.]]])
print(new)
r = torch.stack([t,new])  # invalid argument 0: Sizes of tensors must match except in dimension 0.

I also tried cat, that did not work either.


title: Python (Pytorch) Multiprocessing throwing errors: Connection reset by peer and File Not Found

body:
I'm trying to get something working similarly to keras' "fit_generator" method. Basically, I have a (very) large data file of mini-batches and I want to have my CPU grab mini-batches and populate a queue parallel to my GPU taking mini-batches from the queue and training on them. By having the CPU work in parallel to the GPU (as opposed to having the CPU grab a batch and making the GPU wait for the CPU before it trains on that batch) I should be able to reduce my training time by about half. I have benchmarked how long it takes the CPU to grab a mini-batch, and it's taking a comparable amount of time to how long it takes my GPU to train on one mini-batch, so parallelizing the CPU and GPU should work alright. I haven't found a built-in method in pytorch to do this, if there is one, please let me know.

So I have tried to use the torch.multiprocessing module to do what I want, but I'm not able to complete the training as I always get some sort of error right before training is completed. The torch.multiprocessing module should be a wrapper with essentially all the same functionalities as the regular multiprocessing module except it allows pytorch tensors to be shared between processes. Basically, I have set up my code to have 2 functions, a loader function, and a trainer function like so:

    def data_gen(que,PATH,epochs,steps_per_epoch,batch_size=32):
        for epoch in range(epochs):
            for j in range(steps_per_epoch):
                with h5py.File(PATH,'r') as f:
                    X = f['X'][j*batch_size:(j+1)*batch_size]
                    Y = f['Y'][j*batch_size:(j+1)*batch_size]

                    X = autograd.Variable(torch.Tensor(X).resize_(batch_size,256,25).cpu())
                    Y = autograd.Variable(torch.Tensor(Y).cpu())

                    que.put((X,Y))
            que.put('stop')
        que.close()
        return

    def train_network(que,net,optimizer,epochs):
        print('Training for %s epochs...' %epochs)
        for epoch in range(epochs):
            while(True):
                data = que.get()
                if(data == 'stop'):
                    break
                net.zero_grad()
                net.hid = net.init_hid()
                inp,labels = data
                inp   = inp.cuda()
                labels = labels.cuda()
                out,hid = net(inp)
                loss = F.binary_cross_entropy(out,labels)
                loss.backward()
                optimizer.step()
            print('Epoch end reached')
        return

And then I run the two processes in parallel like so:

    if __name__ == '__main__':
        tmp.set_start_method('spawn')
        que = tmp.Queue(maxsize=10)
        loader = tmp.Process(target=data_gen, args=(que,PATH,epochs,steps), kwargs={'batch_size':batch_size})
        loader.start()
        trainer = tmp.Process(target=train_network, args=(que,net,optimizer,epochs,steps))
        trainer.start()
        loader.join()
        trainer.join()

I have the que put in a 'stop' value at the end of each epoch so I can break out of the loop in the trainer and go to the next epoch. This "poison pill" method appears to work because the code runs for multiple epochs, and the trainer does in fact print the end of epoch verification message. The code runs, and it does appear to speed up the training process (I've been trying to prototype this code on a small subset of the data so it's sometimes hard to tell how much speed up I'm getting), but at the end of training (and always at the end, no matter how many epochs I specify), I always get an error:

    Process Process-2:
    Traceback (most recent call last):
      File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
        self.run()
      File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
        self._target(*self._args, **self._kwargs)
      File "/media/digitalstorm/Storage/RNN_Prototype/Lazuli_rnnprototype.py", line 307, in train_network
        data = que.get()
      File "/usr/lib/python3.6/multiprocessing/queues.py", line 113, in get
        return _ForkingPickler.loads(res)
      File "/usr/local/lib/python3.6/dist-packages/torch/multiprocessing/reductions.py", line 70, in rebuild_storage_fd
        fd = df.detach()
      File "/usr/lib/python3.6/multiprocessing/resource_sharer.py", line 57, in detach
        with _resource_sharer.get_connection(self._id) as conn:
      File "/usr/lib/python3.6/multiprocessing/resource_sharer.py", line 87, in get_connection
        c = Client(address, authkey=process.current_process().authkey)
      File "/usr/lib/python3.6/multiprocessing/connection.py", line 487, in Client
        c = SocketClient(address)
      File "/usr/lib/python3.6/multiprocessing/connection.py", line 614, in SocketClient
        s.connect(address)
    FileNotFoundError: [Errno 2] No such file or directory

or, if I mess around a bit with various options, I sometimes get an error like this:

    Process Process-2:
    Traceback (most recent call last):
      File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
        self.run()
      File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
        self._target(*self._args, **self._kwargs)
      File "/media/digitalstorm/Storage/RNN_Prototype/Lazuli_rnnprototype.py", line 306, in train_network
        data = que.get()
      File "/usr/lib/python3.6/multiprocessing/queues.py", line 113, in get
        return _ForkingPickler.loads(res)
      File "/usr/local/lib/python3.6/dist-packages/torch/multiprocessing/reductions.py", line 70, in rebuild_storage_fd
        fd = df.detach()
      File "/usr/lib/python3.6/multiprocessing/resource_sharer.py", line 58, in detach
        return reduction.recv_handle(conn)
      File "/usr/lib/python3.6/multiprocessing/reduction.py", line 182, in recv_handle
        return recvfds(s, 1)[0]
      File "/usr/lib/python3.6/multiprocessing/reduction.py", line 153, in recvfds
        msg, ancdata, flags, addr = sock.recvmsg(1, socket.CMSG_LEN(bytes_size))
    ConnectionResetError: [Errno 104] Connection reset by peer

I don't know where I'm going wrong. Admittedly, I am a novice at multiprocessing, so it's hard for me to debug what went wrong exactly. Any help would be appreciated, thanks!


title: How to correctly use Numpy's FFT function in PyTorch?

body:
I was recently introduced to PyTorch and began running through the library's documentation and tutorials.
In the "Creating extensions using numpy and scipy" tutorial, under "Parameter-less example", a sample function is created using numpy called BadFFTFunction.
The description for the function states:

"This layer doesn’t particularly do anything useful or mathematically
correct.
It is aptly named BadFFTFunction"

The function and its usage are given as:
from numpy.fft import rfft2, irfft2

class BadFFTFunction(Function):

    def forward(self, input):
        numpy_input = input.numpy()
        result = abs(rfft2(numpy_input))
        return torch.FloatTensor(result)

    def backward(self, grad_output):
        numpy_go = grad_output.numpy()
        result = irfft2(numpy_go)
        return torch.FloatTensor(result)

def incorrect_fft(input):
    return BadFFTFunction()(input)

input = Variable(torch.randn(8, 8), requires_grad=True)
result = incorrect_fft(input)
print(result.data)
result.backward(torch.randn(result.size()))
print(input.grad)

Unfortunately, I was only recently introduced to signal processing as well, and am unsure of where the (likely obvious) error is in this function.
I am wondering, how might one go about fixing this function so that its forward and backward outputs are correct?
How can BadFFTFunction be fixed so that a differentiable FFT function can be used in PyTorch?


title: How to share a list of tensors in PyTorch multiprocessing?

body:
I am programming with PyTorch multiprocessing. I want all the subprocesses can read/write the same list of tensors (no resize). For example the variable can be

m = list(torch.randn(3), torch.randn(5))

Because each tensor has different sizes, I cannot organize them into a single tensor.

A python list has no share_memory_() function, and multiprocessing.Manager cannot handle a list of tensors. How can I share the variable m among multiple subprocesses?


title: Product of PyTorch tensors along arbitrary axes à la NumPy's `tensordot`

body:
NumPy provides the very useful tensordot function. It allows you to compute the product of two ndarrays along any axes (whose sizes match). I'm having a hard time finding anything similar in PyTorch. mm works only with 2D arrays, and matmul has some undesirable broadcasting properties.

Am I missing something? Am I really meant to reshape the arrays to mimic the products I want using mm?


title: How to use multi-gpu during inference in pytorch framework

body:
I am trying to make model prediction from unet3D built on pytorch framework. I am using multi-gpus

import torch
import os
import torch.nn as nn
os.environ['CUDA_DEVICE_ORDER']='PCI_BUS_ID'
os.environ['CUDA_VISIBLE_DEVICES']='0,1,2'

model = unet3d()
model = nn.DataParallel(model)
model = model.to('cuda')

result = model.forward(torch.tensor(input).to('cuda').float())

But the model still uses only 1 GPU (the first one) and I get memory error.

CUDA out of memory. Tried to allocate 64.00 MiB (GPU 0; 11.00 GiB total capacity; 8.43 GiB already allocated; 52.21 MiB free; 5.17 MiB cached)

How shoudl I use Multi-GPUs during inference phase? What is the mistake in my script above?


title: What is the difference between softmax and log-softmax?

body:
The difference between these two functions that has been described in this pytorch post: What is the difference between log_softmax and softmax?
 is: exp(x_i) / exp(x).sum()
and log softmax is: log(exp(x_i) / exp(x).sum()).

But for the Pytorch code below why am I getting different output:

>>> it = autograd.Variable(torch.FloatTensor([0.6229,0.3771]))
>>> op = autograd.Variable(torch.LongTensor([0]))
>>> m  = nn.Softmax()
>>> log = nn.LogSoftmax()
>>> m(it)
Variable containing:
`0.5611  0.4389`
[torch.FloatTensor of size 1x2]
>>>log(it)
Variable containing:
-0.5778 -0.8236
[torch.FloatTensor of size 1x2]

However, the value log(0.5611) is -0.25095973129 and log(0.4389) is -0.35763441915

Why is there such discrepancy?


I am using the following code for machine learning purposes (I am also quite new to python and pytorch). Basically, I think the problem is that multitasking is not happening for some reason.

I am using code from here: https://raw.githubusercontent.com/harryhan618/LaneNet/master/demo_test.py

The purpose of the code is draw lane markings on an image.

import cv2
import torch
import os
os.environ['KMP_DUPLICATE_LIB_OK']='True'

from lane_files.model import LaneNet
from lane_files.utils.transforms import *
from lane_files.utils.postprocess import embedding_post_process

if __name__=='__main__':
    net = LaneNet(pretrained=False, embed_dim=7, delta_v=.5, delta_d=3.)
    transform = Compose(Resize((800, 288)), ToTensor(),
                        Normalize(mean=(0.3598, 0.3653, 0.3662), std=(0.2573, 0.2663, 0.2756)))

    img = cv2.imread('data/train_images/frame0.png')
    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB) # RGB for net model input
    x = transform(img)[0]
    x.unsqueeze_(0)

    save_dict = torch.load('lane_files/experiments/exp0/exp0_best.pth', map_location='cpu')
    net.load_state_dict(save_dict['net'])
    net.eval()

    output = net(x)

    embedding = output['embedding']
    embedding = embedding.detach().cpu().numpy()
    embedding = np.transpose(embedding[0], (1, 2, 0))

    binary_seg = output['binary_seg']
    bin_seg_prob = binary_seg.detach().cpu().numpy()
    bin_seg_pred = np.argmax(bin_seg_prob, axis=1)[0]

    seg_img = np.zeros_like(img)
    lane_seg_img = embedding_post_process(embedding, bin_seg_pred, 0.5)
    color = np.array([[255, 125, 0], [0, 255, 0], [0, 0, 255], [0, 255, 255]], dtype='uint8')

    for i, lane_idx in enumerate(np.unique(lane_seg_img)):
        seg_img[lane_seg_img == lane_idx] = color[i]

    img = cv2.cvtColor(img, cv2.COLOR_RGB2BGR)
    img = cv2.resize(img, (800, 288))
    img = cv2.addWeighted(src1=seg_img, alpha=0.8, src2=img, beta=1., gamma=0.)

    cv2.imshow("", img)
    cv2.waitKey(5000)
    cv2.destroyAllWindows()

Expected result: An image displayed with lane markings on it
Actual result:

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:/Users/sarim/PycharmProjects/thesis/pytorch_learning.py", line 36, in <module>
    lane_seg_img = embedding_post_process(embedding, bin_seg_pred, 0.5)
  File "C:\Users\sarim\PycharmProjects\thesis\lane_files\utils\postprocess.py", line 29, in embedding_post_process
    mean_shift.fit(embedding_reshaped)
  File "C:\Users\sarim\AppData\Local\Programs\Python\Python37\lib\site-packages\sklearn\cluster\mean_shift_.py", line 424, in fit
    cluster_all=self.cluster_all, n_jobs=self.n_jobs)
  File "C:\Users\sarim\AppData\Local\Programs\Python\Python37\lib\site-packages\sklearn\cluster\mean_shift_.py", line 204, in mean_shift
    (seed, X, nbrs, max_iter) for seed in seeds)
  File "C:\Users\sarim\AppData\Local\Programs\Python\Python37\lib\site-packages\joblib\parallel.py", line 934, in __call__
    self.retrieve()
  File "C:\Users\sarim\AppData\Local\Programs\Python\Python37\lib\site-packages\joblib\parallel.py", line 833, in retrieve
    self._output.extend(job.get(timeout=self.timeout))
  File "C:\Users\sarim\AppData\Local\Programs\Python\Python37\lib\site-packages\joblib\_parallel_backends.py", line 521, in wrap_future_result
    return future.result(timeout=timeout)
  File "C:\Users\sarim\AppData\Local\Programs\Python\Python37\lib\concurrent\futures\_base.py", line 435, in result
    return self.__get_result()
  File "C:\Users\sarim\AppData\Local\Programs\Python\Python37\lib\concurrent\futures\_base.py", line 384, in __get_result
    raise self._exception
  File "C:\Users\sarim\AppData\Local\Programs\Python\Python37\lib\site-packages\joblib\externals\loky\_base.py", line 625, in _invoke_callbacks
    callback(self)
  File "C:\Users\sarim\AppData\Local\Programs\Python\Python37\lib\site-packages\joblib\parallel.py", line 309, in __call__
    self.parallel.dispatch_next()
  File "C:\Users\sarim\AppData\Local\Programs\Python\Python37\lib\site-packages\joblib\parallel.py", line 731, in dispatch_next
    if not self.dispatch_one_batch(self._original_iterator):
  File "C:\Users\sarim\AppData\Local\Programs\Python\Python37\lib\site-packages\joblib\parallel.py", line 759, in dispatch_one_batch
    self._dispatch(tasks)
  File "C:\Users\sarim\AppData\Local\Programs\Python\Python37\lib\site-packages\joblib\parallel.py", line 716, in _dispatch
    job = self._backend.apply_async(batch, callback=cb)
  File "C:\Users\sarim\AppData\Local\Programs\Python\Python37\lib\site-packages\joblib\_parallel_backends.py", line 510, in apply_async
    future = self._workers.submit(SafeFunction(func))
  File "C:\Users\sarim\AppData\Local\Programs\Python\Python37\lib\site-packages\joblib\externals\loky\reusable_executor.py", line 151, in submit
    fn, *args, **kwargs)
  File "C:\Users\sarim\AppData\Local\Programs\Python\Python37\lib\site-packages\joblib\externals\loky\process_executor.py", line 1022, in submit
    raise self._flags.broken
joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed to un-serialize. Please ensure that the arguments of the function are all picklable.


title: torch-1.1.0-cp37-cp37m-win_amd64.whl is not a supported wheel on this platform

body:
I need to use pyTorch when developing a RNN and whenever I try to install it I'm getting an error saying torch-1.1.0-cp37-cp37m-win_amd32.whl is not a supported wheel on this platform.

pip3 install https://download.pytorch.org/whl/cpu/torch-1.1.0-cp37-cp37m-win_amd32.whl

This is how I tried to install it according to https://pytorch.org/.

I'm using,

Python 3.7.3 (default, Mar 27 2019, 17:13:21) [MSC v.1915 64 bit (AMD64)] :: Anaconda, Inc. on win32

Any help would be appreciated. Thanks in advance!


title: Pytorch how to stack tensor like for loop

body:
I want to concat tensor generated in for loop, and get 2dTensor.
standard python, like below.

li = []
for i in range(0, len(items)):
    # calc something
    li.append(calc_result)

In my case, in for loop, generate torch.Size([768]) Tensor, and I want to get torch.Size([len(item),768]) Tensor.
How to do this?


I follow the tutorials in pytorch.org
It occurs error：TensorBoard logging requires TensorBoard version 1.15 or above,but I have install TensorBoard already.
Here is the code:
#from torch.utils.tensorboard import SummaryWriter
from tensorboardX import SummaryWriter
writer = SummaryWriter('runs/fashion_mnist_experiment_1')
#get some random training images
dataiter = iter(trainloader)
images , labels = dataiter.next()
#create grid of images
img_grid = torchvision.utils.make_grid(images)
matplotlib_imshow(img_grid,one_channel=True)
writer.add_image('four_fashion_images',img_grid)
writer.add_graph(net, images)
writer.close()

Error:
ImportError                               Traceback (most recent call last)
<ipython-input-12-d38808675cb4> in <module>
----> 1 writer.add_graph(net, images)
      2 writer.close()

~\anaconda3\envs\torch2\lib\site-packages\tensorboardX\writer.py in add_graph(self, model, input_to_model, verbose)
    791
    792         """
--> 793         from torch.utils.tensorboard._pytorch_graph import graph
    794         self._get_file_writer().add_graph(graph(model, input_to_model, verbose))
    795

~\anaconda3\envs\torch2\lib\site-packages\torch\utils\tensorboard\__init__.py in <module>
      2 from distutils.version import LooseVersion
      3 if not hasattr(tensorboard, '__version__') or LooseVersion(tensorboard.__version__) < LooseVersion('1.15'):
----> 4     raise ImportError('TensorBoard logging requires TensorBoard version 1.15 or above')
      5 del LooseVersion
      6 del tensorboard

ImportError: TensorBoard logging requires TensorBoard version 1.15 or above

Environment:
tensorboard               2.3.0                    pypi_0    pypi
tensorboard-plugin-wit    1.7.0                    pypi_0    pypi
tensorboardx              2.1                      pypi_0    pypi
tensorflow                1.2.1                    py36_0    defaults
pytorch                   1.6.0           py3.6_cuda102_cudnn7_0    pytorch
torchvision               0.7.0                py36_cu102    pytorch
future                    0.18.2                   py36_1    defaults
protobuf                  3.12.3           py36h33f27b4_0    defaults

I use from torch.utils.tensorboard import SummaryWriter at the beginning,but it occurs the error as same as above.Then I use from tensorboardX import SummaryWriter


title: How to load the saved tokenizer from pretrained model in Pytorch

body:
I fine-tuned a pretrained BERT model in Pytorch using huggingface transformer. All the training/validation is done on a GPU in cloud.

At the end of the training, I save the model and tokenizer like below:

best_model.save_pretrained('./saved_model/')
tokenizer.save_pretrained('./saved_model/')

This creates below files in the saved_model directory:

config.json
added_token.json
special_tokens_map.json
tokenizer_config.json
vocab.txt
pytorch_model.bin

Now, I download the saved_model directory in my computer and want to load the model and tokenizer. I can load the model like below

model = torch.load('./saved_model/pytorch_model.bin',map_location=torch.device('cpu'))

But how do I load the tokenizer? I am new to pytorch and not sure because there are multiple files. Probably I am not saving the model in the right way?

Please suggest.


title: pip - Unable to install Fastai

body:
Whenever I run:

pip install fastai

I get the error

"Command "python setup.py egg_info" failed with error code 1 in C:\Users\seja9890\AppData\Local\Temp\pip-install-_cw7ve61\torch\".

Can someone please guide me where I might be going wrong?

Ps.: I have tried updating setuptools and it doesn't help in my case.


title: How to use 'collate_fn' with dataloaders?

body:
I am trying to train a pretrained roberta model using 3 inputs, 3 input_masks and a label as tensors of my training dataset.
I do this using the following code:
from torch.utils.data import TensorDataset, DataLoader, RandomSampler, SequentialSampler
batch_size = 32
# Create the DataLoader for our training set.
train_data = TensorDataset(train_AT, train_BT, train_CT, train_maskAT, train_maskBT, train_maskCT, labels_trainT)
train_dataloader = DataLoader(train_data, batch_size=batch_size)

# Create the Dataloader for our validation set.
validation_data = TensorDataset(val_AT, val_BT, val_CT, val_maskAT, val_maskBT, val_maskCT, labels_valT)
val_dataloader = DataLoader(validation_data, batch_size=batch_size)

# Pytorch Training
training_args = TrainingArguments(
    output_dir='C:/Users/samvd/Documents/Master/AppliedMachineLearning/FinalProject/results',          # output directory
    num_train_epochs=1,              # total # of training epochs
    per_device_train_batch_size=32,  # batch size per device during training
    per_device_eval_batch_size=32,   # batch size for evaluation
    warmup_steps=500,                # number of warmup steps for learning rate scheduler
    weight_decay=0.01,               # strength of weight decay
    logging_dir='C:/Users/samvd/Documents/Master/AppliedMachineLearning/FinalProject/logs',            # directory for storing logs
)

trainer = Trainer(
    model=model,                          # the instantiated  Transformers model to be trained
    args=training_args,                   # training arguments, defined above
    train_dataset = train_data,           # training dataset
    eval_dataset = validation_data,       # evaluation dataset
)

trainer.train()

However this gives me the following error:

TypeError: vars() argument must have dict attribute

Now I have found out that it is probably because I don't use collate_fn when using DataLoader, but I can't really find a source that helps me define this correctly so the trainer understands the different tensors I put in.
Can anyone point me in the right direction?


title: Correct way of normalizing and scaling the MNIST dataset

body:
I've looked everywhere but couldn't quite find what I want. Basically the MNIST dataset has images with pixel values in the range [0, 255]. People say that in general, it is good to do the following:

Scale the data to the [0,1] range.
Normalize the data to have zero mean and unit standard deviation (data - mean) / std.

Unfortunately, no one ever shows how to do both of these things. They all subtract a mean of 0.1307 and divide by a standard deviation of 0.3081. These values are basically the mean and the standard deviation of the dataset divided by 255:
from torchvision.datasets import MNIST
import torchvision.transforms as transforms

trainset = torchvision.datasets.MNIST(root='./data', train=True, download=True)
print('Min Pixel Value: {} \nMax Pixel Value: {}'.format(trainset.data.min(), trainset.data.max()))
print('Mean Pixel Value {} \nPixel Values Std: {}'.format(trainset.data.float().mean(), trainset.data.float().std()))
print('Scaled Mean Pixel Value {} \nScaled Pixel Values Std: {}'.format(trainset.data.float().mean() / 255, trainset.data.float().std() / 255))

This outputs the following
Min Pixel Value: 0
Max Pixel Value: 255
Mean Pixel Value 33.31002426147461
Pixel Values Std: 78.56748962402344
Scaled Mean: 0.13062754273414612
Scaled Std: 0.30810779333114624

However clearly this does none of the above! The resulting data 1) will not be between [0, 1] and will not have mean 0 or std 1. In fact this is what we are doing:
[data - (mean / 255)] / (std / 255)
which is very different from this
[(scaled_data) - (mean/255)] / (std/255)
where scaled_data is just data / 255.


title: Does batch normalisation work with a small batch size?

body:
I'm using batch normalization with a batch size of size 10 for face detection, I wanted to know if it is better to remove the batch norm layers or keep them.
And if it is better to remove them what can I use instead?


title: No module named torch.distributed

body:
ImportError: No module named torch.distributed

File "train.py", line 4, in <module>
    import torch.distributed as dist
ImportError: No module named torch.distributed

I installed CUDA AND cuDNN then created env and installed pip3 install torch torchvision but getting error.


title: How to fix "ValueError: Expected input batch_size (1) to match target batch_size (4)."?

body:
I'm training a pytorch neural network on google colab to classify sign langauge alphabets of 29 classes in total.

We've been fixing the code by changing various params but it won't work anyway.

    transform = transforms.Compose([

        #gray scale
        transforms.Grayscale(),

        #resize
        transforms.Resize((128,128)),

        #converting to tensor
        transforms.ToTensor(),

        #normalize
        transforms.Normalize( (0.1307,), (0.3081,)),
    ])

    data_dir = 'data/train/asl_alphabet_train'

    #dataset
    full_dataset = datasets.ImageFolder(root=data_dir, transform=transform)

    #train & test
    train_size = int(0.8 * len(full_dataset))
    test_size = len(full_dataset) - train_size

    #splitting
    train_dataset, test_dataset = torch.utils.data.random_split(full_dataset, [train_size, test_size])

    trainloader = torch.utils.data.DataLoader(train_dataset , batch_size = 4, shuffle = True )
    testloader = torch.utils.data.DataLoader(test_dataset , batch_size = 4, shuffle = False )

    #neural net architecture
    Net(
  (conv1): Conv2d(1, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (conv2): Conv2d(32, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (conv3): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (fc1): Linear(in_features=32768, out_features=128, bias=True)
  (fc2): Linear(in_features=128, out_features=29, bias=True)
  (dropout): Dropout(p=0.5)
   )

    loss_fn = nn.CrossEntropyLoss()
    #optimizer
    opt = optim.SGD(model.parameters(), lr=0.01)
    def train(model, train_loader, optimizer, loss_fn, epoch, device):
        #telling pytorch that training mode is on
        model.train()
        loss_epoch_arr = []

        #epochs
        for e in range(epoch):

            # bach_no, data, target
            for batch_idx, (data, target) in enumerate(train_loader):

                #moving to GPU
                #data, target = data.to(device), target.to(device)

                #Making gradints zero
                optimizer.zero_grad()

                #generating output
                output = model(data)

                #calculating loss
                loss = loss_fn(output, target)

                #backward propagation
                loss.backward()

                #stepping optimizer
                optimizer.step()

                #printing at each 10th epoch
                if batch_idx % 10 == 0:
                    print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                        epoch, batch_idx * len(data), len(train_loader.dataset),
                        100. * batch_idx / len(train_loader), loss.item()))

                #de-allocating memory
                del data,target,output
                #torch.cuda.empty_cache()

            #appending values
            loss_epoch_arr.append(loss.item())

        #plotting loss
        plt.plot(loss_epoch_arr)
        plt.show()

    train(model, trainloader , opt, loss_fn, 10, device)

  ValueError: Expected input batch_size (1) to match target batch_size
  (4).

We're beginners in pytorch and trying to figure out what the problem is.


title: How to convert Onnx model (.onnx) to Tensorflow (.pb) model

body:
I am trying to convert .onxx model to .pb model. I have written the code but i am getting error:

@tf_func(tf.ceil)AttributeError: module 'tensorflow' has no attribute 'ceil'

Code:
import onnx
from tensorflow.python.tools.import_pb_to_tensorboard import import_to_tensorboard
from onnx_tf.backend import prepare
onnx_model = onnx.load("original_3dlm.onnx")
tf_rep = prepare(onnx_model)
tf_rep.export_graph("model_var.pb")
import_to_tensorboard("model_var.pb", "tb_log")

How to resolve this issue? Is there any other way to convert Onxx to Tensorflow?


title: Multiprocessing using cuda

body:
I am trying to run multiprocessing in my python program. I created two processes and passed a neural network in the one process and some heavy computational function in the other. I wanted the neural net to run on GPU and the other function on CPU and thereby I defined neural net using cuda() method.

import cv2
import torch
import torch.nn as nn
import multiprocessing as mp    #I even tried import torch.multiprocessing
from multiprocessing import set_start_method
try:
    set_start_method('spawn')
except RunTimeError:
    pass

class network(nn.Module):
    '''neural net class'''

def func1(img, net):
    '''used to call neural net'''

def func2(img):
    '''image processing part'''

if __name__ == '__main__':
    net = network().cuda() #here network is my neural net class
    img = cv2.imread('img.png')
    p1 mp.Process(target= func1, args=(img, net))    #func1 is used to run neural net
    p2 = mp.Process(target=func2, args=(img,))    #func2 is used for some img-processing
    p1.start()
    p2.start()
    p1.join()
    p2.join()

But when I run the program the got the following error:
RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the ‘spawn’ start method

So I tried with spawn as well as forkserver start method, but then I got the other error:
RuntimeError: cuda runtime error (71) : operation not supported at …/torch/csrc/generic/StorageSharing.cpp:245

I have tried python3 multiprocessing and torch.multiprocessing both but nothing worked for me.


title: RuntimeError: CUDA out of memory in training with pytorch "Pose2Seg"

body:
When I run this code https://github.com/erezposner/Pose2Seg
And I made all steps in this tutorial https://towardsdatascience.com/detection-free-human-instance-segmentation-using-pose2seg-and-pytorch-72f48dc4d23e
but I have this error in cuda:
RuntimeError: CUDA out of memory. Tried to allocate 128.00 MiB (GPU 0; 4.00 GiB total capacity; 2.57 GiB already allocated; 74.77 MiB free; 2.85 GiB reserved in total by PyTorch) (malloc at ..\c10\cuda\CUDACachingAllocator.cpp:289) (no backtrace available)

How can I solve this?
(base) C:\Users\ASUS\Pose2Seg>python train.py
06-23 07:30:01 ===========> loading model <===========
total params in model is 334, in pretrained model is 336, init 334
06-23 07:30:03 ===========> loading data <===========
loading annotations into memory...
Done (t=4.56s)
creating index...
index created!
06-23 07:30:08 ===========> set optimizer <===========
06-23 07:30:08 ===========>   training    <===========
C:\Users\ASUS\Anaconda3\Anaconda\lib\site-packages\torch\nn\functional.py:2796: UserWarning: nn.functional.upsample is deprecated. Use nn.functional.interpolate instead.
  warnings.warn("nn.functional.upsample is deprecated. Use nn.functional.interpolate instead.")
C:\Users\ASUS\Anaconda3\Anaconda\lib\site-packages\torch\nn\functional.py:2973: UserWarning: Default upsampling behavior when mode=bilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details.
  "See the documentation of nn.Upsample for details.".format(mode))
C:\Users\ASUS\Anaconda3\Anaconda\lib\site-packages\torch\nn\functional.py:3289: UserWarning: Default grid_sample and affine_grid behavior has changed to align_corners=False since 1.3.0. Please specify align_corners=True if the old behavior is desired. See the documentation of grid_sample for details.
  warnings.warn("Default grid_sample and affine_grid behavior has changed "
C:\Users\ASUS\Anaconda3\Anaconda\lib\site-packages\torch\nn\functional.py:3226: UserWarning: Default grid_sample and affine_grid behavior has changed to align_corners=False since 1.3.0. Please specify align_corners=True if the old behavior is desired. See the documentation of grid_sample for details.
  warnings.warn("Default grid_sample and affine_grid behavior has changed "
06-23 07:30:13 Epoch: [0][0/56599]      Lr: [6.68e-05]  Time 4.228 (4.228)      Data 0.028 (0.028)      loss 0.85738 (0.85738)
06-23 07:30:22 Epoch: [0][10/56599]     Lr: [6.813333333333334e-05]     Time 0.847 (1.280)      Data 0.012 (0.051)      loss 0.44195 (0.71130)
06-23 07:30:33 Epoch: [0][20/56599]     Lr: [6.946666666666667e-05]     Time 0.882 (1.180)      Data 0.045 (0.037)      loss 0.41523 (0.60743)
Traceback (most recent call last):
  File "train.py", line 157, in <module>
    optimizer, epoch, iteration)
  File "train.py", line 74, in train
    loss.backward()
  File "C:\Users\ASUS\Anaconda3\Anaconda\lib\site-packages\torch\tensor.py", line 198, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "C:\Users\ASUS\Anaconda3\Anaconda\lib\site-packages\torch\autograd\__init__.py", line 100, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: CUDA out of memory. Tried to allocate 128.00 MiB (GPU 0; 4.00 GiB total capacity; 2.57 GiB already allocated; 74.77 MiB free; 2.85 GiB reserved in total by PyTorch) (malloc at ..\c10\cuda\CUDACachingAllocator.cpp:289)
(no backtrace available)

cudatoolkit == 10.1.243

python3.6.5
The version of libs:
>>> import tensorflow
2020-06-23 09:45:01.840827: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
>>> tensorflow.__version__
'2.2.0'
>>> import keras
Using TensorFlow backend.
>>> keras.__version__
'2.3.1'
>>> import torch
>>> torch.__version__
'1.5.1'
>>> import torchvision
>>> torchvision.__version__
'0.6.1'
>>> import pycocotools

train.py code
import os
import sys
import time
import logging
import argparse
import numpy as np
from tqdm import tqdm

import torch
import torch.utils.data

from lib.averageMeter import AverageMeters
from lib.logger import colorlogger
from lib.timer import Timers
from lib.averageMeter import AverageMeters
from lib.torch_utils import adjust_learning_rate
import os
from modeling.build_model import Pose2Seg
from datasets.CocoDatasetInfo import CocoDatasetInfo, annToMask
from test import test

NAME = "release_base"

# Set `LOG_DIR` and `SNAPSHOT_DIR`

def setup_logdir():
    timestamp = time.strftime("%Y-%m-%d_%H_%M_%S", time.localtime())
    LOGDIR = os.path.join(os.getcwd(), 'logs', '%s_%s' % (NAME, timestamp))
    SNAPSHOTDIR = os.path.join(
        os.getcwd(), 'snapshot', '%s_%s' % (NAME, timestamp))
    if not os.path.exists(LOGDIR):
        os.makedirs(LOGDIR)
    if not os.path.exists(SNAPSHOTDIR):
        os.makedirs(SNAPSHOTDIR)
    return LOGDIR, SNAPSHOTDIR

LOGDIR, SNAPSHOTDIR = setup_logdir()

# Set logging
logger = colorlogger(log_dir=LOGDIR, log_name='train_logs.txt')

# Set Global Timer
timers = Timers()

# Set Global AverageMeter
averMeters = AverageMeters()

def train(model, dataloader, optimizer, epoch, iteration):
    # switch to train mode
    model.train()

    averMeters.clear()
    end = time.time()
    for i, inputs in enumerate(dataloader):
        averMeters['data_time'].update(time.time() - end)
        iteration += 1

        lr = adjust_learning_rate(optimizer, iteration, BASE_LR=0.0002,
                                  WARM_UP_FACTOR=1.0/3, WARM_UP_ITERS=1000,
                                  STEPS=(0, 14150*15, 14150*20), GAMMA=0.1)

        # forward
        outputs = model(**inputs)

        # loss
        loss = outputs

        # backward
        averMeters['loss'].update(loss.data.item())
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        # measure elapsed time
        averMeters['batch_time'].update(time.time() - end)
        end = time.time()

        if i % 10 == 0:
            logger.info('Epoch: [{0}][{1}/{2}]\t'
                        'Lr: [{3}]\t'
                        'Time {batch_time.val:.3f} ({batch_time.avg:.3f})\t'
                        'Data {data_time.val:.3f} ({data_time.avg:.3f})\t'
                        'loss {loss.val:.5f} ({loss.avg:.5f})\t'
                        .format(
                            epoch, i, len(dataloader), lr,
                            batch_time=averMeters['batch_time'], data_time=averMeters['data_time'],
                            loss=averMeters['loss'])
                        )

        if i % 10000 == 0:
            torch.save(model.state_dict(), os.path.join(
                SNAPSHOTDIR, '%d_%d.pkl' % (epoch, i)))
            torch.save(model.state_dict(), os.path.join(
                SNAPSHOTDIR, 'last.pkl'))

    return iteration

class Dataset():
    def __init__(self):
        ImageRoot = r'C:\Users\ASUS\Pose2Seg\data\coco2017\train2017'
        AnnoFile = r'C:\Users\ASUS\Pose2Seg\data\coco2017\annotations\person_keypoints_train2017_pose2seg.json'
        self.datainfos = CocoDatasetInfo(
            ImageRoot, AnnoFile, onlyperson=True, loadimg=True)

    def __len__(self):
        return len(self.datainfos)

    def __getitem__(self, idx):
        rawdata = self.datainfos[idx]
        img = rawdata['data']
        image_id = rawdata['id']

        height, width = img.shape[0:2]
        gt_kpts = np.float32(rawdata['gt_keypoints']).transpose(
            0, 2, 1)  # (N, 17, 3)
        gt_segms = rawdata['segms']
        gt_masks = np.array([annToMask(segm, height, width)
                             for segm in gt_segms])

        return {'img': img, 'kpts': gt_kpts, 'masks': gt_masks}

    def collate_fn(self, batch):
        batchimgs = [data['img'] for data in batch]
        batchkpts = [data['kpts'] for data in batch]
        batchmasks = [data['masks'] for data in batch]
        return {'batchimgs': batchimgs, 'batchkpts': batchkpts, 'batchmasks': batchmasks}

if __name__ == '__main__':
    logger.info('===========> loading model <===========')
    model = Pose2Seg().cuda()
    # model.init("")
    model.train()

    logger.info('===========> loading data <===========')
    datasetTrain = Dataset()
    dataloaderTrain = torch.utils.data.DataLoader(datasetTrain, batch_size=1, shuffle=True,
                                                  num_workers=0, pin_memory=False,
                                                  collate_fn=datasetTrain.collate_fn)

    logger.info('===========> set optimizer <===========')
    ''' set your optimizer like this. Normally is Adam/SGD. '''
    #optimizer = torch.optim.SGD(model.parameters(), 0.0002, momentum=0.9, weight_decay=0.0005)
    optimizer = torch.optim.Adam(
        model.parameters(), 0.0002, weight_decay=0.0000)

    iteration = 0
    epoch = 0
    try:
        while iteration < 14150*25:
            logger.info('===========>   training    <===========')
            iteration = train(model, dataloaderTrain,
                              optimizer, epoch, iteration)
            epoch += 1

            logger.info('===========>   testing    <===========')
            test(model, dataset='cocoVal', logger=logger.info)
            test(model, dataset='OCHumanVal', logger=logger.info)

    except (KeyboardInterrupt):
        logger.info('Save ckpt on exception ...')
        torch.save(model.state_dict(), os.path.join(
            SNAPSHOTDIR, 'interrupt_%d_%d.pkl' % (epoch, iteration)))
        logger.info('Save ckpt done.')


title: Pytorch transforms.RandomRotation() does not work on Google Colab

body:
Normally i was working on letter&digit recognition on my computer and I wanted to move my project to Colab but unfortunately there was an error (you can see the error below).
after some debugging i found which line is giving me error.

transforms.RandomRotation(degrees=(90, -90))

below i wrote simple abstract code to show this error.This code does not work on colab but it works fine at my own computer environment.Problem might be about the different versions of pytorch library i have version 1.3.1 on my computer and colab uses version 1.4.0.

import torch
import torchvision
from torchvision import datasets, transforms
import matplotlib.pyplot as plt
    transformOpt = transforms.Compose([
            transforms.RandomRotation(degrees=(90, -90)),
            transforms.ToTensor()
        ])

    train_set = datasets.MNIST(
        root='', train=True, transform=transformOpt, download=True)
    test_set = datasets.MNIST(
        root='', train=False, transform=transformOpt, download=True)

    train_loader = torch.utils.data.DataLoader(
        dataset=train_set,
        batch_size=100,
        shuffle=True)
    test_loader = torch.utils.data.DataLoader(
        dataset=test_set,
        batch_size=100,
        shuffle=False)

    images, labels = next(iter(train_loader))
    plt.imshow(images[0].view(28, 28), cmap="gray")
    plt.show()

  The full error I got when I execute this sample code above on Google Colab.

TypeError                                 Traceback (most recent call last)

<ipython-input-1-8409db422154> in <module>()
     24     shuffle=False)
     25
---> 26 images, labels = next(iter(train_loader))
     27 plt.imshow(images[0].view(28, 28), cmap="gray")
     28 plt.show()

10 frames

/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py in __next__(self)
    343
    344     def __next__(self):
--> 345         data = self._next_data()
    346         self._num_yielded += 1
    347         if self._dataset_kind == _DatasetKind.Iterable and \

/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py in _next_data(self)
    383     def _next_data(self):
    384         index = self._next_index()  # may raise StopIteration
--> 385         data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
    386         if self._pin_memory:
    387             data = _utils.pin_memory.pin_memory(data)

/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py in fetch(self, possibly_batched_index)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---> 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py in <listcomp>(.0)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---> 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

/usr/local/lib/python3.6/dist-packages/torchvision/datasets/mnist.py in __getitem__(self, index)
     95
     96         if self.transform is not None:
---> 97             img = self.transform(img)
     98
     99         if self.target_transform is not None:

/usr/local/lib/python3.6/dist-packages/torchvision/transforms/transforms.py in __call__(self, img)
     68     def __call__(self, img):
     69         for t in self.transforms:
---> 70             img = t(img)
     71         return img
     72

/usr/local/lib/python3.6/dist-packages/torchvision/transforms/transforms.py in __call__(self, img)    1001         angle = self.get_params(self.degrees)    1002
-> 1003         return F.rotate(img, angle, self.resample, self.expand, self.center, self.fill)    1004     1005     def
__repr__(self):

/usr/local/lib/python3.6/dist-packages/torchvision/transforms/functional.py in rotate(img, angle, resample, expand, center, fill)
    727         fill = tuple([fill] * 3)
    728
--> 729     return img.rotate(angle, resample, expand, center, fillcolor=fill)
    730
    731

/usr/local/lib/python3.6/dist-packages/PIL/Image.py in rotate(self, angle, resample, expand, center, translate, fillcolor)    2003         w, h = nw, nh    2004
-> 2005         return self.transform((w, h), AFFINE, matrix, resample, fillcolor=fillcolor)    2006     2007     def save(self,    fp, format=None, **params):

/usr/local/lib/python3.6/dist-packages/PIL/Image.py in transform(self, size, method, data, resample, fill, fillcolor)    2297             raise ValueError("missing method data")    2298
-> 2299         im = new(self.mode, size, fillcolor)    2300         if method == MESH:    2301             # list of quads

/usr/local/lib/python3.6/dist-packages/PIL/Image.py in new(mode, size, color)    2503         im.palette = ImagePalette.ImagePalette()    2504         color = im.palette.getcolor(color)
-> 2505     return im._new(core.fill(mode, size, color))    2506     2507

TypeError: function takes exactly 1 argument (3 given)


title: requires_grad relation to leaf nodes

body:
From the docs:

  requires_grad – Boolean indicating whether the Variable has been
  created by a subgraph containing any Variable, that requires it.  Can
  be changed only on leaf Variables

What does it mean by leaf nodes here? Are leaf nodes only the input nodes?
If it can be only changed at the leaf nodes, how can I freeze layers then?


title: PyTorch can't see GPU (torch.cuda.is_availble() returns False)

body:
I have a problem where
import torch
print(torch.cuda_is_available())

will print False, and I can't use the GPU available. I've tried it on conda environment, where I've installed the PyTorch version corresponding to the NVIDIA driver I have. I've also tried it in docker container, where I've done the same. I've tried both of these options on a remote server, but they both failed. I know that I've installed the correct driver versions because I've checked the version with nvcc --version before installing PyTorch, and I've checked the GPU connection with nvidia-smi which displays the GPUs on the machines correctly.
Also, I've checked this post and tried exporting CUDA_VISIBLE_DEVICES, but had no luck.
On the server I have NVIDIA V100 GPUs with CUDA version 10.0 (for conda environment) and version 10.2 on a docker container I've built. Any help or push in the right direction would be greatly appreciated. Thanks!


title: How to install torch audio on Windows 10 conda?

body:
In Anaconda Python 3.6.7 with PyTorch installed, on Windows 10, I do this sequence:

conda install -c conda-forge librosa
conda install -c groakat sox

then in a fresh download from https://github.com/pytorch/audio I do

python setup.py install

and it runs for a while and ends like this:

torchaudio/torch_sox.cpp(3): fatal error C1083: Cannot open include file: 'sox.h': No such file or directory
error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\Community\\VC\\Tools\\MSVC\\14.15.26726\\bin\\HostX86\\x64\\cl.exe' failed with exit status 2

I am trying to reproduce this OpenNMT-py speech training demo on Windows: http://opennmt.net/OpenNMT-py/speech2text.html


title: How to combine/stack tensors and combine dimensions in PyTorch?

body:
I need to combine 4 tensors, representing greyscale images, of size [1,84,84], into a stack of shape [4,84,84], representing four greyscale images with each image represented as a "channel" in tensor style CxWxH.

I am using PyTorch.

I've tried using torch.stack and torch.cat but if one of these is the solution, I am not having luck figuring out the correct prep/methodology to get my results.

Thank you for any help.

import torchvision.transforms as T

class ReplayBuffer:
    def __init__(self, buffersize, batchsize, framestack, device, nS):
        self.buffer = deque(maxlen=buffersize)
        self.phi = deque(maxlen=framestack)
        self.batchsize = batchsize
        self.device = device

        self._initialize_stack(nS)

    def get_stack(self):
        #t =  torch.cat(tuple(self.phi),dim=0)
        t =  torch.stack(tuple(self.phi),dim=0)
        return t

    def _initialize_stack(self, nS):
        while len(self.phi) < self.phi.maxlen:
            self.phi.append(torch.tensor([1,nS[1], nS[2]]))

a = ReplayBuffer(buffersize=50000, batchsize=64, framestack=4, device='cuda', nS=[1,84,84])
print(a.phi)
s = a.get_stack()
print(s, s.shape)

The above code returns:

print(a.phi)

deque([tensor([ 1, 84, 84]), tensor([ 1, 84, 84]), tensor([ 1, 84, 84]), tensor([ 1, 84, 84])], maxlen=4)

print(s, s.shape)

tensor([[ 1, 84, 84],
        [ 1, 84, 84],
        [ 1, 84, 84],
        [ 1, 84, 84]]) torch.Size([4, 3])

But what I would like is the return to simply be [4, 84, 84]. I suspect this is quite simple but it's escaping me.


title: How to write torch.device('cuda' if torch.cuda.is_available() else 'cpu') as a full if else statement?

body:
I'm a beginner to Pytorch and wanted to type this statement as a whole if else statement:-
torch.device('cuda' if torch.cuda.is_available() else 'cpu')

Can somebody help me?


title: undefined symbol: PySlice_AdjustIndices when importing PyTorch

body:
I am trying to use PyTorch, and I think there is some version of something that isn't lining up.

From what little I can suss out, it seems that there are some functions in the newest version of PyTorch (?) that can't be accessed on my system. I suspect it has something to do with Python version 3.6.1 as opposed to 3.6.0. But I can't figure it out. If anyone has any advice on what I can do to rectify this error:

/home/ubuntu/nbs/torch_utils.py in <module>()
----> 1 import torch
      2 import torch.nn as nn
      3 import torch.nn.parallel
      4 import torch.utils.data
      5 from torch import optim

/home/ubuntu/anaconda3/lib/python3.6/site-packages/torch/__init__.py in <module>()
     51 sys.setdlopenflags(_dl_flags.RTLD_GLOBAL | _dl_flags.RTLD_NOW)
     52
---> 53 from torch._C import *
     54
     55 __all__ += [name for name in dir(_C)

ImportError: /home/ubuntu/anaconda3/lib/python3.6/site-packages/torch/_C.cpython-36m-x86_64-linux-gnu.so: undefined symbol: PySlice_AdjustIndices

Here's the details on my system:

I am running Ubuntu on AWS, Ubuntu 16.04.2 LTS

my Cuda info is Cuda compilation tools, release 8.0, V8.0.61

I have Anaconda, conda 4.3.15

My python version is Python 3.6.0 :: Anaconda custom (64-bit)

Thanks.


title: Cropping a minibatch of images in Pytorch -- each image differently

body:
I have a tensor named input with dimensions 64x21x21. It is a minibatch of 64 images, each 21x21 pixels. I'd like to crop each image down to 11x11 pixels. So the output tensor I want would have dimensions 64x11x11.

I'd like to crop each image around a different "center pixel." The center pixels are given by a 2-dimensional long tensor named center with dimensions 64x2. For image i, center[i][0] gives the row index and center[i][1] gives the column index for the pixel that should be at the center in the output. We can assume that the center pixel is always at least 5 pixels away from the border.

Is there an efficient way to do this in pytorch (on the gpu)?

UPDATE: Let me clarify that the center tensor is formed by a deep neural network. It acts as a "hard attention mechanism," to use the reinforcement learning term for it. After I "crop" an image, that subimage becomes the input to another neural network. That's why I want to do the cropping in Pytorch: because the operations before and after the cropping are in Pytorch. I'd like to avoid having to transfer anything from the GPU back to the CPU.


title: Overflow when unpacking long - Pytorch

body:
I am running the following code

import torch
from __future__ import print_function
x = torch.empty(5, 3)
print(x)

on an Ubuntu machine in CPU mode, which gives me following error, what would be the reason and how to fix

      x = torch.empty(5, 3)
----> print(x)

/usr/local/lib/python3.6/dist-packages/torch/tensor.py in __repr__(self)
     55         # characters to replace unicode characters with.
     56         if sys.version_info > (3,):
---> 57             return torch._tensor_str._str(self)
     58         else:
     59             if hasattr(sys.stdout, 'encoding'):

/usr/local/lib/python3.6/dist-packages/torch/_tensor_str.py in _str(self)
    216             suffix = ', dtype=' + str(self.dtype) + suffix
    217
--> 218         fmt, scale, sz = _number_format(self)
    219         if scale != 1:
    220             prefix = prefix + SCALE_FORMAT.format(scale) + ' ' * indent

/usr/local/lib/python3.6/dist-packages/torch/_tensor_str.py in _number_format(tensor, min_sz)
     94     # TODO: use fmod?
     95     for value in tensor:
---> 96         if value != math.ceil(value.item()):
     97             int_mode = False
     98             break

RuntimeError: Overflow when unpacking long


title: PyTorch running out of memory: DefaultCPUAllocator can't allocate memory

body:
I'm trying to optimize some weighs (weigts) in Pytorch but I keep getting this error:

RuntimeError: [enforce fail at CPUAllocator.cpp:64] . DefaultCPUAllocator: can't allocate memory: you tried to allocate 8000000000000 bytes. Error code 12 (Cannot allocate memory).

Namely, things blow up when I run (weights * col).sum() / weights.sum(). Weights is a tensor of size (1000000,1) and col is also a tensor of size (1000000, 1). Both tensors are decently sized, but it seems odd that I'm using up all the memory in my computer (8GB) for these operations.


title: how to deploy pytorch neural network for production code in Unity

body: I have a neural network trained in pytorch that I'd like to deploy into a Unity app. What's the best way to do it? I'm also interested in allowing the user to further train the neural network in the Unity app, which I guess would require to integrate some part of pytorch into Unity (maybe there's a way to integrate pytorch's C++ / torchscript API with Unity?). If anybody has experience with this, I'd like to know what the best alternatives are.


title: Hidden size vs input size in RNN

body:
Premise 1:

Regarding neurons in a RNN layer - it is my understanding that at "each time step, every neuron receives both the input vector x (t) and the output vector from the previous time step y (t –1)" [1]:

Premise 2:

It is also my understanding that in Pytorch's GRU layer, input_size and hidden_size mean the following:

  input_size – The number of expected features in the input x
  hidden_size – The number of features in the hidden state h

So naturally, hidden_size should represent the number of neurons in a GRU layer.

My question:

Given the following GRU layer:

# assume that hidden_size = 3

class Encoder(nn.Module):
    def __init__(self, src_dictionary_size, hidden_size):
        super(Encoder, self).__init__()
        self.embedding = nn.Embedding(src_dictionary_size, hidden_size)
        self.gru = nn.GRU(input_size = hidden_size, hidden_size = hidden_size)

Assuming a hidden_size of 3, my understanding is that the GRU layer above would have 3 neurons, each which accepts an input vector of size 3 simultaneously for every timestep.

My question is: why do the arguments to hidden_size and input_size have to be equal? I.e. why can't each of the 3 neurons accept say, an input vector of size 5?

Case in point: both of the following produce size mismatch:

self.gru = nn.GRU(input_size = hidden_size, hidden_size = hidden_size-1)
self.gru = nn.GRU(input_size = hidden_size, hidden_size = hidden_size+1)

[1] Géron, Aurélien. Hands-On Machine Learning with Scikit-Learn and TensorFlow (p. 388). O'Reilly Media. Kindle Edition.

[3] https://pytorch.org/docs/stable/nn.html#torch.nn.GRU

Adding full code for reproducibility:

import torch
import torch.nn as nn

class Encoder(nn.Module):
    def __init__(self, src_dictionary_size, hidden_size):
        super(Encoder, self).__init__()
        self.hidden_size = hidden_size
        self.embedding = nn.Embedding(src_dictionary_size, hidden_size)
        self.gru = nn.GRU(input_size = hidden_size, hidden_size = hidden_size-1)

    def forward(self, pad_seqs, seq_lengths, hidden):
        """
        Args:
          pad_seqs of shape (max_seq_length, batch_size, 1): Padded source sequences.
          seq_lengths: List of sequence lengths.
          hidden of shape (1, batch_size, hidden_size): Initial states of the GRU.

        Returns:
          outputs of shape (max_seq_length, batch_size, hidden_size): Padded outputs of GRU at every step.
          hidden of shape (1, batch_size, hidden_size): Updated states of the GRU.
        """
        embedded_sqs = self.embedding(pad_seqs).squeeze(2)
        packed_sqs = pack_padded_sequence(embedded_sqs, seq_lengths)
        packed_output, h_n = self.gru(packed_sqs, hidden)
        output, input_sizes = pad_packed_sequence(packed_output)

        return output, h_n

    def init_hidden(self, batch_size=1):
        return torch.zeros(1, batch_size, self.hidden_size)

def test_Encoder_shapes():
    hidden_size = 5
    encoder = Encoder(src_dictionary_size=5, hidden_size=hidden_size)

    # maximum word count
    max_seq_length = 4

    # num sentences
    batch_size = 2
    hidden = encoder.init_hidden(batch_size=batch_size)

    # these are padded sequences (sentences of words). There are 2 sentences (i.e. 2 batches) with a maximum of 4 words.
    pad_seqs = torch.tensor([
        [1, 2],
        [2, 3],
        [3, 0],
        [4, 0]
    ]).view(max_seq_length, batch_size, 1)

    outputs, new_hidden = encoder.forward(pad_seqs=pad_seqs, seq_lengths=[4, 2], hidden=hidden)
    assert outputs.shape == torch.Size([4, batch_size, hidden_size]), f"Bad outputs.shape: {outputs.shape}"
    assert new_hidden.shape == torch.Size([1, batch_size, hidden_size]), f"Bad new_hidden.shape: {new_hidden.shape}"
    print('Success')

test_Encoder_shapes()


title: How should I save the model of PyTorch if I want it loadable by OpenCV dnn module

body:
I train a simple classification model by PyTorch and load it by opencv3.3, but it throw exception and say

  OpenCV Error: The function/feature is not implemented (Unsupported Lua type) in readObject, file
  /home/ramsus/Qt/3rdLibs/opencv/modules/dnn/src/torch/torch_importer.cpp,
  line 797
  /home/ramsus/Qt/3rdLibs/opencv/modules/dnn/src/torch/torch_importer.cpp:797:
  error: (-213) Unsupported Lua type in function readObject

Model definition

class conv_block(nn.Module):
    def __init__(self, in_filter, out_filter, kernel):
        super(conv_block, self).__init__()

        self.conv1 = nn.Conv2d(in_filter, out_filter, kernel, 1, (kernel - 1)//2)
        self.batchnorm = nn.BatchNorm2d(out_filter)
        self.maxpool = nn.MaxPool2d(2, 2)

    def forward(self, x):
        x = self.conv1(x)
        x = self.batchnorm(x)
        x = F.relu(x)
        x = self.maxpool(x)

        return x

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()

        self.conv1 = conv_block(3, 6, 3)
        self.conv2 = conv_block(6, 16, 3)
        self.fc1 = nn.Linear(16 * 8 * 8, 120)
        self.bn1 = nn.BatchNorm1d(120)
        self.fc2 = nn.Linear(120, 84)
        self.bn2 = nn.BatchNorm1d(84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.conv1(x)
        x = self.conv2(x)
        x = x.view(x.size()[0], -1)
        x = F.relu(self.bn1(self.fc1(x)))
        x = F.relu(self.bn2(self.fc2(x)))
        x = self.fc3(x)
        return x

This model use Conv2d, ReLU, BatchNorm2d, MaxPool2d and Linear layer only, every layers are supported by opencv3.3

I save it by state_dict

torch.save(net.state_dict(), 'cifar10_model')

Load it by c++ as

std::string const model_file("/home/some_folder/cifar10_model");

std::cout<<"read net from torch"<<std::endl;
dnn::Net net = dnn::readNetFromTorch(model_file);

I guess I save the model with the wrong way, what is the proper way to save the model of PyTorch in order to load using OpenCV? Thanks

Edit :

I use another way to save the model, but it cannot be loaded either

torch.save(net, 'cifar10_model.net')

Is this a bug?Or I am doing something wrong?


title: IDE autocomplete for pytorch

body:
I'm using Visual Studio Code. Recently tried out Kite. Both of which doesn't seem to have autocomplete for pytorch.

Is is possible with these tools ? If not, can someone suggest an editor that does ?

Thank you!


title: In pytorch, how to use the weight parameter in F.cross_entropy()?

body:
I'm trying to write some code like below:

x = Variable(torch.Tensor([[1.0,2.0,3.0]]))
y = Variable(torch.LongTensor([1]))
w = torch.Tensor([1.0,1.0,1.0])
F.cross_entropy(x,y,w)
w = torch.Tensor([1.0,10.0,1.0])
F.cross_entropy(x,y,w)

However, the output of cross entropy loss is always 1.4076 whatever w is. What is behind the weight parameter for F.cross_entropy()? How to use it correctly?
I'm using pytorch 0.3