title: PyTorch: Speed up data loading

body:
I am using densenet121 to do cat/dog detection from Kaggle dataset. I enabled cuda and it appears that training is very fast. However, the data loading (or perhaps processing) appears to be very slow. Are there some ways to speed it up? I tried to play witch batch size, that didn't provide much help. I also changed num_workers from 0 to some positive numbers. Going from 0 to 2 reduces loading time by perhaps 1/3, increasing by more doesn't have additional effect. Are there some other ways I can speed loading things up?

This is my rough code (I am focused on learning, so it's not very organized):

import matplotlib.pyplot as plt

import torch
from torch import nn
from torch import optim
import torch.nn.functional as F
from torchvision import datasets, transforms, models

data_dir = 'Cat_Dog_data'

train_transforms = transforms.Compose([transforms.RandomRotation(30),
                                       transforms.RandomResizedCrop(224),
                                       transforms.RandomHorizontalFlip(),
                                       transforms.ToTensor(),
                                       transforms.Normalize([0.5, 0.5, 0.5],
                                                            [0.5, 0.5, 0.5])])
test_transforms = transforms.Compose([transforms.Resize(255),
                                      transforms.CenterCrop(224),
                                      transforms.ToTensor()])

# Pass transforms in here, then run the next cell to see how the transforms look
train_data = datasets.ImageFolder(data_dir + '/train',
                                  transform=train_transforms)
test_data = datasets.ImageFolder(data_dir + '/test', transform=test_transforms)

trainloader = torch.utils.data.DataLoader(train_data, batch_size=64,
                                          num_workers=16, shuffle=True,
                                          pin_memory=True)
testloader = torch.utils.data.DataLoader(test_data, batch_size=64,
                                         num_workers=16)

model = models.densenet121(pretrained=True)

# Freeze parameters so we don't backprop through them
for param in model.parameters():
    param.requires_grad = False

from collections import OrderedDict

classifier = nn.Sequential(OrderedDict([
    ('fc1', nn.Linear(1024, 500)),
    ('relu', nn.ReLU()),
    ('fc2', nn.Linear(500, 2)),
    ('output', nn.LogSoftmax(dim=1))
]))

model.classifier = classifier
model.cuda()
criterion = nn.NLLLoss()
optimizer = optim.Adam(model.parameters(), lr=0.003)

epochs = 30
steps = 0

import time

device = torch.device('cuda:0')

train_losses, test_losses = [], []
for e in range(epochs):
    running_loss = 0
    count = 0
    total_start = time.time()
    for images, labels in trainloader:
        start = time.time()
        images = images.cuda()
        labels = labels.cuda()

        optimizer.zero_grad()

        log_ps = model(images)
        loss = criterion(log_ps, labels)
        loss.backward()
        optimizer.step()
        elapsed = time.time() - start

        if count % 20 == 0:
            print("Optimized elapsed: ", elapsed, "count:", count)
            print("Total elapsed ", time.time() - total_start)
            total_start = time.time()
        count += 1

        running_loss += loss.item()
    else:
        test_loss = 0
        accuracy = 0
        for images, labels in testloader:
            images = images.cuda()
            labels = labels.cuda()
            with torch.no_grad():
                model.eval()
                log_ps = model(images)
                test_loss += criterion(log_ps, labels)
                ps = torch.exp(log_ps)
                top_p, top_class = ps.topk(1, dim=1)
                compare = top_class == labels.view(*top_class.shape)
                accuracy += compare.type(torch.FloatTensor).mean()
        model.train()
        train_losses.append(running_loss / len(trainloader))
        test_losses.append(test_loss / len(testloader))

        print("Epoch: {}/{}.. ".format(e + 1, epochs),
              "Training Loss: {:.3f}.. ".format(
                  running_loss / len(trainloader)),
              "Test Loss: {:.3f}.. ".format(test_loss / len(testloader)),
              "Test Accuracy: {:.3f}".format(accuracy / len(testloader)))


title: Pytorch RuntimeError: expected scalar type Float but found Byte

body:
I am working on the classic example with digits. I want to create a my first neural network that predict the labels of digit images {0,1,2,3,4,5,6,7,8,9}. So the first column of train.txt has the labels and all the other columns are the features of each label. I have defined a class to import my data:
class DigitDataset(Dataset):
    """Digit dataset."""

    def __init__(self, file_path, transform=None):
        """
        Args:
            csv_file (string): Path to the csv file with annotations.
            root_dir (string): Directory with all the images.
            transform (callable, optional): Optional transform to be applied
                on a sample.
        """
        self.data = pd.read_csv(file_path, header = None, sep =" ")
        self.transform = transform

    def __len__(self):
        return len(self.data)

    def __getitem__(self, idx):
        if torch.is_tensor(idx):
            idx = idx.tolist()

        labels = self.data.iloc[idx,0]
        images = self.data.iloc[idx,1:-1].values.astype(np.uint8).reshape((1,16,16))

        if self.transform is not None:
            sample = self.transform(sample)
        return images, labels

And then I am running these commands to split my dataset into batches, to define a model and a loss:
train_dataset = DigitDataset("train.txt")
train_loader = DataLoader(train_dataset, batch_size=64,
                        shuffle=True, num_workers=4)

# Model creation with neural net Sequential model
model=nn.Sequential(nn.Linear(256, 128), # 1 layer:- 256 input 128 o/p
                    nn.ReLU(),          # Defining Regular linear unit as activation
                    nn.Linear(128,64),  # 2 Layer:- 128 Input and 64 O/p
                    nn.Tanh(),          # Defining Regular linear unit as activation
                    nn.Linear(64,10),   # 3 Layer:- 64 Input and 10 O/P as (0-9)
                    nn.LogSoftmax(dim=1) # Defining the log softmax to find the probablities
for the last output unit
                  )

# defining the negative log-likelihood loss for calculating loss
criterion = nn.NLLLoss()

images, labels = next(iter(train_loader))
images = images.view(images.shape[0], -1)

logps = model(images) #log probabilities
loss = criterion(logps, labels) #calculate the NLL-loss

And I take the error:
---------------------------------------------------------------------------
   RuntimeError                              Traceback (most recent call last)
    <ipython-input-2-7f4160c1f086> in <module>
     47 images = images.view(images.shape[0], -1)
     48
---> 49 logps = model(images) #log probabilities
     50 loss = criterion(logps, labels) #calculate the NLL-loss

~/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self,
*input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

~/anaconda3/lib/python3.8/site-packages/torch/nn/modules/container.py in forward(self, input)
    115     def forward(self, input):
    116         for module in self:
--> 117             input = module(input)
    118         return input
    119

~/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self,
*input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

 ~/anaconda3/lib/python3.8/site-packages/torch/nn/modules/linear.py in forward(self, input)
     91
     92     def forward(self, input: Tensor) -> Tensor:
---> 93         return F.linear(input, self.weight, self.bias)
     94
     95     def extra_repr(self) -> str:

 ~/anaconda3/lib/python3.8/site-packages/torch/nn/functional.py in linear(input, weight, bias)
   1688     if input.dim() == 2 and bias is not None:
   1689         # fused op is marginally faster
-> 1690         ret = torch.addmm(bias, input, weight.t())
   1691     else:
   1692         output = input.matmul(weight.t())

RuntimeError: expected scalar type Float but found Byte

Do you know what is wrong? Thank you for your patience and help!


title: Unpickling saved pytorch model throws AttributeError: Can't get attribute 'Net' on <module '__main__' despite adding class definition inline

body:
I'm trying to serve a pytorch model in a flask app. This code was working when I ran this on a jupyter notebook earlier but now I'm running this within a virtual env and apparently it can't get attribute 'Net' even though the class definition is right there. All the other similar questions tell me to add the class definition of the saved model in the same script. But it still doesn't work. The torch version is 1.0.1 (where the saved model was trained as well as the virtualenv)
What am I doing wrong?
Here's my code.

import os
import numpy as np
from flask import Flask, request, jsonify
import requests

import torch
from torch import nn
from torch.nn import functional as F

MODEL_URL = 'https://storage.googleapis.com/judy-pytorch-model/classifier.pt'

r = requests.get(MODEL_URL)
file = open("model.pth", "wb")
file.write(r.content)
file.close()

class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(input_size, hidden_size)
        self.fc2 = nn.Linear(hidden_size, hidden_size)
        self.fc3 = nn.Linear(hidden_size, output_size)

    def forward(self, x):
        x = torch.sigmoid(self.fc1(x))
        x = torch.sigmoid(self.fc2(x))
        x = self.fc3(x)

        return F.log_softmax(x, dim=-1)

model = torch.load('model.pth')

app = Flask(__name__)

@app.route("/")
def hello():
    return "Binary classification example\n"

@app.route('/predict', methods=['GET'])
def predict():

    x_data = request.args['x_data']

    x_data =  x_data.split()
    x_data = list(map(float, x_data))

    sample = np.array(x_data)

    sample_tensor = torch.from_numpy(sample).float()

    out = model(sample_tensor)

    _, predicted = torch.max(out.data, -1)

    if predicted.item() == 0:
         pred_class = "Has no liver damage - ", predicted.item()
    elif predicted.item() == 1:
        pred_class = "Has liver damage - ", predicted.item()

    return jsonify(pred_class)

Here's the full traceback:

Traceback (most recent call last):
  File "/Users/judyraj/Judy/pytorch-deployment/flask_app/liver_disease_finder/bin/flask", line 10, in <module>
    sys.exit(main())
  File "/Users/judyraj/Judy/pytorch-deployment/flask_app/liver_disease_finder/lib/python3.6/site-packages/flask/cli.py", line 894, in main
    cli.main(args=args, prog_name=name)
  File "/Users/judyraj/Judy/pytorch-deployment/flask_app/liver_disease_finder/lib/python3.6/site-packages/flask/cli.py", line 557, in main
    return super(FlaskGroup, self).main(*args, **kwargs)
  File "/Users/judyraj/Judy/pytorch-deployment/flask_app/liver_disease_finder/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/Users/judyraj/Judy/pytorch-deployment/flask_app/liver_disease_finder/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/judyraj/Judy/pytorch-deployment/flask_app/liver_disease_finder/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/judyraj/Judy/pytorch-deployment/flask_app/liver_disease_finder/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/Users/judyraj/Judy/pytorch-deployment/flask_app/liver_disease_finder/lib/python3.6/site-packages/click/decorators.py", line 64, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/Users/judyraj/Judy/pytorch-deployment/flask_app/liver_disease_finder/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/Users/judyraj/Judy/pytorch-deployment/flask_app/liver_disease_finder/lib/python3.6/site-packages/flask/cli.py", line 767, in run_command
    app = DispatchingApp(info.load_app, use_eager_loading=eager_loading)
  File "/Users/judyraj/Judy/pytorch-deployment/flask_app/liver_disease_finder/lib/python3.6/site-packages/flask/cli.py", line 293, in __init__
    self._load_unlocked()
  File "/Users/judyraj/Judy/pytorch-deployment/flask_app/liver_disease_finder/lib/python3.6/site-packages/flask/cli.py", line 317, in _load_unlocked
    self._app = rv = self.loader()
  File "/Users/judyraj/Judy/pytorch-deployment/flask_app/liver_disease_finder/lib/python3.6/site-packages/flask/cli.py", line 372, in load_app
    app = locate_app(self, import_name, name)
  File "/Users/judyraj/Judy/pytorch-deployment/flask_app/liver_disease_finder/lib/python3.6/site-packages/flask/cli.py", line 235, in locate_app
    __import__(module_name)
  File "/Users/judyraj/Judy/pytorch-deployment/flask_app/app.py", line 34, in <module>
    model = torch.load('model.pth')
  File "/Users/judyraj/Judy/pytorch-deployment/flask_app/liver_disease_finder/lib/python3.6/site-packages/torch/serialization.py", line 368, in load
    return _load(f, map_location, pickle_module)
  File "/Users/judyraj/Judy/pytorch-deployment/flask_app/liver_disease_finder/lib/python3.6/site-packages/torch/serialization.py", line 542, in _load
    result = unpickler.load()
AttributeError: Can't get attribute 'Net' on <module '__main__' from '/Users/judyraj/Judy/pytorch-deployment/flask_app/liver_disease_finder/bin/flask'>

This doesn't solve my issue. I do not want to change the way I persist the model. torch.save() worked fine for me outside the virtual env. I don't mind adding the class definition to the script. I'm trying to see what's causing the error despite that.


title: How to multiply a tensor row-wise by a vector in PyTorch?

body: When I have a tensor m of shape [12, 10] and a vector s of scalars with shape [12], how can I multiply each row of m with the corresponding scalar in s?


title: How downsample work in ResNet in pytorch code?

body:
In this pytorch ResNet code example they define downsample as variable in line 44. and line 58 use it as function. How this downsample work here as CNN point of view and as python Code point of view.

code example : pytorch ResNet

i searched for if downsample is any pytorch inbuilt function. but it is not.

class BasicBlock(nn.Module):
    expansion = 1

    def __init__(self, inplanes, planes, stride=1, downsample=None, groups=1, norm_layer=None):
        super(BasicBlock, self).__init__()
        if norm_layer is None:
            norm_layer = nn.BatchNorm2d
        if groups != 1:
            raise ValueError('BasicBlock only supports groups=1')
        # Both self.conv1 and self.downsample layers downsample the input when stride != 1
        self.conv1 = conv3x3(inplanes, planes, stride)
        self.bn1 = norm_layer(planes)
        self.relu = nn.ReLU(inplace=True)
        self.conv2 = conv3x3(planes, planes)
        self.bn2 = norm_layer(planes)
        self.downsample = downsample
        self.stride = stride

    def forward(self, x):
        identity = x

        out = self.conv1(x)
        out = self.bn1(out)
        out = self.relu(out)

        out = self.conv2(out)
        out = self.bn2(out)

        if self.downsample is not None:
            identity = self.downsample(x)

        out += identity
        out = self.relu(out)

return out


title: Gaussian filter in PyTorch

body:
I am looking for a way to apply a Gaussian filter to an image (tensor) only using PyTorch functions. Using numpy, the equivalent code is

import numpy as np
from scipy import signal
import matplotlib.pyplot as plt

# Define 2D Gaussian kernel
def gkern(kernlen=256, std=128):
    """Returns a 2D Gaussian kernel array."""
    gkern1d = signal.gaussian(kernlen, std=std).reshape(kernlen, 1)
    gkern2d = np.outer(gkern1d, gkern1d)
    return gkern2d

# Generate random matrix and multiply the kernel by it
A = np.random.rand(256*256).reshape([256,256])

# Test plot
plt.figure()
plt.imshow(A*gkern(256, std=32))
plt.show()

The closest suggestion I found is based on this post:

import torch.nn as nn

conv = nn.Conv2d(in_channels = 1, out_channels = 1, kernel_size=264, bias=False)
with torch.no_grad():
    conv.weight = gaussian_weights

But it gives me the error NameError: name 'gaussian_weights' is not defined. How can I make it work?


title: RuntimeError: CUDA out of memory. Problem when re-loading the model in a loop

body:
I am running into the classic: CUDA out of memory.

What I want to do: I want to load the same model using a different matrix of embeddings every time. I have to do that 300 times, one for each dimension of the word embeddings.

I am not training the model, that is why I am using model.eval(), I thought that would be enough to keep Pytorch from creating a graph.

Please notice that I never pass the model, nor the data, to cuda. In fact, I wanted to debug the code using cpu before sending the code to be performed by a GPU.

The loop below is executed once, a RuntimeError is raised in the second iteration.

My guess is that the code is loading a new model into a GPU memory at each iteration (which I did not know was possible without explicitly pointing it to do so). The emb_matrix is quite heavy and could cause the GPU memory to crash.

emb_dim = 300
acc_dim = torch.zeros((emb_dim, 4))
for d in range(emb_dim):

    #create embeddings with one dimension shuffled
    emb_matrix = text_f.vocab.vectors.clone()

    #get a random permutation across one of the dimensions
    rand_index = torch.randperm(text_f.vocab.vectors.shape[0])
    emb_matrix[:, d] =  text_f.vocab.vectors[rand_index, d]

    #load model with the scrumbled embeddings
    model = load_classifier(emb_matrix,
                            encoder_type = encoder_type)
    model.eval()
    for batch in batch_iters["test"]:
        x_pre = batch.premise
        x_hyp = batch.hypothesis
        y = batch.label

        #perform forward pass
        y_pred = model.forward(x_pre, x_hyp)

        #calculate accuracies
        acc_dim[d] += accuracy(y_pred, y)/test_batches

        #avoid memory issues
        y_pred.detach()

    print(f"Dimension {d} accuracies: {acc_dim[d]}")

I get the following error:
RuntimeError: CUDA out of memory. Tried to allocate 146.88 MiB (GPU 0; 2.00 GiB total capacity; 374.63 MiB already allocated; 0 bytes free; 1015.00 KiB cached)

I tried passing the model and the data to CPU, but I get precisely the same error.

I looked around for how to fix the problem, but I could not find an obvious solution. Any suggestions on how to load the model and data in the correct place, or how to clean the GPU's memory after each iteration are welcome.


title: `for` loop to a multi dimensional array in PyTorch

body:
I want to implement Q&A systems with attention mechanism. I have two inputs; context and query which shapes are (batch_size, context_seq_len, embd_size) and (batch_size,  query_seq_len, embd_size).
I am following the below paper.
Machine Comprehension Using Match-LSTM and Answer Pointer.   https://arxiv.org/abs/1608.07905

Then, I want to obtain a attention matrix which shape is (batch_size, context_seq_len, query_seq_len, embd_size). In the thesis, they calculate values for each row (it means each context word, G_i, alpha_i in the paper).

My code is below and it is running. But I am not sure my way is good or not. For example, I use for loop for generating sequence data (for i in range(T):). And to obtain each row, I use in-place operator like G[:,i,:,:], embd_context[:,i,:].clone() is a good manner in pytorch? If not, where should I change the code?

And if you notice other points, let me know. I am a new in this field and pytorch. Sorry for my ambiguous question.

class MatchLSTM(nn.Module):
    def __init__(self, args):
        super(MatchLSTM, self).__init__()
        self.embd_size = args.embd_size
        d = self.embd_size
        self.answer_token_len = args.answer_token_len

        self.embd = WordEmbedding(args)
        self.ctx_rnn   = nn.GRU(d, d, dropout = 0.2)
        self.query_rnn = nn.GRU(d, d, dropout = 0.2)

        self.ptr_net = PointerNetwork(d, d, self.answer_token_len) # TBD

        self.w  = nn.Parameter(torch.rand(1, d, 1).type(torch.FloatTensor), requires_grad=True) # (1, 1, d)
        self.Wq = nn.Parameter(torch.rand(1, d, d).type(torch.FloatTensor), requires_grad=True) # (1, d, d)
        self.Wp = nn.Parameter(torch.rand(1, d, d).type(torch.FloatTensor), requires_grad=True) # (1, d, d)
        self.Wr = nn.Parameter(torch.rand(1, d, d).type(torch.FloatTensor), requires_grad=True) # (1, d, d)

        self.match_lstm_cell = nn.LSTMCell(2*d, d)

    def forward(self, context, query):
        # params
        d = self.embd_size
        bs = context.size(0) # batch size
        T = context.size(1)  # context length
        J = query.size(1)    # query length

        # LSTM Preprocessing Layer
        shape = (bs, T, J, d)
        embd_context     = self.embd(context)         # (N, T, d)
        embd_context, _h = self.ctx_rnn(embd_context) # (N, T, d)
        embd_context_ex  = embd_context.unsqueeze(2).expand(shape).contiguous() # (N, T, J, d)
        embd_query       = self.embd(query)           # (N, J, d)
        embd_query, _h   = self.query_rnn(embd_query) # (N, J, d)
        embd_query_ex  = embd_query.unsqueeze(1).expand(shape).contiguous() # (N, T, J, d)

        # Match-LSTM layer
        G = to_var(torch.zeros(bs, T, J, d)) # (N, T, J, d)

        wh_q = torch.bmm(embd_query, self.Wq.expand(bs, d, d)) # (N, J, d) = (N, J, d)(N, d, d)

        hidden     = to_var(torch.randn([bs, d])) # (N, d)
        cell_state = to_var(torch.randn([bs, d])) # (N, d)
        # TODO bidirectional
        H_r = [hidden]
        for i in range(T):
            wh_p_i = torch.bmm(embd_context[:,i,:].clone().unsqueeze(1), self.Wp.expand(bs, d, d)).squeeze() # (N, 1, d) -> (N, d)
            wh_r_i = torch.bmm(hidden.unsqueeze(1), self.Wr.expand(bs, d, d)).squeeze() # (N, 1, d) -> (N, d)
            sec_elm = (wh_p_i + wh_r_i).unsqueeze(1).expand(bs, J, d) # (N, J, d)

            G[:,i,:,:] = F.tanh( (wh_q + sec_elm).view(-1, d) ).view(bs, J, d) # (N, J, d) # TODO bias

            attn_i = torch.bmm(G[:,i,:,:].clone(), self.w.expand(bs, d, 1)).squeeze() # (N, J)
            attn_query = torch.bmm(attn_i.unsqueeze(1), embd_query).squeeze() # (N, d)
            z = torch.cat((embd_context[:,i,:], attn_query), 1) # (N, 2d)

            hidden, cell_state = self.match_lstm_cell(z, (hidden, cell_state)) # (N, d), (N, d)
            H_r.append(hidden)
        H_r = torch.stack(H_r, dim=1) # (N, T, d)

        indices = self.ptr_net(H_r) # (N, M, T) , M means (start, end)
        return indices


title: How to get around in place operation error if index leaf variable for gradient update?

body:
I am encountering In place operation error when I am trying to index a leaf variable to update gradients with customized Shrink function. I cannot work around it. Any help is highly appreciated!

import torch.nn as nn
import torch
import numpy as np
from torch.autograd import Variable, Function

# hyper parameters
batch_size = 100 # batch size of images
ld = 0.2 # sparse penalty
lr = 0.1 # learning rate

x = Variable(torch.from_numpy(np.random.normal(0,1,(batch_size,10,10))), requires_grad=False)  # original

# depends on size of the dictionary, number of atoms.
D = Variable(torch.from_numpy(np.random.normal(0,1,(500,10,10))), requires_grad=True)

# hx sparse representation
ht = Variable(torch.from_numpy(np.random.normal(0,1,(batch_size,500,1,1))), requires_grad=True)

# Dictionary loss function
loss = nn.MSELoss()

# customized shrink function to update gradient
shrink_ht = lambda x: torch.stack([torch.sign(i)*torch.max(torch.abs(i)-lr*ld,0)[0] for i in x])

### sparse reprsentation optimizer_ht single image.
optimizer_ht = torch.optim.SGD([ht], lr=lr, momentum=0.9) # optimizer for sparse representation

## update for the batch
for idx in range(len(x)):
    optimizer_ht.zero_grad() # clear up gradients
    loss_ht = 0.5*torch.norm((x[idx]-(D*ht[idx]).sum(dim=0)),p=2)**2
    loss_ht.backward() # back propogation and calculate gradients
    optimizer_ht.step() # update parameters with gradients
    ht[idx] = shrink_ht(ht[idx])  # customized shrink function.

RuntimeError Traceback (most recent call last) in ()
15 loss_ht.backward() # back propogation and calculate gradients
16 optimizer_ht.step() # update parameters with gradients
—> 17 ht[idx] = shrink_ht(ht[idx]) # customized shrink function.
18
19

/home/miniconda3/lib/python3.6/site-packages/torch/autograd/variable.py in setitem(self, key, value)
85 return MaskedFill.apply(self, key, value, True)
86 else:
—> 87 return SetItem.apply(self, key, value)
88
89 def deepcopy(self, memo):

RuntimeError: a leaf Variable that requires grad has been used in an in-place operation.

Specifically, this line of code below seems give error as it index and update leaf variable at the same time.

ht[idx] = shrink_ht(ht[idx])  # customized shrink function.

Thanks.

W.S.


title: Why detach needs to be called on variable in this example?

body:
I was going through this example - https://github.com/pytorch/examples/blob/master/dcgan/main.py and I have a basic question.

fake = netG(noise)
label = Variable(label.fill_(fake_label))
output = netD(fake.detach()) # detach to avoid training G on these labels
errD_fake = criterion(output, label)
errD_fake.backward()
D_G_z1 = output.data.mean()
errD = errD_real + errD_fake
optimizerD.step()

I understand that why we call detach() on variable fake, so that no gradients are computed for the Generator parameters. My question is, does it matter since optimizerD.step() is going to update the parameters associated with Discriminator only?

OptimizerD is defined as:
optimizerD = optim.Adam(netD.parameters(), lr=opt.lr, betas=(opt.beta1, 0.999))

Besides, in the next step when we will update parameters for Generator, before that we will call netG.zero_grad() which eventually removes all previously computed gradients. Moreover, when we update parameters for G network, we do this - output = netD(fake). Here, we are not using detach. Why?

So, why detaching the variable (line 3) is necessary in the above code?


title: PyTorch - RuntimeError: [enforce fail at inline_container.cc:209] . file not found: archive/data.pkl

body:
Problem
I'm trying to load a file using PyTorch, but the error states archive/data.pkl does not exist.
Code
import torch
cachefile = 'cacheddata.pth'
torch.load(cachefile)

Output
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-4-8edf1f27a4bd> in <module>
      1 import torch
      2 cachefile = 'cacheddata.pth'
----> 3 torch.load(cachefile)

~/opt/anaconda3/envs/matching/lib/python3.8/site-packages/torch/serialization.py in load(f, map_location, pickle_module, **pickle_load_args)
    582                     opened_file.seek(orig_position)
    583                     return torch.jit.load(opened_file)
--> 584                 return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
    585         return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
    586

~/opt/anaconda3/envs/matching/lib/python3.8/site-packages/torch/serialization.py in _load(zip_file, map_location, pickle_module, **pickle_load_args)
    837
    838     # Load the data (which may in turn use `persistent_load` to load tensors)
--> 839     data_file = io.BytesIO(zip_file.get_record('data.pkl'))
    840     unpickler = pickle_module.Unpickler(data_file, **pickle_load_args)
    841     unpickler.persistent_load = persistent_load

RuntimeError: [enforce fail at inline_container.cc:209] . file not found: archive/data.pkl

Hypothesis
I'm guessing this has something to do with pickle, from the docs:

This save/load process uses the most intuitive syntax and involves the
least amount of code. Saving a model in this way will save the entire
module using Python’s pickle module. The disadvantage of this approach
is that the serialized data is bound to the specific classes and the
exact directory structure used when the model is saved. The reason for
this is because pickle does not save the model class itself. Rather,
it saves a path to the file containing the class, which is used during
load time. Because of this, your code can break in various ways when
used in other projects or after refactors.

Versions

PyTorch version: 1.6.0
Python version: 3.8.0


title: How to use a Batchsampler within a Dataloader

body:
I have a need to use a BatchSampler within a pytorch DataLoader instead of calling __getitem__ of the dataset multiple times (remote dataset, each query is pricy). I cannot understand how to use the batchsampler with any given dataset.

e.g

class MyDataset(Dataset):

    def __init__(self, remote_ddf, ):
        self.ddf = remote_ddf

    def __len__(self):
        return len(self.ddf)

    def __getitem__(self, idx):
        return self.ddf[idx] --------> This is as expensive as a batch call

    def get_batch(self, batch_idx):
        return self.ddf[batch_idx]

my_loader = DataLoader(MyDataset(remote_ddf),
           batch_sampler=BatchSampler(Sampler(), batch_size=3))

The thing I do not understand, neither found any example online or in torch docs, is how do I use my get_batch function instead of the __getitem__ function.

Edit:
Following the answer of Szymon Maszke, this is what I tried and yet, \_\_get_item__ gets one index each call, instead of a list of size batch_size

class Dataset(Dataset):

    def __init__(self):
       ...

    def __len__(self):
        ...

    def __getitem__(self, batch_idx):  ------> here I get only one index
        return self.wiki_df.loc[batch_idx]

loader = DataLoader(
                dataset=dataset,
                batch_sampler=BatchSampler(
                    SequentialSampler(dataset), batch_size=self.hparams.batch_size, drop_last=False),
                num_workers=self.hparams.num_data_workers,
            )


title: Speed up datasets loading on Google Colab

body:
I am working on Image classification on the German Traffic Sign Dataset on Google Colab with Pytorch.
Here is the structure of the dataset:

GTSRB

Training

00000/

*.ppmm

…
00043/

*.ppmm

Test

*.ppmm
…
labels.csv

I have managed to upload the whole dataset to my drive(it took a long time!!!).
I have used ImageFolder class and Dataset class to load respectively training and test images.

However, Training my model is really slow, and GPU is not used efficiently. After many searches, I discovered that file transfer from drive to Colab is at fault here.

Does anyone know how I can use hd5 dataset (or others techniques) to first store all training and test images for later preprocessing?


title: Convert CUDA tensor to NumPy

body:
First of all, I tried those solutions: 1, 2, 3, and 4, but did not work for me.

After training and testing the neural network, I am trying to show some examples to verify my work. I named the method predict which I pass the image to it to predict for which class it belongs:

def predict(model, image_path, topk=5):
''' Predict the class (or classes) of an image using a trained deep learning model.
'''

output = process_image(image_path)
output.unsqueeze_(0)
output = output.cuda().float()

model.eval()

with torch.no_grad():
    score = model(output)
    prob, idxs = torch.topk(score, topk)

    # Convert indices to classes
    idxs = np.array(idxs)
    idx_to_class = {val:key for key, val in model.class_to_idx.items()}
    classes = [idx_to_class[idx] for idx in idxs[0]]

    # Map the class name with collected topk classes
    names = []
    for cls in classes:
        names.append(cat_to_name[str(cls)])

    return prob, names

Then there is the final step which displays the final result based on the training of the neural network and done like this:

# TODO: Display an image along with the top 5 classes
x_pos, y_pos = predict(model, img_pil, topk=5)

ax_img = imshow(output)
ax_img.set_title(y_pos[0])

plt.figure(figsize=(4,4))
plt.barh(range(len(y_pos)), np.exp(x_pos[0]))
plt.yticks(range(len(y_pos)), y_pos)

plt.show()

The error is:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-45-e3f9951e9804> in <module>()
----> 1 x_pos, y_pos = predict(model, img_pil, topk=5)
      2
      3 ax_img = imshow(output)
      4 ax_img.set_title(y_pos[0])
      5

1 frames
<ipython-input-44-d77500f31561> in predict(model, image_path, topk)
     14
     15         # Convert indices to classes
---> 16         idxs = np.array(idxs)
     17         idx_to_class = {val:key for key, val in model.class_to_idx.items()}
     18         classes = [idx_to_class[idx] for idx in idxs[0]]

/usr/local/lib/python3.6/dist-packages/torch/tensor.py in __array__(self, dtype)
    456     def __array__(self, dtype=None):
    457         if dtype is None:
--> 458             return self.numpy()
    459         else:
    460             return self.numpy().astype(dtype, copy=False)

TypeError: can't convert CUDA tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.

How do I solve this?

I tried to change idx to idxs = idxs.cpu().numpy() and the error is:

TypeError                                 Traceback (most recent call last)
<ipython-input-62-e3f9951e9804> in <module>()
      5
      6 plt.figure(figsize=(4,4))
----> 7 plt.barh(range(len(y_pos)), np.exp(x_pos[0]))
      8 plt.yticks(range(len(y_pos)), y_pos)
      9

/usr/local/lib/python3.6/dist-packages/torch/tensor.py in __array__(self, dtype)
    456     def __array__(self, dtype=None):
    457         if dtype is None:
--> 458             return self.numpy()
    459         else:
    460             return self.numpy().astype(dtype, copy=False)

TypeError: can't convert CUDA tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.


title: Confused about tensor dimensions and batch sizes in pytorch

body:
So I'm very new to PyTorch and Neural Networks in general, and I'm having some problems creating a Neural Network that classifies names by gender.
I based this off of the PyTorch tutorial for RNNs that classify names by nationality, but I decided not to go with a recurrent approach... Stop me right here if this was the wrong idea!
However, whenever I try to run an input through the network it tells me:

RuntimeError: matrices expected, got 3D, 2D tensors at /py/conda-bld/pytorch_1493681908901/work/torch/lib/TH/generic/THTensorMath.c:1232

I know this has something to do with how PyTorch always expects there to be a batch size or something, and I have my tensor set up that way, but you can probably tell by this point that I have no idea what I'm talking about.
Here's my code:

from future import unicode_literals, print_function, division
from io import open
import glob
import unicodedata
import string
import torch
import torchvision
import torch.nn as nn
import torch.optim as optim
import random
from torch.autograd import Variable
import matplotlib.pyplot as plt
import matplotlib.ticker as ticker

"""------GLOBAL VARIABLES------"""

all_letters = string.ascii_letters + " .,;'"
num_letters = len(all_letters)
all_names = {}
genders = ["Female", "Male"]

"""-------DATA EXTRACTION------"""

def findFiles(path):
    return glob.glob(path)

def unicodeToAscii(s):
    return ''.join(
        c for c in unicodedata.normalize('NFD', s)
        if unicodedata.category(c) != 'Mn'
        and c in all_letters
    )

# Read a file and split into lines
def readLines(filename):
    lines = open(filename, encoding='utf-8').read().strip().split('\n')
    return [unicodeToAscii(line) for line in lines]

for file in findFiles("/home/andrew/PyCharm/PycharmProjects/CantStop/data/names/*.txt"):
    gender = file.split("/")[-1].split(".")[0]
    names = readLines(file)
    all_names[gender] = names

"""-----DATA INTERPRETATION-----"""

def nameToTensor(name):
    tensor = torch.zeros(len(name), 1, num_letters)
    for index, letter in enumerate(name):
        tensor[index][0][all_letters.find(letter)] = 1
    return tensor

def outputToGender(output):
    gender, gender_index = output.data.topk(1)
    if gender_index[0][0] == 0:
        return "Female"
    return "Male"

"""------NETWORK SETUP------"""

class Net(nn.Module):
    def __init__(self, input_size, output_size):
        super(Net, self).__init__()
        #Layer 1
        self.Lin1 = nn.Linear(input_size, int(input_size/2))
        self.ReLu1 = nn.ReLU()
        self.Batch1 = nn.BatchNorm1d(int(input_size/2))
        #Layer 2
        self.Lin2 = nn.Linear(int(input_size/2), output_size)
        self.ReLu2 = nn.ReLU()
        self.Batch2 = nn.BatchNorm1d(output_size)
        self.softMax = nn.LogSoftmax()

    def forward(self, input):
        output1 = self.Batch1(self.ReLu1(self.Lin1(input)))
        output2 = self.softMax(self.Batch2(self.ReLu2(self.Lin2(output1))))
        return output2

NN = Net(num_letters, 2)

"""------TRAINING------"""

def getRandomTrainingEx():
    gender = genders[random.randint(0, 1)]
    name = all_names[gender][random.randint(0, len(all_names[gender])-1)]
    gender_tensor = Variable(torch.LongTensor([genders.index(gender)]))
    name_tensor = Variable(nameToTensor(name))
    return gender_tensor, name_tensor, gender

def train(input, target):
    loss_func = nn.NLLLoss()

    optimizer = optim.SGD(NN.parameters(), lr=0.0001, momentum=0.9)

    optimizer.zero_grad()

    output = NN(input)

    loss = loss_func(output, target)
    loss.backward()
    optimizer.step()

    return output, loss

all_losses = []
current_loss = 0

for i in range(100000):
    gender_tensor, name_tensor, gender = getRandomTrainingEx()
    output, loss = train(name_tensor, gender_tensor)
    current_loss += loss

    if i%1000 == 0:
        print("Guess: %s, Correct: %s, Loss: %s" % (outputToGender(output), gender, loss.data[0]))

    if i%100 == 0:
        all_losses.append(current_loss/10)
        current_loss = 0

# plt.figure()
# plt.plot(all_losses)
# plt.show()

Please help a newbie out!


title: Anaconda Integration with Cuda 9.0 shows Incompatible Package Error

body:
I am trying to install CUDA 9.0  with NVIDIA-SMI: 445.75 in Windows 10.

My Cuda 9.0 installation is successful, as shown from Command-prompt

*(DL) C:\Users\User>nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2017 NVIDIA Corporation
Built on Fri_Sep__1_21:08:32_Central_Daylight_Time_2017
**Cuda compilation tools, release **9.0**, V9.0.176***

(1) I downloaded cudnn-9.0-windows10-x64-v7.zip, extracted it, and moved it to the fold, which was created when Cuda was installed.

(2) In the terminal prompt of the Anaconda,
I input conda install pytorch=1.1.0 torchvision=0.3.0 cudatoolkit=9.0 –c pytorch.

However, Anaconda prompt gives the following error

**Error messages**
*Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: |
Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
failed

UnsatisfiableError: The following specifications were found to be incompatible with the existing python installation in your environment:

Specifications:

  - pytorch=1.1.0 -> python[version='>=3.5,<3.6.0a0|>=3.6,<3.7.0a0|>=3.7,<3.8.0a0']
  - torchvision=0.3.0 -> python[version='>=3.5,<3.6.0a0|>=3.6,<3.7.0a0|>=3.7,<3.8.0a0']

Your python: python=3.8

If python is on the left-most side of the chain, that's the version you've asked for.
When python appears to the right, that indicates that the thing on the left is somehow not available for the python version you are constrained to. Note that conda will not change your python version to a different minor version unless you explicitly specify that.

The following specifications were found to be incompatible with each other:

Output in format: Requested package -> Available versions

Package cudatoolkit conflicts for:

    torchvision=0.3.0 -> cudatoolkit[version='>=10.0,<10.1|>=9.0,<9.1']
    pytorch=1.1.0 -> cudatoolkit[version='>=10.0,<10.1|>=9.0,<9.1']
    torchvision=0.3.0 -> pytorch[version='>=1.1.0'] -> cudatoolkit[version='>=10.1,<10.2|>=9.2,<9.3']The following specifications were found to be incompatible with your CUDA driver:

      - feature:/win-64::__cuda==11.0=0

    Your installed CUDA driver is: 11.0*


title: How to install PyTorch on Python 3.7 / Windows 10 with pip

body:
I am trying to install PyTorch 1.3 using pip :

pip3 install torch==1.3.1+cpu torchvision==0.4.2+cpu -f https://download.pytorch.org/whl/torch_stable.html

I got the command line from this page :
https://pytorch.org/get-started/locally/#start-locally

It fails, with the following error message :

ERROR: Could not find a version that satisfies the requirement
torch==1.3.1+cpu (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)
ERROR: No matching distribution found for torch==1.3.1+cpu

Using verbose mode (-v) give a little more info :

Skipping link: unsupported archive format: .html:
   https://download.pytorch.org/whl/torch_stable.html (from -f)
Analyzing links from page https://pypi.org/simple/torch/

It seems pip cannot use the url I provide and will get packages from another URL.
That one support at best Python 3.6. Since I have installed 3.7.5 (which is support by URL initially provided) none of the packages match and it fails.

I could downgrade to 3.6 but since 3.7 is officially supported I would like to try to fix that issue first.

EDIT : pip is up to date. I got exact same error on both Windows 10 and 7 64 bit machines.
Also: the PyTorch page says  :

  Please ensure that you have met the prerequisites below (e.g., numpy)

But AFAIK does not list any library in that section. During my tests, numpy was installed and up to date.

As requested, here is the full output of pip :
https://pastebin.com/8Wpqp6du


title: Is there a function to extract image patches in PyTorch?

body:
Given a batch of images, I'd like to extract all possible image patches, similar to a convolution. In TensorFlow, we can use tf.extract_image_patches to achieve this. Is there an equivalent function in PyTorch?

Thank you.


title: Fine Tuning Pretrained Model MobileNet_V2 in Pytorch

body:
I am new to pyTorch and I am trying to Create a Classifier where I have around 10 kinds of Images Folder Dataset, for this task I am using Pretrained model( MobileNet_v2 ) but the problem is I am not able to change the FC layer of it. There is not model.fc attribute.
Can anyone help me to do this.
Thanks


title: How can I visualize what happens during loss.backward()?

body:
I am confident in my understanding of the forward pass of my model, how can I control its backward pass?

This is not a theoretical question about what back-propagation is. The question is a practical one, about whether or not there are tools suited to visualize/track/control what happens during back-propagation.

Ideally, this tool would allow to visualize the structure of the computational graph of the model (a graph of the model's operations), its inputs and its trainable parameters.

Now, I do:

loss.backward()

and I would like to visualize what happens in that step.


title: best way of tqdm for data loader

body:
How to use tqdm for data_loader ?
is this the correct way?
for i,j in enumerate(data_loader,total = 100):
           pass


title: Pytorch: Sparse Matrix multiplcation

body:
Given:

self.A = torch.autograd.Variable(random_sparse(n = dim))
self.w = torch.autograd.Variable(torch.Tensor(np.random.normal(0,1,(dim,dim))))

Goal1:

torch.mm(self.A, self.w)

Goal2:

torch.spmm(self.A, self.w)

Result1:

TypeError: Type torch.sparse.FloatTensor doesn't implement stateless method addmm

Result2:

AttributeError: 'module' object has no attribute 'spmm'

My PyTorch version is 0.1.12_2 - would greatly appreciate possible solutions.


title: Reproducibility and performance in PyTorch

body:
The documentation states:

  Deterministic mode can have a performance impact, depending on your model.

My question is, what is meant by performance here. Processing speed or model quality (i.e. minimal loss)? In other words, when setting manual seeds and making the model perform in a deterministic way, does that cause longer training time until minimal loss is found, or is that minimal loss worse than when the model is non-deterministic?

For completeness' sake, I manually make the model deterministic by setting all of these properties:

def set_seed(seed):
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False
    np.random.seed(seed)
    random.seed(seed)
    os.environ['PYTHONHASHSEED'] = str(seed)


On ubuntu14.04,I use pytorch with cudnn.This problem happened:

Traceback (most recent call last):
  File "main.py", line 58, in <module>
    test_detect(test_loader, nod_net, get_pbb, bbox_result_path,config1,n_gpu=config_submit['n_gpu'])
  File "/home/ubuntu/nndl/DSB2017/test_detect.py", line 52, in test_detect
    output = net(input,inputcoord)
  File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/torch/nn/modules/module.py", line 252, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/torch/nn/parallel/data_parallel.py", line 58, in forward
    return self.module(*inputs[0], **kwargs[0])
  File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/torch/nn/modules/module.py", line 252, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ubuntu/nndl/DSB2017/net_detector.py", line 102, in forward
    out = self.preBlock(x)#16
  File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/torch/nn/modules/module.py", line 252, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/torch/nn/modules/container.py", line 67, in forward
    input = module(input)
  File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/torch/nn/modules/module.py", line 252, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/torch/nn/modules/conv.py", line 351, in forward
    self.padding, self.dilation, self.groups)
  File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/torch/nn/functional.py", line 119, in conv3d
    return f(input, weight, bias)
RuntimeError: CUDNN_STATUS_INTERNAL_ERROR

I have google it for severial hours an am really confused.What made this happen?


title: Pytorch: AttributeError: 'function' object has no attribute 'copy'

body:
I am trying to load a model state_dict I trained on Google Colab GPU, here is my code to load the model:

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

model = models.resnet50()
num_ftrs = model.fc.in_features
model.fc = nn.Linear(num_ftrs, n_classes)
model.load_state_dict(copy.deepcopy(torch.load("./models/model.pth",device)))
model = model.to(device)
model.eval()

Here is the error:

  state_dict = state_dict.copy()

  AttributeError: 'function' object has no attribute 'copy'

Pytorch :

>>> import torch
>>> print (torch.__version__)
1.4.0
>>> import torchvision
>>> print (torchvision.__version__)
0.5.0

Please help I have searched everywhere to no avail

[full error details][1] https://i.stack.imgur.com/s22DL.png


title: Pytorch embedding RuntimeError: Expected object of type torch.LongTensor but found type torch.cuda.LongTensor for argument #3 'index'

body:
I'm getting this error saying

RuntimeError: Expected object of type torch.LongTensor but found type torch.cuda.LongTensor for argument #3 'index'

But what does it mean by argument #3 "index"? I can't find "index" argument in torch.embedding (here source : https://pytorch.org/docs/stable/_modules/torch/nn/modules/sparse.html#Embedding)
It seems like I'm passing the embedding the wrong parameters.

I even changed the data type of my input like below but the error persists.

batch['doc_tok'] = batch['doc_tok'].long()
batch['query_tok'] = batch['query_tok'].long()

Any comment (even though it's short!) or just listing keywords to look at will be highly appreciated!

Here is a full traceback.

Traceback (most recent call last):
  File "train_v2.py", line 110, in <module>
    main()
  File "train_v2.py", line 81, in main
    model.update(batch)
  File "/home/aerin/Desktop/squad_vteam/src/model.py", line 129, in update
    loss_adv = self.adversarial_loss(batch, loss, self.network.lexicon_encoder.embedding.weight, y)
  File "/home/aerin/Desktop/squad_vteam/src/model.py", line 104, in adversarial_loss
    start, end, _ = self.network(batch)
  File "/home/aerin/anaconda3/envs/san/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/aerin/Desktop/squad_vteam/src/dreader.py", line 78, in forward
    doc_mask, query_mask = self.lexicon_encoder(batch)
  File "/home/aerin/anaconda3/envs/san/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/aerin/Desktop/squad_vteam/src/encoder.py", line 116, in forward
    doc_emb, query_emb = emb(doc_tok), emb(query_tok)
  File "/home/aerin/anaconda3/envs/san/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/aerin/anaconda3/envs/san/lib/python3.6/site-packages/torch/nn/modules/sparse.py", line 108, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/home/aerin/anaconda3/envs/san/lib/python3.6/site-packages/torch/nn/functional.py", line 1076, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: Expected object of type torch.LongTensor but found type torch.cuda.LongTensor for argument #3 'index'

Update: I even sent the whole model.network to cpu but still getting the same error.

batch['doc_tok']=batch['doc_tok'].long().cpu()
batch['query_tok']=batch['query_tok'].long().cpu()
self.network.cpu()

print(batch['doc_tok'].dtype,  batch['query_tok'].dtype) # They are both torch.int64 torch.int64

start, end, _ = self.network(batch)

At this point, I'm suspecting this might be a bug...

model.py code: https://github.com/byorxyz/san_mrc/blob/master/src/model.py

Network defined: https://github.com/byorxyz/san_mrc/blob/master/src/dreader.py


In following the Pytorch tutorial at https://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html

I received the following error:

(pt_gpu) [martin@A08-R32-I196-3-FZ2LTP2 mlm]$ python pytorch-1.py
Traceback (most recent call last):
  File "pytorch-1.py", line 39, in <module>
    device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
AttributeError: module 'torch' has no attribute 'device'

In my code below, I added this statement:

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
    net.to(device)

But this seems not right or enough. This is the first time for me to run Pytorch with GPU on a linux machine. What else should I do to get right running?

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)

        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 5 * 5)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)

        return x

net = Net()

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
net.to(device)

transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])
print(transform)

trainSet = torchvision.datasets.CIFAR10(root='./data', train=True, download=True, transform=transform)
trainLoader = torch.utils.data.DataLoader(trainSet, batch_size=4, shuffle=True, num_workers=2)

testSet = torchvision.datasets.CIFAR10(root='./data', train=False, download=True, transform=transform)
testLoader = torch.utils.data.DataLoader(testSet, batch_size=4, shuffle=False, num_workers=2)

classes = ('plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck')

import torch.optim as optim

criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)

for epoch in range(2):
    running_loss = 0.0
    for i, data in enumerate(trainLoader, 0):
        inputs, labels = data
        inputs, labels = inputs.to(device), labels.to(device)

        optimizer.zero_grad()

        outputs = net(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        running_loss += loss.item()
        if i % 2000 == 1999:
            print('[%d, %5d] loss %.3f' % (epoch + 1, i + 1, running_loss / 2000))

print('Finished traning!')

def imshow(img):
    img = img / 2 + 0.5
    npimg = img.numpy()
    plt.imshow(numpy.transpose(npimg, (1, 2, 0)))
    plt.show()

dataIter = iter(trainLoader)
images, labels = dataIter.next()
# imshow(torchvision.utils.make_grid(images))

print('GroundTruth: ', ' '.join('%5s' % classes[labels[j]] for j in range(4)))

outputs = net(images)

_, predicted = torch.max(outputs, 1)

print('Predicted: ', ' '.join('%5s' % classes[predicted[j]] for j in range(4)))
dataIter = iter(testLoader)
images, labels = dataIter.next()
# imshow(torchvision.utils.make_grid(images))

correct = 0
total = 0

with torch.no_grad():
    for data in testLoader:
        images, labels = data
        outputs = net(images)
        _, predicted = torch.max(outputs.data, 1)
        total += labels.size(0)

        correct += (predicted == labels).sum().item()

print("accuracy: %d %%", 100 * correct / total)

EDIT:

My conda version is the latest:

(pt_gpu) [martin@A08-R32-I196-3-FZ2LTP2 mlm]$ conda -V
conda 4.6.2

Then I installed pytorch-gpu with:

(pt_gpu) [martin@A08-R32-I196-3-FZ2LTP2 mlm]$ conda install -c anaconda pytorch-gpu

As you can see, the version 0.1.12 is installed:

Collecting package metadata: done
Solving environment: done

## Package Plan ##

  environment location: /home/martin/anaconda3/envs/pt_gpu

  added / updated specs:
    - pytorch-gpu

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    ca-certificates-2018.12.5  |                0         123 KB  anaconda
    certifi-2018.11.29         |           py36_0         146 KB  anaconda
    pytorch-gpu-0.1.12         |           py36_0        16.8 MB  anaconda
    ------------------------------------------------------------
                                           Total:        17.0 MB

The following packages will be UPDATED:

  openssl              pkgs/main::openssl-1.1.1a-h7b6447c_0 --> anaconda::openssl-1.1.1-h7b6447c_0

The following packages will be SUPERSEDED by a higher-priority channel:

  ca-certificates                                 pkgs/main --> anaconda
  certifi                                         pkgs/main --> anaconda
  mkl                    pkgs/main::mkl-2017.0.4-h4c4d0af_0 --> anaconda::mkl-2017.0.1-0
  pytorch-gpu                                     pkgs/free --> anaconda

Proceed ([y]/n)? y

Downloading and Extracting Packages
certifi-2018.11.29   | 146 KB    | ########################################################################################################################## | 100%
ca-certificates-2018 | 123 KB    | ########################################################################################################################## | 100%
pytorch-gpu-0.1.12   | 16.8 MB   | ########################################################################################################################## | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done

To verify the version, I do:

(pt_gpu) [martin@A08-R32-I196-3-FZ2LTP2 mlm]$ python -c "import torch; print(torch.__version__)"
0.1.12

Why does it install such a low version?


title: Spacy similarity warning : "Evaluating Doc.similarity based on empty vectors."

body:
I'm trying to do data enhancement with a FAQ dataset. I change words, specifically nouns, by most similar words with Wordnet checking the similarity with Spacy. I use multiple for loop to go through my dataset.

import spacy
import nltk
from nltk.corpus import wordnet as wn
import pandas as pd

nlp = spacy.load('en_core_web_md')
nltk.download('wordnet')
questions = pd.read_csv("FAQ.csv")

list_questions = []
for question in questions.values:
    list_questions.append(nlp(question[0]))

for question in list_questions:
    for token in question:
        treshold = 0.5
        if token.pos_ == 'NOUN':
            wordnet_syn = wn.synsets(str(token), pos=wn.NOUN)
            for syn in wordnet_syn:
                for lemma in syn.lemmas():
                    similar_word = nlp(lemma.name())
                    if similar_word.similarity(token) != 1. and similar_word.similarity(token) > treshold:
                        good_word = similar_word
                        treshold = token.similarity(similar_word)

However, the following warning is printed several times and I don't understand why :

  UserWarning: [W008] Evaluating Doc.similarity based on empty vectors.

It is my similar_word.similarity(token) which creates the problem but I don't understand why.
The form of my list_questions is :

list_questions = [Do you have a paper or other written explanation to introduce your model's details?, Where is the BERT code come from?, How large is a sentence vector?]

I need to check token but also the similar_word in the loop, for example, I still get the error here :

tokens = nlp(u'dog cat unknownword')
similar_word = nlp(u'rabbit')

if(similar_word):
    for token in tokens:
        if (token):
            print(token.text, similar_word.similarity(token))


title: map function in Pytorch

body:
Is there any map function in Pytorch? (something like map in python).

I need to map a 1xDxhxw tensor variable to a 1x(9D)xhxw tensor, to augment embedding of each pixel with its 8 neighbour embeddings. Is there any functionality in Pytorch that lets me do that efficiently?

I tried using map in Python this way:

n, d, h, w = embedding.size()
padder = nn.ReflectionPad2d(padding=1)
embedding = padder(embedding)
embedding = map(lambda i, j, M: M[:, :, i-1:i+2, j-1:j+2], range(1, h), range(1, w), embedding)

But it does not work for w > 2 and h > 2.


title: dropout(): argument 'input' (position 1) must be Tensor, not str when using Bert with Huggingface

body:
My code was working fine and when I tried to run it today without changing anything I got the following error:
dropout(): argument 'input' (position 1) must be Tensor, not str
Would appreciate if help could be provided. Could be an issue with the data loader?


title: How to run PyTorch on GPU by default?

body:
I want to run PyTorch using cuda. I set model.cuda() and torch.cuda.LongTensor() for all tensors.
Do I have to create tensors using .cuda explicitly if I have used model.cuda()?
Is there a way to make all computations run on GPU by default?


title: BucketIterator throws 'Field' object has no attribute 'vocab'

body:
It's not a new question, references I found without any solution working for me first and second.
I'm a newbie to PyTorch, facing AttributeError: 'Field' object has no attribute 'vocab' while creating batches of the text data in PyTorch using torchtext.

Following up the book Deep Learning with PyTorch I wrote the same example as explained in the book.

Here's the snippet:

from torchtext import data
from torchtext import datasets
from torchtext.vocab import GloVe

TEXT = data.Field(lower=True, batch_first=True, fix_length=20)
LABEL = data.Field(sequential=False)
train, test = datasets.IMDB.splits(TEXT, LABEL)

print("train.fields:", train.fields)
print()
print(vars(train[0]))  # prints the object

TEXT.build_vocab(train, vectors=GloVe(name="6B", dim=300),
                 max_size=10000, min_freq=10)

# VOCABULARY
# print(TEXT.vocab.freqs)  # freq
# print(TEXT.vocab.vectors)  # vectors
# print(TEXT.vocab.stoi)  # Index

train_iter, test_iter = data.BucketIterator.splits(
    (train, test), batch_size=128, device=-1, shuffle=True, repeat=False)  # -1 for cpu, None for gpu

# Not working (FROM BOOK)
# batch = next(iter(train_iter))

# print(batch.text)
# print()
# print(batch.label)

# This also not working (FROM Second solution)
for i in train_iter:
    print (i.text)
    print (i.label)

Here's the stacktrace:

AttributeError                            Traceback (most recent call last)
<ipython-input-33-433ec3a2ca3c> in <module>()
      7
      8
----> 9 for i in train_iter:
     10     print (i.text)
     11     print (i.label)

/anaconda3/lib/python3.6/site-packages/torchtext/data/iterator.py in __iter__(self)
    155                     else:
    156                         minibatch.sort(key=self.sort_key, reverse=True)
--> 157                 yield Batch(minibatch, self.dataset, self.device)
    158             if not self.repeat:
    159                 return

/anaconda3/lib/python3.6/site-packages/torchtext/data/batch.py in __init__(self, data, dataset, device)
     32                 if field is not None:
     33                     batch = [getattr(x, name) for x in data]
---> 34                     setattr(self, name, field.process(batch, device=device))
     35
     36     @classmethod

/anaconda3/lib/python3.6/site-packages/torchtext/data/field.py in process(self, batch, device)
    199         """
    200         padded = self.pad(batch)
--> 201         tensor = self.numericalize(padded, device=device)
    202         return tensor
    203

/anaconda3/lib/python3.6/site-packages/torchtext/data/field.py in numericalize(self, arr, device)
    300                 arr = [[self.vocab.stoi[x] for x in ex] for ex in arr]
    301             else:
--> 302                 arr = [self.vocab.stoi[x] for x in arr]
    303
    304             if self.postprocessing is not None:

/anaconda3/lib/python3.6/site-packages/torchtext/data/field.py in <listcomp>(.0)
    300                 arr = [[self.vocab.stoi[x] for x in ex] for ex in arr]
    301             else:
--> 302                 arr = [self.vocab.stoi[x] for x in arr]
    303
    304             if self.postprocessing is not None:

AttributeError: 'Field' object has no attribute 'vocab'

  If not using BucketIterator, what else I can use to get a similar
  output?


title: How to make early stopping in image classification pytorch

body: I'm new with Pytorch and machine learning I'm follow this tutorial in this tutorial https://www.learnopencv.com/image-classification-using-transfer-learning-in-pytorch/ and use my custom dataset. Then I have same problem in this tutorial but I dont know how to make early stopping in pytorch and if do you have better without create early stopping process please tell me.


title: PyTorch Getting Custom Loss Function Running

body:
I'm trying to use a custom loss function by extending nn.Module, but I can't get past the error

  element 0 of variables does not require grad and does not have a grad_fn

Note: my labels are lists of size: num_samples, but each batch will have the same labels throughout the batch, so we shrink labels for the whole batch to be a single label by calling .diag()

My code is as follows and is based on the transfer learning tutorial:

def train_model(model, criterion, optimizer, scheduler, num_epochs=25):
    since = time.time()

    best_model_wts = copy.deepcopy(model.state_dict())
    best_acc = 0.0

    for epoch in range(num_epochs):
        print('Epoch {}/{}'.format(epoch, num_epochs - 1))
        print('-' * 10)

        # Each epoch has a training and validation phase
        for phase in ['train', 'val']:
            if phase == 'train':
                scheduler.step()
                model.train(True)  # Set model to training mode
            else:
                model.train(False)  # Set model to evaluate mode

            running_loss = 0.0
            running_corrects = 0

            # Iterate over data.
            for data in dataloaders[phase]:
                # get the inputs
                inputs, labels = data
                inputs = inputs.float()

                # wrap them in Variable
                if use_gpu:
                    inputs = Variable(inputs.cuda())
                    labels = Variable(labels.cuda())
                else:
                    inputs = Variable(inputs)
                    labels = Variable(labels)

                # zero the parameter gradients
                optimizer.zero_grad()

                # forward
                outputs = model(inputs)
                #outputs = nn.functional.sigmoid(outputs).round()
                _, preds = torch.max(outputs, 1)
                label = labels.diag().float()
                preds = preds.float()
                loss = criterion(preds, label)
                # backward + optimize only if in training phase
                if phase == 'train':
                    loss.backward()
                    optimizer.step()

                # statistics
                running_loss += loss.data[0] * inputs.size(0)
                running_corrects += torch.sum(pred == label.data)

            epoch_loss = running_loss / dataset_sizes[phase]
            epoch_acc = running_corrects / dataset_sizes[phase]

            print('{} Loss: {:.4f} Acc: {:.4f}'.format(
                phase, epoch_loss, epoch_acc))

            # deep copy the model
            if phase == 'val' and epoch_acc > best_acc:
                best_acc = epoch_acc
                best_model_wts = copy.deepcopy(model.state_dict())

        print()

    time_elapsed = time.time() - since
    print('Training complete in {:.0f}m {:.0f}s'.format(
        time_elapsed // 60, time_elapsed % 60))
    print('Best val Acc: {:4f}'.format(best_acc))

    # load best model weights
    model.load_state_dict(best_model_wts)
    return model

and my loss function is defined below:

class CustLoss(nn.Module):
    def __init__(self):
        super(CustLoss, self).__init__()
    def forward(self, outputs, labels):
        return cust_loss(outputs, labels)

def cust_loss(pred, targets):
    '''preds are arrays of size classes with floats in them'''
    '''targets are arrays of all the classes from the batch'''
    '''we sum the classes from the batch and find the num correct'''
    r = torch.sum(pred == targets)
    return r

Then I run the following to run the model:

model_ft = models.resnet18(pretrained=True)
for param in model_ft.parameters():
    param.requires_grad = False

num_ftrs = model_ft.fc.in_features
model_ft.fc = nn.Linear(num_ftrs, 3)

if use_gpu:
    model_ft = model_ft.cuda()

criterion = CustLoss()

# Observe that all parameters are being optimized
optimizer_ft = optim.SGD(model_ft.fc.parameters(), lr=0.001, momentum=0.9)

# Decay LR by a factor of 0.1 every 7 epochs
exp_lr_scheduler = lr_scheduler.StepLR(optimizer_ft, step_size=7, gamma=0.1)
model_ft = train_model(model_ft, criterion, optimizer_ft, exp_lr_scheduler,num_epochs=25)

I tried getting it to work with other loss functions to no avail. I always get the same error when loss.backward() is called.

It was my understanding that I wouldn't need a custom implementation of loss.backward if I extend nn.Module.


title: Implementing Adagrad in Python

body:
I'm trying to implement Adagrad in Python. For learning purposes, I am using matrix factorisation as an example. I'd be using Autograd for computing the gradients.

My main question is if the implementation is fine.

Problem description

Given a matrix A (M x N) having some missing entries, decompose into W and H having sizes (M x k) and (k X N) respectively. Goal would to learn W and H using Adagrad. I'd be following this guide for the Autograd implementation.

NB: I very well know that ALS based implementation are well-suited. I'm using Adagrad only for learning purposes

Customary imports

import autograd.numpy as np
import pandas as pd

Creating the matrix to be decomposed

A = np.array([[3, 4, 5, 2],
                   [4, 4, 3, 3],
                   [5, 5, 4, 3]], dtype=np.float32).T

Masking one entry

A[0, 0] = np.NAN

Defining the cost function

def cost(W, H):
    pred = np.dot(W, H)
    mask = ~np.isnan(A)
    return np.sqrt(((pred - A)[mask].flatten() ** 2).mean(axis=None))

Decomposition params

rank = 2
learning_rate=0.01
n_steps = 10000

Gradient of cost wrt params W and H

from autograd import grad, multigrad
grad_cost= multigrad(cost, argnums=[0,1])

Main Adagrad routine (this needs to be checked)

shape = A.shape

# Initialising W and H
H =  np.abs(np.random.randn(rank, shape[1]))
W =  np.abs(np.random.randn(shape[0], rank))

# gt_w and gt_h contain accumulation of sum of gradients
gt_w = np.zeros_like(W)
gt_h = np.zeros_like(H)

# stability factor
eps = 1e-8
print "Iteration, Cost"
for i in range(n_steps):

    if i%1000==0:
        print "*"*20
        print i,",", cost(W, H)

    # computing grad. wrt W and H
    del_W, del_H = grad_cost(W, H)

    # Adding square of gradient
    gt_w+= np.square(del_W)
    gt_h+= np.square(del_H)

    # modified learning rate
    mod_learning_rate_W = np.divide(learning_rate, np.sqrt(gt_w+eps))
    mod_learning_rate_H = np.divide(learning_rate, np.sqrt(gt_h+eps))
    W =  W-del_W*mod_learning_rate_W
    H =  H-del_H*mod_learning_rate_H

While the problem converges and I get a reasonable solution, I was wondering if the implementation is correct. Specifically, if the understanding of sum of gradients and then computing the adaptive learning rate is correct or not?


title: PyTorch PermissionError: [Errno 13] Permission denied: '/.torch'

body:
I'm running a PyTorch based ML program for image classification using Resnet50 model for transfer learning. I am getting below error regarding permission.

Traceback (most recent call last):
File "imgc_pytorch.py", line 67, in
  model = models.resnet50(pretrained=True)
File "/opt/conda/lib/python3.6/site-packages/torchvision/models/resnet.py", line 187, in resnet50
  model.load_state_dict(model_zoo.load_url(model_urls['resnet50']))
File "/opt/conda/lib/python3.6/site-packages/torch/utils/model_zoo.py",
line 59, in load_url
  os.makedirs(model_dir)
File "/opt/conda/lib/python3.6/os.py", line 210, in makedirs
  makedirs(head, mode, exist_ok)
File "/opt/conda/lib/python3.6/os.py", line 220, in makedirs
  mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/.torch'

Looked up on this forum and it was suggested to add 'sudo' before the file name, but getting error "/bin/sh: 1: sudo: not found"


title: Pytorch Validating Model Error: Expected input batch_size (3) to match target batch_size (4)

body:
I'm building a NN in Pytorch that is supposed to classify across 102 classes.

I've got the following validation function:

def validation(model, testloader, criterion):
    test_loss = 0
    accuracy = 0

    for inputs, classes in testloader:
        inputs = inputs.to('cuda')
        output = model.forward(inputs)
        test_loss += criterion(output, labels).item()

        ps = torch.exp(output)
        equality = (labels.data == ps.max(dim=1)[1])
        accuracy += equality.type(torch.FloatTensor).mean()

    return test_loss, accuracy

Code for training (calls validation):

epochs = 3
print_every = 40
steps = 0
running_loss = 0
testloader = dataloaders['test']

# change to cuda
model.to('cuda')

for e in range(epochs):
    running_loss = 0
    for ii, (inputs, labels) in enumerate(dataloaders['train']):
        steps += 1

        inputs, labels = inputs.to('cuda'), labels.to('cuda')

        optimizer.zero_grad()

        # Forward and backward passes
        outputs = model.forward(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        running_loss += loss.item()

        if steps % print_every == 0:
            model.eval()
            with torch.no_grad():
                test_loss, accuracy = validation(model, testloader, criterion)

            print("Epoch: {}/{}.. ".format(e+1, epochs),
                  "Training Loss: {:.3f}.. ".format(running_loss/print_every),
                  "Test Loss: {:.3f}.. ".format(test_loss/len(testloader)),
                  "Test Accuracy: {:.3f}".format(accuracy/len(testloader)))

            running_loss = 0
            model.train()

Im getting this error message:

ValueError: Expected input batch_size (3) to match target batch_size (4).

Full Traceback:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-63-f9f67ed13b94> in <module>()
     28             model.eval()
     29             with torch.no_grad():
---> 30                 test_loss, accuracy = validation(model, testloader, criterion)
     31
     32             print("Epoch: {}/{}.. ".format(e+1, epochs),

<ipython-input-62-dbc77acbda5e> in validation(model, testloader, criterion)
      6         inputs = inputs.to('cuda')
      7         output = model.forward(inputs)
----> 8         test_loss += criterion(output, labels).item()
      9
     10         ps = torch.exp(output)

/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    489             result = self._slow_forward(*input, **kwargs)
490         else:
--> 491             result = self.forward(*input, **kwargs)
    492         for hook in self._forward_hooks.values():
    493             hook_result = hook(self, input, result)

/opt/conda/lib/python3.6/site-packages/torch/nn/modules/loss.py in forward(self, input, target)
    191         _assert_no_grad(target)
    192         return F.nll_loss(input, target, self.weight, self.size_average,
--> 193                           self.ignore_index, self.reduce)
    194
    195

/opt/conda/lib/python3.6/site-packages/torch/nn/functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce)
   1328     if input.size(0) != target.size(0):
   1329         raise ValueError('Expected input batch_size ({}) to match target batch_size ({}).'
-> 1330                          .format(input.size(0), target.size(0)))
   1331     if dim == 2:
   1332         return torch._C._nn.nll_loss(input, target, weight, size_average, ignore_index, reduce)

ValueError: Expected input batch_size (3) to match target batch_size (4).

I don't understand where the error comes from. Indeed, without the validation code the training part works perfectly.


title: conv2d(): argument 'input' (position 1) must be Tensor, not str in loop function

body:
I am trying to write a loop function to extract features from my data and save it in the list.

And here is my code:

import pickle

output_features = []

with torch.no_grad():
    for (inputs, labels) in dataloaders.items():
        x=feature_extractor(inputs)
        output_features.append(x)
        output_features = torch.cat(output_features).numpy

pickle.dump(output_features, open("features.pkl", "w"))

But i have an error.

conv2d(): argument 'input' (position 1) must be Tensor, not str

Then I changed again it to

with torch.no_grad():
    for i, (inputs, labels) in enumerate(dataloaders['val']):
        x=feature_extractor(inputs)
        output_features.append(x)
        output_features = torch.cat(output_features).numpy

pickle.dump(output_features, open("features.pkl", "wb"))

Then I got this error

'builtin_function_or_method' object has no attribute 'append'

Please, someone, explain to me why I got this error and how to get features from each input with its label and save them in an array. Thank you


title: Python wheels: cp27mu not supported

body:
I'm trying to install pytorch (http://pytorch.org/) on Linux, and according to my machine configuration, the command I should run is:

pip install https://s3.amazonaws.com/pytorch/whl/torch-0.1.6.post17-cp27-cp27mu-linux_x86_64.whl

On one machine (Linux distribution Slackware 14.1) the installation fails with error: torch-0.1.6.post17-cp27-cp27mu-linux_x86_64.whl is not a supported wheel on this platform., while on another one (Ubuntu 15.10) it succeeds. From what I understood, the problem seems to be the cp27mu in the wheel name.

Using the command import pip; print(pip.pep425tags.get_supported()) from the Python shell, I get this from the Slackware machine:

[('cp27', 'cp27m', 'manylinux1_x86_64'), ('cp27', 'cp27m', 'linux_x86_64'), ('cp27', 'none', 'manylinux1_x86_64'), ('cp27', 'none', 'linux_x86_64'), ('py2', 'none', 'manylinux1_x86_64'), ('py2', 'none', 'linux_x86_64'), ('cp27', 'none', 'any'), ('cp2', 'none', 'any'), ('py27', 'none', 'any'), ('py2', 'none', 'any'), ('py26', 'none', 'any'), ('py25', 'none', 'any'), ('py24', 'none', 'any'), ('py23', 'none', 'any'), ('py22', 'none', 'any'), ('py21', 'none', 'any'), ('py20', 'none', 'any')]

and this from the Ubuntu machine:

[('cp27', 'cp27mu', 'manylinux1_x86_64'), ('cp27', 'cp27mu', 'linux_x86_64'), ('cp27', 'none', 'manylinux1_x86_64'), ('cp27', 'none', 'linux_x86_64'), ('py2', 'none', 'manylinux1_x86_64'), ('py2', 'none', 'linux_x86_64'), ('cp27', 'none', 'any'), ('cp2', 'none', 'any'), ('py27', 'none', 'any'), ('py2', 'none', 'any'), ('py26', 'none', 'any'), ('py25', 'none', 'any'), ('py24', 'none', 'any'), ('py23', 'none', 'any'), ('py22', 'none', 'any'), ('py21', 'none', 'any'), ('py20', 'none', 'any')]

From https://www.python.org/dev/peps/pep-0513/, it seems to me that supporting cp27m or cp27mu depends on an option passed at compile time, --enable-unicode.

Now, maybe at this point I shouldn't even be asking the question, but just to be sure, does this mean that I have to compile Python with the --enable-unicode=ucs4 on the Slackware machine in order to install that wheel?


title: How to find IoU from segmentation masks?

body:
I am doing an image segmentation task and I am using a dataset that only has ground truths but no bounding boxes or polygons.

I have 2 classes( ignoring 0 for background) and the outputs and ground truth labels are in an array like

Predicted--/---Labels

0|0|0|1|2   0|0|0|1|2
 0|2|1|0|0   0|2|1|0|0
 0|0|1|1|1   0|0|1|1|1
 0|0|0|0|1   0|0|0|0|1

How do I calculate IoU from these ?

PS: I am using python3 with pytorch api


I'm using Pytorch  under Ubuntu 18.04 and trying to import torchvision, but I get an error libcudart.so.9.0: cannot open shared object file: No such file or directory.

Someone could help to fix it? Thanks.

The infos below are detailed error logs:

Traceback (most recent call last):
  File "/home/x/.local/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2882, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-6dd351122000>", line 1, in <module>
    import torchvision
  File "/home/x/pycharm-2019.2/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "/home/x/.local/lib/python3.6/site-packages/torchvision/__init__.py", line 1, in <module>
    from torchvision import models
  File "/home/x/pycharm-2019.2/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "/home/x/.local/lib/python3.6/site-packages/torchvision/models/__init__.py", line 11, in <module>
    from . import detection
  File "/home/x/pycharm-2019.2/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "/home/x/.local/lib/python3.6/site-packages/torchvision/models/detection/__init__.py", line 1, in <module>
    from .faster_rcnn import *
  File "/home/x/pycharm-2019.2/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "/home/x/.local/lib/python3.6/site-packages/torchvision/models/detection/faster_rcnn.py", line 7, in <module>
    from torchvision.ops import misc as misc_nn_ops
  File "/home/x/pycharm-2019.2/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "/home/x/.local/lib/python3.6/site-packages/torchvision/ops/__init__.py", line 1, in <module>
    from .boxes import nms, box_iou
  File "/home/x/pycharm-2019.2/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "/home/x/.local/lib/python3.6/site-packages/torchvision/ops/boxes.py", line 2, in <module>
    from torchvision import _C
  File "/home/x/pycharm-2019.2/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
ImportError: libcudart.so.9.0: cannot open shared object file: No such file or directory
import torch
import torch.nn as nn
import torchvision.transforms as transforms
Traceback (most recent call last):
  File "/home/x/.local/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2882, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-4-677acbcfae34>", line 1, in <module>
    import torchvision.transforms as transforms
  File "/home/x/pycharm-2019.2/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "/home/x/.local/lib/python3.6/site-packages/torchvision/__init__.py", line 1, in <module>
    from torchvision import models
  File "/home/x/pycharm-2019.2/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "/home/x/.local/lib/python3.6/site-packages/torchvision/models/__init__.py", line 11, in <module>
    from . import detection
  File "/home/x/pycharm-2019.2/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "/home/x/.local/lib/python3.6/site-packages/torchvision/models/detection/__init__.py", line 1, in <module>
    from .faster_rcnn import *
  File "/home/x/pycharm-2019.2/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "/home/x/.local/lib/python3.6/site-packages/torchvision/models/detection/faster_rcnn.py", line 7, in <module>
    from torchvision.ops import misc as misc_nn_ops
  File "/home/x/pycharm-2019.2/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "/home/x/.local/lib/python3.6/site-packages/torchvision/ops/__init__.py", line 1, in <module>
    from .boxes import nms, box_iou
  File "/home/x/pycharm-2019.2/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "/home/x/.local/lib/python3.6/site-packages/torchvision/ops/boxes.py", line 2, in <module>
    from torchvision import _C
  File "/home/x/pycharm-2019.2/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
ImportError: libcudart.so.9.0: cannot open shared object file: No such file or directory


title: What is running loss in PyTorch and how is it calculated

body:
I had a look at this tutorial in the PyTorch docs for understanding Transfer Learning. There was one line that I failed to understand.

After the loss is calculated using loss = criterion(outputs, labels),  the running loss is calculated using running_loss += loss.item() * inputs.size(0) and finally, the epoch loss is calculated using running_loss / dataset_sizes[phase].

Isn't loss.item() supposed to be for an entire mini-batch (please correct me if I am wrong). i.e, if the batch_size is 4, loss.item() would give the loss for the entire set of 4 images. If this is true, why is loss.item() being multiplied  with inputs.size(0) while calculating running_loss? Isn't this step like an extra multiplication in this case?

Any help would be appreciated. Thanks!


title: BERT Multi-class text classification in Google Colab

body:
I'm working on a data set of social media comments(including youtube links) as input features and the Myers-Biggs Personality Profile as the target label:

    type    posts
0   INFJ    'http://www.youtube.com/watch?v=qsXHcwe3krw|||...
1   ENTP    'I'm finding the lack of me in these posts ver...
2   INTP    'Good one _____ https://www.youtube.com/wat...
3   INTJ    'Dear INTP, I enjoyed our conversation the o...
4   ENTJ    'You're fired.|||That's another silly misconce...

but from what I've found, BERT wants DataFrame's to be in this format:

a   label   posts
0   a   8   'http://www.youtube.com/watch?v=qsXHcwe3krw|||...
1   a   3   'I'm finding the lack of me in these posts ver...
2   a   11  'Good one _____ https://www.youtube.com/wat...
3   a   10  'Dear INTP, I enjoyed our conversation the o...
4   a   2   'You're fired.|||That's another silly misconce...

The resulting output must be a prediction on a test set of comments split into four columns, one for each Personality Profile where, for example, 'Mind' = 1 is the label for Extrovert. Basically splitting a type like INFJ into 'Mind','Energy','Nature','Tactics', like such:

    type    post    Mind    Energy  Nature  Tactics
0   INFJ    'url-web    0   1   0   1
1   INFJ    url-web 0   1   0   1
2   INFJ    enfp and intj moments url-web sportscenter n... 0   1   0   1
3   INFJ    What has been the most life-changing experienc...   0   1   0   1
4   INFJ    url-web url-web On repeat for most of today.    0   1   0   1

I've installed pytorch-pretrained-bert using:

!pip install pytorch-pretrained-bert

I've imported the models and tried to tokenize the 'posts' column using:

import torch
from pytorch_pretrained_bert import BertTokenizer, BertModel, BertForMaskedLM

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

tokenized_train = tokenizer.tokenize(train)

but receive this error:

TypeError: ord() expected a character, but string of length 5 found

I tried this based off the pytorch-pretrained-bert GitHub Repo and a Youtube vidoe.

I am a Data Science intern with no Deep Learning experience at all. I simply want to experiment with the BERT model in the most simplest way to predict the multi-class classified output so I can compare the results to simpler text-classification models we are currently working on. I am working in Google Colab and the resulting output should be a .csv file.

I understand that this is a complicated model and all documentation and examples surrounding the model are complex (fine tuning layers etc.) but any help for a simple implementation(if in fact there is such a thing) for a beginner Data Scientist with minimal Software Engineering experience, would be much appreciated.


The following is from a project that I'm doing in Udacity on Deep Learning. The project is on Generating TV scripts. The error that i encountered is the one below.
The following function is the one after model training.

def generate(rnn, prime_id, int_to_vocab, token_dict, pad_value, predict_len=100):
    """
    Generate text using the neural network
    param decoder: The PyTorch Module that holds the trained neural network
    param prime_id: The word id to start the first prediction
    param int_to_vocab: Dict of word id keys to word values
    param token_dict: Dict of puncuation tokens keys to puncuation values
    param pad_value: The value used to pad a sequence
    param predict_len: The length of text to generate
    return: The generated text
    """
    rnn.eval()

    # create a sequence (batch_size=1) with the prime_id
    current_seq = np.full((1, sequence_length), pad_value)
    current_seq[-1][-1] = prime_id
    predicted = [int_to_vocab[prime_id]]

    for _ in range(predict_len):
        if train_on_gpu:
            current_seq = torch.LongTensor(current_seq).cuda()
        else:
            current_seq = torch.LongTensor(current_seq)

        # initialize the hidden state
        hidden = rnn.init_hidden(current_seq.size(0))

        # get the output of the rnn
        output, _ = rnn(current_seq, hidden)

        # get the next word probabilities
        p = F.softmax(output, dim=1).data
        if(train_on_gpu):
            p = p.cpu() # move to cpu

        # use top_k sampling to get the index of the next word
        top_k = 5
        p, top_i = p.topk(top_k)
        top_i = top_i.numpy().squeeze()

        # select the likely next word index with some element of randomness
        p = p.numpy().squeeze()
        word_i = np.random.choice(top_i, p=p/p.sum())

        # retrieve that word from the dictionary
        word = int_to_vocab[word_i]
        predicted.append(word)

        # the generated word becomes the next "current sequence" and the cycle can continue
        current_seq = np.roll(current_seq, -1, 1)
        current_seq[-1][-1] = word_i

    gen_sentences = ' '.join(predicted)

    # Replace punctuation tokens
    for key, token in token_dict.items():
        ending = ' ' if key in ['\n', '(', '"'] else ''
        gen_sentences = gen_sentences.replace(' ' + token.lower(), key)
    gen_sentences = gen_sentences.replace('\n ', '\n')
    gen_sentences = gen_sentences.replace('( ', '(')

    # return all the sentences
    return gen_sentences

after this the following code is run:

# run the cell multiple times to get different results!
gen_length = 400 # modify the length to your preference
prime_word = 'jerry' # name for starting the script

pad_word = helper.SPECIAL_WORDS['PADDING']
generated_script = generate(trained_rnn, vocab_to_int[prime_word + ':'], int_to_vocab, token_dict, vocab_to_int[pad_word], gen_length)
print(generated_script)

Upon running this code, I get the following error

TypeError                                 Traceback (most recent call last)
<ipython-input-40-68a17c4d1704> in <module>()
      7 """
      8 pad_word = helper.SPECIAL_WORDS['PADDING']
----> 9 generated_script = generate(trained_rnn, vocab_to_int[prime_word + ':'], int_to_vocab, token_dict, vocab_to_int[pad_word], gen_length)
     10 print(generated_script)

3 frames
<ipython-input-39-b86c7a305356> in generate(rnn, prime_id, int_to_vocab, token_dict, pad_value, predict_len)
     53
     54         # the generated word becomes the next "current sequence" and the cycle can continue
---> 55         current_seq = np.roll(current_seq, -1, 1)
     56         current_seq[-1][-1] = word_i
     57

<__array_function__ internals> in roll(*args, **kwargs)

/usr/local/lib/python3.6/dist-packages/numpy/core/numeric.py in roll(a, shift, axis)
   1179
   1180     """
-> 1181     a = asanyarray(a)
   1182     if axis is None:
   1183         return roll(a.ravel(), shift, 0).reshape(a.shape)

/usr/local/lib/python3.6/dist-packages/numpy/core/_asarray.py in asanyarray(a, dtype, order)
    136
    137     """
--> 138     return array(a, dtype, copy=False, order=order, subok=True)
    139
    140

/usr/local/lib/python3.6/dist-packages/torch/tensor.py in __array__(self, dtype)
    490     def __array__(self, dtype=None):
    491         if dtype is None:
--> 492             return self.numpy()
    493         else:
    494             return self.numpy().astype(dtype, copy=False)

TypeError: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.

Can anyone please help me out?


I'm new to pytorch. I took the code from this repository https://github.com/ruotianluo/ImageCaptioning.pytorch and wanted to make captions to the images. Installed "CUDA" and when I run the script to create the annotations produces this:

  $ CUDA_LAUNCH_BLOCKING = 1 python eval.py --model model.pth -
  -infos_path infos.pkl --image_folder blah --num_images 1 /home/azat/anaconda2/lib/python2.7/site-packages/h5py/init.py:36:
  FutureWarning: Conversion of the second argument of issubdtype from
  float tonp.floating is deprecated. In the future, it will be
  treated as np.float64 == np.dtype (float) .type.   from ._conv
  import register_converters as _register_converters
  /home/azat/anaconda2/lib/python2.7/site-packages/torch/cuda/init.py:97:
  UserWarning:
      Found GPU0 GeForce 820M which is of cuda capability 2.1.
      PyTorch no longer supports this GPU because it is too old.
         warnings.warn (old_gpu_warn% (d, name, major, capability 1)) DataLoaderRaw loading images from folder: blah 0 listing all images in
  directory blah DataLoaderRaw found 8 images THCudaCheck FAIL file = /
  pytorch / torch / lib / THC / generic / THCTensorMathPairwise.cu line
  = 40 error = 48: no kernel image is available for execution on the device Traceback (most recent last call last):   File "eval.py", line
  122, in
      vars (opt))   File "/ home / azat / Programing / Python / techno_atom_neuro / Others Implementation /
  ImageCaptioning.pytorch-master / eval_utils.py", line 82, in
  eval_split
      data = loader.get_batch (split)   File "/ home / azat / Programing / Python / techno_atom_neuro / Others Implementation /
  ImageCaptioning.pytorch-master / dataloaderraw.py", line 112, in
  get_batch
      img = Variable (preprocess (img), volatile = True)   File "/home/azat/anaconda2/lib/python2.7/site-packages/torchvision/transforms/transforms.py",
  line 42, in call
      img = t (img)   File "/home/azat/anaconda2/lib/python2.7/site-packages/torchvision/transforms/transforms.py",
  line 118, in call
      return F.normalize (tensor, self.mean, self.std)   File "/home/azat/anaconda2/lib/python2.7/site-packages/torchvision/transforms/functional.py",
  line 161, in normalize
      t.sub_ (m) .div_ (s) RuntimeError: cuda runtime error (48): no kernel image is available for execution on the device at
  /pytorch/torch/lib/THC/generic/THCTensorMathPairwise.cu:40

So, I want to find out what this error is, hardware or software. And how can I solve this problem. Thanks.

PyTorch
OS: Ubuntu 16.04
PyTorch version: 0.3.1 pip
Python version: Python 2.7.14 :: Anaconda custom (64-bit)
CUDA/cuDNN version: 9.1
GPU models and configuration:

$ nvidia-smi
Wed Apr 11 21:34:08 2018
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 390.48                 Driver Version: 390.48                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce 820M        Off  | 00000000:08:00.0 N/A |                  N/A |
| N/A   54C    P0    N/A /  N/A |    114MiB /  1985MiB |     N/A      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0                    Not Supported                                       |
+-----------------------------------------------------------------------------+


title: How to correctly access elements in a 3D-Pytorch-Tensor?

body:
I am trying to access multiple elements in a 3D-Pytorch-Tensor, but the number of elements that are returned is wrong.

This is my code:

import torch

a = torch.FloatTensor(4,3,2)
print("a = {}".format(a))
print("a[:][:][0] = {}".format(a[:][:][0]))

This is the output:

a = tensor([[[-4.8569e+36,  3.0760e-41],
         [ 2.7953e+20,  1.6928e+22],
         [ 3.1692e-40,  7.2945e-15]],

        [[ 2.5011e+24,  1.3173e-39],
         [ 1.7229e-07,  4.1262e-08],
         [ 4.1490e-08,  6.4103e-10]],

        [[ 3.1728e-40,  5.8258e-40],
         [ 2.8776e+32,  6.7805e-10],
         [ 3.1764e-40,  5.4229e+08]],

        [[ 7.2424e-37,  1.3697e+07],
         [-2.0362e-33,  1.8146e+11],
         [ 3.1836e-40,  1.9670e+34]]])
a[:][:][0] = tensor([[-4.8569e+36,  3.0760e-41],
        [ 2.7953e+20,  1.6928e+22],
        [ 3.1692e-40,  7.2945e-15]])

I would expect something like this:

a[:][:][0] = tensor([[-4.8569e+36,  2.7953e+20, 3.1692e-40,
          2.5011e+24, 1.7229e-07, 4.1490e-08,
          3.1728e-40, 2.8776e+32, 3.1764e-40,
          7.2424e-37, -2.0362e-33, 3.1836e-40]])

Can anyone explain to me how I can come to this result?
Thank you very much in advance!

I get exactly the expected result on performing:

for i in range(4):
   for j in range(3):
      print("a[{}][{}][0] = {}".format(i,j, a[i][j][0]))


title: subtraction of scalar from tensor yields 'inconsistent tensor size' in pytorch

body:
I'm using pytorch and my variables are

x = [torch.FloatTensor of size 1x3x32x32]
mean = Variable containing:
1.00000e-02 *
  2.0518
[torch.FloatTensor of size 1]

what I want to do is subtract the scalar mean from x by doing

x = x - mean

However, I'm getting this error:

RuntimeError: inconsistent tensor size at /py/conda-
bld/pytorch_1493670682084/work/torch/lib/TH/generic/THTensorMath.c:831

What am I doing wrong?
Thanks a lot