title: PyTorch: How to print output blob size of each layer in network?

body:
I can print network structure like (also how to print positional index of each 'simple' layer? because in this example we have 3 for Fire module and it's content ('simple' layers) don't have index):

net = models.squeezenet1_1(pretrained=True)
print(net)

SqueezeNet(
  (features): Sequential(
    (0): Conv2d (3, 64, kernel_size=(3, 3), stride=(2, 2))
    (1): ReLU(inplace)
    (2): MaxPool2d(kernel_size=(3, 3), stride=(2, 2), dilation=(1, 1))
    (3): Fire(
      (squeeze): Conv2d (64, 16, kernel_size=(1, 1), stride=(1, 1))
      (squeeze_activation): ReLU(inplace)
      (expand1x1): Conv2d (16, 64, kernel_size=(1, 1), stride=(1, 1))
      (expand1x1_activation): ReLU(inplace)
      (expand3x3): Conv2d (16, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (expand3x3_activation): ReLU(inplace)
    )
    (4): Fire(
      (squeeze): Conv2d (128, 16, kernel_size=(1, 1), stride=(1, 1))
      (squeeze_activation): ReLU(inplace)
      (expand1x1): Conv2d (16, 64, kernel_size=(1, 1), stride=(1, 1))
      (expand1x1_activation): ReLU(inplace)
      (expand3x3): Conv2d (16, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (expand3x3_activation): ReLU(inplace)
    )
    (5): MaxPool2d(kernel_size=(3, 3), stride=(2, 2), dilation=(1, 1))
    (6): Fire(
      (squeeze): Conv2d (128, 32, kernel_size=(1, 1), stride=(1, 1))
      (squeeze_activation): ReLU(inplace)
      (expand1x1): Conv2d (32, 128, kernel_size=(1, 1), stride=(1, 1))
      (expand1x1_activation): ReLU(inplace)
      (expand3x3): Conv2d (32, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (expand3x3_activation): ReLU(inplace)
    )
    (7): Fire(
      (squeeze): Conv2d (256, 32, kernel_size=(1, 1), stride=(1, 1))
      (squeeze_activation): ReLU(inplace)
      (expand1x1): Conv2d (32, 128, kernel_size=(1, 1), stride=(1, 1))
      (expand1x1_activation): ReLU(inplace)
      (expand3x3): Conv2d (32, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (expand3x3_activation): ReLU(inplace)
    )
    (8): MaxPool2d(kernel_size=(3, 3), stride=(2, 2), dilation=(1, 1))
    (9): Fire(
      (squeeze): Conv2d (256, 48, kernel_size=(1, 1), stride=(1, 1))
      (squeeze_activation): ReLU(inplace)
      (expand1x1): Conv2d (48, 192, kernel_size=(1, 1), stride=(1, 1))
      (expand1x1_activation): ReLU(inplace)
      (expand3x3): Conv2d (48, 192, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (expand3x3_activation): ReLU(inplace)
    )
    (10): Fire(
      (squeeze): Conv2d (384, 48, kernel_size=(1, 1), stride=(1, 1))
      (squeeze_activation): ReLU(inplace)
      (expand1x1): Conv2d (48, 192, kernel_size=(1, 1), stride=(1, 1))
      (expand1x1_activation): ReLU(inplace)
      (expand3x3): Conv2d (48, 192, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (expand3x3_activation): ReLU(inplace)
    )
    (11): Fire(
      (squeeze): Conv2d (384, 64, kernel_size=(1, 1), stride=(1, 1))
      (squeeze_activation): ReLU(inplace)
      (expand1x1): Conv2d (64, 256, kernel_size=(1, 1), stride=(1, 1))
      (expand1x1_activation): ReLU(inplace)
      (expand3x3): Conv2d (64, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (expand3x3_activation): ReLU(inplace)
    )
    (12): Fire(
      (squeeze): Conv2d (512, 64, kernel_size=(1, 1), stride=(1, 1))
      (squeeze_activation): ReLU(inplace)
      (expand1x1): Conv2d (64, 256, kernel_size=(1, 1), stride=(1, 1))
      (expand1x1_activation): ReLU(inplace)
      (expand3x3): Conv2d (64, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (expand3x3_activation): ReLU(inplace)
    )
  )
  (classifier): Sequential(
    (0): Dropout(p=0.5)
    (1): Conv2d (512, 1000, kernel_size=(1, 1), stride=(1, 1))
    (2): ReLU(inplace)
    (3): AvgPool2d(kernel_size=13, stride=1, padding=0, ceil_mode=False, count_include_pad=True)
  )
)

And I can print weights size like:

for i, weights in enumerate(list(net.parameters())):
    print('i:',i,'weights:',weights.size())

i: 0 weights: torch.Size([64, 3, 3, 3])
i: 1 weights: torch.Size([64])
i: 2 weights: torch.Size([16, 64, 1, 1])
i: 3 weights: torch.Size([16])
i: 4 weights: torch.Size([64, 16, 1, 1])
i: 5 weights: torch.Size([64])
i: 6 weights: torch.Size([64, 16, 3, 3])
i: 7 weights: torch.Size([64])
i: 8 weights: torch.Size([16, 128, 1, 1])
i: 9 weights: torch.Size([16])
i: 10 weights: torch.Size([64, 16, 1, 1])
i: 11 weights: torch.Size([64])
i: 12 weights: torch.Size([64, 16, 3, 3])
i: 13 weights: torch.Size([64])
i: 14 weights: torch.Size([32, 128, 1, 1])
i: 15 weights: torch.Size([32])
i: 16 weights: torch.Size([128, 32, 1, 1])
i: 17 weights: torch.Size([128])
i: 18 weights: torch.Size([128, 32, 3, 3])
i: 19 weights: torch.Size([128])
i: 20 weights: torch.Size([32, 256, 1, 1])
i: 21 weights: torch.Size([32])
i: 22 weights: torch.Size([128, 32, 1, 1])
i: 23 weights: torch.Size([128])
i: 24 weights: torch.Size([128, 32, 3, 3])
i: 25 weights: torch.Size([128])
i: 26 weights: torch.Size([48, 256, 1, 1])
i: 27 weights: torch.Size([48])
i: 28 weights: torch.Size([192, 48, 1, 1])
i: 29 weights: torch.Size([192])
i: 30 weights: torch.Size([192, 48, 3, 3])
i: 31 weights: torch.Size([192])
i: 32 weights: torch.Size([48, 384, 1, 1])
i: 33 weights: torch.Size([48])
i: 34 weights: torch.Size([192, 48, 1, 1])
i: 35 weights: torch.Size([192])
i: 36 weights: torch.Size([192, 48, 3, 3])
i: 37 weights: torch.Size([192])
i: 38 weights: torch.Size([64, 384, 1, 1])
i: 39 weights: torch.Size([64])
i: 40 weights: torch.Size([256, 64, 1, 1])
i: 41 weights: torch.Size([256])
i: 42 weights: torch.Size([256, 64, 3, 3])
i: 43 weights: torch.Size([256])
i: 44 weights: torch.Size([64, 512, 1, 1])
i: 45 weights: torch.Size([64])
i: 46 weights: torch.Size([256, 64, 1, 1])
i: 47 weights: torch.Size([256])
i: 48 weights: torch.Size([256, 64, 3, 3])
i: 49 weights: torch.Size([256])
i: 50 weights: torch.Size([1000, 512, 1, 1])
i: 51 weights: torch.Size([1000])

How to print output blob size of each layer in network?


title: pytorch:RuntimeError: dimension out of range (expected to be in range of [-1, 0], but got 1)

body:
When i using the cross-entropy loss as a loss function,i get this dimension out of range error.

This is my code:

self.ce = nn.CrossEntropyLoss()
    def forward(self, pred, y):
        loss = 0
        for w_, p_, y_ in zip(self.weights, pred, y):
            loss += w_ * self.ce(p_, y_)
        return loss

when i run this code :

the value of p_:tensor(1.00000e-02 *[-0.7625,  5.8737], device='cuda:0')
the value of w_:tensor(1., device='cuda:0')
the value of y_:tensor(0, device='cuda:0')


title: RuntimeError: Can only calculate the mean of floating types. Got Byte instead. for mean += images_data.mean(2).sum(0)

body:
I have the following pieces of code:
# Device configuration
device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
seed = 42
np.random.seed(seed)
torch.manual_seed(seed)

# split the dataset into validation and test sets
len_valid_set = int(0.1*len(dataset))
len_train_set = len(dataset) - len_valid_set

print("The length of Train set is {}".format(len_train_set))
print("The length of Test set is {}".format(len_valid_set))

train_dataset , valid_dataset,  = torch.utils.data.random_split(dataset , [len_train_set, len_valid_set])

# shuffle and batch the datasets
train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=8, shuffle=True, num_workers=4)
test_loader = torch.utils.data.DataLoader(valid_dataset, batch_size=8, shuffle=True, num_workers=4)

print("LOADERS",
    len(dataloader),
    len(train_loader),
    len(test_loader))

The length of Train set is 720
The length of Test set is 80
LOADERS 267 90 10
mean = 0.0
std = 0.0
nb_samples = 0.0
for data in train_loader:
    images, landmarks = data["image"], data["landmarks"]
    batch_samples = images.size(0)

    images_data = images.view(batch_samples, images.size(1), -1)
    mean += images_data.mean(2).sum(0)
    std += images_data.std(2).sum(0)
    nb_samples += batch_samples

mean /= nb_samples
std /= nb_samples

And I get this error:
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-23-9e47ddfeff5e> in <module>
      7
      8     images_data = images.view(batch_samples, images.size(1), -1)
----> 9     mean += images_data.mean(2).sum(0)
     10     std += images_data.std(2).sum(0)
     11     nb_samples += batch_samples

RuntimeError: Can only calculate the mean of floating types. Got Byte instead.

The fixed code is taken from https://stackoverflow.com/a/64349380/2414957 it worked for dataloader but not train_loader
Also, these are the results of
print(type(images_data))
print(images_data)

We have:
<class 'torch.Tensor'>
tensor([[[74, 74, 74,  ..., 63, 63, 63],
         [73, 73, 73,  ..., 61, 61, 61],
         [75, 75, 75,  ..., 61, 61, 61],
         ...,
         [74, 74, 74,  ..., 38, 38, 38],
         [75, 75, 75,  ..., 39, 39, 39],
         [72, 72, 72,  ..., 38, 38, 38]],

        [[75, 75, 75,  ..., 65, 65, 65],
         [75, 75, 75,  ..., 62, 62, 62],
         [75, 75, 75,  ..., 63, 63, 63],
         ...,
         [71, 71, 71,  ..., 39, 39, 39],
         [74, 74, 74,  ..., 38, 38, 38],
         [73, 73, 73,  ..., 37, 37, 37]],

        [[72, 72, 72,  ..., 62, 62, 62],
         [74, 74, 74,  ..., 63, 63, 63],
         [75, 75, 75,  ..., 61, 61, 61],
         ...,
         [74, 74, 74,  ..., 38, 38, 38],
         [74, 74, 74,  ..., 39, 39, 39],
         [73, 73, 73,  ..., 37, 37, 37]],

        ...,

        [[75, 75, 75,  ..., 63, 63, 63],
         [73, 73, 73,  ..., 63, 63, 63],
         [74, 74, 74,  ..., 62, 62, 62],
         ...,
         [74, 74, 74,  ..., 38, 38, 38],
         [73, 73, 73,  ..., 39, 39, 39],
         [73, 73, 73,  ..., 37, 37, 37]],

        [[73, 73, 73,  ..., 62, 62, 62],
         [75, 75, 75,  ..., 62, 62, 62],
         [74, 74, 74,  ..., 63, 63, 63],
         ...,
         [73, 73, 73,  ..., 39, 39, 39],
         [74, 74, 74,  ..., 38, 38, 38],
         [74, 74, 74,  ..., 38, 38, 38]],

        [[74, 74, 74,  ..., 62, 62, 62],
         [74, 74, 74,  ..., 63, 63, 63],
         [74, 74, 74,  ..., 62, 62, 62],
         ...,
         [74, 74, 74,  ..., 38, 38, 38],
         [73, 73, 73,  ..., 38, 38, 38],
         [72, 72, 72,  ..., 36, 36, 36]]], dtype=torch.uint8)

When I tried
images_data = images_data.float()
mean += images_data.mean(2).sum(0)

I didn't get a tensor for 3 values for mean and 3 values for std like I expected but got a very large tensor (each torch.Size([600]))


title: Data preprocessing for custom dataset in pytorch (transform.Normalize)

body:
I am new to Pytorch and CNN. I am kind of confused about Data Preprocessing. Not sure how to go about transform.Normalising the dataset (in essence how do you calculate mean and std v for your custom dataset ?)

I am loading my data using ImageFolder. The images are of different sizes.

train_transforms = transforms.Compose([transforms.Resize(size=224),
                                       transforms.ToTensor(),  transforms.Normalize((?), (?))
                                       ])
train_dataset = datasets.ImageFolder(root='roota/',
                                     transform=train_transforms)


title: Install pytorch from the source using pip

body:
I am trying to install pytorch on a remote server. It has CentOS 6.5 and according to this link
it has stopped support for CentOS 6. So, I am trying to install it via source.
The recommended method to install is by anaconda but the thing is I am getting plenty of issues while installing anaconda as it is messing with the remote server path's alot so I have decided to use pip. But I have issues regarding converting some conda commands in pip which are as below-

conda install -c pytorch magma-cuda90
The above command is mentioned before the pytorch cloning step and it given me an error that Could not open requirements file: [Errno 2] No such file or directory: 'pytorch'
The other issue I am facing is below-
export CMAKE_PREFIX_PATH=${CONDA_PREFIX:-"$(dirname $(which conda))/../"}
Wht should be the alternative for CMAKE_PREFIX_PATH in pip?


title: How do I list all currently available GPUs with pytorch?

body: I know I can access the current GPU using torch.cuda.current_device(), but how can I get a list of all the currently available GPUs?


title: Adding modules in Pytorch Custom Module

body:
Is it considered bad practice to add modules to a custom pytorch nn.Module using self.add_module()? All of the documentation seems to assign the layers to properties, then access them in the forward() method.

For example:

class ConvLayer(nn.Module):
    def __init__(self):
        self.add_module('conv',nn.Conv2d(...))
        self.add_module('bn',nn.BatchNorm2d(...))


title: How torch.Tensor.backward() works?

body:
I recently study Pytorch and backward function of the package.
I understood how to use it, but when I try

x = Variable(2*torch.ones(2, 2), requires_grad=True)
x.backward(x)
print(x.grad)

I expect

tensor([[1., 1.],
        [1., 1.]])

because it is an identity function. However, it returns

tensor([[2., 2.],
        [2., 2.]]).

Why this happens?


title: Pytorch dataloader for sentences

body:
I have collected a small dataset for binary text classification and my goal is to train a model with the method proposed by Convolutional Neural Networks for Sentence Classification

I started my implementation by using the torch.util.data.Dataset. Essentially every sample in my dataset my_data looks like this (as example):

{"words":[0,1,2,3,4],"label":1},
{"words":[4,9,20,30,4,2,3,4,1],"label":0}

Next I took a look at Writing custom dataloaders with pytorch:
using:

dataloader = DataLoader(my_data, batch_size=2,
                    shuffle=False, num_workers=4)

I would suspect that enumerating over a batch would yield something the following:

{"words":[[0,1,2,3,4],[4,9,20,30,4,2,3,4,1]],"labels":[1,0]}

However it is more like this:

{"words":[[0,4],[1,9],[2,20],[3,30],[4,4]],"label":[1,0]}

I guess it has something to do that they are not equal size.
Do they need to be the same size and if so how can i achieve it? For people knwoing about this paper, what does your training data look like?

edit:

class CustomDataset(Dataset):
def __init__(self, path_to_file, max_size=10, transform=None):

    with open(path_to_file) as f:
        self.data = json.load(f)
    self.transform = transform
    self.vocab = self.build_vocab(self.data)
    self.word2idx, self.idx2word = self.word2index(self.vocab)

def get_vocab(self):
    return self.vocab

def get_word2idx(self):
    return self.word2idx, self.idx2word

def __len__(self):
    return len(self.data)

def __getitem__(self, idx):
    if torch.is_tensor(idx):
        idx = idx.tolist()
    inputs_ = word_tokenize(self.data[idx][0])
    inputs_ = [w for w in inputs_ if w not in stopwords]
    inputs_ = [w for w in inputs_ if w not in punctuation]
    inputs_ = [self.word2idx[w] for w in inputs_]  # convert words to index

    label = {"positive": 1,"negative": 0}
    label_ = label[self.data[idx][1]] #convert label to 0|1

    sample = {"words": inputs_, "label": label_}

    if self.transform:
        sample = self.transform(sample)

    return sample

def build_vocab(self, corpus):
    word_count = {}
    for sentence in corpus:
        tokens = word_tokenize(sentence[0])
        for token in tokens:
            if token not in word_count:
                word_count[token] = 1
            else:
                word_count[token] += 1
    return word_count

def word2index(self, word_count):
    word_index = {w: i for i, w in enumerate(word_count)}
    idx_word = {i: w for i, w in enumerate(word_count)}
    return word_index, idx_word


I'm new to neural nets and am trying to train a simple neural net in Pytorch with labeled data as a classifier. The data contains 6 features and 3 labels. Unfortunately, when I run the code, the error code:

  Assertion `cur_target >= 0 && cur_target < n_classes' failed.

I've read all possible threads on this topic, and haven't found a solution that solves this problem for me.

My labels are 0, 1, and 2 and my feature data is numerical in the range of ~-12 to ~2000.

Any ideas?

#hyperparameters
hl = 10
lr = 0.01
num_epoch = 500

#build model
class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(6, hl)
        self.fc2 = nn.Linear(hl, 3)

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        return x
net = Net()

#choose optimizer and loss function
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(net.parameters(), lr=lr)

#train
for epoch in range(num_epoch):
    X = Variable(torch.Tensor(xtrain).float())
    Y = Variable(torch.Tensor(ytrain).long())

    #feedforward - backprop
    optimizer.zero_grad()
    out = net(X)
    loss = criterion(out, Y)
    loss.backward()
    optimizer.step()

    if (epoch) % 50 == 0:
        print ('Epoch [%d/%d] Loss: %.4f'
                   %(epoch+1, num_epoch, loss.data[0]))


title: How to read a ckpt file with python3, while it is saved using python2？

body:
I try to read a checkpoint file with pyTorch

checkpoint = torch.load(xxx.ckpt)

The file was generated by a program written using python 2.7. I try to read the file using python 3.6 but get the following error

UnicodeDecodeError: 'ascii' codec can't decode byte 0x8c in position 16: ordinal not in range(128)

Is it possible to read the file without downgrade python?


title: Convert pytorch tensor to opencv mat and vice versa in C++

body:
I want to convert pytorch tensors to opencv mat and vice versa in C++. I have these two functions:

cv::Mat TensorToCVMat(torch::Tensor tensor)
{
    std::cout << "converting tensor to cvmat\n";
    tensor = tensor.squeeze().detach().permute({1, 2, 0});
    tensor = tensor.mul(255).clamp(0, 255).to(torch::kU8);
    tensor = tensor.to(torch::kCPU);
    int64_t height = tensor.size(0);
    int64_t width = tensor.size(1);
    cv::Mat mat(width, height, CV_8UC3);
    std::memcpy((void *)mat.data, tensor.data_ptr(), sizeof(torch::kU8) * tensor.numel());
    return mat.clone();
}

torch::Tensor CVMatToTensor(cv::Mat mat)
{
    std::cout << "converting cvmat to tensor\n";
    cv::cvtColor(mat, mat, cv::COLOR_BGR2RGB);
    cv::Mat matFloat;
    mat.convertTo(matFloat, CV_32F, 1.0 / 255);
    auto size = matFloat.size();
    auto nChannels = matFloat.channels();
    auto tensor = torch::from_blob(matFloat.data, {1, size.height, size.width, nChannels});
    return tensor.permute({0, 3, 1, 2});
}

In my code I load two images (image1 and image2) and I want to convert them to pytorch tensors and then back to opencv mat to check if it works. The problem is that I get an memory access error on the first call of TensorToCVMat and I cant figure out whats wrong as I do not have much experience with C++ programming.

cv::Mat image1;
image1 = cv::imread(argv[1]);
if (!image1.data)
{
    std::cout << "no image data\n";
    return -1;
}
cv::Mat image2;
image2 = cv::imread(argv[2]);
if (!image2.data)
{
    std::cout << "no image data\n";
    return -1;
}

torch::Tensor tensor1 = CVMatToTensor(image1);
cv::Mat new_image1 = TensorToCVMat(tensor1); // <<< this is where the memory access error is thrown
torch::Tensor tensor2 = CVMatToTensor(image2);
cv::Mat new_image2 = TensorToCVMat(tensor2);

It would be great if you could give me hints or an explanation to solve this problem.


title: Unable to load model trained on Pytotch version 1.6.0 in Pytorch version 1.5.1

body:
I have recently updated my Pytorch version to 1.6.0 on my local machine to use their mixed-precision training, since then I am encountering this issue, I have tried the solution mentioned here, but it is still throwing below error.
RuntimeError: version_ <= kMaxSupportedFileFormatVersion INTERNAL ASSERT FAILED at /opt/conda/conda-bld/pytorch_1591914880026/work/caffe2/serialize/inline_container.cc:132, please report a bug to PyTorch. Attempted to read a PyTorch file with version 4, but the maximum supported version for reading is 3. Your PyTorch installation may be too old.

Link to reproduce: https://www.kaggle.com/rohitsingh9990/error-reproducing-code?scriptVersionId=37468859
Any help will be appreciated, thanks in advance.


title: No such operator torchvision::nms

body:
When I try to run yoloV3 detect,it happend the error

op = torch._C._jit_get_operation(qualified_op_name)

RuntimeError: No such operator torchvision::nms

Though this code is the source code of torchvision ,I try sevaral time to correct the code by the tips  with failure.


title: How to form a sequence of consecutive numbers in Pytorch?

body:
How to convert the Matlab code

v = [1: n]

to pytorch?

Writing a whole loop for that seems inefficient.

This should be straightforward to find, but I am facing difficulty searching the right keyword to get there.


title: Import error when importing torch

body:
I didn't make any change but it suddenly doesn't work.

>>> import torch
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/cheng/anaconda/envs/py36/lib/python3.6/site-packages/torch/__init__.py", line 56, in <module>
    from torch._C import *
ImportError: numpy.core.multiarray failed to import

I tried pip install numpy -I but it didn't work.

My OS is Mac OS X, and I tried to install by both pip and conda. Reinstall the env didn't work as well.

pytorch version 0.3.1

numpy version 1.14.2

Solution:

Do not name your file tokenize.py!


title: PyTorch: Loading word vectors into Field vocabulary vs. Embedding layer

body:
I'm coming from Keras to PyTorch. I would like to create a PyTorch Embedding layer (a matrix of size V x D, where V is over vocabulary word indices and D is the embedding vector dimension) with GloVe vectors but am confused by the needed steps.

In Keras, you can load the GloVe vectors by having the Embedding layer constructor take a weights argument:

# Keras code.
embedding_layer = Embedding(..., weights=[embedding_matrix])

When looking at PyTorch and the TorchText library, I see that the embeddings should be loaded twice, once in a Field and then again in an Embedding layer. Here is sample code that I found:

# PyTorch code.

# Create a field for text and build a vocabulary with 'glove.6B.100d'
# pretrained embeddings.
TEXT = data.Field(tokenize = 'spacy', include_lengths = True)

TEXT.build_vocab(train_data, vectors='glove.6B.100d')

# Build an RNN model with an Embedding layer.
class RNN(nn.Module):
    def __init__(self, ...):

        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embedding_dim)

        ...

# Initialize the embedding layer with the Glove embeddings from the
# vocabulary. Why are two steps needed???
model = RNN(...)
pretrained_embeddings = TEXT.vocab.vectors
model.embedding.weight.data.copy_(pretrained_embeddings)

Specifically:

Why are the GloVe embeddings loaded in a Field in addition to the Embedding?
I thought the Field function build_vocab() just builds its vocabulary from the training data. How are the GloVe embeddings involved here during this step?

Here are other StackOverflow questions that did not answer my questions:

PyTorch / Gensim - How to load pre-trained word embeddings

Embedding in pytorch

PyTorch LSTM - using word embeddings instead of nn.Embedding()

Thanks for any help.


title: Measuring F1 score for multiclass classification natively in PyTorch

body:
I am trying to implement the macro F1 score (F-measure) natively in PyTorch instead of using the already-widely-used sklearn.metrics.f1_score in order to calculate the measure directly on the GPU.

From what I understand, in order to compute the macro F1 score, I need to compute the F1 score with the sensitivity and precision for all labels, then take the average of all these.

My attempt

My current implementation looks like this:

def confusion_matrix(y_pred: torch.Tensor, y_true: torch.Tensor, n_classes: int):
    conf_matrix = torch.zeros([n_classes, n_classes], dtype=torch.int)
    y_pred = torch.argmax(y_pred, 1)
    for t, p in zip(y_true.view(-1), y_pred.view(-1)):
        conf_matrix[t.long(), p.long()] += 1
    return conf_matrix

def forward(self, y_pred: torch.Tensor, y_true: torch.Tensor) -> torch.Tensor:
    conf_matrix = confusion_matrix(y_pred, y_true, self.classes)
    TP = conf_matrix.diag()
    f1_scores = torch.zeros(self.classes, dtype=torch.float)
    for c in range(self.classes):
        idx = torch.ones(self.classes, dtype=torch.long)
        idx[c] = 0
        FP = conf_matrix[c, idx].sum()
        FN = conf_matrix[idx, c].sum()
        sensitivity = TP[c] / (TP[c] + FN + self.epsilon)
        precision = TP[c] / (TP[c] + FP + self.epsilon)
        f1_scores[c] += 2.0 * ((precision * sensitivity) / (precision + sensitivity + self.epsilon))
    return f1_scores.mean()

self.classes is the number of labels and self.epsilon is a very small value set to 10-e12 which prevents DivisionByZeroError.

When training, I compute the measure for every batch and take the average of all measures as the final score.

Problem

The problem is that when I compare my custom F1 score with sklearn's macro F1 score, they are rarely equal.

# example 1
eval_cce 0.5203, eval_f1 0.8068, eval_acc 81.5455, eval_f1_sci 0.8023,
test_cce 0.4784, test_f1 0.7975, test_acc 82.6732, test_f1_sci 0.8097
# example 2
eval_cce 0.3304, eval_f1 0.8211, eval_acc 87.4955, eval_f1_sci 0.8626,
test_cce 0.3734, test_f1 0.8183, test_acc 85.4996, test_f1_sci 0.8424
# example 3
eval_cce 0.4792, eval_f1 0.7982, eval_acc 81.8482, eval_f1_sci 0.8001,
test_cce 0.4722, test_f1 0.7905, test_acc 82.6533, test_f1_sci 0.8139

While I have tried to scan the internet, most cases cover binary classification. I have yet been able to discover an example to attempts to do what I am trying to.

My Question

Is there any obvious issue with my attempt?

Update (10.06.2020)

I have yet to figure out my mistake. Due to time constraint, I decided to just use the F1 macro score provided by sklearn. While it cannot work directly with GPU tensors, it is fast enough for my case anyway.

However, it would be awesome if anybody can figure this out, so that anybody else that might stumble upon this issue can get their problem resolved.


title: Why is torch.nn.Sigmoid a class instead of a method?

body:
I'm trying to understand how pytorch works a little bit better. Usually, when defining a neural network class, in the init() constructor, people write self.sigmoid = nn.Sigmoid(), so that in the forward() method they can call the sigmoid function multiple times with having to reinstantiate nn.Sigmoid() every time.

But why isn't nn.Sigmoid just a method to begin with, instead of a class?

Also, I was curious what to refer to 'nn' in torch.nn as (package? library?).

Thanks!


title: huggingface transformers: truncation strategy in encode_plus

body:
encode_plus in huggingface's transformers library allows truncation of the input sequence. Two parameters are relevant: truncation and max_length. I'm passing a paired input sequence to encode_plus and need to truncate the input sequence simply in a "cut off" manner, i.e., if the whole sequence consisting of both inputs text and text_pair is longer than max_length it should just be truncated correspondingly from the right.
It seems that neither of the truncation strategies allows to do this, instead longest_first removes tokens from the longest sequence (which could be either text or text_pair, but not just simply from the right or end of the sequence, e.g., if text is longer that text_pair, it seems this would remove tokens from text first), only_first and only_second remove tokens from only the first or second (hence, also not simply from the end), and do_not_truncate does not truncate at all. Or did I misunderstood this and actually longest_first might be what I'm looking for?


title: Cannot import torch - Image not found

body:
I'm trying to import torch but failed because of Image not Found error. Here is the error when I entered import torch:

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
 in
----> 1 import torch

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/torch/__init__.py in
    134     # See Note [Global dependencies]
    135     _load_global_deps()
--> 136     from torch._C import *
    137
    138 __all__ += [name for name in dir(_C)

ImportError: dlopen(/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/torch/_C.cpython-38-darwin.so, 2): Library not loaded: @rpath/libc++.1.dylib
  Referenced from: /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/torch/_C.cpython-38-darwin.so
  Reason: image not found

Thank you very much!

EDIT: This works for me:

>>> install_name_tool -add_rpath /usr/lib /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/torch/_C.cpython-38-darwin.so


title: How to set different learning rate for different layer in pytorch?

body:
I'm doing fine-tuning with pytorch using resnet50 and want to set the learning rate of the last fully connected layer to 10^-3 while the learning rate of other layers be set to 10^-6. I know that I can just follow the method in its document:
optim.SGD([{'params': model.base.parameters()},
           {'params': model.classifier.parameters(), 'lr': 1e-3}],
          lr=1e-2, momentum=0.9)

But is there anyway that I do not need to set the parameters layer by layer


title: How to check if a tensor is on cuda in Pytorch?

body:
I have a tensor
t = torch.zeros((4, 5, 6))
How to check if it is on gpu or not?


title: Conda package install [Errno 13] Permission denied while installing conda-forge::protobuf-3.8.0

body:
I have a conda environment with Python 3.6 and something went wrong with my Pytorch installation so I tried to install it again.  Towards the end of the installation I get this error:

ERROR conda.core.link:_execute(700): An error occurred while installing package 'conda-forge::protobuf-3.8.0-py36h6de7cb9_1'.
Rolling back transaction: done

[Errno 13] Permission denied: '/Users/myusername/anaconda3/envs/torch/lib/python3.6/site-packages/google/protobuf/__init__.py'
()

Also, it says "The environment is inconsistent" which is probably a clue something was already wrong.

Full details:

$ conda install pytorch torchvision -c pytorch
Collecting package metadata (current_repodata.json): done
Solving environment: -
The environment is inconsistent, please check the package plan carefully
The following packages are causing the inconsistency:

  - conda-forge/osx-64::tensorboard==1.14.0=py36_0
  - conda-forge/noarch::tensorboardx==1.9=py_0
done

## Package Plan ##

  environment location: /Users/myusername/anaconda3/envs/torch

  added / updated specs:
    - pytorch
    - torchvision

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    intel-openmp-2019.4        |              233         887 KB
    mkl-2019.4                 |              233       101.9 MB
    pytorch-1.3.1              |          py3.6_0        32.7 MB  pytorch
    torchvision-0.4.2          |         py36_cpu         5.9 MB  pytorch
    ------------------------------------------------------------
                                           Total:       141.4 MB

The following NEW packages will be INSTALLED:

  intel-openmp       pkgs/main/osx-64::intel-openmp-2019.4-233
  libprotobuf        conda-forge/osx-64::libprotobuf-3.8.0-hfbae3c0_0
  mkl                pkgs/main/osx-64::mkl-2019.4-233
  protobuf           conda-forge/osx-64::protobuf-3.8.0-py36h6de7cb9_1
  pytorch            pytorch/osx-64::pytorch-1.3.1-py3.6_0
  torchvision        pytorch/osx-64::torchvision-0.4.2-py36_cpu

The following packages will be UPDATED:

  openssl                                 1.1.1c-h01d97ff_0 --> 1.1.1d-h0b31af3_0

Proceed ([y]/n)? y

Downloading and Extracting Packages
pytorch-1.3.1        | 32.7 MB   | ##################################### | 100%
torchvision-0.4.2    | 5.9 MB    | ##################################### | 100%
mkl-2019.4           | 101.9 MB  | ##################################### | 100%
intel-openmp-2019.4  | 887 KB    | ##################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
ERROR conda.core.link:_execute(700): An error occurred while installing package 'conda-forge::protobuf-3.8.0-py36h6de7cb9_1'.
Rolling back transaction: done

[Errno 13] Permission denied: '/Users/myusername/anaconda3/envs/torch/lib/python3.6/site-packages/google/protobuf/__init__.py'
()

UPDATE:
I tried uninstalling tensorboard and tensorboardx but the same error occurs preventing the uninstall.

ALSO:

Tried pip uninstall protobuf but that caused a similar error:

Uninstalling protobuf-3.10.0:
  Would remove:
...

ERROR: Exception:
Traceback (most recent call last):
  File "/Users/myusername/anaconda3/envs/torch/lib/python3.6/shutil.py", line 544, in move
    os.rename(src, real_dst)
PermissionError: [Errno 13] Permission denied: '/Users/myusername/anaconda3/envs/torch/lib/python3.6/site-packages/google/protobuf/' -> '/Users/myusername/anaconda3/envs/torch/lib/python3.6/site-packages/google/~-otobuf'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/myusername/anaconda3/envs/torch/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 153, in _main
    status = self.run(options, args)
  File "/Users/myusername/anaconda3/envs/torch/lib/python3.6/site-packages/pip/_internal/commands/uninstall.py", line 79, in run
    auto_confirm=options.yes, verbose=self.verbosity > 0,
  File "/Users/myusername/anaconda3/envs/torch/lib/python3.6/site-packages/pip/_internal/req/req_install.py", line 755, in uninstall
    uninstalled_pathset.remove(auto_confirm, verbose)
  File "/Users/myusername/anaconda3/envs/torch/lib/python3.6/site-packages/pip/_internal/req/req_uninstall.py", line 394, in remove
    moved.stash(path)
  File "/Users/myusername/anaconda3/envs/torch/lib/python3.6/site-packages/pip/_internal/req/req_uninstall.py", line 283, in stash
    renames(path, new_path)
  File "/Users/myusername/anaconda3/envs/torch/lib/python3.6/site-packages/pip/_internal/utils/misc.py", line 338, in renames
    shutil.move(old, new)
  File "/Users/myusername/anaconda3/envs/torch/lib/python3.6/shutil.py", line 556, in move
    rmtree(src)
  File "/Users/myusername/anaconda3/envs/torch/lib/python3.6/shutil.py", line 494, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "/Users/myusername/anaconda3/envs/torch/lib/python3.6/shutil.py", line 389, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "/Users/myusername/anaconda3/envs/torch/lib/python3.6/shutil.py", line 387, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [Errno 13] Permission denied: '/Users/myusername/anaconda3/envs/torch/lib/python3.6/site-packages/google/protobuf/descriptor.py'

I have no idea how to fix this other than deleting the whole environment and start again.


title: 'int' object has no attribute 'size'"

body:
F.nll_loss: I am getting

  AttributeError: 'int' object has no attribute 'size'

when I try to run this code. I also get a snippet of the module code.

  raise ValueError('Expected 2 or more dimensions (got {})'.format(dim))
      if input.size(0) != target.size(0):
        raise ValueError('Expected input batch_size ({}) to match target batch_size ({}).'
        format(input.size(0), target.size(0)))

import torch
from torchvision import transforms, datasets
import torch.nn as nn
import torch.nn.functional as F
import matplotlib.pylab as plt

train_dataset = datasets.MNIST(root = '', train =True, download = True,
                                transform =transforms.Compose([transforms.ToTensor()]))

test_dataset = datasets.MNIST(root ='', download =True, train =False,
                               transform =transforms.Compose([transforms.ToTensor()]))
batch_size = 10
train_loader = torch.utils.data.DataLoader(train_dataset, batch_size, shuffle =True)

test_dataset = torch.utils.data.DataLoader(test_dataset, batch_size, shuffle =True)
class Net(nn.Module):

  def __init__(self):
    super().__init__()
    self.fc1 = nn.Linear(28*28, 64)
    self.fc2 = nn.Linear(64,64)
    self.fc3 = nn.Linear(64,64)
    self.fc4 = nn.Linear(64,10)

  def forward(self, x):
      x = F.relu(self.fc1(x))
      x = F.relu(self.fc2(x))
      x = F.relu(self.fc2(x))
      x = self.fc4(x)
      return F.log_softmax(x, dim=1)

x=torch.rand((28,28))
x=x.view(-1,28*28)
net =Net()
out=net(x)
out
import torch.optim as optim

optimizer =optim.Adam(net.parameters(), lr=0.001)

EPOCHS = 3
for epoch in range(EPOCHS):
  for data in train_dataset:
      x, y = data
      net.zero_grad()
      x=x.view(-1, 28*28)
      output = net(x)
      loss = F.nll_loss(output, y)
      loss.backward()
      optimizer.step()
  print(loss)


title: How to batch convert sentence lengths to masks in PyTorch?

body:
For example, from

lens = [3, 5, 4]

we want to get

mask = [[1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1],
        [1, 1, 1, 1, 0]]

Both of which are torch.LongTensors.


I am working on Nvidia Jetson Tx2 (with JETPACK 4.2) and installed the pytorch following this link.
When I am importing torch in python its giving me an error OSError: libcurand.so.10: cannot open shared object file: No such file or directory I have tried all the options but nothing worked.
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-10.0/lib64
export LIBRARY_PATH=$LIBRARY_PATH:/usr/local/cuda-10.0/lib64
export PATH=$PATH:/usr/local/cuda-10.0/lib64

Any guidance to debug the issue is requested. Thanks


title: Trouble with nn.embedding in pytorch, expected scalar type Long, but got torch.cuda.FloatTensor (how to fix)?

body:
so I have an RNN encoder that is part of a larger language model, where the process is encode -> rnn -> decode.
As part of my __init__ for my rnn class I have the following:
self.encode_this = nn.Embedding(self.vocab_size, self.embedded_vocab_dim)

now I am trying to implement a forward class, which takes in batches and performs encoding then decoding,
def f_calc(self, batch):
    #Here, batch.shape[0] is the size of batch while batch.shape[1] is the sequence length

    hidden_states = (torch.zeros(self.num_layers, batch.shape[0], self.hidden_vocab_dim).to(device))
    embedded_states = (torch.zeros(batch.shape[0],batch.shape[1], self.embedded_vocab_dim).to(device))

    o1, h = self.encode_this(embedded_states)

however, my problem is always with the encoder which gives me the following error:
/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
   1465         # remove once script supports set_grad_enabled
   1466         _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-> 1467     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
   1468
   1469

RuntimeError: Expected tensor for argument #1 'indices' to have scalar type Long; but got torch.cuda.FloatTensor instead (while checking arguments for embedding)

Anyone have any idea how to fix? I am completely new to pytorch so please excuse me if this is a stupid question. I know there is some form of type casting involved but I am not sure how to go about doing it...
much appreciated!


I want to utilize deep neural network to classify Hyperspectral Image. But every time I run this code, it gives me this error "TypeError: forward() missing 1 required positional argument: 'negative'".
Code show as below(Not completed):

import numpy as np
import scipy.io as sio
from tqdm import tqdm
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import math

REBUILD_DATA = True

TO read data:

class DATA():
    # 读取样本和标签，并转换为numpy数组格式
    Pavia = sio.loadmat('G:\研究生\Matlab_code\dataset\Classification\paviaU.mat')
    PaviaGT = sio.loadmat('G:\研究生\Matlab_code\dataset\Classification\paviaU_GT.mat')
    # print(sorted(Pavia.keys()))   返回字典中的键值key()
    # print(sorted(PaviaGT.keys()))

    Sample = Pavia['data']
    Sample = np.array(Sample, dtype = np.int32)

    Label = PaviaGT['groundT']
    Label = np.array(Label, dtype = np.int32)

    # 将样本每一维度的数值存到a,b,c中，以便后续使用
    [a,b,c]=Sample.shape

    # 将数据reshape成matlab中的格式
    SampleT = Sample.transpose(1, 0, 2)
    SampleX = SampleT.reshape(-1,103)
    """ sio.savemat('G:\研究生\Sample.mat',{'dataX':SampleX})  """
    LabelT = Label.transpose(1,0)
    Label = LabelT.reshape(-1,1)
    # 如何将样本和标签合并，输入神经网络的数据为[-1,band]
    """ sio.savemat('G:\研究生\Label.mat',{'LabelX':Label}) """
    totalcount = np.zeros((10,1),dtype = np.int32)
    trainset = []
    testset = []

    # 将样本和标签合并
    def integrated_data(self):
        rebuilddata = []
        for i in range(0,self.a*self.b):
            rebuilddata.append([np.array(self.SampleX[i]),np.array(self.Label[i])])
            for j in range(0,10):
                if self.Label[i] == j:
                    self.totalcount[j] += 1
        rebuilddata = np.array(rebuilddata)
        return rebuilddata

    # 并制作训练和测试数据
    def make_trainset_and_testset(self, rebuilddata, ratio):
        TrainIndex = []
        TestIndex = []

        # 取出每一类的训练集和测试集坐标
        for i in range(1,np.max(self.Label)+1):
            class_coor = np.argwhere(self.Label == i)
            index = class_coor[:,0].tolist()
            np.random.shuffle(index)
            VAL_SIZE = int(np.floor(len(index)*ratio))
            ClassTrainIndex = index[:VAL_SIZE]
            ClassTestIndex = index[-VAL_SIZE:]
            TrainIndex += ClassTrainIndex
            TestIndex += ClassTestIndex

        # 返回训练集和测试集样本
        TrainSample = rebuilddata[TrainIndex]
        TestSample = rebuilddata[TestIndex]
        return TrainIndex,TestIndex,TrainSample,TestSample

This is my dnn module:

class DNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(103, 500)
        self.fc2 = nn.Linear(500, 256)
        self.fc3 = nn.Linear(256, 9)

    def forward(self,x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return F.softmax(x,dim=1)

The training and testing function:

def train(dnn):
    BATCH_SIZE = 100
    EPOCHS = 3
    for epoch in range(EPOCHS):
        for i in tqdm(range(0, len(train_X), BATCH_SIZE)):
            batch_X = train_X[i:i+BATCH_SIZE]
            batch_y = train_y[i:i+BATCH_SIZE]

            dnn.zero_grad()
            outputs = dnn(batch_X)
            loss = loss_function(outputs, batch_y)
            loss.backward()
            optimizer.step()
        print(loss)

 def test(net):
     correct = 0
     total = 0
     with torch.no_grad():
         for i in tqdm(range(len(test_X))):
             real_class = torch.argmax(test_y[i]).to(device)
             net_out = dnn(test_X[i].view(-1, 1, 50, 50).to(device))[0]

             predicted_class = torch.argmax(net_out)
             if predicted_class == real_class:
                 correct += 1
             total += 1
     print("Accuracy:", round(correct/total,3))

if REBUILD_DATA:
    Data = DATA()
    datay = Data.integrated_data()
    Trainindex, Testindex, TrainSet, TestSet = Data.make_trainset_and_testset(rebuilddata=datay,ratio=0.1)

train_X = torch.Tensor([i[0] for i in TrainSet])
train_y = torch.Tensor([i[1] for i in TrainSet])
train_X = train_X/3000
test_X = torch.Tensor([i[0] for i in TestSet])
test_y = torch.Tensor([i[1] for i in TestSet])

print(train_X[0])
dnn = DNN()
optimizer = optim.SGD(dnn.parameters(), lr = 0.001)
loss_function = nn.TripletMarginLoss()
train(dnn)


title: How to initialize empty tensor with certain dimension and append to it through a loop without CUDA out of memory?

body:
I am trying to append tensors (t) generated in a for-loop to a list [T] that accumulates all these tensors. Next, the list [T] requires to be converted into a tensor and needs to be loaded onto GPU.

        b_output = []

        for eachInputId, eachMask in zip(b_input_ids, b_input_mask):
            # unrolled into each individual document
            # print(eachInputId.size()) # individual document here

            outputs = model(eachInputId,
                    token_type_ids=None,
                    attention_mask=eachMask)

            # combine the [CLS] output layer to form the document
            doc_output = torch.mean(outputs[1], dim=0) # size = [1, ncol]

            b_output.append( doc_output )

        t_b_output = torch.tensor( b_output )

Another method that I tried was initializing a tensor {T} with fixed dimensions and appending the tensors (t) to it from the for-loop.

        b_output = torch.zeros(batch_size, hidden_units)
        b_output.to(device) # cuda device

        for index, (eachInputId, eachMask) in enumerate(zip(b_input_ids, b_input_mask)):
            # unrolled into each individual document
            # print(eachInputId.size()) # individual document here

            outputs = model(eachInputId,
                    token_type_ids=None,
                    attention_mask=eachMask)

            # combine the [CLS] output layer to form the document
            doc_output = torch.mean(outputs[1], dim=0) # size = [1, ncol]

            b_output[index] = doc_output

Doing either of this produces this error:

RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 11.17 GiB total capacity; 10.65 GiB already allocated; 2.81 MiB free; 10.86 GiB reserved in total by PyTorch)

I assume this is because of appending the tensors (that are on the GPU) to a list (of course not on the GPU) and then trying to convert the list into a tensor (thats not on the GPU).

What could be done to append those tensors to another tensor and then load the tensor to GPU for further processing?

I will be grateful for any hint or information.


title: version `GLIBC_2.7' not found when import pytorch

body:
I have downloaded torch-0.1.12.post2-cp27-none-linux_x86_64.whl(from http://download.pytorch.org/whl/cu75/torch-0.1.12.post2-cp27-none-linux_x86_64.whl ) file and use the following command to install.

pip install torch-0.1.12.post2-cp27-none-linux_x86_64.whl
pip install pip install torchvision

However, the error occurs when import torch

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/my/lib/python2.7/site-packages/torch/__init__.py", line 53, in <module>
    from torch._C import *
ImportError: /lib64/tls/libc.so.6: version `GLIBC_2.7' not found (required by /home/my/lib/python2.7/site-packages/torch/lib/libTH.so.1)

My Linux kernel is 2.6.32_1-19-0-0 and GCC version is 4.8.3

How can I fix this problem?


title: torch.nn has no attribute named upsample

body:
Following this tutorial: https://www.digitalocean.com/community/tutorials/how-to-perform-neural-style-transfer-with-python-3-and-pytorch#step-2-%E2%80%94-running-your-first-style-transfer-experiment

When I run the example in Jupyter notebook, I get the following:

So, I've tried troubleshooting, which eventually got me to running it as per the github example (https://github.com/zhanghang1989/PyTorch-Multi-Style-Transfer) says to via command line:

python main.py eval --content-image images/content/venice-boat.jpg --style-image images/21styles/candy.jpg --model models/21styles.model --content-size 1024 --cuda=0

However, this gives the following error:

Traceback (most recent call last):
File "main.py", line 287, in <module>
main()
File "main.py", line 44, in main
evaluate(args)
File "main.py", line 242, in evaluate
stylemodel = Net(ngf=args.ngf)
File "/root/styletransfer/PyTorch-Style-Transfer/experiments/net.py", line 284, in init
model += [upblock(ngf*expansion, 32, 2, normlayer),
File "/root/styletransfer/PyTorch-Style-Transfer/experiments/net.py", line 127, in init
kernelsize=1, stride=1, upsample=stride)
File "/root/styletransfer/PyTorch-Style-Transfer/experiments/net.py", line 167, in init
self.upsamplelayer = torch.nn.Upsample(scalefactor=upsample)
AttributeError: module 'torch.nn' has no attribute 'Upsample'

I can't figure out how to get past this?


title: Trouble using transforms.FiveCrop()/TenCrop() in PyTorch

body:
I am trying to increase my CNN’s performance and thus i decided to “play” with some transformations in order to see how they affect my model. I read that FiveCrop() and TenCrop() might help because they generate extra data to train on. However, when i try to train the model, using one of the transformations mentioned above, i get the following error:

TypeError: pic should be PIL Image or ndarray. Got < class ‘tuple’>

The documentation of those transformations, only states a note for the test procedure, any idea how to fix this?

Thanks in advance!

train_transform = transforms.Compose(
    [transforms.ColorJitter(),
     transforms.TenCrop(32),
     transforms.ToTensor(),
     transforms.Normalize((0.4914, 0.4822, 0.4465), (0.247, 0.243, 0.261))
     ])

train = datasets.CIFAR10(root = './data',
                  train = True,
                  transform = train_transform,
                  download = True)

train_loader = torch.utils.data.DataLoader(dataset = train,
                                      batch_size = 1250,
                                      shuffle = True)

for epoch in range(num_of_iterations):
  correct = 0
  acc = 0.0
  running_loss = 0.0
  for i, (images, labels) in enumerate(train_loader):
      images = images.requires_grad_().to(device)
      labels = labels.to(device)

The error occurs on the line of the second for-loop


title: Pytorch DataLoader - Choose Class STL10 Dataset

body:
Is it possible to pull only where class = 0 in the STL10 dataset in PyTorch torchvision? I am able to check them in a loop, but need to receive batches of class 0 images

# STL10 dataset
train_dataset = torchvision.datasets.STL10(root='./data/',
                                           transform=transforms.Compose([
                                               transforms.Grayscale(),
                                               transforms.ToTensor()
                                           ]),
                                           split='train',
                                           download=True)

# Data loader
train_loader = torch.utils.data.DataLoader(dataset=train_dataset,
                                           batch_size=batch_size,
                                           shuffle=True)

for i, (images, labels) in enumerate(train_loader):
    if labels[0] == 0:...

edit based on iacolippo's answer - this is now working:

# Set params
batch_size = 25
label_class = 0   # only airplane images

# Return only images of certain class (eg. airplanes = class 0)
def get_same_index(target, label):
    label_indices = []

    for i in range(len(target)):
        if target[i] == label:
            label_indices.append(i)

    return label_indices

# STL10 dataset
train_dataset = torchvision.datasets.STL10(root='./data/',
                                           transform=transforms.Compose([
                                               transforms.Grayscale(),
                                               transforms.ToTensor()
                                           ]),
                                           split='train',
                                           download=True)

# Get indices of label_class
train_indices = get_same_index(train_dataset.labels, label_class)

# Data loader
train_loader = torch.utils.data.DataLoader(dataset=train_dataset,
                                           batch_size=batch_size,
                                           sampler=torch.utils.data.sampler.SubsetRandomSampler(train_indices))


title: what is the inputs to a torch.nn.gru function in pytorch?

body:
I am using a gru function to implement a RNN. This RNN (GRU) is used after some CNN layers. Can someone please tell me what is the input to a GRU function here? Especially, is the hidden size fixed?

self.gru = torch.nn.GRU(
            input_size=input_size,
            hidden_size=128,
            num_layers=1,
            batch_first=True,
            bidirectional=True)

According to my understanding the input size will be the number of features and the hidden size for GRU is always fixed as 128? Can some one please correct me. Or give their feedback


title: ValueError: Error initializing torch.distributed using env:// rendezvous: environment variable MASTER_ADDR expected, but not set

body:
I am not able to initialize the group process in PyTorch for BERT model
I had tried to initialize using following code:

import torch
import datetime

torch.distributed.init_process_group(
    backend='nccl',
    init_method='env://',
    timeout=datetime.timedelta(0, 1800),
    world_size=0,
    rank=0,
    store=None,
    group_name=''
)

and tried to access the get_world_size() function:

num_train_optimization_steps = num_train_optimization_steps // torch.distributed.get_world_size()

full code:

train_examples = None
    num_train_optimization_steps = None
    if do_train:
        train_examples = processor.get_train_examples(data_dir)
        num_train_optimization_steps = int(
            len(train_examples) / train_batch_size / gradient_accumulation_steps) * num_train_epochs
        if local_rank != -1:
            import datetime
            torch.distributed.init_process_group(backend='nccl',init_method='env://', timeout=datetime.timedelta(0, 1800), world_size=0, rank=0, store=None, group_name='')
            num_train_optimization_steps = num_train_optimization_steps // torch.distributed.get_world_size()
            print(num_train_optimization_steps)


title: How to get ouput from a particular layer from pretrained CNN in pytorch

body:
I have pretrained CNN (RESNET18) on imagenet dataset , now what i want is to get output of my input image from a particular layer,

for example.

my input image is of FloatTensor(3, 224, 336) and i send a batch of size = 10 in my resnet model , now what i want is the output returned by model.layer4,

Now what i tried is out = model.layer4(Variable(input)) but it gave me input dimensions mismatch error(As expected), this is the exact error returned

RuntimeError: Need input of dimension 4 and input.size[1] == 64 but got input to be of shape: [10 x 3 x 224 x 336] at /Users/soumith/miniconda2/conda-bld/pytorch_1501999754274/work/torch/lib/THNN/generic/SpatialConvolutionMM.c:47

So i'm confused , how to proceed now to get my layer4 output

PS: My ultimate task is to combine layer4 output and fullyconnected layer output together (Tweeking in CNN, kind of gated CNN ), so if anyone have any insight in this case  then please do tell me, maybe my above approach is not right


title: PyTorch conversion between tensor and numpy array: the addition operation

body:
I am following the 60-minute blitz on PyTorch but have a question about conversion of a numpy array to a tensor. Tutorial example here.
This piece of code:
import numpy as np
a = np.ones(5)
b = torch.from_numpy(a)
np.add(a, 1, out=a)
print(a)
print(b)

yields

[2. 2. 2. 2. 2.]
tensor([2., 2., 2., 2., 2.], dtype=torch.float64)

However
import numpy as np
a = np.ones(5)
b = torch.from_numpy(a)
a = a + 1 #the diff is here
print(a)
print(b)

yields

[2. 2. 2. 2. 2.]
tensor([1., 1., 1., 1., 1.], dtype=torch.float64)

Why are the outputs different?


title: backward, grad function in pytorch

body:
I'm trying to implement backward, grad function in pytorch.

But, I don't know why this value is returned.

Here is my code.

x = Variable(torch.FloatTensor([[1,2],[3,4]]), requires_grad=True)
y = x + 2
z = y * y

gradient = torch.ones(2, 2)
z.backward(gradient)
print(x.grad)

I think that result value should be [[6,8],[10,12]]

Because of dz/dx= 2*(x+2) and x=1,2,3,4

But returned value is [[7,9],[11,13]]

Why this is happened.. I want to know how gradient, grad function is doing.

Help me please..


title: Correct data loading, splitting and augmentation in Pytorch

body:
The tutorial doesn't seem to explain how we should load, split and do proper augmentation.

Let's have a dataset consisting of cars and cats. The folder structure would be:

data
  cat
    0101.jpg
    0201.jpg
    ...
  dogs
    0101.jpg
    0201.jpg
    ...

At first, I loaded the dataset by datasets.ImageFolder function. Image Function has command "TRANSFORM" where we can set some augmentation commands, but we don't want to apply augmentation to test dataset! So let's stay with transform=None.

data = datasets.ImageFolder(root='data')

Apparently, we don't have folder structure train and test and therefore I assume a good approach would be to use split_dataset function

    train_size = int(split * len(data))
    test_size = len(data) - train_size
    train_dataset, test_dataset = torch.utils.data.random_split(data, [train_size, test_size])

Now let's load the data the following way.

    train_loader = torch.utils.data.DataLoader(train_dataset,
                                              batch_size=8,
                                              shuffle=True)
    test_loader = torch.utils.data.DataLoader(test_dataset,
                                              batch_size=8,
                                              shuffle=True)

How can I apply transformations (data augmentation) to the "train_loader" images?

Basically I need to: 1. load data from the folder structure explained above
2. split the data into test/train parts
3. apply augmentations on train part.


title: How to input a matrix to CNN in pytorch

body:
I'm very new to pytorch and I want to figure out how to input a matrix rather than image into CNN.
I have try it in the following way, but some errors occur.
I define my dataset as following:

class FrameDataSet(tud.Dataset):
    def __init__(self, data):
        targets = data['class'].values.tolist()
        features = data.drop('class', axis=1).astype(np.int64).values

        self.datalist = features.reshape((-1, feature_num, frame_size))
        self.labellist = targets

    def __getitem__(self, index):
        return torch.Tensor(self.datalist[index].astype(float)), self.labellist[index]

    def __len__(self):
        return self.datalist.shape[0]

And my CNN is:

self.conv = nn.Sequential(
        nn.Conv2d(1, 12, 3),
        nn.ReLU(True),
        nn.MaxPool2d(3, 3))
self.fc1 = nn.Linear(80, 100)
self.fc2 = nn.Linear(100, 30)
self.fc3 = nn.Linear(30, 5)

But when the data was inputted to CNN, the error brings:

  File "/home/sparks/anaconda2/lib/python2.7/site-packages/torch/nn/functional.py", line 48, in conv2d
      raise ValueError("Expected 4D tensor as input, got {}D tensor instead.".format(input.dim()))
  Expected 4D tensor as input, got 3D tensor instead.


title: How to enable Dict/OrderedDict/NamedTuple support in pytorch 1.1.0 JIT compiler?

body:
From the release highlight of pytorch 1.1.0. It appears that the latest JIT compiler now supports Dict type. (Source: https://jaxenter.com/pytorch-1-1-158332.html)

    Dictionary and list support in TorchScript: Lists and dictionary types behave like Python lists and dictionaries.

Unfortunately I can't find a way to make this improvement to work properly. The following code is a simple example of exporting a Feature Pyramid Network (FPN) into tensorboard, which uses the JIT compiler:

from collections import OrderedDict

import torch
import torchvision
from torch.utils.tensorboard import SummaryWriter

torchWriter = SummaryWriter(log_dir=".tensorboard/example1")

m = torchvision.ops.FeaturePyramidNetwork([10, 20, 30], 5)
# get some dummy data
x = OrderedDict()
x['feat0'] = torch.rand(1, 10, 64, 64)
x['feat2'] = torch.rand(1, 20, 16, 16)
x['feat3'] = torch.rand(1, 30, 8, 8)
# compute the FPN on top of x
output = m.forward(x)
print([(k, v.shape) for k, v in output.items()])

torchWriter.add_graph(m, input_to_model=x)

When I run it I got the following error:

Traceback (most recent call last):
  File "/home/shared/virtualenv/dl-torch/lib/python3.7/site-packages/torch/utils/tensorboard/_pytorch_graph.py", line 276, in graph
    trace, _ = torch.jit.get_trace_graph(model, args)
  File "/home/shared/virtualenv/dl-torch/lib/python3.7/site-packages/torch/jit/__init__.py", line 231, in get_trace_graph
    return LegacyTracedModule(f, _force_outplace, return_inputs)(*args, **kwargs)
  File "/home/shared/virtualenv/dl-torch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/shared/virtualenv/dl-torch/lib/python3.7/site-packages/torch/jit/__init__.py", line 284, in forward
    in_vars, in_desc = _flatten(args)
RuntimeError: Only tuples, lists and Variables supported as JIT inputs, but got collections.OrderedDict

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/peng/git-drone/gate_detection/python/gate_detection/errorcase/tb.py", line 36, in <module>
    torchWriter.add_graph(m, input_to_model=x)
  File "/home/shared/virtualenv/dl-torch/lib/python3.7/site-packages/torch/utils/tensorboard/writer.py", line 534, in add_graph
    self._get_file_writer().add_graph(graph(model, input_to_model, verbose, **kwargs))
  File "/home/shared/virtualenv/dl-torch/lib/python3.7/site-packages/torch/utils/tensorboard/_pytorch_graph.py", line 279, in graph
    _ = model(*args)  # don't catch, just print the error message
  File "/home/shared/virtualenv/dl-torch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
TypeError: forward() takes 2 positional arguments but 4 were given

From the error message it appears that the support is still pending. Can I trust the release highlight? Or I'm not using the API properly?


title: Difference between Parameter vs. Tensor in PyTorch

body:
I would like to know the difference between PyTorch Parameter and Tensor?

The existing answer is for the old PyTorch where variables are being used?


I'm failing to run my GAN on GPU. I call to(device) for all both models and all tensors but still keep getting the following error:

Traceback (most recent call last):
  File "code/a3_gan_template.py", line 185, in <module>
    main(args)
  File "code/a3_gan_template.py", line 162, in main
    train(dataloader, discriminator, generator, optimizer_G, optimizer_D, device)
  File "code/a3_gan_template.py", line 100, in train
    d_x = discriminator.forward(imgs)
  File "code/a3_gan_template.py", line 80, in forward
    out = self.model(img)
  File "/home/lgpu0365/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/lgpu0365/.local/lib/python3.6/site-packages/torch/nn/modules/container.py", line 92, in forward
    input = module(input)
  File "/home/lgpu0365/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/lgpu0365/.local/lib/python3.6/site-packages/torch/nn/modules/linear.py", line 67, in forward
    return F.linear(input, self.weight, self.bias)
  File "/home/lgpu0365/.local/lib/python3.6/site-packages/torch/nn/functional.py", line 1352, in linear
    ret = torch.addmm(torch.jit._unwrap_optional(bias), input, weight.t())
RuntimeError: Expected object of backend CUDA but got backend CPU for argument #4 'mat1'

Source code:

import argparse
import os

import torch
import torch.nn as nn
import torchvision.transforms as transforms
from torchvision.utils import save_image
from torchvision import datasets
from torch.nn.functional import binary_cross_entropy

class Generator(nn.Module):
    def __init__(self, latent_dim):
        super(Generator, self).__init__()

        # Construct generator. You are free to experiment with your model,
        # but the following is a good start:
        #   Linear args.latent_dim -> 128
        #   LeakyReLU(0.2)
        #   Linear 128 -> 256
        #   Bnorm
        #   LeakyReLU(0.2)
        #   Linear 256 -> 512
        #   Bnorm
        #   LeakyReLU(0.2)
        #   Linear 512 -> 1024
        #   Bnorm
        #   LeakyReLU(0.2)
        #   Linear 1024 -> 784
        #   Output non-linearity

        self.latent_dim = latent_dim
        self.model = nn.Sequential(
            nn.Linear(latent_dim, 128),
            nn.LeakyReLU(0.2),
            nn.Linear(128, 256),
            nn.BatchNorm1d(256),
            nn.LeakyReLU(0.2),
            nn.Linear(256, 512),
            nn.BatchNorm1d(512),
            nn.LeakyReLU(0.2),
            nn.Linear(512, 1024),
            nn.BatchNorm1d(1024),
            nn.LeakyReLU(0.2),
            nn.Linear(1024, 784),
            nn.Sigmoid()
        )

    def forward(self, z):
        # Generate images from z
        out = self.model(z)

        return out

class Discriminator(nn.Module):
    def __init__(self):
        super(Discriminator, self).__init__()

        # Construct distriminator. You are free to experiment with your model,
        # but the following is a good start:
        #   Linear 784 -> 512
        #   LeakyReLU(0.2)
        #   Linear 512 -> 256
        #   LeakyReLU(0.2)
        #   Linear 256 -> 1
        #   Output non-linearity

        self.model = nn.Sequential(
            nn.Linear(784, 512),
            nn.LeakyReLU(0.2),
            nn.Linear(512, 256),
            nn.LeakyReLU(0.2),
            nn.Linear(256, 1),
            nn.Sigmoid()
        )

    def forward(self, img):
        # return discriminator score for img
        out = self.model(img)

        return out

def train(dataloader, discriminator, generator, optimizer_G, optimizer_D, device):
    for epoch in range(args.n_epochs):
        for i, (imgs, _) in enumerate(dataloader):
            batch_count = epoch * len(dataloader) + i

            imgs.to(device)
            batch_size = imgs.shape[0]
            imgs = imgs.reshape(batch_size, -1)

            z = torch.rand(batch_size, generator.latent_dim, device=device)
            gen_imgs = generator(z)

            discriminator.to(device)
            d_x = discriminator(imgs)
            d_g_z = discriminator(gen_imgs)

            ones = torch.ones(d_g_z.shape, device=device)

            # Train Generator
            # ---------------
            loss_G = binary_cross_entropy(d_g_z, ones)

            optimizer_G.zero_grad()
            loss_G.backward(retain_graph=True)
            optimizer_G.step()

            # Train Discriminator
            # -------------------
            if batch_count % args.d_train_interval == 0:
                loss_D = binary_cross_entropy(d_x, 0.9 * ones) + binary_cross_entropy(d_g_z, 0. * ones)

                optimizer_D.zero_grad()
                loss_D.backward()
                optimizer_D.step()

            # Save Images
            # -----------
            if batch_count % args.save_interval == 0:
                print(f'epoch: {epoch} batches: {batch_count} L_G: {loss_G.item():0.3f} L_D: {loss_D.item():0.3f}')
                # You can use the function save_image(Tensor (shape Bx1x28x28),
                # filename, number of rows, normalize) to save the generated
                # images, e.g.:
                save_image(gen_imgs[:25],
                           f'images/{batch_count}.png',
                           nrow=5, normalize=True)

def main(args):
    # Create output image directory
    os.makedirs('images', exist_ok=True)

    # Set device
    if torch.cuda.is_available():
        device = torch.device('cuda')
    else:
        device = torch.device('cpu')

    # load data
    dataloader = torch.utils.data.DataLoader(
        datasets.MNIST('./data/mnist', train=True, download=True,
                       transform=transforms.Compose([
                           transforms.ToTensor(),
                           transforms.Normalize( (0.5,), (0.5,) )
                           ])),
        batch_size=args.batch_size, shuffle=True)

    # Initialize models and optimizers
    generator = Generator(args.latent_dim)
    generator.to(device)
    discriminator = Discriminator()
    discriminator.to(device)
    optimizer_G = torch.optim.Adam(generator.parameters(), lr=args.lr)
    optimizer_D = torch.optim.Adam(discriminator.parameters(), lr=args.lr)

    # Start training
    train(dataloader, discriminator, generator, optimizer_G, optimizer_D, device)

    # You can save your generator here to re-use it to generate images for your
    # report, e.g.:
    torch.save(generator.state_dict(), "mnist_generator.pt")

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument('--n_epochs', type=int, default=200,
                        help='number of epochs')
    parser.add_argument('--batch_size', type=int, default=64,
                        help='batch size')
    parser.add_argument('--lr', type=float, default=0.0002,
                        help='learning rate')
    parser.add_argument('--latent_dim', type=int, default=100,
                        help='dimensionality of the latent space')
    parser.add_argument('--save_interval', type=int, default=500,
                        help='save every SAVE_INTERVAL iterations')
    parser.add_argument('--d_train_interval', type=int, default=25,
                        help='train discriminator (only) every D_TRAIN_INTERVAL iterations')
    args = parser.parse_args()

    main(args)

Any ideas on how to figure out what is missing? Thanks!


title: EOFError: Ran out of input, using torch.load()

body:
I saw this error being posted a lot and often it was due to the file not being closed properly after opening. But since I'm using the integrated torch.load() function, I'm not sure what I could do different.

First the saving part:

    torch.save({
            'model_state_dict': agent.dqn.state_dict(),
             ...
            'loss_history': agent.losshistory
            }, modelpath)

and here the loading part, where I also get the error message:

if os.path.exists(modelpath):
    checkpoint = torch.load(modelpath)
    agent.dqn.load_state_dict(checkpoint['model_state_dict'])
    ...
    agent.losshistory = checkpoint['loss_history']

and here the error:

Traceback (most recent call last):
  File "c:/Users/levin/Desktop/programming/main.py", line 33, in <module>
    checkpoint = torch.load(modelpath)
  File "C:\Users\levin\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\serialization.py", line 529, in load
    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
  File "C:\Users\levin\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\serialization.py", line 702, in _legacy_load
    result = unpickler.load()
EOFError: Ran out of input

One more thing I want to mention is that I used this exact code several times without a problem. I can't remember changing anything that could have caused the error.


title: How to normalize convolutional weights in pytorch?

body:
I have a CNN in pytorch and I need to normalize the convolution weights (filters) with L2 norm in each iteration. What is the most efficient way to do this?

Basically, in my particular experiment I need to replace the filters with their normalized value in the model (during both training and test).


title: Why should be the function backward be called only on 1 element tensor or with gradients w.r.t to Variable?

body:
I am new to pytorch. I want to understand as to why we can't call the backward function on a variable containing a tensor of say size say [2,2].
And if we do want to call it on a variable containing tensor of say size say [2,2], we have to do that by first defining a gradient tensor and then calling the backward function on the variable containing the tensor w.r.t the defined gradients.


title: Pytorch How to convert images into tensor

body:
I stored multiple images as below.

for name in image_names:
    images.append(cv2.imread("./train_mini/"+name))

And I’d like to use those images in CNN training later. However, when I stored those data in “torch.utils.data.TensorDataset” like below, it shows error:

“RuntimeError: can’t convert a given np.ndarray to a tensor - it has an invalid type. The only supported types are: double, float, int64, int32, and uint8.”

So I checked the data type of images, and it was “object”.

train = torch.utils.data.TensorDataset(torch.from_numpy(X_train), torch.from_numpy(Y_train))

How can I solve this problem? I am completely stuck…


title: pytorch AttributeError: module 'torch.optim.lr_scheduler' has no attribute 'CosineAnnealingLR'

body:
When trying to run a python notebook on google colab, mentioned in https://medium.com/@markn_67491/run-allennlp-models-on-free-gpus-using-googles-colab-notebooks-4db9359970c1 available from: https://drive.google.com/file/d/1JH6dz8GJbwh9GhPoZQKwR-EipeR5JBrV/view

after installing pytorch and AllenNlp I run

from allennlp.models.archival import load_archive
from allennlp.service.predictors import Predictor
import pprint

pretty_print = pprint.PrettyPrinter(indent=4)

which crashes on the second line due to ...

AttributeError: module 'torch.optim.lr_scheduler' has no attribute 'CosineAnnealingLR'

note: https://github.com/pytorch/pytorch/issues/3214 suggest upgrading to version 3.0 but even with
pip3 install http://download.pytorch.org/whl/cu80/torch-0.3.1-cp36-cp36m-linux_x86_64.whl
i.e. version 3.1 the issues persists. Any suggestions


title: Why aren't torch.functional.sigmoid and torch.nn.functional.relu deprecated like torch.nn.functional.tanh?

body:
Now when torch.autograd.Variable is merged with torch.tensor and obsolete, why did they deprecate some functions in torch.nn.functional but not others?  Namely, tanhis deprecated but not sigmoid or relu.

>>> torch.__version__
'1.1.0'
>>> u
tensor(2., grad_fn=<MeanBackward0>)
>>> torch.nn.functional.tanh(u)
C:\Users\mlearning\AppData\Local\Continuum\anaconda3\lib\site-packages\torch\nn\functional.py:1374: UserWarning: nn.functional.tanh is deprecated. Use torch.tanh instead.
  warnings.warn("nn.functional.tanh is deprecated. Use torch.tanh instead.")
tensor(0.9640, grad_fn=<TanhBackward>)
>>> torch.nn.functional.sigmoid(u)
tensor(0.8808, grad_fn=<SigmoidBackward>)
>>> torch.nn.functional.relu(u)
tensor(2., grad_fn=<ReluBackward0>)

Is there any difference between torch.nn.functional.relu and torch.relu, or I can use them interchangeably?


title: Pytorch autograd.grad how to write the parameters for multiple outputs?

body:
In the documentation of torch.autograd.grad, it is stated that, for parameters,

  parameters:

  outputs (sequence of Tensor) – outputs of the differentiated function.

  inputs (sequence of Tensor) – Inputs w.r.t.
  which the gradient will be returned (and not accumulated into .grad).

I try the following:

a = torch.rand(2, requires_grad=True)
b = torch.rand(2, requires_grad=True)
c = a+b
d = a-b

torch.autograd.grad([c, d], [a, b]) #ValueError: only one element tensors can be converted to Python scalars
torch.autograd.grad(torch.tensor([c, d]), torch.tensor([a, b])) #RuntimeError: grad can be implicitly created only for scalar outputs

I would like to get gradients of a list of tensors w.r.t another list of tensors. What is the correct way to feed the parameters?


title: PyTorch: When using backward(), how can I retain only part of the graph?

body:
I have a PyTorch computational graph, which consists of a sub-graph performing some calculation, and the result of this calculation (let's call it x) is then branched into two other sub-graphs. Each of these two sub-graphs yields some scalar results (lets call them y1 and y2). I want to do a backward pass for each of these two results (that is, I want to accumulate the gradients of the two sub-graphs. I do not want to perform the actual optimization step).

Now, since memory is an issue here, I would like to perform the operations in the following order:
First, calculate x. Then, calculate y1, and perform y1.backward() while (and this is the key point) retaining the graph that leads to x, but freeing the graph from x to y1. Then, calculate y2, and perform y2.backward().

In other words, to save memory without sacrificing too much speed, I want to keep x without needing to recalculate it, but I want to drop all calculations leading from x to y1 after I don't need them anymore.

The problem is that the argument retain_graph of the function backward() will retain the entire graph leading to y1, whereas I need to retain only the part of the graph leading to x.

Here is an example of what I would ideally want:

import torch

w = torch.tensor(1.0)
w.requires_grad_(True)

# sub-graph for calculating `x`
x = w+10

# sub-graph for calculating `y1`
x1 = x*x
y1 = x1*x1
y1.backward(retain_graph=x) # this would not work, since retain_graph is a boolean and can either retain the entire graph or free it.

# sub-graph for calculating `y2`
x2 = torch.sqrt(x)
y2 = x2/2
y2.backward()

How can this be done?


Does batch_first affect hidden tensors in Pytorch LSTMs?

That is if batch_first parameter is true,
Will the hidden state be (numlayer*direction,num_batch,encoding_dim)
or (num_batch,numlayer*direction,encoding_dim)

I've tested both, both give no error.


title: Pytorch: Error in DataParallel for RNN model

body:
I'm trying to use torch.nn.DataParallel for a RNN model. My model looks like this:

class EncoderRNN(nn.Module):
def __init__(self, vocal_size, hidden_size):
    super(EncoderRNN, self).__init__()
    self.hidden_size = hidden_size
    self.embedding = nn.Embedding(vocal_size, hidden_size)
    self.gru = nn.GRU(hidden_size, hidden_size, batch_first=True)

def forward(self, input_batch, input_batch_length, hidden):
    embedded = self.embedding(input_batch)
    packed_input = nn.utils.rnn.pack_padded_sequence(embedded, input_batch_length.cpu().numpy(), batch_first=True)
    output, hidden = self.gru(packed_input, hidden)
    return output, hidden

class DecoderRNN(nn.Module):
def __init__(self, hidden_size, vocab_size):
    super(DecoderRNN, self).__init__()
    self.hidden_size = hidden_size
    self.embedding = nn.Embedding(vocab_size, hidden_size)
    self.gru = nn.GRU(hidden_size, hidden_size, batch_first=True)
    self.out = nn.Linear(hidden_size, vocab_size)
    self.softmax = nn.LogSoftmax(dim=1)

def forward(self, target_batch, target_batch_length, hidden, train=False):
    embedded = self.embedding(target_batch)
    output = F.relu(embedded)

    if train:
        # minus 1 to eliminate <EOS>
        packed_target = nn.utils.rnn.pack_padded_sequence(output, (target_batch_length - 1).cpu().numpy(),
                                                          batch_first=True)

    output, hidden = self.gru(packed_target, hidden)
    output = self.softmax(self.out(output[0]))
    return output, hidden

And I implemented DataParallel like this when declaring the model:

encoder = nn.DataParallel(encoder)
decoder = nn.DataParallel(decoder)

The code runs on a server with 4 GPUs, and I received following error message:

/home/cjunjie/NLP/DocSummarization/model.py:18: UserWarning: RNN module weights are not part of single contiguous chunk of memory. This means they need to be compacted at every call, possibly greatly increasing memory usage. To compact weights again call flatten_parameters().
output, hidden = self.gru(packed_input, hidden)
Traceback (most recent call last):
  File "train.py", line 144, in <module>
    train_iteration(encoder, decoder, fileDataSet)
  File "train.py", line 110, in train_iteration
    target_indices, encoder, decoder, encoder_optimizer, decoder_optimizer, criterion)
  File "train.py", line 41, in train
    encoder_output, encoder_hidden = encoder(input_batch, input_batch_length, encoder_hidden)
  File "/home/cjunjie/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 357, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/cjunjie/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 74, in forward
return self.gather(outputs, self.output_device)
  File "/home/cjunjie/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 86, in gather
    return gather(outputs, output_device, dim=self.dim)
  File "/home/cjunjie/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 65, in gather
    return gather_map(outputs)
  File "/home/cjunjie/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 60, in gather_map
    return type(out)(map(gather_map, zip(*outputs)))
  File "/home/cjunjie/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 60, in gather_map
    return type(out)(map(gather_map, zip(*outputs)))
  File "/home/cjunjie/anaconda3/lib/python3.6/site-packages/torch/nn/utils/rnn.py", line 39, in __new__
    return super(PackedSequence, cls).__new__(cls, *args[0])
  File "/home/cjunjie/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 57, in gather_map
    return Gather.apply(target_device, dim, *outputs)
  File "/home/cjunjie/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/_functions.py", line 58, in forward
    assert all(map(lambda i: i.is_cuda, inputs))
AssertionError

I searched for the same problem, but none of them have a solution. Can anyone help?


title: Is it possible to load a pretrained Pytorch model from a GCS bucket URL without first persisting locally?

body:
I'm asking this in the context of Google Dataflow, but also generally.

Using PyTorch, I can reference a local directory containing multiple files that comprise a pretrained model. I happen to be working with a Roberta model, but the interface is the same for others.

ls some-directory/
      added_tokens.json
      config.json
      merges.txt
      pytorch_model.bin
      special_tokens_map.json vocab.json

from pytorch_transformers import RobertaModel

# this works
model = RobertaModel.from_pretrained('/path/to/some-directory/')

However, my pretrained model is stored in a GCS bucket. Let's call it gs://my-bucket/roberta/.

In the context of loading this model in Google Dataflow, I'm trying to remain stateless and avoid persisting to disk, so my preference would be to get this model straight from GCS. As I understand it, the PyTorch general interface method from_pretrained() can take the string representation of a local dir OR a URL. However, I can't seem to load the model from a GCS URL.

# this fails
model = RobertaModel.from_pretrained('gs://my-bucket/roberta/')
# ValueError: unable to parse gs://mahmed_bucket/roberta-base as a URL or as a local path

If I try to use the public https URL of the directory blob, it will also fail, although that is likely due to lack of authentication since the credentials referenced in the python environment that can create clients don't translate to public requests to https://storage.googleapis

# this fails, probably due to auth
bucket = gcs_client.get_bucket('my-bucket')
directory_blob = bucket.blob(prefix='roberta')
model = RobertaModel.from_pretrained(directory_blob.public_url)
# ValueError: No JSON object could be decoded

# and for good measure, it also fails if I append a trailing /
model = RobertaModel.from_pretrained(directory_blob.public_url + '/')
# ValueError: No JSON object could be decoded

I understand that GCS doesn't actually have subdirectories and it's actually just being a flat namespace under the bucket name. However, it seems like I'm blocked by the necessity of authentication and a PyTorch not speaking gs://.

I can get around this by persisting the files locally first.

from pytorch_transformers import RobertaModel
from google.cloud import storage
import tempfile

local_dir = tempfile.mkdtemp()
gcs = storage.Client()
bucket = gcs.get_bucket(bucket_name)
blobs = bucket.list_blobs(prefix=blob_prefix)
for blob in blobs:
    blob.download_to_filename(local_dir + '/' + os.path.basename(blob.name))
model = RobertaModel.from_pretrained(local_dir)

But this seems like such a hack, and I keep thinking I must be missing something. Surely there's a way to stay stateless and not have to rely on disk persistence!

So is there a way to load a pretrained model stored in GCS?
Is there a way to authenticate when doing the public URL request in this context?
Even if there is a way to authenticate, will the non-existence of subdirectories still be an issue?

Thanks for the help! I'm also happy to be pointed to any duplicate questions 'cause I sure couldn't find any.

Edits and Clarifications

My Python session is already authenticated to GCS, which is why I'm able to download the blob files locally and then point to that local directory with load_frompretrained()
load_frompretrained() requires a directory reference because it needs all the files listed at the top of the question, not just pytorch-model.bin
To clarify question #2, I was wondering if there's some way of giving the PyTorch method a request URL that had encrypted credentials embedded or something like that. Kind of a longshot, but I wanted to make sure I hadn't missed anything.
To clarify question #3 (in addition to the comment on one answer below), even if there's a way to embed credentials in the URL that I don't know about, I still need to reference a directory rather than a single blob, and I don't know if the GCS subdirectory would be recognized as such because (as the Google docs state) subdirectories in GCS are an illusion and they don't represent a real directory structure. So I think this question is irrelevant or at least blocked by question #2, but it's a thread I chased for a bit so I'm still curious.


title: Pytorch nn.embedding error

body:
I was reading pytorch documentation on Word Embedding.

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

torch.manual_seed(5)

word_to_ix = {"hello": 0, "world": 1, "how":2, "are":3, "you":4}
embeds = nn.Embedding(2, 5)  # 2 words in vocab, 5 dimensional embeddings
lookup_tensor = torch.tensor(word_to_ix["hello"], dtype=torch.long)
hello_embed = embeds(lookup_tensor)
print(hello_embed)

Output:

tensor([-0.4868, -0.6038, -0.5581,  0.6675, -0.1974])

This looks good but if I replace  line lookup_tensor by

lookup_tensor = torch.tensor(word_to_ix["how"], dtype=torch.long)

I am getting the error as:

RuntimeError: index out of range at /Users/soumith/minicondabuild3/conda-bld/pytorch_1524590658547/work/aten/src/TH/generic/THTensorMath.c:343

I don't understand why it gives RunTime error on line hello_embed = embeds(lookup_tensor).


title: Correctly converting a NumPy array to a PyTorch tensor running on the gpu

body:
I have created a DataLoader that looks like this

class ToTensor(object):
    def __call__(self, sample):
        return torch.from_numpy(sample).to(device)

class MyDataset(Dataset):
    def __init__(self, data, transform=None):
        self.data = data
        self.transform = transform

    def __len__(self):
        return len(self.data)

    def __getitem__(self, idx):
        sample = self.data[idx, :]

        if self.transform:
            sample = self.transform(sample)

        return sample

I am using this data loader like so

dataset = MLBDataset(
        data=data,
        transform=transforms.Compose([
            ToTensor()
        ]))
dataloader = DataLoader(dataset, batch_size=BATCH_SIZE, shuffle=True, num_workers=4)
dataiter = iter(dataloader)
x = dataiter.next()

This fails with the message

THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1549628766161/work/aten/src/THC/THCGeneral.cpp line=55 error=3 : initialization error
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1549628766161/work/aten/src/THC/THCGeneral.cpp line=55 error=3 : initialization error
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1549628766161/work/aten/src/THC/THCGeneral.cpp line=55 error=3 : initialization error
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1549628766161/work/aten/src/THC/THCGeneral.cpp line=55 error=3 : initialization error
...
    torch._C._cuda_init()
RuntimeError: cuda runtime error (3) : initialization error at /opt/conda/conda-bld/pytorch_1549628766161/work/aten/src/THC/THCGeneral.cpp:55

For the return command inside ToTensor(), in fact any attempt to move the tensor te the GPU will fail inside that class. I have tried:

a = np.array([[[1, 2, 3, 4], [5, 6, 7, 8], [25, 26, 27, 28]],
             [[11, 12, np.nan, 14], [15, 16, 17, 18], [35, 36, 37, 38]]])
print(torch.from_numpy(a).to(device))

inside the body of __call__ in ToTensor() and it fails with the same message, whereas it succeeds everywhere else.

Why is this error generated and how can I resolve this?


title: padding a list of torch tensors (or numpy arrays)

body:
Let's say I have a list as the following:

l = [torch.randn(2,3), torch.randn(2,4),torch.randn(2,5)]

I want to zero pad all of them in the second dimension, so they will extend as far as 5 elements (5 being the max number between the three of the elements in the second dimension). How can I do this. I tried this but failed:

from torch.nn.utils.rnn import pad_sequence
pad_sequence(l, batch_first=True, padding_value=0)

which caused the following error:

RuntimeError: The expanded size of the tensor (3) must match the existing size (4) at non-singleton dimension 1.  Target sizes: [2, 3].  Tensor sizes: [2, 4]

The equivalent answer in Numpy would also be appreciated.


title: Pytorch : How .grad() function returning result?

body:
I am trying to understand grad() function in python, I know about backpropagation but having some doubt in .grad() function result.

So if i have a very simple network say with one single input and one single weight :

import torch
from torch.autograd import Variable
from torch import FloatTensor

a_tensor=Variable(FloatTensor([1]))
weight=Variable(FloatTensor([1]),requires_grad=True)

Now i am running this in ipython cell:

net_out=a_tensor*weight
loss=5-net_out
loss.backward()
print("atensor",a_tensor)
print('weight',weight)
print('net_out',net_out)
print('loss',loss)
print(weight.grad)

During first run it returns :

atensor tensor([ 1.])
weight tensor([ 1.])
net_out tensor([ 1.])
loss tensor([ 4.])
tensor([-1.])

Which is correct because if i am right then computing gradient equation would be here :

Now netout/w would be (w*a) w.r.t to w ==> 1*a
And loss/netout  (5-netout) w.r.t to netout ==> (0-1)

Which would be 1*a*-1 ==> -1

But problem is if i press same cell again without modifying anything then i am getting grad -2 , -3 ,-4 ...etc

atensor tensor([ 1.])
weight tensor([ 1.])
net_out tensor([ 1.])
loss tensor([ 4.])
tensor([-2.])

next run:

atensor tensor([ 1.])
weight tensor([ 1.])
net_out tensor([ 1.])
loss tensor([ 4.])
tensor([-3.])

so on..

I am not getting what's happening there why and how the value of grad is increasing?


title: Pytorch inceptionV3 transfer learning gives error - max() received an invalid combination of arguments

body:
The program for transfer learning inception_v3 in pytorch that i am using is here : https://drive.google.com/file/d/1zn4z7nOp_wJne0En6zq4WJfwHVVftERT/view?usp=sharing

I am getting the following error upon running the program :

Epoch 0/24
    ---------------------------------------------------------------------------
    TypeError                                 Traceback (most recent call last)
    <ipython-input-20-cc88ea5f8bd3> in <module>()
          1 model_ft = train_model(model_ft, criterion, optimizer_ft, exp_lr_scheduler,
    ----> 2                        num_epochs=25)

    <ipython-input-17-812cf3c4576a> in train_model(model, criterion, optimizer, scheduler, num_epochs)
         33                     outputs = model(inputs)
         34                     print(outputs)
    ---> 35                     _, preds = torch.max(outputs, 1)
         36                     loss = criterion(outputs, labels)
         37

    TypeError: max() received an invalid combination of arguments - got (tuple, int), but expected one of:
     * (Tensor input)
     * (Tensor input, Tensor other, Tensor out)
     * (Tensor input, int dim, bool keepdim, tuple of Tensors out)

How can this be fixed ?
Thank you


title: How to deal with mini-batch loss in Pytorch?

body:
I feed mini-batch data to model, and I just want to know how to deal with the loss. Could I accumulate the loss, then call the backward like:

    ...
    def neg_log_likelihood(self, sentences, tags, length):
        self.batch_size = sentences.size(0)

        logits = self.__get_lstm_features(sentences, length)
        real_path_score = torch.zeros(1)
        total_score = torch.zeros(1)
        if USE_GPU:
            real_path_score = real_path_score.cuda()
            total_score = total_score.cuda()

        for logit, tag, leng in zip(logits, tags, length):
            logit = logit[:leng]
            tag = tag[:leng]
            real_path_score += self.real_path_score(logit, tag)
            total_score += self.total_score(logit, tag)
        return total_score - real_path_score
    ...
loss = model.neg_log_likelihood(sentences, tags, length)
loss.backward()
optimizer.step()

I wonder that if the accumulation could lead to gradient explosion?

So, should I call the backward in loop:

for sentence, tag , leng in zip(sentences, tags, length):
    loss = model.neg_log_likelihood(sentence, tag, leng)
    loss.backward()
    optimizer.step()

Or, use the mean loss just like the reduce_mean in tensorflow

loss = reduce_mean(losses)
loss.backward()


title: Upsampling an autoencoder in pytorch

body:
I have defined my autoencoder in pytorch as following (it gives me a 8-dimensional bottleneck at the output of the encoder which works fine torch.Size([1, 8, 1, 1])):

self.encoder = nn.Sequential(
    nn.Conv2d(input_shape[0], 32, kernel_size=8, stride=4),
    nn.ReLU(),
    nn.Conv2d(32, 64, kernel_size=4, stride=2),
    nn.ReLU(),
    nn.Conv2d(64, 8, kernel_size=3, stride=1),
    nn.ReLU(),
    nn.MaxPool2d(7, stride=1)
)

self.decoder = nn.Sequential(
    nn.ConvTranspose2d(8, 64, kernel_size=3, stride=1),
    nn.ReLU(),
    nn.Conv2d(64, 32, kernel_size=4, stride=2),
    nn.ReLU(),
    nn.Conv2d(32, input_shape[0], kernel_size=8, stride=4),
    nn.ReLU(),
    nn.Sigmoid()
)

What I cannot do is train the autoencoder with

def forward(self, x):
    x = self.encoder(x)
    x = self.decoder(x)
    return x

The decoder gives me an error that the decoder cannot upsample the tensor:

Calculated padded input size per channel: (3 x 3). Kernel size: (4 x 4). Kernel size can't be greater than actual input size


title: Calculate mean across one specific dimension of a 4D tensor in Pytorch

body:
I have a PyTorch video feature tensor of shape [66,7,7,1024] and I need to convert it to [1024,66,7,7]. How to rearrange a tensor shape? Also, how to perform mean across dimension=1? i.e., after performing mean of the dimension with size 66, I need the tensor to be [1024,1,7,7].

I have tried to calculate the mean of dimension=1 but I failed to replace it with the mean value. And I could not imagine a 4D tensor in which one dimension is replaced by its mean.

Edit:
 I tried torch.mean(my_tensor, dim=1). But this returns me a tensor of shape [1024,7,7]. The 4D tensor is being converted to 3D. But I want it to remain 4D with shape [1024,1,7,7].

Thank you very much.


title: matrix product of the two tensors

body:
How would you calculate matrix product of the two tensors in PyTorch?

x =  torch.Tensor([[1, 2, 3], [1, 2, 3]]).view(-1, 2)
y =  torch.Tensor([[2, 1]]).view(2, -1)

I am confused between these options.