title: TypeError: forward() takes 1 positional argument but 2 were given

body:
class DeConv2d(nn.Module):
    def __init__(self, in_channel, out_channel, kernel_size, stride, padding, dilation):
        super().__init__()

        self.up = nn.Upsample(scale_factor=2, mode='nearest')
        self.conv = nn.Conv2d(in_channel, out_channel, kernel_size=kernel_size, \
                              stride=stride, padding=padding, dilation=dilation)

    def forward(self, x):

        output = self.up(x)
        output = self.conv(output)

        return output

class EncoderDecoder(nn.Module):
    def __init__(self, pretrained_net, n_class):
        super().__init__()
        self.n_class = n_class
        self.pretrained_net = pretrained_net
        self.relu = nn.ReLU(inplace=True)

        self.deconv1 = DeConv2d(512, 512, kernel_size=3, stride=1, padding=1, dilation=1)
        self.bn1 = nn.BatchNorm2d(512)

        self.deconv2 = DeConv2d(512, 256, kernel_size=3, stride=1, padding=1, dilation=1)
        self.bn2 = nn.BatchNorm2d(256)

        self.deconv3 = DeConv2d(256, 128, kernel_size=3, stride=1, padding=1, dilation=1)
        self.bn3 = nn.BatchNorm2d(128)

        self.deconv4 = DeConv2d(128, 64, kernel_size=3, stride=1, padding=1, dilation=1)
        self.bn4 = nn.BatchNorm2d(64)

        self.classifier = nn.Conv2d(64, n_class, kernel_size=1)

    def forward(self, x):

        output=self.pretrained_net.layers(x)

        output=self.relu(self.deconv1(output))
        output=self.bn1(output)

        output=self.relu(self.deconv2(output))
        output=self.bn2(output)

        output=self.relu(self.deconv3(output))
        output=self.bn3(output)

        output=self.relu(self.deconv4(output))
        output=self.bn4(output)

        output=self.classifier(output)

        return output

this is my code and I don't know why the type error exist . Does somebody know how to fix these problems?


title: How to use PyTorch to print out the prediction accuracy of every class?

body:
I am trying to use PyTorch to print out the prediction accuracy of every class based on the official tutorial link

But things seem to go wrong. My code intends to do this work is as following:

    for epoch in range(num_epochs):

    # Each epoch has a training and validation phase
    for phase in ['train', 'val']:
       ... (this is given by the tutorial)

    (my code)

    class_correct = list(0. for i in range(3))
    class_total = list(0. for i in range(3))

    for data in dataloaders['val']:
        images, labels = data
        outputs = model(inputs)
        _, predicted = torch.max(outputs.data, 1)
        c = (predicted == labels.data).squeeze()

        for i in range(4):
            label = labels.data[i]
            class_correct[label] += c[i]
            class_total[label] += 1

    for i in range(3):
        print('Accuracy of {} : {} / {} = {:.4f} %'.format(i,
class_correct[i], class_total[i], 100 * class_correct[i].item() /
class_total[i]))

    print(file = f)
    print()

For example, the output of epoch 1/1 is :

I think the following equation should be satisfied:

  running_corrects := 2 + 2

But things does not happen as I think.

What's wrong there?
Hope someone can point out my fault and teach me how to do this correctly.

Thx!


title: Pytorch: Normalize Image data set

body: I want to normalize custom dataset of images. For that i need to compute mean and standard deviation by iterating over the dataset. How can I normalize my entire dataset before creating the data set?


title: Is it possible to freeze only certain embedding weights in the embedding layer in pytorch?

body:
When using GloVe embedding in NLP tasks, some words from the dataset might not exist in GloVe. Therefore, we instantiate random weights for these unknown words.

Would it be possible to freeze weights gotten from GloVe, and train only the newly instantiated weights?

I am only aware that we can set:
model.embedding.weight.requires_grad = False

But this makes the new words untrainable..

Or are there better ways to extract semantics of words..


title: For a given condition, get indices of values in 2D tensor A, use those to index a 3D tensor B

body:
For a given 2D tensor I want to retrieve all indices where the value is 1. I expected to be able to simply use torch.nonzero(a == 1).squeeze(), which would return tensor([1, 3, 2]). However, instead, torch.nonzero(a == 1) returns a 2D tensor (that's okay), with two values per row (that's not what I expected). The returned indices should then be used to index the second dimension (index 1) of a 3D tensor, again returning a 2D tensor.

import torch

a = torch.Tensor([[12, 1, 0, 0],
                  [4, 9, 21, 1],
                  [10, 2, 1, 0]])

b = torch.rand(3, 4, 8)

print('a_size', a.size())
# a_size torch.Size([3, 4])
print('b_size', b.size())
# b_size torch.Size([3, 4, 8])

idxs = torch.nonzero(a == 1)
print('idxs_size', idxs.size())
# idxs_size torch.Size([3, 2])

print(b.gather(1, idxs))

Evidently, this does not work, leading to aRunTimeError:

  RuntimeError: invalid argument 4: Index tensor must have same
  dimensions as input tensor at
  C:\w\1\s\windows\pytorch\aten\src\TH/generic/THTensorEvenMoreMath.cpp:453

It seems that idxs is not what I expect it to be, nor can I use it the way I thought. idxs is

tensor([[0, 1],
        [1, 3],
        [2, 2]])

but reading through the documentation I don't understand why I also get back the row indices in the resulting tensor. Now, I know I can get the correct idxs by slicing idxs[:, 1] but then still, I cannot use those values as indices for the 3D tensor because the same error as before is raised. Is it possible to use the 1D tensor of indices to select items across a given dimension?


I already saw many people with this same error. I tried adding some lines about my CUDA and gpu and cpu at the virtual environment (that I created) python file, but any answer that I saw could help me

I am trying to install this Python tool, but I am having a problem when I am executing this software.

https://github.com/CorentinJ/Real-Time-Voice-Cloning

I have the CUDA at 10.2, Pytorch is at the last version for CUDA 10.2, and I have Python at 3.7

When I click at a button to use vocoder at pretrained, I got this lines before the error:

Building Wave-RNN
Trainable Parameters: 4.481M
Traceback (most recent call last):
  File "C:\Program Files\DeepVoiceFiles\Real-Time-Voice-Cloning-master\toolbox\__init__.py", line 187, in vocode
    self.init_vocoder()
  File "C:\Program Files\DeepVoiceFiles\Real-Time-Voice-Cloning-master\toolbox\__init__.py", line 249, in init_vocoder
    vocoder.load_model(model_fpath)
  File "C:\Program Files\DeepVoiceFiles\Real-Time-Voice-Cloning-master\vocoder\inference.py", line 25, in load_model
    mode=hp.voc_mode
  File "E:\PROGRAMAS\Anaconda\envs\[Test_voice]\lib\site-packages\torch\nn\modules\module.py", line 304, in cuda
    return self._apply(lambda t: t.cuda(device))
  File "E:\PROGRAMAS\Anaconda\envs\[Test_voice]\lib\site-packages\torch\nn\modules\module.py", line 201, in _apply
    module._apply(fn)
  File "E:\PROGRAMAS\Anaconda\envs\[Test_voice]\lib\site-packages\torch\nn\modules\module.py", line 201, in _apply
    module._apply(fn)
  File "E:\PROGRAMAS\Anaconda\envs\[Test_voice]\lib\site-packages\torch\nn\modules\module.py", line 201, in _apply
    module._apply(fn)
  File "E:\PROGRAMAS\Anaconda\envs\[Test_voice]\lib\site-packages\torch\nn\modules\module.py", line 223, in _apply
    param_applied = fn(param)
  File "E:\PROGRAMAS\Anaconda\envs\[Test_voice]\lib\site-packages\torch\nn\modules\module.py", line 304, in <lambda>
    return self._apply(lambda t: t.cuda(device))
  File "E:\PROGRAMAS\Anaconda\envs\[Test_voice]\lib\site-packages\torch\cuda\__init__.py", line 199, in _lazy_init
    torch._C._cuda_init()
RuntimeError: cuda runtime error (100) : no CUDA-capable device is detected at ..\aten\src\THC\THCGeneral.cpp:50

Please, someone, help me, I don't understand much about Python, CUDA and virtual environment.

I installed this following a YouTube video.


title: AutoTokenizer.from_pretrained fails to load locally saved pretrained tokenizer (PyTorch)

body:
I am new to PyTorch and recently, I have been trying to work with Transformers. I am using pretrained tokenizers provided by HuggingFace.
 I am successful in downloading and running them. But if I try to save them and load again, then some error occurs.  If I use  AutoTokenizer.from_pretrained to download a tokenizer, then it works.

[1]:    tokenizer = AutoTokenizer.from_pretrained('distilroberta-base')
        text = "Hello there"
        enc = tokenizer.encode_plus(text)
        enc.keys()

Out[1]: dict_keys(['input_ids', 'attention_mask'])

But if I save it using tokenizer.save_pretrained("distilroberta-tokenizer") and try to load it locally, then it fails.

[2]:    tmp = AutoTokenizer.from_pretrained('distilroberta-tokenizer')

---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
/opt/conda/lib/python3.7/site-packages/transformers/configuration_utils.py in get_config_dict(cls, pretrained_model_name_or_path, **kwargs)
    238                 resume_download=resume_download,
--> 239                 local_files_only=local_files_only,
    240             )

/opt/conda/lib/python3.7/site-packages/transformers/file_utils.py in cached_path(url_or_filename, cache_dir, force_download, proxies, resume_download, user_agent, extract_compressed_file, force_extract, local_files_only)
    266         # File, but it doesn't exist.
--> 267         raise EnvironmentError("file {} not found".format(url_or_filename))
    268     else:

OSError: file distilroberta-tokenizer/config.json not found

During handling of the above exception, another exception occurred:

OSError                                   Traceback (most recent call last)
<ipython-input-25-3bd2f7a79271> in <module>
----> 1 tmp = AutoTokenizer.from_pretrained("distilroberta-tokenizer")

/opt/conda/lib/python3.7/site-packages/transformers/tokenization_auto.py in from_pretrained(cls, pretrained_model_name_or_path, *inputs, **kwargs)
    193         config = kwargs.pop("config", None)
    194         if not isinstance(config, PretrainedConfig):
--> 195             config = AutoConfig.from_pretrained(pretrained_model_name_or_path, **kwargs)
    196
    197         if "bert-base-japanese" in pretrained_model_name_or_path:

/opt/conda/lib/python3.7/site-packages/transformers/configuration_auto.py in from_pretrained(cls, pretrained_model_name_or_path, **kwargs)
    194
    195         """
--> 196         config_dict, _ = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
    197
    198         if "model_type" in config_dict:

/opt/conda/lib/python3.7/site-packages/transformers/configuration_utils.py in get_config_dict(cls, pretrained_model_name_or_path, **kwargs)
    250                 f"- or '{pretrained_model_name_or_path}' is the correct path to a directory containing a {CONFIG_NAME} file\n\n"
    251             )
--> 252             raise EnvironmentError(msg)
    253
    254         except json.JSONDecodeError:

OSError: Can't load config for 'distilroberta-tokenizer'. Make sure that:

- 'distilroberta-tokenizer' is a correct model identifier listed on 'https://huggingface.co/models'

- or 'distilroberta-tokenizer' is the correct path to a directory containing a config.json file

Its saying 'config.josn' is missing form the directory. On checking the directory, I am getting list of these files:

[3]:    !ls distilroberta-tokenizer

Out[3]: merges.txt  special_tokens_map.json  tokenizer_config.json  vocab.json

I know this problem has been posted earlier but none of them seems to work. I have also tried to follow the docs but still can't make it work.
Any help would be appreciated.


title: How to define specific number of convolutional kernels/filters in pytorch?

body:
On pytorch website they have the following model in their tutorial

class BasicCNN(nn.Module):
    def __init__(self):
        super(BasicCNN, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = x.permute(0, 3, 1, 2)
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 5 * 5)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

How many kernels/filters this model has? Is it two - e.g conv1 and conv2. How do I easily create many filters by specifying the number of them? For example 100 filters.

Thanks!


title: PyTorch: Comparing predicted label and target label to compute accuracy

body:
I'm trying to implement this loop to get the accuracy of my PyTorch CNN (The complete code of it is here)
My version of the loop is so far:

correct = 0
    test_total = 0
    for itera, testdata2 in enumerate(test_loader, 0):
        test_images2, test_labels2 = testdata2
        if use_gpu:
            test_images2 = Variable(test_images2.cuda())
        else:
            test_images2 = Variable(test_images2)
        outputs = model(test_images2)
        _, predicted = torch.max(outputs.data, 1)
        test_total += test_labels2.size(0)
        test_labels2 = test_labels2.type_as(predicted)
        correct += (predicted == test_labels2[0]).sum()
    print('Accuracy of the network on all the test images: %d %%' % (
        100 * correct / test_total))

If I run it like this, I get:

> Traceback (most recent call last):   File
> "c:/python_code/Customized-DataLoader-master_two/multi_label_classifier_for2classes.py",
> line 186, in <module>
>     main()   File "c:/python_code/Customized-DataLoader-master_two/multi_label_classifier_for2classes.py",
> line 177, in main
>     correct += (predicted == test_labels2[0]).sum()   File "C:\anaconda\envs\pytorch_cuda\lib\site-packages\torch\tensor.py",
> line 360, in __eq__
>     return self.eq(other) RuntimeError: invalid argument 3: sizes do not match at
> c:\anaconda2\conda-bld\pytorch_1519501749874\work\torch\lib\thc\generated\../THCTensorMathCompareT.cuh:65

I used test_labels2 = test_labels2.type_as(predicted) to have both tensors as LongTensors, which seems to work fine to avert the "Expected this...but got..." errors. They look like this now:

test_labels2 after conversion:
 0  1
 1  0
 1  0
[torch.cuda.LongTensor of size 3x2 (GPU 0)]

predicted:
 1
 1
 1
[torch.cuda.LongTensor of size 3 (GPU 0)]

I supppose the problem now is, that test_labels2[0] is returning a row but not the column.

How do I get this to work?


title: What is the meaning of in-place in dropout

body:
def dropout(input, p=0.5, training=True, inplace=False)

  inplace: If set to True, will do this operation in-place.

I would like to ask what is the meaning of in-place in dropout. What does it do?
Any performance changes when performing these operation?

Thanks


title: Pytorch: load dataset of grayscale images

body:
I want to load a dataset of grayscale images. I used ImageFolder but this doesn't load gray images by default as it converts images to RGB.

I found solutions that load images with ImageFolder and after convert images in grayscale, using:

transforms.Grayscale(num_output_channels=1)

or

ImageOps.grayscale(image)

Is it correct?
How can I load grayscale imaged without conversion? I try ImageDataBunch, but I have problems to import fastai.vision


title: How does batching work in a seq2seq model in pytorch?

body:
I am trying to implement a seq2seq model in Pytorch and I am having some problem with the batching.
For example I have a batch of data whose dimensions are

  [batch_size, sequence_lengths, encoding_dimension]

where the sequence lengths are different for each example in the batch.

Now, I managed to do the encoding part by padding each element in the batch to the length of the longest sequence.

This way if I give as input to my net a batch with the same shape as said, I get the following outputs:

  output, of shape [batch_size, sequence_lengths, hidden_layer_dimension]

  hidden state, of shape [batch_size, hidden_layer_dimension]

  cell state, of shape [batch_size, hidden_layer_dimension]

Now, from the output, I take for each sequence the last relevant element, that is the element along the sequence_lengths dimension corresponding to the last non padded element of the sequence. Thus the final output I get is of shape [batch_size, hidden_layer_dimension].

But now I have the problem of decoding it from this vector. How do I handle a decoding of sequences of different lengths in the same batch? I tried to google it and found this, but they don't seem to address the problem. I thought of doing element by element for the whole batch, but then I have the problem to pass the initial hidden states, given that the ones from the encoder will be of shape [batch_size, hidden_layer_dimension], while the ones from the decoder will be of shape [1, hidden_layer_dimension].

Am I missing something? Thanks for the help!


title: Can't convert Pytorch to ONNX

body:
Trying to convert this pytorch model with ONNX gives me this error. I've searched github and this error came up before in version 1.1.0 but was apparently rectified. Now I'm on torch 1.4.0. (python 3.6.9) and I see this error.

File "/usr/local/lib/python3.6/dist-packages/torch/onnx/init.py", line 148, in export
strip_doc_string, dynamic_axes, keep_initializers_as_inputs)
File "/usr/local/lib/python3.6/dist-packages/torch/onnx/utils.py", line 66, in export
dynamic_axes=dynamic_axes, keep_initializers_as_inputs=keep_initializers_as_inputs)
File "/usr/local/lib/python3.6/dist-packages/torch/onnx/utils.py", line 416, in _export
fixed_batch_size=fixed_batch_size)
File "/usr/local/lib/python3.6/dist-packages/torch/onnx/utils.py", line 296, in _model_to_graph
fixed_batch_size=fixed_batch_size, params_dict=params_dict)
File "/usr/local/lib/python3.6/dist-packages/torch/onnx/utils.py", line 135, in _optimize_graph
graph = torch._C._jit_pass_onnx(graph, operator_export_type)
File "/usr/local/lib/python3.6/dist-packages/torch/onnx/init.py", line 179, in _run_symbolic_function
return utils._run_symbolic_function(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/torch/onnx/utils.py", line 657, in _run_symbolic_function
return op_fn(g, *inputs, **attrs)
File "/usr/local/lib/python3.6/dist-packages/torch/onnx/symbolic_helper.py", line 128, in wrapper
args = [_parse_arg(arg, arg_desc) for arg, arg_desc in zip(args, arg_descriptors)]
File "/usr/local/lib/python3.6/dist-packages/torch/onnx/symbolic_helper.py", line 128, in
args = [_parse_arg(arg, arg_desc) for arg, arg_desc in zip(args, arg_descriptors)]
File "/usr/local/lib/python3.6/dist-packages/torch/onnx/symbolic_helper.py", line 81, in _parse_arg
"', since it's not constant, please try to make "
RuntimeError: Failed to export an ONNX attribute 'onnx::Gather', since it's not constant, please try to make things (e.g., kernel size) static if possible

How to fix it? I've also tried latest nightly build, same error comes up.

My code:

from model import BiSeNet
import torch.onnx
import torch

net = BiSeNet(19)
net.cuda()
net.load_state_dict(torch.load('/content/drive/My Drive/Collab/fp/res/cp/79999_iter.pth'))
net.eval()

dummy = torch.rand(1,3,512,512).cuda()
torch.onnx.export(net, dummy, "Model.onnx", input_names=["image"], output_names=["output"])

I added print (v.node ()) to symbolic_helper.py just before the runtime error is raised to see what's causing the error.

This is the output: %595 : Long() = onnx::Gather[axis=0](%592, %594) # /content/drive/My Drive/Collab/fp/model.py:111:0

And that line in 111 in model.py is: avg = F.avg_pool2d(feat32, feat32.size()[2:])

This source suggests that tensor.size method in pytorch cannot be recognized by onnx and needs to be modified into a constant.


title: How to realize a polynomial regression in Pytorch / Python

body:
I want my neural network to solve a polynomial regression problem like y=(x*x) + 2x -3.

So right now I created a network with 1 input node, 100 hidden nodes and 1 output node and gave it a lot of epochs to train with a high test data size. The problem is that the prediction after like 20000 epochs is okayish, but much worse then the linear regression predictions after training.

import torch
from torch import Tensor
from torch.nn import Linear, MSELoss, functional as F
from torch.optim import SGD, Adam, RMSprop
from torch.autograd import Variable
import numpy as np

# define our data generation function
def data_generator(data_size=1000):
    # f(x) = y = x^2 + 4x - 3
    inputs = []
    labels = []

    # loop data_size times to generate the data
    for ix in range(data_size):
        # generate a random number between 0 and 1000
        x = np.random.randint(1000) / 1000

        # calculate the y value using the function x^2 + 4x - 3
        y = (x * x) + (4 * x) - 3

        # append the values to our input and labels lists
        inputs.append([x])
        labels.append([y])

    return inputs, labels

# define the model
class Net(torch.nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = Linear(1, 100)
        self.fc2 = Linear(100, 1)

    def forward(self, x):
        x = F.relu(self.fc1(x)
        x = self.fc2(x)
        return x

model = Net()
# define the loss function
critereon = MSELoss()
# define the optimizer
optimizer = SGD(model.parameters(), lr=0.01)

# define the number of epochs and the data set size
nb_epochs = 20000
data_size = 1000

# create our training loop
for epoch in range(nb_epochs):
    X, y = data_generator(data_size)
    X = Variable(Tensor(X))
    y = Variable(Tensor(y))

    epoch_loss = 0;

    y_pred = model(X)

    loss = critereon(y_pred, y)

    epoch_loss = loss.data
    optimizer.zero_grad()

    loss.backward()

    optimizer.step()

    print("Epoch: {} Loss: {}".format(epoch, epoch_loss))

# test the model
model.eval()
test_data = data_generator(1)
prediction = model(Variable(Tensor(test_data[0][0])))
print("Prediction: {}".format(prediction.data[0]))
print("Expected: {}".format(test_data[1][0]))

Is their a way to get way better results? I wondered if I should try to get 3 outputs, call them a, b and c, such that y= a(x*x)+b(x)+c. But I have no idea how to implement that and train my neural network.


title: Can't send pytorch tensor to cuda

body:
I create a torch tensor and I want it to go to GPU but it doesn't. This is so broken. What's wrong?
def test_model_works_on_gpu():
    with torch.cuda.device(0) as cuda:
        some_random_d_model = 2 ** 9
        five_sentences_of_twenty_words = torch.from_numpy(np.random.random((5, 20, T * d))).float()
        five_sentences_of_twenty_words_mask = torch.from_numpy(np.ones((5, 1, 20))).float()
        pytorch_model = make_sentence_model(d_model=some_random_d_model, T_sgnn=T, d_sgnn=d)

        five_sentences_of_twenty_words.to(cuda)
        five_sentences_of_twenty_words_mask.to(cuda)
        print(type(five_sentences_of_twenty_words), type(five_sentences_of_twenty_words_mask))
        print(five_sentences_of_twenty_words.is_cuda, five_sentences_of_twenty_words_mask.is_cuda)
        pytorch_model.to(cuda)
        output_before_match = pytorch_model(five_sentences_of_twenty_words, five_sentences_of_twenty_words_mask)

        assert output_before_match.shape == (5, some_random_d_model)
        print(type(output_before_match))
        print(output_before_match.is_cuda, output_before_match.get_device())

tests/test_model.py:58: RuntimeError

<class 'torch.Tensor'> <class 'torch.Tensor'>
False False
<class 'torch.Tensor'>

>       print(output_before_match.is_cuda, output_before_match.get_device())
E       RuntimeError: get_device is not implemented for tensors with CPU backend

Also:
>>> torch.cuda.is_available()
True
>>> torch.cuda.device_count()
2

And:
pip freeze | grep -i torch
torch==1.0.0
torchvision==0.2.1


title: How to install an older version of gcc on Fedora

body:
I tried to install CUDA on Fedora 31 but got stuck at the last step because CUDA officially supports Fedora 29 (gcc 8.2) while the version shipped with Fedora 31 is 9.2, I then installed Pytorch with CUDA support and without much surprise, CUDA support is not present:
>>> import torch
>>> device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
>>> device
device(type='cpu')

I then tried to search for gcc, g++ and gfortran on pkgs, but got stuck in a dependency hell, something like:
sudo dnf install gcc-8.2.1-2.fc29.x86_64.rpm gcc-gfortran-8.2.1-2.fc29.x86_64.rpm gcc-c++-8.2.1-2.fc29.x86_64.rpm -y

Error:
 Problem 1: conflicting requests
  - nothing provides cpp = 8.2.1-2.fc29 needed by gcc-8.2.1-2.fc29.x86_64
  - nothing provides libgomp = 8.2.1-2.fc29 needed by gcc-8.2.1-2.fc29.x86_64
 Problem 2: conflicting requests
  - nothing provides libgfortran = 8.2.1-2.fc29 needed by gcc-gfortran-8.2.1-2.fc29.x86_64
  - nothing provides libquadmath = 8.2.1-2.fc29 needed by gcc-gfortran-8.2.1-2.fc29.x86_64
  - nothing provides libquadmath-devel = 8.2.1-2.fc29 needed by gcc-gfortran-8.2.1-2.fc29.x86_64
 Problem 3: conflicting requests
  - nothing provides libstdc++ = 8.2.1-2.fc29 needed by gcc-c++-8.2.1-2.fc29.x86_64
  - nothing provides libstdc++-devel = 8.2.1-2.fc29 needed by gcc-c++-8.2.1-2.fc29.x86_64

I found a bunch helpful links on this post but unfortunately the last version is gcc 7.3
My last option is building from source, but apart from that is there any other option? I'm not sure my computer can handle the compilation in a decent amount of time.


title: Cannot import BertModel from transformers

body:
I am trying to import BertModel from transformers, but it fails. This is code I am using

from transformers import BertModel, BertForMaskedLM

This is the error I get

ImportError: cannot import name 'BertModel' from 'transformers'

Can anyone help me fix this?


title: Pytorch expected type Long but got type int

body:
I revived an error

 Expected object of scalar type Long but got scalar type Int for argument #3 'index'

This is from this line.

targets = torch.zeros(log_probs.size()).scatter_(1, targets.unsqueeze(1).data.cpu(), 1)

I am not sure what to do as I tried to convert this to a long using several places. I tried putting a

.long

at the end as well as setting the dtype to be torch.long which still didn't work.

Very similar to this but he didn't do anything to get the answer
"Expected Long but got Int" while running PyTorch script

I have change a lot of the code and here is my last rendition, but is now giving me the same issue.

    def forward(self, inputs, targets):
            """
            Args:
                inputs: prediction matrix (before softmax) with shape (batch_size, num_classes)
                targets: ground truth labels with shape (num_classes)
            """
            log_probs = self.logsoftmax(inputs)
            targets = torch.zeros(log_probs.size()).scatter_(1, targets.unsqueeze(1).data.cpu(), 1)
            if self.use_gpu: targets = targets.to(torch.device('cuda'))
            targets = (1 - self.epsilon) * targets + self.epsilon / self.num_classes
            loss = (- targets * log_probs).mean(0).sum()
            return loss


title: Advantages and Disadvantages of MXNet compared to other Deep Learning APIs

body:
Recently I decided to learn MXNet, as some code I need to use, is written using this API.

However, I would like to know which are the advantages and disadvantages of MXNet compared to the other Deep Learning Libraries out there.


title: GKE - Unable to make cuda work with pytorch

body:
I have setup a kubernetes node with a nvidia tesla k80 and followed this tutorial to try to run a pytorch docker image with nvidia drivers and cuda drivers working.

My nvidia drivers and cuda drivers are all accessible inside my pod at /usr/local:

$> ls /usr/local
bin  cuda  cuda-10.0  etc  games  include  lib  man  nvidia  sbin  share  src

And my GPU is also recongnized by my image nvidia/cuda:10.0-runtime-ubuntu18.04:

$> /usr/local/nvidia/bin/nvidia-smi
Fri Nov  8 16:24:35 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.79       Driver Version: 410.79       CUDA Version: 10.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K80           Off  | 00000000:00:04.0 Off |                    0 |
| N/A   73C    P8    35W / 149W |      0MiB / 11441MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

But after installing pytorch 1.3.0 i'm not able to make pytorch recognize my cuda installation even with LD_LIBRARY_PATH set to /usr/local/nvidia/lib64:/usr/local/cuda/lib64:

$> python3 -c "import torch; print(torch.cuda.is_available())"
False

$> python3
Python 3.6.8 (default, Oct  7 2019, 12:59:55)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> print ('\t\ttorch.cuda.current_device()    =', torch.cuda.current_device())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/dist-packages/torch/cuda/__init__.py", line 386, in current_device
    _lazy_init()
  File "/usr/local/lib/python3.6/dist-packages/torch/cuda/__init__.py", line 192, in _lazy_init
    _check_driver()
  File "/usr/local/lib/python3.6/dist-packages/torch/cuda/__init__.py", line 111, in _check_driver
    of the CUDA driver.""".format(str(torch._C._cuda_getDriverVersion())))
AssertionError:
The NVIDIA driver on your system is too old (found version 10000).
Please update your GPU driver by downloading and installing a new
version from the URL: http://www.nvidia.com/Download/index.aspx
Alternatively, go to: https://pytorch.org to install
a PyTorch version that has been compiled with your version
of the CUDA driver.

The error above is strange because my cuda version for my image is 10.0 and Google GKE mentions that:

  The latest supported CUDA version is 10.0

Also, it's GKE's daemonsets that automatically installs NVIDIA drivers

  After adding GPU nodes to your cluster, you need to install NVIDIA's device drivers to the nodes.

  Google provides a DaemonSet that automatically installs the drivers for you.
  Refer to the section below for installation instructions for Container-Optimized OS (COS) and Ubuntu nodes.

  To deploy the installation DaemonSet, run the following command:
  kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/master/nvidia-driver-installer/cos/daemonset-preloaded.yaml

I have tried everything i could think of, without success...


title: Understanding log_prob for Normal distribution in pytorch

body:
I'm currently trying to solve Pendulum-v0 from the openAi gym environment which has a continuous action space. As a result, I need to use a Normal Distribution to sample my actions. What I don't understand is the dimension of the log_prob when using it :

import torch
from torch.distributions import Normal

means = torch.tensor([[0.0538],
        [0.0651]])
stds = torch.tensor([[0.7865],
        [0.7792]])

dist = Normal(means, stds)
a = torch.tensor([1.2,3.4])
d = dist.log_prob(a)
print(d.size())

I was expecting a tensor of size 2 (one log_prob for each actions) but it output a tensor of size(2,2).

However, when using a Categorical distribution for discrete environment the log_prob has the expected size:

logits = torch.tensor([[-0.0657, -0.0949],
        [-0.0586, -0.1007]])

dist = Categorical(logits = logits)
a = torch.tensor([1, 1])
print(dist.log_prob(a).size())

give me a tensor a size(2).

Why is the log_prob for Normal distribution of a different size ?


title: Implementing a simple ResNet block with PyTorch

body:
I'm trying to implement following ResNet block, which ResNet consists of blocks with two convolutional layers and a skip connection. For some reason it doesn't add the output of skip connection, if applied, or input to the output of convolution layers.

The ResNet block has:

Two convolutional layers with:

3x3 kernel
no bias terms
padding with one pixel on both sides
2d batch normalization after each convolutional layer

The skip connection:

simply copies the input if the resolution and the number of channels do not change.
if either the resolution or the number of channels change, the skip connection should have one convolutional layer with:

1x1 convolution without bias
change of the resolution with stride (optional)
different number of input channels and output channels (optional)
the 1x1 convolutional layer is followed by 2d batch normalization.

The ReLU nonlinearity is applied after the first convolutional layer and at the end of the block.

My code:

class Block(nn.Module):
    def __init__(self, in_channels, out_channels, stride=1):
        """
        Args:
          in_channels (int):  Number of input channels.
          out_channels (int): Number of output channels.
          stride (int):       Controls the stride.
        """
        super(Block, self).__init__()

        self.skip = nn.Sequential()

        if stride != 1 or in_channels != out_channels:
          self.skip = nn.Sequential(
            nn.Conv2d(in_channels=in_channels, out_channels=out_channels, kernel_size=1, stride=stride, bias=False),
            nn.BatchNorm2d(out_channels))
        else:
          self.skip = None

        self.block = nn.Sequential(
            nn.Conv2d(in_channels=in_channels, out_channels=out_channels, kernel_size=3, padding=1, stride=1, bias=False),
            nn.BatchNorm2d(out_channels),
            nn.ReLU(),
            nn.Conv2d(in_channels=in_channels, out_channels=out_channels, kernel_size=3, padding=1, stride=1, bias=False),
            nn.BatchNorm2d(out_channels))

    def forward(self, x):
        out = self.block(x)

        if self.skip is not None:
          out = self.skip(x)
        else:
          out = x

        out += x

        out = F.relu(out)
        return out


I am running this extremely simple PyTorch example NN from the documentation as is, with nothing at all changed.

I get this error:

Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/opt/conda/envs/fastai/lib/python3.6/site-packages/torch/nn/modules/module.py", line 357, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/envs/fastai/lib/python3.6/site-packages/torch/nn/modules/container.py", line 67, in forward
    input = module(input)
  File "/opt/conda/envs/fastai/lib/python3.6/site-packages/torch/nn/modules/module.py", line 357, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/envs/fastai/lib/python3.6/site-packages/torch/nn/modules/linear.py", line 55, in forward
    return F.linear(input, self.weight, self.bias)
  File "/opt/conda/envs/fastai/lib/python3.6/site-packages/torch/nn/functional.py", line 835, in linear
    return torch.addmm(bias, input, weight.t())
RuntimeError: addmm(): argument 'mat1' (position 1) must be Variable, not torch.FloatTensor

Apparently during the matrix multiplication, there is some data type error.
Why would the matrices I'm trying to multiply need to be Variable anyway?

I can do
x = Variable(torch.randn(N, D_in))
y = Variable(torch.randn(N, D_out))
but get
AttributeError: 'Variable' object has no attribute 'item'
so that didn't help.

I am running PyTorch version 0.3.1.post2.


title: Any example of torch 0.4.0 nn.LayerNorm example for nn.LSTMCell?

body:
In pytorch 0.4.0 release, there is a nn.LayerNorm module.

I want to implement this layer to my LSTM network, though I cannot find any implementation example on LSTM network yet.

And the pytorch Contributor implies that this nn.LayerNorm is only applicable through nn.LSTMCells.

It will be a great help if I can get any git repo or some code that implements nn.LayerNorm on nn.LSTMcell or any torch LSTM network.

Thanks in advance


title: fastai.vision Import Error: How to fix the import error so I can use ImageDataBunch.from_folder?

body:
I'm working on google colab ( python 3.6 and GPU), I import torch (1.2.0) nicely and I use the following to import fastai:

    import fastai
    print(fastai.__version__)
    from fastai import *
    from fastai.vision import *

I get the following error:

ImportError: /usr/local/lib/python3.6/dist-packages/torchvision/_C.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN3c106Device8validateEv

I have tried to install different torch versions like 1.0.0 and work with earlier python versions. I have also tried to install fastai and its dependencies manually using !pip, but nothing worked.

This is the complete code I used to install torch:

    from os.path import exists
    from wheel.pep425tags import get_abbr_impl, get_impl_ver, get_abi_tag

    platform = '{}{}-{}'.format(get_abbr_impl(), get_impl_ver(),
    get_abi_tag())
    cuda_output = !ldconfig -p|grep cudart.so|sed -e 's/.*\.\([0-9]*\)\.\
    ([0-9]*\)$/cu\1\2/'
    accelerator = cuda_output[0] if exists('/dev/nvidia0') else 'cpu'

    !pip install torch_nightly -f
 https://download.pytorch.org/whl/nightly/{accelerator}/torch_nightly.html
    !pip install fastai

   import torch

   print(torch.__version__)
   print(torch.cuda.is_available())
   print(torch.backends.cudnn.enabled)

And this is the full error message I get:

    ImportError
    Traceback (most recent call last)
    <ipython-input-5-4b8b8d8134df> in <module>()
  2 print(fastai.__version__)
  3 from fastai import *
  ----> 4 from fastai.vision import *
    8 frames
  /usr/local/lib/python3.6/dist-packages/torchvision/ops/boxes.py in
  <module>()
  1 import torch
  ---->  2 from torchvision import _C
  3
  4
  5 def nms(boxes, scores, iou_threshold):

  ImportError: /usr/local/lib/python3.6/dist-
  packages/torchvision/_C.cpython-36m-x86_64-linux-gnu.so: undefined
  symbol: _ZN3c106Device8validateEv

I'm unable to use ImageDataBunch.from_folder because of this fastai import error. I get the error NameError: name 'ImageDataBunch' is not defined when I do.

Note: I did use the same code before and I was able to use fastai and  ImageDataBunch.from_folder with no import errors , but I'm guessing that an update to fastai or torch happened.


As I've read through some previous questions, I'm getting this error that probably has something to do with a discrepancy between dimensions of tensors but since this is my very first attempt of running PyTorch so I'm coming here because I have far to little intuition about it. Wanted to run a non-standard dataset (which I'm pretty sure I'm loading fine) on a basic MNIST setup to mess with it and see what moves what.

Traceback (most recent call last):

File "C:/Users/Administrator/Desktop/pytong/proj/pytorch_cnnv2.py", line 110, in <module>
loss = loss_func(output, b_y)
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\nn\modules\module.py", line 547, in __call__
result = self.forward(*input, **kwargs)
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\nn\modules\loss.py", line 916, in forward
ignore_index=self.ignore_index, reduction=self.reduction)
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\nn\functional.py", line 1995, in cross_entropy
return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\nn\functional.py", line 1316, in log_softmax
ret = input.log_softmax(dim)

IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

The line making the fuss is the loss function:

loss = loss_func(output, b_y)

The rest of the code, sans imports and load:

class CNNModel(nn.Module):
def __init__(self):
    super(CNNModel, self).__init__()

    # Convolution 1
    self.cnn1 = nn.Conv2d(in_channels=3, out_channels=16, kernel_size=5, stride=1, padding=0)
    self.relu1 = nn.ReLU()

    # Max pool 1
    self.maxpool1 = nn.MaxPool2d(kernel_size=2)

    # Convolution 2
    self.cnn2 = nn.Conv2d(in_channels=16, out_channels=32, kernel_size=5, stride=1, padding=0)
    self.relu2 = nn.ReLU()

    # Max pool 2
    self.maxpool2 = nn.MaxPool2d(kernel_size=2)

    # Fully connected
    self.fc1 = nn.Linear(338 * 4 * 4, 5)

def forward(self, x):
    # Convolution 1
    out = self.cnn1(x)
    print(out.shape)
    out = self.relu1(out)
    print(out.shape)

    # Max pool 1
    out = self.maxpool1(out)
    print(out.shape)

    # Convolution 2
    out = self.cnn2(out)
    print(out.shape)
    out = self.relu2(out)
    print(out.shape)

    # Max pool 2
    out = self.maxpool2(out)
    print('++++++++++++++ out')
    print(out.shape)
    # out = out.reshape(-1, 169 * 4 * 4)
    out = out.view(out.size(0), -1)
    print(out.shape)
    print('-----------------------')
    # Linear function (readout)
    out = self.fc1(out)
    print(out.shape)
    print('=======================')

    return out

if __name__ == '__main__':

print("Number of train samples: ", len(train_data))
print("Number of test samples: ", len(test_data))
print("Detected Classes are: ", train_data.class_to_idx) # classes are detected by folder structure

model = CNNModel()
optimizer = torch.optim.Adam(model.parameters(), lr=LEARNING_RATE)
loss_func = nn.CrossEntropyLoss()

# Training and Testing
for epoch in range(EPOCHS):
    print(enumerate(train_data_loader))
    for step, (x, y) in enumerate(train_data_loader):
        b_x = Variable(x)   # batch x (image)
        b_y = Variable(y)   # batch y (target)
        # print('============ b_x')
        # print(len(b_x))
        # print(b_x.data)
        # print('============ b_y')
        # print(len(b_y))
        # print(b_y.data)

        output = model(b_x)[0]
        loss = loss_func(output, b_y)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if step % 50 == 0:
            test_x = Variable(test_data_loader)
            test_output, last_layer = model(test_x)
            pred_y = torch.max(test_output, 1)[1].data.squeeze()
            accuracy = sum(pred_y == test_y) / float(test_y.size(0))
            print('Epoch: ', epoch, '| train loss: %.4f' % loss.data[0], '| test accuracy: %.2f' % accuracy)

Plus the output of my prints that I've tried to use for diagnostics:

torch.Size([100, 16, 60, 60])
torch.Size([100, 16, 60, 60])
torch.Size([100, 16, 30, 30])
torch.Size([100, 32, 26, 26])
torch.Size([100, 32, 26, 26])
++++++++++++++ out
torch.Size([100, 32, 13, 13])
torch.Size([100, 5408])
-----------------------
torch.Size([100, 5])
=======================


title: PyTorch optimizer.step() function doesn't update weights

body:
The code can be seen below.
The problem is, that the optimizer.step() part doesn't work. I'm printing model.parameters() before and after the training, and the weights don't change.

I'm trying to make a perceptron that can solve the AND-problem. I've been successful in doing this with my own tiny library, where I've implemented a perceptron with the two functions predict() and train().

Just to clarify, I've just started learning deep learning using PyTorch, so it's probably a very newbie problem. I've tried searching for a solution, but without luck. I've also compared my code with other codes that work, but I don't know what I'm doing wrong.

import torch
from torch import nn, optim
from random import randint

class NeuralNet(nn.Module):
  def __init__(self):
    super(NeuralNet, self).__init__()
    self.layer1 = nn.Linear(2, 1)

  def forward(self, input):
    out = input
    out = self.layer1(out)
    out = torch.sign(out)
    out = torch.clamp(out, 0, 1) # 0=false, 1=true
    return out

data = torch.Tensor([[0, 0], [0, 1], [1, 0], [1, 1]])
target = torch.Tensor([0, 0, 0, 1])
model = NeuralNet()
epochs = 1000
lr = 0.01

print(list(model.parameters()))
print() # Print parameters before training
loss_func = nn.L1Loss()
optimizer = optim.Rprop(model.parameters(), lr)
for epoch in range(epochs + 1):
  optimizer.zero_grad()
  rand_int = randint(0, len(data) - 1)
  x = data[rand_int]
  y = target[rand_int]

  pred = model(x)
  loss = loss_func(pred, y)

  loss.backward()
  optimizer.step()

# Print parameters again
# But they haven't changed
print(list(model.parameters()))


title: LSTM's expected hidden state dimensions doesn't take batch size into account

body:
I have this decoder model, which is supposed to take batches of sentence embeddings (batchsize = 50, hidden size=300) as input and output a batch of one hot representation of predicted sentences:

class DecoderLSTMwithBatchSupport(nn.Module):
        # Your code goes here
        def __init__(self, embedding_size,batch_size, hidden_size, output_size):
            super(DecoderLSTMwithBatchSupport, self).__init__()
            self.hidden_size = hidden_size
            self.batch_size = batch_size
            self.lstm = nn.LSTM(input_size=embedding_size,num_layers=1, hidden_size=hidden_size, batch_first=True)
            self.out = nn.Linear(hidden_size, output_size)
            self.softmax = nn.LogSoftmax(dim=1)

        def forward(self, my_input, hidden):
            print(type(my_input), type(hidden))
            output, hidden = self.lstm(my_input, hidden)
            output = self.softmax(self.out(output[0]))
            return output, hidden

        def initHidden(self):
            return Variable(torch.zeros(1, self.batch_size, self.hidden_size)).cuda()

However, when I run it using:

decoder=DecoderLSTMwithBatchSupport(vocabularySize,batch_size, 300, vocabularySize)
decoder.cuda()
decoder_input=np.zeros([batch_size,vocabularySize])
    for i in range(batch_size):
        decoder_input[i] = embeddings[SOS_token]
    decoder_input=Variable(torch.from_numpy(decoder_input)).cuda()
    decoder_hidden = (decoder.initHidden(),decoder.initHidden())
        for di in range(target_length):
            decoder_output, decoder_hidden = decoder(decoder_input.view(1,batch_size,-1), decoder_hidden)

I get he following error:

  Expected hidden[0] size (1, 1, 300), got (1, 50, 300)

What am I missing in order to make the model expect batched hidden states?


title: Pyinstaller executable fails importing torchvision

body:
This is my main.py:

import torchvision
input("Press key")

It runs correctly in the command line: python main.py

I need an executable for windows. So I did : pyinstaller main.py

But when I launched the main.exe, inside /dist/main I got this error:

Traceback (most recent call last):
  File "main.py", line 1, in <module>

  ... (omitted)

  File "site-packages\torchvision\ops\misc.py", line 135, in <module>
  File "site-packages\torchvision\ops\misc.py", line 148, in FrozenBatchNorm2d
  File "site-packages\torch\jit\__init__.py", line 850, in script_method
  File "site-packages\torch\jit\frontend.py", line 152, in get_jit_def
  File "inspect.py", line 973, in getsource
  File "inspect.py", line 955, in getsourcelines
  File "inspect.py", line 786, in findsource
OSError: could not get source code
[2836] Failed to execute script main

It seems that some source code is not correctly imported from pyinstaller. I am not sure if the problems is the torch module or torchvision.

Additional info:

I recently installed Visual Studio 2019

System info:

Window 10
Python 3.7
torch-1.1.0
torchvision-0.3.0

[EDIT]

I found that the problem is in the definition of the class FrozenBatchNorm2d inside torchvision. The following script produce the same error as the one before posted:

main.py

import torch

class FrozenBatchNorm2d(torch.jit.ScriptModule):

    def __init__(self, n):
        super(FrozenBatchNorm2d, self).__init__()

    @torch.jit.script_method

    def forward(self):
        pass

I copied all the torch source file. But I still got the error...


title: PyTorch replace torch.nn.Conv2d with torch.nn.functional.conv2d

body:
So I have this MNIST example for PyTorch.
I wanted to replace conv2d with functional method. But got unexpected error.

I replace self.conv1 = nn.Conv2d(1, 32, 5, padding=2) with self.w_conv1 = Variable(torch.randn(1, 32, 5))

In the forward method I replace x = F.max_pool2d(F.relu(self.conv1(x)), 2) with x = F.max_pool2d(F.relu(F.conv2d(x, self.w_conv1, padding=2),2))

And then it will give me an error:

Expected 4-dimensional input for 4-dimensional weight [1, 32, 5], but got input of size [50, 1, 28, 28] instead

The code worked before, and I thought I'd replace the class with it's functional equivalent.


title: Loading custom dataset of images using PyTorch

body:
I'm using the coil-100 dataset which has images of 100 objects, 72 images per object taken from a fixed camera by turning the object 5 degrees per image. Following is the folder structure I'm using:

  data/train/obj1/obj01_0.png, obj01_5.png ... obj01_355.png
  .
  .
  data/train/obj85/obj85_0.png, obj85_5.png ... obj85_355.png
  .
  .
  data/test/obj86/obj86_0.ong, obj86_5.png ... obj86_355.png
  .
  .
  data/test/obj100/obj100_0.ong, obj100_5.png ... obj100_355.png

I have used the imageloader and dataloader classes. The train and test datasets loaded properly and I can print the class names.

train_path = 'data/train/'
test_path = 'data/test/'
data_transforms = {
    transforms.Compose([
    transforms.Resize(224, 224),
    transforms.ToTensor(),
    transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
    ])
}

train_data = torchvision.datasets.ImageFolder(
    root=train_path,
    transform= data_transforms
)
test_data = torchvision.datasets.ImageFolder(
    root = test_path,
    transform = data_transforms
)
train_loader = torch.utils.data.DataLoader(
    train_data,
    batch_size=None,
    num_workers=1,
    shuffle=False
)
test_loader = torch.utils.data.DataLoader(
    test_data,
    batch_size=None,
    num_workers=1,
    shuffle=False
)

print(len(train_data))
print(len(test_data))
classes = train_data.class_to_idx
print("detected classes: ", classes)

In my model I wish to pass every image through pretrained resnet and make a dataset from the output of resnet to feed into a biderectional LSTM.
For which I need to access the images by classname and index.
for ex. pre_resnet_train_data['obj01'][0] should be obj01_0.png and post_resnet_train_data['obj01'][0] should be the resnet output of obj01_0.png and so on.
I'm a beginner in Pytorch and for the past 2 days, I have read many tutorials and stackoverflow questions about creating a custom dataset class but couldn't figure out how to achieve what I want.
please help!


title: how to add list of arrays (tensors)

body:
I am defining a simple conv2d function to calculate the cross-correlation between input and kernel (both 2D tensor) as below:

import torch

def conv2D(X, K):
    h = K.shape[0]
    w = K.shape[1]
    ĥ = X.shape[0] - h + 1
    ŵ = X.shape[1] - w + 1
    Y = torch.zeros((ĥ, ŵ))
    for i in range (ĥ):
        for j in range (ŵ):
            Y[i, j] = (X[i: i+h, j: j+w]*K).sum()

    return Y

When X and K are of rank-3 tensor, I calculate the conv2d for each channel and then add them together as below:

def conv2D_multiple(X, K):
    cross = []
    result = 0
    for x, k in zip(X, K):
        cross.append(conv2D(x,k))

    for t in cross:
        result += t

    return result

To test my function:

X_2 = torch.tensor([[[0, 1, 2], [3, 4, 5], [6, 7, 8]],
                    [[1, 2, 3], [4, 5, 6], [7, 8, 9]]], dtype=torch.float32)
K_2 = torch.tensor([[[0, 1], [2, 3]], [[1, 2], [3, 4]]], dtype=torch.float32)

conv2D_multiple(X_2, K_2)

The results is:

tensor([[ 56.,  72.],
        [104., 120.]])

The result is as expected, however, I believe my second
for loop inside conv2D_multiple(X, K) function is redundant.  My question is how to sum (element wise)
tensors (arrays) in the list so I omit the second for loop.


title: Torch code producing CUDA Runtime Error

body:
a friend of mine implemented a sparse version of torch.bmm that actually works, but when I try a test, I have a runtime error (that has nothing to do with this implementation), that I don't understand. I have seen a few topics about if but couldn't find a solution. Here is the code, and the error:

if __name__ == "__main__":
     tmp = torch.zeros(1).cuda()
     batch_csr = BatchCSR()
     sparse_bmm = SparseBMM()

     i=torch.LongTensor([[0,5,8], [1,5,8], [2,5,8]])
     v=torch.FloatTensor([4,3,8])
     s=torch.Size([3,500,500])

     indices, values, size = i,v,s

     a_ = torch.sparse.FloatTensor(indices, values, size).cuda().transpose(2, 1)
     batch_size, num_nodes, num_faces = a_.size()

     a = a_.to_dense()

     for _ in range(10):
        b = torch.randn(batch_size, num_faces, 16).cuda()
        torch.cuda.synchronize()
        time1 = time.time()
        result = torch.bmm(a, b)
        torch.cuda.synchronize()
        time2 = time.time()
        print("{} CuBlas dense bmm".format(time2 - time1))

        torch.cuda.synchronize()
        time1 = time.time()
        col_ind, col_ptr = batch_csr(a_.indices(), a_.size())
        my_result = sparse_bmm(a_.values(), col_ind, col_ptr, a_.size(), b)
        torch.cuda.synchronize()
        time2 = time.time()
        print("{} My sparse bmm".format(time2 - time1))

        print("{} Diff".format((result-my_result).abs().max()))

And the error:

Traceback (most recent call last):
  File "sparse_bmm.py", line 72, in <module>
    b = torch.randn(3, 500, 16).cuda()
  File "/home/bizeul/virtual_env/lib/python2.7/site-packages/torch/_utils.py", line 65, in _cuda
    return new_type(self.size()).copy_(self, async)
RuntimeError: cuda runtime error (59) : device-side assert triggered at /b/wheel/pytorch-src/torch/lib/THC/generic/THCTensorCopy.c:18

When running with the command CUDA_LAUNCH_BLOCKING=1, I get the error :

/b/wheel/pytorch-src/torch/lib/THC/THCTensorIndex.cu:121: void indexAddSmallIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 1, SrcDim = 1, IdxDim = -2]: block: [0,0,0], thread: [0,0,0] Assertion `dstIndex < dstAddDimSize` failed.
THCudaCheck FAIL file=/b/wheel/pytorch-src/torch/lib/THCS/generic/THCSTensorMath.cu line=292 error=59 : device-side assert triggered
Traceback (most recent call last):
  File "sparse_bmm.py", line 69, in <module>
    a = a_.to_dense()
RuntimeError: cuda runtime error (59) : device-side assert triggered at /b/wheel/pytorch-src/torch/lib/THCS/generic/THCSTensorMath.cu:292


title: How to use different data augmentation for Subsets in PyTorch

body:
How to use different data augmentation (transforms) for different Subsets in PyTorch?

For instance:

train, test = torch.utils.data.random_split(dataset, [80000, 2000])

train and test will have the same transforms as dataset. How to use custom transforms for these subsets?


title: Pytorch custom dataset: ValueError: some of the strides of a given numpy array are negative

body:
I wrote a custom pytorch dataset, but ran into an error thhat seems quite unintelligible.

My custom dataset,

class data_from_xlsx(Dataset):
    def __init__(self, xlsx_fp, path_col, class_cols_list):
        self.xlsx_file = pd.read_excel(xlsx_fp)
        self.path_col = path_col
        self.class_cols_list = class_cols_list

    def __len__(self):
        return get_xlsx_length(self.xlsx_file)

    def __getitem__(self, index):
        file_path = cols_from_xlsx(self.xlsx_file, index, 1, self.path_col)
        feature = load_nii_file(file_path) # get 3D volume (x, y, z)
        feature = np.expand_dims(feature, axis=0) # add channel (c, x, y, z)
        label = cols_from_xlsx(self.xlsx_file, index, 1, self.class_cols_list) # get label
        return feature, label.astype(np.bool)

def main():
dataset = data_from_xlsx("train.xlsx", "file_path", ["pos", "neg"], transformations, aug=True)
    data_loader = DataLoader(dataset, batch_size=4, shuffle=True)

    for (f, l) in data_loader:
        print("f shape", f.shape)
        print("l shape", l.shape)

An error is reported when I ran main(),

 File "d:\pytorch\lib\site-packages\torch\utils\data\dataloader.py", line 346, in __next__
    data = self.dataset_fetcher.fetch(index)  # may raise StopIteration
  File "d:\pytorch\lib\site-packages\torch\utils\data\_utils\fetch.py", line 47, in fetch
    return self.collate_fn(data)
  File "d:\pytorch\lib\site-packages\torch\utils\data\_utils\collate.py", line 80, in default_collate
    return [default_collate(samples) for samples in transposed]
  File "d:\pytorch\lib\site-packages\torch\utils\data\_utils\collate.py", line 80, in <listcomp>
    return [default_collate(samples) for samples in transposed]
  File "d:\pytorch\lib\site-packages\torch\utils\data\_utils\collate.py", line 65, in default_collate
    return default_collate([torch.as_tensor(b) for b in batch])
  File "d:\pytorch\lib\site-packages\torch\utils\data\_utils\collate.py", line 65, in <listcomp>
    return default_collate([torch.as_tensor(b) for b in batch])
ValueError: some of the strides of a given numpy array are negative. This is currently not supported, but will be added in future release

The reported error does't make sense to me, so I googled it. At first I thought I didn't change the feature from numpy.array to tensor, so I tried feature = torch.from_array(feature.copy()) and also tried transforms.TOTensor() but both attempts failed.


title: Resnet-18 as backbone in Faster R-CNN

body:
I code with pytorch and I want to use resnet-18 as backbone of Faster R-RCNN. When I print structure of resnet18, this is the output:

>>import torch
>>import torchvision
>>import numpy as np
>>import torchvision.models as models

>>resnet18 = models.resnet18(pretrained=False)
>>print(resnet18)

ResNet(
  (conv1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
  (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (relu): ReLU(inplace=True)
  (maxpool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)
  (layer1): Sequential(
    (0): BasicBlock(
      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
    (1): BasicBlock(
      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
  )
  (layer2): Sequential(
    (0): BasicBlock(
      (conv1): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (downsample): Sequential(
        (0): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False)
        (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    (1): BasicBlock(
      (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
  )
  (layer3): Sequential(
    (0): BasicBlock(
      (conv1): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (downsample): Sequential(
        (0): Conv2d(128, 256, kernel_size=(1, 1), stride=(2, 2), bias=False)
        (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    (1): BasicBlock(
      (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
  )
  (layer4): Sequential(
    (0): BasicBlock(
      (conv1): Conv2d(256, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (downsample): Sequential(
        (0): Conv2d(256, 512, kernel_size=(1, 1), stride=(2, 2), bias=False)
        (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    (1): BasicBlock(
      (conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
  )
  (avgpool): AdaptiveAvgPool2d(output_size=(1, 1))
  (fc): Linear(in_features=512, out_features=1000, bias=True)
)

My question is, until which layer it is feature extractor?  is AdaptiveAvgPool2d should be part of backbone of Faster R-CNN?

In this toturial, it is shown how to train a Mask R-CNN with an arbitrary backbone, I want to do the same thing with Faster R-CNN and train a Faster R-CNN with resnet-18 but until which layer should be part of feature extractor is confusing for me.

I know how to use resnet+Feature Pyramid Network as backbone, My question is about resent.


title: Can ReduceLrOnPlateau scheduler in pytorch use test set metric for decreasing learning rate?

body:
Hi I am currently learning the use of scheduler in deep learning in pytroch. I came across the following code :

import torch
import torch.nn as nn
import torchvision.transforms as transforms
import torchvision.datasets as dsets

# Set seed
torch.manual_seed(0)

# Where to add a new import
from torch.optim.lr_scheduler import ReduceLROnPlateau

'''
STEP 1: LOADING DATASET
'''

train_dataset = dsets.MNIST(root='./data',
                            train=True,
                            transform=transforms.ToTensor(),
                            download=True)

test_dataset = dsets.MNIST(root='./data',
                           train=False,
                           transform=transforms.ToTensor())

'''
STEP 2: MAKING DATASET ITERABLE
'''

batch_size = 100
n_iters = 6000
num_epochs = n_iters / (len(train_dataset) / batch_size)
num_epochs = int(num_epochs)

train_loader = torch.utils.data.DataLoader(dataset=train_dataset,
                                           batch_size=batch_size,
                                           shuffle=True)

test_loader = torch.utils.data.DataLoader(dataset=test_dataset,
                                          batch_size=batch_size,
                                          shuffle=False)

'''
STEP 3: CREATE MODEL CLASS
'''
class FeedforwardNeuralNetModel(nn.Module):
    def __init__(self, input_dim, hidden_dim, output_dim):
        super(FeedforwardNeuralNetModel, self).__init__()
        # Linear function
        self.fc1 = nn.Linear(input_dim, hidden_dim)
        # Non-linearity
        self.relu = nn.ReLU()
        # Linear function (readout)
        self.fc2 = nn.Linear(hidden_dim, output_dim)

    def forward(self, x):
        # Linear function
        out = self.fc1(x)
        # Non-linearity
        out = self.relu(out)
        # Linear function (readout)
        out = self.fc2(out)
        return out
'''
STEP 4: INSTANTIATE MODEL CLASS
'''
input_dim = 28*28
hidden_dim = 100
output_dim = 10

model = FeedforwardNeuralNetModel(input_dim, hidden_dim, output_dim)

'''
STEP 5: INSTANTIATE LOSS CLASS
'''
criterion = nn.CrossEntropyLoss()

'''
STEP 6: INSTANTIATE OPTIMIZER CLASS
'''
learning_rate = 0.1

optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate, momentum=0.9, nesterov=True)

'''
STEP 7: INSTANTIATE STEP LEARNING SCHEDULER CLASS
'''
# lr = lr * factor
# mode='max': look for the maximum validation accuracy to track
# patience: number of epochs - 1 where loss plateaus before decreasing LR
        # patience = 0, after 1 bad epoch, reduce LR
# factor = decaying factor
scheduler = ReduceLROnPlateau(optimizer, mode='max', factor=0.1, patience=0, verbose=True)

'''
STEP 7: TRAIN THE MODEL
'''
iter = 0
for epoch in range(num_epochs):
    for i, (images, labels) in enumerate(train_loader):
        # Load images as Variable
        images = images.view(-1, 28*28).requires_grad_()

        # Clear gradients w.r.t. parameters
        optimizer.zero_grad()

        # Forward pass to get output/logits
        outputs = model(images)

        # Calculate Loss: softmax --> cross entropy loss
        loss = criterion(outputs, labels)

        # Getting gradients w.r.t. parameters
        loss.backward()

        # Updating parameters
        optimizer.step()

        iter += 1

        if iter % 500 == 0:
            # Calculate Accuracy
            correct = 0
            total = 0
            # Iterate through test dataset
            for images, labels in test_loader:
                # Load images to a Torch Variable
                images = images.view(-1, 28*28)

                # Forward pass only to get logits/output
                outputs = model(images)

                # Get predictions from the maximum value
                _, predicted = torch.max(outputs.data, 1)

                # Total number of labels
                total += labels.size(0)

                # Total correct predictions
                # Without .item(), it is a uint8 tensor which will not work when you pass this number to the scheduler
                correct += (predicted == labels).sum().item()

            accuracy = 100 * correct / total

            # Print Loss
            # print('Iteration: {}. Loss: {}. Accuracy: {}'.format(iter, loss.data[0], accuracy))

    # Decay Learning Rate, pass validation accuracy for tracking at every epoch
    print('Epoch {} completed'.format(epoch))
    print('Loss: {}. Accuracy: {}'.format(loss.item(), accuracy))
    print('-'*20)
    scheduler.step(accuracy)

I am using the above strategy. The only thing that I am not able to understand is that how are they using test data to step up the accuracy and decreasing learning rate on the basis of that via scheduler? It is the last line of the code. Can we during training show the test accuracy to the scheduler and ask that it to reduce the learning rate? I found the similar thing on github resnet main.py
too. Can someone please clarify ?


title: EXE using Pytorch and packed with PyInstaller cannot run on the other PC

body:
I built an exe of a program using pytorch. It works well on the PC where the exe was built on but not on the other PC. An ImportError raised, DLL load failed.

I'm new to pytorch and pyinstaller so I can't get what happend.

The ImportError:

F:\test>test D:\testData.xml D:\testResult.xml
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0\libnvvp;D:\Program Files\Java\jdk-9.0.4\bin;D:\Program Files\Java\jdk-9.0.4\jre\bin;C:\ProgramData\Oracle\Java\javapath;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;D:\Program Files\MATLAB\R2016a\runtime\win64;D:\Program Files\MATLAB\R2016a\bin;D:\Program Files\MATLAB\R2016a\polyspace\bin;D:\FFMPEG\bin;C:\WINDOWS\System32\OpenSSH\;D:\Anaconda3;D:\Anaconda3\Scripts;D:\Anaconda3;D:\Anaconda3\Library\mingw-w64\bin;D:\Anaconda3\Library\usr\bin;D:\Anaconda3\Library\bin;D:\Anaconda3\Scripts;D:\Python\Python36\Scripts\;D:\Python\Python36\;C:\Users\snow-\AppData\Local\Microsoft\WindowsApps;D:\Program Files\modeltech64_10.4\win64;D:\modeltech64_10.4\win64;D:\Modeltech_pe_edu_10.4a\win32pe_edu;F:\test;F:\test\torch\lib
Traceback (most recent call last):
  File "test.py", line 10, in <module>
  File "C:\Users\yuech\AppData\Local\conda\conda\envs\cy_pytorch\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module
  File "Lib\site-packages\torch\__init__.py", line 102, in <module>
ImportError: DLL load failed: 找不到指定的模块。
[10812] Failed to execute script test

I print all the 'PATH'

the line 6 is

import torch

the line 102 is

from torch._C import *

I searched on internet and I found it might be caused by openmp or numpy. I checked and I do have them in the folder.

I tried to add variable environment 'PATH' but it doesn't work.

I use pytorch 1.0.1, python 3.6.8 and pyinstaller 3.4

I built my exe with this line

pyinstaller test.py --exclude-module PyQt5 -p dataLoader_cy.py --hidden-import dataLoader_cy -p C:\Users\yuech\AppData\Local\conda\conda\envs\cy_pytorch\Lib\site-packages --hidden-import
 torchvision --hidden-import numpy

I tried adding the environment variable with this

import os
os.environ['PATH'] += (";" + os.path.abspath('.'))
os.environ['PATH'] += (";" + os.path.join(os.path.abspath('.'), "torch\lib"))

Since my experience in python and pytorch is limited, I can' get why it can work on one PC but not on the others. I want my exe works on the other windows. Please help me if you have some idea. Thank you!


title: How to compute cross entropy loss for binary classification in Pytorch ？

body:
For binary classification, my output and label is like this

output = [0.7, 0.3, 0.1, 0.9 ... ]
label = [1, 0, 0, 1 ... ]

where the output is the probability for precited label = 1

And I want a cross entropy like this:

def cross_entropy(output, label):
    return sum(-label * log(output) - (1 - label) * log(1 - output))

However, this gives me a NaN error because that in log(output) the output might be zero.

I know there is torch.nn.CrossEntropyLoss however it does not apply for my data format here.


title: How to install pytorch on windows subsystem for linux

body: my windows10 has subsystem for linux of 14.04. I tried  to install pytorch on the preinstalled python2 but couldn't work.The error is: torch-0.2.0.post1-cp27-cp27m-manylinux1_x86_64.whl is not a supported wheel on this platform. I tried to install python3.6 then install pytorch with it, but still couldn't work.The error is missing module 'apt_pkg'. Anyone has idea on this?


title: How can I run both TensorFlow and Torch on Mac M1 MacBook Pro?

body:
I encounter some problems on my MacBook Pro M1. I thought it would be easier to start coding on it, apparently it's not a ML beast yet...
I need to use both PyTorch and TensorFlow on Python. I have installed TensorFlow 2.0 for Mac OS.
The problem is: TensorFlow won't work when you use a x86_64 terminal. (So it doesn't work with PyCharm). However, I can import TensorFlow 2.0 from an arm terminal.
Paradoxically, PyTorch won't install on a arm terminal, only on a x86_64 terminal. So, on the same Python terminal, I'm not able to import both torch and TensorFlow 2.0.
Since HuggingFace transformers is crucial for me, and transformers needs both TensorFlow 2.0 and PyTorch, I need to go back on my old computer to code. I'm very disappointed!
Anyone successfully imported both PyTorch and TensorFlow on a Mac M1 device?
And does anyone know if there is a way to force PyCharm to use an arm terminal, so I can use TensorFlow 2.0 on PyCharm on my M1 MPB?
Thank you!


title: How does one create a data set in pytorch and save it into a file to later be used?

body:
I want to extract data from cifar10 in a specific order according to some criterion f(Image,label) (for the sake of an example lets say f(Image,label) simply computes the sum of all the pixels in Image). Then it I want to generate 1 file for the train set and 1 file for the test set that I can later load in a dataloader to use for training a neural net.

How do I do this? My current idea was simply to loop through the data with data loader with shuffle off and remember the indices of the images and the score and then sort the indices according to the score and then loop through everything again and create some giant numpy array and save it. After I save it I’d use torch.utils.data.TensorDataset(X_train, X_test) to wrap with TensorDataset and feed to DataLoader.

I think it might work for a small data set like cifar10 at the very least, right?

Another very important thing for me is that I also want to only train on the first K images (especially since I already sorted them the first K have a special meaning which I want to keep) so respecting but training only with a fraction will be important.

https://discuss.pytorch.org/t/how-does-one-create-a-data-set-in-pytorch-and-save-it-into-a-file-to-later-be-used/16742


title: pytorch "log_softmax_lastdim_kernel_impl" not implemented for 'torch.LongTensor'

body:
I am trying to use my own dataset to classify text according to https://github.com/bentrevett/pytorch-sentiment-analysis/blob/master/5%20-%20Multi-class%20Sentiment%20Analysis.ipynb. My dataset is a csv of sentences and a class associated with it. there are 6 different classes:

sent                      class
'the fox is brown'        animal
'the house is big'        object
'one water is drinkable'  water
...

When running:

N_EPOCHS = 5

best_valid_loss = float('inf')

for epoch in range(N_EPOCHS):

    start_time = time.time()
    print(start_time)
    train_loss, train_acc = train(model, train_iterator, optimizer, criterion)
    print(train_loss.type())
    print(train_acc.type())
    valid_loss, valid_acc = evaluate(model, valid_iterator, criterion)

    end_time = time.time()

    epoch_mins, epoch_secs = epoch_time(start_time, end_time)

    if valid_loss < best_valid_loss:
        best_valid_loss = valid_loss
        torch.save(model.state_dict(), 'tut5-model.pt')

    print(f'Epoch: {epoch+1:02} | Epoch Time: {epoch_mins}m {epoch_secs}s')
    print(f'\tTrain Loss: {train_loss:.3f} | Train Acc: {train_acc*100:.2f}%')
    print(f'\t Val. Loss: {valid_loss:.3f} |  Val. Acc: {valid_acc*100:.2f}%')

, I receive the following error

RuntimeError: "log_softmax_lastdim_kernel_impl" not implemented for 'torch.LongTensor'

pointing to:

<ipython-input-38-9c6cff70d2aa> in train(model, iterator, optimizer, criterion)
     14         print('pred'+ predictions.type())
     15         #batch.label = batch.label.type(torch.LongTensor)
---> 16         loss = criterion(predictions.long(), batch.label)**

The solution posted here https://github.com/pytorch/pytorch/issues/14224 suggests I need to use long/int.

I had to add .long() at line ** in order to fix this earlier error:

RuntimeError: Expected object of scalar type Long but got scalar type Float for argument #2 'target'

The specific lines of code are:

  def train(model, iterator, optimizer, criterion):
    epoch_loss = 0
    epoch_acc = 0

    model.train()

    for batch in iterator:

        optimizer.zero_grad()

        predictions = model(batch.text)
        print('pred'+ predictions.type())
        #batch.label = batch.label.type(torch.LongTensor)
        loss = criterion(predictions.long(), batch.label)**

        acc = categorical_accuracy(predictions, batch.label)

        loss.backward()

        optimizer.step()

        epoch_loss += loss.item()
        epoch_acc += acc.item()

    return epoch_loss / len(iterator), epoch_acc / len(iterator)

Note, the ** was originally loss = criterion(predictions, batch.label)

Any other suggestions to fix this issue?


title: How to take a transpose for each matrix in a batch in Pytorch?

body: Say I have a 4 batch of 5x3 matrixes. So the dimensions of these tensor are 4x5x3. How do I take the transpose of each matrix within each batch. So converting it to 4x3x5?


title: AttributeError: 'Image' object has no attribute 'new' occurs when trying to use Pytorchs AlexNet Lighting preprocessing

body:
I tried to train my model on ImageNet using inception and Alexnet like preprocessing. I used Fast-ai imagenet training script provided script.   Pytorch has support for inception like preprocessing but for AlexNets Lighting, we have to implement it ourselves :

__imagenet_pca = {
    'eigval': torch.Tensor([0.2175, 0.0188, 0.0045]),
    'eigvec': torch.Tensor([
        [-0.5675,  0.7192,  0.4009],
        [-0.5808, -0.0045, -0.8140],
        [-0.5836, -0.6948,  0.4203],
    ])
}

# Lighting data augmentation taken from here - https://github.com/eladhoffer/convNet.pytorch/blob/master/preprocess.py
class Lighting(object):
    """Lighting noise(AlexNet - style PCA - based noise)"""

    def __init__(self, alphastd, eigval, eigvec):
        self.alphastd = alphastd
        self.eigval = eigval
        self.eigvec = eigvec

    def __call__(self, img):
        if self.alphastd == 0:
            return img

        alpha = img.new().resize_(3).normal_(0, self.alphastd)
        rgb = self.eigvec.type_as(img).clone()\
            .mul(alpha.view(1, 3).expand(3, 3))\
            .mul(self.eigval.view(1, 3).expand(3, 3))\
            .sum(1).squeeze()
        return img.add(rgb.view(3, 1, 1).expand_as(img))

which is finally used like this :

train_tfms = transforms.Compose([
        transforms.RandomResizedCrop(size),
        transforms.RandomHorizontalFlip(),
        transforms.ColorJitter(.4,.4,.4),
        transforms.ToTensor(),
        Lighting(0.1, __imagenet_pca['eigval'], __imagenet_pca['eigvec']),
        normalize,
    ])
    train_dataset = datasets.ImageFolder(traindir, train_tfms)
    train_sampler = (torch.utils.data.distributed.DistributedSampler(train_dataset)
                     if args.distributed else None)
    train_loader = torch.utils.data.DataLoader(
        train_dataset, batch_size=args.batch_size, shuffle=(train_sampler is None),
        num_workers=args.workers, pin_memory=True, sampler=train_sampler)

However, the problem is, whenever I run the script I get the:

  'AttributeError: 'Image' object has no attribute 'new''

Which complains about this line:

alpha = img.new().resize_(3).normal_(0, self.alphastd)

I am clueless as to why this is happening. I'm using Pytorch 0.4 by the way.


title: How does torch.distributed.barrier() work

body:
I've read all the documentations I could find about torch.distributed.barrier(), but still having trouble understanding how it's being used in this script and would really appreciate some help.

So the official doc of torch.distributed.barrier says it "Synchronizes all processes.This collective blocks processes until the whole group enters this function, if async_op is False, or if async work handle is called on wait()."

It's used in two places in the script:

First place

    if args.local_rank not in [-1, 0] and not evaluate:
        torch.distributed.barrier()  # Make sure only the first process in distributed training process the dataset, and the others will use the cache

        ... (preprocesses the data and save the preprocessed data)

    if args.local_rank == 0 and not evaluate:
        torch.distributed.barrier()

Second place

    if args.local_rank not in [-1, 0]:
        torch.distributed.barrier()  # Make sure only the first process in distributed training will download model & vocab

        ... (loads the model and the vocabulary)

    if args.local_rank == 0:
        torch.distributed.barrier()  # Make sure only the first process in distributed training will download model & vocab

I'm having trouble relating the comment in the code to the functionality of this function stated in the official doc. How does it make sure only the first process executes the code between the two calls of torch.distributed.barrier() and why it only checks whether the local rank is 0 before the second call?

Thanks in advance!


title: How to calculate output sizes after a convolution layer in a configuration file?

body:
I'm new to convolutional neural networks and wanted to know how to calculate or figure out the output sizes between layers of a model given a configuration file for pytorch similar to those following instructions in this link.

Most of the stuff I've already looked at hasn't been very clear and concise. How am I supposed to calculate the sizes through each layer?
Below is a snippet of a configuration file that would be parsed.

# (3, 640, 640)
[convolutional]
batch_normalize=1
filters=16
size=3
stride=1
pad=1
activation=leaky

[maxpool]
size=2
stride=2

# (16, 320, 320)


title: Pytorch tutorial LSTM

body:
I was trying to implement the exercise about Sequence Models and Long-Short Term Memory Networks with Pytorch. The idea is to add an LSTM part-of-speech tagger character-level features but I can't seem to work it out. They gave as a hint that there should be two LSTMs involved, one that will output a character level representation and another one that will be in charge of predicting the Part-of-speech tag. I just can't figure out how to loop over the words level (in a sentence) and over the character (in each word of a sentence) and implement it in the forward function. Does anyone know how to do it ? Or encounter a similar situation ?

Here is my code:

class LSTMTaggerAug(nn.Module):
def __init__(self, embedding_dim_words, embedding_dim_chars, hidden_dim_words, hidden_dim_chars, vocab_size, tagset_size, charset_size):
    super(LSTMTaggerAug, self).__init__()
    self.hidden_dim_words = hidden_dim_words
    self.hidden_dim_chars = hidden_dim_chars
    self.word_embeddings = nn.Embedding(vocab_size, embedding_dim_words)
    self.char_embeddings = nn.Embedding(charset_size, embedding_dim_chars)
    self.lstm_char = nn.LSTM(embedding_dim_chars, hidden_dim_chars)
    self.lstm_words = nn.LSTM(embedding_dim_words + hidden_dim_chars, hidden_dim_words)
    self.hidden2tag = nn.Linear(hidden_dim_words, tagset_size)
    self.hidden_char = self.init_hidden(c=False)
    self.hidden_words = self.init_hidden(c=True)

def init_hidden(self, c=True):
    if c:
        return (autograd.Variable(torch.zeros(1, 1, self.hidden_dim_words)),
                autograd.Variable(torch.zeros(1, 1, self.hidden_dim_words)))
    else:
        return (autograd.Variable(torch.zeros(1, 1, self.hidden_dim_chars)),
                autograd.Variable(torch.zeros(1, 1, self.hidden_dim_chars)))

def forward(self, sentence, words):
    # embeds = self.word_embeddings(sentence)
    for ix, word in enumerate(sentence):
        chars = words[ix]
        char_embeds = self.char_embeddings(chars)
        lstm_char_out, self.hidden_char = self.lstm_char(
            char_embeds.view(len(chars), 1, -1), self.hidden_char)
        char_rep = lstm_char_out[-1]
        embeds = self.word_embeddings(word)
        embeds_cat = torch.cat((embeds, char_rep), dim=1)
        lstm_out, self.hidden_words = self.lstm_words(embeds_cat, self.hidden_words)
        tag_space = self.hidden2tag(lstm_out.view(1, -1))
        tag_score = F.log_softmax(tag_space, dim=1)
        if ix==0:
            tag_scores = tag_score
        else:
            tag_scores = torch.cat((tag_scores, tag_score), 0)

    return tag_scores


title: 1D CNN on Pytorch: mat1 and mat2 shapes cannot be multiplied (10x3 and 10x2)

body:
I have a time series with sample of 500 size and 2 types of labels and want to construct a 1D CNN with pytorch on them:
class Simple1DCNN(torch.nn.Module):
    def __init__(self):
        super(Simple1DCNN, self).__init__()
        self.layer1 = torch.nn.Conv1d(in_channels=50,
                                      out_channels=20,
                                      kernel_size=5,
                                      stride=2)
        self.act1 = torch.nn.ReLU()
        self.layer2 = torch.nn.Conv1d(in_channels=20,
                                      out_channels=10,
                                      kernel_size=1)

        self.fc1 = nn.Linear(10* 1 * 1, 2)
    def forward(self, x):
        x = x.view(1, 50,-1)
        x = self.layer1(x)
        x = self.act1(x)
        x = self.layer2(x)
        x = self.fc1(x)

        return x

model = Simple1DCNN()
model(torch.tensor(np.random.uniform(-10, 10, 500)).float())

But got this error message:
Traceback (most recent call last):
  File "so_pytorch.py", line 28, in <module>
    model(torch.tensor(np.random.uniform(-10, 10, 500)).float())
  File "/Users/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "so_pytorch.py", line 23, in forward
    x = self.fc1(x)
  File "/Users/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/Users/lib/python3.8/site-packages/torch/nn/modules/linear.py", line 93, in forward
    return F.linear(input, self.weight, self.bias)
  File "/Users/lib/python3.8/site-packages/torch/nn/functional.py", line 1692, in linear
    output = input.matmul(weight.t())
RuntimeError: mat1 and mat2 shapes cannot be multiplied (10x3 and 10x2)

what am I doing wrong?


title: Why there are different output between model.forward(input) and model(input)

body:
I'm using pytorch to build a simple model like VGG16,and I have overloaded the function forward in my model.

I found everyone tends to use model(input) to get the output rather than model.forward(input), and I am interested in the difference between them. I try to input the same data, but the result is different. I'm confused.

I have output the layer_weight before I input data, the weight not be changed, and I know when we using model(input) it using __call__ function, and this function will call model.forward.

   vgg = VGG()
   vgg.double()
   for layer in vgg.modules():
      if isinstance(layer,torch.nn.Linear):
         print(layer.weight)
   print("   use model.forward(input)     ")
   result = vgg.forward(array)

   for layer in vgg.modules():
     if isinstance(layer,torch.nn.Linear):
       print(layer.weight)
   print("   use model(input)     ")
   result_2 = vgg(array)
   print(result)
   print(result_2)

output:

    Variable containing:1.00000e-02 *
    -0.2931  0.6716 -0.3497 -2.0217 -0.0764  1.2162  1.4983 -1.2881
    [torch.DoubleTensor of size 1x8]

    Variable containing:
    1.00000e-02 *
    0.5302  0.4494 -0.6866 -2.1657 -0.9504  1.0211  0.8308 -1.1665
    [torch.DoubleTensor of size 1x8]


title: Why error FileNotFoundError: [Errno 2] No such file or directory: './ML/model.pth' occurs?

body:
Error

l_1      |     from Ml import demo_model
ml_1      |   File "/app/Ml/demo.py", line 84, in
ml_1      |     model = torch.load('./ML/model.pth',  map_location='cpu')
ml_1      |   File "/usr/local/lib/python3.7/site-packages/torch/serialization.py", line 584, in load
ml_1      |     with _open_file_like(f, 'rb') as opened_file:
ml_1      |   File "/usr/local/lib/python3.7/site-packages/torch/serialization.py", line 234, in _open_file_like
ml_1      |     return _open_file(name_or_buffer, mode)
ml_1      |   File "/usr/local/lib/python3.7/site-packages/torch/serialization.py", line 215, in init
ml_1      |     super(_open_file, self).init(open(name, mode))
FileNotFoundError: [Errno 2] No such file or directory: './ML/model.pth'

occurs when starting the docker container.
Dockerfile:
FROM python:3.7

WORKDIR /app

COPY ./server/requirements.txt /app

COPY ./server/Ml/model.pth /app/Ml/

COPY ./server/Ml/demo.py /app/Ml/

RUN python -m pip install --no-cache-dir -r requirements.txt

CMD gunicorn -b=0.0.0.0:8443 -w=4 project.app:app --timeout 2000

The line of code where the error occurs:
best_model = torch.load('./ML/model.pth',  map_location='cpu')


title: Using tensorboard in pytorch, but get blank page?

body:
I am using tensorboard in pytorch 1.3.1, and I did exactly the same in the pytorch docs for tensorboard.
After running tensorboard --logdir=runs, I got this enter image description here.

$ tensorboard --logdir=runs
TensorFlow installation not found - running with reduced feature set.
Serving TensorBoard on localhost; to expose to the network, use a proxy or pass --bind_all
TensorBoard 2.1.0 at http://localhost:6006/ (Press CTRL+C to quit)

And after opening http://localhost:6006/, I got blank page like this

I also tried tensorboardX, and got the same result. Could you please tell how to solve the problem? thx.


title: pytorch when do I need to use `.to(device)` on a model or tensor?

body:
I am new to Pytorch, but it seems pretty nice. My only question was when to use tensor.to(device) or Module.nn.to(device).
I was reading the documentation on this topic, and it indicates that this method will move the tensor or model to the specified device. But I was not clear for what operations this is necessary, and what kind of errors I will get if I don't use .to() at the right time?
For example, if I just create a tensor, I imagine that the tensor is stored in CPU accessible memory until I move the tensor to the GPU. Once the tensor is on the GPU, then the GPU will execute any mathematical operations on that tensor.
However, do I have to worry about accidentally transferring the data tensor to the GPU while not transferring the model to the GPU? Will this just give me straight errors, or will it engage in a lot of expensive data transfer behind the scenes. This example is easy enough for me to test, but I was just wondering about other cases where it might not be so obvious.
Any guidance would be helpful.


title: Advanced indexing in 2d tensor in pytorch

body:
I have a 2d tensor X. and two lists of indexes that is first index and second call a and b. I want to do

X[a[i],b[i]] = 0 for i in range(len(a))

How can I do this. If i directly do X[a,b] the error is IndexError: The advanced indexing objects could not be broadcast


title: Pytorch device and .to(device) method

body:
I'm trying to learn RNN and Pytorch.

So I saw some codes for RNN where in the forward probagation method, they did a check like this:

def forward(self, inputs, hidden):
    if inputs.is_cuda:
        device = inputs.get_device()
    else:
        device = torch.device("cpu")
    embed_out = self.embeddings(inputs)
    logits = torch.zeros(self.seq_len, self.batch_size, self.vocab_size).to(device)

I think the point of the check is to see if we can run the code on faster GPU instead of CPU?
To understand the code a bit more, I did the following:

ex= torch.zeros(3,10,5)
ex1= torch.tensor(np.array([[0,0,0,1,0], [1,0,0,0,0],[0,1,0,0,0]]))

print(ex)
print("device is")
print(ex1.get_device())
print(ex.to(ex1.get_device()))

And the output was:

        ...
        [[0., 0., 0., 0., 0.],
         [0., 0., 0., 0., 0.],
         [0., 0., 0., 0., 0.],
         [0., 0., 0., 0., 0.],
         [0., 0., 0., 0., 0.],
         [0., 0., 0., 0., 0.],
         [0., 0., 0., 0., 0.],
         [0., 0., 0., 0., 0.],
         [0., 0., 0., 0., 0.],
         [0., 0., 0., 0., 0.]]])
device is
-1
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-2-b09342e2ba0f> in <module>()
     67 print("device is")
     68 print(ex1.get_device())
---> 69 print(ex.to(ex1.get_device()))

RuntimeError: Device index must not be negative

I don't understand the "device" in the code and I don't understand the .to(device) method. Can you help me understand it?


title: How to convert caffe prototxt to pytorch model?

body:
So far I was using the pytorch-caffe-darknet-convert repository. After overcoming numerous problems (concat and eltwise layers not convertible) I ended up with something that looks like a darknet config file:

python caffe2darknet.py my_prototxt.txt my_caffemodel.caffemodel new_net_file.cfg new_model.weights

Does someone know how to convert the output new_net_file.cfg into pytorch? Alternatively is there another way of converting caffe prototxt files into pytorch?
I would like to have the same behaviour as caffe-tensorflow
I'll post both my caffe prototxt and the output new_net_file.cfg below as reference.

my_prototxt:

input: "data"
input_shape {
  dim: 1
  dim: 240
  dim: 144
  dim: 240
}

layer {
  name: "conv1_1"
  type: "Convolution"
  bottom: "data"
  top: "conv1_1"
  convolution_param {
    num_output: 16
    pad: 3
    pad: 3
    pad: 3
    kernel_size: 7
    kernel_size: 7
    kernel_size: 7
    stride: 2
    weight_filler {
      type: "xavier"
    }
    bias_filler {
      type: "constant"
    }
    engine: CUDNN
    axis: 1
  }
}
layer {
  name: "relu1_1"
  type: "ReLU"
  bottom: "conv1_1"
  top: "conv1_1"
}
layer {
  name: "reduction2_1"
  type: "Convolution"
  bottom: "conv1_1"
  top: "reduction2_1"
  convolution_param {
    num_output: 32
    bias_term: false
    pad: 0
    kernel_size: 1
    stride: 1
    weight_filler {
      type: "xavier"
    }
    bias_filler {
      type: "constant"
    }
  }
}
layer {
  name: "conv2_1"
  type: "Convolution"
  bottom: "conv1_1"
  top: "conv2_1"
  convolution_param {
    num_output: 32
    pad: 1
    pad: 1
    pad: 1
    kernel_size: 3
    kernel_size: 3
    kernel_size: 3
    stride: 1
    weight_filler {
      type: "xavier"
    }
    bias_filler {
      type: "constant"
    }
    engine: CUDNN
    axis: 1
  }
}
layer {
  name: "relu2_1"
  type: "ReLU"
  bottom: "conv2_1"
  top: "conv2_1"
}
layer {
  name: "conv2_2"
  type: "Convolution"
  bottom: "conv2_1"
  top: "conv2_2"
  convolution_param {
    num_output: 32
    pad: 1
    pad: 1
    pad: 1
    kernel_size: 3
    kernel_size: 3
    kernel_size: 3
    stride: 1
    weight_filler {
      type: "xavier"
    }
    bias_filler {
      type: "constant"
    }
    axis: 1
  }
}
layer {
  name: "res2_2"
  type: "Eltwise"
  bottom: "reduction2_1"
  bottom: "conv2_2"
  top: "res2_2"
  eltwise_param { operation: SUM }
}
layer {
  name: "add2_2"
  type: "ReLU"
  bottom: "res2_2"
  top: "res2_2"
}
layer {
  name: "pool2"
  type: "Pooling"
  bottom: "res2_2"
  top: "pool2"
  pooling_param {
    pool: MAX
    kernel_size: 2
    stride: 2
    engine: CUDNN
  }
}
[...] # I cropped it here, since file is too lengthy

the (darknet) config file:

[net]
batch=1
channels=240
height=144
width=240

[convolutional]
filters=16
size=['7', '7', '7']
stride=2
pad=1
activation=relu

[convolutional]
filters=32
size=1
stride=1
pad=1
activation=linear

[route]
layers=-2

[convolutional]
filters=32
size=['3', '3', '3']
stride=1
pad=1
activation=relu

[convolutional]
filters=32
size=['3', '3', '3']
stride=1
pad=1
activation=linear

[shortcut]
from=-4
activation=relu

[maxpool]
size=2
stride=2

[...] # I cropped it here, since file is too lengthy


title: How to enable CUDA for using pytorch?

body:
I have a CUDA based GPU with 384 CUDA cores. But still I am not getting "True" if I command "torch.cuda.is_available()". I would like to know how to connect pytorch and CUDA.

I tried installing CUDA Toolkit.

torch.cuda.is_available()

Expected
torch.cuda.is_available()
True

Actual
torch.cuda.is_available()
False


title: Reset parameters of a neural network in pytorch

body:
I have a neural network with the following structure:
class myNetwork(nn.Module):
    def __init__(self):
        super(myNetwork, self).__init__()
        self.bigru = nn.GRU(input_size=2, hidden_size=100, batch_first=True, bidirectional=True)
        self.fc1 = nn.Linear(200, 32)
        torch.nn.init.xavier_uniform_(self.fc1.weight)
        self.fc2 = nn.Linear(32, 2)
        torch.nn.init.xavier_uniform_(self.fc2.weight)

I need to reinstate the model to an unlearned state by resetting the parameters of the neural network. I can do so for nn.Linear layers by using the method below:
def reset_weights(self):
    torch.nn.init.xavier_uniform_(self.fc1.weight)
    torch.nn.init.xavier_uniform_(self.fc2.weight)

But, to reset the weight of the nn.GRU layer, I could not find any such snippet.
My question is how does one reset the nn.GRU layer? Any other way of resetting the network is also fine. Any help is appreciated.