title: Pytorch: multi-target error with CrossEntropyLoss

body:
So I was training a Conv. Neural Network. Following are the essential details:

original label dim  = torch.Size([64, 1])
output from the net dim = torch.Size([64, 2])
loss type = nn.CrossEntropyLoss()
error = RuntimeError: multi-target not supported at /pytorch/aten/src/THCUNN/generic/ClassNLLCriterion.cu:15

WHERE AM I WRONG..?

training:

EPOCHS        = 5
LEARNING_RATE = 0.0001
BATCH_SIZE    = 64

net = Net().to(device)
optimizer = optim.Adam(net.parameters(), lr=LEARNING_RATE)

loss_log = []
loss_log = train(net, trainSet, loss_log, EPOCHS, LEARNING_RATE, BATCH_SIZE)

train function:

def train(net, train_set, loss_log=[], EPOCHS=5, LEARNING_RATE=0.001, BATCH_SIZE=32):
  print('Initiating Training..')
  loss_func = nn.CrossEntropyLoss()

  # Iteration Begins
  for epoch in tqdm(range(EPOCHS)):
    # Iterate over every sample in the batch
    for data in tqdm(trainSet, desc=f'Iteration > {epoch+1}/{EPOCHS} : ', leave=False):
        x, y = data
        net.zero_grad()

        #Compute the output
        output, sm = net(x)

        # Compute Train Loss
        loss = loss_func(output, y.to(device))

        # Backpropagate
        loss.backward()

        # Update Parameters
        optimizer.step()

        # LEARNING_RATE -= LEARNING_RATE*0.0005

    loss_log.append(loss)
    lr_log.append(LEARNING_RATE)

  return loss_log, lr_log

FULL ERROR:

---------------------------------------------------------------------------

RuntimeError                              Traceback (most recent call last)

<ipython-input-20-8deb9a27d3b4> in <module>()
     13
     14 total_epochs += EPOCHS
---> 15 loss_log = train(net, trainSet, loss_log, EPOCHS, LEARNING_RATE, BATCH_SIZE)
     16
     17 plt.plot(loss_log)

4 frames

<ipython-input-9-59e1d2cf0c84> in train(net, train_set, loss_log, EPOCHS, LEARNING_RATE, BATCH_SIZE)
     21         # Compute Train Loss
     22         # print(output, y.to(device))
---> 23         loss = loss_func(output, y.to(device))
     24
     25         # Backpropagate

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--> 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/loss.py in forward(self, input, target)
    914     def forward(self, input, target):
    915         return F.cross_entropy(input, target, weight=self.weight,
--> 916                                ignore_index=self.ignore_index, reduction=self.reduction)
    917
    918

/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction)
   2019     if size_average is not None or reduce is not None:
   2020         reduction = _Reduction.legacy_get_string(size_average, reduce)
-> 2021     return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
   2022
   2023

/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)
   1836                          .format(input.size(0), target.size(0)))
   1837     if dim == 2:
-> 1838         ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
   1839     elif dim == 4:
   1840         ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)

RuntimeError: multi-target not supported at /pytorch/aten/src/THCUNN/generic/ClassNLLCriterion.cu:15


title: Pytorch and Polynomial Linear Regression issue

body:
I have modified the code hat I found on the Pytorch github to suit my data, but my loss results are huge and with each iteration they get bigger and later become nan.Code doesn't give me any errors, just nor loss results and no predictions.
I have another code that deals withe the simple Linear Regression and all works fine. I guess I'm missing something simple here, but I'm unable to see it. any help would be appreciated.
Code:
import sklearn.linear_model as lm
from sklearn.preprocessing import PolynomialFeatures
import torch
import torch.autograd
import torch.nn.functional as F
from torch.autograd import Variable

train_data = torch.Tensor([
   [40,  6,  4],
   [44, 10,  4],
   [46, 12,  5],
   [48, 14,  7],
   [52, 16,  9],
   [58, 18, 12],
   [60, 22, 14],
   [68, 24, 20],
   [74, 26, 21],
   [80, 32, 24]])
test_data = torch.Tensor([
    [6, 4],
    [10, 5],
    [4, 8]])

x_train = train_data[:,1:3]
y_train = train_data[:,0]

POLY_DEGREE = 3
input_size = 2
output_size = 1

poly = PolynomialFeatures(input_size * POLY_DEGREE, include_bias=False)
x_train_poly = poly.fit_transform(x_train.numpy())

class Model(torch.nn.Module):

    def __init__(self):
        super(Model, self).__init__()
        self.fc = torch.nn.Linear(poly.n_output_features_, output_size)

    def forward(self, x):
        return self.fc(x)

model = Model()
criterion = torch.nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.001)

losses = []

for i in range(10):
    optimizer.zero_grad()
    outputs = model(Variable(torch.Tensor(x_train_poly)))
    print(outputs)
    loss = criterion(outputs, Variable(y_train))
    print(loss.data[0])
    losses.append(loss.data[0])
    loss.backward()
    optimizer.step()
    if loss.data[0] < 1e-4:
        break

print('n_iter', i)
print(loss.data[0])
plt.plot(losses)
plt.show()

output:

[393494300459008.0, inf, inf, inf, nan, nan, nan, nan, nan, nan]
n_iter
9 nan


title: Get probability of multi-token word in MASK position

body:
It is relatively easy to get a token's probability according to a language model, as the snippet below shows. You can get the output of a model, restrict yourself to the output of the masked token, and then find the probability of your requested token in the output vector. However, this only works with single-token words, e.g. words that are themselves in the tokenizer's vocabulary. When a word does not exist in the vocabulary, the tokenizer will chunk it up into pieces that it does know (see the bottom of the example). But since the input sentence consists of only one masked position, and the requested token has more tokens than that, how can we get its probability? Ultimately I am looking for a solution that works regardless of the number of subword units a word has.

In the code below I have added many comments explaining what is going on, as well as printing out the given output of print statements. You'll see that predicting tokens such as 'love' and 'hate' is straightforward because they are in the tokenizer's vocabulary. 'reprimand' is not, though, so it cannot be predicted in a single masked position - it consists of three subword units. So how can we predict 'reprimand' in the masked position?

from transformers import BertTokenizer, BertForMaskedLM
import torch

# init model and tokenizer
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForMaskedLM.from_pretrained('bert-base-uncased')
model.eval()
# init softmax to get probabilities later on
sm = torch.nn.Softmax(dim=0)
torch.set_grad_enabled(False)

# set sentence with MASK token, convert to token_ids
sentence = f"I {tokenizer.mask_token} you"
token_ids = tokenizer.encode(sentence, return_tensors='pt')
print(token_ids)
# tensor([[ 101, 1045,  103, 2017,  102]])
# get the position of the masked token
masked_position = (token_ids.squeeze() == tokenizer.mask_token_id).nonzero().item()

# forward
output = model(token_ids)
last_hidden_state = output[0].squeeze(0)
# only get output for masked token
# output is the size of the vocabulary
mask_hidden_state = last_hidden_state[masked_position]
# convert to probabilities (softmax)
# giving a probability for each item in the vocabulary
probs = sm(mask_hidden_state)

# get probability of token 'hate'
hate_id = tokenizer.convert_tokens_to_ids('hate')
print('hate probability', probs[hate_id].item())
# hate probability 0.008057191967964172

# get probability of token 'love'
love_id = tokenizer.convert_tokens_to_ids('love')
print('love probability', probs[love_id].item())
# love probability 0.6704086065292358

# get probability of token 'reprimand' (?)
reprimand_id = tokenizer.convert_tokens_to_ids('reprimand')
# reprimand is not in the vocabulary, so it needs to be split into subword units
print(tokenizer.convert_ids_to_tokens(reprimand_id))
# [UNK]

reprimand_id = tokenizer.encode('reprimand', add_special_tokens=False)
print(tokenizer.convert_ids_to_tokens(reprimand_id))
# ['rep', '##rim', '##and']
# but how do we now get the probability of a multi-token word in a single-token position?


title: Pytorch: Weight in cross entropy loss

body:
I was trying to understand how weight is in CrossEntropyLoss works by a practical example. So I first run as standard PyTorch code and then manually both. But the losses are not the same.

from torch import nn
import torch
softmax=nn.Softmax()
sc=torch.tensor([0.4,0.36])
loss = nn.CrossEntropyLoss(weight=sc)
input = torch.tensor([[3.0,4.0],[6.0,9.0]])
target = torch.tensor([1,0])
output = loss(input, target)
print(output)
>>1.7529

Now for manual Calculation, first softmax the input:

print(softmax(input))
>>
tensor([[0.2689, 0.7311],
        [0.0474, 0.9526]])

and then negetive log of the correct class probality and multiply with the respective weight:

((-math.log(0.7311)*0.36) - (math.log(0.0474)*0.4))/2
>>
0.6662

What I am missing here?


title: RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True

body:
I'm a student and a beginner in Python and PyTorch both. I have a very basic Neural Network for which I am encountering the mentioned RunTimeError. The code to reproduce the error is this:

import torch
from torch import nn
from torch import optim
import torch.nn.functional as F
import matplotlib.pyplot as plt

# Ensure Reproducibility
torch.manual_seed(0)

# Data Generation
x = torch.randn((100,1), requires_grad = True)
y = 1 + 2 * x + 0.3 * torch.randn(100,1)
# Shuffles the indices
idx = np.arange(100)
np.random.shuffle(idx)

# Uses first 80 random indices for train
train_idx = idx[:70]
# Uses the remaining indices for validation
val_idx = idx[70:]

# Generates train and validation sets
x_train, y_train = x[train_idx], y[train_idx]
x_val, y_val = x[val_idx], y[val_idx]

class OurFirstNeuralNetwork(nn.Module):
    def __init__(self):
        super(OurFirstNeuralNetwork, self).__init__()
        # Here we "define" our Neural Network Architecture
        self.fc1 = nn.Linear(1, 5)
        self.non_linearity_fc1 = nn.ReLU()
        self.fc2 = nn.Linear(5,1)
        #self.non_linearity_fc2 = nn.ReLU()

    def forward(self, x):
        # The forward pass
        # Here we define how activations "flow" between neurons. We've already discussed the "Sum" and "Transformation" steps of the forward pass.
        sum_fc1 = self.fc1(x)
        transformation_fc1 = self.non_linearity_fc1(sum_fc1)
        sum_fc2 = self.fc2(transformation_fc1)
        #transformation_fc2 = self.non_linearity_fc2(sum_fc2)
        # The transformation_fc2 is also the output of our model which symbolises the end of our forward pass.
        return sum_fc2

# Instantiate the model and train

model = OurFirstNeuralNetwork()
print(model)
print(model.state_dict())
n_epochs = 1000
loss_fn = nn.MSELoss(reduction='mean')
optimizer = optim.Adam(model.parameters())

for epoch in range(n_epochs):

    model.train()
    optimizer.zero_grad()
    prediction = model(x_train)
    loss = loss_fn(y_train, prediction)
    print(epoch, loss)
    loss.backward(retain_graph=True)
    optimizer.step()

print(model.state_dict())

Everything is basic and standard and this works fine.

However, when I take out the "retain_graph=True" argument, it throws the RunTimeError. From reading various forums, I understand that this is to do with the graph getting thrown away after the first iteration but I have seen many tutorials and blogs where loss.backward() is the way to go especially since it conserves memory. But I am not able to conceptually grasp why the same does not work for me.

Any help is appreciated and my apologies if the way in which I have asked my question is not in the expected format. I am open to feedback and will oblige to include more details or rephrase the question so that it is easier for everyone. Thank you in advance!


title: OverflowError: Python int too large to convert to C long torchtext.datasets.text_classification.DATASETS['AG_NEWS']()

body:
I have 64 bit windows 10 OS
I have installed python 3.6.8
I have installed torch and torchtext using pip.
torch version is 1.2.0

I am trying to load AG_NEWS dataset using below code:

import torch
import torchtext
from torchtext.datasets import text_classification
NGRAMS = 2
import os
if not os.path.isdir('./.data'):
    os.mkdir('./.data')
train_dataset, test_dataset = text_classification.DATASETS['AG_NEWS'](root='./.data', ngrams=NGRAMS, vocab=None)

On the last statement of above code, I am getting below error:

---------------------------------------------------------------------------
OverflowError                             Traceback (most recent call last)
<ipython-input-1-7e8544fdaaf6> in <module>
      6 if not os.path.isdir('./.data'):
      7     os.mkdir('./.data')
----> 8 train_dataset, test_dataset = text_classification.DATASETS['AG_NEWS'](root='./.data', ngrams=NGRAMS, vocab=None)
      9 # BATCH_SIZE = 16
     10 # device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

c:\users\pramodp\appdata\local\programs\python\python36\lib\site-packages\torchtext\datasets\text_classification.py in AG_NEWS(*args, **kwargs)
    168     """
    169
--> 170     return _setup_datasets(*(("AG_NEWS",) + args), **kwargs)
    171
    172

c:\users\pramodp\appdata\local\programs\python\python36\lib\site-packages\torchtext\datasets\text_classification.py in _setup_datasets(dataset_name, root, ngrams, vocab, include_unk)
    126     if vocab is None:
    127         logging.info('Building Vocab based on {}'.format(train_csv_path))
--> 128         vocab = build_vocab_from_iterator(_csv_iterator(train_csv_path, ngrams))
    129     else:
    130         if not isinstance(vocab, Vocab):

c:\users\pramodp\appdata\local\programs\python\python36\lib\site-packages\torchtext\vocab.py in build_vocab_from_iterator(iterator)
    555     counter = Counter()
    556     with tqdm(unit_scale=0, unit='lines') as t:
--> 557         for tokens in iterator:
    558             counter.update(tokens)
    559             t.update(1)

c:\users\pramodp\appdata\local\programs\python\python36\lib\site-packages\torchtext\datasets\text_classification.py in _csv_iterator(data_path, ngrams, yield_cls)
     33     with io.open(data_path, encoding="utf8") as f:
     34         reader = unicode_csv_reader(f)
---> 35         for row in reader:
     36             tokens = ' '.join(row[1:])
     37             tokens = tokenizer(tokens)

c:\users\pramodp\appdata\local\programs\python\python36\lib\site-packages\torchtext\utils.py in unicode_csv_reader(unicode_csv_data, **kwargs)
    128             maxInt = int(maxInt / 10)
    129
--> 130     csv.field_size_limit(sys.maxsize)
    131
    132     if six.PY2:

OverflowError: Python int too large to convert to C long

I think the issue is with either windows os or torchtext because I am getting same error for below code as well.

pos = data.TabularDataset( path='data/pos/pos_wsj_train.tsv', format='tsv', fields=[('text', data.Field()),
        ('labels', data.Field())])

Can somebody please help? and mainly I don't have any large numerical values in the file.


title: Number of instances per class in pytorch dataset

body:
I'm trying to make a simple image classifier using PyTorch.
This is how I load the data into a dataset and dataLoader:

batch_size = 64
validation_split = 0.2
data_dir = PROJECT_PATH+"/categorized_products"
transform = transforms.Compose([transforms.Grayscale(), CustomToTensor()])

dataset = ImageFolder(data_dir, transform=transform)

indices = list(range(len(dataset)))

train_indices = indices[:int(len(indices)*0.8)]
test_indices = indices[int(len(indices)*0.8):]

train_sampler = SubsetRandomSampler(train_indices)
test_sampler = SubsetRandomSampler(test_indices)

train_loader = torch.utils.data.DataLoader(dataset, batch_size=batch_size, sampler=train_sampler, num_workers=16)
test_loader = torch.utils.data.DataLoader(dataset, batch_size=batch_size, sampler=test_sampler, num_workers=16)

I want to print out the number of images in each class in training and test data separately, something like this:

In train data:

shoes: 20
shirts: 14

In test data:

shoes: 4
shirts: 3

I tried this:

from collections import Counter
print(dict(Counter(sample_tup[1] for sample_tup in dataset.imgs)))

but I got this error:

AttributeError: 'MyDataset' object has no attribute 'img'


title: Tensorboard Graph Visualization with PyTorch

body:
I'm trying to visualize the graph of a PyTorch model using: https://github.com/lanpa/tensorboardX

However when I follow instructions on the GitHub README page for the demo-example:

  Run the demo script: python demo.py
  Use TensorBoard with tensorboard --logdir runs (needs to install    TensorFlow)

Most of the options do work - except for the Graph tab. When going there it shows this error message:

  No graph definition files were found.

  To store a graph, create a tf.summary.FileWriter and pass the graph
  either via the constructor, or by calling its add_graph() method. You
  may want to check out the graph visualizer tutorial.

  If you’re new to using TensorBoard, and want to find out how to add
  data and set up your event files, check out the README and perhaps the
  TensorBoard tutorial.

  If you think TensorBoard is configured properly, please see the
  section of the README devoted to missing data problems and consider
  filing an issue on GitHub.

While the shell outputs these errors:

W0817 13:51:54.464755 Thread-1 application.py:272] path /data/plugin/debugger/debugger_grpc_host_port not found, sending 404
W0817 13:51:54.493842 Thread-1 application.py:272] path /data/plugin/debugger/debugger_grpc_host_port not found, sending 404
W0817 13:51:54.507973 Thread-1 application.py:272] path /data/plugin/debugger/debugger_grpc_host_port not found, sending 404
W0817 13:51:54.518329 Thread-1 application.py:272] path /data/plugin/debugger/debugger_grpc_host_port not found, sending 404
W0817 13:51:54.527345 Thread-1 application.py:272] path /data/plugin/debugger/comm not found, sending 404
W0817 13:51:54.536046 Thread-1 application.py:272] path /data/plugin/debugger/comm not found, sending 404
W0817 13:51:54.557759 Thread-1 application.py:272] path /data/plugin/debugger/comm not found, sending 404
W0817 13:51:54.570233 Thread-1 application.py:272] path /data/plugin/debugger/comm not found, sending 404

I installed the tensorboadX master-branch using:

pip install git+https://github.com/lanpa/tensorboardX

(I Tried also tensorboardX version 1.2 before without success)

I tried switching to different versions but without success,also those which were 'recommended' in combination with tensorboardX:

  Tested on anaconda2 / anaconda3, with PyTorch 0.4 / torchvision 0.2 /
  tensorboard 1.7.0

There are plenty similar issues out there with the same error message stating: No graph definition files were found., but all of which I found where related to Tensorflow itself, not PyTorch.

It seems to me this is just an issue of locating the correct files, but I haven't found any solution yet.

Any help is greatly appreciated!


title: Add a constant variable to a cuda.FloatTensor

body:
I have two question:

  1) I'd like to know how can I add/subtract a constante torch.FloatTensor of size 1 to all of the elemets of a torch.FloatTensor of size 30.

  2) How can I multiply each element of a torch.FloatTensor of size 30 by a random value (different or not for each).

My code:

import torch
dtype = torch.cuda.FloatTensor
def main():
     pop, xmax, xmin   = 30, 5, -5
     x                 = (xmax-xmin)*torch.rand(pop).type(dtype)+xmin
     y                 = torch.pow(x, 2)
     [miny, indexmin]  = y.min(0)
     gxbest            = x[indexmin]
     pxbest            = x
     pybest            = y
     v = torch.rand(pop)
     vnext = torch.rand()*v + torch.rand()*(pxbest - x) + torch.rand()*(gxbest - x)

main()

What is the best way to do it? I think I should so how convert the gxbest into a torch.FloatTensor of size 30 but how can I do that?
I've try to create a vector:

  Variable(torch.from_numpy(np.ones(pop)))*gxbest

But it did not work. The multiplication is not working also.

RuntimeError: inconsistent tensor size

Thank you all for your help!


title: PyTorch: purpose of addmm function

body:
What is the purpose of the following PyTorch function (doc):

torch.addmm(beta=1, mat, alpha=1, mat1, mat2, out=None)

More specifically, is there any reason to prefer this function instead of just using

beta * mat + alpha * (mat1 @ mat2)


title: Understanding TypeError: '<' not supported between instances of 'Example' and 'Example'

body:
I am working on a project of text simplification using a multi-head attention transformer model. For the same, I am using torchtext for tokenisation and numericalization. The dataset contains two aligned files for training and two aligned files for testing. In the training files, one file contains the complex sentences while the other contains the corresponding simplified sentences.

I read the files as such:

training_sentences = open(path + "train.en" , encoding = "utf-8").read().split("\n")
target_sentences = open(path + "train.sen" , encoding = "utf-8").read().split("\n")

Next, I tokenised them as such:

complicated = spacy.load('en')
simple = spacy.load('en')

def tokenize_complicated(sentence):
   return [tok.text for tok in complicated.tokenizer(sentence)]

def tokenize_simple(sentence):
    return [tok.text for tok in simple.tokenizer(sentence)]

C_TEXT = Field(tokenize=tokenize_complicated, fix_length = 100)
S_TEXT = Field(tokenize=tokenize_simple, fix_length = 100, init_token = "<sos>", eos_token = "<eos>")

I then converted into TabularDataset object of torchtext.

import pandas as pd
raw_data = {'Complicated' : [line for line in training_sentences],
        'Simple': [line for line in target_sentences]}

df = pd.DataFrame(raw_data, columns=["Complicated", "Simple"])

df.to_csv("df.csv", index=False)
data_fields = [('Complicated', C_TEXT), ('Simple', S_TEXT)]

train = torchtext.data.TabularDataset.splits(path='./', train = "df.csv", format='csv', fields=data_fields, skip_header = True)

And then created vocabulary

C_TEXT.build_vocab(train)
S_TEXT.build_vocab(train)

However, on doing so I got this error:

  TypeError: '<' not supported between instances of 'Example' and
  'Example'

On searching, I came across this solution here and the error disappeared. However, I am not understanding whether this makes the model take only one instance or it takes all of the dataset?
I would like to know the significance of the index [0] so that I can manipulate it effectively for my model.


title: Pytorch CUDA error: no kernel image is available for execution on the device on RTX 3090 with cuda 11.1

body:
If I run the following:
import torch
import sys
print('A', sys.version)
print('B', torch.__version__)
print('C', torch.cuda.is_available())
print('D', torch.backends.cudnn.enabled)
device = torch.device('cuda')
print('E', torch.cuda.get_device_properties(device))
print('F', torch.tensor([1.0, 2.0]).cuda())

I get this:
A 3.7.5 (default, Nov  7 2019, 10:50:52)
[GCC 8.3.0]
B 1.8.0.dev20210115+cu110
C True
D True
E _CudaDeviceProperties(name='GeForce RTX 3090', major=8, minor=6, total_memory=24267MB, multi_processor_count=82)
F
<stacktrace>
CUDA error: no kernel image is available for execution on the device

More info about my system:

Nvidia version: NVIDIA-SMI 455.38 Driver Version: 455.38 CUDA Version: 11.1
python 3.7, Ubuntu 18.04


title: pyTorch can backward twice without setting retain_graph=True

body:
As indicated in pyTorch tutorial,

  if you even want to do the backward on some part of the graph twice,
  you need to pass in retain_graph = True during the first pass.

However, I found the following codes snippet actually worked without doing so. I'm using pyTorch-0.4

x = torch.ones(2, 2, requires_grad=True)
y = x + 2
y.backward(torch.ones(2, 2)) # Note I do not set retain_graph=True
y.backward(torch.ones(2, 2)) # But it can still work!
print x.grad

output:

tensor([[ 2.,  2.],
        [ 2.,  2.]])

Could anyone explain? Thanks in advance!


title: Pytorch: How to save the previous parameters and gradient temporally for use them in the next step/loop ？

body:
I am wondering how to save the parameters/weights and their dependent gradient (after using backward() from the object optimizer (e.g. optimizer = optim.SGD....) to them at each step/loop), in order to re-load them back into the optimizer at next step. Since I need to save and re-load the parameters & their gradients every step/loop time, I do not want to save them out to a separate pt or pth file and load time at every loop.

I tried to implement the code like:

    for i, data in enumerate(trainloader, 0):
        inputs, labels = data
        inputs, labels = Variable(inputs), Variable(labels)
        def closure():
            optimizer.zero_grad()
            outputs = net(inputs)
            loss = criterion(outputs, labels)
            loss.backward()
            return loss.data[0]
        # I saved parameters & their gradients here! I also modified the return stuff in sgd.py as well, which is not shown here.
        running_loss, param_groups_temp_prev, param_groups_grad_temp_prev = optimizer.step(closure)
        # Here I tried to re-assign the above saved two lists back to optimizer again for the second call of optimizer.step(closure)
        optimizer.param_groups[0]['params'] = copy.deepcopy([pgt_prev for pgt_prev in param_groups_temp_prev])
        for ii in range(len(param_groups_grad_temp_prev)):
            pggt_prev = copy.deepcopy(param_groups_grad_temp_prev[ii])
            optimizer.param_groups[0]['params'][ii].grad = copy.deepcopy(Variable(pggt_prev))
        running_loss_temp, param_groups_temp1, param_groups_grad_temp1 = optimizer.step(closure)

I save them as param_groups_temp_prev and param_groups_grad_temp_prev, in order to re-load them in the second time to call optimizer.step(closure) in the same loop (first time loop), everything went well.

However, when it goes to the loop run to the 2nd time, after optimizer.zero_grad() is called, still good, gradients of all parameters were cleared to zero. But, after loss.backward() was called, the gradients remained zero still, which seemed that the backward() was out of work!

Could anyone help me out of this issue? Appreciate!


title: TypeError: 'module' object is not callable in pytorch

body:
using pytorch and python on implementing a self driving car i am getting this error.

File "C:\Users\Ganesh Bhat\Desktop\AI\Self_Driving_Car\ai.py", line 86, in update
    self.memory.push((self.last_state,new_state,torch.LongTensor([int(self.last_action)])),torch.tensor([self.last_reward]))

TypeError: 'module' object is not callable

here is my code of update function

def update(self,reward, new_signal):
    new_state = torch.Tensor(new_signal).float().unsqueeze(0)
    self.memory.push((self.last_state,new_state,torch.LongTensor([int(self.last_action)]),torch.tensor([self.last_reward])))
    action = self.select_action(new_state)
    if len(self.memory.memory) > 100:
        batch_state, batch_next_state, batch_reward, batch_action = self.memory.sample(100)
        self.learn(batch_state, batch_next_state, batch_reward, batch_action)
    self.last_action = action
    self.last_state = new_state
    self.last_reward = reward
    self.reward_window.append(reward)
    if len(self.reward_window)>1000:
        del self.reward_window[0]
    return action


title: Training Detectron2 on part of COCO dataset

body:
I'm trying to train model with Detectron2 and COCO dataset for vehicle and person detection and I'm having problems with model loading.
I've used posts here on SO and https://github.com/immersive-limit/coco-manager (filter.py file) code to filter COCO dataset to only include annotations and images from classes "person", "car", "bike", "truck" and "bicycle". Now my directory structure is:
main
  - annotations:
    - instances_train2017_filtered.json
    - instances_val2017_filtered.json
  - images:
    - train2017_filtered (lots of images inside)
    - val2017_filtered (lots of images inside)

Basically, the only thing that I've done here was to remove documents and images not corresponding to those classes, and changed their IDs (so they are from 1 to 5).
Then I've used code from Detectron2 tutorial:
import random

import cv2
from detectron2.data import MetadataCatalog, DatasetCatalog
from detectron2.data.datasets import register_coco_instances
from detectron2.engine import DefaultTrainer, DefaultPredictor
from detectron2.config import get_cfg
import os

from detectron2.model_zoo import model_zoo
from detectron2.utils.visualizer import Visualizer

register_coco_instances("train",
                        {},
                        "/home/jakub/Projects/coco/annotations/instances_train2017_filtered.json",
                        "/home/jakub/Projects/coco/images/train2017_filtered/")

register_coco_instances("val",
                        {},
                        "/home/jakub/Projects/coco/annotations/instances_val2017_filtered.json",
                        "/home/jakub/Projects/coco/images/val2017_filtered/")

metadata = MetadataCatalog.get("train")
dataset_dicts = DatasetCatalog.get("train")

cfg = get_cfg()
cfg.merge_from_file(model_zoo.get_config_file("COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml"))
cfg.DATASETS.TRAIN = ("train",)
cfg.DATASETS.TEST = ()
cfg.DATALOADER.NUM_WORKERS = 2
cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url("COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml")
cfg.SOLVER.IMS_PER_BATCH = 2
cfg.SOLVER.BASE_LR = 0.00025
cfg.SOLVER.MAX_ITER = 300
cfg.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = 512
cfg.MODEL.ROI_HEADS.NUM_CLASSES = 5

os.makedirs(cfg.OUTPUT_DIR, exist_ok=True)
trainer = DefaultTrainer(cfg)
trainer.resume_or_load(resume=False)
trainer.train()

cfg.MODEL.WEIGHTS = os.path.join(cfg.OUTPUT_DIR, "model_final.pth")
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.5
cfg.DATASETS.TEST = ("val", )
predictor = DefaultPredictor(cfg)

img = cv2.imread("demo/input.jpg")
outputs = predictor(img)

for d in random.sample(dataset_dicts, 1):
    im = cv2.imread(d["file_name"])
    outputs = predictor(im)
    v = Visualizer(im[:, :, ::-1],
                   metadata=metadata,
                   scale=0.8)
    out = v.draw_instance_predictions(outputs["instances"].to("cpu"))
    cv2.imwrite('demo/output_retrained.jpg', out.get_image()[:, :, ::-1])

During training, I get the following errors:
Unable to load 'roi_heads.box_predictor.cls_score.weight' to the model due to incompatible shapes: (81, 1024) in the checkpoint but (6, 1024) in the model!
Unable to load 'roi_heads.box_predictor.cls_score.bias' to the model due to incompatible shapes: (81,) in the checkpoint but (6,) in the model!
Unable to load 'roi_heads.box_predictor.bbox_pred.weight' to the model due to incompatible shapes: (320, 1024) in the checkpoint but (20, 1024) in the model!
Unable to load 'roi_heads.box_predictor.bbox_pred.bias' to the model due to incompatible shapes: (320,) in the checkpoint but (20,) in the model!
Unable to load 'roi_heads.mask_head.predictor.weight' to the model due to incompatible shapes: (80, 256, 1, 1) in the checkpoint but (5, 256, 1, 1) in the model!
Unable to load 'roi_heads.mask_head.predictor.bias' to the model due to incompatible shapes: (80,) in the checkpoint but (5,) in the model!

The model cannot predict anything useful after training, despite reducing total_loss during training. I get that I should get warnings because of size mismatch (I've reduced number of classes), which is normal from what I've seen on the internet, but I don't get "Skipped" after each error line. I think that model is actually not loading anything here, and I wonder why and how can I fix this.
EDIT
For comparison, a similar behaviour in almost identical situation was reported as an Issue, but it had "Skipped" at the end of each error line, making them effectively warnings, not errors:
https://github.com/facebookresearch/detectron2/issues/196


title: pytorch cnn stride error

body:
I am now using pytorch 0.4.0 in windows to build a CNN and here is my code:

class net(nn.Module):
    def __init__(self):
        super(net, self).__init__()
        self.conv1 = nn.Conv2d(in_channels=1, out_channels=16, kernel_size=(1,3),stride=1 )

        self.conv2 = nn.Conv2d(in_channels=16, out_channels=32, kernel_size=(1,3), stride=1)

        self.dense1 = nn.Linear(32 * 28 * 24, 60)
        self.out = nn.Linear(60,3)

    def forward(self, input):
        x = F.relu(self.conv1(input))
        x = F.relu(self.conv2(x))
        x = x.view(x.size(0), -1) # flatten（batch，32*7*7）
        x = self.dense1(x)
        output = self.out(x)
        return output

but I get the error that

File "D:\Anaconda\lib\site-packages\torch\nn\modules\conv.py", line 301, in forward
    self.padding, self.dilation, self.groups)

RuntimeError: expected stride to be a single integer value or a list of 1 values to match the convolution dimensions, but got stride=[1, 1]

I think it shows that I made some mistakes in the code above, but I don't know how to fix it, can any one help me out? Thanks in advance!


title: pytorch DataLoader extremely slow first epoch

body:
When I create a PyTorch DataLoader and start iterating -- I get an extremely slow first epoch (x10--x30 slower then all next epochs). Moreover, this problem occurs only with the train dataset from the Google landmark recognition 2020 from Kaggle. I can't reproduce this on synthetic images, also, I tried to create a folder with 500k images from GLR2020, and everything worked well. Found few similar problems in the PyTorch forum without any solutions.
import argparse
import pandas as pd
import numpy as np
import os, sys
import multiprocessing, ray
import time
import cv2
import logging
import albumentations as albu
from torch.utils.data import Dataset, DataLoader

samples = 50000 # count of samples to speed up test
bs = 64 # batch size
dir = '/hdd0/datasets/ggl_landmark_recognition_2020/train' # directory with train data
all_files = pd.read_csv('/hdd0/datasets/ggl_landmark_recognition_2020/train.csv')
files = np.random.choice(all_files.id.values, 50000)
files = [os.path.join(_[0], _[1], _[2], _+'.jpg') for _ in files]

# augmentations
aug =  albu.Compose([albu.Resize(400, 400),
        albu.Rotate(limit=15),
        albu.ChannelDropout(p=0.1),
        albu.Normalize(),])

class ImgDataset:
    def __init__(self, path, files, augmentation = None):
        self.path = path
        self.files = {k:v for k, v in enumerate(files)}
        self.augmentation = augmentation

    def __len__(self):
        return len(self.files)

    def __getitem__(self, idx):
        img_name = self.files[idx]
        img = np.array(cv2.imread(os.path.join(self.path, img_name)))
        if self.augmentation is not None:
            return self.augmentation(image=img)['image']

dtset = ImgDataset(dir,files, aug)
torchloader = DataLoader(dataset= dtset, batch_size=64, num_worker=16, shuffle=True)
for _ in range(3):
   t1 = time.time()
   for idx, val in enumerate(torchloader):
       pass
   t2 = time.time()
   print(str(t2-t1) +' sec')

Here are some examples of execution speed with different num_workers in DataLoader
#num_workers=0
273.1584792137146 sec
83.15653467178345 sec
83.67923021316528 sec

# num_workers = 8
165.62366938591003 sec
10.405716896057129 sec
10.495309114456177 sec

# num_workers = 16
156.60744667053223 sec
8.051618099212646 sec
7.922858238220215 sec

Looks like the problem is not with DataLoader, but with dataset. When I delete and reinitialise DataLoader object after first "long" iteration, everything still works fine. When I reinitialise dataset -- long first iteration appears again.
Moreover, I tracked my cpu utilisation via htop during this epochs with num_workers setted to 32, and during the first epoch, utilisation is really low; only 1-2 of 32 cores are working, during other epochs ~all cores are working.


title: Normalising images before learning in pytorch

body:
I want to apply a transform to standardise the images in my dataset before learning in pytorch. I hear this improves learning dramatically. I think Pytorch by default divides all image pixel values by 255 before puttint them in tensors, does this pose a problem for standardization?. The online guide recommends we proceed in the following way.

transform = transforms.Compose(
    [transforms.ToTensor(),
     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

However, how the 0.5 here is just an example I found, it is not the mean or variance for the channels of my data.

So my question is how is the mean and standard deviation derived? Do we need to flatten and append all the green pixel values of the input pictures then calculate the mean and standard deviation? Then repeat for all the other colours. is that how it's done?

I heard there was another approach that tries to calculate an "average picture" to standardise with. What is the difference in result?


title: In PyTorch, what is the input into nll_loss?

body:
I am looking at the tutorial here: https://pytorch.org/tutorials/beginner/fgsm_tutorial.html

import torch.nn.functional as F
loss = F.nll_loss(output, target)

In the above two lines of code, what exactly is "target"? They load the data set for target but never discuss what it is exactly. The documentation is also hard to understand.


title: Understanding Pytorch Grid Sample

body:
I have a input tensor of size [1,32,296,400]

and I have a pixel set of [1, 56000, 400, 2]

After applying grid_sample with mode=‘bilinear’ I have [1, 32, 56000, 400]

Can I know what exactly happened here? I know that grid_sample is suppose to effective transform pixels to a new location in a differentiable manner, but these dimensions don't make it clear what is happening.


I am following this doc for hstack.
a = torch.tensor([1, 2, 3])
b = torch.tensor([4, 5, 6])
torch.hstack((a,b))

But I keep getting the error:
AttributeError: module 'torch' has no attribute 'hstack'

Here is the torch version that results in this error:
torch.__version__
'1.6.0+cpu'

What am I doing wrong?


title: When does dataloader shuffle happen for Pytorch?

body:
I have beening using shuffle option for pytorch dataloader for many times. But I was wondering when this shuffle happens and whether it is performed dynamically during iteration. Take the following code as an example:

namesDataset = NamesDataset()
namesTrainLoader = DataLoader(namesDataset, batch_size=16, shuffle=True)
for batch_data in namesTrainLoader:
    print(batch_data)

When we define "namesTrainLoader", does that mean the shuffling is finished and the following iteration will be based on a fixed order of data? Will there be any randomness in the for loop after namesTrainLoader was defined?

I was trying to replace half of "batch_data" with some special value:

for batch_data in namesTrainLoader:
    batch_data[：8] = special_val
    pre = model(batch_data)

Let us say there will be infinite number of epoches, will "model" eventually see all the data in "namesTrainLoader"? Or half of the data of "namesTrainLoader" is actually lost to "model"?


title: Pytorch : GPU Memory Leak

body:
I speculated that I was facing a GPU memory leak in the training of Conv nets using PyTorch framework. Below image

To resolve it, I added -

os.environ['CUDA_LAUNCH_BLOCKING'] = "1"

which resolved the memory problem, as shown below -

but as I was using torch.nn.DataParallel, so I expect my code to utilise all the GPUs, but now it is utilising only the GPU:1.

Before using os.environ['CUDA_LAUNCH_BLOCKING'] = "1", the GPU utilisation was below (which is equally bad)-

On digging further, I come to know that, when we use torch.nn.DataParallel, we are supposed to not use CUDA_LAUNCH_BLOCKING', because it puts the network in some deadlock mechanism.
So, now I have come back again in GPU memory issue, because I think my code is not utilising that much memory which it is showing without setting CUDA_LAUNCH_BLOCKING=1.
My code to use torch.nn.DataParallel-

device =  torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
if torch.cuda.device_count() > 1:
  print("Let's use", torch.cuda.device_count(), "GPUs!")
  # dim = 0 [30, xxx] -> [10, ...], [10, ...], [10, ...] on 3 GPUs
  model_transfer = nn.DataParallel(model_transfer.cuda(),device_ids=range(torch.cuda.device_count()))

model_transfer.to(device)

How to resolve the GPU memory issue?
Edit:
Minimal code -

image_dataset = datasets.ImageFolder(train_dir_path,transform = transform)
train_loader = torch.utils.data.DataLoader(image_dataset['train'], batch_size=batch_size,shuffle = True)
model_transfer = models.resnet18(pretrained=True)

device =  torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
if torch.cuda.device_count() > 1:
  print("Let's use", torch.cuda.device_count(), "GPUs!")
  # dim = 0 [30, xxx] -> [10, ...], [10, ...], [10, ...] on 3 GPUs
  model_transfer = nn.DataParallel(model_transfer.cuda(),device_ids=range(torch.cuda.device_count()))
model_transfer.to(device)

## Training function
for epoch in range(1, n_epochs+1):
    for batch_idx, (data, target) in enumerate(train_loader):
                    if use_cuda:
            data, target = data.to('cuda',non_blocking = True), target.to('cuda',non_blocking = True)
        optimizer.zero_grad()
        output = model(data)
        loss = criterion(output,target)
        loss.backward()
        optimizer.step()
        train_loss += ((1 / (batch_idx + 1)) * (loss.item() - train_loss))

   ## Validation loop same as training loop so not mentioning here

    print('Epoch: {} \tTraining Loss: {:.6f} \tValidation Loss: {:.6f}'.format(
        epoch,
        train_loss,
        valid_loss
        ))
        if valid_loss <= valid_loss_min:
           valid_loss_min,valid_loss))
           torch.save(model.state_dict(), 'case_3_model.pt')
           valid_loss_min = valid_loss


title: Pytorch: Understand how nn.Module class internally work

body:
Generally, a nn.Module can be inherited by a subclass as below.

def init_weights(m):
    if type(m) == nn.Linear:
        torch.nn.init.xavier_uniform(m.weight)  #

class LinearRegression(nn.Module):
    def __init__(self):
        super(LinearRegression, self).__init__()
        self.fc1 = nn.Linear(20, 1)
        self.apply(init_weights)

    def forward(self, x):
        x = self.fc1(x)
        return x

My 1st question is, why I can simply run the code below even my __init__ doesn't have any positinoal arguments for training_signals and it looks like that training_signals is passed to forward() method. How does it work?

model = LinearRegression()
training_signals = torch.rand(1000,20)
model(training_signals)

The second question is that how does self.apply(init_weights) internally work? Is it executed before calling forward method?


title: PyTorch DataLoader

body:
I'm trying to use multiple torch.utils.data.DataLoaders to create datasets that have different transforms applied to them. Currently, my code is roughly

d_transforms = [
    transforms.RandomHorizontalFlip(),
    # Some other transforms...
]
loaders = []
for i in range(len(d_transforms)):
    dataset = datasets.MNIST('./data',
            train=train,
            download=True,
            transform=d_transforms[i]
    loaders.append(
        DataLoader(dataset,
            shuffle=True,
            pin_memory=True,
            num_workers=1)
        )

This works, but it's extremely slow. kernprof shows that nearly all of the time in my code is spent on lines like

x, y = next(iter(train_loaders[i]))

I suspect that this is due to the fact that i'm using multiple instances of DataLoader, each with their own worker, which tries to read the same data files.

My question is, what is a better way to do this? Ideally, I would subclass torch.utils.data.DataSet and specify the transform i'd like to apply when sampling, but this doesn't seem possible due to __getitem__ not being able to take arguments.


title: Select/Mask different column index in every row

body:
In pytorch I have a multi-dimensional tensor, call it X

X = [[1, 2, 3], [4, 5, 6], [7, 8, 9], [10, 11, 12], ...]

Now I would like to select a different column index for each row like so

indices = [[0], [1], [0], [2], ...]
# now I expect following values to be returned:
[[1], [5], [7], [12], ...]

also I would like to achieve the opposite so that for the given indices I get

[[2, 3], [4, 6], [8, 9], [10, 11]]

Is there a "simple" way to achieve this without a for loop? I would be grateful for any ideas.


RuntimeError: Only Tensors of floating point dtype can require gradients

got following error from

  input = Variable(preprocessed_img, requires_grad = True)

img=train_loader.dataset.data[0]
print(type(img))
img_tensor = torch.tensor(img)
preprocess_image(img)

> def preprocess_image(img):    means=[0.485, 0.456, 0.406]     stds=[0.229,
> 0.224, 0.225]
>
>   preprocessed_img = img.copy()[: , :, ::-1]  for i in range(3):
>       preprocessed_img[:, :, i] = preprocessed_img[:, :, i] - means[i]
>       preprocessed_img[:, :, i] = preprocessed_img[:, :, i] / stds[i]
>   preprocessed_img = \
>       np.ascontiguousarray(np.transpose(preprocessed_img, (2, 0, 1)))
>   preprocessed_img = torch.from_numpy(preprocessed_img)
>   preprocessed_img.unsqueeze_(0)  input = Variable(preprocessed_img,
> requires_grad = True)     return input


title: How to convert 35 classes of cityscapes dataset to 19 classes?

body:
The following is a small snippet of my code. Using this, I can train my model called 'lolnet' on cityscapes dataset. But the dataset contains 35 classes/labels [0-34].

imports ***

trainloader = torch.utils.data.DataLoader(
    datasets.Cityscapes('/media/farshid/DataStore/temp/cityscapes/', split='train', mode='fine',
                    target_type='semantic', target_transform =trans,
                    transform=input_transform ), batch_size = batch_size, num_workers = 2)

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
net = lolNet()
criterion = CrossEntropyLoss2d()

net.to(device)
num_of_classes = 34

for epoch in range(int(0), 200000):

    lr = 0.0001

    for batch, data in enumerate(trainloader, 0):

        inputs, labels = data
        labels = labels.long()
        inputs, labels = inputs.to(device), labels.to(device)

        labels = labels.view([-1, ])

        optimizer = optim.Adam(net.parameters(), lr=lr)

        optimizer.zero_grad()
        outputs = net(inputs)

        outputs = outputs.view(-1, num_of_class)

        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()
        running_loss += loss.item()

        outputs = outputs.to('cpu')
        outputs = outputs.data.numpy()
        outputs = outputs.reshape([-1, num_of_class])

        mask = np.zeros([outputs.shape[0]])
        #
        for i in range(len(outputs)):
            mask[i] = np.argmax(outputs[i])

        mask = mask.reshape([-1, 1])

        IoU = jaccard_score(labels.to('cpu').data, mask, average='micro')

But I want to train my model only on the 19 classes. These 19 classes are found here . The labels to train for are stored as "ignoreInEval" = True. This pytorch Dataloader helper for this dataset doesnt provide any clue.

So my question is how can I train my model on the desired 19 classes of this dataset using pytorch's "datasets.Cityscapes" api.


title: Weight Normalization in PyTorch

body:
An important weight normalization technique was introduced in this paper and has been included in PyTorch since long as follows:

    from torch.nn.utils import weight_norm
    weight_norm(nn.Conv2d(in_channles, out_channels))

From the docs I get to know, weight_norm does re-parametrization before each forward() pass. But I am not sure if this re-parameterization is also happening during the inference when everything is running inside with torch.no_grad() and the model is set to eval() mode.

Can someone please help me know if weight_norm is active only during training or during the inference mode as described above?

Thank you


title: How to train a reverse embedding, like vec2word?

body:
how do you train a neural network to map from a vector representation, to one hot vectors? The example I'm interested in is where the vector representation is the output of a word2vec embedding, and I'd like to map onto the the individual words which were in the language used to train the embedding, so I guess this is vec2word?

In a bit more detail; if I understand correctly, a cluster of points in embedded space represents similar words. Thus if you sample from points in that cluster, and use it as the input to vec2word, the output should be a mapping to similar individual words?

I guess I could do something similar to an encoder-decoder, but does it have to be that complicated/use so many parameters?

There's this TensorFlow tutorial, how to train word2vec, but I can't find any help to do the reverse? I'm happy to do it using any deeplearning library, and it's OK to do it using sampling/probabilistic.

Thanks a lot for your help, Ajay.


title: What is the difference between Tensor.size and Tensor.shape in PyTorch?

body:
What is the difference between Tensor.size and Tensor.shape in Pytorch?
I want to get the number of elements and the dimensions of Tensor. For example for a tensor with the dimensions of 2 by 3 by 4 I expect 24 for number of elements and (2,3,4) for dimension.
Thanks.


title: multiprocessing error with pytorch on windows 10

body:
i get the following error when i try to execute my code, which clearly shows its an mulitprocessing error:

An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.

On Linux, the following code runs fine, but Im wondering why i cant get it running on Windows 10. However, here is the code:

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torchvision import datasets, transforms
from torch.autograd import Variable

kwargs = {'num_workers': 1, 'pin_memory': True}
train_data = torch.utils.data.DataLoader(datasets.MNIST('data', train=True,                 download=True,
                                            transform=transforms.Compose([transforms.ToTensor,
                                                            transforms.Normalize((0.1307,), (0.3081,))])),
                                 batch_size=64, shuffle=True, **kwargs)

test_data = torch.utils.data.DataLoader(datasets.MNIST('data', train=False,
                                                transform=transforms.Compose([transforms.ToTensor,
                                                            transforms.Normalize((0.1307,), (0.3081,))])),
                                 batch_size=64, shuffle=True, **kwargs)

class Netz(nn.Module):
    def __init__(self):
        super(Netz, self).__init__()
        self.conv1 = nn.Conv2d(1, 10, kernel_size=5)
        self.conv2 = nn.Conv2d(10, 20, kernel_size=5)
        self.conv_dropout = nn.Dropout2d()
        self.fc1 = nn.Linear(320, 60)
        self.fc2 = nn.Linear(60, 10)

    def forward(self, x):
        x = self.conv1(x)
        x = F.max_pool2d(x, 2)
        x = F.relu(x)
        x = self.conv2(x)
        x = self.conv_dropout(x)
        x = F.max_pool2d(x, 2)
        x = F.relu(x)
        print(x.size())
        exit()

model = Netz()
model.cuda()

optimizer = optim.SGD(model.parameters(), lr=0.1, momentum=0.8)

def train(epoch):
    model.train()
    for batch_id, (data, target) in enumerate(train_data):
        data = data.cuda()
        target = target.cuda()
        data = Variable(data)
        target = Variable(target)
        optimizer.zero_grad()
        out = model(data)
        criterion = F.nll_loss
        loss = criterion(out, target)
        loss.backward()
        optimizer.step()
        print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(epoch, batch_id * len(data), len(train_data.dataset),
                                                                   100. * batch_id / len(train_data), loss.data[0]))

for epoch in range(1, 30):
    print(epoch)
    train(epoch)

I tried to fix it with:

if __name__ == '__main__':
    for epoch in range(1, 30):
        train(epoch)

But also without success.
Anyone has an idea how i can fix this multiprocessing error?
Any help would be appreciated.
(Yes, i know pytorch is not officially released for windows, but i dont think this is causing the error here.)

Thanks!


model_save_name = 'classifier.pt'
path = F"/content/gdrive/My Drive/Others/{model_save_name}"

model.load_state_dict(torch.load(path), strict = False)

trying to load into model state_dict from path. Later the same model state_dict will be used for saving the checkpoint like torch.save(model.state_dict(), path)

Error Traceback:

UnpicklingError                           Traceback (most recent call last)
<ipython-input-69-7cb7940c24e3> in <module>()
----> 1 model.load_state_dict(torch.load(path), strict = False)

/usr/local/lib/python3.6/dist-packages/torch/serialization.py in load(f, map_location, pickle_module)
    356         f = open(f, 'rb')
    357     try:
--> 358         return _load(f, map_location, pickle_module)
    359     finally:
    360         if new_fd:

/usr/local/lib/python3.6/dist-packages/torch/serialization.py in _load(f, map_location, pickle_module)
    530             f.seek(0)
    531
--> 532     magic_number = pickle_module.load(f)
    533     if magic_number != MAGIC_NUMBER:
    534         raise RuntimeError("Invalid magic number; corrupt file?")

UnpicklingError: invalid load key, '\x0a'.


title: PyInstaller executable fails to get source code of TorchScript

body:
I'm trying to make Windows executable of my script that includes PyTorch. My script's imports are:

import numpy.core.multiarray  # which is a workaround for "ImportError: numpy.core.multiarray failed to import"
import six # which is workaround for "ModuleNotFoundError: No module named 'six'"
import torch
import torch.nn as nn
import warnings
import argparse
import json
import math
import numpy as np
import jsonschema
import os
from datetime import datetime
from sklearn.mixture import GaussianMixture
from scipy.io import wavfile
from scipy.signal import get_window
from scipy.signal import spectrogram

I'm using command:

pyinstaller --hidden-import pkg_resources.py2_warn extractor.py

PyInstaller throws no error while creating .exe, but when I run the .exe i get:

Traceback (most recent call last):
  File "site-packages\torch\_utils_internal.py", line 46, in get_source_lines_and_file
  File "inspect.py", line 967, in getsourcelines
  File "inspect.py", line 798, in findsource
OSError: could not get source code

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "extractor.py", line 3, in <module>
    import torch
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "C:\ProgramData\Anaconda3\envs\forexe2\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 623, in exec_module
    exec(bytecode, module.__dict__)
  File "site-packages\torch\__init__.py", line 367, in <module>
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "C:\ProgramData\Anaconda3\envs\forexe2\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 623, in exec_module
    exec(bytecode, module.__dict__)
  File "site-packages\torch\distributions\__init__.py", line 112, in <module>
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "C:\ProgramData\Anaconda3\envs\forexe2\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 623, in exec_module
    exec(bytecode, module.__dict__)
  File "site-packages\torch\distributions\von_mises.py", line 55, in <module>
  File "site-packages\torch\jit\__init__.py", line 1287, in script
  File "site-packages\torch\jit\frontend.py", line 164, in get_jit_def
  File "site-packages\torch\_utils_internal.py", line 53, in get_source_lines_and_file
OSError: Can't get source for <function _rejection_sample at 0x0000000006892F70>. TorchScript requires source access in order to carry out compilation, make sure original .py files are available. Original error: could not get source code
[5704] Failed to execute script extractor

Which I don't understand. This may be similar issue to this question. What is causing the problem?

I'm using conda env, with torch installed via pip (which is a workaround for the torch to be correctly hooked).

Windows 10
Python 3.8.2
torch 1.5.0+cu101
torchvision 0.6.0+cu101 (also tried with 0.2.2)
PyInstaller 3.6


title: LBFGS never converges in large dimensions in pytorch

body:
I am playing with Rule 110 of Wolfram cellular automata. Given line of zeroes and ones, you can calculate next line with these rules:

Starting with 00000000....1 in the end you get this sequence:

Just of curiosity I decided to approximate these rules with a polynomial, so that cells could be not only 0 and 1, but also gray color in between:

def triangle(x,y,z,v0):
    v=(y + y * y + y * y * y - 3. * (1. + x) * y * z + z * (1. + z + z * z)) / 3.
    return (v-v0)*(v-v0)

so if x,y,z and v0 matches any of these rules from the table, it will return 0, and positive nonzero value otherwise.

Next I've added all possible groups of 4 neighbors into single sum, which will be zero for integer solutions:

def eval():
    s = 0.
    for i in range(W - 1):
        for j in range(1, W + 1):
            xx = x[i, (j - 1) % W]
            yy = x[i, j % W]
            zz = x[i, (j + 1) % W]
            r = x[i + 1, j % W]
            s += triangle(xx, yy, zz, r)
    for j in range(W - 1): s += x[0, j] * x[0, j]
    s += (1 - x[0, W - 1]) * (1 - x[0, W - 1])
    return torch.sqrt(s)

Also in the bottom of this function I add ordinary conditions for first line, so that all elements are 0 except last one, which is 1. Finally I've decided to minimize this sum of squares on W*W matrix with pytorch:

x = Variable(torch.DoubleTensor(W,W).zero_(), requires_grad=True)
opt = torch.optim.LBFGS([x],lr=.1)
for i in range(15500):
    def closure():
        opt.zero_grad()
        s=eval()
        s.backward()
        return s
    opt.step(closure)

Here is full code, you can try it yourself. The problem is that for 10*10 it converges to correct solution in ~20 steps:

But if I take 15*15 board, it never finishes convergence:

The graph on the right shows how sum of squares is changing with each next iteration and you can see that it never reaches zero. My question is why this happens and how can I fix this. Tried different pytorch optimisers, but all of them perform worse then LBFGS. Tried different learning rates. Any ideas why this happen and how I can reach final point during optimisation?

UPD: improved convergence graph, log of SOS:

UPD2: I also tried doing same in C++ with dlib, and I don't have any convergency issues there, it goes much deeper in much less time:

I am using this code for optimisation in C++:

find_min_using_approximate_derivatives(bfgs_search_strategy(),
        objective_delta_stop_strategy(1e-87),
        s, x, -1)


title: Use pretrained embedding in Spanish with Torchtext

body:
I am using Torchtext in an NLP project. I have a pretrained embedding in my system, which I'd like to use. Therefore, I tried:

my_field.vocab.load_vectors(my_path)

But, apparently, this only accepts the names of a short list of pre-accepted embeddings, for some reason. In particular, I get this error:

Got string input vector "my_path", but allowed pretrained vectors are ['charngram.100d', 'fasttext.en.300d', ..., 'glove.6B.300d']

I found some people with similar problems, but the solutions I can find so far are "change Torchtext source code", which I would rather avoid if at all possible.

Is there any other way in which I can work with my pretrained embedding? A solution that allows to use another Spanish pretrained embedding is acceptable.

Some people seem to think it is not clear what I am saking. So, if the title and final question are not enough: "I need help using a pre-trained Spanish word embedding in Torchtext".


title: Variational Autoencoder gives same output image for every input mnist image when using KL divergence

body:
When not using KL divergence term, the VAE reconstructs mnist images almost perfectly but fails to generate new ones properly when provided with random noise.

When using KL divergence term, the VAE gives the same weird output both when reconstructing and generating images.

Here's the pytorch code for the loss function:

def loss_function(recon_x, x, mu, logvar):
    BCE = F.binary_cross_entropy(recon_x, x.view(-1, 784), size_average=True)
    KLD = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp())
    return (BCE+KLD)

recon_x is the reconstructed image, x is the original_image, mu is the mean vector while logvar is the vector containing the log of variance.

What is going wrong here? Thanks in advance :)


title: Stacking tensors in a list of tuples of tensors

body:
I have a list of tuples of PyTorch tensors. It looks like this:

[
    (tensor([1, 2, 3]),  tensor([4, 5, 6, 7]),  tensor([8])),
    (tensor([9, 10,11]), tensor([11,12,13,14]), tensor([15])),
    (tensor([16,17,18]), tensor([19,20,21,22]), tensor([23])),
    ...
]

Tensors in each column (that is, tensors that position k of their respective tuple) share the same shape. I want to stack the tensors in each column so that I end up with a single tuple, each value being the tensors concatenated along the dimension of the column.

In this case, the output tuple would have three values, and look like this:

(
 tensor([[1,2,3], [9,10,11], [16,17,18]]),

 tensor([[4,5,6,7], [11,12,13,14], [19,20,21,22]],

 tensor([[8],[15],[23])
)

This is a made-up example. I want to do this for tuples of any length, and tensors of arbitrary size. What is the best way to do this type of concatenation quickly using PyTorch?


title: ModuleNotFoundError: No module named ‘tools.nnwrap’ (windows)

body:
I am trying to install PyTorch but every time it throws the same error ( ModuleNotFoundError: No module named ‘tools.nnwrap’).

This is what is typed: pip install torch

This is what I get every time:

Collecting torch
Using cached https://files.pythonhosted.org/packages/f8/02/880b468bd382dc79896eaecbeb8ce95e9c4b99a24902874a2cef0b562cea/torch-0.1.2.post2.tar.gz

Requirement already satisfied: pyyaml in c:\users\user\appdata\local\programs\python\python37-32\lib\site-packages (from torch) (5.1.2)

Installing collected packages: torch

Running setup.py install for torch … error

ERROR: Command errored out with exit status 1:

command: ‘c:\users\user\appdata\local\programs\python\python37-32\python.exe’ -u -c ‘import sys, setuptools, tokenize; sys.argv[0] = ‘"’"‘C:\Users\User\AppData\Local\Temp\pip-install-oegjniuy\torch\setup.py’"’"’; file=’"’"‘C:\Users\User\AppData\Local\Temp\pip-install-oegjniuy\torch\setup.py’"’"’;f=getattr(tokenize, ‘"’"‘open’"’"’, open)(file);code=f.read().replace(’"’"’\r\n’"’"’, ‘"’"’\n’"’"’);f.close();exec(compile(code, file, ‘"’"‘exec’"’"’))’ install --record ‘C:\Users\User\AppData\Local\Temp\pip-record-b_c49sh\install-record.txt’ --single-version-externally-managed --compile

cwd: C:\Users\User\AppData\Local\Temp\pip-install-oegjniuy\torch

Complete output (23 lines):

running install

running build_deps

Traceback (most recent call last):

File “<string>”, line 1, in <module>

File “C:\Users\User\AppData\Local\Temp\pip-install-oegjniuy\torch\setup.py”, line 265, in
description=“Tensors and Dynamic neural networks in Python with strong GPU acceleration”,

File "C:\Users\User\AppData\Local\Programs\Python\Python37-32\Lib\site-packages\setuptools_init.py", line 145, in setup
return distutils.core.setup(**attrs)

File “C:\Users\User\AppData\Local\Programs\Python\Python37-32\Lib\distutils\core.py”, line 148, in setup
dist.run_commands()

File “C:\Users\User\AppData\Local\Programs\Python\Python37-32\Lib\distutils\dist.py”, line 966, in run_commands
self.run_command(cmd)

File “C:\Users\User\AppData\Local\Programs\Python\Python37-32\Lib\distutils\dist.py”, line 985, in run_command
cmd_obj.run()

File “C:\Users\User\AppData\Local\Temp\pip-install-oegjniuy\torch\setup.py”, line 99, in run
self.run_command(‘build_deps’)

File “C:\Users\User\AppData\Local\Programs\Python\Python37-32\Lib\distutils\cmd.py”, line 313, in run_command
self.distribution.run_command(command)

File “C:\Users\User\AppData\Local\Programs\Python\Python37-32\Lib\distutils\dist.py”, line 985, in run_command
cmd_obj.run()

File “C:\Users\User\AppData\Local\Temp\pip-install-oegjniuy\torch\setup.py”, line 51, in run
from tools.nnwrap import generate_wrappers as generate_nn_wrappers

ModuleNotFoundError: No module named ‘tools.nnwrap’
----------------------------------------
ERROR: Command errored out with exit status 1: ‘c:\users\user\appdata\local\programs\python\python37-32\python.exe’ -u -c ‘import sys, setuptools, tokenize; sys.argv[0] = ‘"’"‘C:\Users\User\AppData\Local\Temp\pip-install-oegjniuy\torch\setup.py’"’"’; file=’"’"‘C:\Users\User\AppData\Local\Temp\pip-install-oegjniuy\torch\setup.py’"’"’;f=getattr(tokenize, ‘"’"‘open’"’"’, open)(file);code=f.read().replace(’"’"’\r\n’"’"’, ‘"’"’\n’"’"’);f.close();exec(compile(code, file, ‘"’"‘exec’"’"’))’ install --record ‘C:\Users\User\AppData\Local\Temp\pip-record-_b_c49sh\install-record.txt’ --single-version-externally-managed --compile Check the logs for full command output.

Do anyone knows how to fix this error!!


title: How to get class_to_idx map for Custom Dataset in Pytorch

body:
I am attempting transfer learning with a CNN (vgg19) on the Oxford102 category dataset consisting of 8189 samples of flowers labeled from 1 through 102. Instead of loading the data with ImageFolder, which requires a tedious process of  structuring my data into train, valid and test folders with each class being a sub-folder holding my images, I decided to load it in using the Custom Dataset class following

https://pytorch.org/tutorials/beginner/data_loading_tutorial.html

A subset of the code I wrote up for my project

    data_dir_path = 'data/images/'
    labels_path = 'data/imagelabels.mat'
    class_label_path = 'data/class_label_map'

    # standard normalization for Imagenet models mean: [0.485, 0.456,0.406],
    # std :[0.229, 0.224, 0.225]

    data_transforms = {
'train': transforms.Compose([
    transforms.RandomRotation(45),
    transforms.RandomResizedCrop(224),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(),
    transforms.Normalize([0.485, 0.456, 0.406],
                         [0.229, 0.224, 0.225])
]),
'valid': transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize([0.485, 0.456, 0.406],
                         [0.229, 0.224, 0.225])
]),
'test': transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize([0.485, 0.456, 0.406],
                         [0.229, 0.224, 0.225])
]),
   }

    class MyDataset(Dataset):

        def __init__(self, image_labels, data_dir, transform=None):

    """

    :param image_labels_path: path to our labels
    :param root_dir: the directory which houses our images
    :param transform: apply any transform on our sample
    """

    self.image_labels = image_labels
    self.root_dir = data_dir
    self.transform = transform

        def __len__(self):
    label_dict = scipy.io.loadmat(self.image_labels)
    return len(label_dict['labels'][0])

        def __getitem__(self, idx):

    image_path_list = [os.path.join(self.root_dir, filename) for filename in os.listdir(self.root_dir)]

    image = Image.open(image_path_list[idx])

    label_dict = scipy.io.loadmat(self.image_labels)
    label_list = label_dict['labels'][0]

    # label index for pytorch should start form zero
    # so subtract -1 from each class
    label_list[:] = [i - 1 for i in label_list]
    label = label_list[idx]

    if self.transform:
        image = self.transform(image)

    return image, label

    image_datasets = {x: MyDataset(image_labels=labels_path, data_dir=data_dir_path, transform=data_transforms[x]) for x in
              ['train', 'valid', 'test']}

my model class instance inherited from nn.Module is

    classifier = Neural(25088, [4096], 102)

I have subtracted -1 from my label list since Pytorch expects labels to start from 0. Thus 0 through 101 for 102 labels. Correct me if I am wrong cause I get a "current target >=0 and current target <= n_classes failed" error if I don't subtract one.

class_label_map is a dict which maps class labels to flowers names

    {
"1": "pink primrose",
"2": "hard-leaved pocket orchid",
"3": "canterbury bells",
"4": "sweet pea",
"5": "english marigold",
"6": "tiger lily",
"7": "moon orchid",
"8": "bird of paradise",
"9": "monkshood",
"10": "globe thistle",
"11": "snapdragon",
    }

My big problem is to get a class_to_idx mapping, how do I do this, my flower names do not match the images if I visualize them, I get totally different flower names for my flowers.

I first created a mapping by having a dict with key my original label before I subtracted 1, and value the one after. Example

    class_to_idx = {77:76, 73:72, 1:0, 65:64......102:101...65:54}

This is beyond doubt wrong as I was getting totally wrong label for my images.

The first label of my image in my data_dir_path = 'data/images' is 77, upon subtracting one I get 76.
Would this mean the index for all labels 76 is 0, and if the next class is 72 , would that mean the index for all classes 72 is 1? So...

    class_to_idx = {76:0, 72:1, 0:2, 65:3....and so on}

The ImageFolder seems to have a class_to_idx attribute which if used on my Dataset throws an error,

    image_datasets['train'].class_to_idx

   AttributeError: 'MyDataset' object has no attribute 'class_to_idx'

This is obviously the case because my Dataset class does not contain any such attribute.

But seriously though, How do I map my classes to my index? This is super important as I need to checkpoint my model and load it back again to fire out predictions.It may sound really silly, but I really don't know what to do here, please help?


title: Pytorch : AttributeError: 'function' object has no attribute 'cuda'

body:
import torch
import models
model_names = sorted(name for name in models.__dict__
                     if name.islower() and not name.startswith("__")
                     and callable(models.__dict__[name]))
model = models.__dict__['resnet18']
model = torch.nn.DataParallel(model,device_ids = [0])  #PROBLEM CAUSING LINE
model.to('cuda:0')

To run this code you need to clone this repository : https://github.com/SoftwareGift/FeatherNets_Face-Anti-spoofing-Attack-Detection-Challenge-CVPR2019.git

Please run this piece of code inside the root folder of the cloned directory.

I am getting the follow error AttributeError: 'function' object has no attribute 'cuda'
I have tried using torch.device object as well for the same function and it results in the same error.
Please ask for any other details that are required. PyTorch newbie here
python:3.7 pytorch:1.3.1


title: PyTorch, apply different functions element-wise

body:
I defined a tensor like this

t_shape = [4, 1]
data = torch.rand(t_shape)

I want to apply different functions to each row.

funcs = [lambda x: x+1, lambda x: x**2, lambda x: x-1, lambda x: x*2]  # each function for each row.

I can do it with the following code

d = torch.tensor([f(data[i]) for i, f in enumerate(funcs)])

How can I do it in a proper way with more advanced APIs defined in PyTorch?


title: Pytorch Adam optimizer's awkward behavior? better with restart?

body:
I'm trying to train a CNN text classifier with Pytorch. I'm using the Adam optimizer like this.

optimizer = torch.optim.Adam(CNN_Text.parameters(), lr=args.lr)

I figured out that the optimizer converges really fast, and then it keeps on slowly dropping on accuracy. (the validation loss decreases a lot in 1-2 minutes, then it keeps on increasing slowly)

So, I implemented learning-rate decay,

If curr_loss > val_loss:
   prev_lr = param_group['lr']
   param_group['lr'] = prev_lr/10

I found out that it didn't really help a lot. But if I manually save the model, load it, and run the training with decreased learning rate, it really gets way better performance!

This gets me in hard time because I need to keep on watching the gradient descent and manually change the options. I tried SGD and other optimizers because I thought this was Adam's problem, but I couldn't find out a good way.

Can anyone help me with it?


title: How to make Intel GPU available for processing through pytorch?

body:
I'm using a laptop which has Intel Corporation HD Graphics 520.
Does anyone know how to it set up for Deep Learning, specifically Pytorch? I have seen if you have Nvidia graphics I can install cuda but what to do when you have intel GPU?


title: pytorch backprop through volatile variable error

body:
I'm trying to minimize some input relative to some target by running it through several backward pass iterations and updating the input at each step. The first pass runs successfully but I get the following error on the second pass:

RuntimeError: element 0 of variables tuple is volatile

This code snippet demonstrates the problem

import torch
from torch.autograd import Variable
import torch.nn as nn

inp = Variable(torch.Tensor([1]), requires_grad=True)
target = Variable(torch.Tensor([3]))

loss_fn = nn.MSELoss()

for i in range(2):
    loss = loss_fn(inp, target)
    loss.backward()
    gradient = inp.grad
    inp = inp - inp.grad * 0.01

When I inspect the value of inp, before it is reassigned on the last line, inp.volatile => False and inp.requires_grad => True but after it is reassigned those switch to True and False, respectively. Why does being a volatile variable prevent the second backprop run?


title: Sharing parameters between certain layers of different instances of the same pytorch model

body:
I have a pytorch model with multiple layers that looks something like this

class CNN(nn.Module):
    def __init__(self):
        super(CNN).__init__()
        self.layer1 = nn.Conv2d(#parameters)
        self.layer2 = nn.Conv2d(#different_parameters)
        self.layer3 = nn.Conv2d(#other_parameters)
        self.layer4 = nn.Conv2d(#final_parameters)

    def forward(self, x):
        out1 = self.layer2(F.relu(self.layer1(x)))
        out2 = self.layer4(F.relu(self.layer3(x)))
        return torch.cat((out1, out2), 0)

I then want to instantiate multiple instances of this class (cnn1, cnn2), and share the parameters of the first path (layer1, layer2) across instances while keeping the other parameters separate.

Is there an optimal/supported way to do this?


title: How to solve "No module named 'tools'" in google colab?

body:
I am trying to follow this tutorial to apply Bert in my data: https://medium.com/swlh/a-simple-guide-on-using-bert-for-text-classification-bbf041ac8d04

And Getting error on imports

This is what I am importing

import torch
import pickle
from torch.utils.data import (DataLoader, RandomSampler, SequentialSampler, TensorDataset)
from torch.nn import CrossEntropyLoss, MSELoss

from tqdm import tqdm_notebook, trange
import os
from pytorch_pretrained_bert import BertTokenizer, BertModel, BertForMaskedLM, BertForSequenceClassification
from pytorch_pretrained_bert.optimization import BertAdam, WarmupLinearSchedule

from multiprocessing import Pool, cpu_count
from tools import *
import convert_examples_to_features

and getting error on:  from tools import * as "No module named 'tools'".

I tried to follow some answers like:
Python/Caffe2: ImportError: No module named tools.setup_helpers.env

pyserial: No module named tools

ModuleNotFoundError: No module named 'tools.nnwrap'

However, these answers didn't solve the error for me.

Please help!


title: How can I build an LSTM AutoEncoder with PyTorch?

body:
I have my data as a DataFrame:

       dOpen     dHigh      dLow    dClose   dVolume  day_of_week_0  day_of_week_1  ...  month_6  month_7  month_8  month_9  month_10  month_11  month_12
639 -0.002498 -0.000278 -0.005576 -0.002228 -0.002229              0              0  ...        0        0        1        0         0         0         0
640 -0.004174 -0.005275 -0.005607 -0.005583 -0.005584              0              0  ...        0        0        1        0         0         0         0
641 -0.002235  0.003070  0.004511  0.008984  0.008984              1              0  ...        0        0        1        0         0         0         0
642  0.006161 -0.000278 -0.000281 -0.001948 -0.001948              0              1  ...        0        0        1        0         0         0         0
643 -0.002505  0.001113  0.005053  0.002788  0.002788              0              0  ...        0        0        1        0         0         0         0
644  0.004185  0.000556 -0.000559 -0.001668 -0.001668              0              0  ...        0        0        1        0         0         0         0
645  0.002779  0.003056  0.003913  0.001114  0.001114              0              0  ...        0        0        1        0         0         0         0
646  0.000277  0.004155 -0.002227 -0.002782 -0.002782              1              0  ...        0        0        1        0         0         0         0
647 -0.005540 -0.007448 -0.003348  0.001953  0.001953              0              1  ...        0        0        1        0         0         0         0
648  0.001393 -0.000278  0.001960 -0.003619 -0.003619              0              0  ...        0        0        1        0         0         0         0

My input will be 10 rows (already one-hot encoded). I want to create an n-dimensional auto encoded representation. So as I understand it, my input and output should be the same.

I've seen some examples to construct this, but am still stuck on the first step. Is my training data just a lot of those samples as to make a matrix? What then?

I apologize for the general nature of the question. Any questions, just ask and I will clarify in the comments.

Thank you.


title: Does PyTorch support variable with dynamic dimension?

body:
I've updated my question based upon the variable dimension of variables.

Suppose the input tensor stores the 3d points with dimension 10x3, 10 means the #points and 3 is the feature dimension (say x,y,z coordinates). The dimension of the variable depends on the input tensor, say its dimension is 10x10. When the input tensor changes its dimension to 50x3, then the dimension of the variable will also have to change to 50x50.

I know in Tensorflow, if the input dimension is changing/unknown, we can declare it as tf.placeholder(None,3). However, I never meet the situation where the size of variable is changing/unknown, it seems that the variable will always have the fixed dimension.

I am currently learning PyTorch and don't know whether PyTorch supports this function. Any information would be appreciated!

========= Original question ========

I have a variable in which the size is changeable when input dimension changes. For example, if input is 10x2, then the variable should be 10x10. If input is 25x2, then the variable should be 25x25. As my understanding, the variable is used to store weights, which normally has fixed dimension. However in my case, the dimension of the variable depends on input data, which can change. Does PyTorch currently supports this kind of function?

Thanks!


title: PyTorch version of as simple Keras LSTM model

body:
Trying to translate a simple LSTM model in Keras to PyTorch code. The Keras model converges after just 200 epochs, while the PyTorch model:

needs many more epochs to reach the same loss level (200 vs. ~8000)
seems to overfit the inputs because the predicted value is not near 100

This is the Keras code:

from numpy import array
from keras.models import Sequential
from keras.layers import LSTM
from keras.layers import Dense

X = array([10,20,30,20,30,40,30,40,50,40,50,60,50,60,70,60,70,80]).reshape((6,3,1))
y = array([40,50,60,70,80,90])
model = Sequential()
model.add(LSTM(50, activation='relu', recurrent_activation='sigmoid',  input_shape=(3, 1)))
model.add(Dense(1))
model.compile(optimizer='adam', loss='mse')
model.fit(X, y, epochs=200, verbose=1)
x_input = array([70, 80, 90]).reshape((1, 3, 1))
yhat = model.predict(x_input, verbose=0)
print(yhat)

And this is the equivalent PyTorch code:

from numpy import array
import torch
import torch.nn as nn
import torch.nn.functional as F

X = torch.tensor([10,20,30,20,30,40,30,40,50,40,50,60,50,60,70,60,70,80]).float().reshape(6,3,1)
y = torch.tensor([40,50,60,70,80,90]).float().reshape(6,1)

class Model(nn.Module):
  def __init__(self):
    super(Model, self).__init__()
    self.lstm = nn.LSTM(input_size=1, hidden_size=50, num_layers=1, batch_first=True)
    self.fc = nn.Linear(50, 1)

  def forward(self, x):
    batches = x.size(0)
    h0 = torch.zeros([1, batches, 50])
    c0 = torch.zeros([1, batches, 50])
    (x, _) = self.lstm(x, (h0, c0))
    x = x[:,-1,:]  # Keep only the output of the last iteration. Before shape (6,3,50), after shape (6,50)
    x = F.relu(x)
    x = self.fc(x)
    return x

model = Model()
criterion = nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters())

n_epochs = 8000
for epoch in range(n_epochs):
  model.train()
  optimizer.zero_grad()
  y_ = model(X)
  loss = criterion(y_, y)
  loss.backward()
  optimizer.step()
  print(f"Epoch {epoch+1}/{n_epochs}, loss = {loss.item()}")

model.eval()
x_input = torch.tensor([70, 80, 90]).float().reshape((1, 3, 1))
yhat = model(x_input)
print(yhat)

The only possible difference is the initial weight and bias values, but I don't think that slightly different weights and biases may account for such a big difference in behavior.
What am I missing in the PyTorch code?


title: Why does roi_align not seem to work in pytorch?

body:
I am a pytorch beginner. It seems that there is a bug in the RoIAlign module in pytorch. The code is simple but the result is out of my expectation.

code:

import torch
from torchvision.ops import RoIAlign

if __name__ == '__main__':
    output_size = (3,3)
    spatial_scale = 1/4
    sampling_ratio = 2

    #x.shape:(1,1,6,6)
    x = torch.FloatTensor([[
        [[1,2,3,4,5,6],
        [7,8,9,10,11,12],
        [13,14,15,16,17,18],
        [19,20,21,22,23,24],
        [25,26,27,28,29,30],
        [31,32,33,34,35,36],],
    ]])

    rois = torch.tensor([
        [0,0.0,0.0,20.0,20.0],
    ])
    channel_num = x.shape[1]
    roi_num = rois.shape[0]

    a = RoIAlign(output_size, spatial_scale=spatial_scale, sampling_ratio=sampling_ratio)
    ya = a(x, rois)
    print(ya)

output:

tensor([[[[ 6.8333,  8.5000, 10.1667],
          [16.8333, 18.5000, 20.1667],
          [26.8333, 28.5000, 30.1667]]]])

But in this case shouldn't it be an average pooling operation on every 2x2 cell, like:

tensor([[[[ 4.5000,  6.5000, 8.5000],
          [16.5000, 18.5000, 20.5000],
          [28.5000, 30.5000, 32.5000]]]])

My torch version is 1.3.0 with python3.6 and cuda 10.1, on Ubuntu16. I have been troubled for two days and I couldn't appreciate it more if anyone could help me.


title: error: unrecognized arguments: - pytorch code in colab

body:
I tried to run my code on google colab. but I get this message (error: unrecognized arguments)when I'm trying to call this function :

def parse_opts():
 parser = argparse.ArgumentParser()
 parser.add_argument(
     '--root_path',
     default='/root/data/ActivityNet',
     type=str,
     help='Root directory path of data')
  parser.add_argument(
     '--video_path',
     default='video_kinetics_jpg',
     type=str,
     help='Directory path of Videos')
 args = parser.parse_args()

return args

but this is failed and I get this error

  tester_video.py: error: unrecognized arguments:
  cifar_comp_20_200_0.01_0.1 20 10 0.01 0.1

I tried to use Easydict but it seems its not working
thinks


title: RuntimeError: unable to open shared memory object, OSError: [Errno 24] Too many open files

body:
I having trouble with loading indexes of document.
I am testing my code, so I set

batch_size = 4
number_of_sentences_in_document = 84
number_of_words_in_sentence = 80

that sums up one mini_batch with 80 * 84 * 4 indexes of documents.

The problem is that when I transform those indexes dataset into a DataLoader as below
and try to loop over trainloader, it results out so many error messages.

DataManager = DS.NewsDataset(data_examples_gen, Vocab)
trainloader = torch.utils.data.DataLoader(DataManager, batch_size=Args.args.batch_size, shuffle=True, num_workers=32)

The error messages are below.

Traceback (most recent call last):   File "/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()   File "/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)   File "/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 61, in _worker_loop
    data_queue.put((idx, samples))   File "/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/multiprocessing/queues.py", line 341, in put   File "/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/multiprocessing/reduction.py", line 51, in dumps   File "/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/site-packages/torch/multiprocessing/reductions.py", line 125, in reduce_storage   File "/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/multiprocessing/reduction.py", line 191, in DupFd   File "/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/multiprocessing/resource_sharer.py", line 48, in __init__ OSError: [Errno 24] Too many open files

During handling of the above exception, another exception occurred:

Traceback (most recent call last):   File "/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/multiprocessing/util.py", line 262, in _run_finalizers   File "/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/multiprocessing/util.py", line 186, in __call__   File "/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/shutil.py", line 476, in rmtree   File "/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/shutil.py", line 474, in rmtree OSError: [Errno 24] Too many open files: '/tmp/pymp-be4nmgxw' Process Process-2: Traceback (most recent call last):   File "/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()   File "/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)   File "/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 61, in _worker_loop
    data_queue.put((idx, samples))   File "/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/multiprocessing/queues.py", line 341, in put   File "/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/multiprocessing/reduction.py", line 51, in dumps   File "/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/site-packages/torch/multiprocessing/reductions.py", line 125, in reduce_storage   File "/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/multiprocessing/reduction.py", line 191, in DupFd   File "/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/multiprocessing/resource_sharer.py", line 48, in __init__ OSError: [Errno 24] Too many open files Traceback (most recent call last):   File "/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()   File "/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)   File "/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 61, in _worker_loop
    data_queue.put((idx, samples))   File "/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/multiprocessing/queues.py", line 341, in put   File "/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/multiprocessing/reduction.py", line 51, in dumps   File "/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/site-packages/torch/multiprocessing/reductions.py", line 121, in reduce_storage RuntimeError: unable to open shared memory object </torch_54415_3383444026> in read-write mode at /opt/conda/conda-bld/pytorch_1525909934016/work/aten/src/TH/THAllocator.c:342

During handling of the above exception, another exception occurred:

Traceback (most recent call last):   File "/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/multiprocessing/util.py", line 262, in _run_finalizers   File "/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/multiprocessing/util.py", line 186, in __call__   File "/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/shutil.py", line 476, in rmtree   File "/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/shutil.py", line 474, in rmtree OSError: [Errno 24] Too many open files: '/tmp/pymp-abguy87b' Process Process-1: Traceback (most recent call last):   File "/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()   File "/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)   File "/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 61, in _worker_loop
    data_queue.put((idx, samples))   File "/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/multiprocessing/queues.py", line 341, in put   File "/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/multiprocessing/reduction.py", line 51, in dumps   File "/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/site-packages/torch/multiprocessing/reductions.py", line 121, in reduce_storage RuntimeError: unable to open shared memory object </torch_54415_3383444026> in read-write mode at /opt/conda/conda-bld/pytorch_1525909934016/work/aten/src/TH/THAllocator.c:342 Traceback (most recent call last):   File "/home/nlpgpu3/LinoHong/FakeNewsByTitle/main.py", line 26, in <module>
    for mini_batch in trainloader :   File "/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 280, in __next__
    idx, batch = self._get_batch()   File "/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 259, in _get_batch
    return self.data_queue.get()   File "/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/multiprocessing/queues.py", line 335, in get
    res = self._reader.recv_bytes()   File "/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/multiprocessing/connection.py", line 216, in recv_bytes
    buf = self._recv_bytes(maxlength)   File "/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/multiprocessing/connection.py", line 407, in _recv_bytes
    buf = self._recv(4)   File "/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)   File "/home/nlpgpu3/anaconda3/envs/linohong3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 178, in handler
    _error_if_any_worker_fails() RuntimeError: DataLoader worker (pid 54416) exited unexpectedly with exit code 1.

Process finished with exit code 1

I thought this is some kind of memory problem so I tried the same thing
only having two sentences for a document and it worked.
However, I am expecting this to get pretty much larger with
batch_size up to 32 or 64,
the number of sentences per document up to 84
the number of words per sentence up to 84.

I tried

$ ulimit -n 10000

but that one didn't work.

How can I manage this problem?
Any Idea???


title: Pytorch Exception in Thread: ValueError: signal number 32 out of range

body:
I'm getting this error:

Exception in Thread: ValueError: signal number 32 out of range

The specific tutorial that raises an issue for me is the training a classifier (https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html), the specific line is:

dataiter = iter(trainloader)

and the full error traceback is:

Exception in thread Thread-5:
Traceback (most recent call last):
File "/home/chenchen/anaconda3/lib/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/home/chenchen/anaconda3/lib/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/home/chenchen/anaconda3/lib/python3.6/multiprocessing/resource_sharer.py", line 139, in _serve
signal.pthread_sigmask(signal.SIG_BLOCK, range(1, signal.NSIG))
File "/home/chenchen/anaconda3/lib/python3.6/signal.py", line 60, in pthread_sigmask
sigs_set = _signal.pthread_sigmask(how, mask)
ValueError: signal number 32 out of range

My operation system is Ubuntu 18.10 and my python env is Anaconda3 for python 3.6. I installed pytorch from the latest source. My cuda version is 10.0.


title: pytorch questions: how to add bias term and extract its value? class vs sequential model? and softmax

body:
I have a basic neural network model in pytorch like this:

import torch
import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):
    def __init__(self, input_dim, hidden_dim, output_dim):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(input_dim, hidden_dim)
        self.sigmoid = nn.Sigmoid()
        self.fc2 = nn.Linear(hidden_dim, output_dim)

    def forward(self, x):
        out = self.fc1(x)
        out = self.sigmoid(out)
        out = self.fc2(out)
        return out
net = Net(400, 512,10)

How can I extract bias/intercept term from net.parameters()?
And is this model equivalent to using sequential()?

net = nn.Sequential(nn.Linear(input_dim, hidden_dim[0]),
                      nn.Sigmoid(),
                      nn.Linear(hidden_dim[0], hidden_dim[1]),
                      nn.Sigmoid(),
                      nn.Linear(hidden_dim[1], output_dim))

Is nn.Softmax() optional at the end of either model for multi-class classification? If I understand correctly, with software it outputs probability of a certain class but without it returns predicted output?

Thanks in advance for answering my newbie questions.


title: How to efficiently run multiple Pytorch Processes / Models at once ? Traceback: The paging file is too small for this operation to complete

body:
Background
I have a very small network which I want to test with different random seeds.
The network barely uses 1% of my GPUs compute power so i could in theory run 50 processes at once to try many different seeds at once.
Problem
Unfortunately i can't even import pytorch in multiple processes. When the nr of processes exceeds 4 I get a Traceback regarding a too small paging file.
Minimal reproducable code§ - dispatcher.py
from subprocess import Popen
import sys

procs = []
for seed in range(50):
    procs.append(Popen([sys.executable, "ml_model.py", str(seed)]))

for proc in procs:
    proc.wait()

§I increased the number of seeds so people with better machines can also reproduce this.
Minimal reproducable code - ml_model.py
import torch
import time
time.sleep(10)

 Traceback (most recent call last):
  File "ml_model.py", line 1, in <module>
    import torch
  File "C:\Users\user\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\__init__.py", line 117, in <module>
    import torch
  File "C:\Users\user\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\__init__.py", line 117, in <module>
    raise err
 OSError: [WinError 1455] The paging file is too small for this operation to complete. Error loading "C:\Users\user\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\lib\cudnn_cnn_infer64_8.dll" or one of its dependencies.
    raise err

Further Investigation
I noticed that each process loads a lot of dll's into RAM. And when i close all other programs which use a lot of RAM i can get up to 10 procesess instead of 4. So it seems like a resource constraint.
Questions
Is there a workaround ?
What's the recommended way to train many small networks with pytorch on a single gpu ?
Should i write my own CUDA Kernel instead, or use a different framework to achieve this ?
My goal would be to run around 50 processes at once (on a 16GB RAM Machine, 8GB GPU RAM)