title: Pytorch to ONNX export function fails and causes legacy function error

body:
I am trying to convert the pytorch model in this link to onnx model using the code below :

device=t.device('cuda:0' if t.cuda.is_available() else 'cpu')
print(device)

faster_rcnn = FasterRCNNVGG16()
trainer = FasterRCNNTrainer(faster_rcnn).cuda()
#trainer = FasterRCNNTrainer(faster_rcnn).to(device)
trainer.load('./checkpoints/model.pth')

dummy_input = t.randn(1, 3, 300, 300, device = 'cuda')
#dummy_input = dummy_input.to(device)
t.onnx.export(faster_rcnn, dummy_input, "model.onnx", verbose = True)

But I get the following error  (Sorry for the block quote below stackoverflow wouldn't let the whole trace be in code format and wouldn't let the question be posted otherwise):

  Traceback (most recent call last):
     small_object_detection_master_samirsen\onnxtest.py", line 44, in <module>
       t.onnx.export(faster_rcnn, dummy_input, "fasterrcnn_10120119_06025842847785781.onnx", verbose = True)
     File "C:\Users\HP\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\onnx\__init__.py",

  line 132, in export
             strip_doc_string, dynamic_axes)
           File "C:\Users\HP\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\onnx\utils.py",
  line 64, in export
             example_outputs=example_outputs, strip_doc_string=strip_doc_string, dynamic_axes=dynamic_axes)
           File "C:\Users\HP\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\onnx\utils.py",
  line 329, in _export
             _retain_param_name, do_constant_folding)
           File "C:\Users\HP\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\onnx\utils.py",
  line 213, in _model_to_graph
             graph, torch_out = _trace_and_get_graph_from_model(model, args, training)
           File "C:\Users\HP\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\onnx\utils.py",
  line 171, in _trace_and_get_graph_from_model
             trace, torch_out = torch.jit.get_trace_graph(model, args, _force_outplace=True)
           File "C:\Users\HP\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\jit__init__.py",
  line 256, in get_trace_graph
             return LegacyTracedModule(f, _force_outplace, return_inputs)(*args, **kwargs)
           File "C:\Users\HP\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\nn\modules\module.py",
  line 547, in call
             result = self.forward(*input, **kwargs)
           File "C:\Users\HP\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\jit__init__.py",
  line 323, in forward
             out = self.inner(*trace_inputs)
           File "C:\Users\HP\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\nn\modules\module.py",
  line 545, in call
             result = self._slow_forward(*input, **kwargs)
           File "C:\Users\HP\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\nn\modules\module.py",
  line 531, in _slow_forward
          File "C:\Users\HP\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\nn\modules\module.py",
  line 531, in _slow_forward
             result = self.forward(*input, **kwargs)
           File "D:\smallobject2\export test s\small_object_detection_master_samirsen\model\faster_rcnn.py", line
  133, in forward
             h, rois, roi_indices)
           File "C:\Users\HP\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\nn\modules\module.py",
  line 545, in call
             result = self._slow_forward(*input, **kwargs)
           File "C:\Users\HP\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\nn\modules\module.py",
  line 531, in _slow_forward
             result = self.forward(*input, **kwargs)
           File "D:\smallobject2\export test s\small_object_detection_master_samirsen\model\faster_rcnn_vgg16.py",
  line 142, in forward
             pool = self.roi(x, indices_and_rois)
           File "C:\Users\HP\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\nn\modules\module.py",
  line 545, in call
             result = self._slow_forward(*input, **kwargs)
           File "C:\Users\HP\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\nn\modules\module.py",
  line 531, in _slow_forward
             result = self.forward(*input, **kwargs)
           File "D:\smallobject2\export test s\small_object_detection_master_samirsen\model\roi_module.py", line
  85, in forward
             return self.RoI(x, rois)
          RuntimeError: Attempted to trace RoI, but tracing of legacy functions is not supported


title: Pytorch: merging two models (nn.Module)

body:
I have a model that's quite complicated and I therefore can't just call self.fc.weight etc. so I want to iterate over the model in some way.

The goal is to merge models this way: m = alpha * n + (1 - alpha) * o where m n and o are instances of the same class but trained differently. So for each parameter in these models, I want to assign initial values to m based on n and o as described in the equation and then continue the training procedure with m only.

I tried:

for p1, p2, p3 in zip(m.parameters(), n.parameters(), o.parameters()):
    p1 = alpha * p2 + (1 - alpha) * p3

But this does not assign new values within m.

for p1, p2, p3 in zip(m.parameters(), n.parameters(), o.parameters()):
    p1.fill_(alpha * p2 + (1 - alpha) * p3)

But this throws

  RuntimeError: a leaf Variable that requires grad has been used in an in-place operation.

And so I resorted to a working

m.load_state_dict({
    k: alpha * v1 + (1 - alpha) * v2
    for (k, v1), (_, v2) in zip(n.state_dict().items(), o.state_dict().items())
})

Is there a better way to do this in Pytorch? Is it possible that I get gradient errors?


title: Training a BERT model and using the BERT embeddings

body:
I've been reading up on BERT and using BERT embeddings for a classification task. I've read many articles but my understanding of it still is not 100% (I have self-taught myself NLP, so my access to resources can be a bit restricted). First I'll describe my task.
I was planning on using BERT embeddings for classification because of how it encapsulates the meaning and language. Unfortunately, there are no BERT models in my language (Irish), so I looked into training my own. I know that BERT is basically an 'extension' of sorts to a Transformer Encoder.
Here are my issues/questions:

I presume this is fairly obvious, but to check, pre-trained BERT embeddings cannot be applied to different languages (the standard embedding model is trained on the wiki dataset for English, I presume it may not be used on other languages for obvious reasons)?

My datset contains about 850k sentences in Irish (around 22M words). Would that be enough to train a decent BERT model? I could find a bit more data but to get significantly more in Irish would be very hard.

Would one recommend to make a BERT model 'from scratch' in PyTorch or TensorFlow, or are models from the likes of Fairseq and OpenNMT good to use?

Apologies for such a disjointed question, but in summary, I'm all over the place trying to make complete sense of BERT, specifically the training process and tuning it just for embeddings. If I've got this all wrong, or just have advice, I'd appreciate the feedback.


title: How to convert opencv mat grayscale image to pytorch tensor?

body: I want to convert cv::Mat class into torch::Tensor class and put them into CNN module of libtorch for prediction. How should I convert Mat class to Tensor class? My CNN module is a pt file generated using the mnist training set. The training image of this mnist training set is single-channel gray image. If I want to test, it should load a gray mat image, so I want to change the single-channel gray image converted to tensor. The language I use is c++。


title: parallelize for loop in tensorflow for training multiple models in parallel

body:
I have multiple models that I want to train in parallel in tensorflow. The models are currently inside a for loop. When I tried to train the model, the training happened sequentially rather than in parallel. My question is similar to the one asked here (but nobody answered this question). However, I am training the model and not only doing inference. One of the comments on that question is "The way to make it faster is to use a single process with single TensorFlow runtime. TensorFlow was not optimized for the case of multiple runtimes sharing a GPU".
In line with the above comment, I noticed that if I just make a single large model (e.g., even a single model with larger total number of parameters than multiple models in the loop combined), it executes faster (e.g., 0.19 sec/step for this large model). This is in contrast with a for loop training four different models (which takes about 0.55 sec/step for the entire for loop -- almost 4 times the larger model). Can anyone advise me how to parallelize this for loop so that all iterations of the for loop execute in parallel (i.e., in almost 0.19 sec/step for the entire loop instead of 0.55 sec/step)? All of my models are able to fit in the GPU memory.
PS: Due to the nature of my work, I am unable to use tf.while_loop since it requires the loss functions to be defined within the while_loop. Is there any other way to do this kind of parallelization of for loop on the GPU (e.g., maybe using threading)? I am also not using "per_process_gpu_memory_fraction" in tensorflow since the other question above indicated that it makes things slow.
Also, is it possible to do this kind of parallelization in keras or pytorch?
Any help will be much appreciated. Thanks.


title: Extracting 3D patches from 3D images in both overlapping and nonoverlapping process and recovering the image back

body:
I am working with 172x220x156 shaped 3D images. To feed the image into the network for output I need to extract patches of size 32x32x32 from the image and add those back to get the image again.
Since my image dimension are not multiples of patch size thus I have to get overlapping patches.
I want to know how to do that.

I am working in PyTorch, there are some options like unfold and fold but I am not sure how they work.


title: How to Debug Saving Model TypeError: can't pickle SwigPyObject objects?

body:
I'm trying to save a model, which is an object of a class that inherits from nn.Module. It trains fine without problem, but when I try running the code:

    torch.save(
        obj=model,
        f=os.path.join(tensorboard_writer.get_logdir(), 'model.ckpt'))

I receive the error: TypeError: can't pickle SwigPyObject objects

I have no idea what a SwigPyObject object is. How do I debug this error to save my model?

Full Traceback:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/torch/serialization.py", line 149, in _with_file_like
    return body(f)
  File "/usr/local/lib/python3.6/dist-packages/torch/serialization.py", line 224, in <lambda>
    return _with_file_like(f, "wb", lambda f: _save(obj, f, pickle_module, pickle_protocol))
  File "/usr/local/lib/python3.6/dist-packages/torch/serialization.py", line 296, in _save
    pickler.dump(obj)
TypeError: can't pickle SwigPyObject objects

In case it helps, my model is an object of the following class:

class RecurrentModel(nn.Module):

    def __init__(self,
                 core_str,
                 core_kwargs,
                 tensorboard_writer=None,
                 input_size=1,
                 hidden_size=32,
                 output_size=2):

        super(RecurrentModel, self).__init__()
        self.tensorboard_writer = tensorboard_writer
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.output_size = output_size

        self.core = self._create_core(core_str=core_str, core_kwargs=core_kwargs)
        self.core_hidden = None
        self.linear = nn.Linear(
            in_features=hidden_size,
            out_features=output_size,
            bias=True)
        self.softmax = nn.Softmax(dim=-1)

        # converts all weights into doubles i.e. float64
        # this prevents PyTorch from breaking when multiplying float32 * flaot64
        self.double()

        # TODO figure out why writing the model to tensorboard doesn't work
        # dummy_input = torch.zeros(size=(10, 1, 1), dtype=torch.double)
        # tensorboard_writer.add_graph(
        #     model=self,
        #     input_to_model=dict(stimulus=dummy_input))

    def _create_core(self, core_str, core_kwargs):
        if core_str == 'lstm':
            core_constructor = nn.LSTM
        elif core_str == 'rnn':
            core_constructor = nn.RNN
        elif core_str == 'gru':
            core_constructor = nn.GRU
        else:
            raise ValueError('Unknown core string')

        core = core_constructor(
            input_size=self.input_size,
            hidden_size=self.hidden_size,
            batch_first=True,
            **core_kwargs)
        return core

    def forward(self, model_input):

        if self.core_hidden is None:
            core_output, self.core_hidden = self.core(
                model_input['stimulus'])
        else:
            core_output, self.core_hidden = self.core(
                model_input['stimulus'],
                self.core_hidden)

        linear_output = self.linear(core_output)

        softmax_output = self.softmax(linear_output)

        forward_output = dict(
            core_output=core_output,
            core_hidden=self.core_hidden,
            linear_output=linear_output,
            softmax_output=softmax_output)

        return forward_output

    def reset_core_hidden(self):
        self.core_hidden = None

I then construct and save the model:

    tensorboard_writer = SummaryWriter()

    model = RecurrentModel(
        core_str='rnn',
        core_kwargs={},
        tensorboard_writer=tensorboard_writer)

    torch.save(
        obj=model,
        f=os.path.join(tensorboard_writer.get_logdir(), 'model.ckpt')
    )


title: PyTorch RNN gradients for variable length input sequences are very small

body:
I'm aiming to do multiclass classification on sentences. The input to the my RNN(LSTM or GRU) is a batched input of variable length sequences(which are indexed using Glove embeddings). This input is right padded with zeros. The redefined forward for my GRU RNN is:

def last_timestep(self, unpacked, lengths):
    # Index of the last output for each sequence
    idx = (lengths - 1).view(-1, 1).expand(unpacked.size(0), unpacked.size(2)).unsqueeze(1)
    return unpacked.gather(1, idx).squeeze()

def forward(self, x, lengths, **kwargs):
    """Forward propagation of activations"""

    if self.gpu:
        x = Variable(x).cuda()
        lengths = Variable(lengths).cuda()
    else:
        x = Variable(x)
        lengths = Variable(lengths)

    # batch_size = int(x.size()[0])
    # h_0 = Variable(torch.zeros(self.total_layers, batch_size, self.hidden_size)).cuda()

    # Embed and pack the padded sequence
    embs = self.embeddings(x)
    packed = pack_padded_sequence(embs, list(lengths.data), batch_first=True)
    out_packed, _ = self.gru(packed)
    out_unpacked, _ = pad_packed_sequence(out_packed, batch_first=True)
    out_last = self.last_timestep(out_unpacked, lengths)
    output = self.lin(out_last)
    return output

As for training, I'm using CrossEntropyLoss. However, when I test the prediction, it always predicts the same class irrespective of the sentence input. Moreover, the final output from the RNN(variable output) is almost the same! On closer inspection, I've discovered that the problem is in backpropagation.

The gradients are very very low(in the order of 10^-3 and some are much lower) for many of the parameters. Moreover, I'm not sure if the packing or padding is helping at all?? I've tried running the code without any packing(just running forward on the padded input) and I get the same output, which leads me to believe that I'm doing something wrong with packing and unpacking. I'd really appreciate any help. Thank you!


title: With the HuggingFace transformer, how can I return multiple samples when generating text?

body:
I'm going off of https://github.com/cortexlabs/cortex/blob/master/examples/pytorch/text-generator/predictor.py

But if I pass num_samples=5, I get:

    generated = torch.cat((generated, next_token.unsqueeze(0)), dim=1)
RuntimeError: Sizes of tensors must match except in dimension 1. Got 5 and 1 in dimension 0

the code is:

def sample_sequence(
    model,
    length,
    context,
    num_samples=1,
    temperature=1,
    top_k=0,
    top_p=0.9,
    repetition_penalty=1.0,
    device="cpu",
):
    context = torch.tensor(context, dtype=torch.long, device=device)
    context = context.unsqueeze(0).repeat(num_samples, 1)
    print('context_size', context.shape)
    generated = context
    print('context', context)
    with torch.no_grad():
        for _ in trange(length):
            inputs = {"input_ids": generated}
            outputs = model(
                **inputs
            )  # Note: we could also use 'past' with GPT-2/Transfo-XL/XLNet/CTRL (cached hidden-states)
            next_token_logits = outputs[0][0, -1, :] / (temperature if temperature > 0 else 1.0)

            # reptition penalty from CTRL (https://arxiv.org/abs/1909.05858)
            for _ in set(generated.view(-1).tolist()):
                next_token_logits[_] /= repetition_penalty

            filtered_logits = top_k_top_p_filtering(next_token_logits, top_k=top_k, top_p=top_p)
            if temperature == 0:  # greedy sampling:
                next_token = torch.argmax(filtered_logits).unsqueeze(0)
            else:
                next_token = torch.multinomial(F.softmax(filtered_logits, dim=-1), num_samples=1)
            generated = torch.cat((generated, next_token.unsqueeze(0)), dim=1)
    return generated


title: How to use smac for hyper-parameter optimization of Convolution Neural Network?

body:
Note: Long Post. Please bear with me

I have implemented a convolution neural network in PyTorch on KMNIST dataset. I need to use SMAC to optimize the learning rate and the momentum of Stochastic Gradient Descent of the CNN. I am new in hyperparameter optimization and what I learnt from the smac documentation is,

SMAC evaluates the algorithm to be optimized by invoking it through a Target Algorithm Evaluator (TAE).
We need a Scenario-object to configure the optimization process.
run_obj parameter in Scenario object specifies what SMAC is supposed to optimize.

My Ultimate goal is to get a good accuracy or low loss

This is what I have done so far:

Convolution Neural Network

import numpy as np
import torch
import torch.nn as nn
import torchvision
import torchvision.transforms as transforms
import torchvision.datasets as datasets
from torch.autograd import Variable
from datasets import *
import torch.utils.data
import torch.nn.functional as F
import matplotlib.pyplot as plt

# Create the model class

class CNN(nn.Module):
    def __init__(self):

        super(CNN, self).__init__() # to inherent the features of nn.Module

        self.cnn1 = nn.Conv2d(in_channels = 1, out_channels = 8, kernel_size = 3, stride = 1, padding =1)

        # in_channels =1 because of grey scale image
        # kernel_size = feature_size
        # padding = 1 because for same padding = [(filter_size -1)/2]
        # the output size of the 8 feature maps is [(input_size - filter_size +2(padding)/stride)+1]

        #Batch Normalization

        self.batchnorm1 = nn.BatchNorm2d(8)

        # RELU

        self.relu = nn.ReLU()
        self.maxpool1 = nn.MaxPool2d(kernel_size =2)

        # After maxpooling, the output of each feature map is 28/2 =14

        self.cnn2 = nn.Conv2d(in_channels = 8, out_channels = 32, kernel_size = 5, stride = 1, padding =2)

        #Batch Normalization

        self.batchnorm2 = nn.BatchNorm2d(32)

        # RELU

        #self.relu = nn.ReLU()
        self.maxpool2 = nn.MaxPool2d(kernel_size =2)

        # After maxpooling , the output of each feature map is 14/2 =7of them is of size 7x7 --> 32*7*7=1568
        # Flatten the feature maps. You have 32 feature maps, each
        self.fc1 = nn.Linear(in_features=1568, out_features = 600)
        self.dropout = nn.Dropout(p=0.5)
        self.fc2 = nn.Linear(in_features=600, out_features = 10)

    def forward(self,x):

        out = self.cnn1(x)
        #out = F.relu(self.cnn1(x))
        out = self.batchnorm1(out)
        out = self.relu(out)
        out = self.maxpool1(out)

        out = self.cnn2(out)
        out = self.batchnorm2(out)
        out = self.relu(out)
        out = self.maxpool2(out)

        #Now we have to flatten the output. This is where we apply the feed forward neural network as learned
        #before!

        #It will the take the shape (batch_size, 1568) = (100, 1568)

        out = out.view(-1, 1568)

        #Then we forward through our fully connected layer

        out = self.fc1(out)
        out = self.relu(out)
        out = self.dropout(out)
        out = self.fc2(out)

        return out

def train(model, train_loader, optimizer, epoch, CUDA, loss_fn):
        model.train()
        cum_loss=0
        iter_count = 0

        for i, (images, labels) in enumerate(train_load):

            if CUDA:

               images = Variable(images.cuda())
               images = images.unsqueeze(1)
               images = images.type(torch.FloatTensor)
               images = images.cuda()

               labels = Variable(labels.cuda())
               labels = labels.type(torch.LongTensor)
               labels = labels.cuda()

            else:

               images = Variable(images)
               images = images.unsqueeze(1)
               images = images.type(torch.DoubleTensor)

               labels = Variable(labels)
               labels = labels.type(torch.DoubleTensor)

            optimizer.zero_grad()
            outputs = model(images)
            loss = loss_fn(outputs, labels)
            loss.backward()
            optimizer.step()
            cum_loss += loss

            if (i+1) % batch_size == 0:
               correct = 0
               total = 0
               acc = 0
               _, predicted = torch.max(outputs.data,1)
               total += labels.size(0)
               if CUDA:
                  correct += (predicted.cpu()==labels.cpu()).sum()
               else:
                  correct += (predicted==labels).sum()

               accuracy = 100*correct/total

            if i % len(train_load) == 0:

               iter_count += 1
               ave_loss = cum_loss/batch_size
        return ave_loss

batch_size = 100
epochs = 5
e = range(epochs)
#print(e)

#Load datasets

variable_name=KMNIST()

train_images = variable_name.images
train_images = torch.from_numpy(train_images)

#print(train_images.shape)
#print(type(train_images))

train_labels = variable_name.labels
train_labels = torch.from_numpy(train_labels)

#print(train_labels.shape)
#print(type(train_labels))

train_dataset = torch.utils.data.TensorDataset(train_images, train_labels)

# Make the dataset iterable

train_load = torch.utils.data.DataLoader(dataset = train_dataset, batch_size = batch_size, shuffle = True)

print('There are {} images in the training set' .format(len(train_dataset)))
print('There are {} images in the loaded training set' .format(len(train_load)))

def net(learning_rate, Momentum):
    model = CNN()
    CUDA = torch.cuda.is_available()
    if CUDA:
        model = model.cuda()

    loss_fn = nn.CrossEntropyLoss()
    optimizer = torch.optim.SGD(model.parameters(), lr = learning_rate,momentum = Momentum, nesterov= True)

    iteration = 0
    total_loss=[]

    for epoch in range(epochs):
        ave_loss = train(model, train_load, optimizer, epoch, CUDA, loss_fn)

        total_loss.append(ave_loss)

    return optimizer, loss_fn, model, total_loss

optimizer, loss_fn, model, total_loss = net(learning_rate= 0.01, Momentum = 0.09)

# Print model's state_dict

print("---------------")

print("Model's state_dict:")

for param_tensor in model.state_dict():
    print(param_tensor, "\t", model.state_dict()[param_tensor].size())

print("---------------")

#print("Optimizer's state_dict:")

#for var_name in optimizer.state_dict():
 #   print(var_name, "\t", optimizer.state_dict()[var_name])

torch.save(model.state_dict(), "kmnist_cnn.pt")

plt.plot(e, (np.array(total_loss)))
plt.xlabel("# Epoch")
plt.ylabel("Loss")
plt.show()

print('Done!')

smac hyperparameter optimization:

from smac.configspace import ConfigurationSpace
from ConfigSpace.hyperparameters import CategoricalHyperparameter, \
    UniformFloatHyperparameter, UniformIntegerHyperparameter

from smac.configspace.util import convert_configurations_to_array
#from ConfigSpace.conditions import InCondition

# Import SMAC-utilities
from smac.tae.execute_func import ExecuteTAFuncDict
from smac.scenario.scenario import Scenario
from smac.facade.smac_facade import SMAC

# Build Configuration Space which defines all parameters and their ranges
cs = ConfigurationSpace()

# We define a few possible types of SVM-kernels and add them as "kernel" to our cs

lr = UniformFloatHyperparameter('learning_rate', 1e-4, 1e-1, default_value='1e-2')
momentum = UniformFloatHyperparameter('Momentum', 0.01, 0.1, default_value='0.09')

cs.add_hyperparameters([lr, momentum])

def kmnist_from_cfg(cfg):

    cfg = {k : cfg[k] for k in cfg if cfg[k]}
    print('Config is', cfg)

    #optimizer, loss_fn, model, total_loss = net(**cfg)
    #optimizer, loss_fn, model, total_loss = net(learning_rate= cfg["learning_rate"], Momentum= cfg["Momentum"])

    optimizer, loss_fn, model, total_loss = net(learning_rate= 0.02, Momentum= 0.05)

    return optimizer, loss_fn, model, total_loss

# Scenario object
scenario = Scenario({"run_obj": "quality",   # we optimize quality (alternatively runtime)
                     "runcount-limit": 200,  # maximum function evaluations
                     "cs": cs,               # configuration space
                     "deterministic": "true"
                     })

#def_value = kmnist_from_cfg(cs.get_default_configuration())
#print("Default Value: %.2f" % (def_value))

# Optimize, using a SMAC-object

print("Optimizing! Depending on your machine, this might take a few minutes.")
smac = SMAC(scenario=scenario,tae_runner=kmnist_from_cfg) #rng=np.random.RandomState(42)
smac.solver.intensifier.tae_runner.use_pynisher = False

print("SMAC", smac)
incumbent = smac.optimize()

inc_value = kmnist_from_cfg(incumbent)

print("Optimized Value: %.2f" % (inc_value))

When I give loss as the run_obj parameter, I get the error message

  ArgumentError: argument --run-obj/--run_obj: invalid choice: 'total_loss' (choose from 'runtime', 'quality')

To be honest, I do not know what does "quality" means. Anyways, when I give quality as the run_obj parameter, I get the error message

  TypeError: ufunc 'isfinite' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''

If I understood it correctly, the above error message is obtained when an int is expected but str is given. To check whether the problem was with configuration space, I tried

optimizer, loss_fn, model, total_loss = net(learning_rate= 0.02, Momentum= 0.05)

instead of these:

optimizer, loss_fn, model, total_loss = net(**cfg)
optimizer, loss_fn, model, total_loss = net(learning_rate= cfg["learning_rate"], Momentum= cfg["Momentum"])

the error remains the same.

Any ideas on how to use smac to optimize hyperparameters of CNN and why do I get this error message? I tried looking for similar problems online. This post was a little helpful. Unfortunately, since there is no implementation of smac on NN (at least I did not find it), I cannot figure out the solution. I ran out of all ideas.

Any help, ideas or useful link is appreciated.

Thank you!


title: Explanation of CPU and GPU time with torch autograd profiler in CUDA mode

body:
I am trying to profile a network with torch.autograd.profiler and I need some explanation regarding the CPU and GPU time reported. I assume that the timings are nearly equal because CPU time includes the time the kernel launch + execution. However I see inconsistency when trying to find a relationship between CPU and GPU times.

As shown below some ops report approximately same time , some ops report CPU time larger than GPU time and some have the CPU time smaller than GPU time. Could someone please explain the difference in time.

OpName CPUTime GPUTime
relu 14.700us 15.936us
sub  112.447us 93.504us
mm 43.501us  46.912us
CatBackward  84.912us 84.704us

Thanks


title: Stopping criteria/rule for ADAM optimization in pytorch?

body:
In the code below, we define two functions, and then do some optimization using adam and pytorch. The code seems to work. However, we do a pre-defined number of iterations for the adam optimization (see the while loop). How can we put a stopping criteria based on the output of the optimization? An example for such criteria we are trying to use is that the improvement of the objective function is smaller than a given threshold or anything similar.

import torch
import torch.optim as optim

def myFirstFunction(parameter_current_here):
    optimalValue=100000000000000
    for j in range(2,10):
        i= torch.ones(1,requires_grad=True)*j
        optimalValueNow=i*parameter_current_here.sum()
        if (optimalValueNow<optimalValue):
            optimalValue=optimalValueNow

    return optimalValueNow,i

def mySecondFunction(Current):
    y=(20*Current)/2 + (Current**2)/10
    return y

parameter_current = torch.randn(2, 2,requires_grad=True)
optimizer = optim.Adam([parameter_current], lr = 0.1)
counter=0
while counter<5:
    optimizer.zero_grad()
    outputMyFirstFunction=myFirstFunction(parameter_current)
    outputmySecondFunction=mySecondFunction(outputMyFirstFunction[0])
    outputmySecondFunction.backward()
    optimizer.step()
    print("outputMyFirstFunction after backward:",outputMyFirstFunction)
    print("outputmySecondFunction after backward:",outputmySecondFunction)
    print("parameter_current Gradient after backward:",parameter_current.grad)

    counter=counter+1

Your help will be greatly appreciate.
Thanks,
Aly


title: Append 1D tensors iteratively to make 2D tensor

body:
I have this tensor that's of length 64 and I'm trying to append to the 1st dimension to make a 2D tensor recursively. Essentially each iteration it creates a new 1D 64 length tensor and it iterates 6 times so by the end I should have a 6 x 64 tensor. How do I do this?

I tried

names_tensor = torch.cat((names_tensor, sampled_indexes), dim=1)

Where name_tensor is initiated as torch.zeros(0) and sampled_indexes is the 64 length tensor that gets appended each iteration


title: what is the difference between if-else statement and torch.where in pytorch?

body:
See the code snippet:

import torch
x = torch.tensor([-1.], requires_grad=True)
y = torch.where(x > 0., x, torch.tensor([2.], requires_grad=True))
y.backward()
print(x.grad)

The output is tensor([0.]), but

import torch
x = torch.tensor([-1.], requires_grad=True)
if x > 0.:
    y = x
else:
    y = torch.tensor([2.], requires_grad=True)
y.backward()
print(x.grad)

The output is None.

I'm confused that why the output of torch.where is tensor([0.])?

update

import torch
a = torch.tensor([[1,2.], [3., 4]])
b = torch.tensor([-1., -1], requires_grad=True)
a[:,0] = b

(a[0, 0] * a[0, 1]).backward()
print(b.grad)

The output is tensor([2., 0.]). (a[0, 0] * a[0, 1]) is not in any way related to b[1], but the gradient of b[1] is 0 not None.


title: Cannot find in-place operation causing "RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation:"

body:
I'm relatively new to PyTorch and am trying to reproduce an algorithm from an academic paper that approximates a term using the Hessian matrix.  I've set up a toy problem so that I can compare the results of the full Hessian with the approximation.  I found this gist and have been playing with it to compute the full Hessian part of the algorithm.

I am getting the error: "RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation."

I've scoured through the simple example code, documentation, and many, many forum posts about this issue and cannot find any in-place operations.  Any help would be greatly appreciated!

Here is my code:

import torch
import torch.autograd as autograd
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import numpy as np

torch.set_printoptions(precision=20, linewidth=180)

def jacobian(y, x, create_graph=False):
    jac = []
    flat_y = y.reshape(-1)
    grad_y = torch.zeros_like(flat_y)

    for i in range(len(flat_y)):
        grad_y[i] = 1.
        grad_x, = torch.autograd.grad(flat_y, x, grad_y, retain_graph=True, create_graph=create_graph)
        jac.append(grad_x.reshape(x.shape))
        grad_y[i] = 0.
    return torch.stack(jac).reshape(y.shape + x.shape)

def hessian(y, x):
    return jacobian(jacobian(y, x, create_graph=True), x)

def f(x):
    return x * x

np.random.seed(435537698)

num_dims = 2
num_samples = 3

X = [np.random.uniform(size=num_dims) for i in range(num_samples)]
print('X: \n{}\n\n'.format(X))

mean = torch.Tensor(np.mean(X, axis=0))
mean.requires_grad = True
print('mean: \n{}\n\n'.format(mean))

cov = torch.Tensor(np.cov(X, rowvar=False))
print('cov: \n{}\n\n'.format(cov))

with autograd.detect_anomaly():
    hessian_matrices = hessian(f(mean), mean)
    print('hessian: \n{}\n\n'.format(hessian_matrices))

And here is the output with the stack trace:

X:
[array([0.81700949, 0.17141617]), array([0.53579366, 0.31141496]), array([0.49756485, 0.97495776])]

mean:
tensor([0.61678934097290039062, 0.48592963814735412598], requires_grad=True)

cov:
tensor([[ 0.03043144382536411285, -0.05357056483626365662],
        [-0.05357056483626365662,  0.18426130712032318115]])

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-3-5a1c492d2873> in <module>()
     42
     43 with autograd.detect_anomaly():
---> 44     hessian_matrices = hessian(f(mean), mean)
     45     print('hessian: \n{}\n\n'.format(hessian_matrices))

2 frames
<ipython-input-3-5a1c492d2873> in hessian(y, x)
     21
     22 def hessian(y, x):
---> 23     return jacobian(jacobian(y, x, create_graph=True), x)
     24
     25 def f(x):

<ipython-input-3-5a1c492d2873> in jacobian(y, x, create_graph)
     15     for i in range(len(flat_y)):
     16         grad_y[i] = 1.
---> 17         grad_x, = torch.autograd.grad(flat_y, x, grad_y, retain_graph=True, create_graph=create_graph)
     18         jac.append(grad_x.reshape(x.shape))
     19         grad_y[i] = 0.

/usr/local/lib/python3.6/dist-packages/torch/autograd/__init__.py in grad(outputs, inputs, grad_outputs, retain_graph, create_graph, only_inputs, allow_unused)
    155     return Variable._execution_engine.run_backward(
    156         outputs, grad_outputs, retain_graph, create_graph,
--> 157         inputs, allow_unused)
    158
    159

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [2]] is at version 4; expected version 3 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!


title: GPU out of memory on evaluation : Pytorch

body:
The model trains fine when I only train and don't validate, however it runs out of memory during evaluation, but I don't understand why this might be a problem especially since I am using torch.no_grad() any ideas?
def test(epoch,net,testloader,optimizer):
    net.eval()
    test_loss = 0
    correct = 0
    total = 0
    idx = 0
    features_all = []
    for batch_idx, (inputs, targets) in enumerate(testloader):
        with torch.no_grad():
            idx = batch_idx
            # inputs, targets = inputs.cpu(), targets.cpu()
            if use_cuda:
                inputs, targets = inputs.cuda(), targets.cuda()
            inputs, targets = Variable(inputs), Variable(targets)
            save_features, out, ce_loss = net(inputs,targets)
            test_loss += ce_loss.item()
            _, predicted = torch.max(out.data, 1)
            total += targets.size(0)
            correct += predicted.eq(targets.data).cpu().sum().item()
            features_all.append((save_features, predicted, targets.data))
    test_acc = 100.*correct/total
    test_loss = test_loss/(idx+1)
    logging.info('test, test_acc = %.4f,test_loss = %.4f' % (test_acc,test_loss))
    print('test, test_acc = %.4f,test_loss = %.4f' % (test_acc,test_loss))
    return features_all, test_acc


title: How to add a multiclass multilabel layer on top of pretrained BERT model?

body:
I am trying to do a multitask multiclass sentence classification task using the pretrained BERT model from the huggingface transformers library . I have tried to use the BERTForSequenceClassification model from there but the issue I am having is that I am not able to extend it for multiple tasks . I will try to make it more informative through this example.
Suppose we have four different tasks and for each sentence and for each task we have labels like this as follows in the examples:

A :[ 'a' , 'b' , 'c' , 'd' ]
B :[ 'e' , 'f' , 'g' , 'h' ]
C :[ 'i' , 'j' , 'k' , 'l' ]
D :[ 'm' , 'n' , 'o' , 'p' ]

Now , if I have a sentence for this model , I want the output to give me output for all the four different tasks (A,B,C,D).
This is what I was doing earlier
   model = BertForSequenceClassification.from_pretrained(
    "bert-base-uncased", # Use the 12-layer BERT model, with an uncased vocab.
    num_labels = 4, # The number of output labels--2 for binary classification.
                    # You can increase this for multi-class tasks.
    output_attentions = False, # Whether the model returns attentions weights.
    output_hidden_states = False, # Whether the model returns all hidden-states.
)

Then I tried to implement a CustomBERT model like this :
class CustomBERTModel(nn.Module):
    def __init__(self):
          super(CustomBERTModel, self).__init__()
          self.bert = BertModelForSequenceClassification.from_pretrained("bert-base-uncased")
          ### New layers:
          self.linear1 = nn.Linear(768, 256)
          self.linear2 = nn.Linear(256, num_classes) ## num_classes is the number of classes in this example

    def forward(self, ids, mask):
          sequence_output, pooled_output = self.bert(
               ids,
               attention_mask=mask)

          # sequence_output has the following shape: (batch_size, sequence_length, 768)
          linear1_output = self.linear1(sequence_output[:,0,:].view(-1,768))
          linear2_output = self.linear2(linear2_output)

          return linear2_output

I have went through the answers to questions similar to it available earlier but none of them appeared to
answer my question . I have tried to get through all the points which I think can be helpful for the understanding of my problem and would try to clear further more in case of any descrepancies made by me in the explaination of the question . Any answers related to this will be very much helpful .


title: Neural network in pytorch predict two binary variables

body:
Suppose I want to have the general neural network architecture:

                     ---> BinaryOutput_A
                    /
Input --> nnLayer -
                    \
                     ---> BinaryOutput_B

An input is put through a neural network layer which then goes to predict two binary variables (i.e., A is [0 or 1] and B is [0 or 1].

In pytorch, you can make such a network with:

class NN(nn.Module):

    def __init__(self, inputs):
        super(NN, self).__init__()

        # -- first layer
        self.lin = nn.Linear(inputs,10)

        # -- firstLayer --> binaryOutputA
        self.l2a = nn.Linear(10,2)

        # -- firstLayer --> binaryOutputB
        self.l2b = nn.Linear(10,2)

    def forward(self, inputs):
        o = self.lin(inputs)
        o1 = F.log_softmax(self.l2a(o))
        o2 = F.log_softmax(self.l2b(o))
        return o1, o2

In my train function, I calculate the loss with loss = loss_function(output, target).  If that's the case, to properly backpropagate the loss to both the l2a and l2b layers using loss.backward(), could I simply concat the target with the proper label for l2a and l2b?  In that sense, the output would be [outputPredictionA, outputPredictionB] and I could make the target be [labelA, labelB], Would pytorch know to properly assign the loss to each layer?


title: Cuda out of memory despite consuming only 60% memory

body:
I'm getting cuda out of memory error. The error is shown below

RuntimeError: CUDA out of memory. Tried to allocate 256.00 MiB (GPU 0;
23.65 GiB total capacity; 21.65 GiB already allocated; 242.88 MiB free; 22.55 GiB reserved in total by PyTorch)

I'm able to train the model after I reduce the batch_size. When I checked the output of nvidia-smi, I see a 40% of memory is still free. Here is the output.

What could be possible reason for this?
pytorch: 1.4
cuda: 10.2
input_size: (512, 512, 4)
using half-precision
More information:
The plot of gpu-utilization is shown below

The numbers at each peak represent the batch_size. It seems the initial memory requirement is much higher than memory needed afterward. Can someone explain?


title: matrix multiplication for complex numbers in PyTorch

body:
I am trying to multiply two complex matrices in PyTorch and it seems the torch.matmul functions is not added yet to PyTorch library for complex numbers.
Do you have any recommendation or is there another method to multiply complex matrices in PyTorch?


title: Pytorch .backward() method without CUDA

body:
I am trying to run the code in the Pytorch tutorial on the autograd module. However, when I run the .backwards() call, I get the error:

cuda runtime error (38) : no CUDA-capable device is detected at torch/csrc/autograd/engine.cpp:359

I admittedly have no CUDA-capable device set up at the moment, but it was my understanding that this wasn't strictly necessary (at least I didn't find it specified anywhere in the tutorial). So I was wondering if there is a way to still run the code without a CUDA-enabled GPU.


title: Compute the number of epoch from iteration in training?

body:
I have a Caffe prototxt as follows:

stepsize: 20000
iter_size: 4
batch_size: 10
gamma =0.1

in which, the dataset has 40.000 images. It means after 20000 iters, the learning rate will decrease 10 times. In pytorch, I want to compute the number of the epoch to have the same behavior in caffe (for learning rate). How many epoch should I use to decrease learning rate 10 times (note that, we have iter_size=4 and batch_size=10). Thanks

Ref: Epoch vs Iteration when training neural networks

My answer: Example: if you have 40000 training examples, and batch size is 10, then it will take 40000/10 =4000 iterations to complete 1 epoch. Hence, 20000 iters to reduce learning rate in caffe will same as 5 epochs in pytorch.


title: Use Tensorflow/PyTorch to speed up minimisation of a custom function

body:
I do a lot of simulations, for which I often need to minimise complicated user-defined functions, for which I generally use numpy and scipy.optimize.minimize(). However, the problem with this is that I need to explicitly write down a gradient function, which can sometimes be very difficult/impossible, to find. And for large dimensional vectors, the numerical derivatives calculated by scipy are prohibitively expensive.

So, I am trying to switch to Tensorflow or PyTorch to take advantage both of their automatic differentiation capabilities, and to be able to exploit GPUs freely. Let me give an explicit example of a function who derivative is somewhat complicated to write down (would required a lot of chain rule), and thus seems ripe for Tensorflow or PyTorch -- calculating the dihedral angle between the two triangles formed by four points in 3d space:

def dihedralAngle(xyz):
## calculate dihedral angle between 4 nodes

    p1, p2, p3, p4 = 0, 1, 2, 3

    ## get unit normal vectors
    N1 = np.cross(xyz[p1]-xyz[p3] , xyz[p2]-xyz[p3])
    N2 = - np.cross(xyz[p1]-xyz[p4] , xyz[p2]-xyz[p4])
    n1, n2 = N1 / np.linalg.norm(N1), N2 / np.linalg.norm(N2)

    angle = np.arccos(np.dot(n1, n2))

    return angle

xyz1 = np.array([[0.2       , 0.        , 0.        ],
       [0.198358  , 0.02557543, 0.        ],
       [0.19345897, 0.05073092, 0.        ],
       [0.18538335, 0.0750534 , 0.        ]]) # or any (4,3) ndarray

print(dihedralAngle(xyz1)) >> 3.141

I could easily minimise this using scipy.optimize.minimize(), and I should get 0. For such a small function, I don't really the need a gradient (explicit or numerical). However, if I wish to iterate over many, many nodes, and minimise some function that depends on all the dihedral angles, then the overhead is much higher?

My questions then --

How would I implement this minimisation problem using TensorFlow or PyTorch? Both for a single dihedral angle, and for a list of such angles (i.e. we need to account for looping over lists).
Also, could I just get the gradient using automatic differentiation, to plug back into scipy.optimize.minimize() if desired? For example, scipy.optimize.minimize() allows easily for bounds and constraints, something that I haven't noticed in the Tensorflow or PyToch optimisation modules.


title: How to deploy a deep learning model in a windows 32bit machine?

body:
I have trained a crnn model for ocr with tensorflow. Then I deployed it with tensorflow c++ API on a windows 64 bit machine and it works well.
Now I want to deploy the model on a windows 32 bit machine , but I notice that most most deep learning frameworks like tensorflow and pytorch don't support 32 bit machine.
Then I tried opencv dnn module and ncnn framework, but they both don't support the squeeze and unsqueeze operation which are used in my model.
Now I have no idea, could any one give me some suggestions?
Thanks.


title: ImportError when importing pytorch

body:
When I try importing pytorch inside a jupyter notebook i get the following error:

  ImportError: dlopen: cannot load any more object with static TLS
  pytorch

import torch

No error when i import torch from command line (not jupyter notebook)


title: After Deleting some Python Files, I can't install pytorch any more via pip

body:
Unfortunately I deleted some files of Python. I got macOS Catalina and I want to install pytorch with the command:

pip3 install torch

If I enter this in my terminal I'll get

  Building wheel for torch (setup.py) ... error
    ERROR: Command errored out with exit status 1:
     command: /Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/sb/4h99w30940zcbdvy_csrksxc0000gn/T/pip-install-kb_zrdjk/torch/setup.py'"'"'; file='"'"'/private/var/folders/sb/4h99w30940zcbdvy_csrksxc0000gn/T/pip-install-kb_zrdjk/torch/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' bdist_wheel -d /private/var/folders/sb/4h99w30940zcbdvy_csrksxc0000gn/T/pip-wheel-qh35w11o --python-tag cp38
         cwd: /private/var/folders/sb/4h99w30940zcbdvy_csrksxc0000gn/T/pip-install-kb_zrdjk/torch/
    Complete output (30 lines):
    running bdist_wheel
    running build
    running build_deps
    Traceback (most recent call last):
      File "", line 1, in
      File "/private/var/folders/sb/4h99w30940zcbdvy_csrksxc0000gn/T/pip-install-kb_zrdjk/torch/setup.py", line 225, in
        setup(name="torch", version="0.1.2.post2",
      File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/init.py", line 145, in setup
        return distutils.core.setup(**attrs)
      File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/dist.py", line 966, in run_commands
        self.run_command(cmd)
      File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 192, in run
        self.run_command('build')
      File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/command/build.py", line 135, in run
        self.run_command(cmd_name)
      File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/dist.py", line 985, in run_command
        cmd_obj.run()
      File "/private/var/folders/sb/4h99w30940zcbdvy_csrksxc0000gn/T/pip-install-kb_zrdjk/torch/setup.py", line 51, in run
        from tools.nnwrap import generate_wrappers as generate_nn_wrappers
    ModuleNotFoundError: No module named 'tools.nnwrap'

  ERROR: Failed building wheel for torch
    Running setup.py clean for torch
    ERROR: Command errored out with exit status 1:
     command: /Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/sb/4h99w30940zcbdvy_csrksxc0000gn/T/pip-install-kb_zrdjk/torch/setup.py'"'"'; file='"'"'/private/var/folders/sb/4h99w30940zcbdvy_csrksxc0000gn/T/pip-install-kb_zrdjk/torch/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' clean --all
         cwd: /private/var/folders/sb/4h99w30940zcbdvy_csrksxc0000gn/T/pip-install-kb_zrdjk/torch
    Complete output (2 lines):
    running clean
    error: [Errno 2] No such file or directory: '.gitignore'

  ERROR: Failed cleaning build dir for torch
  Failed to build torch
  Installing collected packages: torch
      Running setup.py install for torch ... error
      ERROR: Command errored out with exit status 1:
       command: /Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/sb/4h99w30940zcbdvy_csrksxc0000gn/T/pip-install-kb_zrdjk/torch/setup.py'"'"'; file='"'"'/private/var/folders/sb/4h99w30940zcbdvy_csrksxc0000gn/T/pip-install-kb_zrdjk/torch/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /private/var/folders/sb/4h99w30940zcbdvy_csrksxc0000gn/T/pip-record-cerpeh7h/install-record.txt --single-version-externally-managed --compile
           cwd: /private/var/folders/sb/4h99w30940zcbdvy_csrksxc0000gn/T/pip-install-kb_zrdjk/torch/
      Complete output (23 lines):
      running install
      running build_deps
      Traceback (most recent call last):
        File "", line 1, in
        File "/private/var/folders/sb/4h99w30940zcbdvy_csrksxc0000gn/T/pip-install-kb_zrdjk/torch/setup.py", line 225, in
          setup(name="torch", version="0.1.2.post2",
        File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/setuptools/init.py", line 145, in setup
          return distutils.core.setup(**attrs)
        File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/core.py", line 148, in setup
          dist.run_commands()
        File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/dist.py", line 966, in run_commands
          self.run_command(cmd)
        File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/private/var/folders/sb/4h99w30940zcbdvy_csrksxc0000gn/T/pip-install-kb_zrdjk/torch/setup.py", line 99, in run
          self.run_command('build_deps')
        File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/private/var/folders/sb/4h99w30940zcbdvy_csrksxc0000gn/T/pip-install-kb_zrdjk/torch/setup.py", line 51, in run
          from tools.nnwrap import generate_wrappers as generate_nn_wrappers
      ModuleNotFoundError: No module named 'tools.nnwrap'
      ----------------------------------------
  ERROR: Command errored out with exit status 1: /Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/sb/4h99w30940zcbdvy_csrksxc0000gn/T/pip-install-kb_zrdjk/torch/setup.py'"'"'; file='"'"'/private/var/folders/sb/4h99w30940zcbdvy_csrksxc0000gn/T/pip-install-kb_zrdjk/torch/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /private/var/folders/sb/4h99w30940zcbdvy_csrksxc0000gn/T/pip-record-cerpeh7h/install-record.txt --single-version-externally-managed --compile Check the logs for full command output.

Is there a way to solve this problem? Do you need further Information?

UPDATE

I changed the way of the enviroment (without anaconda) and so I get this error message:

  ERROR: Command errored out with exit status 1: /Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/sb/4h99w30940zcbdvy_csrksxc0000gn/T/pip-install-zelcwulm/torch/setup.py'"'"'; file='"'"'/private/var/folders/sb/4h99w30940zcbdvy_csrksxc0000gn/T/pip-install-zelcwulm/torch/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /private/var/folders/sb/4h99w30940zcbdvy_csrksxc0000gn/T/pip-record-i1_7akf_/install-record.txt --single-version-externally-managed --compile Check the logs for full command output.


title: Max function in PyTorch versus in NumPy

body:
I am developing an algorithm that involves a CPU case, allowing for NumPy, and a GPU case, allowing for PyTorch. The object will almost always be 4D. Two versions of the object are as follows.
B = [
    [[[0.5000, 0.5625],
      [0.5000, 0.5625]],

     [[1.2500, 0.5000],
      [0.5625, 0.6875]],

     [[0.5625, 0.6250],
      [0.5000, 0.5625]]]
]

B_array = np.array(B)

B_tensor = torch.Tensor(B)

I want to take the max of each 2D matrix, such that I get a result of:
max_array_fn(B_array)   # returns array([0.5625, 1.250, 0.6250])
max_tensor_fn(B_tensor) # returns tensor([0.5625, 1.250, 0.6250])

Part of the solution was discussed here, but this is only for NumPy on CPU:
Max of each 2D matrix in 4D NumPy array
However, on the GPU, it seems that PyTorch does not use the same convention as NumPy.
If this was defined as a NumPy array, we could solve the problem using np.array(B, axis=(0,2,3)). The usage of axis in PyTorch similar to the NumPy example is not supported, as suggested here:
PyTorch torch.max over multiple dimensions
Is there an alternative vectorization of the method? Why is it only able to be vectorized on the CPU with NumPy and not on GPU with PyTorch?
The correct solution should not use any loops and ideally one function call.


title: Display image in a PIL format from torch.Tensor

body:
I’m quite new to Pytorch. I was wondering how I could convert my tensor of size torch.Size([1, 3, 224, 224]) to display in an image format on a Jupyter notebook. A PIL format or a CV2 format should be fine.

I tried using transforms.ToPILImage(x) but it resulted in a different format like this: ToPILImage(mode=ToPILImage(mode=tensor([[[[1.3034e-16, 1.3034e-16, 1.3034e-16, ..., 1.4475e-16,.

Maybe I’m doing something wrong :no_mouth:


title: Word embeddings with BERT and map tensors to words

body:
I try to aggregate BERT embeddings on the token level. For each token in the corpus vocabulary, I would like to create a list of all their contextual embeddings and average them to get one representation for each token in the vocabulary.
The code is pasted below.
Question: How to map output tensors (see object token_vecs_sum in the last line of the code below) to particular tokens?
Preprocess data
!pip install transformers
import torch
from transformers import BertTokenizer
from nltk import tokenize
import nltk
nltk.download('punkt')
import re

MAX_LEN = 64

sentences = ['Some sentences. Some sentences are.', 'Some sentences are really.', 'Some sentences are really hard.']

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True)

def preprocessing_for_bert(data):
  input_ids = []
  attention_masks = []
  for row in data:
    sents = tokenize.sent_tokenize(row)
    print(sents)
    for sent in sents:
      encoded_sent = tokenizer.encode_plus(text=sent,
                                          add_special_tokens=True,
                                          max_length=MAX_LEN,
                                          pad_to_max_length=True,
                                          return_attention_mask=True,
                                          truncation=True)
      input_ids.append(encoded_sent.get('input_ids'))
      attention_masks.append(encoded_sent.get('attention_mask'))
  # Convert lists to tensors
  input_ids = torch.tensor(input_ids)
  attention_masks = torch.tensor(attention_masks)

  return input_ids, attention_masks

tokens_tensor, segments_tensor = preprocessing_for_bert(sentences)

Load pretrained model
import torch
from transformers import BertTokenizer, BertModel

# Load pre-trained model (weights)
model = BertModel.from_pretrained('bert-base-uncased', output_hidden_states = True)

# Put the model in "evaluation" mode, meaning feed-forward operation.
model.eval()

Run through BERT
# Run the text through BERT, and collect all of the hidden states produced
# from all 12 layers.
with torch.no_grad():

    outputs = model(tokens_tensor, segments_tensor)

    # Evaluating the model will return a different number of objects based on
    # how it's  configured in the `from_pretrained` call earlier. In this case,
    # becase we set `output_hidden_states = True`, the third item will be the
    # hidden states from all layers.
    hidden_states = outputs[2]

The output results
print ("Number of layers:", len(hidden_states), "  (initial embeddings + 12 BERT layers)")

layer_i = 0
print ("Number of batches:", len(hidden_states[layer_i]))

batch_i = 0
print ("Number of tokens:", len(hidden_states[layer_i][batch_i]))

token_i = 0
print ("Number of hidden units:", len(hidden_states[layer_i][batch_i][token_i]))

Number of layers: 13   (initial embeddings + 12 BERT layers)
Number of batches: 4
Number of tokens: 64
Number of hidden units: 768

Aggregation
token_embeddings = torch.stack(hidden_states, dim=0)
# Average over batches
token_embeddings = token_embeddings.mean(1)
token_embeddings = token_embeddings.permute(1,0,2)
token_embeddings.size()
## -> torch.Size([64, 13, 768])

Prepare token embedding matrix
token_vecs_sum = []
# For each token in the sentence...
for token in token_embeddings:
  # `token` is a [13 x 768] tensor
  # Sum the vectors from the last four layers.
  sum_vec = torch.sum(token[-4:], dim=0)
  # Use `sum_vec` to represent `token`.
  token_vecs_sum.append(sum_vec)

print ('Shape is: %d x %d' % (len(token_vecs_sum), len(token_vecs_sum[0])))


title: Issues converting Keras code into PyTorch code (shaping)

body:
I have some keras code that I need to convert to Pytorch. I am new to pytorch and I am having trouble wrapping my head around how to take in input the same way that I did in keras. I have spent many hours on this any tips or help is very appreciated.

Here is the keras code I am dealing with. The input shape is (5000,1)

    def build(input_shape, classes):
        model = Sequential()

        filter_num = ['None',32,64,128,256]
        kernel_size = ['None',8,8,8,8]
        conv_stride_size = ['None',1,1,1,1]
        pool_stride_size = ['None',4,4,4,4]
        pool_size = ['None',8,8,8,8]

        # Block1
        model.add(Conv1D(filters=filter_num[1], kernel_size=kernel_size[1], input_shape=input_shape,
                         strides=conv_stride_size[1], padding='same',
                         name='block1_conv1'))
        model.add(BatchNormalization(axis=-1))
        model.add(ELU(alpha=1.0, name='block1_adv_act1'))
        model.add(Conv1D(filters=filter_num[1], kernel_size=kernel_size[1],
                         strides=conv_stride_size[1], padding='same',
                         name='block1_conv2'))
        model.add(BatchNormalization(axis=-1))
        model.add(ELU(alpha=1.0, name='block1_adv_act2'))
        model.add(MaxPooling1D(pool_size=pool_size[1], strides=pool_stride_size[1],
                               padding='same', name='block1_pool'))
        model.add(Dropout(0.1, name='block1_dropout'))

        # Block 2
        model.add(Conv1D(filters=filter_num[2], kernel_size=kernel_size[2],
                         strides=conv_stride_size[2], padding='same',
                         name='block2_conv1'))
        model.add(BatchNormalization())
        model.add(Activation('relu', name='block2_act1'))

        model.add(Conv1D(filters=filter_num[2], kernel_size=kernel_size[2],
                         strides=conv_stride_size[2], padding='same',
                         name='block2_conv2'))
        model.add(BatchNormalization())
        model.add(Activation('relu', name='block2_act2'))
        model.add(MaxPooling1D(pool_size=pool_size[2], strides=pool_stride_size[3],
                               padding='same', name='block2_pool'))
        model.add(Dropout(0.1, name='block2_dropout'))

        # Block 3
        model.add(Conv1D(filters=filter_num[3], kernel_size=kernel_size[3],
                         strides=conv_stride_size[3], padding='same',
                         name='block3_conv1'))
        model.add(BatchNormalization())
        model.add(Activation('relu', name='block3_act1'))
        model.add(Conv1D(filters=filter_num[3], kernel_size=kernel_size[3],
                         strides=conv_stride_size[3], padding='same',
                         name='block3_conv2'))
        model.add(BatchNormalization())
        model.add(Activation('relu', name='block3_act2'))
        model.add(MaxPooling1D(pool_size=pool_size[3], strides=pool_stride_size[3],
                               padding='same', name='block3_pool'))
        model.add(Dropout(0.1, name='block3_dropout'))

        # Block 4
        model.add(Conv1D(filters=filter_num[4], kernel_size=kernel_size[4],
                         strides=conv_stride_size[4], padding='same',
                         name='block4_conv1'))
        model.add(BatchNormalization())
        model.add(Activation('relu', name='block4_act1'))
        model.add(Conv1D(filters=filter_num[4], kernel_size=kernel_size[4],
                         strides=conv_stride_size[4], padding='same',
                         name='block4_conv2'))
        model.add(BatchNormalization())
        model.add(Activation('relu', name='block4_act2'))
        model.add(MaxPooling1D(pool_size=pool_size[4], strides=pool_stride_size[4],
                               padding='same', name='block4_pool'))
        model.add(Dropout(0.1, name='block4_dropout'))

        # FC #1
        model.add(Flatten(name='flatten'))
        model.add(Dense(512, kernel_initializer=glorot_uniform(seed=0), name='fc1'))
        model.add(BatchNormalization())
        model.add(Activation('relu', name='fc1_act'))

        model.add(Dropout(0.7, name='fc1_dropout'))

        #FC #2
        model.add(Dense(512, kernel_initializer=glorot_uniform(seed=0), name='fc2'))
        model.add(BatchNormalization())
        model.add(Activation('relu', name='fc2_act'))

        model.add(Dropout(0.5, name='fc2_dropout'))

        # Classification
        model.add(Dense(classes, kernel_initializer=glorot_uniform(seed=0), name='fc3'))
        model.add(Activation('softmax', name="softmax"))
        return model

Here are the results of model.summary() from the keras code

Layer (type)                 Output Shape              Param #
=================================================================
block1_conv1 (Conv1D)        (None, 5000, 32)          288
_________________________________________________________________
batch_normalization_1 (Batch (None, 5000, 32)          128
_________________________________________________________________
block1_adv_act1 (ELU)        (None, 5000, 32)          0
_________________________________________________________________
block1_conv2 (Conv1D)        (None, 5000, 32)          8224
_________________________________________________________________
batch_normalization_2 (Batch (None, 5000, 32)          128
_________________________________________________________________
block1_adv_act2 (ELU)        (None, 5000, 32)          0
_________________________________________________________________
block1_pool (MaxPooling1D)   (None, 1250, 32)          0
_________________________________________________________________
block1_dropout (Dropout)     (None, 1250, 32)          0
_________________________________________________________________
block2_conv1 (Conv1D)        (None, 1250, 64)          16448
_________________________________________________________________
batch_normalization_3 (Batch (None, 1250, 64)          256
_________________________________________________________________
block2_act1 (Activation)     (None, 1250, 64)          0
_________________________________________________________________
block2_conv2 (Conv1D)        (None, 1250, 64)          32832
_________________________________________________________________
batch_normalization_4 (Batch (None, 1250, 64)          256
_________________________________________________________________
block2_act2 (Activation)     (None, 1250, 64)          0
_________________________________________________________________
block2_pool (MaxPooling1D)   (None, 313, 64)           0
_________________________________________________________________
block2_dropout (Dropout)     (None, 313, 64)           0
_________________________________________________________________
block3_conv1 (Conv1D)        (None, 313, 128)          65664
_________________________________________________________________
batch_normalization_5 (Batch (None, 313, 128)          512
_________________________________________________________________
block3_act1 (Activation)     (None, 313, 128)          0
_________________________________________________________________
block3_conv2 (Conv1D)        (None, 313, 128)          131200
_________________________________________________________________
batch_normalization_6 (Batch (None, 313, 128)          512
_________________________________________________________________
block3_act2 (Activation)     (None, 313, 128)          0
_________________________________________________________________
block3_pool (MaxPooling1D)   (None, 79, 128)           0
_________________________________________________________________
block3_dropout (Dropout)     (None, 79, 128)           0
_________________________________________________________________
block4_conv1 (Conv1D)        (None, 79, 256)           262400
_________________________________________________________________
batch_normalization_7 (Batch (None, 79, 256)           1024
_________________________________________________________________
block4_act1 (Activation)     (None, 79, 256)           0
_________________________________________________________________
block4_conv2 (Conv1D)        (None, 79, 256)           524544
_________________________________________________________________
batch_normalization_8 (Batch (None, 79, 256)           1024
_________________________________________________________________
block4_act2 (Activation)     (None, 79, 256)           0
_________________________________________________________________
block4_pool (MaxPooling1D)   (None, 20, 256)           0
_________________________________________________________________
block4_dropout (Dropout)     (None, 20, 256)           0
_________________________________________________________________
flatten (Flatten)            (None, 5120)              0
_________________________________________________________________
fc1 (Dense)                  (None, 512)               2621952
_________________________________________________________________
batch_normalization_9 (Batch (None, 512)               2048
_________________________________________________________________
fc1_act (Activation)         (None, 512)               0
_________________________________________________________________
fc1_dropout (Dropout)        (None, 512)               0
_________________________________________________________________
fc2 (Dense)                  (None, 512)               262656
_________________________________________________________________
batch_normalization_10 (Batc (None, 512)               2048
_________________________________________________________________
fc2_act (Activation)         (None, 512)               0
_________________________________________________________________
fc2_dropout (Dropout)        (None, 512)               0
_________________________________________________________________
fc3 (Dense)                  (None, 101)               51813
_________________________________________________________________
softmax (Activation)         (None, 101)               0
=================================================================
Total params: 3,985,957
Trainable params: 3,981,989
Non-trainable params: 3,968

Here is what I have made in pytorch

class model(torch.nn.Module):
    def __init__(self, input_channels, kernel_size, stride, pool_kernel, pool_stride, dropout_p, dropout_inplace=False):
        super(model, self).__init__()
        self.encoder = nn.Sequential(
            BasicBlock1(input_channels, kernel_size, stride, pool_kernel, pool_stride, dropout_p),
            BasicBlock(input_channels//4, kernel_size, stride, pool_kernel, pool_stride, dropout_p),
            BasicBlock(input_channels//16, kernel_size, stride, pool_kernel, pool_stride, dropout_p),
            BasicBlock(input_channels//16//4, kernel_size, stride, pool_kernel, pool_stride, dropout_p)
        )

        self.decoder = nn.Sequential(
            nn.Linear(5120, 512),
            nn.BatchNorm1d(512),
            nn.ReLU(),
            nn.Dropout(p=dropout_p, inplace=dropout_inplace),
            nn.Linear(512, 512),
            nn.BatchNorm1d(512),
            nn.ReLU(),
            nn.Dropout(p=dropout_p, inplace=dropout_inplace),
            nn.Linear(512, 101),
            nn.Softmax(dim=101)
        )
    def forward(self, x):
        x = self.encoder(x)

        x = x.view(x.size(0), -1)  # flatten

        x = self.decoder(x)
        return x

def BasicBlock(input_channels, kernel_size, stride, pool_kernel, pool_stride, dropout_p, dropout_inplace=False):
    return nn.Sequential(
        nn.Conv1d(in_channels=input_channels, out_channels=input_channels, kernel_size=kernel_size, stride=stride,
                  padding=get_pad_size(input_channels, input_channels, kernel_size)),
        nn.BatchNorm1d(32),
        nn.ReLU(),
        nn.Conv1d(in_channels=input_channels, out_channels=input_channels, kernel_size=kernel_size, stride=stride,
                  padding=get_pad_size(input_channels, input_channels, kernel_size)),
        nn.BatchNorm1d(32),
        nn.ReLU(),
        nn.MaxPool1d(kernel_size=pool_kernel, stride=pool_stride,
                     padding=get_pad_size(input_channels, input_channels/4, kernel_size)),
        nn.Dropout(p=dropout_p, inplace=dropout_inplace)
    )

def BasicBlock1(input_channels, kernel_size, stride, pool_kernel, pool_stride, dropout_p, dropout_inplace=False):
    return nn.Sequential(
        nn.Conv1d(in_channels=1, out_channels=input_channels, kernel_size=kernel_size, stride=stride,
                  padding=get_pad_size(input_channels, input_channels, kernel_size)),
        nn.BatchNorm1d(32),
        nn.ReLU(),
        nn.Conv1d(in_channels=input_channels, out_channels=input_channels, kernel_size=kernel_size, stride=stride,
                  padding=get_pad_size(input_channels, input_channels, kernel_size)),
        nn.BatchNorm1d(32),
        nn.ReLU(),
        nn.MaxPool1d(kernel_size=pool_kernel, stride=pool_stride,
                     padding=get_pad_size(input_channels, input_channels/4, kernel_size)),
        nn.Dropout(p=dropout_p, inplace=dropout_inplace)
    )

def get_pad_size(input_shape, output_shape, kernel_size, stride=1, dilation=1):
    """
    Gets the right padded needed to maintain same shape in the conv layers
    BEWARE: works only on odd size kernel size
    :param input_shape: the input shape to the conv layer
    :param output_shape: the desired output shape of the conv layer
    :param kernel_size: the size of the kernel window, has to be odd
    :param stride: Stride of the convolution
    :param dilation: Spacing between kernel elements
    :return: the appropriate pad size for the needed configuration
    :Author: Aneesh
    """

    if kernel_size % 2 == 0:
        raise ValueError(
            "Kernel size has to be odd for this function to work properly. Current Value is %d." % kernel_size)

    return (int((output_shape * stride - stride + kernel_size - input_shape + (kernel_size - 1) * (dilation - 1)) / 2))

Lastly here is the model summary for what my pytorch model creates

model(
  (encoder): Sequential(
    (0): Sequential(
      (0): Conv1d(1, 5000, kernel_size=(7,), stride=(1,), padding=(3,))
      (1): BatchNorm1d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (2): ReLU()
      (3): Conv1d(5000, 5000, kernel_size=(7,), stride=(1,), padding=(3,))
      (4): BatchNorm1d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (5): ReLU()
      (6): MaxPool1d(kernel_size=8, stride=4, padding=-1872, dilation=1, ceil_mode=False)
      (7): Dropout(p=0.1)
    )
    (1): Sequential(
      (0): Conv1d(1250, 1250, kernel_size=(7,), stride=(1,), padding=(3,))
      (1): BatchNorm1d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (2): ReLU()
      (3): Conv1d(1250, 1250, kernel_size=(7,), stride=(1,), padding=(3,))
      (4): BatchNorm1d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (5): ReLU()
      (6): MaxPool1d(kernel_size=8, stride=4, padding=-465, dilation=1, ceil_mode=False)
      (7): Dropout(p=0.1)
    )
    (2): Sequential(
      (0): Conv1d(312, 312, kernel_size=(7,), stride=(1,), padding=(3,))
      (1): BatchNorm1d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (2): ReLU()
      (3): Conv1d(312, 312, kernel_size=(7,), stride=(1,), padding=(3,))
      (4): BatchNorm1d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (5): ReLU()
      (6): MaxPool1d(kernel_size=8, stride=4, padding=-114, dilation=1, ceil_mode=False)
      (7): Dropout(p=0.1)
    )
    (3): Sequential(
      (0): Conv1d(78, 78, kernel_size=(7,), stride=(1,), padding=(3,))
      (1): BatchNorm1d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (2): ReLU()
      (3): Conv1d(78, 78, kernel_size=(7,), stride=(1,), padding=(3,))
      (4): BatchNorm1d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (5): ReLU()
      (6): MaxPool1d(kernel_size=8, stride=4, padding=-26, dilation=1, ceil_mode=False)
      (7): Dropout(p=0.1)
    )
  )
  (decoder): Sequential(
    (0): Linear(in_features=5120, out_features=512, bias=True)
    (1): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (2): ReLU()
    (3): Dropout(p=0.1)
    (4): Linear(in_features=512, out_features=512, bias=True)
    (5): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (6): ReLU()
    (7): Dropout(p=0.1)
    (8): Linear(in_features=512, out_features=101, bias=True)
    (9): Softmax()
  )
)


title: Pytorch simple model not improving

body:
I am making a simple PyTorch neural net to approximate the sine function on x = [0, 2pi]. This is a simple architecture I use with different deep learning libraries to test whether I understand how to use it or not. The neural net, when untrained, always produces a straight horizontal line, and when trained, produces a straight line at y = 0. In general, it always produces a straight line at y = (The mean of the function). This leads me to believe something is wrong with the forward prop portion of it, as the boundary should not just be a straight line when untrained. Here is the code for the net:

class Net(nn.Module):
    def __init__(self):
      super(Net, self).__init__()
      self.model = nn.Sequential(
      nn.Linear(1, 20),
      nn.Sigmoid(),
      nn.Linear(20, 50),
      nn.Sigmoid(),
      nn.Linear(50, 50),
      nn.Sigmoid(),
      nn.Linear(50, 1)
      )

    def forward(self, x):
        x = self.model(x)
        return x

Here is the training loop

def train(net, trainloader, valloader, learningrate, n_epochs):
    net = net.train()
    loss = nn.MSELoss()
    optimizer = torch.optim.SGD(net.parameters(), lr = learningrate)

    for epoch in range(n_epochs):

        for X, y in trainloader:
            X = X.reshape(-1, 1)
            y = y.view(-1, 1)
            optimizer.zero_grad()

            outputs = net(X)

            error   = loss(outputs, y)
            error.backward()
            #net.parameters()  net.parameters() * learningrate
            optimizer.step()

        total_loss = 0
        for X, y in valloader:
            X = X.reshape(-1, 1).float()
            y = y.view(-1, 1)
            outputs = net(X)
            error   = loss(outputs, y)
            total_loss += error.data

        print('Val loss for epoch', epoch, 'is', total_loss / len(valloader) )

it is called as:

net = Net()
losslist = train(net, trainloader, valloader, .0001, n_epochs = 4)

Where trainloader and valloader are the training and validation loaders. Can anyone help me see what's wrong with this? I know its not the learning rate since its the one I use in other frameworks, and I know its not the fact im using SGD or sigmoid activation functions, although I have a suspicion the error is in the activation functions somewhere.

Does anyone know how to fix this? Thanks.


How I solve this error, I am trying to train a ticket classification model

I'm trying to make a ticket sorter with the Pytorch library, but I have this error. I can't understand what I've done wrong
Can you help me?

data_transforms = {
    'train' : transforms.Compose([
        transforms.RandomResizedCrop(224),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
    'val' : transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ])

}

data_dir = 'dataset_billete_argentino'
image_datasets = {x: datasets.ImageFolder(os.path.join(data_dir, x),
                                          data_transforms[x])
                  for x in ['train', 'val']}
dataloaders = {x: torch.utils.data.DataLoader(image_datasets[x], batch_size=4,
                                              shuffle=True, num_workers=4)
               for x in ['train', 'val']}
dataset_sizes = {x: len(image_datasets[x]) for x in ['train', 'val']}
class_name = image_datasets['train'].classes

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3,6,5)
        self.pool = nn.MaxPool2d(2,2)
        self.conv2 = nn.Conv2d(6,16,5)
        self.fc1 = nn.Linear(16 * 53 * 53, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 2)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(x.size(0), 16* 53 * 53)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3
        return x

net = Net()

def train_model(model, criterion, optimizer, scheduler, num_epochs=25):
    since = time.time()

    best_model_wts = copy.deepcopy(model.state_dict())
    best_acc = 0.0

    for epoch in range(num_epochs):
        print('Epoch {}/{}'.format(epoch, num_epochs - 1))
        print('-' * 10)

        # Each epoch has a training and validation phase
        for phase in ['train', 'val']:
            if phase == 'train':
                model.train()  # Set model to training mode
            else:
                model.eval()   # Set model to evaluate mode

            running_loss = 0.0
            running_corrects = 0

            # Iterate over data.
            for inputs, labels in dataloaders[phase]:
                inputs = inputs.to(device)
                labels = labels.to(device)

                # zero the parameter gradients
                optimizer.zero_grad()

                # forward
                # track history if only in train
                with torch.set_grad_enabled(phase == 'train'):
                    outputs = model(inputs)
                    _, preds = torch.max(outputs, 1)
                    loss = criterion(outputs, labels)

                    # backward + optimize only if in training phase
                    if phase == 'train':
                        loss.backward()
                        optimizer.step()

                # statistics
                running_loss += loss.item() * inputs.size(0)
                running_corrects += torch.sum(preds == labels.data)
            if phase == 'train':
                scheduler.step()

            epoch_loss = running_loss / dataset_sizes[phase]
            epoch_acc = running_corrects.double() / dataset_sizes[phase]

            print('{} Loss: {:.4f} Acc: {:.4f}'.format(
                phase, epoch_loss, epoch_acc))

            # deep copy the model
            if phase == 'val' and epoch_acc > best_acc:
                best_acc = epoch_acc
                best_model_wts = copy.deepcopy(model.state_dict())

        print()

    time_elapsed = time.time() - since
    print('Training complete in {:.0f}m {:.0f}s'.format(
        time_elapsed // 60, time_elapsed % 60))
    print('Best val Acc: {:4f}'.format(best_acc))

    # load best model weights
    model.load_state_dict(best_model_wts)
    return model

from torch.optim import lr_scheduler

exp_lr_scheduler = lr_scheduler.StepLR(optimizer, step_size=7, gamma=0.1)

net = train_model(net, criterion, optimizer, exp_lr_scheduler,
                  num_epochs=25)

And give this error
TypeError: max() received an invalid combination of arguments - got (Linear, int), but expected one of:
 * (Tensor input)
 * (Tensor input, name dim, bool keepdim, tuple of Tensors out)
 * (Tensor input, Tensor other, Tensor out)
 * (Tensor input, int dim, bool keepdim, tuple of Tensors out)

Epoch 0/24
----------
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-27-29dfe3459d8a> in <module>
      4
      5 net = train_model(net, criterion, optimizer, exp_lr_scheduler,
----> 6                   num_epochs=25)

<ipython-input-19-1a5d4f162548> in train_model(model, criterion, optimizer, scheduler, num_epochs)
     31                 with torch.set_grad_enabled(phase == 'train'):
     32                     outputs = model(inputs)
---> 33                     _, preds = torch.max(outputs, 1)
     34                     loss = criterion(outputs, labels)
     35

TypeError: max() received an invalid combination of arguments - got (Linear, int), but expected one of:
 * (Tensor input)
 * (Tensor input, name dim, bool keepdim, tuple of Tensors out)
 * (Tensor input, Tensor other, Tensor out)
 * (Tensor input, int dim, bool keepdim, tuple of Tensors out)


title: What does *variable.shape mean in python

body:
I know "*variable_name" assists in packing and unpacking.

But how does variable_name.shape work? Unable to visualize why the second dimension is squeezed out when prefixing with ""?

print("top_class.shape {}".format(top_class.shape))
top_class.shape torch.Size([64, 1])

print("*top_class.shape {}".format(*top_class.shape))
*top_class.shape 64


title: pytorch subtracting variables with different dimension

body:
conf_loss = cross_entropy_loss(conf_preds.view(-1,num_classes),conf_targets.view(-1))

The shape of x and y are

X: torch.Size([69856, 40]) ,  Y: torch.Size([69856])

respectively.
Author has mentioned the size as x:[N,D] and y:[N,]
but my y size is [N] I need to calculate the difference but I am getting out of memory . Can anyone help with the dimension? I should get the final shape after taking the difference as [N,]. I need to calculate this print(log_sum_exp - x.gather(1, y.view(-1,1)))

def cross_entropy_loss(x, y):
        '''Cross entropy loss w/o averaging across all samples.
        Args:
          x: (tensor) sized [N,D].
          y: (tensor) sized [N,].
        Return:
          (tensor) cross entroy loss, sized [N,].
        '''
        print("X:",x.shape)
        print("Y:",y.shape)
        xmax = x.data.max()

        log_sum_exp = torch.log(torch.sum(torch.exp(x-xmax), 1)) + xmax
        print(log_sum_exp.shape)   # torch.Size([69856])
        print(x.gather(1,y.view(-1,1)).shape)   #torch.Size([69856, 1])
        #print(log_sum_exp - x.gather(1, y.view(-1,1)))

        #return log_sum_exp - x.gather(1, y.view(-1,1))


title: Multi-task regression problem using Pytorch (Problem: Same output for all test data)

body:
I am trying to multi-task regression problem.
Input shape: 200 * 60000, Output shape: 200*3 (here, 200 = Total number of data, 60000 = number of features)
So, for each data point, I have to predict 3 values (continuous).
Sample Code:

class Classifier(nn.Module):
    def __init__(self,input_nodes):
        super(Classifier, self).__init__()
        self.input_nodes = input_nodes

        self.sharedlayer = nn.Sequential(
            nn.Linear(input_nodes, 300),
            nn.ReLU(),
            nn.Dropout(),
            nn.Linear(300, 100),
            nn.ReLU(),
            nn.Dropout(),
        )

        self.att1 = nn.Sequential(
            nn.Linear(100, 40),
            nn.ReLU(),
            nn.Dropout(),
            nn.Linear(40, 20),
            nn.ReLU(),
            nn.Dropout(),
            nn.Linear(20, 1)
        )
        self.att2 = nn.Sequential(
            nn.Linear(100, 40),
            nn.ReLU(),
            nn.Dropout(),
            nn.Linear(40, 20),
            nn.ReLU(),
            nn.Dropout(),
            nn.Linear(20, 1)
        )
        self.att3 = nn.Sequential(
            nn.Linear(100, 40),
            nn.ReLU(),
            nn.Dropout(),
            nn.Linear(40, 20),
            nn.ReLU(),
            nn.Dropout(),
            nn.Linear(20, 1)
        )

    def forward(self, x):

        h_shared = self.sharedlayer(x)
        out1 = self.att1(h_shared)
        out2 = self.att2(h_shared)
        out3 = self.att3(h_shared)

        return out1, out2, out3

criterion = nn.MSELoss()
optimizer = optim.Adam(model.parameters(), lr=0.01)

for epoch in range(n_epochs):
            running_loss = 0
            i = 0
            model.train()
            for data, label in trainloader:
                 i = i + 1
                 out1, out2, out3 = model(data)

                 l1 = criterion(out1, label[:,0].view(-1,1))
                 l2 = criterion(out2, label[:,1].view(-1,1))
                 l3 = criterion(out3, label[:,2].view(-1,1))

                 loss = (l1 + l2 + l3)
                 optimizer.zero_grad()
                 loss.backward()
                 optimizer.step()

Problem: The model always produces same value for all test data.

Example: Suppose, 3 Test data:
For output1: 3.5 3.5 3.5
For output2: 9.5 9.5 9.5
For output3: 0.2 0.2 0.2

Can you please help me to figure out what is the problem here?
Why does it generate the same value for all test data?


title: system shows All requested packages already installed.however i get error message when i import pytorch

body:
I installed pytorch using the following command as specified on their site: conda install pytorch torchvision cudatoolkit=10.2 -c pytorch.
However, I tried using import torch and I get the following error:

'Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\__init__.py", line 81, in <module>
    ctypes.CDLL(dll)
  File "C:\ProgramData\Anaconda3\lib\ctypes\__init__.py", line 364, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: [WinError 126] The specified module could not be found'

How do I resolve this?


title: How to fix this loss is NaN problem in PyTorch of this RNN with GRU?

body:
I'm completely new to PyTorch and tried out some models. I wanted to make an easy prediction rnn of stock market prices and found the following code:
I load the data set with pandas then split it into training and test data and load it into a pytorch DataLoader for later usage in training process. The model is defined in the GRU class. But the actual problem seems to be the optimisation. I think the problem could be gradient explosion. I thought about adding gradient clipping but the GRU design should actually prevent gradient explosion or am I wrong? What could cause the loss to be instantly NaN (already in the first epoch)
from sklearn.preprocessing import MinMaxScaler

import time
import pandas as pd
import numpy as np

import torch
import torch.nn as nn
from torch.utils.data import TensorDataset, DataLoader

batch_size = 200
input_dim = 1
hidden_dim = 32
num_layers = 2
output_dim = 1
num_epochs = 10

nvda = pd.read_csv('dataset/stocks/NVDA.csv')
price = nvda[['Close']]
scaler = MinMaxScaler(feature_range=(-1, 1))
price['Close'] = scaler.fit_transform(price['Close'].values.reshape(-1, 1))

def split_data(stock, lookback):
    data_raw = stock.to_numpy()  # convert to numpy array
    data = []

    # create all possible sequences of length seq_len
    for index in range(len(data_raw) - lookback):
        data.append(data_raw[index: index + lookback])

    data = np.array(data)
    test_set_size = int(np.round(0.2 * data.shape[0]))
    train_set_size = data.shape[0] - (test_set_size)

    x_train = data[:train_set_size, :-1, :]
    y_train = data[:train_set_size, -1, :]

    x_test = data[train_set_size:, :-1]
    y_test = data[train_set_size:, -1, :]

    return [x_train, y_train, x_test, y_test]

lookback = 20  # choose sequence length
x_train, y_train, x_test, y_test = split_data(price, lookback)

train_data = TensorDataset(torch.from_numpy(x_train).float(), torch.from_numpy(y_train).float())
train_data = DataLoader(train_data, shuffle=True, batch_size=batch_size, drop_last=True)

test_data = TensorDataset(torch.from_numpy(x_test).float(), torch.from_numpy(y_test).float())
test_data = DataLoader(test_data, shuffle=True, batch_size=batch_size, drop_last=True)

class GRU(nn.Module):
    def __init__(self, input_dim, hidden_dim, num_layers, output_dim):
        super(GRU, self).__init__()
        self.hidden_dim = hidden_dim
        self.num_layers = num_layers

        self.gru = nn.GRU(input_dim, hidden_dim, num_layers, batch_first=True, dropout=0.2)
        self.fc = nn.Linear(hidden_dim, output_dim)
        self.relu = nn.ReLU()

    def forward(self, x, h):

        out, h = self.gru(x, h)
        out = self.fc(self.relu(out[:, -1]))
        return out, h

    def init_hidden(self, batch_size):
        weight = next(self.parameters()).data
        hidden = weight.new(self.num_layers, batch_size, self.hidden_dim).zero_()
        return hidden

model = GRU(input_dim=input_dim, hidden_dim=hidden_dim, output_dim=output_dim, num_layers=num_layers)
criterion = nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.0000000001)
model.train()

start_time = time.time()

h = model.init_hidden(batch_size)
for epoch in range(1, num_epochs+1):
    for x, y in train_data:
        h = h.data
        model.zero_grad()
        y_train_pred, h = model(x, h)
        loss = criterion(y_train_pred, y)
        print("Epoch ", epoch, "MSE: ", loss.item())
        loss.backward()
        optimizer.step()

training_time = time.time() - start_time
print("Training time: {}".format(training_time))

This is the dataset which I used.


title: How to handle Multi Label DataSet from Directory for image captioning in PyTorch

body:
I need  a  help in PyTorch,
Regarding Dataloader, and dataset
Can someone aid/guide me

Here is my query :
I am trying for Image Captioning using https://github.com/yunjey/pytorch-tutorial/tree/master/tutorials/03-advanced/image_captioning.

Here they have used Standard COCO Dataset.

I have dataset as images/ and captions/ directory .

Example

Directory Structure:

images/T001.jpg
images/T002.jpg
...
...
captions/T001.txt
captions/T002.txt
....
....

The above is the relation. Caption file has 'n' number of captions in each separate line.

I am able to create a custom Dataset class, in that the complete caption file content is being returned. But I want only one line alone gas to be returned.

Any guidance/suggestion on how to achieving this.

++++++++++++++++++++++++++++++++++++++++++++++++
Here is the class that i have designed:

from __future__ import print_function
import torch
from torchvision import datasets, models, transforms
from torchvision import transforms
from torch.autograd import Variable
from torch.nn.utils.rnn import pack_padded_sequence
import torch.optim as optim
import torch.nn as nn
#from torch import np
import numpy as np
import utils_c
from data_loader_c import get_cust_data_loader
from models import CNN, RNN
from vocab_custom import Vocabulary, load_vocab
import os

class ImageCaptionDataSet(data.Dataset):
    def __init__(self, path, json, vocab=None, transform=None):
        self.vocab = vocab
        self.transform = transform
        self.img_dir_path = path
        self.cap_dir_path = json
        self.all_imgs_path = glob.glob(os.path.join(self.img_dir_path,'*.jpg'))
        self.all_caps_path = glob.glob(os.path.join(self.cap_dir_path,'*.txt'))
        pass

    def __getitem__(self,index):
        vocab = self.vocab

        img_path = self.all_imgs_path[index]
        img_base_name = os.path.basename(img_path)
        cap_base_name = img_base_name.replace(".jpg",".txt")
        cap_path  = os.path.join(self.cap_dir_path,cap_base_name)

        caption_all_for_a_image = open(cap_path).read().split("\n")

        image = Image.open(img_path)
        image = image.convert('RGB')

        if self.transform != None:
            # apply image preprocessing
            image = self.transform(image)

        #captions_combined = []
        #max_len = 0
        #for caption in caption_all_for_a_image:
        #    caption_str = str(caption).lower()
        #    tokens = nltk.tokenize.word_tokenize(caption_str)
        #    m = len(tokens) + 2
        #    if m>max_len:
        #        max_len = m
        #    caption = torch.Tensor([vocab(vocab.start_token())] +
        #                           [vocab(token) for token in tokens] +
        #                           [vocab(vocab.end_token())])
        #    captions_combined.append(caption)
        #    #yield image, caption
        #return image,torch.Tensor(captions_combined)

        caption_str = str(caption_all_for_a_image).lower()
        tokens = nltk.tokenize.word_tokenize(caption_str)
        caption = torch.Tensor([vocab(vocab.start_token())] +
                                   [vocab(token) for token in tokens] +
                                   [vocab(vocab.end_token())])

        return image,caption

    def __len__(self):
        return len(self.all_imgs_path)

+++++++++++++++++++++++++++++++++