title: Inconsistent output between Python and LibTorch C++ when exporting for iOS

body:
I've trained the HuggingFace RoBERTa model for my data (it's a very particular usage — hence the small model/vocabulary!) and tested successfully on Python. I exported the traced model to LibTorch for iOS, but prediction results on device do not match those in Python (giving different argmax token indices). My conversion script:
# torch = 1.5.0
# transformers = 3.2.0

config = RobertaConfig(
    vocab_size=858,
    max_position_embeddings=258,
    num_attention_heads=6,
    num_hidden_layers=4,
    type_vocab_size=1,
    torchscript=True,
)

model = RobertaForMaskedLM(config=config).from_pretrained('./trained_RoBERTa')
model.cpu()
model.eval()

example_input = torch.LongTensor(1, 256).random_(0, 857).cpu()
traced_model = torch.jit.trace(model, example_input)
traced_model.save('./exports/trained_RoBERTa.pt')

I have had problems in the past with another (vision) model that I trained in Python+GPU and converted to LibTorch for iOS, which were solved by adding map_location={'cuda:0': 'cpu'} to the torch.load() call in my conversion script. So I'm wondering whether: 1) that makes sense as a possible explanation in this situation?, and 2) how I can add the map_location option when loading using the .from_pretrained() syntax?
Just in case my Obj-C++ handling of the prediction results is to blame, here's the Obj-C++ code run on device:
- (NSArray<NSArray<NSNumber*>*>*)predictText:(NSArray<NSNumber*>*)tokenIDs {
    try {
        long count = tokenIDs.count;
        long* buffer = new long[count];
        for(int i=0; i < count;  i++) {
            buffer[i] = tokenIDs[i].intValue;
        }
        at::Tensor tensor = torch::from_blob(buffer, {1, (int64_t)count}, at::kLong);
        torch::autograd::AutoGradMode guard(false);
        at::AutoNonVariableTypeMode non_var_type_mode(true);
        auto outputTuple = _impl.forward({tensor}).toTuple();

        auto outputTensor = outputTuple->elements()[0].toTensor();
        auto sizes = outputTensor.sizes();
        // len will be tokens * vocab size -- sizes[1] * sizes[2] (sizes[0] is batch_size = 1)
        auto positions = sizes[1];
        auto tokens = sizes[2];
        float* floatBuffer = outputTensor.data_ptr<float>();
        if (!floatBuffer) {
            return nil;
        }
        // MARK: This is probably a slow way to create this 2D NSArray
        NSMutableArray* results = [[NSMutableArray alloc] initWithCapacity: positions];
        for (int i = 0; i < positions; i++) {
            NSMutableArray* weights = [[NSMutableArray alloc] initWithCapacity: tokens];
            for (int j = 0; j < tokens; j++) {
                [weights addObject:@(floatBuffer[i*positions + j])];
            }
            [results addObject:weights];
        }
        return [results copy];
    } catch (const std::exception& exception) {
        NSLog(@"%s", exception.what());
    }
    return nil;
}

Note that my init code in iOS does call eval() on the TorchScript model.
UPDATE: One observation; the way I've attempted to use my config when loading the trained model above results in the torchscript flag not being set — I assume it's ignoring my config entirely and getting it from the pretrained file. So I've moved it to from_pretrained('./trained_RoBERTa', torchscript=True), as outlined in the docs. Same problem with output on iOS, mind you...
UPDATE 2: I thought I'd try testing the traced model in Python. Not sure it's expected that this should work, but the output does match the same test in the original model:
traced_test = traced_model(input)
pred = torch.argmax(traced_test[0], dim=2).squeeze(0)
pred_str = tokenizer.decode(pred[1:-1].tolist())
print(pred_str)

Which makes me think there's something going with the iOS Obj-C++ execution. The code that loads the traced model/export does call .eval() on the model, btw (I realize that comes up as a possible explanation for differing outputs):
- (nullable instancetype)initWithFileAtPath:(NSString*)filePath {
    self = [super init];
    if (self) {
        try {
            auto qengines = at::globalContext().supportedQEngines();
            if (std::find(qengines.begin(), qengines.end(), at::QEngine::QNNPACK) != qengines.end()) {
                at::globalContext().setQEngine(at::QEngine::QNNPACK);
            }
            _impl = torch::jit::load(filePath.UTF8String);
            _impl.eval();
        } catch (const std::exception& exception) {
            NSLog(@"%s", exception.what());
            return nil;
        }
    }
    return self;
}

UPDATE 3: Uhhhmmm... This is definitely a face-palm moment (following a wasted weekend)... I decided to return a flat NSArray from Obj-C and do the 2D array reshape in Swift, and aside from a shift of one token (I think it's just the [CLS]), the output is now correct. I guess my Obj-C really is that rusty. Sadly, I still don't see the issue, but it's working now so I'm going to surrender.


title: Tensorflow Extended: Is it possible to use pytorch training loop in Tensorflow extended flow

body:
I have trained an image classification model using pytorch.
Now, I want to move it from research to production pipeline.
I am thinking of using TensorFlow extended. I have a very noob doubt that will I'll be able to use my PyTorch trained model in the TensorFlow extended pipeline(I can convert the trained model to ONNX and then to Tensorflow compatible format).
I don't want to rewrite and retrain the training part to TensorFlow as it'll be a great overhead.
Is it possible or Is there any better way to productionize the PyTorch trained models?


title: What is the difference between an Embedding Layer with a bias immediately afterwards and a Linear Layer in PyTorch

body:
I am reading the "Deep Learning for Coders with fastai & PyTorch" book. I'm still a bit confused as to what the Embedding module does. It seems like a short and simple network, except I can't seem to wrap my head around what Embedding does differently than Linear without a bias. I know it does some faster computational version of a dot product where one of the matrices is a one-hot encoded matrix and the other is the embedding matrix. It does this to in effect select a piece of data? Please point out where I am wrong. Here is one of the simple networks shown in the book.
class DotProduct(Module):
    def __init__(self, n_users, n_movies, n_factors):
        self.user_factors = Embedding(n_users, n_factors)
        self.movie_factors = Embedding(n_movies, n_factors)

    def forward(self, x):
        users = self.user_factors(x[:,0])
        movies = self.movie_factors(x[:,1])
        return (users * movies).sum(dim=1)


title: Using an autoencoder to reduce dimensionality

body:
Here is my version of an autoencoder written using PyTorch :

import warnings
warnings.filterwarnings('ignore')
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
from matplotlib import pyplot as plt
from sklearn import metrics
import datetime
from sklearn.preprocessing import MultiLabelBinarizer
import seaborn as sns
sns.set_style("darkgrid")
from ast import literal_eval
import numpy as np
from sklearn.preprocessing import scale
import seaborn as sns
sns.set_style("darkgrid")
import torch

%matplotlib inline

f = []
f.append(np.random.uniform(0,10,(1 , 10)).flatten())
f.append(np.random.uniform(10,20,(1 , 10)).flatten())
f.append(np.random.uniform(20,30,(1 , 10)).flatten())
x_data = torch.FloatTensor(np.array(f))
x_data

dimensions_input = 10
hidden_layer_nodes = 5
output_dimension = 10

class Model(torch.nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.linear = torch.nn.Linear(dimensions_input,hidden_layer_nodes)
        self.sigmoid = torch.nn.Sigmoid()
        self.linear2 = torch.nn.Linear(hidden_layer_nodes,output_dimension)

    def forward(self, x):
        l_out1 = self.linear(x)
        l_out2 = self.sigmoid(l_out1)
        y_pred = self.linear2(l_out2)
        return y_pred

model = Model()

criterion = torch.nn.MSELoss(size_average = False)
optim = torch.optim.SGD(model.parameters(), lr = 0.00001)

def train_model():
    y_data = x_data.clone()
    for i in range(150000):
        y_pred = model(x_data)
        loss = criterion(y_pred, y_data)

        if i % 5000 == 0:
            print(loss)
        optim.zero_grad()

        loss.backward()
        optim.step()

Using x_data.clone() I train the network to learn a feature representation of the input data.

I'm attempting to generate hidden layer weights that match the dimensionality of rows of the input data so that each vector of x_data has a corresponding encoding. But the hidden later is of is a vector of size 5. How to change this network so that a matrix is generated that represents a reduced dimensionality of the input data ?


title: Translating LSTM model from Keras to Pytorch

body:
I am having a hard time translating a quite simple LSTM model from Keras to Pytorch. X (get it here) corresponds to 1152 samples of 90 timesteps, each timestep has only 1 dimension. y (here) is a single prediction at t = 91 for all 1152 samples.

In Keras:

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Dropout, LSTM
import numpy as np
import pandas as pd

X = pd.read_csv('X.csv', header = None).values
X.shape

y = pd.read_csv('y.csv', header = None).values
y.shape

# From Keras documentation [https://keras.io/layers/recurrent/]:
# Input shape 3D tensor with shape (batch_size, timesteps, input_dim).
X = np.reshape(X, (1152, 90, 1))

regressor = Sequential()
regressor.add(LSTM(units = 100, return_sequences = True, input_shape = (90, 1)))
regressor.add(Dropout(0.3))
regressor.add(LSTM(units = 50, return_sequences = True))
regressor.add(Dropout(0.3))
regressor.add(LSTM(units = 50, return_sequences = True))
regressor.add(Dropout(0.3))
regressor.add(LSTM(units = 50))
regressor.add(Dropout(0.3))
regressor.add(Dense(units = 1, activation = 'linear'))
regressor.compile(optimizer = 'rmsprop', loss = 'mean_squared_error', metrics = ['mean_absolute_error'])
regressor.fit(X, y, epochs = 10, batch_size = 32)

... leads me to:

# Epoch 10/10
# 1152/1152 [==============================] - 33s 29ms/sample - loss: 0.0068 - mean_absolute_error: 0.0628

Then in Pytorch:

import torch
from torch import nn, optim
from sklearn.metrics import mean_absolute_error

X = pd.read_csv('X.csv', header = None).values
y = pd.read_csv('y.csv', header = None).values

X = torch.tensor(X, dtype = torch.float32)
y = torch.tensor(y, dtype = torch.float32)

dataset = torch.utils.data.TensorDataset(X, y)
loader = torch.utils.data.DataLoader(dataset, batch_size = 32, shuffle = True)

class regressor_LSTM(nn.Module):
    def __init__(self):
        super().__init__()
        self.lstm1 = nn.LSTM(input_size = 1, hidden_size = 100)
        self.lstm2 = nn.LSTM(100, 50)
        self.lstm3 = nn.LSTM(50, 50, dropout = 0.3, num_layers = 2)
        self.dropout = nn.Dropout(p = 0.3)
        self.linear = nn.Linear(in_features = 50, out_features = 1)

    def forward(self, X):
        # From the Pytorch documentation [https://pytorch.org/docs/stable/_modules/torch/nn/modules/rnn.html]:
        # **input** of shape `(seq_len, batch, input_size)`
        X = X.view(90, 32, 1)
        # I am discarding hidden/cell states since in Keras I am using a stateless approach
        # [https://keras.io/examples/lstm_stateful/]
        X, _ = self.lstm1(X)
        X = self.dropout(X)
        X, _ = self.lstm2(X)
        X = self.dropout(X)
        X, _ = self.lstm3(X)
        X = self.dropout(X)
        X = self.linear(X)

        return X

regressor = regressor_LSTM()
criterion = nn.MSELoss()
optimizer = optim.RMSprop(regressor.parameters())

for epoch in range(10):
    running_loss = 0.
    running_mae = 0.

    for i, data in enumerate(loader):
        inputs, labels = data
        optimizer.zero_grad()
        outputs = regressor(inputs)
        outputs = outputs[-1].view(*labels.shape)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()
        running_loss += loss.item()
        mae = mean_absolute_error(labels.detach().cpu().numpy().flatten(), outputs.detach().cpu().numpy().flatten())
        running_mae += mae

    print('EPOCH %3d: loss %.5f - MAE %.5f' % (epoch+1, running_loss/len(loader), running_mae/len(loader)))

... leads me to:

# EPOCH  10: loss 0.04220 - MAE 0.16762

You can notice that both loss and MAE are quite different (Pytorch's are much higher). If I use Pytorch's model to predict the values, they all return as a constant.

What am I doing wrong?


title: How to process input data for audio classification using CNN with PyTorch?

body:
As an engineer student works towards DSP and ML fields, I am working on an audio classification project with inputs being short clips (4 sec.) of instruments like bass, keyboard, guitar, etc. (NSynth Dataset by the Magenta team at Google).

The idea is to convert all the short clips (.wav files) to spectrograms or melspectrograms then apply a CNN to train the model.

However, my questions is since the entire dataset is large (approximately 23GB), I wonder if I should firstly convert all the audio files to images like PNG then apply CNN. I feel like this can take a lot of time, and it will double the storage space for my input data as now it is audio + image (maybe up to 70GB).

Thus, I wonder if there is any workaround here that can speed the process.

Thanks in advance.


title: generate an FBX for smplx

body:
I need to generate an FBX file for maya from a .pkl pytorch model (smplx model).I'm quite new to 3d graphics(I know python and pytorch). My question is how can I get it done? like general steps. Also if something to read or topic to learn. I would really appreciate your help.

FBX for another similar model (smpl) is available from here in the downloads (you'll have to sign up) https://smpl.is.tue.mpg.de/
but I need smplx


I am doing training and put the dataset inside the data folder. The Structure looks like this.
--data
-----mars
---------bbox_train
---------bbox_test
---------info

Many developers said that this is a label problem but I am not sure because labels are in the right place.
Traceback (most recent call last):
Args:Namespace(arch='resnet50graphpoolparthyper', concat=False, dataset='mars', dropout=0.1, eval_step=100, evaluate=False, gamma=0.1, gpu_devices='0', height=256, htri_only=False, lr=0.0003, margin=0.3, max_epoch=800, nheads=8, nhid=512, num_instances=4, part1=4, part2=8, part3=2, pool='avg', pretrained_model='/home/jiyang/Workspace/Works/video-person-reid/3dconv-person-reid/pretrained_models/resnet-50-kinetics.pth', print_freq=80, save_dir='log_hypergraphsagepart', seed=1, seq_len=8, start_epoch=0, stepsize=200, test_batch=1, train_batch=32, use_cpu=False, warmup=True, weight_decay=0.0005, width=128, workers=4, xent_only=False)
==========
Currently using GPU 0
Initializing dataset mars
=> MARS loaded
Dataset statistics:
  ------------------------------
  subset   | # ids | # tracklets
  ------------------------------
  train    |   625 |     8298
  query    |   626 |     1980
  gallery  |   622 |     9330
  ------------------------------
  total    |  1251 |    19608
  number of images per tracklet: 2 ~ 920, average 59.5
  ------------------------------
Initializing model: resnet50graphpoolparthyper
Model size: 44.17957M
==> Epoch 1/800  lr:1.785e-05
Traceback (most recent call last):
  File "main_video_person_reid_hypergraphsage_part.py", line 357, in <module>
    main()
  File "main_video_person_reid_hypergraphsage_part.py", line 220, in main
    train(model, criterion_xent, criterion_htri, optimizer, trainloader, use_gpu)
  File "main_video_person_reid_hypergraphsage_part.py", line 257, in train
    outputs, features = model(imgs)
  File "/home/khawar/anaconda3/envs/hypergraph_reid/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/khawar/anaconda3/envs/hypergraph_reid/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 165, in forward
    return self.module(*inputs[0], **kwargs[0])
  File "/home/khawar/anaconda3/envs/hypergraph_reid/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/media/khawar/HDD_Khawar1/hypergraph_reid/models/ResNet_hypergraphsage_part.py", line 621, in forward
    x = self.base(x)
  File "/home/khawar/anaconda3/envs/hypergraph_reid/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/media/khawar/HDD_Khawar1/hypergraph_reid/models/resnet.py", line 213, in forward
    x = self.conv1(x)
  File "/home/khawar/anaconda3/envs/hypergraph_reid/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/khawar/anaconda3/envs/hypergraph_reid/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 399, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "/home/khawar/anaconda3/envs/hypergraph_reid/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 396, in _conv_forward
    self.padding, self.dilation, self.groups)
RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED


title: How can I use three Conv1d on the three axis of my 3*n matrix in Pytorch?

body:
The following is my CNN. The input of it is a (3,64) matrix, I want to use three convolution kernels to process the x,y,z axis respectively.

class Char_CNN(nn.Module):
    def __init__(self):
        super(Char_CNN, self).__init__()
        self.convdx = nn.Conv1d(1, 12, 20)
        self.convdy = nn.Conv1d(1, 12, 20)
        self.convdz = nn.Conv1d(1, 12, 20)
        self.fc1 = nn.Linear(540, 1024)
        self.fc2 = nn.Linear(1024, 30)
        self.fc3 = nn.Linear(30, 13)

    def forward(self, x):
        after_convd = [self.convdx(x[:, :, 0]), self.convdy(x[:, :, 1]), self.convdz(x[:, :, 2])]
        after_pool = [F.max_pool1d(F.relu(value), 3) for value in after_convd]

        x = torch.cat(after_pool, 1)
        x = x.view(x.size(0), -1)
        x = self.fc1(x)
        x = self.fc2(x)
        x = self.fc3(x)
        x = F.softmax(x)
        return x

But during the running of loss = criterion(out, target), a RunTime Error occurs:

  RuntimeError: Assertion `cur_target >= 0 && cur_target < n_classes' failed.

I'm very new to pytorch so that I cannot find out the mistake of my code.
Can you help me?


title: Model taking long time to train

body:
I have added an LSTM layer after a convolution in the VGG-16 model using PyTorch. Overtime, the model learns just fine. However, after adding just one LSTM layer, which consists of 32 LSTM cells, the process of training and evaluating takes about 10x longer.

I added the LSTM layer to a VGG framework as follows

def make_layers(cfg, batch_norm=False):
   # print("Making layers!")
    layers = []
    in_channels = 3
    count=0
    for v in cfg:
        count+=1
        if v == 'M':
            layers += [nn.MaxPool2d(kernel_size=2, stride=2)]
        else:
            conv2d = nn.Conv2d(in_channels, v, kernel_size=3, padding=1)
            if batch_norm:
                layers += [conv2d, nn.BatchNorm2d(v), nn.ReLU(inplace=True)]
            else:
                layers += [conv2d, nn.ReLU(inplace=True)]
                in_channels=v
        if count==5:
            rlstm =RLSTM(v)
            rlstm=rlstm.cuda()
            layers+=[rlstm]

RLSTM is my custom class, which implements RowLSTM, from Google’s Pixel RNN paper.

Is this a common issue?  Do LSTM layers just take long to train in general?


title: PyTorch - removing batch norm gives different model results in inference

body:
I removed the batch norm layers from the model and loaded the weights of all the other layers for inference. The predictions of the original model vs models without batch norm are not the same.

Is the difference caused by the removal of the batch norm?

#load pretrained model
checkpoint = torch.load(restore_file)
pretrained_model = checkpoint['model']
pretrained_dict = pretrained_model.state_dict()

#create new model without batch norm
model = no_batch_model()
model_dict = model.state_dict()

# 1. filter out unnecessary keys
pretrained_dict = {k: v for k, v in pretrained_dict.items() if k in model_dict}

# 2. overwrite entries in the existing state dict
model_dict.update(pretrained_dict)

# 3. load the new state dict
model.load_state_dict(pretrained_dict)


title: SpaCy - ValueError: operands could not be broadcast together with shapes (1,2) (1,5)

body:
In relation to the previous post on stackoverflow
Model() got multiple values for argument 'nr_class' - SpaCy multi-classification model (BERT integration) in which my problem partialy have beed resolved I wanted to share the issue which comes up after implementing the solution.

if I take out the nr_class argument, I get this error here:

ValueError: operands could not be broadcast together with shapes (1,2) (1,5)

I actually thought this would happen because I didn't specify the nr_class
 argument. Is this correct?

one more time my code for the multi-class model:

nlp = spacy.load('en_pytt_bertbaseuncased_lg')
textcat = nlp.create_pipe(
    'pytt_textcat',
    config={
        "nr_class":5,
        "exclusive_classes": True,
    }
)
nlp.add_pipe(textcat, last = True)

textcat.add_label("class1")
textcat.add_label("class2")
textcat.add_label("class3")
textcat.add_label("class4")
textcat.add_label("class5")

The code for the training is as follows and is based on the example from here(https://pypi.org/project/spacy-pytorch-transformers/):

def extract_cat(x):
    for key in x.keys():
        if x[key]:
            return key

# get names of other pipes to disable them during training
n_iter = 250 # number of epochs

train_data = list(zip(train_texts, [{"cats": cats} for cats in train_cats]))

dev_cats_single   = [extract_cat(x) for x in dev_cats]
train_cats_single = [extract_cat(x) for x in train_cats]
cats = list(set(train_cats_single))
recall = {}
for c in cats:
    if c is not None:
        recall['dev_'+c] = []
        recall['train_'+c] = []

optimizer = nlp.resume_training()
batch_sizes = compounding(1.0, round(len(train_texts)/2), 1.001)

for i in range(n_iter):
    random.shuffle(train_data)
    losses = {}
    batches = minibatch(train_data, size=batch_sizes)
    for batch in batches:
        texts, annotations = zip(*batch)
        nlp.update(texts, annotations, sgd=optimizer, drop=0.2, losses=losses)
    print(i, losses)

So the structure of my data looks like this:

[('TEXT TEXT TEXT',
  {'cats': {'class1': False,
    'class2': False,
    'class3': False,
    'class4': True,
    'class5': False}}), ... ]


title: Pytorch - how to extract features of an MLP network (weights, biases, number of nodes, hidden layers)?

body:
I would be interested to extract the weights, biases, number of nodes and number of hidden layers from an MLP/neural network built in pytorch. I wonder if anyone may be able to point me in the right direction?

Many thanks,

Max


title: Text generation using huggingface's distilbert models

body:
I've been struggling with huggingface's DistilBERT model for some time now, since the documentation seems very unclear and their examples (e.g. https://github.com/huggingface/transformers/blob/master/notebooks/Comparing-TF-and-PT-models-MLM-NSP.ipynb and https://github.com/huggingface/transformers/tree/master/examples/distillation) are extremely thick and the thing they are showcasing doesn't seem well documented.

I'm wondering if anyone here has any experience and knows of some good code example for basic in-python usage of their models. Namely:

How to properly decode the output of the model into actual text (no matter how I change its shape the tokenizer seems willing to decode it and always yields some sequence of [UNK] tokens)
How to actually use their schedulers+optimizers to train a model for a simple text to text task.


title: Installing torch using pip / Visual Studio

body:
I'm trying every which way to install torch via Visual Studio, yet keep getting the below errors.

Any suggestions on what to do? Have seen some mention that the normal pip install torch command does not work as intended, but as I'm fairly new to Python I'm not sure where to go from here.

----- Installing 'torch' -----
Collecting torch
  Using cached torch-0.1.2.post2.tar.gz (128 kB)
Requirement already satisfied, skipping upgrade: pyyaml in c:\users\`\desktop\python projects\deeprl\deeprl\lib\site-packages (from torch) (5.3)
Installing collected packages: torch
    Running setup.py install for torch: started
    Running setup.py install for torch: finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: 'C:\Users\`\Desktop\Python Projects\DeepRL\DeepRL\Scripts\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\`\\AppData\\Local\\Temp\\pip-install-6ouq_hzc\\torch\\setup.py'"'"'; __file__='"'"'C:\\Users\\`\\AppData\\Local\\Temp\\pip-install-6ouq_hzc\\torch\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\`\AppData\Local\Temp\pip-record-lfjewv4e\install-record.txt' --single-version-externally-managed --compile --install-headers 'C:\Users\`\Desktop\Python Projects\DeepRL\DeepRL\include\site\python3.7\torch'
         cwd: C:\Users\`\AppData\Local\Temp\pip-install-6ouq_hzc\torch\
    Complete output (23 lines):
    running install
    running build_deps
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\Users\`\AppData\Local\Temp\pip-install-6ouq_hzc\torch\setup.py", line 265, in <module>
        description="Tensors and Dynamic neural networks in Python with strong GPU acceleration",
      File "C:\Users\`\Desktop\Python Projects\DeepRL\DeepRL\lib\site-packages\setuptools\__init__.py", line 144, in setup
        return distutils.core.setup(**attrs)
      File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python37_64\lib\distutils\core.py", line 148, in setup
        dist.run_commands()
      File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python37_64\lib\distutils\dist.py", line 966, in run_commands
        self.run_command(cmd)
      File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python37_64\lib\distutils\dist.py", line 985, in run_command
        cmd_obj.run()
      File "C:\Users\`\AppData\Local\Temp\pip-install-6ouq_hzc\torch\setup.py", line 99, in run
        self.run_command('build_deps')
      File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python37_64\lib\distutils\cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python37_64\lib\distutils\dist.py", line 985, in run_command
        cmd_obj.run()
      File "C:\Users\`\AppData\Local\Temp\pip-install-6ouq_hzc\torch\setup.py", line 51, in run
        from tools.nnwrap import generate_wrappers as generate_nn_wrappers
    ModuleNotFoundError: No module named 'tools.nnwrap'
    ----------------------------------------
ERROR: Command errored out with exit status 1: 'C:\Users\`\Desktop\Python Projects\DeepRL\DeepRL\Scripts\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\`\\AppData\\Local\\Temp\\pip-install-6ouq_hzc\\torch\\setup.py'"'"'; __file__='"'"'C:\\Users\\`\\AppData\\Local\\Temp\\pip-install-6ouq_hzc\\torch\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\`\AppData\Local\Temp\pip-record-lfjewv4e\install-record.txt' --single-version-externally-managed --compile --install-headers 'C:\Users\`\Desktop\Python Projects\DeepRL\DeepRL\include\site\python3.7\torch' Check the logs for full command output.
----- Failed to install 'torch' -----


title: Pytorch rename labels of a trained model

body:
I have a trained model. During training there was some problem with string characters. So i converted my labels into numbers like:
red : 0
blue: 1
green: 2
Now is it possible to rename my label back to actual label names.
Hours of training. Would be  helpful if anyone has an idea.
Train and validate the model
for epoch in range(1, epoch_num + 1):
    loss_train, acc_train = train(train_loader, model, criterion, optimizer, epoch)
    loss_val, acc_val = validate(val_loader, model, criterion, epoch)
    total_loss_val.append(loss_val)
    total_acc_val.append(acc_val)

Test Single Image:
def eval_image(file_path):
    model = torch.load(file_path)
    model.eval()
    device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
    X = Image.open('red.jpeg')
    test_transforms = transforms.Compose([transforms.ToTensor()])
    image_tensor = test_transforms(X).float()
    image_tensor = image_tensor.unsqueeze_(0)
    input = Variable(image_tensor)
    input = input.to(device)
    output = model(input)
    index = output.data.cpu().numpy().argmax()
    print(index)

Once the training is done, The evaluation script cannot be made generic. I will have to pass following code
idx_to_class = {
        0: "red",
        1: "blue",
        2: "green",
    }
    class_name = idx_to_class[index]

But I do not want to pass the above code. As my evaluation script needs to generic.


title: How to perform batch multiplication in pytorch without bmm?

body:
I'm doing the online Computer Vision course by UMich and am new to PyTorch. One of the assignment questions is on batch matrix multiplication, where we have to find the batch matrix product with and without the bmm function. Here is the code.
    def batched_matrix_multiply(x, y, use_loop=True):
  """
  Perform batched matrix multiplication between the tensor x of shape (B, N, M)
  and the tensor y of shape (B, M, P).

  If use_loop=True, then you should use an explicit loop over the batch
  dimension B. If loop=False, then you should instead compute the batched
  matrix multiply without an explicit loop using a single PyTorch operator.

  Inputs:
  - x: Tensor of shape (B, N, M)
  - y: Tensor of shape (B, M, P)
  - use_loop: Whether to use an explicit Python loop.

  Hint: torch.stack, bmm

  Returns:
  - z: Tensor of shape (B, N, P) where z[i] of shape (N, P) is the result of
       matrix multiplication between x[i] of shape (N, M) and y[i] of shape
       (M, P). It should have the same dtype as x.
  """
  z = None
  #############################################################################
  #                    TODO: Implement this function                          #
  #############################################################################
  # Replace "pass" statement with your code
  z = torch.zeros(x.shape[0], x.shape[1], y.shape[2])
  if use_loop == True:
    for i in range(x.shape[0]):
      z[i] = torch.mm(x[i], y[i])
  else:
    z = torch.bmm(x,y)
  #############################################################################
  #                            END OF YOUR CODE                               #
  #############################################################################
  return z

I've managed to do it without bmm, but without using the torch.stack hint. I initialized a zeroes matrix 'z' with the dimensions of the output matrix and performed normal matrix multiplication for each batch using the for loop.
I'd like to know what the more efficient answer using torch.stack is.


title: onnxjs opset versions and MaxPool

body:
I'm trying to export an FCN from torchvision using the following code:
model= models.segmentation.fcn_resnet101(pretrained=True, progress=True, num_classes=21, aux_loss=None)
model.eval()
x = torch.randn(1,3,512,512)
torch_out = model(x)
torch.onnx.export(model, x, "seg_rn.onnx",
                    export_params=True,
                    opset_version=11,
                    do_constant_folding=True,
                    verbose=True
                    )

When exporting the model, I need minimum opset 11 to support the way pytorch's interpolation works, and this is confirmed by the output of the onnx model when running in the python onnx runtime.
Running in the python onnx runtime is fine, but when I load the model in onnxjs, like this:
var session = new InferenceSession();
const modelURL = "./models/seg_rn.onnx";
await session.loadModel(modelURL);

I get Uncaught (in promise) TypeError: cannot resolve operator 'Shape' with opsets: ai.onnx v11
If I go and create my own copies of bits from torchvision.models.segmentation, I can get rid of the error about Shape (by speciying a static shape for the input and telling the interpolation what the resizing factor should be), but then I get basically the same error: Uncaught (in promise) TypeError: cannot resolve operator 'MaxPool' with opsets: ai.onnx v11 but this time in reference to MaxPool. Ignoring tests and outputting with opset v10 results in a loadable model, but one which will be incorrect.
What is going on? Is there any way forward, or am I basically stuck?


title: How do I make sure Vulkan is using the same GPU as CUDA?

body:
I'm using an application that uses both vulkan and cuda (specifically pytorch) on an HPC cluster (univa grid engine).
When a job is submitted, the cluster scheduler sets an environment variable SGE_HGR_gpu which contains a GPU ID for the job to use (so other jobs run by other users do not use the same GPU)
The typical way to tell an application that uses CUDA to use a specific GPU is to set CUDA_VISIBLE_DEVICES=n
As i'm also using Vulkan, I dont know how to make sure that I choose the same device from those that are listed with vkEnumeratePhysicalDevices.
I think that the order of the values that 'n' can take is the same as the order of the devices on the PCI BUS, however I dont know if the order of the devices returned by vkEnumeratePhysicalDevices are in this order, and the documentation does not specify what this order is.
So how can I go about making sure i'm choosing the same physical GPU for both Vulkan and CUDA?


title: Error Doing Inference with Huggingface model in PySpark

body:
I have a PySpark dataframe containing text and am trying to run a Huggingface model on it to get logits which I could then use downstream for classification. Here is the code I am using:
tokenizer = AutoTokenizer.from_pretrained("gpt2")
model = AutoModelWithLMHead.from_pretrained("gpt2")
str2tokens = udf(lambda s: tokenizer.tokenize(s, return_tensors='pt'), ArrayType(StringType()))
tokens2ids = udf(lambda s: tokenizer.convert_tokens_to_ids(s), ArrayType(IntegerType()))
vectorize_ids = udf(lambda s: list(model(tensor(s)).logits.mean(axis=0).detach().numpy()), ArrayType(DoubleType()))
df = spark.read.json("path/to/data").sample(False, 0.001)
df = df.withColumn('tokens', str2tokens('text'))
df = df.withColumn('ids', tokens2ids('tokens'))

Up until here, there is no error and the output is what you would expect (the text is from twitter):
Row(text='my timeline the writer of the hangover movies and an original snl castmember ', user_id=230*****, username='NU***', tokens=['my', 'Ġtimeline', 'Ġthe', 'Ġwriter', 'Ġof', 'Ġthe', 'Ġhang', 'over', 'Ġmovies', 'Ġand', 'Ġan', 'Ġoriginal', 'Ġsn', 'l', 'Ġcast', 'member', 'Ġ'], ids=[1820, 15264, 262, 6260, 286, 262, 8181, 2502, 6918, 290, 281, 2656, 3013, 75, 3350, 19522, 220])

Where the trouble I'm running into occurs is vectorizing the ids:
df = df.withColumn('logits', vectorize_ids('ids'))

Error:
    An error was encountered:
An error occurred while calling o197.collectToPython.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 9.0 failed 4 times, most recent failure: Lost task 0.3 in stage 9.0 (TID 260, ip-172-18-8-49.us-west-2.compute.internal, executor 1): net.razorvine.pickle.PickleException: expected zero arguments for construction of ClassDict (for numpy.dtype)
    at net.razorvine.pickle.objects.ClassDictConstructor.construct(ClassDictConstructor.java:23)
    at net.razorvine.pickle.Unpickler.load_reduce(Unpickler.java:707)
    at net.razorvine.pickle.Unpickler.dispatch(Unpickler.java:175)
    at net.razorvine.pickle.Unpickler.load(Unpickler.java:99)
    at net.razorvine.pickle.Unpickler.loads(Unpickler.java:112)
    at org.apache.spark.sql.execution.python.BatchEvalPythonExec$$anonfun$evaluate$1.apply(BatchEvalPythonExec.scala:90)
    at org.apache.spark.sql.execution.python.BatchEvalPythonExec$$anonfun$evaluate$1.apply(BatchEvalPythonExec.scala:89)
    at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:435)
    at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:441)
    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
    at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage2.processNext(Unknown Source)
    at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
    at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:585)
    at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:297)
    at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:289)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:858)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:858)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:310)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:310)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
    at org.apache.spark.scheduler.Task.run(Task.scala:123)
    at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1405)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

Driver stacktrace:
    at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:2043)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:2031)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:2030)
    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2030)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:967)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:967)
    at scala.Option.foreach(Option.scala:257)
    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:967)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2264)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2213)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2202)
    at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:778)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2061)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2082)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2101)
    at org.apache.spark.sql.execution.SparkPlan.executeTake(SparkPlan.scala:407)
    at org.apache.spark.sql.execution.CollectLimitExec.executeCollect(limit.scala:38)
    at org.apache.spark.sql.Dataset$$anonfun$collectToPython$1.apply(Dataset.scala:3267)
    at org.apache.spark.sql.Dataset$$anonfun$collectToPython$1.apply(Dataset.scala:3264)
    at org.apache.spark.sql.Dataset$$anonfun$52.apply(Dataset.scala:3370)
    at org.apache.spark.sql.execution.SQLExecution$.org$apache$spark$sql$execution$SQLExecution$$executeQuery$1(SQLExecution.scala:83)
    at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1$$anonfun$apply$1.apply(SQLExecution.scala:94)
    at org.apache.spark.sql.execution.QueryExecutionMetrics$.withMetrics(QueryExecutionMetrics.scala:141)
    at org.apache.spark.sql.execution.SQLExecution$.org$apache$spark$sql$execution$SQLExecution$$withMetrics(SQLExecution.scala:178)
    at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:93)
    at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:200)
    at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:92)
    at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3369)
    at org.apache.spark.sql.Dataset.collectToPython(Dataset.scala:3264)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    at py4j.Gateway.invoke(Gateway.java:282)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:238)
    at java.lang.Thread.run(Thread.java:748)
Caused by: net.razorvine.pickle.PickleException: expected zero arguments for construction of ClassDict (for numpy.dtype)
    at net.razorvine.pickle.objects.ClassDictConstructor.construct(ClassDictConstructor.java:23)
    at net.razorvine.pickle.Unpickler.load_reduce(Unpickler.java:707)
    at net.razorvine.pickle.Unpickler.dispatch(Unpickler.java:175)
    at net.razorvine.pickle.Unpickler.load(Unpickler.java:99)
    at net.razorvine.pickle.Unpickler.loads(Unpickler.java:112)
    at org.apache.spark.sql.execution.python.BatchEvalPythonExec$$anonfun$evaluate$1.apply(BatchEvalPythonExec.scala:90)
    at org.apache.spark.sql.execution.python.BatchEvalPythonExec$$anonfun$evaluate$1.apply(BatchEvalPythonExec.scala:89)
    at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:435)
    at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:441)
    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
    at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage2.processNext(Unknown Source)
    at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
    at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:585)
    at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:297)
    at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:289)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:858)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:858)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:310)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:310)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
    at org.apache.spark.scheduler.Task.run(Task.scala:123)
    at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1405)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    ... 1 more

Traceback (most recent call last):
  File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/dataframe.py", line 1256, in head
    rs = self.head(1)
  File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/dataframe.py", line 1258, in head
    return self.take(n)
  File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/dataframe.py", line 574, in take
    return self.limit(num).collect()
  File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/dataframe.py", line 535, in collect
    sock_info = self._jdf.collectToPython()
  File "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
    answer, self.gateway_client, self.target_id, self.name)
  File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 63, in deco
    return f(*a, **kw)
  File "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 328, in get_return_value
    format(target_id, ".", name), value)
py4j.protocol.Py4JJavaError: An error occurred while calling o197.collectToPython.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 9.0 failed 4 times, most recent failure: Lost task 0.3 in stage 9.0 (TID 260, ip-172-18-8-49.us-west-2.compute.internal, executor 1): net.razorvine.pickle.PickleException: expected zero arguments for construction of ClassDict (for numpy.dtype)
    at net.razorvine.pickle.objects.ClassDictConstructor.construct(ClassDictConstructor.java:23)
    at net.razorvine.pickle.Unpickler.load_reduce(Unpickler.java:707)
    at net.razorvine.pickle.Unpickler.dispatch(Unpickler.java:175)
    at net.razorvine.pickle.Unpickler.load(Unpickler.java:99)
    at net.razorvine.pickle.Unpickler.loads(Unpickler.java:112)
    at org.apache.spark.sql.execution.python.BatchEvalPythonExec$$anonfun$evaluate$1.apply(BatchEvalPythonExec.scala:90)
    at org.apache.spark.sql.execution.python.BatchEvalPythonExec$$anonfun$evaluate$1.apply(BatchEvalPythonExec.scala:89)
    at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:435)
    at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:441)
    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
    at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage2.processNext(Unknown Source)
    at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
    at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:585)
    at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:297)
    at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:289)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:858)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:858)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:310)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:310)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
    at org.apache.spark.scheduler.Task.run(Task.scala:123)
    at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1405)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

Driver stacktrace:
    at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:2043)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:2031)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:2030)
    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2030)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:967)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:967)
    at scala.Option.foreach(Option.scala:257)
    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:967)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2264)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2213)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2202)
    at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:778)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2061)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2082)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2101)
    at org.apache.spark.sql.execution.SparkPlan.executeTake(SparkPlan.scala:407)
    at org.apache.spark.sql.execution.CollectLimitExec.executeCollect(limit.scala:38)
    at org.apache.spark.sql.Dataset$$anonfun$collectToPython$1.apply(Dataset.scala:3267)
    at org.apache.spark.sql.Dataset$$anonfun$collectToPython$1.apply(Dataset.scala:3264)
    at org.apache.spark.sql.Dataset$$anonfun$52.apply(Dataset.scala:3370)
    at org.apache.spark.sql.execution.SQLExecution$.org$apache$spark$sql$execution$SQLExecution$$executeQuery$1(SQLExecution.scala:83)
    at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1$$anonfun$apply$1.apply(SQLExecution.scala:94)
    at org.apache.spark.sql.execution.QueryExecutionMetrics$.withMetrics(QueryExecutionMetrics.scala:141)
    at org.apache.spark.sql.execution.SQLExecution$.org$apache$spark$sql$execution$SQLExecution$$withMetrics(SQLExecution.scala:178)
    at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:93)
    at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:200)
    at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:92)
    at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3369)
    at org.apache.spark.sql.Dataset.collectToPython(Dataset.scala:3264)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    at py4j.Gateway.invoke(Gateway.java:282)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:238)
    at java.lang.Thread.run(Thread.java:748)
Caused by: net.razorvine.pickle.PickleException: expected zero arguments for construction of ClassDict (for numpy.dtype)
    at net.razorvine.pickle.objects.ClassDictConstructor.construct(ClassDictConstructor.java:23)
    at net.razorvine.pickle.Unpickler.load_reduce(Unpickler.java:707)
    at net.razorvine.pickle.Unpickler.dispatch(Unpickler.java:175)
    at net.razorvine.pickle.Unpickler.load(Unpickler.java:99)
    at net.razorvine.pickle.Unpickler.loads(Unpickler.java:112)
    at org.apache.spark.sql.execution.python.BatchEvalPythonExec$$anonfun$evaluate$1.apply(BatchEvalPythonExec.scala:90)
    at org.apache.spark.sql.execution.python.BatchEvalPythonExec$$anonfun$evaluate$1.apply(BatchEvalPythonExec.scala:89)
    at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:435)
    at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:441)
    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
    at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage2.processNext(Unknown Source)
    at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
    at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:585)
    at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:297)
    at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:289)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:858)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:858)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:310)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:310)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
    at org.apache.spark.scheduler.Task.run(Task.scala:123)
    at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1405)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    ... 1 more


title: Couple 2 different GPU cards for mini batching

body:
I just bought a GTX 1080Ti and I wanted to know if I can use both my old GTX 1070 and GTX 1080Ti in parallel for mini batching with either TensorFlow or PyTorch.

My main concern is:

Would the GTX 1070 bottleneck the GTX 1080Ti or the power of each cards will be used to their maximum?

I know that in a SLI configuration the amount of total VRAM will be equal to the card with the lowest amount (the GTX 1070 here with 8GB or VRAM) but does the same thing happen during training with tensorflow/pytorch with no SLI involved?


title: PyTorch Broadcasting Failing. "Rules" Followed

body:
I have a few PyTorch examples that confuse me and am hoping to clear it up.

Firstly, as per the PyTorch page, I would expect these example to work as do their numpy equivalents namely these. The first example is very intuitive. These are compatible for broadcasting:

Image  (3d array): 256 x 256 x 3
Scale  (1d array):             3
Result (3d array): 256 x 256 x 3

Take for instance these:

torch.Tensor([[1,2,3]])/torch.Tensor([1,2,3])
Out[5]:

 1  1  1
[torch.FloatTensor of size 1x3]
torch.Tensor([[1,2,3]])/torch.Tensor([1,2,3])
Out[6]:

  1  1  1
[torch.FloatTensor of size 1x3]

torch.Tensor([[[1,2,3]]])/torch.Tensor([1,2,3])
Out[7]:

(0 ,.,.) =
  1  1  1
[torch.FloatTensor of size 1x1x3]

However, this is the result of the numpy example:

torch.randn(256,256,3)/torch.Tensor([1,2,3])
Traceback (most recent call last):

  File "<ipython-input-12-4c328d453e24>", line 1, in <module>
    torch.randn(256,256,3)/torch.Tensor([1,2,3])

  File "/home/nick/anaconda3/lib/python3.6/site-packages/torch/tensor.py", line 313, in __div__
    return self.div(other)

RuntimeError: inconsistent tensor size at /opt/conda/conda-bld/pytorch_1501971235237/work/pytorch-0.1.12/torch/lib/TH/generic/THTensorMath.c:873

Here is an excerpt which says this ought to work:

  Two tensors are “broadcastable” if the following rules hold:

  Each tensor has at least one dimension.
  When iterating over the dimension sizes, starting at the trailing dimension, the dimension sizes must either be equal, one of them is 1, or one of them does not exist.

If one converts the tensors to numpy arrays the arithmetic works as expected.

What's going wrong? Have I misinterpreted the docs and if so what syntax results in the same results?


title: How to import Python package (Pytorch-neat) that is not installable from pip/conda repositories?

body:
I am trying to used Pytorch-neat package https://github.com/uber-research/PyTorch-NEAT but I don't understand the workflow of using it. I already installed python-neat package and I can import it using import neat in my Jupyter notebook. But what should I do with Pytroch-neat code? There is no pytorch-neat package in Conda or pip repositories, so, I guess that this Pytroch-neat code is not compiled and distributed as the Python package for Jupyter notebook. But what should I do with this code? E.g. sample script contains the code:

import neat

from pytorch_neat.multi_env_eval import MultiEnvEvaluator

So - neat is package and I am importing it. But how to understand the from clause? Should I load Pytroch-neat scripts somehow in the previous cells of my notebook and then I can call this from clause? Or maybe I should compile Pytroch-neat package locally and install it from local repository and import it similarly to neat package. But if so, why the examples use from clause?

I am starting to use Python and I am greatly confused with all of this!


title: PyTorch and Chainer implementations of the Linear layer- are they equivalent?

body:
I want to use a Linear, Fully-Connected Layer as one of the input layers in my network. The input has shape (batch_size, in_channels, num_samples). It is based on the Tacotron paper: https://arxiv.org/pdf/1703.10135.pdf, the Enocder prenet part.
It feels to me as if Chainer and PyTorch have different implementations of the Linear layer - are they really performing the same operations or am I misunderstanding something?
In PyTorch, behavior of the Linear layer follows the documentations: https://pytorch.org/docs/0.3.1/nn.html#torch.nn.Linear
according to which, the shape of the input and output data are as follows:

Input: (N,∗,in_features) where * means any number of additional dimensions
Output: (N,∗,out_features) where all but the last dimension are the same shape as the input.

Now, let's try creating a linear layer in pytorch and performing the operation. I want an output with 8 channels, and the input data will have 3 channels.
import numpy as np
import torch
from torch import nn
linear_layer_pytorch = nn.Linear(3, 8)

Let's create some dummy input data of shape (1, 4, 3) - (batch_size, num_samples, in_channels:
data = np.array([1, 1, 1, 2, 2, 2, 3, 3, 3, 4, 4, 4], dtype=np.float32).reshape(1, 4, 3)
data_pytorch = torch.from_numpy(data)

and finally, perform the operation:
results_pytorch = linear_layer_pytorch(data_pytorch)
results_pytorch.shape

the shape of the output is as follows: Out[27]: torch.Size([1, 4, 8])
Taking a look at the source of the PyTorch implementation:
def linear(input, weight, bias=None):
    # type: (Tensor, Tensor, Optional[Tensor]) -> Tensor
    r"""
    Applies a linear transformation to the incoming data: :math:`y = xA^T + b`.

    Shape:

        - Input: :math:`(N, *, in\_features)` where `*` means any number of
          additional dimensions
        - Weight: :math:`(out\_features, in\_features)`
        - Bias: :math:`(out\_features)`
        - Output: :math:`(N, *, out\_features)`
    """
    if input.dim() == 2 and bias is not None:
        # fused op is marginally faster
        ret = torch.addmm(bias, input, weight.t())
    else:
        output = input.matmul(weight.t())
        if bias is not None:
            output += bias
        ret = output
    return ret

It transposes the weight matrix that is passed to it, broadcasts it along the batch_size axis and performs a matrix multiplications. Having in mind how a linear layer works, I imagine it as 8 nodes, connected through a synapse, holding a weight, with every channel in an input sample, thus in my case it has 3*8 weights. And that is exactly the shape I see in debugger (8, 3).
Now, let's jump to Chainer. The Chainer's linear layer documentation is available here: https://docs.chainer.org/en/stable/reference/generated/chainer.links.Linear.html#chainer.links.Linear. According to this documentation, the Linear layer wraps the function linear, which according to the docs, flattens the input along the non-batch dimensions and the shape of it's weight matrix is (output_size, flattend_input_size)
import chainer
linear_layer_chainer = chainer.links.Linear(8)
results_chainer = linear_layer_chainer(data)
results_chainer.shape
Out[21]: (1, 8)

Creating the layer as linear_layer_chainer = chainer.links.Linear(3, 8) and calling it causes a size mismatch. So in case of chainer, I have gotten a totally different results, because this time around I have a weight matrix that is of shape (8, 12) and my results have a shape of (1, 8). So now, here is my question : since the results are clearly different,both the weight matrices and the outputs have different shapes, how can I make them equivalent and what should be the desired output? In the PyTorch implementation of Tacotron it seems that the PyTorch approach is used as is (https://github.com/mozilla/TTS/blob/master/layers/tacotron.py) - Prenet. If that is the case, how can I make the Chainer produce the same results (I have to implement this in Chainer). I will be grateful for any inshight, sorry that the post has gotten this long.


title: How to parallelize regularization loss calculation on multiple GPUs using pytorch?

body:
After wrapping my model in DataParallel

model20 = FourLayerConvNetWithPool()
model20 = nn.DataParallel(model20)

I can see utilization of all available GPUs. But after closer inspection noticed that "cuda:0" still has higher utilization (and as a result is a bottleneck {potentially minor for huge networks}).

Here is a snippet of training iteration:

model.train()  # put model to training mode
x = x.to(device=self.device, dtype=self.dtype)  # move to device, e.g. GPU
y = y.to(device=self.device, dtype=torch.long)

scores = model(x)
loss = F.cross_entropy(scores, y)

if (reg > 0):
    l2_regularization = torch.tensor(0).to(device=self.device, dtype=self.dtype)
    for param in model.parameters():
        l2_regularization += torch.norm(param, 2)
    loss += reg * l2_regularization

# Zero out all of the gradients for the variables which the optimizer
# will update.
optimizer.zero_grad()

# This is the backwards pass: compute the gradient of the loss with
# respect to each  parameter of the model.
loss.backward()

# Actually update the parameters of the model using the gradients
# computed by the backwards pass.
optimizer.step()

By commenting various parts of above snippet found that F.cross_entropy and torch.norm contribute to the difference between "cuda:0" and "cuda:1". This makes sense because these parts are not parallelized.

I understand that I can move F.cross_entropy to my model interface.

But what can be done with L2 regularization loss?


title: cosine similarity for attention decoder in nmt

body:
I am implementing a neural machine translation model, and for the decoder part (with attention mechanism) I would like to calculate the cosine similarity for finding the scores.
Here is the function:

score(a,b) =  / ||a|| ||b||

In my case:

        a = htilde_t (N, H)

        b = h (S, N, H)

        the output should be (S, N)

My confusion is about their dimensions and I don't know how to solve that in pytorch.


title: How to visualize a torch_geometric graph in Python?

body:
Let's consider as an example that I have the following adjacence matrix in coordinate format:

> edge_index.numpy() = array([[    0,     1,     0,   3,   2],
                              [    1,     0,     3,   2,   1]], dtype=int64)

which means that the node 0 is linked toward the node 1, and vice-versa, the node 0 is linked to 3 etc...

Do you know a way to draw this graph as in networkx with nx.draw() ?

Thank you.


title: PyTorch save prediction as CSV for each epoch

body:
I'm trying save the predictions I am getting from a model in PyTorch as csv. The following code. However, the code seem to overwrite the prediction at each epoch and the final .csv file only contains 3 values. Any idea what I am doing wrong please:
output = model(images)
preds = output.sum(dim=[1,2,3])
np.savetxt('preds.csv', preds.detach())


title: Is there something like torch.utils.data.TensorDataset(*tensors) for TensorFlow/Keras?

body:
I´m "translating" a notebook made in Pytorch to one made in Keras. And they use that app to pack the data from a tensor into the dataset that will be used for the network. But I can't find something that fulfills that function. I would greatly appreciate the help!
Pytorch documentation says that torch.utils.data.TensorDataset (* tensors) does:
"Dataset wrapping tensors. Each sample will be retrieved by indexing Tensor a along the first dimension."
Thank you everybody!


I just start to learn about the YOLO v5 PyTorch version and I was able to build a model, so then I tried to implement a flask application for real-time prediction using this trained model.
class for load model and predict
class Model(object):

    def __init__(self, model):

        self.device = torch_utils.select_device()
        print(self.device)
        model = torch.load(model, map_location=self.device)['model']

        self.half = False and self.device.type != 'cpu'
        print('half = ' + str(self.half))

        if self.half:
            model.half()

        # model  = model.to(self.device).eval()
        model.cuda()

        self.loaded_model = model

    def predict(self, img):
        global session
        # img1 = torch.from_numpy(img).to(self.device)
        # img = img1.reshape(1, 3, 640, 640)
        img = img.half() if self.half else img.float()  # uint8 to fp16/32
        img /= 255.0  # 0 - 255 to 0.0 - 1.0
        print(img.ndimension())
        if img.ndimension() == 3:
            img = img.unsqueeze(0)
        print(self.loaded_model)
        img = img.to(self.device)
        # img = img.half()
        self.preds = self.loaded_model(img, augment=False)[0]
        print(self.predict())
        return self.preds

Camera class for reading frames from camera or video
model = Model("weights/best.pt")
class Camera(object):
    def __init__(self):
        # self.video = cv2.VideoCapture('facial_exp.mkv')
        self.video = cv2.VideoCapture(0)

    def __del__(self):
        self.video.release()

    def get_frame(self):
        _, fr = self.video.read()
        loader = transforms.Compose([transforms.ToTensor()])

        image = cv2.resize(fr, (640, 640), interpolation=cv2.INTER_AREA)
        input_im = image.reshape(1, 640, 640, 3)

        pil_im = Image.fromarray(fr)
        image = loader(pil_im).float()
        # image = Variable(image, requires_grad=True)
        image = image.unsqueeze(0)

        pred = model.predict(input_im)
        pred = model.predict(image)
        print(pred)

        _, jpeg = cv2.imencode('.jpg', fr)
        return jpeg.tobytes()

Some of the commented lines are the ways which I tried but in all times bellow line
self.preds = self.loaded_model(img, augment=False)[0]  throws below error
RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.cuda.HalfTensor) should be the same

any idea or guidance for solving this error thank you.


AttributeError: 'str' object has no attribute 'shape' while encoding tensor using BertModel with PyTorch (Hugging Face). Below is the code
bert_model = BertModel.from_pretrained(r'downloads\bert-pretrained-model')
input_ids

Output is:
tensor([[  101,   156, 13329,  ...,     0,     0,     0],
        [  101,   156, 13329,  ...,     0,     0,     0],
        [  101,  1302,  1251,  ...,     0,     0,     0],
        ...,
        [  101, 25456,  1200,  ...,     0,     0,     0],
        [  101,   143,  9664,  ...,     0,     0,     0],
        [  101,  2586,  7340,  ...,     0,     0,     0]])

Followed by code below
last_hidden_state, pooled_output = bert_model(
  input_ids=encoding['input_ids'],
  attention_mask=encoding['attention_mask']
)

Followed by code below
last_hidden_state.shape

Output is
AttributeError                            Traceback (most recent call last)
<ipython-input-70-9628339f425d> in <module>
----> 1 last_hidden_state.shape

AttributeError: 'str' object has no attribute 'shape'

Complete Code link is 'https://colab.research.google.com/drive/1FY4WtqCi2CQ9RjHj4slZwtdMhwaWv2-2?usp=sharing'


title: why Gradient Descent doesn't work as expected with pytorch

body:
so I'm starting with Pytorch and tried to start with an easy Linear Regression Example. Actually I made an easy Implementation of Linear Regression with Pytorch to calculate the equation 2*x+1 but the loss stay stuck at 120 and there is a Problem with Gradient Descent because it doesn't converge to a small loss value. I don't know why this is happening and it made me crazy because I don't see what's wrong. actually this example should be very easy to solve. this is the Code I'm using

import torch
import torch.nn.functional as F
from torch.utils.data import TensorDataset, DataLoader
import numpy as np

X = np.array([i for i in np.arange(1, 20)]).reshape(-1, 1)
X = torch.tensor(X, dtype=torch.float32, requires_grad=True)
y = np.array([2*i+1 for i in np.arange(1, 20)]).reshape(-1, 1)
y = torch.tensor(y, dtype=torch.float32, requires_grad=True)
print(X.shape, y.shape)

class LR(torch.nn.Module):
    def __init__(self, n_features, n_hidden1, n_out):
        super(LR, self).__init__()
        self.linear = torch.nn.Linear(n_features, n_hidden1)

        self.predict = torch.nn.Linear(n_hidden1, n_out)

    def forward(self, x):
        x = F.relu(self.linear(x))
        x = self.predict(x)
        return x

model = LR(1, 10, 1)
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
loss_fn = torch.nn.MSELoss()

def train(epochs=100):
    for e in range(epochs):

        pred = model(X)
        loss = loss_fn(pred, y)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        print(f"epoch: {e} and loss= {loss}")

desired output is a small loss value and that the model train to give a good prediction later.


title: softmax dims and variable volatile in PyTorch

body:
I have a code for previous version of PyTorch and I receive 2 warning for the 3nd line of it:

import torch.nn.functional as F

def select_action(self, state):
        probabilities = F.softmax(self.model(Variable(state, volatile = True))*100) # T=100
        action = probs.multinomial(num_samples=1)
        return action.data[0,0]

  UserWarning: volatile was removed and now has no effect. Use with
  torch.no_grad(): instead.

  UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X > as an argument.

I found that:

  Volatile is recommended for purely inference mode, when you’re sure
  you won’t be even calling .backward(). It’s more efficient than any
  other autograd setting - it will use the absolute minimal amount of
  memory to evaluate the model. volatile also determines that
  requires_grad is False.

Am I right that I should just remove it?
And because I want to get probabilities therefore should I use dim=1 ?
and the 3nd line of my code should look like:

    probabilities = F.softmax(self.model(Variable(state), dim=1)*100) # T=100

state is created here:

def update(self, reward, new_signal):
   new_state = torch.Tensor(new_signal).float().unsqueeze(0)
   self.memory.push((self.last_state, new_state, torch.LongTensor([int(self.last_action)]), torch.Tensor([self.last_reward])))
   action = self.select_action(new_state)
   if len(self.memory.memory) > 100:
       batch_state, batch_next_state, batch_action, batch_reward = self.memory.sample(100)
       self.learn(batch_state, batch_next_state, batch_reward, batch_action)
   self.last_action = action
   self.last_state = new_state
   self.last_reward = reward
   self.reward_window.append(reward)
   if len(self.reward_window) > 1000:
       del self.reward_window[0]
   return action


title: Pytorch Loaded model giving inconsistent results

body:
I am playing around with code from this Github repository https://github.com/jindongwang/Pytorch-CapsuleNet.

After training the model for 5 epochs, I got an accuracy of 99.2% on the test dataset. So I saved model using the following code:

torch.save(capsule_net.state_dict(),"capsnet_mnist_state.pt")

I tried loading the model back in another machine with the below code:

capsnet = CapsNet(Config())
capsnet.load_state_dict(torch.load('capsnet_mnist_state.pt'))
capsnet.eval()

Now the model predicts the 0 has the output for every input. Is there anything wrong with way I saved the model or loaded the model?.


title: Pytorch/Ktrain import name BartForConditionalGeneration from transformers error?

body:
I am trying to use the package ktrain's text function

 text.TransformerSummarizer()

However, when I try to use it I get the error that

 TransformerSummarizer() requires PyTorch to be installed.

I installed PyTorch using different variations of conda and pip, as well as the command given on the pytorch website. After it installs and I try to use it I get:

from ktrain import text
import torch
from transformers import BartForConditionalGeneration

ts = text.TransformerSummarizer()

ImportError: cannot import name 'BartForConditionalGeneration' from 'transformers' (C:\Users\user\AppData\Roaming\Python\Python37\site-packages\transformers\__init__.py)

Any suggestions of how to fix this? Thank you!


title: Problem of exporting batchnorm weight from pytorch to Keras

body:
I follow Pytorch Batchnorm layer different from Keras Batchnorm,
Pytorch Batchnorm implementation, but they do not solve my problem.
I also read Wiki about Batchnorm.
And search source code from tensorflow batchnorm and from pytorch source code.
Below is my testing code, and the results between pytorch and keras are different in 1e-2 to 1e-3 order of error which is big. Function b0, b1 similar to torch result but still not quite accurate.
And b2 tries to follow the formula used in tensorflow batchnorm.
Convolution part yields the same result, but I am stucking at batchnorm layer. I also use eval(), no_grad() for pytorch and model.predict for keras model to make sure they are in inference stage.
Tensorflow implementation does not use 1/sqrt(var+eps), but sqrt(var+eps) instead. I try to transfer 1/running_var to keras.BN.moving_var but still fail the result.
import tensorflow as tf
import tensorflow.keras.layers as L
from tensorflow.keras import Model as KModel
import torch.nn as nn
import torch

def KM():
    x = L.Input((None,None,3))
    y0 = L.Concatenate(axis=-1)([x[:,::2,::2,:],x[:,::2,1::2,:],x[:,1::2,::2,:],x[:,1::2,1::2,:]])
    y1 = L.Conv2D(32,3,1,"same",use_bias=False)(y0)
    y2 = L.BatchNormalization()(y1)
    y3 = L.LeakyReLU(0.1)(y2)
    return KModel(x, [y1, y2, y3])

class YM(nn.Module):
    def __init__(self):
        super(YM, self).__init__()
        self.cat = lambda x : torch.cat([x[:,:,::2,::2],x[:,:,::2,1::2],x[:,:,1::2,::2],x[:,:,1::2,1::2]],axis=1)
        self.conv = nn.Conv2d(12,32,3,1,1,bias=False)
        self.bn = nn.BatchNorm2d(32)
        self.act = nn.LeakyReLU(0.1)

    def forward(self, x):
        y0 = ym.cat(x)
        y0 = ym.conv(y0)
        y1 = ym.bn(y0)
        y2 = ym.act(y1)
        return [y0, y1, y2]

np.random.seed(0)
img = np.random.randint(0,255,(1,12,14,3)).astype(np.float32)
img_torch = torch.from_numpy(img.transpose(0,3,1,2).astype(np.float32))
w1 = np.random.rand(32,12,3,3).astype(np.float32)*0.1
bw1 = np.random.rand(32).astype(np.float32)*0.1
bb1 = np.random.rand(32).astype(np.float32)
bm1 = np.random.rand(32).astype(np.float32)
bv1 = np.abs(np.random.rand(32).astype(np.float32))*0.1

ym = YM()
km = KM()

ym.conv.weight = nn.Parameter(torch.from_numpy(w1))
ym.bn.weight = nn.Parameter(torch.from_numpy(bw1))
ym.bn.bias = nn.Parameter(torch.from_numpy(bb1))
ym.bn.running_mean = torch.from_numpy(bm1)
ym.bn.running_var = torch.from_numpy(bv1)

km.layers[6].set_weights([w1.transpose(2,3,1,0)])
km.layers[7].set_weights([bw1, bb1, bm1, bv1])

ym.eval()
ym.bn.track_running_stats = True
with torch.no_grad():
    t0 = Ym(ym, img_torch/255.-0.5)
k0 = km.predict(img/255.-0.5)

for i in range(len(t0)):
    print(t0[i].shape, k0[i].shape)

Key = 1
print(t0[Key][0,0,:,:].detach().numpy())
print(k0[Key][0,:,:,0])

>>>>>>>>>>>
[[    0.71826     0.72964     0.73189     0.70224     0.74954     0.72928      0.7524]
 [    0.71305     0.68717     0.68581      0.7242     0.73491     0.71925     0.70781]
 [    0.70145     0.66769      0.6857     0.70804     0.73533     0.73165     0.72006]
 [     0.6758     0.69231     0.71173     0.71325     0.72097     0.71414     0.75782]
 [    0.68255     0.72283     0.71273      0.7226     0.71788     0.68119     0.72556]
 [    0.70452     0.68088     0.74389     0.73558     0.72853      0.7174     0.74389]]
[[    0.71953     0.73082     0.73306     0.70365     0.75056     0.73046     0.75339]
 [    0.71437      0.6887     0.68736     0.72543     0.73605     0.72052     0.70918]
 [    0.70287     0.66939     0.68724      0.7094     0.73647     0.73282     0.72133]
 [    0.67743      0.6938     0.71306     0.71457     0.72223     0.71545     0.75877]
 [    0.68413     0.72407     0.71405     0.72384     0.71916     0.68278     0.72678]
 [    0.70592     0.68246     0.74495     0.73671     0.72972     0.71868     0.74496]]```

tt = t0[Key].detach().numpy().transpose(0,2,3,1)
kk = k0[Key]
np.abs(tt-kk).max()
>>>>>>>>>>
0.078752756

gamma, beta = bw1[0], bb1[0]
mu, var = bm1[0], bv1[0]
x_p = t0[0][0,0,0,0]

print(gamma,beta,mu,var,x_p)

eps = 1e-10
def bn0(x_p, mu, var, gamma, beta):
    # wiki
    xhat = (x_p - mu)/np.sqrt(var + eps)
    _x = xhat * gamma + beta
    return _x

def bn1(x_p, mu, var, gamma, beta):
    # pytorch cpp
    inv_var = 1/ np.sqrt(var + eps)
    alpha_d = gamma * inv_var
    beta_d = beta - mu * inv_var * gamma
    return x_p * alpha_d + beta_d

def bn2(x_p, mu, var, gamma, beta):
    # tensorflow cpp
    inv_var = np.sqrt(var + eps)
    xhat = (x_p - mu)*inv_var
    _x = xhat * gamma + beta
    return _x
print(bn0(x_p, mu, var, gamma, beta))
print(bn1(x_p, mu, var, gamma, beta))
print(bn2(x_p, mu, var, gamma, beta))
print(bn2(x_p, mu, 1/var, gamma, beta))

>>>>>>>>
0.048011426 0.87305844 0.67954195 0.059197646 tensor(-0.26256)
tensor(0.68715)
tensor(0.68715)
tensor(0.86205)
tensor(0.68715)


title: What should be the Query Q, Key K and Value V vectors/matrics in torch.nn.MultiheadAttention?

body: Following an amazing blog, I implemented my own self-attention module. However, I found PyTorch has already implemented a multi-head attention module. The input to the forward pass of the MultiheadAttention module includes Q (which is query vector) , K (key vector), and V (value vector). It is strange that PyTorch wouldn't just take the input embedding and compute the Q, K, V vectors on the inside. In the self-attention module that I implemented, I compute this Q, K, V vectors from the input embeddings multiplied by the Q, K, V weights. At this point, I am not sure what the Q, K, and V vector inputs that MultiheadAttention module requires. Should they be Q, K, and V weights or vectors and should these be normal vectors, or should these be Parameters?


title: Is max operation differentiable in Pytorch?

body:
I am using Pytorch to training some neural networks. The part I am confused about is:
prediction = myNetwork(img_batch)
max_act = prediction.max(1)[0].sum()
loss = softcrossentropy_loss - alpha * max_act

In the above codes, "prediction" is the output tensor of "myNetwork".
I hope to maximize the larget output of "prediction" over a batch.
For example:
[[-1.2, 2.0, 5.0, 0.1, -1.5] [9.6, -1.1, 0.7, 4,3, 3.3]]
For the first prediction vector, the 3rd element is the larget, while for the second vector, the 1st element is the largets. And I want to maximize "5.0+9.6", although we cannot know what index is the larget output for a new input data.
In fact, my training seems to be successful, because the "max_act" part was really increased, which is the desired behavior to me. However, I heard some discussion about whether max() operation is differentiable or not:
Some says, mathmatically, max() is not differentiable.
Some says, max() is just an identity function to select the largest element, and this largest element is differentiable.

So I got confused now, and I am worried if my idea of maximizing "max_act" is wrong from the beginning.
Could someone provide some guidance if max() operation is differentiable in Pytorch?


title: Does using FP16 help accelerate generation? (HuggingFace BART)

body:
I follow the guide below to use FP16 in PyTorch.
https://pytorch.org/blog/accelerating-training-on-nvidia-gpus-with-pytorch-automatic-mixed-precision/
Basically, I'm using BART in HuggingFace for generation

During the training phase, I'm able to get 2x speedup and less GPU memory consumption

But.

I found out there is no speedup when I call model.generate under torch.cuda.amp.autocast().

with torch.cuda.amp.autocast():
   model.generate(...)

When I save the model by:

model.save_pretrained("model_folder")

the size does not decrease to half. But I have to call model.half() before saving in order to make the model half size.
Thus, my questions:

Is the issue in 1. expected or there should be something I did wrong?
Is the operation I did in 2. proper?


title: Docker python error. urllib.error.URLError:

body:
I was trying to run a script in a docker container and got this.
Traceback (most recent call last):
  File "experiments/caffemodel2pytorch.py", line 387, in <module>
    net_param = initialize(args.caffe_proto).NetParameter()
  File "experiments/caffemodel2pytorch.py", line 35, in initialize
    mybytes = urlopen(caffe_proto).read()
  File "/usr/lib/python3.6/urllib/request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.6/urllib/request.py", line 526, in open
    response = self._open(req, data)
  File "/usr/lib/python3.6/urllib/request.py", line 544, in _open
    '_open', req)
  File "/usr/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.6/urllib/request.py", line 1361, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/usr/lib/python3.6/urllib/request.py", line 1320, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 99] Cannot assign requested address>

Part of the code is here.
def initialize(caffe_proto = 'https://raw.githubusercontent.com/BVLC/caffe/master/src/caffe/proto/caffe.proto', codegen_dir = tempfile.mkdtemp(), shadow_caffe = True):
    global caffe_pb2
    if caffe_pb2 is None:
        local_caffe_proto = os.path.join(codegen_dir, os.path.basename(caffe_proto))
        with open(local_caffe_proto, 'w') as f:
            mybytes = urlopen(caffe_proto).read()
            mystr = mybytes.decode('ascii', 'ignore')
            f.write(mystr)
            #f.write((urlopen if 'http' in caffe_proto else open)(caffe_proto).read())
        subprocess.check_call(['protoc', '--proto_path', os.path.dirname(local_caffe_proto), '--python_out', codegen_dir, local_caffe_proto])
        sys.path.insert(0, codegen_dir)
        old_pool = google.protobuf.descriptor._message.default_pool
        old_symdb = google.protobuf.symbol_database._DEFAULT
        google.protobuf.descriptor._message.default_pool = google.protobuf.descriptor_pool.DescriptorPool()
        google.protobuf.symbol_database._DEFAULT = google.protobuf.symbol_database.SymbolDatabase(pool = google.protobuf.descriptor._message.default_pool)
        import caffe_pb2 as caffe_pb2
        google.protobuf.descriptor._message.default_pool = old_pool
        google.protobuf.symbol_database._DEFAULT = old_symdb
        sys.modules[__name__ + '.proto'] = sys.modules[__name__]
        if shadow_caffe:
            sys.modules['caffe'] = sys.modules[__name__]
            sys.modules['caffe.proto'] = sys.modules[__name__]
    return caffe_pb2

I think it has something to do with the urlopen, but i don't know how to fix this in a docker container, any help would be appreciated, plz.
BTW, I start the container like this:
sudo nvidia-docker run -itv /home/ljh/mobilepose:/home/mobilepose -p 7777:8888 ufoym/deepo:all-py36-jupyter /bin/bash


title: Pytorch LSTM each epoch starts from 0 accuracy

body:
I'm training a LSTM model for time series prediction and at each epoch my accuracy restarts from 0 as if I'm training for the first time.

I attach below the training method snippet:

def train(model, loader, epoch, mini_batch_size, sequence_size):
model.train()
correct = 0
padded_size = 0
size_input = mini_batch_size * sequence_size
for batch_idx, (inputs, labels, agreement_score) in enumerate(loader):

    if(inputs.size(0) == size_input):
        inputs = inputs.clone().reshape(mini_batch_size, sequence_size, inputs.size(1))
        labels = labels.clone().squeeze().reshape(mini_batch_size*sequence_size)
        agreement_score = agreement_score.clone().squeeze().reshape(mini_batch_size*sequence_size)
    else:
        padded_size = size_input - inputs.size(0)
        (inputs, labels, agreement_score) = padd_incomplete_sequences(inputs, labels, agreement_score, mini_batch_size, sequence_size)

    inputs, labels, agreement_score = Variable(inputs.cuda()), Variable(labels.cuda()), Variable(agreement_score.cuda())

    output = model(inputs)

    loss = criterion(output, labels)
    loss = loss * agreement_score
    loss = loss.mean()

    optimizer.zero_grad()
    loss.backward()

    optimizer.step()

    pred = output.data.max(1, keepdim = True)[1]
    correct += pred.eq(labels.data.view_as(pred)).cuda().sum()

    accuracy = 100. * correct / (len(loader.dataset) + padded_size)

    print("Train: Epoch: {}, [{}/{} ({:.0f}%)]\t loss: {:.6f}, Accuracy: {}/{} ({:.0f}%)".format(
        epoch,
        batch_idx * len(output),
        (len(loader.dataset) + padded_size),
        100. * batch_idx / (len(loader.dataset)+padded_size),
        loss.item(),
        correct,
        (len(loader.dataset) + padded_size),
        accuracy))

accuracy = 100. * correct / (len(loader.dataset) + padded_size)

train_accuracy.append(accuracy)
train_epochs.append(epoch)
train_loss.append(loss.item())

According to that my loop looks like:

for epoch in range(1, 10):
    train(audio_lstm_model, train_rnn_audio_loader, epoch, MINI_BATCH_SIZE, SEQUENCE_SIZE_AUDIO)
    evaluation(audio_lstm_model,validation_rnn_audio_loader, epoch, MINI_BATCH_SIZE, SEQUENCE_SIZE_AUDIO)

Consequently, my accuracy and loss restarts at every epoch:

Train: Epoch: 1, [0/1039079 (0%)]    loss: 0.921637, Accuracy: 0/1039079 (0%)
...
Train: Epoch: 1, [10368/1039079 (0%)]    loss: 0.523242, Accuracy: 206010/1039079 (19%)
Test set: loss: 151.4845, Accuracy: 88222/523315 (16%)
Train: Epoch: 2, [0/1039079 (0%)]    loss: 0.921497, Accuracy: 0/1039079 (0%)

If anyone has any clue about it, your help is welcomed!
Have a nice day!


title: Auto-encoder for vector encodings

body:
Here is an autoencoder I wrote to encode two vectors :

[1,2,3] & [1,2,3]

The vectors are created in : features = torch.tensor(np.array([ [1,2,3],[1,2,3] ]))

This works as per code :

%reset -f

epochs = 1000
from pylab import plt
plt.style.use('seaborn')
import torch.utils.data as data_utils
import torch
import torchvision
import torch.nn as nn
from torch.autograd import Variable

cuda = torch.cuda.is_available()
FloatTensor = torch.cuda.FloatTensor if cuda else torch.FloatTensor
import numpy as np
import pandas as pd
import datetime as dt

features = torch.tensor(np.array([ [1,2,3],[1,2,3] ]))

print(features)

batch = 2
data_loader = torch.utils.data.DataLoader(features, batch_size=2, shuffle=True)

encoder = nn.Sequential(nn.Linear(3,batch), nn.Sigmoid())
decoder = nn.Sequential(nn.Linear(batch,3), nn.Sigmoid())
autoencoder = nn.Sequential(encoder, decoder)

optimizer = torch.optim.Adam(params=autoencoder.parameters(), lr=0.001)

encoded_images = []
for i in range(epochs):
    for j, (images, _) in enumerate(data_loader):
    #     images = images.view(images.size(0), -1)
        images = Variable(images).type(FloatTensor)
        optimizer.zero_grad()
        reconstructions = autoencoder(images)
        loss = torch.dist(images, reconstructions)
        loss.backward()
        optimizer.step()

        encoded_images.append(encoder(images))

But when I add a new vector :

features = torch.tensor(np.array([ [1,2,3],[1,2,3],[1,2,3] ]))

I receive error :

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-223-3ca45519e975> in <module>
     32 encoded_images = []
     33 for i in range(epochs):
---> 34     for j, (images, _) in enumerate(data_loader):
     35     #     images = images.view(images.size(0), -1)
     36         images = Variable(images).type(FloatTensor)

ValueError: not enough values to unpack (expected 2, got 1)

Have I setup my data loader correctly ?


title: Getting 5 random crops - TypeError: pic should be PIL Image or ndarray. Got

body:
I do transformations on images as below (which works with RandCrop): (it is from this dataloader script: https://github.com/jeffreyhuang1/two-stream-action-recognition/blob/master/dataloader/motion_dataloader.py)

def train(self):
training_set = motion_dataset(dic=self.dic_video_train, in_channel=self.in_channel, root_dir=self.data_path,
mode=‘train’,
transform = transforms.Compose([
transforms.Resize([256,256]),
transforms.FiveCrop([224, 224]),
#transforms.RandomCrop([224, 224]),
transforms.ToTensor(),
#transforms.Normalize([0.5], [0.5])
]))
print ‘==> Training data :’,len(training_set),’ videos’,training_set[1][0].size()

    train_loader = DataLoader(
        dataset=training_set,
        batch_size=self.BATCH_SIZE,
        shuffle=True,
        num_workers=self.num_workers,
        pin_memory=True
        )

    return train_loader

But when I do try to get Five Crops, I get this error:

Traceback (most recent call last):
File “motion_cnn.py”, line 267, in
main()
File “motion_cnn.py”, line 51, in main
train_loader,test_loader, test_video = data_loader.run()
File “/media/d/DATA_2/two-stream-action-recognition-master/dataloader/motion_dataloader.py”, line 120, in run
train_loader = self.train()
File “/media/d/DATA_2/two-stream-action-recognition-master/dataloader/motion_dataloader.py”, line 156, in train
print ‘==> Training data :’,len(training_set),’ videos’,training_set[1][0].size()
File “/media/d/DATA_2/two-stream-action-recognition-master/dataloader/motion_dataloader.py”, line 77, in getitem
data = self.stackopf()
File “/media/d/DATA_2/two-stream-action-recognition-master/dataloader/motion_dataloader.py”, line 51, in stackopf
H = self.transform(imgH)
File “/media/d/DATA_2/two-stream-action-recognition-master/venv/local/lib/python2.7/site-packages/torchvision/transforms/transforms.py”, line 60, in call
img = t(img)
File “/media/d/DATA_2/two-stream-action-recognition-master/venv/local/lib/python2.7/site-packages/torchvision/transforms/transforms.py”, line 91, in call
return F.to_tensor(pic)
File “/media/d/DATA_2/two-stream-action-recognition-master/venv/local/lib/python2.7/site-packages/torchvision/transforms/functional.py”, line 50, in to_tensor
raise TypeError(‘pic should be PIL Image or ndarray. Got {}’.format(type(pic)))
TypeError: pic should be PIL Image or ndarray. Got <type ‘tuple’>

Getting 5 random crops, I should handle a tuple of images instead of a PIL image - so I use Lambda, but then I get the error, at line 55, in stackopf
flow[2*(j),:,:] = H

  RuntimeError: expand(torch.FloatTensor{[5, 1, 224, 224]}, size=[224,
  224]): the number of sizes provided (2) must be greater or equal to
  the number of dimensions in the tensor (4)

and when I try to set flow = torch.FloatTensor(5, 2*self.in_channel,self.img_rows,self.img_cols)

I get motion_dataloader.py", line 55, in stackopf
flow[:,2*(j),:,:] = H

  RuntimeError: expand(torch.FloatTensor{[5, 1, 224, 224]}, size=[5,
  224, 224]): the number of sizes provided (3) must be greater or equal
  to the number of dimensions in the tensor (4)

when I multiply the train batchsize by 5 that is returned, I also get the same error.


title: How can I build a good approximation of an unknown distribution when only having samples from it in order to draw from it in torch?

body:
Say I just have random samples from the Distribution and no other data - e.g. a list of numbers - [1,15,30,4,etc.]. What's the best way to estimate the distribution to draw more samples from it in pytorch?

I am currently assuming that all samples come from a Normal distribution and just using the mean and std of the samples to build it and draw from it. The function, however, can be of any distribution.

samples = torch.Tensor([1,2,3,4,3,2,2,1])
Normal(samples.mean(), samples.std()).sample()


title: Colab pro timeouts

body:
I'm implementing a paper in colab rn with 160000 iterations where each iteration takes about 2-3 seconds which means with normal colab i should be finished in about 5-6 days, but i get timed out most of the time at around 5 hours, probably cause I'm using alot of ressources, then I have to wait 12 hours to train again.
My question is, If I'd get colab pro, will i get full 24 hours most of the time or will I get timed out much earlier and is the timeout duration the same as in colab free?


title: PyTorch: NVIDIA NGC image or Docker Hub image?

body:
What are the differences between the official PyTorch image on Docker Hub and the PyTorch image on NVIDIA NGC?
The NGC page is more documented than the Docker Hub page, which has no description. But the NGC image is also heavier by a few gigabytes, and it seems to require a CUDA 10.2-compatible driver.
Is there any advantage in using the NVIDIA NGC image instead of the one from Docker Hub?


title: convert nn.Softmax to torch.tensor

body:
I have a softmax function at the end of my Neural Network.
I want to have the probabilities as torch.tensor.For it I am using  torch.tensor(nn.softmax(x)) and getting error RuntimeError: Could not infer dtype of Softmax.
May I please know what I am doing wrong here or is there any other way to do it.


title: How to combine train and test set for IMDB Dataset in Torchtext / Pytorch

body:
I want to use the examples in the test set of the IMDB Sentiment Analysis Dataset for training, as I have built my own benchmark with which I will compare the performance of various Models (my Matura Thesis)
So after trying, I got the appending working and also managed ot split it, so that I have a validation set as well. The code is the following:
 from torchtext import datasets

    train_data, test_data = datasets.IMDB.splits(TEXT, LABEL)

    import random
    train_data, valid_data = train_data.split(random_state = random.seed(SEED))
    from torch.utils.data import ConcatDataset
    data_list = list()
    data_list.append(train_data)
    data_list.append(test_data)
    train_data = ConcatDataset(data_list)
    print(f'Number of validation examples: {len(valid_data)}')
    print(f'Number of training examples: {len(train_data)}')

And I get the following split (which is my goal):
Number of validation examples: 7500
Number of training examples: 42500

Now when I want to built the vocab with the following code, I get this error:
MAX_VOCAB_SIZE = 25_000
LABEL.build_vocab(train_data)

TEXT.build_vocab(train_data, max_size = MAX_VOCAB_SIZE)
~\.conda\envs\matura-ml\lib\collections\__init__.py in update(*args, **kwds)
    654             else:
--> 655                 _count_elements(self, iterable)
    656         if kwds:

TypeError: 'Example' object is not iterable

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
 in
      1 MAX_VOCAB_SIZE = 25_000
      2
----> 3 TEXT.build_vocab(train_data, max_size = MAX_VOCAB_SIZE)
      4 LABEL.build_vocab(train_data)

~\.conda\envs\matura-ml\lib\site-packages\torchtext\data\field.py in build_vocab(self, *args, **kwargs)
    299                     counter.update(x)
    300                 except TypeError:
--> 301                     counter.update(chain.from_iterable(x))
    302         specials = list(OrderedDict.fromkeys(
    303             tok for tok in [self.unk_token, self.pad_token, self.init_token,

TypeError: 'Example' object is not iterable

How can I combine the train and test split in the correct way?