title: Skorch training object from scratch

body:
I'm trying to use skorch class to execut GridSearch on a classifier.
I tried running with the vanilla NeuralNetClassifier object, but I haven't found a way to pass the Adam optimizer only the trainable weights (I'm using pre-trained embeddings and I would like to keep them frozen). It's doable if a module is initialized, and then pass those weights with the optimizer__params option, but module needs an uninitialized model. Is there a way around this?

net = NeuralNetClassifier(module=RNN, module__vocab_size=vocab_size, module__hidden_size=hidden_size,
                          module__embedding_dim=embedding_dim, module__pad_id=pad_id,
                          module__dataset=ClaimsDataset, lr=lr, criterion=nn.CrossEntropyLoss,
                          optimizer=torch.optim.Adam, optimizer__weight_decay=35e-3, device='cuda',
                          max_epochs=nb_epochs, warm_start=True)

The code above works. However, with the batch_size set at 64, I've got to run the model for the specified number of epochs on every batch! Which is not the behavior I'm seeking. I'd be grateful if someone could suggest a nicer way to do this.

My other issue is with subclassing skorch.NeuralNet. I run into a similar issue: figuring out a way to pass only the trainable weights to Adam optimizer. The code below is what I've got so far.

class Train(skorch.NeuralNet):
def __init__(self, module, lr, norm, *args, **kwargs):
    self.module = module
    self.lr = lr
    self.norm = norm
    self.params = [p for p in self.module.parameters(self) if p.requires_grad]
    super(Train, self).__init__(*args, **kwargs)

def initialize_optimizer(self):
    self.optimizer = torch.optim.Adam(params=self.params, lr=self.lr, weight_decay=35e-3, amsgrad=True)

def train_step(self, Xi, yi, **fit_params):
    self.module.train()

    self.optimizer.zero_grad()
    yi = variable(yi)

    output = self.module(Xi)

    loss = self.criterion(output, yi)
    loss.backward()

    nn.utils.clip_grad_norm_(self.params, max_norm=self.norm)
    self.optimizer.step()

def score(self, y_t, y_p):
    return accuracy_score(y_t, y_p)

Initializing the class gives the error:

Traceback (most recent call last):
File "/snap/pycharm-community/74/helpers/pydev/pydevd.py", line 1664, in <module>
    main()
File "/snap/pycharm-community/74/helpers/pydev/pydevd.py", line 1658, in main
globals = debugger.run(setup['file'], None, None, is_module)
File "/snap/pycharm-community/74/helpers/pydev/pydevd.py", line 1068, in run
pydev_imports.execfile(file, globals, locals)  # execute the script
File "/snap/pycharm-community/74/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "/home/l/Documents/Bsrc/cv.py", line 115, in <module>
    main()
File "/home/l/B/src/cv.py", line 86, in main
    trainer = Train(module=RNN, criterion=nn.CrossEntropyLoss, lr=lr, norm=max_norm)
File "/home/l/B/src/cv.py", line 22, in __init__
   self.params = [p for p in self.module.parameters(self) if p.requires_grad]
File "/home/l/B/src/cv.py", line 22, in <listcomp>
   self.params = [p for p in self.module.parameters(self) if p.requires_grad]
File "/home/l/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 739, in parameters
    for name, param in self.named_parameters():
AttributeError: 'Train' object has no attribute 'named_parameters'


title: what is the right usage of _extra_files arg in torch.jit.save

body:
one option I tried is pickling vocab and saving with extrafiles arg

import torch
import pickle

class Vocab(object):
    pass

vocab = Vocab()
pickle.dump(open('path/to/vocab.pkl','w'))

m = torch.jit.ScriptModule()

## I am not sure about the usage of this arg, the docs didn't help me
extra_files = torch._C.ExtraFilesMap()
extra_files['vocab.pkl'] = 'path/to/vocab.pkl'
# I also tried  pickle.dumps(vocab), and directly vocab

torch.jit.save(m, 'scriptmodule.pt', _extra_files=extra_files)

## Load with extra files.
files = {'vocab.pkl': ''}
torch.jit.load('scriptmodule.pt', _extra_files = files)

this gives

TypeError: import_ir_module(): incompatible function arguments. The following argument types are supported:
    1. (arg0: Callable[[List[str]], torch._C.ScriptModule], arg1: str, arg2: object, arg3: torch._C.ExtraFilesMap) -> None

other option is obviously to load the pickle separately, but I was looking for single file option.

it would be nice if one could just add vocab to to the torchscript ... it would also be nice to know if there is some reason for not doing this that I am obviously not aware of.


title: How to retrieve topk values across multiple rows along with their respective indices in PyTorch?

body:
I have a tensor like this:

tensor([[[ 7.3478, -1.8058, -2.6140,  ..., -0.2719, -0.3171, -0.4737]],

    [[ 7.3606, -1.8269, -1.9825,  ..., -0.8680,  0.4894,  0.2708]]],
   grad_fn=<CatBackward>)

I want to get the topk values across both the rows. Currently what I am able to do is the following:

ipdb>  stacked.topk(2)
torch.return_types.topk(
values=tensor([[[14.3902, 14.3039]],

        [[14.8927, 12.1973]]], grad_fn=<TopkBackward>),
indices=tensor([[[60, 12]],

        [[12, 23]]]))

From the output, you can see that the top 2 values were retrieved from both rows. I want to get an output as follows:

14.8927 that maps to index 12
14.3902 that maps to index 60

Note that if the top 2 values were in the first row, it will only return the values from there and completely ignore the second row and vice versa.

Need help in this regard.

A very hacky way of doing what I am trying to say would be the following but it is very hacky and shown for a BEAM_WIDTH of 2:

BEAM_WIDTH = 2
top_k = stacked.data.topk(BEAM_WIDTH, dim=2)

v1, i1 = top_k[0][0][0], top_k[1][0][0]
v2, i2 = top_k[0][1][0], top_k[1][1][0]

i = j = 0
final = []
for _ in range(BEAM_WIDTH):
    if v1[i] >= v2[j]:
        final.append((v1[i], i1[i]))
        i += 1
    else:
        final.append((v2[j], i2[j]))
        j += 1


title: "MisconfigurationError: No TPU devices were found" even when TPU is connected in PyTorch Lightning

body:
Have been frustrated over the past few hours over a problem, though It's likely its a problem I started myself hah.
I'm trying to connect to the TPU in Colab. I'm pretty sure I've gotten all the import stuff down. My code is here. I'm not completely set on everything, so the entire document isn't functional, but you should be able to see my attempts at TPU connection.
I'm running Pytorch in version 1.5.0 and torchvision in 0.6.0 because I was finding I couldn't install XLA with anything later than 1.5.0. I'm running XLA in version 20200325.
This is the image that seems so confusing: It states that we have a connection with xla: 1 yet when trying to flag it in the trainer I get an error saying no TPUs can be found.

If anyone could help me, that would be amazing.
Thanks,
A


title: In pytorch loss is the value nn.MSELoss(yhat, y), you can then call loss.data[0], what does that do?

body:
I am following a tutorial from GitHub and I am confused on this piece of code. I am trying to find documentation for it but I couldn't find any. I tried to trace my way back to nn.MSELoss as well. What does indexing into loss.data[0] do? What about the other indexes?

criterion = nn.MSELoss()

loss = criterion(pred, y)

print('loss: ', loss.data[0])


title: My pytorch model concatenation is throwing an error

body:
I'm trying to concatenate two models, consisting of images (resnet) and numerical categorical data. I have categorical and numerical columns to bring in. I have pulled them from my dataframe, and turned them into tensors. I’ve then pulled in the resnet-50 model with a modification (to add an output of 2 instead of 1000) at the end for my images.

I defined a model for the categorical and numerical images and tried to concatenate with resnet. I’m getting the following error:

NotImplementedError Traceback (most recent call last)
in
20
21
—> 22 y_pred = combined_model(image, categorical_data, numerical_data)
23 single_loss = loss_function(y_pred, label)
24 aggregated_losses.append(single_loss)

C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py in call(self, *input, **kwargs)
539 result = self._slow_forward(*input, **kwargs)
540 else:
–> 541 result = self.forward(*input, **kwargs)
542 for hook in self._forward_hooks.values():
543 hook_result = hook(self, input, result)

C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py in forward(self, *input)
95 registered hooks while the latter silently ignores them.
96 “”"
—> 97 raise NotImplementedError
98
99 def register_buffer(self, name, tensor):

NotImplementedError:

Here is how I defined my model, the loss and the training loop:

class Image_Embedd(nn.Module):
    def __init__(self, model, embedding_size, num_numerical_cols, output_size, layers, p = 0.4):
    '''
    Args
    ---------------------------
    embedding_size: Contains the embedding size for the categorical columns
    num_numerical_cols: Stores the total number of numerical columns
    output_size: The size of the output layer or the number of possible outputs.
    layers: List which contains number of neurons for all the layers.
    p: Dropout with the default value of 0.5

    '''
        super().__init__()

        self.model = model
        #list of ModuleList objects for all categorical columns
        self.all_embeddings = nn.ModuleList([nn.Embedding(ni, nf) for ni, nf in embedding_size])

        #drop out value for all layers
        self.embedding_dropout = nn.Dropout(p)

        #list of 1 dimension batch normalization objects for all numerical columns
        self.batch_norm_num = nn.BatchNorm1d(num_numerical_cols)

        #the number of categorical and numerical columns are added together and stored in input_size
        all_layers = []
        num_categorical_cols = sum((nf for ni, nf in embedding_size))
        input_size = num_categorical_cols + num_numerical_cols

         #loop iterates to add corresonding layers to all_layers list above
       for i in layers:
            all_layers.append(nn.Linear(input_size, i))
            all_layers.append(nn.ReLU(inplace=True))
            all_layers.append(nn.BatchNorm1d(i))
            all_layers.append(nn.Dropout(p))
            input_size = i

        #append output layer to list of layers
        all_layers.append(nn.Linear(layers[-1], output_size))

        #pass all layers to the sequential class
        self.layers = nn.Sequential(*all_layers)

        #define the forward method

    def forward(self, x_categorical, x_numerical):
        #this starts the embedding of categorical columns
        embeddings = []
        for i,e in enumerate(self.all_embeddings):
            embeddings.append(e(x_categorical[:,i]))
        x = torch.cat(embeddings, 1)
        x = self.embedding_dropout(x)

        #normalizing numerical columns
        x_numerical = self.batch_norm_num(x_numerical)

        #concatenating numerical and categorical columns
        x = torch.cat([x, x_numerical], 1)
        x = self.layers(x)

        x2 = model(x2)
        x_final = torch.concat(x, x2)
        x_final = F.softmax(x_final, dim = 1)
        return x

Instantiate Model

combined_model = Image_Embedd(model = CNNmodel, embedding_size=categorical_embedding_sizes
,num_numerical_cols=numerical_data.shape[1], output_size = 2, layers = [256,128,64,32,2]
, p = 0.4)

Loss, optimizer

torch.manual_seed(101) criterion = nn.CrossEntropyLoss().cuda()
optimizer = torch.optim.Adam(combined_model.parameters(), lr=0.001)
exp_lr_scheduler = lr_scheduler.StepLR(optimizer, step_size=7,
gamma=0.1) combined_model = combined_model.cuda()

Training Loop

epochs = 1
aggregated_losses = []

max_trn_batch = 25

for i in range(epochs):
for b, (image, label, policy) in enumerate(train_loader):
image = image.cuda()
label = label.cuda()
categorical_data = categorical_data.cuda()
numerical_data = numerical_data.cuda()
#print(image, label, categorical_data, numerical_data)

    #count batches
    b += 1

    #throttle teh batches
    if b == max_trn_batch:
        break

    y_pred = combined_model(image, categorical_data, numerical_data)
    single_loss = loss_function(y_pred, label)
    aggregated_losses.append(single_loss)

    # statistics
    running_loss += single_loss.item() * image.size(0)
    running_corrects += torch.sum(y_pred == label.data)

    print(f'train-epoch: {i}, train-batch: {b}')

    optimizer.zero_grad()
    single_loss.backward()
    optimizer.step()

I'm not sure where in the process I'm messing this up. I want the categorical embeddings and numerical data to help inform the image classification.


title: Cuda Runtime error out of memory - Pytorch using 8GB

body:
I'm trying to train a model with my GTX 1080Ti that has 11GB memory but it seems that Pytorch reserved about 8GB memory to itself any way I can reduce that number? (Super new to this)

 RuntimeError: CUDA out of memory. Tried to allocate 2.53 GiB (GPU 0; 10.91 GiB total capacity; 4.15 GiB already allocated; 1.73 GiB free; 8.19 GiB reserved in total by PyTorch) (malloc at /pytorch/c10/cuda/CUDACachingAllocator.cpp:289)

frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x46 (0x7f54646f7536 in /home/yasin/.local/lib/python3.6/site-packages/torch/lib/libc10.so)
    frame #1: <unknown function> + 0x1cf1e (0x7f5464940f1e in /home/yasin/.local/lib/python3.6/site-packages/torch/lib/libc10_cuda.so)
    frame #2: <unknown function> + 0x1df9e (0x7f5464941f9e in /home/yasin/.local/lib/python3.6/site-packages/torch/lib/libc10_cuda.so)
    frame #3: at::native::empty_cuda(c10::ArrayRef<long>, c10::TensorOptions const&, c10::optional<c10::MemoryFormat>) + 0x135 (0x7f54005f7535 in /home/yasin/.local/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so)
    frame #4: <unknown function> + 0xf7a66b (0x7f53febef66b in /home/yasin/.local/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so)
    frame #5: <unknown function> + 0xfc3f57 (0x7f53fec38f57 in /home/yasin/.local/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so)
    frame #6: <unknown function> + 0x1075389 (0x7f543b173389 in /home/yasin/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
    frame #7: <unknown function> + 0x10756c7 (0x7f543b1736c7 in /home/yasin/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
    frame #8: <unknown function> + 0xf1557e (0x7f53feb8a57e in /home/yasin/.local/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so)
    frame #9: <unknown function> + 0xf18043 (0x7f53feb8d043 in /home/yasin/.local/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so)
    frame #10: at::native::cudnn_convolution_backward_input(c10::ArrayRef<long>, at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<long>, long, bool, bool) + 0xb2 (0x7f53feb8d882 in /home/yasin/.local/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so)
    frame #11: <unknown function> + 0xf7f3a0 (0x7f53febf43a0 in /home/yasin/.local/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so)
    frame #12: <unknown function> + 0xfc3c38 (0x7f53fec38c38 in /home/yasin/.local/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so)
    frame #13: at::native::cudnn_convolution_backward(at::Tensor const&, at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<long>, long, bool, bool, std::array<bool, 2ul>) + 0x4fa (0x7f53feb8ef1a in /home/yasin/.local/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so)
    frame #14: <unknown function> + 0xf7f6cb (0x7f53febf46cb in /home/yasin/.local/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so)
    frame #15: <unknown function> + 0xfc3c94 (0x7f53fec38c94 in /home/yasin/.local/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so)
    frame #16: <unknown function> + 0x2c809b6 (0x7f543cd7e9b6 in /home/yasin/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
    frame #17: <unknown function> + 0x2cd0444 (0x7f543cdce444 in /home/yasin/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
    frame #18: torch::autograd::generated::CudnnConvolutionBackward::apply(std::vector<at::Tensor, std::allocator<at::Tensor> >&&) + 0x378 (0x7f543c996918 in /home/yasin/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
    frame #19: <unknown function> + 0x2d89c05 (0x7f543ce87c05 in /home/yasin/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
    frame #20: torch::autograd::Engine::evaluate_function(std::shared_ptr<torch::autograd::GraphTask>&, torch::autograd::Node*, torch::autograd::InputBuffer&) + 0x16f3 (0x7f543ce84f03 in /home/yasin/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
    frame #21: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&, bool) + 0x3d2 (0x7f543ce85ce2 in /home/yasin/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
    frame #22: torch::autograd::Engine::thread_init(int) + 0x39 (0x7f543ce7e359 in /home/yasin/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
    frame #23: torch::autograd::python::PythonEngine::thread_init(int) + 0x38 (0x7f545f97b4d8 in /home/yasin/.local/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
    frame #24: <unknown function> + 0xbd6df (0x7f54675c86df in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
    frame #25: <unknown function> + 0x76db (0x7f546c2ff6db in /lib/x86_64-linux-gnu/libpthread.so.0)
    frame #26: clone + 0x3f (0x7f546c63888f in /lib/x86_64-linux-gnu/libc.so.6)

Super Resolution with CNNs and GANs

train_cnn.py


title: What could cause a VAE(Variational AutoEncoder) to output random noise even after training?

body:
I have trained a VAE on CIFAR10 data-set. However, when I try to generate images from the VAE all I get is a bunch of gray noise back. The implementation of this VAE follows the implementation from the book Generative Deep Learning, but instead of TensorFlow the code uses PyTorch.
The notebook containing the training as well as the generation can be found here, while the actual implementation of the VAE can be found here.
I have tried:

Disabling dropouts.
Increasing the dimension of the latent space.

None of the methods show any improvement at all.
I have verified that:

The input size matches the output size
Back-propagation runs successfully as the loss decreases during training.


title: AssertionError: No inf checks were recorded for this optimizer in Pytorch's AutomaticMixedPrecision

body:
I'm using AutomaticMixedPrecision feature of PyTorch to train a network with smaller footprint and precision.
At a certain point some embeddings from the network have NaNs in their tensors, so I'd like to replace those with 0s in order to perform online hard negative samples mining.
However, after replacing the NaNs in the tensor like this:
tensor[torch.isnan(tensor)] = 0

I get the following error while doing the next scaler ste (scaler.step(optimizer):
    assert len(optimizer_state["found_inf_per_device"]) > 0, "No inf checks were recorded for this optimizer."
AssertionError: No inf checks were recorded for this optimizer.

What's the correct way to zero out NaNs while getting rid of this error?


title: Cannot fix ImportError: cannot import name 'ToTensorV2' from 'albumentations.pytorch' on Colab

body:
I tried all three solutions from this question to solve the problem, but still receive an error:
ImportError: cannot import name 'ToTensorV2' from 'albumentations.pytorch' (/usr/local/lib/python3.7/dist-packages/albumentations/pytorch/__init__.py)

My current code:
!pip install -U albumentations
import albumentations
from albumentations.pytorch import ToTensorV2


title: Pytorch visdom: can`t start visdom

body:
When I run python -m visdom.server, then it shows:

for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution
    logging.error('Error {} while downloading {}'.format(exc.code, key))
AttributeError: 'URLError' object has no attribute 'code'

What does it mean? How can I deal with it?


title: Why feature extraction of text don't return all possible feature names?

body:
Here is the snippet of code from the book
Natural Language Processing with PyTorch:

import numpy as np
from sklearn.feature_extraction.text import CountVectorizer
import seaborn as sns

corpus = ['Time flies flies like an arrow.', 'Fruit flies like a banana.']
one_hot_vectorizer = CountVectorizer()
vocab = one_hot_vectorizer.get_feature_names()

The value of vocab :

vocab = ['an', 'arrow', 'banana', 'flies', 'fruit', 'like', 'time']

Why is not there an 'a' among the extracted feature names? If it is excluded as too common word automatically, why "an" is not excluded for the same reasons? How to make .get_feature_names() filter other words as well?


title: Multiple outputs in Pytorch, Keras style

body:
How could you implement these 2 Keras models (inspired by the Datacamp course 'Advanced Deep Learning with Keras in Python') in Pytorch:

Classification with 1 input, 2 outputs:

from keras.layers import Input, Concatenate, Dense
from keras.models import Model

input_tensor = Input(shape=(1,))
output_tensor = Dense(2)(input_tensor)

model = Model(input_tensor, output_tensor)
model.compile(optimizer='adam', loss='categorical_crossentropy')
X = ... # e.g. a pandas series
y = ... # e.g. a pandas df with 2 columns
model.fit(X, y, epochs=100)

A model with classification and regression:

from keras.layers import Input, Dense
from keras.models import Model

input_tensor = Input(shape=(1,))
output_tensor_reg = Dense(1)(input_tensor)
output_tensor_class = Dense(1, activation='sigmoid')(output_tensor_reg)

model.compile(loss=['mean_absolute_error','binary_crossentropy']
X = ...
y_reg = ...
y_class = ...
model.fit(X, [y_reg, y_class], epochs=100)


title: How do I effectively parallelize AlphaZero on the GPU?

body:
I'm implementing a version of AlphaZero (AlphaGo's most recent incarnation) to be applied to some other domain.

The crux of the algorithm is a Monte Carlo Tree Search of the state space (CPU) interleaved with 'intuition' (probabilities) from a neural network in eval mode (GPU). The MCTS result is then used to train the neural network.

I already parallelized the CPU execution by launching multiple processes which each build up their own tree. This is effective and has now lead to a GPU bottleneck! (nvidia-smi showing the GPU at 100% all the time)

I have devised 2 strategies to parallelize GPU evaluations, however both of them have problems.

Each process evaluates the network only on batches from its own tree. In my initial naive implementation, this meant a batch size of 1. However, by refactoring some code and adding a 'virtual loss' to discourage (but not completely block) the same node from being picked twice we can get larger batches of size 1-4. The problem here is that we cannot allow large delays until we evaluate the batch or accuracy suffers, so a small batch size is key here.
Send the batches to a central "neural network worker" thread which combines and evaluates them. This could be done in a large batch of 32 or more, so the GPU could be used very efficiently. The problem here is that the tree workers send CUDA tensors 'round-trip' which is not supported by PyTorch. It is supported if I clone them first, but all that constant copying makes this approach slower than the first one.

I was thinking maybe a clever batching scheme that I'm not seeing could make the first approach work. Using multiple GPUs could speed up the first approach too, but the kind of parallelism I want is not natively supported by PyTorch. Maybe keeping all tensors in the NN worker and only sending ids around could improve the second approach, however the difficulty here is how to synchronize effectively to get a large batch without making the CPU threads wait too long.

I found next to no information on how AlphaZero or AlphaGo Zero were parallelized in their respective papers. I was able to find limited information online however which lead me to improve the first approach.

I would be grateful for any advice on this, particularly if there's some point or approach I missed.


title: Constant loss during LSTM training - PyTorch

body:
I'm trying to implement an LSTM network for predicting the next word in a sentence. This is my first time building a neural network and I'm confused by all the information I found on the Internet.

I'm trying to use the following architecture:

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

class WordLSTM(nn.Module):
    def __init__(self, vocabulary_size, embedding_dim, hidden_dim):
        super().__init__()

        # Word embeddings
        self.encoder = nn.Embedding(vocabulary_size, embedding_dim)

        # LSTM input dim is embedding_dim, output dim is hidden_dim
        self.lstm = nn.LSTM(embedding_dim, hidden_dim)

        # Linear layer to map hidden states to vocabulary space
        self.decoder = nn.Linear(hidden_dim, vocabulary_size)

    def forward(self, sentence):
        encoded = self.encoder(sentence)
        output, _ = self.lstm(
            encoded.view(len(sentence), 1, -1))
        decoded = self.decoder(output)
        word_scores = F.softmax(decoded, dim=1)

        return word_scores[-1].view(1, -1)

I've created a dictionary with all the sentences from my dataset, and each word is encoded with their respective indices from the dictionary. They're followed by an encoded next word (target vector). Here's a bunch of training examples that I'm trying to use:

[tensor([39]), tensor([13698])],
[tensor([   39, 13698]), tensor([11907])],
[tensor([   39, 13698, 11907]), tensor([70])]

I'm passing one sentence at a time during training, so my batch size is always 1.

NUM_EPOCHS = 100
LEARNING_RATE = 0.0005

rnn = WordLSTM(vocab_size, 64, 32)

optimizer = optim.SGD(rnn.parameters(), lr=LEARNING_RATE)

for epoch in range(NUM_EPOCHS):
    training_example = generate_random_training_example(training_ds)
    optimizer.zero_grad()

    for sentence, next_word in training_example:
        output = rnn(sentence)
        loss = F.cross_entropy(output, next_word)
        loss.backward()

    optimizer.step()

    print(f"Epoch: {epoch}/{NUM_EPOCHS} Loss: {loss:.4f}")

However, when I start the training, the loss does not change with time:

Epoch: 0/100 Loss: 10.3929
Epoch: 1/100 Loss: 10.3929
Epoch: 2/100 Loss: 10.3929
Epoch: 3/100 Loss: 10.3929
Epoch: 4/100 Loss: 10.3929
Epoch: 5/100 Loss: 10.3929
Epoch: 6/100 Loss: 10.3929

I've tried placing optimizer.zero_grad() and optimizer.step() in a different places already, but didn't help either.

What could be the problem in this case? Am I calculating the loss in the wrong way, or do I pass the tensors in the wrong format?


title: Pytorch - Caught StopIteration in replica 1 on device 1 error while Training on GPU

body:
I am trying to train a BertPunc model on the train2012 data used in the git link: https://github.com/nkrnrnk/BertPunc.
While running on the server, with 4 GPUs enabled, below is the error I get:
StopIteration: Caught StopIteration in replica 1 on device 1.
Original Traceback (most recent call last):
  File "/home/stenoaimladmin/.local/lib/python3.8/site-packages/torch/nn/parallel/parallel_apply.py", line 61, in _worker
    output = module(*input, **kwargs)
  File "/home/stenoaimladmin/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/stenoaimladmin/notebooks/model_BertPunc.py", line 16, in forward
    x = self.bert(x)
  File "/home/stenoaimladmin/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/stenoaimladmin/anaconda3/lib/python3.8/site-packages/pytorch_pretrained_bert/modeling.py", line 861, in forward
    sequence_output, _ = self.bert(input_ids, token_type_ids, attention_mask,
  File "/home/stenoaimladmin/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/stenoaimladmin/anaconda3/lib/python3.8/site-packages/pytorch_pretrained_bert/modeling.py", line 727, in forward
    extended_attention_mask = extended_attention_mask.to(dtype=next(self.parameters()).dtype) # fp16 compatibility
StopIteration

From the link: https://github.com/huggingface/transformers/issues/8145, this appears to be happening when the data gets moved back and forth between multiple GPUs.
As per the git link: https://github.com/interpretml/interpret-text/issues/117, we need to downgrade PyTorch version to 1.4 from 1.7 which I use currently. For me downgrading the version isnt an option as I have other scripts that use Torch 1.7 version. What should I do to overcome this error?
I cant put the whole code here as there are too many lines, but here is the snippet that gives me the error:
bert_punc, optimizer, best_val_loss = train(bert_punc, optimizer, criterion, epochs_top,
        data_loader_train, data_loader_valid, save_path, punctuation_enc, iterations_top, best_val_loss=1e9)

Here is my DataParallel code:
   bert_punc = nn.DataParallel(BertPunc(segment_size, output_size, dropout)).cuda()

I tried changing the Dataparallel line to divert the training to only 1 GPU , out of 4 present. But that gave me a space issue, and hence had to revert the code back to default.
Here is the link to all scripts that I am using: https://github.com/nkrnrnk/BertPunc
Please advice.


title: Torch JIT Trace = TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect

body:
I am following this tutorial: https://huggingface.co/transformers/torchscript.html
to create a trace of my custom BERT model, however when running the exact same dummy_input I receive an error:
TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect.
We cant record the data flow of Python values, so this value will be treated as a constant in the future.

Having loaded in my model and tokenizer, the code to create the trace is the following:
text = "[CLS] Who was Jim Henson ? [SEP] Jim Henson was a puppeteer [SEP]"
tokenized_text = tokenizer.tokenize(text)

# Masking one of the input tokens
masked_index = 8
tokenized_text[masked_index] = '[MASK]'
indexed_tokens = tokenizer.convert_tokens_to_ids(tokenized_text)
segments_ids = [0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1]

tokens_tensor = torch.tensor([indexed_tokens])
segments_tensors = torch.tensor([segments_ids])
dummy_input = [tokens_tensor, segments_tensors]

traced_model = torch.jit.trace(model, dummy_input)

The dummy_input is a list of tensors so I'm not sure where the Boolean type is coming into play here. Does anyone understand why this error is occurring and whether the Boolean conversion is happening?
Many Thanks


title: Is there a nice way to to check if numpy array and torch tensor point to same underlying data?

body:
I want to check if numpy array and torch tensor point to same underlying memory.
So far I've came up with a simple check but it doesn't look super elegant.

import numpy as np
import torch

# example
a = np.random.randn(3,3)
b = torch.from_numpy(a)

assert a.__array_interface__['data'][0] == b.data_ptr()

Is there a nicer way to do it? Also, could some potential undefined/incorrect behaviour occur if using this assertion?

Thanks in advance for the answers :)


title: How to use torch.nn.CrossEntropyLoss as autoencoder's reconstruction loss?

body:
I want to compute the reconstruction accuracy of my autoencoder using CrossEntropyLoss:

ae_criterion = nn.CrossEntropyLoss()
ae_loss = ae_criterion(X, Y)

where X is the autoencoder's reconstruction and Y is the target (since it is an autoencoder, Y is the same as the original input X).
Both X and Y have shape [42, 32, 130] = [batch_size, timesteps, number_of_classes]. When I run the code above I get the following error:

  ValueError: Expected target size (42, 130), got torch.Size([42, 32,
  130])

After looking the docs, I'm still unsure on how should I call nn.CrossEntropyLoss() in the appropriate way. It seems that I should change Y to be of shape [42, 32, 1], with each element being a scalar in the interval [0, 129] (or [1, 130]), am I right?

Is there a way to avoid this? Since X and Y are between 0 and 1, could I just use binary cross-entropy loss element-wise in an equivalent way?


title: Where is the PatchGAN implementation in the official PyTorch CycleGAN repo?

body:
There's mention in the paper and in the code comments of a 70x70 PatchGAN, although I can't find anywhere in the code this is explicitly implemented.

Does anyone know how it is implemented?


I have an input layer of shape (16,12,18,768)  ie (batch_size, channel_out, sentence_lenght, embedding_dim) but when I pass it through a conv layer with input channel = 12 , output channel = 16 and kernel size = 3 x 768, i get an output layer of shape
(16,192,18,1).

Then i get the error RuntimeError: Calculated padded input size per channel: (18 x 1). Kernel size: (3 x 768). Kernel size can't be greater than actual input size. why does the input size per channel become (18 x 1)

Why and how does this happen? Can some explain?


title: Projector tab is blank in PyTorch Tensorboard

body:
I'm trying to visualize embeddings per epoch. But the Projector page shows this -

Here's the text from the page -
No checkpoint was found.

Probable causes:

    No checkpoint has been saved yet. Please refresh the page periodically.
    You are not saving any checkpoint. To save your model, create a tf.train.Saver and save your model periodically by calling saver.save(session, LOG_DIR/model.ckpt, step).

If you’re new to using TensorBoard, and want to find out how to add data and set up your event files, check out the README and perhaps the TensorBoard tutorial.

If you think TensorBoard is configured properly, please see the section of the README devoted to missing data problems and consider filing an issue on GitHub.

To simplify things, I use the following to generate an embedding inside my training loop; still did not work.
for e in epochs:
    .
    .
    writer.add_embedding(torch.randn(100, 5), global_step=e)

What am I missing here?


title: How to save a Pytorch Model directly in s3 Bucket?

body:
The title says it all - I want to save a pytorch model in an s3 bucket. What I tried was the following:
import boto3

s3 = boto3.client('s3')
saved_model = model.to_json()
output_model_file = output_folder + "pytorch_model.json"
s3.put_object(Bucket="power-plant-embeddings", Key=output_model_file, Body=saved_model)

Unfortunately this doesn't work, as .to_json() only works for tensorflow models. Does anyone know how to do it in pytorch?


title: ValueError: Unknown CUDA arch (8.6) or GPU not supported

body:
when I build DCNv2 in my conda environment,I got this message.
I have checked cuda by nvidia-smi:
Tue Apr  6 20:03:13 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.67       Driver Version: 460.67       CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce RTX 3070    Off  | 00000000:01:00.0  On |                  N/A |
|  0%   45C    P8    17W / 220W |    448MiB /  7979MiB |     23%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A       955      G   /usr/lib/xorg/Xorg                 53MiB |
|    0   N/A  N/A      1555      G   /usr/lib/xorg/Xorg                210MiB |
|    0   N/A  N/A      1690      G   /usr/bin/gnome-shell               61MiB |
|    0   N/A  N/A      3564      G   ...AAAAAAAAA= --shared-files      108MiB |

and nvcc -V:
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243

Any help?


title: PyTorch equivalent of `numpy.unpackbits`?

body:
I am training a neural net on GPU. It uses a lot of binary input features.

Since moving data to/from GPU is expensive, I am looking for ways to make the initial representation more compact. Now, I encode my features as int8, move them over to GPU and then expand as float32:

# create int8
features = torch.zeros(*dims, dtype=torch.int8)

# fill in some data (set some features to 1.)
…

# move int8 to GPU
features = features.to(device=cuda, non_blocking=True)

# expand int8 as float32
features = features.to(dtype=float32)

Now, I am looking for ways to compress those binary features to bits instead of bytes.

NumPy has functions packbits and unpackbits

>>> a = np.array([[2], [7], [23]], dtype=np.uint8)
>>> b = np.unpackbits(a, axis=1)
>>> b
array([[0, 0, 0, 0, 0, 0, 1, 0],
       [0, 0, 0, 0, 0, 1, 1, 1],
       [0, 0, 0, 1, 0, 1, 1, 1]], dtype=uint8)

Is there any way to unpack bits in PyTorch on GPU?


title: What can be the cause of the validation loss increasing and the accuracy remaining constant to zero while the train loss decreases?

body:
I am trying to solve a multiclass text classification problem. Due to specific requirements from my project I am trying to use skorch (https://skorch.readthedocs.io/en/stable/index.html) to wrap pytorch for the sklearn pipeline. What I am trying to do is fine-tune a pretrained version of BERT from Huggingface (https://huggingface.co) with my dataset. I have tried, in the best of my knowledge, to follow the instructions from skorch on how I should input my data, structure the model etc. Still during the training the train loss decreases until the 8th epoch where it starts fluctuating, all while the validation loss increases from the beginning and the validation accuracy remains constant to zero. My pipeline setup is

    from sklearn.pipeline import Pipeline

    pipeline = Pipeline(
    [
        ("tokenizer", Tokenizer()),
        ("classifier", _get_new_transformer())
    ]

in which I am using a tokenizer class to preprocess my dataset, tokenizing it for BERT and creating the attention masks. It looks like this

    import torch
    from transformers import AutoTokenizer, AutoModel
    from torch import nn
    import torch.nn.functional as F
    from sklearn.base import BaseEstimator, TransformerMixin
    from tqdm import tqdm
    import numpy as np

    class Tokenizer(BaseEstimator, TransformerMixin):
        def __init__(self):
            super(Tokenizer, self).__init__()

            self.tokenizer = AutoTokenizer.from_pretrained(/path/to/model)

        def _tokenize(self, X, y=None):
            tokenized = self.tokenizer.encode_plus(X, max_length=20, add_special_tokens=True, pad_to_max_length=True)
            tokenized_text = tokenized['input_ids']
            attention_mask = tokenized['attention_mask']
            return np.array(tokenized_text), np.array(attention_mask)

        def fit(self, X, y=None):
            return self

        def transform(self, X, y=None):
            word_tokens, attention_tokens = np.array([self._tokenize(string)[0] for string in tqdm(X)]), \
                                    np.array([self._tokenize(string)[1] for string in tqdm(X)])
            X = word_tokens, attention_tokens
            return X

        def fit_transform(self, X, y=None, **fit_params):
            self = self.fit(X, y)
            return self.transform(X, y)

then I initialize the model I want to fine-tune as

    class Transformer(nn.Module):
        def __init__(self, num_labels=213, dropout_proba=.1):
            super(Transformer, self).__init__()

            self.num_labels = num_labels
            self.model = AutoModel.from_pretrained(/path/to/model)
            self.dropout = torch.nn.Dropout(dropout_proba)
            self.classifier = torch.nn.Linear(768, num_labels)

        def forward(self, X, **kwargs):
            X_tokenized, attention_mask = torch.stack([x.unsqueeze(0) for x in X[0]]),\
                                  torch.stack([x.unsqueeze(0) for x in X[1]])
            _, X = self.model(X_tokenized.squeeze(), attention_mask.squeeze())
            X = F.relu(X)
            X = self.dropout(X)
            X = self.classifier(X)
            return X

I initialize the model and create the classifier with skorch as follows

    from skorch import NeuralNetClassifier
    from skorch.dataset import CVSplit
    from skorch.callbacks import ProgressBar
    import torch
    from transformers import AdamW

    def _get_new_transformer() -> NeuralNetClassifier:
        transformer = Transformer()
        net = NeuralNetClassifier(
            transformer,
            lr=2e-5,
            max_epochs=10,
            criterion=torch.nn.CrossEntropyLoss,
            optimizer=AdamW,
            callbacks=[ProgressBar(postfix_keys=['train_loss', 'valid_loss'])],
            train_split=CVSplit(cv=2, random_state=0)
            )
            return net

and I use fit like that

   pipeline.fit(X=dataset.training_samples, y=dataset.training_labels)

in which my training samples are lists of strings and my labels are the an array containing the indexes of each class, as pytorch requires.

This is a sample of what happens

training history

I have tried to keep train only the fully connected layer and not BERT but I have the same issue again. I also tested the train accuracy after the training process and it was only 0,16%. I would be grateful for any advice or insight on how to solve my problem! I am pretty new with skorch and not so comfortable with pytorch yet and I believe that I am missing something really simple. Thank you very much in advance!


I am in a weird scenario were I am forced to use torch 1.3.1 (due to hardware see: https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/#/). I read from the pytorch docs that it's corresponding version of torchvision is 0.4.1 (https://pypi.org/project/torchvision/):

Installation
We recommend Anaconda as Python package management system. Please refer to pytorch.org for the detail of PyTorch (torch) installation. The following is the corresponding torchvision versions and supported Python versions.
Installation
We recommend Anaconda as Python package management system. Please refer to pytorch.org for the detail of PyTorch (torch) installation. The following is the corresponding torchvision versions and supported Python versions.

torch torchvision python
master / nightly  master / nightly    >=3.6
1.5.0 0.6.0   >=3.5
1.4.0 0.5.0   ==2.7, >=3.5, <=3.8
1.3.1 0.4.2   ==2.7, >=3.5, <=3.7
1.3.0 0.4.1   ==2.7, >=3.5, <=3.7
1.2.0 0.4.0   ==2.7, >=3.5, <=3.7
1.1.0 0.3.0   ==2.7, >=3.5, <=3.7
<=1.0.1   0.2.2   ==2.7, >=3.5, <=3.7

but for some reason I have the wrong version of it:
torchvision 0.2.2 pypi_0 pypi

is there a way to install the right version of torchvision?

What I've tried:
First I tried force installing the right version with conda. Conda couldn't find the version of torchvision that I need:
$ conda install torchvision==0.4.2
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.

PackagesNotFoundError: The following packages are not available from current channels:

  - torchvision==0.4.2

Current channels:

  - https://repo.anaconda.com/pkgs/main/linux-ppc64le
  - https://repo.anaconda.com/pkgs/main/noarch
  - https://repo.anaconda.com/pkgs/r/linux-ppc64le
  - https://repo.anaconda.com/pkgs/r/noarch

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.

Then I proceeded to try to install it regardless with pip
$ pip install torchvision==0.4.2
Defaulting to user installation because normal site-packages is not writeable
ERROR: Could not find a version that satisfies the requirement torchvision==0.4.2 (from versions: 0.1.6, 0.1.7, 0.1.8, 0.1.9, 0.2.0, 0.2.1, 0.2.2, 0.2.2.post2, 0.2.2.post3)
ERROR: No matching distribution found for torchvision==0.4.2

got an error too.
Is there anything else to try?

I tried but it failed:
conda install torchvision==0.4.2 -c pytorch

Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.

PackagesNotFoundError: The following packages are not available from current channels:

  - torchvision==0.4.2

Current channels:

  - https://conda.anaconda.org/pytorch/linux-ppc64le
  - https://conda.anaconda.org/pytorch/noarch
  - https://repo.anaconda.com/pkgs/main/linux-ppc64le
  - https://repo.anaconda.com/pkgs/main/noarch
  - https://repo.anaconda.com/pkgs/r/linux-ppc64le
  - https://repo.anaconda.com/pkgs/r/noarch

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.

related:

crossposted SO: Force installing torchvision 0.4.2 when I am forced to use pytorch 1.3.1 due to hardware constraints (ppc64le IBM)
crossposted pytorch forum: https://discuss.pytorch.org/t/force-installing-torchvision/97279
crossposted reddit pytorch: https://www.reddit.com/r/pytorch/comments/iyf2qn/force_installing_torchvision/
crossposted reddit ibm: https://www.reddit.com/r/IBM/comments/iyhzex/force_installing_torchvision_042_when_i_am_forced/
real problem is installing torchmeta: https://github.com/tristandeleu/pytorch-meta/issues/95
https://www.ibm.com/mysupport/s/forumsquestion?id=0D50z00006gaxV9CAI
quora: https://www.quora.com/unanswered/How-does-one-install-specific-Python-packages-in-Conda-from-IBM-architectures
reddit ibm2: https://www.reddit.com/r/newIBM/comments/iyij10/force_installing_torchvision_042_when_i_am_forced/
gitissue for ibm: https://github.com/IBM/powerai/issues/268
gitissue in pytorch


title: How to map element in pytorch tensor to id?

body:
Given a tensor:
A = torch.tensor([2., 3., 4., 5., 6., 7.])

Then, give each element in A an id:
id = torch.arange(A.shape[0], dtype = torch.int)   # tensor([0,1,2,3,4,5])

In other words, id of 2. in A is 0 and  id of 3. in A is 1:
2. -> 0
3. -> 1
4. -> 2
5. -> 3
6. -> 4
7. -> 5

Then, I have a new tensor:
B = torch.tensor([3., 6., 6., 5., 4., 4., 4.])

In pytorch, is there any way in Pytorch to map each element in B to id?
In other words, I want to obtain tensor([1, 4, 4, 3, 2, 2, 2]), in which each element is id of the element in B.


title: Using edge features for GCN in DGL

body: I'm trying to implement a graph convolutional network (GCN) in the Deep Graph Learning (DGL) package for Python. In many papers, edges have discrete features, and each possible value is associated with a different weight matrix or set of weight matrices. An example would be here. Is anyone familiar with how to implement a model like this in DGL? The DGL team's example of GCNs for graph classification, as does another example I found online.


title: Pytorch - View/Flatten (4, 10, 3, 368, 368) to (40, 3, 368, 368)

body: How does one convert a tensor of shape  (4, 10, 3, 368, 368) to (40, 3, 368, 368). I am not able to find any documentation on this. I could hand code it but I'm sure there is any easier way to flatten the first two indexes.


title: Loading original images besides the transformed ones using ImageFolder

body:
I'm trying to train a GAN to colorize images. For that, I'm using ImageFolder of torchvision to load grayscale images but I also need the original data alongwith the transformed ones.
I want it in the fastest way as the data is large. I want to make ImageFolder load both at the same time to reduce the time complexity.
def load_data_bw(opt):
    datapath = '/content/gdrive/My Drive/faces/2003'

    dataset = torchvision.datasets.ImageFolder(datapath,
                       transform=transforms.Compose([
                                                    transforms.Grayscale(num_output_channels=3), #load images as grayscale with three channels
                                                    transforms.RandomChoice(
                                                       [transforms.Resize(opt['loadSize'], interpolation=1),
                                                        transforms.Resize(opt['loadSize'], interpolation=2),
                                                        transforms.Resize(opt['loadSize'], interpolation=3),
                                                        transforms.Resize((opt['loadSize'], opt['loadSize']),
                                                                          interpolation=1),
                                                        transforms.Resize((opt['loadSize'], opt['loadSize']),
                                                                          interpolation=2),
                                                        transforms.Resize((opt['loadSize'], opt['loadSize']),
                                                                          interpolation=3)]
                                                    ),
                                                    transforms.RandomChoice(
                                                       [transforms.RandomResizedCrop(opt['fineSize'], interpolation=1),
                                                        transforms.RandomResizedCrop(opt['fineSize'], interpolation=2),
                                                        transforms.RandomResizedCrop(opt['fineSize'], interpolation=3)]
                                                    ),
                                                    transforms.ColorJitter(brightness=0.1, contrast=0.1),
                                                    transforms.RandomHorizontalFlip(),
                                                    transforms.ToTensor()
                                               ]))
    return dataset

I'm expecting to get:
for iteration, orig_data, gray_data in enumerate(training_data_loader, 1):
    # code..


title: Whitespace tokenizer for training BERT language model from scratch with Huggingface

body:
I am trying to train a BERT language model from scratch using Huggingface API. For that I need to build a tokenizer that tokenize the text data based on white spaces only, nothing else. I understand that there are multiple tonkenizers available in Huggingface (such as BPE, WordPiece) that produce good results for language models but for my use case I want to tokenize text input based on whitespace only and generate vocabs that should not have any kind of special characters viz "##" in front of words.

For example: the input Hello, y'all! How are you? should be tonkenized as:

Hello,, y'all!, How, are, you?

I checked the documentation [1] and [2] but did not find a way to achieve this.


title: How to get Class label from Mosaic augmentation in Object Detection Dataloader?

body:
NOTE: I couldn't think any better Title name, you're most welcome
to edit or suggestion.

Update
Direct Colab Link. Just grab the given dummy data set and load it to colab.

I'm trying to train an object detection model for a multi-class problem. In my training, I am using the Mosaic augmentation, Paper, for this task.
In my training mechanism, I'm a bit stuck to properly retrieve the class labels of each category, as the augmentation mechanism randomly picks the sub-portion of a sample. However, below is a result of a mosaic augmentation that we've achieved with a relevant bounding box until now.

Data Set
I've created a dummy data set. Link here. The df.head():

It has 4 class in total and df.object.value_counts():
human    23
car      13
cat       5
dog       3

Data Loader and Mosaic Augmentation
The data loader is defined as follows. However, the mosaic augmentation should be defined inside but for now, I'll create a separate code snippet for better demonstration.

IMG_SIZE = 2000

class DatasetRetriever(Dataset):

    def __init__(self, main_df, image_ids, transforms=None, test=False):
        super().__init__()

        self.image_ids = image_ids
        self.main_df = main_df
        self.transforms = transforms
        self.size_limit = 1
        self.test = test

    def __getitem__(self, index: int):
        image_id = self.image_ids[index]
        image, boxes, labels = self.load_mosaic_image_and_boxes(index)

        # labels = torch.tensor(labels, dtype=torch.int64) # for multi-class
        labels = torch.ones((boxes.shape[0],), dtype=torch.int64) # for single-class

        target = {}
        target['boxes'] = boxes
        target['cls'] = labels
        target['image_id'] = torch.tensor([index])

        if self.transforms:
            for i in range(10):
                sample = self.transforms(**{
                    'image' : image,
                    'bboxes': target['boxes'],
                    'labels': target['cls']
                })

                assert len(sample['bboxes']) == target['cls'].shape[0], 'not equal!'
                if len(sample['bboxes']) > 0:
                    # image
                    image = sample['image']

                    # box
                    target['boxes'] = torch.tensor(sample['bboxes'])
                    target['boxes'][:,[0,1,2,3]] = target['boxes'][:,[1,0,3,2]]

                    # label
                    target['cls'] = torch.stack(sample['labels'])
                    break

        return image, target

    def __len__(self) -> int:
        return self.image_ids.shape[0]

Basic Transform
def get_transforms():
    return A.Compose(
        [
            A.Resize(height=IMG_SIZE, width=IMG_SIZE, p=1.0),
            ToTensorV2(p=1.0),
        ],
        p=1.0,
        bbox_params=A.BboxParams(
            format='pascal_voc',
            min_area=0,
            min_visibility=0,
            label_fields=['labels']
        )
    )

Mosaic Augmentation
Note, It should be defined inside the data loader. The main issue is, in this augmentation, while iterating will all 4 samples to create such augmentation, image and bounding_box is rescaled as follows:
mosaic_image[y1a:y2a, x1a:x2a] = image[y1b:y2b, x1b:x2b]

offset_x = x1a - x1b
offset_y = y1a - y1b
boxes[:, 0] += offset_x
boxes[:, 1] += offset_y
boxes[:, 2] += offset_x
boxes[:, 3] += offset_y

In this way, how would I select the relevant class labels for those selected bounding_box? Please, see the full code below:
def load_mosaic_image_and_boxes(self, index, s=3000,
                                    minfrac=0.25, maxfrac=0.75):
        self.mosaic_size = s
        xc, yc = np.random.randint(s * minfrac, s * maxfrac, (2,))

        # random other 3 sample
        indices = [index] + random.sample(range(len(self.image_ids)), 3)

        mosaic_image = np.zeros((s, s, 3), dtype=np.float32)
        final_boxes  = [] # box for the sub-region
        final_labels = [] # relevant class labels

        for i, index in enumerate(indices):
            image, boxes, labels = self.load_image_and_boxes(index)

            if i == 0:    # top left
                x1a, y1a, x2a, y2a =  0,  0, xc, yc
                x1b, y1b, x2b, y2b = s - xc, s - yc, s, s # from bottom right
            elif i == 1:  # top right
                x1a, y1a, x2a, y2a = xc, 0, s , yc
                x1b, y1b, x2b, y2b = 0, s - yc, s - xc, s # from bottom left
            elif i == 2:  # bottom left
                x1a, y1a, x2a, y2a = 0, yc, xc, s
                x1b, y1b, x2b, y2b = s - xc, 0, s, s-yc   # from top right
            elif i == 3:  # bottom right
                x1a, y1a, x2a, y2a = xc, yc,  s, s
                x1b, y1b, x2b, y2b = 0, 0, s-xc, s-yc    # from top left

            # calculate and apply box offsets due to replacement
            offset_x = x1a - x1b
            offset_y = y1a - y1b
            boxes[:, 0] += offset_x
            boxes[:, 1] += offset_y
            boxes[:, 2] += offset_x
            boxes[:, 3] += offset_y

            # cut image, save boxes
            mosaic_image[y1a:y2a, x1a:x2a] = image[y1b:y2b, x1b:x2b]
            final_boxes.append(boxes)

            '''
            ATTENTION:
            Need some mechanism to get relevant class labels
            '''
            final_labels.append(labels)

        # collect boxes
        final_boxes  = np.vstack(final_boxes)
        final_labels = np.hstack(final_labels)

        # clip boxes to the image area
        final_boxes[:, 0:] = np.clip(final_boxes[:, 0:], 0, s).astype(np.int32)
        w = (final_boxes[:,2] - final_boxes[:,0])
        h = (final_boxes[:,3] - final_boxes[:,1])

        # discard boxes where w or h <10
        final_boxes = final_boxes[(w>=self.size_limit) & (h>=self.size_limit)]

        return mosaic_image, final_boxes, final_labels

That's it. I hope, I make my query clear. Your suggestion would be highly appreciated.

With this query, I've also update another very related query which I've asked a few days ago but didn't get enough response. I update that query too and make it more clear. In case you're interested, please, Link: Stratified K-Fold For Multi-Class Object Detection?


title: how to calculate mahalanobis distance in pytorch?

body: What is the most efficient way to calculate the mahalanobis distance: in pytorch?


title: what's wrong might be if it accurs in pytorch&&tqdm __init__() missing 1 required positional argument: 'code'

body:
I use pytorch:0.2.0.post3 and tqdm:4.19.4 to train my image dataset. It works at first but as it runs, error accured

I don't know how to do and I think it might be code/gpu/version's fault or not??

related code as below.

def train(train_loader, model, optimizer, criterion, epoch):
    losses = []
    model.train()
    lrs = [param_group['lr'] for param_group in optimizer.param_groups]

    logging.info('Epoch: {} | lrs: {}'.format(epoch, lrs))
    for i, (inputs, O, targets) in tqdm.tqdm(enumerate(train_loader), total=len(train_loader)):
        inputs, O, targets = variable(inputs), variable(O), variable(targets)
        outputs = model(inputs, O)
        loss = criterion(outputs, targets)
        optimizer.zero_grad()
        batch_size = inputs.size(0)
        (batch_size * loss).backward()
        optimizer.step()

        losses.append(loss.data[0])

        if (i + 1) % 100 == 0:
            logging.info('Step: {}, train_loss: {}'.format(i + 1, np.mean(losses[-100:])))

    train_loss = np.mean(losses)
    logging.info('train_loss: {}'.format(train_loss))

the error as below.

100%|█████████████████████████████████████████████████████████████████████████████████████████████████████| 1000/1000 [08:44<00:00,  1.91it/s]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████| 1000/1000 [07:42<00:00,  2.16it/s]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████| 1000/1000 [07:34<00:00,  2.20it/s]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████| 1000/1000 [07:34<00:00,  2.20it/s]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████| 1000/1000 [07:29<00:00,  2.22it/s]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████| 1000/1000 [07:37<00:00,  2.19it/s]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████| 1000/1000 [07:31<00:00,  2.21it/s]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████| 1000/1000 [07:32<00:00,  2.21it/s]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████| 1000/1000 [07:40<00:00,  2.17it/s]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████| 1000/1000 [19:49<00:00,  1.19s/it]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████| 1000/1000 [19:58<00:00,  1.20s/it]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████| 1000/1000 [18:28<00:00,  1.11s/it]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████| 1000/1000 [19:42<00:00,  1.18s/it]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████| 1000/1000 [19:19<00:00,  1.16s/it]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████| 1000/1000 [19:33<00:00,  1.17s/it]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████| 1000/1000 [19:33<00:00,  1.17s/it]
 29%|█████████████████████████████▋                                                                        | 291/1000 [06:03<14:46,  1.25s/it]Traceback (most recent call last):
  File "init_train.py", line 15, in <module>
    main()
  File "init_train.py", line 11, in main
    train_model(model_name, use_pseudo=False)
  File "/hd2/maoyuzhe/kaggle2018/kaggle-camera-model-identification-master/src/ilya/pipeline/core/train_core.py", line 101, in train_model
    run_training(model_name, model, ids_train, ids_val, batch_size, config.STEPS, use_d4)
  File "/hd2/maoyuzhe/kaggle2018/kaggle-camera-model-identification-master/src/ilya/pipeline/core/train_core.py", line 60, in run_training
    optimizer, scheduler, criterion, model_name)
  File "/hd2/maoyuzhe/kaggle2018/kaggle-camera-model-identification-master/src/ilya/pipeline/core/train_utils.py", line 69, in train_and_validate
    optimizer, loss_fn, epoch)
  File "/hd2/maoyuzhe/kaggle2018/kaggle-camera-model-identification-master/src/ilya/pipeline/core/train_utils.py", line 111, in train
    for i, (inputs, O, targets) in tqdm.tqdm(enumerate(train_loader), total=len(train_loader)):
  File "/home/maoyuzhe/anaconda2/envs/py36/lib/python3.6/site-packages/tqdm/_tqdm.py", line 953, in __iter__
    for obj in iterable:
  File "/home/maoyuzhe/anaconda2/envs/py36/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 201, in __next__
    return self._process_next_batch(batch)
  File "/home/maoyuzhe/anaconda2/envs/py36/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 221, in _process_next_batch
    raise batch.exc_type(batch.exc_msg)
TypeError: __init__() missing 1 required positional argument: 'code'


title: Error while converting pytorch model to core-ml

body:
C = torch.cat((A,B),1)

shape of tensors:

A is (1, 128, 128, 256)
B is (1, 1, 128, 256)

Expected C value is (1, 129, 128, 256)

This code is working on pytorch, but while converting to core-ml it gives me below error:

"Error while converting op of type: {}. Error message: {}\n".format(node.op_type, err_message, )
TypeError: Error while converting op of type: Concat. Error message: unable to translate constant array shape to CoreML shape"


title: How do I make the bounding boxes in yolo v3 tighter (closer to the objects)?

body:
I'm following this Repo on creating Yolo v3 model from scratch in PyTorch. The only problem is that the bounding boxes are not as tight (close to the objects) in most images I tried. I compared them to the tutorial on creating Yolo v3 model but using TensorFlow. The tensorflow model produces excellent bounding boxed that are as tight as possible to the objects.
I tried to understand how the calculations are different between the two, but I'm finding myself getting stuck with the differences between torch and tf.
I believe the code for the bounding boxes in the tf tutorial comes from here:

def yolo_layer(inputs, n_classes, anchors, img_size, data_format):
    """Creates Yolo final detection layer.

    Detects boxes with respect to anchors.

    Args:
        inputs: Tensor input.
        n_classes: Number of labels.
        anchors: A list of anchor sizes.
        img_size: The input size of the model.
        data_format: The input format.

    Returns:
        Tensor output.
    """
    n_anchors = len(anchors)

    inputs = tf.layers.conv2d(inputs, filters=n_anchors * (5 + n_classes),
                              kernel_size=1, strides=1, use_bias=True,
                              data_format=data_format)

    shape = inputs.get_shape().as_list()
    grid_shape = shape[2:4] if data_format == 'channels_first' else shape[1:3]
    if data_format == 'channels_first':
        inputs = tf.transpose(inputs, [0, 2, 3, 1])
    inputs = tf.reshape(inputs, [-1, n_anchors * grid_shape[0] * grid_shape[1],
                                 5 + n_classes])

    strides = (img_size[0] // grid_shape[0], img_size[1] // grid_shape[1])

    box_centers, box_shapes, confidence, classes = \
        tf.split(inputs, [2, 2, 1, n_classes], axis=-1)

    x = tf.range(grid_shape[0], dtype=tf.float32)
    y = tf.range(grid_shape[1], dtype=tf.float32)
    x_offset, y_offset = tf.meshgrid(x, y)
    x_offset = tf.reshape(x_offset, (-1, 1))
    y_offset = tf.reshape(y_offset, (-1, 1))
    x_y_offset = tf.concat([x_offset, y_offset], axis=-1)
    x_y_offset = tf.tile(x_y_offset, [1, n_anchors])
    x_y_offset = tf.reshape(x_y_offset, [1, -1, 2])
    box_centers = tf.nn.sigmoid(box_centers)
    box_centers = (box_centers + x_y_offset) * strides

    anchors = tf.tile(anchors, [grid_shape[0] * grid_shape[1], 1])
    box_shapes = tf.exp(box_shapes) * tf.to_float(anchors)

    confidence = tf.nn.sigmoid(confidence)

    classes = tf.nn.sigmoid(classes)

    inputs = tf.concat([box_centers, box_shapes,
                        confidence, classes], axis=-1)

    return inputs

While the code for the bounding boxes for the pytorch model comes from here, and the explanation:

def bbox_iou(box1, box2):
    """
    Returns the IoU of two bounding boxes

    """
    #Get the coordinates of bounding boxes
    b1_x1, b1_y1, b1_x2, b1_y2 = box1[:,0], box1[:,1], box1[:,2], box1[:,3]
    b2_x1, b2_y1, b2_x2, b2_y2 = box2[:,0], box2[:,1], box2[:,2], box2[:,3]

    #get the corrdinates of the intersection rectangle
    inter_rect_x1 =  torch.max(b1_x1, b2_x1)
    inter_rect_y1 =  torch.max(b1_y1, b2_y1)
    inter_rect_x2 =  torch.min(b1_x2, b2_x2)
    inter_rect_y2 =  torch.min(b1_y2, b2_y2)

    #Intersection area
    inter_area = torch.clamp(inter_rect_x2 - inter_rect_x1 + 1, min=0) * torch.clamp(inter_rect_y2 - inter_rect_y1 + 1, min=0)

    #Union Area
    b1_area = (b1_x2 - b1_x1 + 1)*(b1_y2 - b1_y1 + 1)
    b2_area = (b2_x2 - b2_x1 + 1)*(b2_y2 - b2_y1 + 1)

    iou = inter_area / (b1_area + b2_area - inter_area)

    return iou

def predict_transform(prediction, inp_dim, anchors, num_classes, CUDA = True):

    batch_size = prediction.size(0)
    stride =  inp_dim // prediction.size(2)
    grid_size = inp_dim // stride
    bbox_attrs = 5 + num_classes
    num_anchors = len(anchors)

    prediction = prediction.view(batch_size, bbox_attrs*num_anchors, grid_size*grid_size)
    prediction = prediction.transpose(1,2).contiguous()
    prediction = prediction.view(batch_size, grid_size*grid_size*num_anchors, bbox_attrs)
    anchors = [(a[0]/stride, a[1]/stride) for a in anchors]

    #Sigmoid the  centre_X, centre_Y. and object confidencce
    prediction[:,:,0] = torch.sigmoid(prediction[:,:,0])
    prediction[:,:,1] = torch.sigmoid(prediction[:,:,1])
    prediction[:,:,4] = torch.sigmoid(prediction[:,:,4])

    #Add the center offsets
    grid = np.arange(grid_size)
    a,b = np.meshgrid(grid, grid)

    x_offset = torch.FloatTensor(a).view(-1,1)
    y_offset = torch.FloatTensor(b).view(-1,1)

    if CUDA:
        x_offset = x_offset.cuda()
        y_offset = y_offset.cuda()

    x_y_offset = torch.cat((x_offset, y_offset), 1).repeat(1,num_anchors).view(-1,2).unsqueeze(0)

    prediction[:,:,:2] += x_y_offset

    #log space transform height and the width
    anchors = torch.FloatTensor(anchors)

    if CUDA:
        anchors = anchors.cuda()

    anchors = anchors.repeat(grid_size*grid_size, 1).unsqueeze(0)
    prediction[:,:,2:4] = torch.exp(prediction[:,:,2:4])*anchors

    prediction[:,:,5: 5 + num_classes] = torch.sigmoid((prediction[:,:, 5 : 5 + num_classes]))

    prediction[:,:,:4] *= stride

    return prediction


title: Using the full PyTorch Transformer Module

body:
I tried asking this question on the PyTorch forums but didn't get any response so I am hoping someone here can help me. Additionally, if anyone has a good example of using the transformer module please share it as the documentation only shows using a simple linear decoder. For the transformer I'm aware that we generally feed in the actual target sequence. Therefore, my first question is that prior to the transformer I have a standard linear layer to transform my time series sequence along with positional encodings. According to the documentation the transformer module code the src and trg sequence need to be the same dimension.
    from torch.nn.modules.transformer import Transformer
    class TransformerTimeSeries(torch.nn.Module):
       def __init__(self, n_time_series, d_model=128):
          super()._init__()
          self.dense_shape = torch.nn.Linear(n_time_series, d_model)
          self.pe = SimplePositionalEncoding(d_model)
          self.transformer = Transformer(d_model, nhead=8)

So I was wondering can I simply do something like this or will this somehow leak information about the target? I'm still not actually sure how loss.backward() works so I'm not sure if this will cause problems.
    def forward(self, x, t):
       x = self.dense_shape(x)
       x = self.pe(x)
       t = self.dense_shape(t)
       t = self.pe(t)
       x = self.transformer(x, t)

Secondly, does the target sequence need any sort of offset? So for instance if I have the time series [0,1,2,3,4,5,6,7] and I want to feed in [0,1,2,3] to predict [4,5,6,7] (tgt)? Would I simply feed it in like that or is it more complicated? Typically BERT and those models have [CLS] and [SEP] tokens to denote the beginning and end of sentences however, for time series I assume I don't need a separator time step.


title: Different behavior of gather() function as seen in tensorflow and pytorch

body:
I have a tensor of shape (16, 4096, 3). I have another tensor of indices of shape (16, 32768, 3). I am trying to collect the values along dim=1. This was initially done in pytorch using gather function as shown below-

# a.shape (16L, 4096L, 3L)
# idx.shape (16L, 32768L, 3L)
b = a.gather(1, idx)
# b.shape (16L, 32768L, 3L)

Please note that the size of output b is the same as that of idx. However, when I apply gather function of tensorflow, I get a completely different output. The output dimension was found mismatching as shown below-

b = tf.gather(a, idx, axis=1)
# b.shape (16, 16, 32768, 3, 3)

I also tried using tf.gather_nd but got in vain. See below-

b = tf.gather_nd(a, idx)
# b.shape (16, 32768)

Why am I getting different shapes of tensors? I want to get the tensor of the same shape as calculated by pytorch.

How to achieve the same result as given by pytorch?


title: Pytorch - Apply pooling on specific dimension

body:
I have a 3 dimension vector. I would like to perform a 1d max pool on the second dimension.

According to the documentation of pytorch the pooling is always performed on the last dimension.

  https://pytorch.org/docs/stable/nn.html#maxpool1d

For example:

>>> x = torch.rand(5, 64, 32)
>>> pool = nn.MaxPool1d(2, 2)
>>> pool(x).shape
torch.Size([5, 64, 16])

My desired output:

torch.Size([5, 32, 32])

How can i do that?


title: LSTM example with multiple GPU error: module 'torch' has no attribute 'long'

body:
There is an example of LSTM for pytorch.

The below code works fine when using CPU or 1 GPU. However, when I use more than 1 GPU, it gives an error:

  AttributeError: module 'torch' has no attribute 'long'

The code that caused the error:

def prepare_sequence(seq, to_ix):
    idxs = [to_ix[w] for w in seq]
    return torch.tensor(idxs, dtype=torch.long)

Why it doesn't work for multiple GPUs?


title: Pytorch Double DQN not working properly

body:
I'm trying to make a double dqn network for cartpole-v0, but the network doesn't seem to be working as expected and stagnates at around 8-9 reward. What am I doing wrong?

Each step in the learning phase:

def make_step(model, target_model, optimizer, criterion, observation, action, reward, next_observation):
    inp_obv = torch.Tensor(observation)
    q = model(inp_obv)
    q_argmax = torch.argmax(q.data)
    q = q[action]

    inp_next_obv = torch.Tensor(next_observation)
    q_next = target_model(inp_next_obv)
    q_a_next = q_next[q_argmax]

    #LHS of the double DQN equation
    obv_reward = q

    #RHS of the double DQN equation
    target_reward = torch.Tensor([reward]) + GAMMA*q_a_next.detach()

    #Backprop
    loss = criterion(obv_reward, target_reward) #MSELoss
    loss.backward()

Code wrapping make_step:

optimizer.zero_grad() #RMSprop on net
if e%2 == 0:
    target_net.load_state_dict(net.state_dict())
for i in range(len(data)):
    observation, action, reward, next_observation = data[i]
    make_step(net, target_net, optimizer, criterion, observation, action, reward, next_observation)

GAMMA *= GAMMA
optimizer.step()

What am I doing wrong? Thank you.


title: How to create a custom loss function in Pytorch

body:
I’m trying to train a network by my own loss function.
I can train a network with loss functions are included in the PyTorch.

But, I meet the challenge when I am trying to define my own loss function.
Indeed, I need to a correct example to train a network by custom loss function in details

My loss function is:


title: PyTorch: unable to use the Gloo backend

body:
I am trying to use PyTorch to train a model in a (multi-CPU) distributed manner. However, when I ran the code, I got the following error message

RuntimeError: the Gloo backend is not available; try to recompile the THD package with Gloo support at /opt/conda/conda-bld/pytorch-cpu_1544218188686/work/torch/lib/THD/process_group/General.cpp:20

I tried this both on my own Macbook and the university's slurm cluster (Red Hat 4.8.5-16). The Python version is 3.6. I installed PyTorch using the command conda install pytorch-cpu torchvision-cpu -c pytorch (the version without CUDA support).

I was wondering if I have to re-install PyTorch from the source or install Gloo manually. I was a little confused since according to PyTorch's documentation,

  Since version 0.2.0, the Gloo backend is automatically included with the pre-compiled binaries of PyTorch

The code is exactly the example code available at https://github.com/seba-1511/dist_tuto.pth/blob/gh-pages/train_dist.py

import os
import torch
import torch.distributed.deprecated as dist
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

from math import ceil
from random import Random
from torch.multiprocessing import Process
from torch.autograd import Variable
from torchvision import datasets, transforms

class Partition(object):
    """ Dataset-like object, but only access a subset of it. """

    def __init__(self, data, index):
        self.data = data
        self.index = index

    def __len__(self):
        return len(self.index)

    def __getitem__(self, index):
        data_idx = self.index[index]
        return self.data[data_idx]

class DataPartitioner(object):
    """ Partitions a dataset into different chuncks. """

    def __init__(self, data, sizes=[0.7, 0.2, 0.1], seed=1234):
        self.data = data
        self.partitions = []
        rng = Random()
        rng.seed(seed)
        data_len = len(data)
        indexes = [x for x in range(0, data_len)]
        rng.shuffle(indexes)

        for frac in sizes:
            part_len = int(frac * data_len)
            self.partitions.append(indexes[0:part_len])
            indexes = indexes[part_len:]

    def use(self, partition):
        return Partition(self.data, self.partitions[partition])

class Net(nn.Module):
    """ Network architecture. """

    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 10, kernel_size=5)
        self.conv2 = nn.Conv2d(10, 20, kernel_size=5)
        self.conv2_drop = nn.Dropout2d()
        self.fc1 = nn.Linear(320, 50)
        self.fc2 = nn.Linear(50, 10)

    def forward(self, x):
        x = F.relu(F.max_pool2d(self.conv1(x), 2))
        x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))
        x = x.view(-1, 320)
        x = F.relu(self.fc1(x))
        x = F.dropout(x, training=self.training)
        x = self.fc2(x)
        return F.log_softmax(x)

def partition_dataset():
    """ Partitioning MNIST """
    dataset = datasets.MNIST(
        './data',
        train=True,
        download=True,
        transform=transforms.Compose([
            transforms.ToTensor(),
            transforms.Normalize((0.1307, ), (0.3081, ))
        ]))
    size = dist.get_world_size()
    bsz = int(128 / float(size))
    partition_sizes = [1.0 / size for _ in range(size)]
    partition = DataPartitioner(dataset, partition_sizes)
    partition = partition.use(dist.get_rank())
    train_set = torch.utils.data.DataLoader(
        partition, batch_size=bsz, shuffle=True)
    return train_set, bsz

def average_gradients(model):
    """ Gradient averaging. """
    size = float(dist.get_world_size())
    for param in model.parameters():
        dist.all_reduce(param.grad.data, op=dist.reduce_op.SUM, group=0)
        param.grad.data /= size

def run(rank, size):
    """ Distributed Synchronous SGD Example """
    torch.manual_seed(1234)
    train_set, bsz = partition_dataset()
    model = Net()
    model = model
#    model = model.cuda(rank)
    optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.5)

    num_batches = ceil(len(train_set.dataset) / float(bsz))
    for epoch in range(10):
        epoch_loss = 0.0
        for data, target in train_set:
            data, target = Variable(data), Variable(target)
#            data, target = Variable(data.cuda(rank)), Variable(target.cuda(rank))
            optimizer.zero_grad()
            output = model(data)
            loss = F.nll_loss(output, target)
            epoch_loss += loss.data.item()
            loss.backward()
            average_gradients(model)
            optimizer.step()
        print('Rank ',
              dist.get_rank(), ', epoch ', epoch, ': ',
              epoch_loss / num_batches)

def init_processes(rank, size, fn, backend='gloo'):
    """ Initialize the distributed environment. """
    os.environ['MASTER_ADDR'] = '127.0.0.1'
    os.environ['MASTER_PORT'] = '29500'
    dist.init_process_group(backend, rank=rank, world_size=size)
    fn(rank, size)

if __name__ == "__main__":
    size = 2
    processes = []
    for rank in range(size):
        p = Process(target=init_processes, args=(rank, size, run))
        p.start()
        processes.append(p)

    for p in processes:
        p.join()


title: Getting alignment/attention during translation in OpenNMT-py

body: Does anyone know how to get the alignments weights when translating in Opennmt-py? Usually the only output are the resulting sentences and I have tried to find a debugging flag or similar for the attention weights. So far, I have been unsuccessful.


title: Cuda 10.2 and pytorch 1.5 – OSError: [WinError 126] The specified module could not be found

body:
Getting below issue when after installing cuda 10.2 and installing pytorch 1.5 ‘conda install pytorch torchvision cudatoolkit=10.2 -c pytorch’ . Any ideas how to fix this issue?

OSError Traceback (most recent call last)
in
----> 1 import torch

~\Documents\Anaconda3\lib\site-packages\torch_init_.py in
79 dlls = glob.glob(os.path.join(th_dll_path, ‘*.dll’))
80 for dll in dlls:
—> 81 ctypes.CDLL(dll)
82
83

~\Documents\Anaconda3\lib\ctypes_init_.py in init(self, name, mode, handle, use_errno, use_last_error)
362
363 if handle is None:
–> 364 self._handle = _dlopen(self._name, mode)
365 else:
366 self._handle = handle

OSError: [WinError 126] The specified module could not be found


title: How to get padding mask from input ids?

body:
Considering a batch of 4 pre-processed sentences (tokenization, numericalizing and padding) shown below:

batch = torch.tensor([
                      [1, 2, 0, 0],
                      [4, 0, 0, 0],
                      [3, 5, 6, 7]
])

where 0 states for [PAD] token.

Thus, what would be an efficient approach to generate a padding masking tensor of the same shape as the batch assigning zero at [PAD] positions and assigning one to other input data (sentence tokens)?

In the example above it would be something like:

padding_masking=
tensor([
                      [1, 1, 0, 0],
                      [1, 0, 0, 0],
                      [1, 1, 1, 1]
])


title: Modifying a pretrained model in PyTorch

body:
I am attempting to modify this particular section of code in the mobilenetv2 model
(17): InvertedResidual(
      (conv): Sequential(
        (0): ConvBNReLU(
          (0): Conv2d(160, 960, kernel_size=(1, 1), stride=(1, 1), bias=False)
          (1): BatchNorm2d(960, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (2): ReLU6(inplace=True)
        )

After the conv2d section, I want to add a max-pooling layer but I am having trouble figuring out to do so. I suspect it would be similar to doing something like this:
MobileNet.features = nn.Sequential(nn.Linear(1280, 1000), nn.LeakyReLU(), nn.Dropout(0.5), nn.Linear(1000,3), nn.LogSoftmax(dim=1))

Where I would do something like:
MobileNet.features[17].conv[0] = nn.ConvBRELU(nn.Conv2d(),nn.maxpool,nn.BatchNorm(),nn.ReLU())

but when I tried that I got the error message module 'torch.nn' has no attribute 'ConvBNReLU'
How can I go about modifying the provided section of code?


title: Run into OSError: [Errno 5] Input/output error. when reading images

body:
I'm using python to train my models and run into this problem when i try to transform a specific picture. This picture does exist, it can be successfully read by PIL， but will report this error when going through transform. I'm not so sure how to deal with this situation.
Traceback (most recent call last):
  File "/home/wangwenlong/文件保存/img_read.py", line 31, in <module>
    img = train_trans(img)
  File "/home/wangwenlong/anaconda3/envs/pt1.11/lib/python3.6/site-packages/torchvision/transforms/transforms.py", line 61, in __call__
    img = t(img)
  File "/home/wangwenlong/anaconda3/envs/pt1.11/lib/python3.6/site-packages/torchvision/transforms/transforms.py", line 196, in __call__
    return F.resize(img, self.size, self.interpolation)
  File "/home/wangwenlong/anaconda3/envs/pt1.11/lib/python3.6/site-packages/torchvision/transforms/functional.py", line 246, in resize
    return img.resize(size[::-1], interpolation)
  File "/home/wangwenlong/anaconda3/envs/pt1.11/lib/python3.6/site-packages/PIL/Image.py", line 1817, in resize
    self.load()
  File "/home/wangwenlong/anaconda3/envs/pt1.11/lib/python3.6/site-packages/PIL/ImageFile.py", line 224, in load
    s = read(self.decodermaxblock)
  File "/home/wangwenlong/anaconda3/envs/pt1.11/lib/python3.6/site-packages/PIL/PngImagePlugin.py", line 668, in load_read
    return self.fp.read(read_bytes)
OSError: [Errno 5] Input/output error


title: Pytorch transformer forward function masks implementation for decoder forward function

body:
I am trying to use and learn PyTorch Transformer with DeepMind math dataset. I have tokenized (char not word) sequence that is fed into model. Models forward function is doing once forward for encoder and multiple forwards for decoder (till all batch outputs reach  token, this is still TODO).
I am struggling with Transformer masks and decoder forward as it throws the error:
    k = k.contiguous().view(-1, bsz * num_heads, head_dim).transpose(0, 1)
    RuntimeError: shape '[-1, 24, 64]' is invalid for input of size 819200.

Source is N = 32, S = 50, E = 512. Target is N = 32, S = 3, E = 512.
It is possible that I have wrong implementation of masks or that source and target lengths are different, not realy sure.
class PositionalEncoding(nn.Module):
# function to positionally encode src and target sequencies
def __init__(self, d_model, dropout=0.1, max_len=5000):
    super(PositionalEncoding, self).__init__()
    self.dropout = nn.Dropout(p=dropout)
    pe = torch.zeros(max_len, d_model)
    position = torch.arange(0, max_len, dtype=torch.float).unsqueeze(1)
    div_term = torch.exp(torch.arange(0, d_model, 2).float() * (-math.log(10000.0) / d_model))
    pe[:, 0::2] = torch.sin(position * div_term)
    pe[:, 1::2] = torch.cos(position * div_term)
    pe = pe.unsqueeze(0).transpose(0, 1)
    self.register_buffer('pe', pe)

def forward(self, x):
    x = x + self.pe[:x.size(0), :]
    return self.dropout(x)

class MyTransformerModel(nn.Module):
# should implement init and forward function
# define separate functions for masks
# define forward function with
# implement:
#  embedding layer
#  positional encoding
#  encoder layer
#  decoder layer
#  final classification layer
# encoder -> forward once
# decoder -> forward multiple times (for one encoder forward)
# decoder output => concatenate to input e.g. decoder_input = torch.cat([decoder_input], [decoder_output])
# early stopping => all in batch reach <eos> token
def __init__(self, vocab_length = 30, sequence_length = 512, num_encoder_layers = 3, num_decoder_layers = 2, num_hidden_dimension = 256, feed_forward_dimensions = 1024, attention_heads = 8, dropout = 0.1, pad_idx = 3, device = "CPU", batch_size = 32):
    super(MyTransformerModel, self).__init__()
    self.src_embedding = nn.Embedding(vocab_length, sequence_length)
    self.pos_encoder = PositionalEncoding(sequence_length, dropout)
    self.src_mask = None # attention mask
    self.memory_mask = None # attention mask
    self.pad_idx = pad_idx
    self.device = device
    self.batch_size = batch_size
    self.transformer = nn.Transformer(
        sequence_length,
        attention_heads,
        num_encoder_layers,
        num_decoder_layers,
        feed_forward_dimensions,
        dropout,
    )

def src_att_mask(self, src_len):
    mask = (torch.triu(torch.ones(src_len, src_len)) == 1).transpose(0, 1)
    mask = mask.float().masked_fill(mask == 0, float('-inf')).masked_fill(mask == 1, float(0.0))
    return mask

def no_peak_att_mask(self, batch_size, src_len, time_step):
    mask = np.zeros((batch_size, src_len), dtype=bool)
    mask[:, time_step: ] = 1 # np.NINF
    mask = torch.from_numpy(mask)
    return mask

def make_src_key_padding_mask(self, src):
    # mask "<pad>"
    src_mask = src.transpose(0, 1) == self.pad_idx
    return src_mask.to(self.device)

def make_trg_key_padding_mask(self, trg):
    tgt_mask = trg.transpose(0, 1) == self.pad_idx
    return tgt_mask.to(self.device)

def forward(self, src, trg):
    src_seq_length, N = src.shape
    trg_seq_length, N = trg.shape
    embed_src = self.src_embedding(src)
    position_embed_src =  self.pos_encoder(embed_src)
    embed_trg = self.src_embedding(trg)
    position_embed_trg = self.pos_encoder(embed_trg)
    src_padding_mask = self.make_src_key_padding_mask(src)
    trg_padding_mask = self.make_trg_key_padding_mask(trg)
    trg_mask = self.transformer.generate_square_subsequent_mask(trg_seq_length).to(self.device)
    time_step = 1
    att_mask = self.no_peak_att_mask(self.batch_size, src_seq_length, time_step).to(self.device)
    encoder_output = self.transformer.encoder.forward(position_embed_src, src_key_padding_mask = src_padding_mask)
    # TODO : implement loop for transformer decoder forward fn, implement early stopping
    # where to feed decoder_output?
    decoder_output = self.transformer.decoder.forward(position_embed_trg, encoder_output, trg_mask, att_mask, trg_padding_mask, src_padding_mask)
    return decoder_output

Can anyone pin point where I have made a mistake?


title: Pytorch: 1.7.1+cu110- torch.nn.modules.module.ModuleAttributeError: 'LSTM' object has no attribute '_flat_weights_names'

body:
I am trying to implement a Generative Adversarial Network, however while testing I am coming across this error.
This is the code I am attempting to run: Obj-GAN
import os
import argparse
import logging
import numpy as np
import math
import torch
from torch.optim.lr_scheduler import StepLR

import seq2seq
from seq2seq.trainer import SupervisedTrainer
from seq2seq.models import PreEncoderRNN, DecoderRNN, Seq2seq
from seq2seq.loss import Perplexity, BBLoss
from seq2seq.optim import Optimizer
from seq2seq.dataset.prepare_dataset import prepare_data, prepare_test_data, \
    random_batch, get_class_sta
from seq2seq.evaluator import Evaluator
from seq2seq.util.checkpoint import Checkpoint

import sys
if sys.version_info[0] == 2:
    import cPickle as pickle
else:
    import pickle

os.environ["CUDA_VISIBLE_DEVICES"] = "0"

parser = argparse.ArgumentParser()
parser.add_argument('--train_path', action='store', dest='train_path',
    default=r'C:\Users\niti\OneDrive\Desktop\ObjectGAN\data\coco\box_label\input_train2014.txt',
    help='Path to train data')
parser.add_argument('--dev_path', action='store', dest='dev_path',
    default=r'C:\Users\niti\OneDrive\Desktop\ObjectGAN\data\coco\box_label\input_val2014.txt',
    help='Path to dev data')
parser.add_argument('--train_filename_path', action='store', dest='train_filename_path',
    default=r'C:\Users\niti\OneDrive\Desktop\ObjectGAN\data\coco\box_label\filenames_train2014.txt',
    help='Path to train filename data')
parser.add_argument('--dev_filename_path', action='store', dest='dev_filename_path',
    default=r'C:\Users\niti\OneDrive\Desktop\ObjectGAN\data\coco\box_label\filenames_val2014.txt',
    help='Path to dev filename data')
parser.add_argument('--mean_std_path', action='store', dest='mean_std_path',
    default=r'C:\Users\niti\OneDrive\Desktop\ObjectGAN\data\coco\box_label\mean_std_train2014.txt',
    help='Path to dev data')
parser.add_argument('--gaussian_dict_path', action='store', dest='gaussian_dict_path',
    default=r'C:\Users\niti\OneDrive\Desktop\ObjectGAN\data\coco\box_label\gaussian_dict.npy',
    help='Path to gaussian dict')
parser.add_argument('--vocab_path', action='store', dest='vocab_path',
    default=r'C:\Users\niti\OneDrive\Desktop\ObjectGAN\data\coco\captions.pickle',
    help='Path to the vocab path')
parser.add_argument('--box_saving_folder', action='store', dest='box_saving_folder',
    default=r'C:\Users\niti\OneDrive\Desktop\ObjectGAN\data\coco\gen_masks',
    help='Path to box saving folder')
parser.add_argument('--expt_dir', action='store', dest='expt_dir', default='experiment',
    help='Path to experiment directory. If load_checkpoint is True, \
    then path to checkpoint directory has to be provided')
parser.add_argument('--load_checkpoint', action='store', dest='load_checkpoint',
    default=r'C:\Users\niti\OneDrive\Desktop\ObjectGAN\data\coco\pretrained\box_ckpt', help='The name of the checkpoint to load, \
    usually an encoded time string')
parser.add_argument('--resume', action='store_true', dest='resume', default=False,
    help='Indicates if training has to be resumed from the latest checkpoint')
parser.add_argument('--log-level', dest='log_level', default='info', help='Logging level.')
parser.add_argument('--batch_size', type=int, default=1, help='The batch size.')
parser.add_argument('--gmm_comp_num', type=int, default=5, help='The number of GMM components.')
parser.add_argument('--lamda1', type=float, default=1.0,
    help='The balancing parameter for label loss.')
parser.add_argument('--lamda2', type=float, default=1.0,
    help='The balancing parameter for box loss.')
parser.add_argument('--count_smooth', type=float, default=1e5,
    help='The balancing parameter for box loss.')
parser.add_argument('--is_training', type=int, default=1, help='The state for training or test')
parser.add_argument('--max_len', type=int, default=150, help='The max length for sequences')
parser.add_argument('--min_len', type=int, default=1, help='The min length for sequences')
parser.add_argument('--early_stop_len', type=int, default=10,
    help='The early-stop length for generation')
parser.add_argument('--output_opt', type=int, default=0, help='The output option (0/1)')
parser.add_argument('--embedding_dim', type=int, default=256, help='The embedding dimension')
parser.add_argument('--encoder_path', type=str,
    default=r'C:\Users\niti\OneDrive\Desktop\ObjectGAN\data\coco\pretrained\text_encoder100.pth',
    help='encoder path.')

opt = parser.parse_args()

LOG_FORMAT = '%(asctime)s %(name)-12s %(levelname)-8s %(message)s'
logging.basicConfig(format=LOG_FORMAT, level=getattr(logging, opt.log_level.upper()))
logging.info(opt)

if opt.is_training:
    # load vocab
    x = pickle.load(open(opt.vocab_path, 'rb'))
    ixtoword, wordtoix = x[2], x[3]
    del x

    # prepare dataset
    train_cap_lang, train_label_lang, train_tuples, dev_cap_lang, dev_label_lang, dev_tuples, \
        x_mean_std, y_mean_std, w_mean_std, r_mean_std = prepare_data(opt.train_path,
            opt.dev_path, opt.mean_std_path, opt.max_len, opt.min_len, ixtoword, wordtoix)

    weight = torch.ones(len(train_label_lang.word2index))
    for word in train_label_lang.word2index:
        if train_label_lang.word2count[word] == 0:
            continue
        index = train_label_lang.word2index[word]
        weight[index] = weight[index]*opt.count_smooth/float(math.pow(
            train_label_lang.word2count[word], 0.8))

    # Prepare loss
    pad = train_label_lang.word2index["<pad>"]
    lloss = Perplexity(weight, pad, opt.lamda1)
    bloss = BBLoss(opt.batch_size, opt.gmm_comp_num, opt.lamda2)
    if torch.cuda.is_available():
        lloss.cuda()
        bloss.cuda()

    print('train_label_lang.index2word:')
    for index in train_label_lang.index2word:
        print('{} : {} '.format(index, train_label_lang.index2word[index]))

    print('train_label_lang.word2count:')
    for word in train_label_lang.word2count:
        print('{} : {} '.format(word, train_label_lang.word2count[word]))

    hidden_size = opt.embedding_dim
    encoder = PreEncoderRNN(train_cap_lang.n_words, nhidden=opt.embedding_dim)
    state_dict = torch.load(opt.encoder_path, map_location=lambda storage, loc: storage)
    encoder.load_state_dict(state_dict)
    for p in encoder.parameters():
        p.requires_grad = False
    encoder.eval()

    if torch.cuda.is_available():
        encoder.cuda()
    torch.nn.Module.dump_patches=True
    seq2seq = None
    optimizer = None
    bidirectional = True
    if not opt.resume:
        # Initialize model
        decoder = DecoderRNN(train_label_lang.word2index, x_mean_std[0], y_mean_std[0],
            w_mean_std[0], r_mean_std[0], opt.batch_size, opt.max_len, hidden_size,
            opt.gmm_comp_num, dropout_p=0.2, use_attention=False, bidirectional=bidirectional)

        for param in decoder.parameters():
            param.data.uniform_(-0.08, 0.08)

        if torch.cuda.is_available():
            decoder.cuda()

    # train
    t = SupervisedTrainer(lloss=lloss, bloss=bloss, batch_size=opt.batch_size,
        checkpoint_every=100, print_every=50, expt_dir=opt.expt_dir,
        train_cap_lang=train_cap_lang, train_label_lang=train_label_lang, x_mean_std=x_mean_std,
        y_mean_std=y_mean_std, w_mean_std=w_mean_std, r_mean_std=r_mean_std)

    seq2seq = t.train(encoder, decoder, train_tuples, num_epochs=10, dev_data=dev_tuples,
        optimizer=optimizer, resume=opt.resume, is_training=opt.is_training)

elif not opt.is_training and opt.load_checkpoint is not None:
    opt.box_saving_folder = '%s_%s\ '%(opt.box_saving_folder, opt.load_checkpoint)

    logging.info("loading checkpoint from {}".format(os.path.join(opt.expt_dir,
        Checkpoint.CHECKPOINT_DIR_NAME, opt.load_checkpoint)))
    checkpoint_path = os.path.join(opt.expt_dir, Checkpoint.CHECKPOINT_DIR_NAME,
        opt.load_checkpoint)
    checkpoint = Checkpoint.load(checkpoint_path)
    decoder = checkpoint.model
    decoder.eval()

    cap_word2index = checkpoint.cap_word2index
    cap_index2word = checkpoint.cap_index2word
    label_word2index = checkpoint.label_word2index
    label_index2word = checkpoint.label_index2word

    if not os.path.isfile(opt.gaussian_dict_path):
        print('calculating means and stds of box positions and sizes...')
        get_class_sta(opt.train_path, opt.gaussian_dict_path)

    gaussian_dict = np.load(opt.gaussian_dict_path,allow_pickle='True').item()

    hidden_size = opt.embedding_dim
    encoder = PreEncoderRNN(len(cap_word2index), nhidden=opt.embedding_dim)
    state_dict = torch.load(opt.encoder_path, map_location=lambda storage, loc: storage)
    encoder.load_state_dict(state_dict)
    encoder.eval()

    if torch.cuda.is_available():
        encoder.cuda()
        decoder.cuda()

    # prepare dataset
    dev_cap_lang, dev_label_lang, dev_tuples, x_mean_std, y_mean_std, w_mean_std, r_mean_std, \
    keys = prepare_test_data(opt.dev_path, opt.mean_std_path, opt.max_len, opt.min_len,
        cap_word2index, cap_index2word, label_word2index, label_index2word, opt.dev_filename_path)

    evaluator = Evaluator(opt.batch_size, opt.early_stop_len, opt.expt_dir, dev_cap_lang,
        dev_label_lang, x_mean_std, y_mean_std, w_mean_std, r_mean_std, gaussian_dict,
        opt.box_saving_folder, opt.output_opt)
    evaluator.evaluate(encoder, decoder, dev_tuples, keys)

Traceback (most recent call last):   File "sample.py", line 186, in

decoder.cuda()   File "C:\Users\niti\AppData\Roaming\Python\Python38\site-packages\torch\nn\modules\module.py",
line 463, in cuda
return self._apply(lambda t: t.cuda(device))   File "C:\Users\niti\AppData\Roaming\Python\Python38\site-packages\torch\nn\modules\module.py",
line 359, in _apply
module._apply(fn)   File "C:\Users\niti\AppData\Roaming\Python\Python38\site-packages\torch\nn\modules\rnn.py",
line 160, in _apply
self._flat_weights = [(lambda wn: getattr(self, wn) if hasattr(self, wn) else None)(wn) for wn in self._flat_weights_names]
File
"C:\Users\niti\AppData\Roaming\Python\Python38\site-packages\torch\nn\modules\module.py",
line 778, in getattr
raise ModuleAttributeError("'{}' object has no attribute '{}'".format( torch.nn.modules.module.ModuleAttributeError: 'LSTM'
object has no attribute '_flat_weights_names'

I read in another forum about this issue arising in earlier versions of PyTorch (1.3 to be specific). But since I'm using 1.7 I cannot figure out what is causing the error.