title: Regarding number of epochs for torchvision models

body:
I was trying to find, how many epochs was the pretrained Alexnet model (available from torchvision) trained for on Imagenet and also what learning rate was used? I tried checking the checkpoint keys to see if any epoch info was stored.
Any suggestions on how to find it out?


title: Sagemaker endpoint inference with third-party libraries

body:
I'm trying to create an endpoint with custom libraries, particularly trying to install fastai.
Going over this tutorial which is apparently incomplete. Endpoint deployment was failing at health check because of the missing library.
This doc says I can pass a list of third-party libraries in requirements.txt file. The file should be in code folder inside the model.tar.gz archive. This actually lets me deploy the endpoint with no issues but still fails when running predictions: Received server error (500) from model with message "No module named 'fastai'".
Using PyTorch 1.3.1 estimator.
Contents of the requirements.txt is just fastai.
Wondering what could have gone wrong.


title: Breaking down a batch in pytorch leads to different results, why?

body:
I was trying something with batch processing in pytorch. In my code below, you may think of x as a batch of batch size 2 (each sample is a 10d vector). I use x_sep to denote the first sample in x.

import torch
import torch.nn as nn

class net(nn.Module):
    def __init__(self):
        super(net, self).__init__()
        self.fc1 = nn.Linear(10,10)

    def forward(self, x):
        x = self.fc1(x)
        return x

f = net()

x = torch.randn(2,10)
print(f(x[0])==f(x)[0])

Ideally, f(x[0])==f(x)[0] should give a tensor with all true entries. But the output on my computer is

tensor([False, False,  True,  True, False, False, False, False,  True, False])

Why does this happen? Is it a computational error? Or is it related to how the batch precessing in implemented in pytorch?

Update: I simplified the code a bit. The question remains the same.

My reasoning:
I believe f(x)[0]==f(x[0]) should have all its entries True because the law of matrix multiplication says so. Let us think of x as a 2x10 matrix, and think of the linear transformation f() as represented by matrix B (ignoring the bias for a moment). Then f(x)=xB by our notations. The law of matrix multiplication tells us that xB is equal to first multiply the two rows by B on the right separately, and then put the two rows back together. Translated back to the code, it is f(x[0])==f(x)[0] and f(x[1])==f(x)[1].

Even if we consider the bias, every row should have the same bias and the equality should still hold.

Also note that no training is done here. Hence how the weights are initialized shouldn't matter.


title: Resume plotting scalars to tensorboard once training is resumed in pytorch

body:
I am training my model using PyTorch, and I have already added saving and resume training functionalities, but the problem is that when I want to override the step value on tensorboard, the graph gets missed up (connecting the last step to the current step)

Example:

I am logging scalars to tensorboard every 10 iterations, and saving models every 100 iterations where iteration=step.

In the image below, I trained the model for 150 iterations (so I now have 15 step points in the tensorboard graph), and then I resume training from another 100 iterations and start logging in step 100, I was expected to override all step points after step 110 but it seems to connect the last step point (150) to step 110, how I can override all steps after step 100, and fix the graph ???

logging to tensorboard code:

if (OLD_ITERATIONS + i)%10 is 0 and i is not 0:
            wirter.add_scalar(tag='Accuracy/Train', scalar_value=accs/10, global_step=OLD_ITERATIONS+i)
            wirter.add_scalar(tag='Accuracy/Valid', scalar_value=val_accs/10, global_step=OLD_ITERATIONS+i)
            wirter.add_scalar(tag='Loss/Train', scalar_value=losses/10, global_step=OLD_ITERATIONS+i)
            wirter.add_scalar(tag='Loss/Valid', scalar_value=val_losses/10, global_step=OLD_ITERATIONS+i)

            accs = 0
            losses = 0
            val_accs = 0
            val_losses = 0


title: How to compute second-order derivative with respect to a vector in python (pytorch?)

body:
I am trying to write up a code that performs metropolized iid sampling, and I am having troubles computing the second-order derivative of a function with respect to a numpy ndarray. The code I wrote so far are shown below:
import numpy as np
from scipy.optimize import newton

w = np.array([4,5,3,6,2])
y = np.array([1,0,0,1,1])
lamb = np.array([0.001, 0.0045, 0.0072,0.0083, 0.0069)]
tau = np.array([0.0002, 0.00045, 0.000378, 0.00467, 0.00235])

# define function f
def f(gamma):

    return (-1) * (np.dot(y, np.dot(w, gamma)) - np.sum(np.log(np.exp(np.dot(w,gamma)) + 1)) - np.sum((np.subtract(gamma,lamb)) * (np.subtract(gamma,lamb)) / (2*tau*tau)))

# mode for the metropolized iid sampler
gamma_hat = newton(f, gamma, fprime=None, args=(), tol=1.48e-08, maxiter=50, fprime2=None)

# not sure how to calculate the variance for the same sampler

What I want to do:
To calculate the variance for the sampler, I want to take the second-order derivative of the function f with respect to the vector variable gamma, and then compute the value of the second derivative after substituting gamma = gamma_hat.
Can I do this in python? If needed, I am willing to use the pytorch for calculating the value of the second order derivative.
Thank you,


title: Converting numpy code to pytorch removing for loop

body:
My understanding is, using for loops in python is going to make our code slower. Hence, they should be avoided as far as possible. I have this piece of code which I am trying to convert to PyTorch. Is there a smart way to code it such that we do not have the for loop in it?
def nms(dets, scores, thresh):
    '''
    dets is a numpy array : num_dets, 6
    scores ia  nump array : num_dets,
    '''
    x1 = dets[:, 0]
    y1 = dets[:, 1]
    z1 = dets[:, 2]
    x2 = dets[:, 3]
    y2 = dets[:, 4]
    z2 = dets[:, 5]

    volume = (x2 - x1 + 1) * (y2 - y1 + 1) * (z2 - z1 + 1)
    order = scores.argsort()[::-1]  # get boxes with more ious first

    keep = []
    while order.size > 0:
        i = order[0]  # pick maxmum iou box
        keep.append(i)
        xx1 = np.maximum(x1[i], x1[order[1:]])
        yy1 = np.maximum(y1[i], y1[order[1:]])
        zz1 = np.maximum(z1[i], z1[order[1:]])
        xx2 = np.minimum(x2[i], x2[order[1:]])
        yy2 = np.minimum(y2[i], y2[order[1:]])
        zz2 = np.minimum(z2[i], z2[order[1:]])

        w = np.maximum(0.0, xx2 - xx1 + 1)  # maximum width
        h = np.maximum(0.0, yy2 - yy1 + 1)  # maxiumum height
        l = np.maximum(0.0, zz2 - zz1 + 1)  # maxiumum length
        inter = w * h * l
        ovr = inter / (volume[i] + volume[order[1:]] - inter)

        inds = np.where(ovr <= thresh)[0]
        order = order[inds + 1]

    return keep

I would really appreciate some help to see if we can use indexing to somehow to replace the for loop.


title: How to find the source code of torch.solve?

body: I am concerned whether torch.solve() examine the condition of the coefficient matrix for a linear system and employ desirable preconditionings; thus I am curious about its implementation details. I have read through several answers trying to track down the source file but in vain. I hope somebody can help me to locate its definition in the ATen library.


title: CUDA out of memory when running Bert with Pytorch (Previously worked)

body:
I am building a BERT binary classification on SageMaker using Pytorch.
Previously when I ran the model, I set the Batch size to 16 and the model were able to run successfully. However, yesterday after I stopped SageMaker and restarted the this morning, I can't run the model with Batch size as 16 any more. I am able to run the model with batch size 8.
However, the model is not producing the same result (of course). I didn't change anything else in between. All other settings are the same. (Except I change the SageMaker volume from 30GB to 200GB.)
Does anyone know what may cause this problem? I really want to reproduce the result with batch size 16.
Any answers will help and thank you in advance!


title: How to remove the NameError: module 'utils' has no attribute 'collate_fn'

body:
I use the fine tuning Maskrcnn in Pytorch
and I encounter the problem is
"module 'utils' has no attribute 'collate_fn"

but import utils no error
please help me!.


title: How can i visualize all the predictions with tensorboard in Pytorch?

body:
I tried to visualize the actual images with the predictions to figure out how my algorithm is performing and which labels are being predicted incorrectly. But when I set the step, during visualization on the tensorboard, it does not show all the steps. So, it does not show all the training images and their labels. Instead, I can only see a few examples out of all the training images.
writer = SummaryWriter(log_dir='graphs')

def matplotlib_imshow(img):
    npimg = img.cpu().numpy()
    npimg = np.transpose(npimg, (1, 2, 0))
    plt.imshow((npimg * 255).astype(np.uint8))

def images_to_probs(net, images):
    output = net(images)
    _, preds_tensor = torch.max(output, 1)
    preds = np.squeeze(preds_tensor.cpu().numpy())
    return preds, [F.softmax(el, dim=0)[i].item() for i, el in zip(preds, output)]

def plot_classes_preds(net, images, labels):
    preds, probs = images_to_probs(net, images)
    fig = plt.figure(figsize=(6, 6))
    for idx in np.arange(4):
        ax = fig.add_subplot(1, 4, idx+1, xticks=[], yticks=[])

        matplotlib_imshow(images[idx])
        ax.set_title("{0}, {1:.1f}%\n(label: {2})".format(
            classes[preds[idx]],
            probs[idx] * 100.0,
            classes[labels[idx]]),
                    color=("green" if preds[idx]==labels[idx].item() else "red"))
    return fig

The following is my training loop where I used the global step as step.
for epoch in range(epochs):
    epoch_start_time = time.time()
    losses = []
    total_batch_images = 0
    batch_correct_pred = 0
    step = 0
    #save model
    # if batch_accuracy>best_acc:
    #     best_acc = batch_accuracy
    #     checkpoint = {'state_dict': model.state_dict(),'acc' : batch_accuracy, 'epoch' : epoch,  'optimizer': optimizer.state_dict()}
    #     save_checkpoint(checkpoint)

    model.train()
    for batch_idx, (images, labels) in enumerate(train_loader):
        # Get data to cuda if possible

        images = images.to(device=device)
        labels = labels.to(device=device)

        # forward
        scores = model(images)
        loss = criterion(scores, labels)

        losses.append(loss.item())

        # backward
        optimizer.zero_grad()
        loss.backward()

        # gradient descent or adam step
        optimizer.step()

        # visualizing Dataset images
        # img_grid = torchvision.utils.make_grid(images)
        # writer.add_image('Xray_images', img_grid, global_step = step)

        # calculation running accuracy
        model.eval()
        _, predictions = scores.max(1)
        num_correct = (predictions == labels).sum()
        batch_correct_pred += float(num_correct)
        total_batch_images += predictions.size(0)

        writer.add_figure('predictions vs. actuals',
                           plot_classes_preds(model, images, labels),
                           global_step=step)
        step += 1

I think that the problem is in the last line of writer.add_figure() where the global_step is defined. However, any help in this regard would be really appreciated.


title: PyTorch is tiling images when loaded with Dataloader

body:
I am trying to load an Images Dataset using the PyTorch dataloader, but the resulting transformations are tiled, and don't have the original images cropped to the center as I am expecting them.
transform = transforms.Compose([transforms.Resize(224),
                             transforms.CenterCrop(224),
                             transforms.ToTensor()])

dataset = datasets.ImageFolder('ml-models/downloads/', transform=transform)
dataloader = torch.utils.data.DataLoader(dataset, batch_size=32, shuffle=True)

images, labels = next(iter(dataloader))
import matplotlib.pyplot as plt
plt.imshow(images[6].reshape(224, 224, 3))

The resulting image is tiled, and not center cropped.[![as shown in the Jupyter snapshot here][1]][1]
Is there something wrong in the provided transformation? (Image shown below on link: )
[1]: https://i.stack.imgur.com/HtrIa.png


title: How do I link in all of PyTorch with a pybind11 .so?

body:
I have a pybind11 c++ project which uses the pytorch c++ api:
#include <pybind11/pybind11.h>
#include <pybind11/numpy.h>
#include <math.h>
#include <torch/torch.h>

...

void f()
{
...
   torch::Tensor dynamic_parameters = torch::full({1}, /*value=*/0.5, torch::dtype(torch::kFloat64).requires_grad(true));
   torch::optim::SGD optimizer({dynamic_parameters}, /*lr=*/0.01);
...
}

PYBIND11_MODULE(reson8, m)
{
    m.def("my_function", &my_function, "");
}

I use distutils to compile this into a .so that can be imported in Python:
from distutils.core import setup, Extension

def configuration(parent_package='', top_path=None):
      import numpy
      from numpy.distutils.misc_util import Configuration
      from numpy.distutils.misc_util import get_info

      #Necessary for the half-float d-type.
      info = get_info('npymath')

      config = Configuration('',
                             parent_package,
                             top_path)
      config.add_extension('reson8',
                           ['reson8.cpp'],
                           extra_info=info,
                           include_dirs=["/home/ian/anaconda3/lib/python3.7/site-packages/pybind11/include",
                                          "/home/ian/anaconda3/lib/python3.8/site-packages/pybind11/include",
                                          "/home/ian/dev/hedgey/Engine/lib/libtorch/include",
                                          "/home/ian/dev/hedgey/Engine/lib/libtorch/include/torch/csrc/api/include"])

      return config

if __name__ == "__main__":
      from numpy.distutils.core import setup
      setup(configuration=configuration)

It compiles with no error, but on running "import reson8" in python I get this error:
importerror: undefined symbol: _ZTVN5torch5optim9OptimizerE

I'm unsure if it's whether pytorch hasn't been linked into my so (although the .so is 10mb which is rather large if it doesn't include pytorch, but maybe all pybind11 .so files are large.)
How can I solve this issue?


While executing iNltk library, I am getting an error. I have latest versions of pytorch and torchvision.
'LSTM' object has no attribute '_flat_weights_names'

After re-searching on some blogs some people suggested to downgrade the version to 1.2 So i tried below installation from https://pytorch.org/get-started/previous-versions/
pip install torch==1.2.0+cu92 torchvision==0.4.0+cu92 -f https://download.pytorch.org/whl/torch_stable.html

However, getting errors
ERROR: Could not find a version that satisfies the requirement torch==1.2.0+cpu
ERROR: No matching distribution found for torch==1.2.0+cpu

Also, 1.3.1 version is missing.
Anybody has any idea about how to downgrade to 1.3.1 or 1.2.0?
Thanks in advance
PD


title: Fast write Pytorch CUDA tensors to memory mapped files on GPU

body:
I saw that is possible to use CUDA to write to memory mapped files (reference cuda - Zero-copy memory, memory-mapped file )

I am wonder if it is somehow possible in Pytorch to write a cuda mounted tensor directory to a mem mapped stored on GPU.

The purpose of this is to speed up writing tensors after each training step. Currently,

with torch.no_grad():
    numpyMemmap[arrayOfRandomIndexes] = u_embeddings.weight.data.detach().cpu().numpy()

takes 6 seconds. I think it’s because the numpy memory map is stored on CPU. I need something that would write in a fraction of a second since I will be storing the tensors after each training step, and there will be hundreds of thousands of training steps.


title: Balanced sampling between classes in PyTorch

body:
I'm splitting the data using PyTorch's data.random_split

train_idx, validate_idx, test_idx = data.random_split(X,
                         [num_training_sample, num_validation_sample, num_testing_sample])

Which returns the indexes for the random split. Then I'm doing the following

train = [X[train_idx.indices], y[train_idx.indices]]
validate = [X[validate_idx.indices], y[validate_idx.indices]]
test = [X[test_idx.indices], y[test_idx.indices]]

Though they split have 6:2:2 ratio in total but the sampling between classes is not same. When I printed the counts, it shows

{0: 4251, 1: 4168, 2: 4223, 3: 4192, 4: 4180, 5: 4133, 6: 4226, 7: 4249, 8: 4238, 9: 4140}

Now, I can iterate over the samples and make balanced sub-samples, but I was wondering if there was a way in PyTorch(or Numpy) to do this efficiently.


title: PyTorch path generation with RNN - confusion with input, output, hidden and batch sizes

body:
I'm new to pytorch, I followed a tutorial on sentence generation with RNN and I'm trying to modify it to generate sequences of positions, however I'm having trouble with defining the correct model parameters such as input_size, output_size, hidden_dim, batch_size.

Background:
I have 596 sequences of x,y positions, each looking like [[x1,y1],[x2,y2],...,[xn,yn]]. Each sequence represents the 2D path of a vehicle. I would like to to train a model that, given a starting point (or a partial sequence), could generate one of these sequences.

-I have padded/truncated the sequences so that they all have length 50, meaning each sequence is an array of shape [50,2]

-I then divided this data into input_seq and target_seq:

input_seq: tensor of torch.Size([596, 49, 2]). contains all the 596 sequences, each without its last position.

target_seq: tensor of torch.Size([596, 49, 2]). contains all the 596 sequences, each without its first position.

The model class:

class Model(nn.Module):
def __init__(self, input_size, output_size, hidden_dim, n_layers):
    super(Model, self).__init__()
    # Defining some parameters
    self.hidden_dim = hidden_dim
    self.n_layers = n_layers
    #Defining the layers
    # RNN Layer
    self.rnn = nn.RNN(input_size, hidden_dim, n_layers, batch_first=True)
    # Fully connected layer
    self.fc = nn.Linear(hidden_dim, output_size)

def forward(self, x):
    batch_size = x.size(0)
    # Initializing hidden state for first input using method defined below
    hidden = self.init_hidden(batch_size)
    # Passing in the input and hidden state into the model and obtaining outputs
    out, hidden = self.rnn(x, hidden)
    # Reshaping the outputs such that it can be fit into the fully connected layer
    out = out.contiguous().view(-1, self.hidden_dim)
    out = self.fc(out)
    return out, hidden

def init_hidden(self, batch_size):
    # This method generates the first hidden state of zeros which we'll use in the forward pass
    # We'll send the tensor holding the hidden state to the device we specified earlier as well
    hidden = torch.zeros(self.n_layers, batch_size, self.hidden_dim)
    return hidden

I instantiate the model with the following parameters:

input_size of 2 (an [x,y] position)

output_size of 2 (an [x,y] position)

hidden_dim of 2 (an [x,y] position) (or should this be 50 as in the length of a full sequence?)

model = Model(input_size=2, output_size=2, hidden_dim=2, n_layers=1)
n_epochs = 100
lr=0.01
# Define Loss, Optimizer
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=lr)

# Training Run
for epoch in range(1, n_epochs + 1):
    optimizer.zero_grad() # Clears existing gradients from previous epoch
    output, hidden = model(input_seq)
    loss = criterion(output, target_seq.view(-1).long())
    loss.backward() # Does backpropagation and calculates gradients
    optimizer.step() # Updates the weights accordingly
    if epoch%10 == 0:
        print('Epoch: {}/{}.............'.format(epoch, n_epochs), end=' ')
        print("Loss: {:.4f}".format(loss.item()))

When I run the training loop, it fails with this error:

ValueError                                Traceback (most recent call last)
<ipython-input-9-ad1575e0914b> in <module>
      3     optimizer.zero_grad() # Clears existing gradients from previous epoch
      4     output, hidden = model(input_seq)
----> 5     loss = criterion(output, target_seq.view(-1).long())
      6     loss.backward() # Does backpropagation and calculates gradients
      7     optimizer.step() # Updates the weights accordingly
...

ValueError: Expected input batch_size (29204) to match target batch_size (58408).

I tried modifying input_size, output_size, hidden_dim and batch_size and reshaping the tensors, but the more I try the more confused I get. Could someone point out what I am doing wrong?

Furthermore, since batch size is defined as x.size(0) in Model.forward(self,x), this means I only have a single batch of size 596 right? What would be the correct way to have multiple smaller batches?


title: Convolution - Deconvolution for even and odd size

body:
I have two different size tensors to put in the network.
C = nn.Conv1d(1, 1, kernel_size=1, stride=2)
TC = nn.ConvTranspose1d(1, 1, kernel_size=1, stride=2)

a = torch.rand(1, 1, 100)
b = torch.rand(1, 1, 101)

a_out, b_out = TC(C(a)), TC(C(b))

The results are
a_out = torch.size([1, 1, 99]) # What I want is [1, 1, 100]
b_out = torch.size([1, 1, 101])

Is there any method to handle this problem?
I need your help.
Thanks


title: Size mismatch in fully connected layers

body:
I build the following simply model in pytorch as a first run and I am gettign a size mismatch error that does not make sense as out_feat always equals in_feat for the subsequent layer...
class Network(nn.Module):

    def __init__(self):
        super(Network,self).__init__()
        #first linear block

        self.fc1=nn.Linear(32,1024)
        self.b1=nn.BatchNorm1d(1024)

        #Two Linear 1
        self.fc2=nn.Linear(1024,1024)
        self.b2=nn.BatchNorm1d(1024)
        self.fc3=nn.Linear(1024,1024)
        self.b3=nn.BatchNorm1d(1024)

        #Two Linear 2
        self.fc4=nn.Linear(1024,1024)
        self.b4=nn.BatchNorm1d(1024)
        self.fc5=nn.Linear(1024,1024)
        self.b5=nn.BatchNorm1d(1024)

        #Final Linear Layer
        self.fc6=nn.Linear(1024,48)

    def forward(self,x):
        x1=self.fc1(x)
        x1=self.b1(x1)
        x1=nn.ReLU(x1)

        x2=self.fc2(x1)
        x2=self.b2(x2)
        x2=nn.ReLU(x2)
        x2=self.fc3(x2)
        x2=self.b3(x2)
        x2=nn.ReLU(x2)

        x3=x1+x2

        x4=self.fc4(x3)
        x4=self.b4(x4)
        x4=nn.ReLU(x4)
        x4=self.fc5(x4)
        x4=self.b5(x4)
        x4=nn.ReLU(x4)

        x5=x3+x4

        x6=self.fc6(x5)

        return x6

model=Network()
zeros=np.zeros((1,32))
outputs=model(torch.FloatTensor(zeros))
RuntimeError: size mismatch, m1: [1 x 32], m2: [1024 x 32] at ..\aten\src\TH/generic/THTensorMath.cpp:41

I do not understand how I am getting this error when all the dimensions match does anyone see the issue?
=================================================================
Layer (type:depth-idx)                   Param #
=================================================================
├─Linear: 1-1                            33,792
├─BatchNorm1d: 1-2                       4,096
├─Linear: 1-3                            1,049,600
├─BatchNorm1d: 1-4                       4,096
├─Linear: 1-5                            1,049,600
├─BatchNorm1d: 1-6                       4,096
├─Linear: 1-7                            1,049,600
├─BatchNorm1d: 1-8                       4,096
├─Linear: 1-9                            1,049,600
├─BatchNorm1d: 1-10                      4,096
├─Linear: 1-11                           49,200
=================================================================
Total params: 4,301,872
Trainable params: 4,301,872
Non-trainable params: 0

here is model summary


title: How to create tensor directly on GPU, or on device of another tensor?

body:
I found this discussion about this, in which the code
if std.is_cuda:
    eps = torch.FloatTensor(std.size()).cuda().normal_()
else:
    eps = torch.FloatTensor(std.size()).normal_()

becomes the nice
eps = std.new().normal_()

but it is said there that

The new() method is deprecated.

How to create a new tensor directly on a specific device?
How to create a new tensor on the same device like another tensor without the ugly if?


title: How to use a Pytorch DataLoader for a dataset with multiple labels

body: I'm wondering how to create a DataLoader that supports multiple types of labels in Pytorch. How do I do this?


title: What is the difference between src and tgt in nn.Transformer for PyTorch?

body:
From the docs it says to create a transformer model like this:
transformer_model = nn.Transformer(nhead=16, num_encoder_layers=12)
src = torch.rand((10, 32, 512))
tgt = torch.rand((20, 32, 512)) # What is tgt??
out = transformer_model(src, tgt)

What is tgt mean't to be? Should tgt be the same as the src?


title: Size mismatch for DNN for the MNIST dataset in pytorch

body:
I have to find a way to create a neural network model and train it on the MNIST dataset. I need there to be 5 layers, with 100 neurons each. However, when I try to set this up I get an error that there is a size mismatch. Can you please help? I am hoping that I can train on the model below:

class Mnist_DNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.layer1 = nn.Linear(784, 100)
        self.layer2 = nn.Linear(100, 100)
        self.layer3 = nn.Linear(100, 100)
        self.layer4 = nn.Linear(100, 100)
        self.layer5 = nn.Linear(100, 10)

    def forward(self, xb):
        xb = xb.view(-1, 1, 28, 28)
        xb = F.relu(self.layer1(xb))
        xb = F.relu(self.layer2(xb))
        xb = F.relu(self.layer3(xb))
        xb = F.relu(self.layer4(xb))
        xb = F.relu(self.layer5(xb))
        return self.layer5(xb)


title: Some mistakes when I reload "torch.nn.functioanl"?

body:
I want to modify some functions in "torch.nn.functional", and then I use "importlib.reload" to reload the new "torch.nn.functional".But I encounter a Runtime error:function 'avg_pool2d' already has a docstring.

I am trying to do some work on excitationbp(https://github.com/greydanus/excitationbp), and there is a  operation named "eb.use_eb()"  to modify "torch.nn.functional". The functiona eb.use_eb() is for modifying the "torch.nn.linear","torch.nn.functional.conv1d" and so on.I try to modify the code including many eb.use_eb() in the utils.py.When I want to run the original "torch.nn.linear",I must run the eb.use_eb(False).But the module has loaded in the head of the file(import torch.nn.funtional). So I want to reload the module "torch.nn.funtional".
When I use the "importlib.reload", I encounter a error.

eb.use_eb(False,True)
importlib.reload(torch)
importlib.reload(torch.nn)
importlib.reload(torch.nn.functional)
print("gradient_evidence:"+str(gradient_evidence))
return torch.autograd.grad(contr_h_, target_h_, grad_outputs=gradient_evidence)[0]

RuntimeError         Traceback (most recent call last)
ipython-input-15-29d0fca29882 in module()
----> 1 prob_inputs_one = eb.excitation_backprop(model, inputs, prob_outputs_one, contrastive=2)
      2 #pdb.set_trace()
      3 prob_inputs_true = eb.excitation_backprop(model, inputs, prob_outputs_true, contrastive=2)

~/code/excitationbp/excitationbp-master/excitationbp/utils.py in excitation_backprop(model, inputs, prob_outputs, contrastive, target_layer)
     75         importlib.reload(torch)
     76         importlib.reload(torch.nn)
---> 77         importlib.reload(torch.nn.functional)
     78
     79         gradient_evidence = torch.autograd.grad(top_h_, contr_h_, grad_outputs=prob_outputs.clone())[0]

~/anaconda3/envs/pytorch0.3/lib/python3.6/importlib/__init__.py in reload(module)
    164         target = module
    165         spec = module.__spec__ = _bootstrap._find_spec(name, pkgpath, target)
--> 166         _bootstrap._exec(spec, module)
    167         # The module may have replaced itself in sys.modules!
    168         return sys.modules[name]

~/anaconda3/envs/pytorch0.3/lib/python3.6/importlib/_bootstrap.py in _exec(spec, module)

~/anaconda3/envs/pytorch0.3/lib/python3.6/importlib/_bootstrap_external.py in exec_module(self, module)

~/anaconda3/envs/pytorch0.3/lib/python3.6/importlib/_bootstrap.py in _call_with_frames_removed(f, *args, **kwds)

~/anaconda3/envs/pytorch0.3/lib/python3.6/site-packages/torch/nn/functional.py in <module>()
    286     count_include_pad: when True, will include the zero-padding in th
    287         averaging calculation. Default: ``True``
--> 288 """)
    289
    290 avg_pool3d = _add_docstr(torch._C._nn.avg_pool3d, r"""

RuntimeError: function 'avg_pool2d' already has a docstring


title: Backward algorithm Hidden Markov Model, 0th index (termination step) yields wrong result

body:
I'm implementing a backward HMM algorithm in PyTorch. I used this link as reference. This link contains the results of the numerical example used (I am attempting to implement that and compare my generated results to it). Page 3, section 2. Backward probability, there is a table containing the calculated results.

Here is my code:

# Initial Transition matrix as shown in page 2 of above link
A = np.array([[0.6, 0.4], [0.3, 0.7]])
A = torch.from_numpy(A)
# Initial State Probability (page 2)
pi = np.array([0.8, 0.2])
pi = torch.from_numpy(pi)
# Output probabilities (page 2)
emission_matrix = np.array([[0.3, 0.4, 0.3, 0.3], [0.4, 0.3, 0.3, 0.3]])
emission_matrix = torch.from_numpy(emission_matrix)
# Initialize empty 2x4 matrix (dimensions of emission matrix)
backward = torch.zeros(emission_matrix.shape, dtype=torch.float64)

# Backward algorithm
def _backward(emission_matrix):
    # Initialization: A(i, j) * B(T, i) * B(Ot+1, j) , where B(Ot+1, j)  = 1
    backward[:, -1] = torch.matmul(A, emission_matrix[:, -1])
    # I reversed the emission matrix so as to start from the last column
    rev_emission_mat = torch.flip(emission_matrix[:, :-1], [1])
    # I transposed the reversed emission matrix such that each iterable in the for
    # loop is the observation sequence probability
    T_rev_emission_mat = torch.transpose(rev_emission_mat, 1, 0)
    # This step is so that I assign a reverse index enumeration to each iterable in the
    # emission matrix starts from time T to 0, rather than the opposite
    zipped_cols = list(zip(range(len(T_rev_emission_mat)-1, -1, -1), T_rev_emission_mat))

    for i, obs_prob in zipped_cols:
        # Induction: Σ A(i, j) * B(j)(Ot+1) * β(t+1, j)
        if i != 0:
            backward[:, i] = torch.matmul(A * obs_prob, backward[:, i+1])
    # Termination: Σ π(i) * bi * β(1, i)
    backward[:, 0] = torch.matmul(pi * obs_prob, backward[:, 1])

# run backward algorithm
_backward(emission_matrix)
# check results, backward is an all zero matrix that was initialized above
print(backward)
>>> tensor([[0.0102, 0.0324, 0.0900, 0.3000],
           [0.0102, 0.0297, 0.0900, 0.3000]], dtype=torch.float64)

As you can see, the 0-th index does not match the result in page 3 of the previous link. What did I do wrong? If there is anything I can clarify, please let me know. Thanks in advance!


title: How to sample Logits and Probabilties from a transformer seq2seq model for reinforcement learning?

body:
skipping the formalities:

I am trying to apply reinforcement learning to a transformer based seq2seq model (for abstractive summarization purposes) in Pytorch.

My current setup looks something like this:

I am getting a greedy distribution (summary) from the model by inferring one token at a time in a loop

def get_greedy_distribution(model, batch):
    src, (shift_tgt, lbl_tgt), segs, clss, mask_src, mask_tgt, mask_cls = batch

    # the mock targets are just torch.zeros tensors to store inferred tokens
    mock_tgt = get_mock_tgt(shift_tgt)
    mock_return = get_mock_tgt(shift_tgt)

    max_length = shift_tgt.shape[1]

    with torch.no_grad():
        for i in range(0, max_length-1):
            prediction = model(src, mock_tgt, segs, clss, mask_src, mask_tgt, mask_cls)
            prediction = F.softmax(prediction, dim=2)
            val, ix = prediction.data.topk(1)
            mock_tgt[:, i+1] = ix.squeeze()[:, i].detach()
            mock_return[:, i] = ix.squeeze()[:, i].detach()

    return mock_return

I am getting a sample distribution, with probabilities, from the model in a similar way:

def get_distribution(model, batch):
    src, (shift_tgt, lbl_tgt), segs, clss, mask_src, mask_tgt, mask_cls = batch
    mock_tgt = get_mock_tgt(shift_tgt)
    mock_return = get_mock_tgt(shift_tgt)

    max_length = shift_tgt.shape[1]

    log_probs = []

    for i in range(0, max_length-1):
        prediction = model(src, mock_tgt, segs, clss, mask_src, mask_tgt, mask_cls)
        prediction = F.softmax(prediction, dim=2)

        multi_dist = Categorical(prediction[:, i])
        x_t = multi_dist.sample()
        log_prob = multi_dist.log_prob(x_t)

        mock_tgt[:, i+1] = x_t
        mock_return[:, i] = x_t

        log_probs.append(log_prob)

    return mock_return, log_probs

However, I am a bit unsure if I am inferring the sample distribution correctly. This would work well in an RNN context where I can sample logits and probabilities during the typical RNN loop, but it feels slightly wrong when using a Transformer.

How would you suggest to approach the Transformer for a typical baseline-sampled reinforcement learning setup (I am guessing it is a policy gradient)?
Pytorch code is preferred but if you have Tensorflow examples I am sure I can figure it out.


title: How to split a 3D grid of coordinate values into blocks in PyTorch?

body:
If I have a batch of uniform 3D grids of coordinate locations, with shape (for example), [1, 32, 32, 32, 3], what is the best way for me to split this up into multiple even chunks, so I could end up with something such as [1, 4096, 2, 2, 2, 3]? In other words, I’m splitting up that one big 32 x 32 x 32 cube where each point is an x, y, z coordinate location into 4096 smaller 2 x 2 x 2 cubes? Does a simple view operation make sense here, or would it throw off the coordinate values? I was looking into operations like torch.chunk, but they require a specific dimension to split along, which I’m not sure applies here.

My use case for this is that I have a smaller [1, 16, 16, 16, 3] cube, so I’m trying to match up points from this smaller shape into the corresponding cubes in the upsampled [1, 32, 32, 32, 3] shape (since a single coordinate point in the 16^3 shape corresponds to 8 points in the 32^3 shape).

For additional context, this is how I generate my 3D grid right now:

pxs = torch.linspace(-1, 1, 32)
pys = torch.linspace(-1, 1, 32)
pzs = torch.linspace(-1, 1, 32)

pxs = pxs.view(-1, 1, 1).expand(*shape).contiguous().view(size)
pys = pys.view(1, -1, 1).expand(*shape).contiguous().view(size)
pzs = pzs.view(1, 1, -1).expand(*shape).contiguous().view(size)
points = torch.stack([pxs, pys, pzs], dim=1)

grid_3d = torch.reshape(points, (32, 32, 32, 3))


title: Parallel analog to torch.nn.Sequential container

body: Just wondering, why I can't find subj in torch.nn? nn.Sequential is pretty convinient, it allows to define networks in one place, clear and visual, but restricted to very simple ones! With parallel analog (and litle help of "identity" nodes for residual connections) it forms a complete method to construct any feedforward net combinatorial way. Am I missing something?


title: Kornia rotation not quite rotating as expected

body:
I'm trying to use the kornia.geometry.transform.rotate function, in Python, to rotate a PyTorch tensor by arbitrary angles. However if I do a simple 90 degree rotation, the resulting tensor doesn't look like it's been fully rotated.

Here's some sample code:

import torch
from kornia.geometry.transform import rotate
import matplotlib.pyplot as plt

a = torch.ones((1,64,64))
a[0,:,2] += 1

angle = torch.tensor([90])

c = rotate(a,angle)

plt.figure()
plt.subplot(121)
plt.imshow(a[0].detach().numpy())
plt.subplot(122)
plt.imshow(c[0].detach().numpy())

And the results before and after the rotation:

Am I missing a subtlety due to the tensor being too coarse here, which causes interpolation issues or something that would be alleviated with a much finer grained tensor?

Many thanks in advance!

Note I am using:

python 3.6.10
kornia 0.2.0
pytorch 1.4.0


title: Is the forward definition of a model executed sequentially in PyTorch or in parallel?

body:
I wanted to know if instructions in forward definition of deep models class are executed sequentially? For example:

class Net(nn.Module):
    ...
    def forward(self,x):

        #### Group 1
        y = self.conv1(x)
        y = self.conv2(y)
        y = self.conv3(y)

        ### Group 2
        z = self.conv4(x)
        z = self.conv5(z)
        z = self.conv6(z)

        out = torch.cat((y,z),dim=1)
        return out

In this case Group1 and Group2 instructions can be parallelized. But will the forward definition understand this automatically or will they be executed sequentially? If no, then how to run them in parallel?

I am running PyTorch 1.3.1

Thankyou very much


title: Profiling of PyTorch Custom Dataset for Dataloader

body:
I implemented a custom PyTorch Dataset for my project's Dataloader to use. However, it is running slower than expected so profiling was chosen to troubleshoot the bottleneck.
Looked into vprof but I am unsure of the function to be profiled if I am interested in only profiling the Dataset implementation. Using PyTorch 1.5 with Python 3.7 on Ubuntu 18.04.
How can we perform performance profiling (CPU and memory) on just the custom Dataset implementation?
Thanks!


title: How to train Pytorch CNN with two or more inputs

body:
I have a big image, multiple events in the image can impact the classification. I am thinking to split big image into small chunks and get features from each chunk and concatenate outputs together for prediction.
My code is like:
train_load_1 = DataLoader(dataset=train_dataset_1, batch_size=100, shuffle=False)
train_load_2 = DataLoader(dataset=train_dataset_2, batch_size=100, shuffle=False)
train_load_3 = DataLoader(dataset=train_dataset_3, batch_size=100, shuffle=False)

test_load_1 = DataLoader(dataset=test_dataset_1, batch_size=100, shuffle=True)
test_load_2 = DataLoader(dataset=test_dataset_2, batch_size=100, shuffle=True)
test_load_3 = DataLoader(dataset=test_dataset_3, batch_size=100, shuffle=True)

class Net(nn.Module):
   def __init__(self):
      super(Net, self).__init__()
      self.conv = nn.Conv2d( ... )  # set up your layer here
      self.fc1 = nn.Linear( ... )  # set up first FC layer
      self.fc2 = nn.Linear( ... )  # set up the other FC layer

   def forward(self, x1, x2, x3):
      o1 = self.conv(x1)
      o2 = self.conv(x2)
      o3 = self.conv(x3)
      combined = torch.cat((o1.view(c.size(0), -1),
                            o2.view(c.size(0), -1),
                            o3.view(c.size(0), -1)), dim=1)
      out = self.fc1(combined)
      out = self.fc2(out)
      return F.softmax(x, dim=1)

model = Net().to(device)
optimizer = optim.SGD(model.parameters(), lr=0.01)

for epoch in epochs:
   model.train()

   for batch_idx, (inputs, labels) in enumerate(train_loader_1):
   **### I am stuck here, how to enumerate all three train_loader to pass input_1, input_2, input_3 into model and share the same label? Please note in train_loader I have set shuffle=False, this is to make sure train_loader_1, train_loader_2, train_loader_3 are getting the same label **

Thank you for your help!


title: Libtorch C++ - no matching member function for call to 'size' for InterpolateFuncOptions

body:
Using Libtorch 1.6.0 in C++, I get the following error:
error: no matching member function for call to 'size'

My line is the following:
image = F::interpolate(image, F::InterpolateFuncOptions().size({target_height, target_width}).mode(torch::kNearest));

But in the documentation it seems correct... Any idea?
Thanks in advance


title: What do you use to access CSV data on S3 and other object storage providers as a PyTorch Dataset?

body:
My dataset is stored as a collection of CSV files in an Amazon Web Services (AWS) Simple Storage Service (S3) bucket. I'd like to train a PyTorch model based on this data but the built-in Dataset classes do not provide native support for object storage services like S3 or Google Cloud Storage (GCS), Azure Blob storage, and such. I checked the PyTorch documentation here https://pytorch.org/docs/stable/data.html# about the available Dataset classes and it comes up short when it comes to public cloud object storage support.
It looks like I have to create my own custom Dataset according to the following instructions: https://pytorch.org/tutorials/beginner/data_loading_tutorial.html#dataset-class but the effort seems overwhelming: I need to figure out how to download data from the object storage to local node, parse the CSV files to read them into PyTorch tensors, and then deal with the possibility of running out of disk space since my dataset is 100s of GBs.
Since PyTorch models are trained using gradient descent and I only need to store just a small batch of data (less than 1GB) in memory at once, is there a custom dataset implementation that can help?


title: Pytorch Attention model not training

body:
I have been building a Deep Learning model based on Attention. First I built a seq2seq model for the same, which works good, gives good results. Later I applied Attention mechanism, and the loss for it is not decreasing. I have tried various solutions available online, but none helped. Could some one help me find where could the issue be ?
The model architecture:
import torch
from torch import nn
import torch.nn.functional as F
from torch.autograd import Variable

class AttentionDecoder(nn.Module):
    def __init__(self, nIn, hidden_size, output_size, dropout_p=0.1):
        super(AttentionDecoder, self).__init__()
        #layers and layers

    def forward(self, input, hidden, encoder_outputs):
        embedded = self.embedding(input)

        batch_size = encoder_outputs.shape[1]
        alpha = hidden + encoder_outputs
        alpha = alpha.reshape(-1, alpha.shape[-1])
        attn_weights = self.vat( torch.tanh(alpha))
        attn_weights = attn_weights.view(-1, 1, batch_size).permute((2,1,0))
        attn_weights = F.softmax(attn_weights, dim=2)
        attn_applied = torch.matmul(attn_weights, encoder_outputs.permute((1, 0, 2)))
        output = torch.cat((embedded, attn_applied.squeeze(1) ), 1)
        output = self.attn_combine(output).unsqueeze(0)
        output, hidden = self.gru(output, hidden)
        output = self.out(output)
        output = F.softmax(output, dim=1)
        return output, hidden, attn_weights

    def initHidden(self, batch_size):
        result = Variable(torch.zeros(1, batch_size, self.hidden_size))
        return result

class Decoder(nn.Module):

    def __init__(self, nIn, nHidden, nOut):
        super(Decoder, self).__init__()
        # LSTM layers defined

    def forward(self, input):
        recurrent, _ = self.rnn(input)
        output = self.embedding(t_rec)
        return output

class Encoder(nn.Module):
    def __init__(self, cnnOutSize, nc, nclass, nh):
        super(Encoder, self).__init__()
        # CNN and RNN layers defined
    def forward(self, input):
        conv = self.cnn(input)
        encoder_outputs = self.rnn(conv)
        return encoder_outputs

The train script:
from __future__ import print_function
from builtins import range

import json
import character_set
import sys
import hw_dataset
from hw_dataset import HwDataset
import attention_model_from_scratch_modded as model
import os
import torch
import numpy as np
from torch.utils.data import DataLoader
from torch.autograd import Variable
from warpctc_pytorch import CTCLoss
import error_rates
import string_utils

config_path = sys.argv[1]

with open(config_path) as f:
    config = json.load(f)

idx_to_char, char_to_idx = character_set.load_char_set(config['character_set_path'])

NUM_CHANNELS = 3
NUM_HIDDEN_LAYERS = 1024
NUM_OUTPUTS = len(idx_to_char)+1
BATCH_SIZE = 8
NUM_WORKERS = 1
SOS_LABEL = 1
EOS_LABEL = 2

device = ""

if torch.cuda.is_available():
    device = 'cuda'
    dtype = torch.cuda.FloatTensor
    print("Using GPU")
else:
    device = 'cpu'
    dtype = torch.FloatTensor
    print("No GPU detected")

train_dataset = HwDataset(config['training_set_path'], char_to_idx, img_height=config['network']['input_height'], root_path=config['image_root_directory'], augmentation=True)
train_dataloader = DataLoader(train_dataset, batch_size=BATCH_SIZE, shuffle=False, num_workers=NUM_WORKERS, collate_fn=hw_dataset.collate)

test_dataset = HwDataset(config['validation_set_path'], char_to_idx, img_height=config['network']['input_height'], root_path=config['image_root_directory'])
test_dataloader = DataLoader(test_dataset, batch_size=BATCH_SIZE, shuffle=False, num_workers=NUM_WORKERS, collate_fn=hw_dataset.collate)

print('Size of Train dataset: ', len(train_dataset))
print('Size of Test dataset: ', len(test_dataset))

criterion = CTCLoss()
encoder = model.Encoder(config['network']['cnn_out_size'], NUM_CHANNELS, NUM_OUTPUTS, NUM_HIDDEN_LAYERS)
decoder = model.AttentionDecoder(config['network']['cnn_out_size'], NUM_HIDDEN_LAYERS, NUM_OUTPUTS)
encoder.to(device)
decoder.to(device)
criterion.to(device)

encoder_optimizer = torch.optim.Adam(encoder.parameters(), lr=config['network']['learning_rate'])
decoder_optimizer = torch.optim.Adam(decoder.parameters(), lr=config['network']['learning_rate'])

def cer_loss_func(x, out):
    #calculates Character Error rate

def validation(encoder, decoder, criterion):
    for e, d in zip(encoder.parameters(), decoder.parameters()):
        e.requires_grad = False
        d.requires_grad = False
    encoder.eval()
    decoder.eval()
    for x in test_dataloader:
        #processing
        encoder_output = encoder(line_imgs)
        decoder_input = text[0].cuda()
        decoder_hidden = decoder.initHidden(BATCH_SIZE).cuda()
        decoded_labels = []
        for di in range(1, text.shape[0]):
            decoder_output, decoder_hidden, decoder_attention = decoder(decoder_input, decoder_hidden, encoder_output)
            preds_size = Variable(torch.IntTensor([decoder_output.size(0)] * decoder_output.size(1)))
            decoder_input = text[di].cuda()
        sum_loss, steps = cer_loss_func(x, decoded_labels)
    return sum_loss, steps

def train(encoder, decoder, criterion, encoder_optimizer, decoder_optimizer):
    for e, d in zip(encoder.parameters(), decoder.parameters()):
        e.requires_grad = True
        d.requires_grad = True
    encoder.train()
    decoder.train()
    for x in train_dataloader:
        encoder_optimizer.zero_grad()
        decoder_optimizer.zero_grad()
        #processing
        encoder_output = encoder(input)
        decoder_input = torch.tensor([1]*BATCH_SIZE).cuda()
        decoder_hidden = decoder.initHidden(BATCH_SIZE).cuda()
        loss = 0.0
        decoded_labels = []
        for di in range(1, text.shape[0]):
            decoder_output, decoder_hidden, decoder_attention = decoder(decoder_input, decoder_hidden, encoder_output)
            preds_size = Variable(torch.IntTensor([decoder_output.size(0)] * decoder_output.size(1)))
            loss += criterion(decoder_output, text[di], preds_size, label_lengths)
            decoder_input = text[di].cuda()
        loss.backward()
        encoder_optimizer.step()
        decoder_optimizer.step()

        sum_loss, steps = cer_loss_func(x, decoded_labels)
    return sum_loss, steps

def main():
    lowest_loss = float('inf')
    for epoch in range(1000):
        print('\nStarting for epoch:', (epoch+1))

        sum_loss, steps = train(encoder, decoder, criterion, encoder_optimizer, decoder_optimizer)
        print("Training CER", sum_loss / steps)
        sum_loss, steps = validation(encoder, decoder, criterion)
        print("Testing CER", sum_loss / steps)

        if lowest_loss > sum_loss/steps:
            lowest_loss = sum_loss/steps
            print("Saving Best")
            dirname = os.path.dirname(config['model_save_path'])
            if len(dirname) > 0 and not os.path.exists(dirname):
                os.makedirs(dirname)

            torch.save(encoder.state_dict(), 'encoder.pt')
            torch.save(decoder.state_dict(), 'decoder.pt')

if __name__ == "__main__":
    main()

I am using the CTCLoss() function, and also CER rate, as I am generating string output.
Can someone help me figure out why could be the errors or issues due to which the loss isn't decreasing ?


title: Filtering rows in pytorch tensor

body:
If i have tensor which holds information on detected images in following shape:
        [[595.00000, 179.62500, 628.00000, 283.00000,   0.89062,   0.00000],
        [142.87500, 167.62500, 201.62500, 324.00000,   0.88086,   0.00000],
        [311.75000, 170.50000, 368.75000, 320.50000,   0.87549,   0.00000],
        [555.50000, 173.75000, 593.50000, 280.50000,   0.85791,   0.00000],
        [398.50000, 179.00000, 425.50000, 265.00000,   0.84180,   0.00000],
        [445.75000, 177.75000, 479.25000, 270.75000,   0.82129,   0.00000]]

where each row represents image with following parameters :
[   top,    bottom, left,   right, confidence, class    ]

What is most effective way of dropping images (rows) that are in size smaller than some user defined input for height which is top-bottom?
Naturally I would iterate over rows and than drop each row where top-bottom < someValue with some list comprehension but I suspect there could be better way of doing this.


title: Computing Cosine Distance with Differently shaped tensors

body:
I have the following tensor  representing a word vector
A = (2, 500)

Where the first dimension is the BATCH dimension (i.e. A contains two word vectors each with 500 elements)
I also have the following tensor
B = (10, 500)

I want to compute the cosine distance between A and B such that I get
C = (2, 10, 1)

i.e for each row in A compute the cosine distance with each row in B
I looked at using torch.nn.functional.F.cosine_similarity however this doesn't work as  the dimensions must be the same.
Whats the best efficient way of achieving this in pytorch?


title: One hot encoding in pytorch

body:
I am really new to coding, right now I am trying to turn my label to one hot encoding. I have already done transferring the np.array to tensor as shown below

tensor([4., 4., 4., 4., 4., 4., 4., 4., 4., 4., 4., 4., 4., 4., 4., 1., 1., 1.,
        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 3., 3., 3., 3., 3., 3.,
        3., 3., 3., 3., 3., 3., 3., 3., 3., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2.,
        2., 2., 2.], dtype=torch.float64)

and I am using code to do one hot encoding

aaa = F.one_hot(torch_qyh, num_classes=5)

However, there is an error showing "RuntimeError: one_hot is only applicable to index tensor." Any help would appreciated.


title: How to split a big HDF5 file into multiple small HDF5 dataset

body:
I have a big HDF5 file with the images and its corresponding ground truth density map.
I want to put them into the network CRSNet and it requires the images in separate files.
How can I achieve that? Thank you very much.
-- Basic info I have a HDF5 file with two keys "images" and "density_maps". Their shapes are (300, 380, 676, 1).
300 stands for the number of images, 380 and 676 refer to the height and width respectively.
-- What I need to put into the CRSNet network  are the images (jpg) with their corresponding HDF5 files.  The shape of them would be (572, 945).
Thanks a lot for any comment and discussion!


title: PyTorch warning about using a non-full backward hook when the forward contains multiple autograd Nodes

body:
I am relatively new to pytorch and building neural networks.
After a recent upgrade, when running my pytorch loop, I now get the warning "using a non-full backward hook when the forward contains multiple autograd Nodes".
The training still runs and completes, but I am unsure where I am supposed to place the register_full_backward_hook function.
I have tried adding it to each of the layers in my neural network but this gives further errors about using different hooks.
Can anyone please advise?


title: datalloader function ,and the iter function is cleared, but the next() function causes errors

body:
The datalloader function is used in pytorch, and the iter function is cleared, but the next() function causes errors. If you know the solution or the cause, please help me.

I've changed the batch size.

import torch
import torchvision
import torchvision.transforms as transforms
from torchvision.datasets import ImageFolder

transform = transforms.Compose([transforms.Resize(224,224),
                                transforms.ToTensor(),
                                transforms.Normalize((0.5,0.5,0.5),(0.5,0.5,0.5))])

train = ImageFolder('/home/minseok/study/dog_cat/train',transform)
valid = ImageFolder('/home/minseok/study/dog_cat/valid',transform)

train_loader = torch.utils.data.DataLoader(train,batch_size=256,num_workers=8,shuffle=True)
valid_loader = torch.utils.data.DataLoader(valid,batch_size=2,num_workers=1,shuffle=True)

dataiter = iter(valid_loader)  #It's safe up to here
images, labels = dataiter.next() # error

I am sorry for my poor English explanation because it is non-English speaking.


title: Pytorch : how to run code on several machines in cluster

body:
I am using a cluster to train a recurrent neural network developed using PyTorch. PyTorch automatically threads, which allows to use all the cores of a machine in parallel without having to explicitly program for it. This is great !

Now when I try to use several nodes at the same time using a script like this one :

#$ -S /bin/bash
#$ -N comparison_of_architecture
#$ -pe mvapich2-rostam 32
#4 -tc 4
#$ -o /scratch04.local/cnelias/Deep-Jazz/logs/out_comparison_training.txt
#$ -e /scratch04.local/cnelias/Deep-Jazz/logs/err_comparison_training.txt
#$ -t 1
#$ -cwd

I see that 4 nodes are being used but only one is actually doing work, so "only" 32 cores are in use.

I have no knowledge of parallel programming and I don't understand a thing in the tutorial provided on PyTorch's website, I am afraid this is completely out of my scope.

Are you aware of a simple way to let a PyTorch program run on several machines without having to explicitly program the exchanges of the messages and computation between these machines ?

PS : I unfortunately don't have a GPU and the cluster I am using also doesn't, otherwise I would have tried it.


title: Why do we need the second parameter for add_graph() when using tensorboard?

body: As we know, there are two parameters for add_graph(), a first parameter is an object of the model that we want to visualize, and the second parameter is just an input for the model. The input can be quite casual, i.e., we can simply randomly generate an input with the shape we want and feed it to add_graph(). So what’s the point to have the input as a parameter? It seems to me that this parameter only provides the shape information of an input tensor, but we can directly get this information from the model definition, right? And no matter what input we feed to add_graph(), the computation graph printed by tensorboard is always the same. I am really confused about this.


I hope everything is alright with you all. I am new with PyTorch and I came across an online code where it had the following code:

cuda = True if torch.cuda.is_available() else False
num_class = opt.num_class
num_repeats = opt.num_repeats
total=0
correct=0
acc=[]
acc1=[]
model = SHMnet(num_class)
# model = AlexNet1D(num_class)
learning_rate = opt.lr
print(model)
loss = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate,betas=(opt.b1,opt.b2))
if cuda:
    model=model.cuda()
    loss=loss.cuda()
FloatTensor = torch.cuda.FloatTensor if cuda else torch.FloatTensor
LongTensor = torch.cuda.LongTensor if cuda else torch.LongTensor

traindataset = []
for label in range(num_class):
    epdataset = epDataset(label,num_repeats)
    traindataset = ConcatDataset([traindataset,epdataset])

testdataset =[]
for label in range(num_class):
    eptdataset = eptDataset(label,6)
    testdataset = ConcatDataset([testdataset,eptdataset])
dataloader = DataLoader(traindataset, batch_size=opt.batch_size)
viddataloader = DataLoader(testdataset,batch_size = opt.batch_size)
loss_r=[]
start = time.time()

for epoch in range(opt.n_epochs):
    print('Epoch: %d' % epoch)
    total = 0
    correct = 0
    total1 = 0
    correct1 = 0
    for i, (sdata, label) in enumerate(dataloader):
        batch_size = sdata.shape[0]
        sdata = Variable(sdata.type(FloatTensor))
       noise=Variable(torch.from_numpy(np.random.normal(1, 0.2, sdata.shape))).type(FloatTensor)
        x = torch.mul(sdata,noise).view(sdata.size(0), 1, sdata.size(1))
        print(x)
        print(x.size())
        label = Variable (label.type(LongTensor))
        print(label)
        optimizer.zero_grad()
        output = model(x)
        print(output)
        print(output.size())
        loss1 = loss(input=output, target=label)
        loss1.backward()
        optimizer.step()
        _, predicted = torch.max(model(x).data, 1)
        temp = (predicted == label).sum()
        correct += temp
        total += batch_size
        print(i, loss1.item())
        loss_r.append(loss1.item())
    print('Accuracy of training data is: %d %%' % (100 * correct / total))
    acc.append(100 * correct / total)

I managed to find a dataset, but, I got the following error when I run the code:

RuntimeError: “dimension out of range (expected to be in range of [-1, 0], but got 1)”

So, I do not understand this error. What could be the reason and how to solve it?

The batch size is 60 and the number of classes is 6

Edit 1

The output of the output and label variables are:

Labels:  tensor([0, 0, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5])
Labels Size/Shape:  torch.Size([12])
torch.Size([12, 512, 2])
Output:  tensor([0.0288, 0.0125, 0.0104, 0.0222, 0.0106, 0.0121],  grad_fn=<AddBackward0>)
Output Size/Shape:  torch.Size([6])

Edit 2

I managed to change the batch size to 6, however, the output of the output and label variables are:

Input Size/Shape:  torch.Size([6, 1, 2192])
Labels:  tensor([0, 1, 2, 3, 4, 5])
Labels Size/Shape:  torch.Size([6])
torch.Size([6, 512, 2])
Output:  tensor([ 0.0228, -0.0088,  0.0047,  0.0224,  0.0146,  0.0357], grad_fn=<AddBackward0>)
Output Size/Shape:  torch.Size([6])

But, I get the same error and the batch size is 6 and the number of classes is 6
Thank you


I am writing custom dataset but the dataloader does not return correct target tensors.

where the label is 010 I am getting 10.

I am converting the labels into int before converting to tensor.

  y_label = torch.tensor(int(self.annotations.iloc[index, 1]))

iterating through dataloader:

Batch idx 0, data shape torch.Size([1, 1, 56, 128, 128]), target shape torch.Size([1])
tensor([1])
Batch idx 1, data shape torch.Size([1, 1, 56, 128, 128]), target shape torch.Size([1])
tensor([100])
Batch idx 2, data shape torch.Size([1, 1, 56, 128, 128]), target shape torch.Size([1])
tensor([10])
Batch idx 3, data shape torch.Size([1, 1, 56, 128, 128]), target shape torch.Size([1])
tensor([1])

csv file looks like the following:

1 p0.npy, 100
2 pl.npy, 001
3 p2.npy, 001
4 p3.npy, 001
5 p4.npy, 100
6 p5.npy, 010
7 p6.npy, 100
8 p7.npy, 100
9 p8.npy, 100
10 p9.npy, 010
11 plO.npy, 010
12 pll.npy, 010
13 p12.npy, 010
14 p13.npy, 100

code:

class patientdataset(Dataset):

    def __init__(self, csv_file, root_dir, transform=None):

            self.annotations = pd.read_csv(csv_file)
            self.root_dir = root_dir
            self.transform = transform

    def __len__(self):
        return len(self.annotations)

    def __getitem__(self, index):

        img_path = os.path.join(self.root_dir, self.annotations.iloc[index,0])
        # np_load_old = np.load
        # np.load = lambda *a, **k: np_load_old(*a, allow_pickle=True, **k)

        image= np.array(np.load(img_path))

        # y_label = torch.tensor(np.asarray(self.annotations.iloc[index,1]))

        y_label = torch.tensor(int(self.annotations.iloc[index, 1]))

        if self.transform:
            imagearrays = self.transform(image)
            image = imagearrays[None, :, :, :]
            imaget = np.transpose(image, (0, 2, 1, 3))
            image = imaget

        return (image, y_label)


title: How to shuffle labels on ImageFolder (Pytorch)

body:
I have to randomly shuffle a fraction of the labels of a dataset. I've always done it doing the following (Example on CIFAR10):

train_dataset = dsets.CIFAR10(root='./data',
                        train=True,
                        transform=transform_train,
                        download=True)

test_dataset = dsets.CIFAR10(root='./data',
                       train=False,
                       transform=transform_test)

for i in range(len(train_dataset.targets)):
    if (i % 3 == 0):
        train_dataset.targets[i] = random.randint(0, 9)

Particularly, the code above shuffles 33% of the labels. I want to do the same using ImageFolder, prebuilt on torchvision, but somehow it fails to shuffle the labels without giving any error.

In fact, I have checked the training set, and what I noticed was that the label train_dataset[0][1] was unchanged. Furthermore, if I manually try to change it by doing

train_dataset[0][1] = random.randint(0,9)

It gives an error (which makes sense, it is a Tuple)

How can I shuffle a percentage of the labels? Thank you!


title: Moving Tensors to the GPU in Pytorch (when using Image Folder)

body:
I am trying to move my tensors to the GPU after loading them in by using ImageFolder. Below is the relevant code:
    train_transform = transforms.Compose([
        transforms.RandomRotation(10),      # rotate +/- 10 degrees
        transforms.RandomHorizontalFlip(),  # reverse 50% of images
        transforms.Resize(224),             # resize shortest side to 224 pixels
        transforms.CenterCrop(224),         # crop longest side to 224 pixels at center
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406],
                             [0.229, 0.224, 0.225])
    ])

    train_data = datasets.ImageFolder(os.path.join(root, 'train'), transform=train_transform)

This creates tuples of tensors, but when I then try to iterate through the train_data and move the tensors to the GPU with .cuda(), I get an error because tuples are immutable.
Any ideas to help out? Thank you!!


title: How can i know the architecture of pre-trained model in Pytorch?

body: I have downloaded this pre-trained model "model_ir_se50.pth" for face recognition. It is giving very good results. How can i know its architecture?


title: Index out of range in self when I predict all tokens using BERT

body:
When I try to predict all tokens in my text using BERT with this function:

        def load_predict_BERT(self, masked_text):
            """
            Look for the [MASK] tokens and then attempts to predict the original value of the masked words

            :param masked_text: str
                Text containing [MASK] tokens for each word to predict
            :return: predictions:
            :return: MASKIDS: list
            """
            # Load, train and predict using pre-trained model
            tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
            tokenized_text = tokenizer.tokenize(masked_text)
            indexed_tokens = tokenizer.convert_tokens_to_ids(tokenized_text)
            MASKIDS = [i for i, e in enumerate(tokenized_text) if e == '[MASK]']
            # Create the segments tensors
            segs = [i for i, e in enumerate(tokenized_text) if e == "."]
            segments_ids = []
            prev = -1
            for k, s in enumerate(segs):
                segments_ids = segments_ids + [k] * (s - prev)
                prev = s
            segments_ids = segments_ids + [len(segs)] * (len(tokenized_text) - len(segments_ids))
            segments_tensors = torch.tensor([segments_ids])
            # Prepare Torch inputs
            tokens_tensor = torch.tensor([indexed_tokens])
            # Load pre-trained model
            model = BertForMaskedLM.from_pretrained('bert-base-uncased')
            model.resize_token_embeddings(len(tokenizer))
            # Predict all tokens
            with torch.no_grad():
                predictions = model(tokens_tensor, segments_tensors)
            return predictions, MASKIDS, tokenizer

I get this error and I am not able to fix it. I am a beginner with this kind of method but I hope someone could help me.
Traceback (most recent call last):
  File "/Users/AppleDreeaMz/opt/anaconda3/envs/OCRNLP_env/lib/python3.8/site-packages/IPython/core/interactiveshell.py", line 3417, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-9-c1e97a1e3497>", line 3, in <module>
    text = doc.extract_text(lang='la_Lat')
  File "<ipython-input-2-669f80886eff>", line 79, in extract_text
    prediction, MASKIDS, tokenizer = self.load_predict_BERT(masked_text=masked_text)
  File "<ipython-input-2-669f80886eff>", line 170, in load_predict_BERT
    predictions = model(tokens_tensor, segments_tensors)
  File "/Users/AppleDreeaMz/opt/anaconda3/envs/OCRNLP_env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/Users/AppleDreeaMz/opt/anaconda3/envs/OCRNLP_env/lib/python3.8/site-packages/pytorch_pretrained_bert/modeling.py", line 861, in forward
    sequence_output, _ = self.bert(input_ids, token_type_ids, attention_mask,
  File "/Users/AppleDreeaMz/opt/anaconda3/envs/OCRNLP_env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/Users/AppleDreeaMz/opt/anaconda3/envs/OCRNLP_env/lib/python3.8/site-packages/pytorch_pretrained_bert/modeling.py", line 730, in forward
    embedding_output = self.embeddings(input_ids, token_type_ids)
  File "/Users/AppleDreeaMz/opt/anaconda3/envs/OCRNLP_env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/Users/AppleDreeaMz/opt/anaconda3/envs/OCRNLP_env/lib/python3.8/site-packages/pytorch_pretrained_bert/modeling.py", line 269, in forward
    token_type_embeddings = self.token_type_embeddings(token_type_ids)
  File "/Users/AppleDreeaMz/opt/anaconda3/envs/OCRNLP_env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/Users/AppleDreeaMz/opt/anaconda3/envs/OCRNLP_env/lib/python3.8/site-packages/torch/nn/modules/sparse.py", line 124, in forward
    return F.embedding(
  File "/Users/AppleDreeaMz/opt/anaconda3/envs/OCRNLP_env/lib/python3.8/site-packages/torch/nn/functional.py", line 1814, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
IndexError: index out of range in self

I know that there is a problem with the size of the token but I do not know why.


title: what's the difference between "self-attention mechanism" and "full-connection" layer?

body: I am confused with these two structures. In theory, the output of them are all connected to their input. what magic make 'self-attention mechanism' is more powerful than the full-connection layer?


title: word synonym / antonym detection

body:
I need to create a classifier that takes 2 words and determines if they are synonyms or antonyms. I tried nltk's antsyn-net but it doesn't have enough data.
example:

capitalism <-[antonym]-> socialism
capitalism =[synonym]= free market
god <-[antonym]-> atheism
political correctness <-[antonym]-> free speach
advertising =[synonym]= marketing

I was thinking about taking a BERT model, because may be some of the relations would be embedded  in it and transfer-learn on a data-set that I found.


title: Error using torch.normal with invalid combination of arguments

body:
I got some error when using torch.normal
I don't know what is the problem.
torch.normal(mean=0,std=1,size=(128,100)).cuda()

 TypeError: normal() received an invalid combination of arguments - got (std=int, mean=int, size=tuple, ), but expected one of:
 * (Tensor mean, Tensor std, torch.Generator generator, Tensor out)
 * (Tensor mean, float std, torch.Generator generator, Tensor out)
 * (float mean, Tensor std, torch.Generator generator, Tensor out)

What is the problem?


title: How to apply LayerNorm (pytorch) to both time and feature dimension when the length of time varies in each batch?

body:
For example, if I use RNNs and have inputs of shape (B, T*, F). T* is the maximum length of the time dimension for each batch (to which the rest are padded), so for the first batch and the second batch, T1 and T2 may be different.
I think if only normalize over the feature (last) dimension we can set the LN layer to LN=nn.LayerNorm(F), but what if I want to normalize over both time and feature dimensions, how to initialize the LN layer in this case while T* is changing?
And, do I really need to normalize over the time dimension? Will it be useful or harmful?
Thanks in advance.


title: How to strip a pretrained network and add some layers to it using pytorch lightning?

body:
I am trying to use transfer learning for an image segmentation task, and my plan is to use the first few layers of a pretrained model (VGG16 for example) as an encoder and then will add my own decoder.
So, I can load the model and see the structure by printing it:
model = torch.hub.load('pytorch/vision:v0.6.0', 'resnet18', pretrained=True)
print(model)

I get like this:
ResNet(
  (conv1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
  (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (relu): ReLU(inplace=True)
  (maxpool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)
  (layer1): Sequential(
    (0): BasicBlock(
      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
    (1): BasicBlock(
      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
  )
  .....
  .....
  .....

I can also access the specific layers with model.layer3 for example. Now, I am struggling with certain things.

How to cut the model and take every module from the beginning to the end of any layer (model.layer3 for example)?
How to freeze only this stripped part, and keep the newly added modules available for training?


title: Rearrange torch 2D tensors ("Tiles") to be in a particular order

body:
I have something that looks like this...
import torch
X = torch.cat([torch.ones((2,4)).unsqueeze(0), torch.ones((2,4)).unsqueeze(0)*2, torch.ones((2,4)).unsqueeze(0)*3, torch.ones((2,4)).unsqueeze(0)*4]).unsqueeze(0)

which is
tensor([[[[1., 1., 1., 1.],
          [1., 1., 1., 1.]],

         [[2., 2., 2., 2.],
          [2., 2., 2., 2.]],

         [[3., 3., 3., 3.],
          [3., 3., 3., 3.]],

         [[4., 4., 4., 4.],
          [4., 4., 4., 4.]]]])

And I want to rearrange them to look like this
tensor([[[1., 1., 1., 1., 2., 2., 2., 2.],
         [1., 1., 1., 1., 2., 2., 2., 2.],
         [3., 3., 3., 3., 4., 4., 4., 4.],
         [3., 3., 3., 3., 4., 4., 4., 4.]]])

However the solution I was hoping would work obviously doesn't and while I understand why it doesn't work I'll share it.
b, t, x, y = X.shape
num_x_splits = num_y_splits = 2
X = X.reshape(-1,x*num_x_splits,y*num_y_splits)

tensor([[[1., 1., 1., 1., 1., 1., 1., 1.],
         [2., 2., 2., 2., 2., 2., 2., 2.],
         [3., 3., 3., 3., 3., 3., 3., 3.],
         [4., 4., 4., 4., 4., 4., 4., 4.]]])

Is there a computationally efficient way I can accomplish what I want?

I did some more research and looks like the ideal solution might actually be a combinatoin of the accepted answer and to use the einops package which provides a high-level, easy-to-read, efficient solution


title: Neural Network pruning mechanism

body:
I am working on SqueezeNet pruning . I have some questions regarding the pruning code which is based on the paper : PRUNING CONVOLUTIONAL NEURAL NETWORKS FOR RESOURCE EFFICIENT INFERENCE

def compute_rank(self, grad):
    activation_index = len(self.activations) - self.grad_index - 1
    activation = self.activations[activation_index]
    values = \
        torch.sum((activation * grad), dim = 0, keepdim=True).\
            sum(dim=2, keepdim=True).sum(dim=3, keepdim=True)[0, :, 0, 0].data

    values = \
        values / (activation.size(0) * activation.size(2) * activation.size(3))

    if activation_index not in self.filter_ranks:
        self.filter_ranks[activation_index] = \
            torch.FloatTensor(activation.size(1)).zero_().cuda()

    self.filter_ranks[activation_index] += values
    self.grad_index += 1

1) Why 'values' uses only in_height (2) and in_width (3) of the activation ? What about in_channels (1) ?

2) Why does filter_ranks[activation_index] depend on in_channels (1) only ?

3) Why activation multiplied with gradient ? and why sum them up ?


title: How to convert trained PyTorch model to Keras model?

body:
I am aware of ONNX package but I am not sure how to use it here. The tutorial on their Github page helps convert to Tensorflow but I want to convert it to Keras!
Any help would be appreciated.


title: Why can't I get this Runge-Kutta solver to converge as the time step decreases?

body:
For reasons, I need to implement the Runge-Kutta4 method in PyTorch (so no, I'm not going to use scipy.odeint). I tried and I get weird results on the simplest test case, solving x'=x with x(0)=1 (analytical solution: x=exp(t)). Basically, as I reduce the time step, I cannot get the numerical error to go down. I'm able to do it with a simpler Euler method, but not with the Runge-Kutta 4 method, which makes me suspect some floating point issue here (maybe I'm missing some hidden conversion from double precision to single)?

import torch
import numpy as np
import matplotlib.pyplot as plt

def Euler(f, IC, time_grid):
    y0 = torch.tensor([IC])
    time_grid = time_grid.to(y0[0])
    values = y0

    for i in range(0, time_grid.shape[0] - 1):
        t_i = time_grid[i]
        t_next = time_grid[i+1]
        y_i = values[i]
        dt = t_next - t_i
        dy = f(t_i, y_i) * dt
        y_next = y_i + dy
        y_next = y_next.unsqueeze(0)
        values = torch.cat((values, y_next), dim=0)

    return values

def RungeKutta4(f, IC, time_grid):

    y0 = torch.tensor([IC])
    time_grid = time_grid.to(y0[0])
    values = y0

    for i in range(0, time_grid.shape[0] - 1):
        t_i = time_grid[i]
        t_next = time_grid[i+1]
        y_i = values[i]
        dt = t_next - t_i
        dtd2 = 0.5 * dt
        f1 = f(t_i, y_i)
        f2 = f(t_i + dtd2, y_i + dtd2 * f1)
        f3 = f(t_i + dtd2, y_i + dtd2 * f2)
        f4 = f(t_next, y_i + dt * f3)
        dy = 1/6 * dt * (f1 + 2 * (f2 + f3) +f4)
        y_next = y_i + dy
        y_next = y_next.unsqueeze(0)
        values = torch.cat((values, y_next), dim=0)

    return values

# differential equation
def f(T, X):
    return X

# initial condition
IC = 1.

# integration interval
def integration_interval(steps, ND=1):
    return torch.linspace(0, ND, steps)

# analytical solution
def analytical_solution(t_range):
    return np.exp(t_range)

# test a numerical method
def test_method(method, t_range, analytical_solution):
    numerical_solution = method(f, IC, t_range)
    L_inf_err = torch.dist(numerical_solution, analytical_solution, float('inf'))
    return L_inf_err

if __name__ == '__main__':

    Euler_error = np.array([0.,0.,0.])
    RungeKutta4_error = np.array([0.,0.,0.])
    indices = np.arange(1, Euler_error.shape[0]+1)
    n_steps = np.power(10, indices)
    for i, n in np.ndenumerate(n_steps):
        t_range = integration_interval(steps=n)
        solution = analytical_solution(t_range)
        Euler_error[i] = test_method(Euler, t_range, solution).numpy()
        RungeKutta4_error[i] = test_method(RungeKutta4, t_range, solution).numpy()

    plots_path = "./plots"
    a = plt.figure()
    plt.xscale('log')
    plt.yscale('log')
    plt.plot(n_steps, Euler_error, label="Euler error", linestyle='-')
    plt.plot(n_steps, RungeKutta4_error, label="RungeKutta 4 error", linestyle='-.')
    plt.legend()
    plt.savefig(plots_path + "/errors.png")

The result:

As you can see, the Euler method converges (slowly, as expected of a first order method). However, the Runge-Kutta4 method does not converge as the time step gets smaller and smaller. The error goes down initially, and then up again. What's the issue here?


title: Memory leak issue using PyTorch IterableDataset with zarr

body:
I'm trying to build a pytorch project on an IterableDataset with zarr as storage backend.
class Data(IterableDataset):
    def __init__(self, path, start=None, end=None):
        super(Data, self).__init__()
        store = zarr.DirectoryStore(path)
        self.array = zarr.open(store, mode='r')

        if start is None:
            start = 0
        if end is None:
            end = self.array.shape[0]

        assert end > start

        self.start = start
        self.end = end

    def __iter__(self):
        return islice(self.array, self.start, self.end)

This works quite nicely with small test-datasets but once i move to my actual dataset (480 000 000 x 290) i'm running into a memory leak. I've tried logging out the python heap periodically as everything slows to a crawl, but i couldn't see anything increasing in size abnormally, so the lib i used (pympler) didn't actually catch the memory leak.
I'm kind of at my wits end, so if anybody has any idea how to further debug this, it would be greatly appreciated.
Cross-posted on PyTorch Forums.


title: Is there a way to use a 2d tensor as the input for a MLP in Pytorch?

body:
I’m brand new to Pytorch, and I'm trying to train a MLP in Pytorch to classify a variable from multiple sets of timeseries data. <
 Example[time, a, b, c]:
    {[0, 10, 15, 20] [1, 11, 17, 10] [2, 12, 12, 6]} -> 1
    {[0, 10, 12, 9] [1, 15, 13, 0] [2, 18, 11, 7]} -> 2

Right now, I'm reshaping each of these 2d tensors to be 1d (ie [0, 10, 15, 20, 1, 11, 17, 10, 2, 12, 12, 6]) and then placing them in a Data Loader, but I'm wondering if it's possible to keep them 2d, or if there's a different way to do it?
Also, could this be the reason for my awful accuracy(both training and testing)?


title: Inference error:Expected at most 2 argument(s) for operator 'forward', but received 3 argumen

body:
Here is my cnn network:
class CNN(nn.Module):
    def __init__(self):
        super(CNN, self).__init__()
        self.conv1 = nn.Sequential(  # input shape (1, 28, 28)
            nn.Conv2d(
                in_channels=1,
                out_channels=16,
                kernel_size=5,
                stride=1,
                padding=2,
            ),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2),
        )
        self.conv2 = nn.Sequential(
            nn.Conv2d(16, 32, 5, 1, 2),
            nn.ReLU(),
            nn.MaxPool2d(2),
        )
        self.out = nn.Linear(32 * 7 * 7, 10)

    def forward(self, x):
        x = self.conv1(x)
        x = self.conv2(x)
        x = x.view(x.size(0), -1)
        output = self.out(x)
        return output, x

I save it as:
1  sm = torch.jit.script(cnn)
2  sm.save("cnn.pt")

Then I load it in C++ and I am trying to read some data to make some inference:
vector<vector<double>>images;
read_Mnist_Images("/home/heye/python/mnist/MNIST/raw/t10k-images-idx3-ubyte", images);
// Create a vector of inputs.
std::vector<torch::jit::IValue> inputs;
inputs.push_back(torch::ones({10, 1, 28, 28}));

auto tensor = torch::empty(10 * 1 * 28 * 28);
float *data = tensor.data<float>();

for (int i = 0; i < 10; i++) {
  for (int j = 0; j < 1; j++) {
    for (int k = 0; k < 28; k++) {
      for (int l = 0; l < 28; l++) {
        *data++ = images[i][28 * k + l] / 255;
      }
    }
  }
}
inputs.emplace_back(tensor.resize_({10, 1, 28, 28}));
// Run inference
auto output = module.forward(inputs).toTensor();
cout << output.slice(/*dim=*/1, /*start=*/0, /*end=*/5) << '\n';
std::cout << "ok\n";

and I get the error message, when I run the code:
terminate called after throwing an instance of 'c10::Error'
  what():  Expected at most 2 argument(s) for operator 'forward', but received 3 argument(s). Declaration: forward(ClassType<CNN> self, Tensor x) -> ((Tensor, Tensor)) (checkAndNormalizeInputs at /pytorch/aten/src/ATen/core/function_schema_inl.h:245)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x33 (0x7fa4d08f2813 in /home/heye/torch1.3_cpu/torch/lib/libc10.so)
frame #1: <unknown function> + 0x323f830 (0x7fa4d3d56830 in /home/heye/torch1.3_cpu/torch/lib/libtorch.so)
frame #2: torch::jit::Function::operator()(std::vector<c10::IValue, std::allocator<c10::IValue> >, std::unordered_map<std::string, c10::IValue, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, c10::IValue> > > const&) + 0x36 (0x7fa4d3d54ec6 in /home/heye/torch1.3_cpu/torch/lib/libtorch.so)
frame #3: torch::jit::script::Method::operator()(std::vector<c10::IValue, std::allocator<c10::IValue> >, std::unordered_map<std::string, c10::IValue, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, c10::IValue> > > const&) + 0xc9 (0x7fa4d3d13529 in /home/heye/torch1.3_cpu/torch/lib/libtorch.so)
frame #4: torch::jit::script::Module::forward(std::vector<c10::IValue, std::allocator<c10::IValue> >) + 0xe2 (0x42fb76 in ./read_pt)
frame #5: main + 0x43e (0x4299ab in ./read_pt)
frame #6: __libc_start_main + 0xf5 (0x7fa4d00e8495 in /lib64/libc.so.6)
frame #7: ./read_pt() [0x428849]

How to solve this problem, most of the code following from https://pytorch.org/tutorials/advanced/cpp_export.html, any help will be appreciated.


title: Specifying number of cells in LSTM layer in PyTorch

body: I don't fully understand the LSTM layer in PyTorch. When I instantiate an LSTM layer how can I specify the number of LSTM cells inside the layer? My first thought was that it was the "num_layers" argument, if we assume that LSTM cells are connected vertically. But if that is the case how can we implement stacked LSTM with for example two layers with 8 cells each?


title: Creating a generator over a zarr array with start and end for pytorch dataloader

body:
I'm working on a pytorch project where my data is saved in zarr.
Random access on zarr is costly, but thanks to zarr using a blockwise cache, iteration is really quick. To harness this fact, I use an IterableDataset together with multiple workers:
class Data(IterableDataset):
    def __init__(self, path, start=None, end=None):
        super(Data, self).__init__()
        store = zarr.DirectoryStore(path)
        self.array = zarr.open(store, mode='r')

        if start is None:
            start = 0
        if end is None:
            end = self.array.shape[0]

        assert end > start

        self.start = start
        self.end = end

    def __iter__(self):
        return islice(self.array, self.start, self.end)

The issue is that self.array has on the order of 10e9 rows and for consecutive workers, as self.start and self.end naturally get bigger, creating the generators like itertools.islice(array, start, end) takes a significant time out of my training/validation processes, because islice still has to iterate over the unneeded elements until it gets to start. Once a generator is created per each worker, this works like a charm, but to get there takes too long.
Is there a better way to create such a generator? Or maybe there's a smarter way to use zarr in pytorch?


title: How to access module attributes such as the stride of convolutions from jit::script::module

body:
I am currently writing a C++ program which needs to do some analyses of the structure of a CNN model in torchScript format. I am using the C++ torch library the way it is shown on torch.org, loading in the model like so:
#include <torch/script.h>
#include <torch/torch.h>

#include <iostream>
#include <memory>

int main(int argc, const char* argv[]) {
    if (argc != 2) {
        std::cerr << "usage: example-app <path-to-exported-script-module>\n";
        return -1;
    }

    torch::jit::script::Module module;
    try {
        // Deserialize the ScriptModule from a file using torch::jit::load().
        module = torch::jit::load(argv[1]);
    }
    catch (const c10::Error& e) {
        std::cerr << "error loading the model\n";
        return -1;
    }

    return 0;
}

As far as I was able to figure out module consists of a collection of nested torch::jit::script::Module's the lowest of which represent the built in functions. Im accessing those lowest modules as follows:
void print_modules(const torch::jit::script::Module& imodule) {

    for (const auto& module : imodule.named_children()) {

        if(module.value.children().size() > 0){
            print_modules(module.value);

        }
        else{

            std::cout << module.name << "\n";

        }
    }
}

This function recursively goes trough the modules and prints the name of the lowest level ones, which correspond to the builtin functions of torch script.
My question now is, how do I access the details for those builtin functions like for example the stride length for convolutions?
I can't figure out for the life of me to get access to those basic attributes of modules.


title: Time complexity of torch.topk

body:
When m and n of n.topk(m) exceed 20 million and 200,000 respectively, the sorting becomes very slow(over 3 hours). I want to know the time complexity of torch.topk and improvement measures of the sorting.
topv, topi = outline.topk(beam_size)  # beam_size = 200,000, outline: 1 × 20,000,000


title: jsonpickle.decode() not executing in Flask

body:
I've got a Flask app run with Apache that uses some custom object that's serialized with jsonpickle. When deserializing with jsonpickle.decode(my_json_string), there's nothing happening, meaning the method does not return anything nor does it throw an error. It's as if it just takes infinitely (haven't observed it finishing).

I've tried with a toy object that's just a few KB big, and I monitor hardware usage, and it's not a memory nor CPU problem.

What's more puzzling, using the same code to load the object in the Python console rather than Flask (using the same virtualenv), everything works smoothly, and the object is decoded in a matter of milliseconds.

import jsonpickle

with open("serialized_object.json") as f:
  object_as_json = f.read()  # this works, print(object_as_json) as expected
my_object = jsonpickle.decode(object_as_json)

This is Python 3.6.8, Flask 0.12.2, jsonpickle 1.2, Apache 2.4.

Line 5 in the code snippet above never finishes execution when run in Flask app/Apache. In the normal Python console, all works fine.

I'm afraid I don't know what more info to provide, but any help or pointers are greatly appreciated!

EDIT: I should perhaps add that the serialized object contains a Pytorch Module, but as I said that's really quite small.

UPDATE: I tried using pickle instead and noticed exactly the same behaviour, strangely. I then removed some of the Pytorch components of the object and it all worked fine, so it seems like this is not a jsonpickle/pickle problem at all.


title: Translate torch code intended for GPU computation to run on CPU only

body:
I have a pytorch-based code that runs on GPU due to his multiple calls to CUDA, the code is intended for object detection and the model has already been trained to do this. So what I do is testing a model on a dataset. However, my GPU is not enough powerful to make the model run without Out Of Memory (OOM) errors.

My GPU is Nvidia P400 and I set up cuda 9.0 for it. I'm running on the 396.54 Nvidia driver version. I didn't try to upgrade it because it causes troubles to my settings. My operating system is Linux Ubuntu 18.04.

More specifically, my question is the following. In order to make my pytorch code run on CPU, can I just remove the occurrence of cuda in my code ?
For example replacing this following code :

torch.cuda.manual_seed(11)

By

torch.manual_seed(11)

EDIT :
When I set up pytorch on my conda environment, I installed a gpu compatible pytorch.
So now that I tried to remove all the occurrences of cuda, it doesn't work and I get the following error :

THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=74 error=38 : no CUDA-capable device is detected
Traceback (most recent call last):
  File "main.py", line 1111, in <module>
    train()
  File "main.py", line 1039, in train
    log_path=model_save_path)
  File "main.py", line 858, in train_networks
    train_losses, valid_losses, test_losses = pickle.load(lossfile, encoding='latin1')
  File ".../anaconda3/envs/pytorch-gpu/lib/python3.6/site-packages/torch/cuda/__init__.py", line 492, in _lazy_new
    _lazy_init()
  File ".../anaconda3/envs/pytorch-gpu/lib/python3.6/site-packages/torch/cuda/__init__.py", line 161, in _lazy_init
    torch._C._cuda_init()
RuntimeError: cuda runtime error (38) : no CUDA-capable device is detected at /pytorch/aten/src/THC/THCGeneral.cpp:7


title: Google Cloud Function Build timeout - all requirements have been loaded

body:
I have the following code on my cloud function -
import os
import numpy as np
import requests

import torch
from torch import nn
from torch.nn import functional as F
import math
from torch.nn import BCEWithLogitsLoss
from torch.utils.data import TensorDataset
from transformers import AdamW, XLNetTokenizer, XLNetModel, XLNetLMHeadModel, XLNetConfig
from keras.preprocessing.sequence import pad_sequences
import numpy as np
import pandas as pd

def polarization(request):

   MODEL_URL = 'https://polarization.s3-us-west-1.amazonaws.com/classifier_state_dict.pt'
   print(MODEL_URL)
   r = requests.get(MODEL_URL)
   print(r)

   #Cloud function vm is a read only s/m. The only writable place is the tmp folder
   file = open("/tmp/model.pth", "wb")
   file.write(r.content)
   file.close()

   print("Wrote to the tmp file")
   # State dict requires model object
   model = XLNetForPolarizationClassification(num_labels=1)
   model.load_state_dict(torch.load('/tmp/model.pth'))

   # Tokenize the embedded article
   embeddedArticle = request["embeddedArticle"]
   tokenizer = XLNetTokenizer.from_pretrained('xlnet-base-cased', do_lower_case=True)
   textIds = tokenize_inputs(embeddedArticle, tokenizer, num_embeddings=250)

   # Generate the attention masks and padding
   masks = create_attn_masks(textIds)
   article = pd.DataFrame()
   article["features"] = textIds.tolist()
   article["masks"] = masks

   # Call generate_predictions
   pred = generate_predictions(model, article, 1)

   return pred

## Extracting parameter and returning prediction
def generate_predictions(model, df, num_labels, device="cpu"):

   model.eval()

   X = df_subset["features"].values.tolist()
   masks = df_subset["masks"].values.tolist()

   X = torch.tensor(X)
   masks = torch.tensor(masks, dtype=torch.long)
   with torch.no_grad():
       # Run the model with the input_ids and attention_masks separately
       logits = model(input_ids=X, attention_mask=masks)
       # Get the logits for each class
       logits = logits.sigmoid().detach().cpu().numpy()

   return round(logits)

class XLNetForPolarizationClassification(torch.nn.Module):

   def __init__(self, num_labels=2):
       super(XLNetForPolarizationClassification, self).__init__()
       self.num_labels = num_labels
       self.xlnet = XLNetModel.from_pretrained('xlnet-base-cased')
       self.classifier = torch.nn.Linear(768, 1)
       torch.nn.init.xavier_normal_(self.classifier.weight)

   def forward(self, input_ids, token_type_ids=None, attention_mask=None, labels=None):

       last_hidden_state = self.xlnet(input_ids=input_ids,
                                       attention_mask=attention_mask,
                                       token_type_ids=token_type_ids)

       mean_last_hidden_state = self.pool_hidden_state(last_hidden_state)
       logits = self.classifier(mean_last_hidden_state)

       # If you know the labels, compute the loss otherwise
       if labels is not None:
           loss_fct = BCEWithLogitsLoss()
           loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1, self.num_labels))
           return loss
       else:
           return logits

   def pool_hidden_state(self, last_hidden_state):
       last_hidden_state = last_hidden_state[0]
       mean_last_hidden_state = torch.mean(last_hidden_state, 1)
       return mean_last_hidden_state

def create_attn_masks(input_ids):
 """
 This will set a 1 or 0 based on if it is a mask or an actual input it for the word
 """
 attention_masks = []

 for seq in input_ids:
   seq_mask = [float(i>0) for i in seq]
   attention_masks.append(seq_mask)
 return attention_masks

def tokenize_inputs(text, tokenizer, num_embeddings=250):

   # tokenize the text, then truncate sequence to the desired length minus 2 for
   # the 2 special characters
   tokenized_texts = list(map(lambda t: tokenizer.tokenize(t)[:num_embeddings-2], text))

   # convert tokenized text into numeric ids for the appropriate LM
   input_ids = [tokenizer.convert_tokens_to_ids(x) for x in tokenized_texts]

   # append special token "<s>" and </s> to end of sentence
   input_ids = [tokenizer.build_inputs_with_special_tokens(x) for x in input_ids]

   # pad sequences
   input_ids = pad_sequences(input_ids, maxlen=num_embeddings, dtype="long", truncating="post", padding="post")
   return input_ids

and requirements.txt
certifi==2020.6.20
chardet==3.0.4
click==7.1.2
cycler==0.10.0
filelock==3.0.12
future==0.18.2
h5py==2.10.0
idna==2.10
joblib==0.16.0
Keras==2.4.3
kiwisolver==1.2.0
matplotlib==3.3.0
numpy==1.19.1
packaging==20.4
Pillow==7.2.0
pyparsing==2.4.7
python-dateutil==2.8.1
PyYAML==5.3.1
regex==2020.7.14
requests==2.24.0
sacremoses==0.0.43
scipy==1.5.2
sentencepiece==0.1.91
six==1.15.0
tokenizers==0.8.1rc1
torch==1.6.0
tqdm==4.48.2
transformers==3.0.2
urllib3==1.25.10

However when I deploy, it gives me build timed out. The logs don't show any errors and show that each of the dependencies in the requirements.txt file have built.  The first print statement doesn't get logged either. I don't see what part of the model/ requirements is causing the time out issue.
Here's a screenshot of the logs - there are no errors all the logs are info until theres a Conext deadline exceeded statement. I can't share the actual logs without giving access to the function I believe.  I've set the timeout to 9 minutes (540 seconds)1


title: Using Autograd .backward() function to calculate an intermediate value in the forward pass of Pytorch model

body:
Hello I am new to Pytorch. I have a simple pytorch module where the output of the module is scalar loss function that depends on the derivative of some polynomial functions. Let's say the output of the forward pass is: input*derivative(x^2+y^2).
One way to implement this, is to explicitly write down the deriviates of the polynomials used and have that be part of the forward model. So output=inputs*(2x+2y). However, this is not robust as if I include more polynomials, I have to manually add more derivative functions which can be time consuming and prone to errors.
I want to initialize the polynomials, use Autograd to get their derivatives, plug that derivative into the output formula. let's say the polynomial function is called n. I do n.backward(retain_graph=True) inside the forward pass. However, it does not seem to work properly as I get very different answers (of the magnitude of the derivatives of loss function vs the model parameters) as when I use the analytic expression in the forward pass.
Note, that both the output of the f.backward and the analytic expression of the derivative match. So it is computing the derivative of the polynomials correctly, but it is having a hard time associating this with the final loss function. Meaning that the backward() call is also messing up the model parameters while it is trying to get the derivatives for the polynomial coefficients. I am sure this is because of my poor understanding of pytorch and adding the f.backward() call inside the forward pass is somehow messing up the loss.backward() call.
Here is a simplified example: The problem is that the value model.learn.grad is not the same when using the analytic method and the autograd .backward() method

class Model(nn.Module):

    def __init__(self, grin_type='radial',loss_type='analytic', device='cpu', dtype=torch.float32):
        super(Model, self).__init__()

        self.dtype=dtype
        self.device=device
        self.loss_type=loss_type
        self.grin_type=grin_type
        self.x=torch.tensor(2.,dtype = dtype, device=self.device) #mm

        self.learn=nn.Parameter(torch.tensor(5.,dtype = dtype, device=self.device))

    def forward(self,inputs,plotting=0):

        if self.loss_type=='analytic':

            outputs=inputs*self.learn*(2.*self.x)

        elif self.loss_type=='autograd':

            self.der=self.calc_derivative(self.x)
            outputs=inputs*self.der

        return outputs

    def poly_fun(self,x):

        return self.learn*torch.square(x)

    def calc_derivative(self,x):
        xn=x.clone().detach().requires_grad_(True)
        n=self.poly_fun(xn)

        dloss_dx=torch.autograd.grad(outputs=n, inputs=xn,create_graph=True)[0]*n/n

        return dloss_dx


title: do I have to add softmax in def forward when I use torch.nn.CrossEntropyLoss

body:
https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html
When I read the contents above, I understood that torch.nn.CrossEntropy already computes exp score of the last layer. So I thought the forward function doesn't have to include softmax. For example,
return self.fc(x) rather than return nn.softmax(self.fc(x)). However, I'm confused, for I've seen several implementations of ConvNet Classifiers that use both ways (they return with or without softmax while both use cross entropy loss).
Do this issues affect the performance of Classifier?? Which way is correct?