title: PyTorch gradient doesn't flow through a clone of a tensor

body:
I'm trying to have my model learn a certain function. I have parameters self.a, self.b, self.c that are trainable. I'm trying to force self.b to be in a certain range by using tanh. However, when I run the code it appears as the gradient is flowing through the original parameter (self.b), but not through the clone (self.b_opt)
import torch
import torch.nn as nn
import torch.optim as optim

class model(nn.Module):
        def __init__(self):
            super(model, self).__init__()
            self.a = torch.nn.Parameter(torch.rand(1, requires_grad=True))
            self.b = torch.nn.Parameter(torch.rand(1, requires_grad=True))
            self.c = torch.nn.Parameter(torch.rand(1, requires_grad=True))

            self.b_opt = torch.tanh(self.b.clone())

model_net = model()
#function to learn = 5 * (r > 2) * (3)
optimizer = optim.Adam(model_net.parameters(), lr = 0.1)

for epoch in range(10):
  for r in range(5):
    optimizer.zero_grad()
    loss = 5 * (r > 2) * (3) - model_net.a * torch.sigmoid((r - model_net.b_opt)) * (model_net.c)

    loss.backward(retain_graph=True)
    optimizer.step()
  #print(model_net.a)
  print(model_net.b)
  print(model_net.b_opt)
  #print(model_net.c)
  print()

>>> Parameter containing:
tensor([0.4298], requires_grad=True)
tensor([0.7229], grad_fn=<TanhBackward>)

Parameter containing:
tensor([-0.0277], requires_grad=True)
tensor([0.7229], grad_fn=<TanhBackward>)

Parameter containing:
tensor([-0.5007], requires_grad=True)
tensor([0.7229], grad_fn=<TanhBackward>)


title: importing likelihoods fails

body:
I am trying to get in touch with Gaussian Process Classification and try to reproduce the example from https://docs.gpytorch.ai/en/stable/examples/01_Exact_GPs/GP_Regression_on_Classification_Labels.html
Following their code, I want to import the Dirichlet Classification Likelihood by
from gpytorch.likelihoods import DirichletClassificationLikelihood

However, this seems not to work, and I get the Error:
ImportError: cannot import name 'DirichletClassificationLikelihood'
from 'gpytorch.likelihoods' (/Library/Frameworks/Python.framework/Versions/3.8/
lib/python3.8/site-packages/gpytorch/likelihoods/__init__.py)

also other likelihoods as FixedNoiseGaussianLikelihood does not work.
I tried to reinstall GPyTorch in Conda, which did not help.
but importing i.e GaussianLikelihood works without problems. Did somebody had similar problems, or knows how to solve this Error?
EDIT: It only does not work for Jupyter Notebook.
Best


title: Monkey-patching `__reduce__` to customize pickling of a single object

body:
In Python, it's usually easy to monkey-patch a method of an object:
import types

class MyClass:
    def foo(self):
        print("hi")

obj = MyClass()
obj.foo()  # => hi

def myfoo(self):
    print("hello")

obj.foo = types.MethodType(myfoo, obj)
obj.foo()  # => hello

This behavior is not so simple for magic (__*__) methods, and I assume something similar to the linked question is happening when this fails:
import pickle, io

class MyClass:
    def __reduce__(self):
        return (MyClass, tuple())

buffer = io.BytesIO()
pickler = pickle.Pickler(buffer)

obj = MyClass()
pickler.dump(obj)

def myreduce(self):
    print("hello")
    return (MyClass, tuple())

obj.__reduce__ = types.MethodType(myreduce, obj)
pickler.dump(obj)

Where the above does not output hello.
Bizarely, the above does output hello if you change pickler.dump to pickle.dumps. Unfortunately, this is not actually a solution for me: I'm trying to customize the serialization of an object in PyTorch, whose torch.save uses a Pickler object.
Is it possible customize the pickling behavior of a single object?


title: I want to confirm which of these methods to calculate Dice Loss is correct

body:
so I have 4 methods to calculate dice loss and 3 of them are returning the same results, so I can conclude that 1 of them is calculating it wrong, but I would to confirm it with you guys:
import torch
torch.manual_seed(0)

inputs = torch.rand((3,1,224,224))
target = torch.rand((3,1,224,224))

Method 1: flatten tensors
def method1(inputs, target):

    inputs = inputs.reshape( -1)

    target = target.reshape( -1)

    intersection = (inputs * target).sum()
    union = inputs.sum() + target.sum()
    dice = (2. * intersection) / (union + 1e-8)
    dice = dice.sum()

    print("method1", dice)

Method 2: flatten tensors except for batch size, sum all dims
def method2(inputs, target):
    num = target.shape[0]
    inputs = inputs.reshape(num, -1)

    target = target.reshape(num, -1)

    intersection = (inputs * target).sum()
    union = inputs.sum() + target.sum()
    dice = (2. * intersection) / (union + 1e-8)
    dice = dice.sum()/num

    print("method2", dice)

Method 3: flatten tensors except for batch size, sum dim 1
def method3(inputs, target):
    num = target.shape[0]
    inputs = inputs.reshape(num, -1)

    target = target.reshape(num, -1)

    intersection = (inputs * target).sum(1)
    union = inputs.sum(1) + target.sum(1)
    dice = (2. * intersection) / (union + 1e-8)
    dice = dice.sum()/num

    print("method3", dice)

Method 4: don't flatten tensors
def method4(inputs, target):

    intersection = (inputs * target).sum()
    union = inputs.sum() + target.sum()
    dice = (2. * intersection) / (union + 1e-8)

    print("method4", dice)

method1(inputs, target)
method2(inputs, target)
method3(inputs, target)
method4(inputs, target)

method 1,3 and 4 print: 0.5006
method 2 print: 0.1669
and it makes sense, since I am flattening the inputs and targets on 3 dimensions leaving out batch size, and then I am summing all 2 dimensions that result from the flattening instead of just dim 1
Method 4 seems to be the most optimized one


title: Expected 4 dimensional Input but got 3-dimensional input

body:
There are two discriminators, d_l1 and d_l2, and two generators, g_l1, and g_l2. The flow of logic in the original framework goes something like this:
In order to pretrain the discriminator, we have positive data, but we need to generate negative data from the generator. So a function, "generate_samples()" is called.
def generate_samples(model_dict, negative_file, batch_size,
                     use_cuda=False, temperature=1.0):

    neg_data = []
    for i in range(batch_size):
        sample = get_sample(model_dict, use_cuda, temperature)

        if i < 25:
            print("Generated: %s" % sample)
        elif i == 25:
            sample = sample.cpu()
            neg_data.append(sample.data.numpy())
    neg_data = np.concatenate(neg_data, axis=0)

    print("Saving generated samples for reuse.")

    np.save(negative_file, neg_data)

This function uses "get_sample", which receives model_dict, which contains a dictionary of the current set of models in training. In this case, it's the d_l2 and the g_l2. "get_sample" is defined in utils.py, and returns a function which receives the same parameters. Ultimately, the code executed at this stage is a recurrent function that is defined and returns the following code:
elif f_type == "gen":
    def func(model_dict, use_cuda=False, temperature=1.0):
        generator = model_dict["generator"]
        discriminator = model_dict["discriminator"]
        h_w_t, c_w_t, h_m_t, c_m_t, last_goal, real_goal, x_t = \
            init_vars(generator, discriminator, use_cuda)
        t = 0
        gen_token_list = []
        batch_size = generator.worker.batch_size
        seq_len = discriminator.seq_len
        step_size = generator.step_size
        goal_out_size = generator.worker.goal_out_size
        vocab_size = discriminator.vocab_size

        ##################################################################

        vector_size = discriminator.num_filters_total

        ##################################################################

        #G forward
        while t < seq_len:
            #Extract f_t
            if t == 0:
                cur_sen = Variable(nn.init.constant_(
                    torch.zeros(batch_size, seq_len, vector_size), vocab_size)
                ).long()
                if use_cuda:
                    cur_sen = cur_sen.cuda(non_blocking=True)
            else:
                cur_sen = torch.stack(gen_token_list).permute(1, 0)
                cur_sen = F.pad(
                    cur_sen, (0, seq_len - t), value=vocab_size
                )
            f_t = discriminator(cur_sen)["feature"]
            #G forward step
            x_t, h_m_t, c_m_t, h_w_t, c_w_t, last_goal, real_goal, sub_goal, probs, t_ = generator(x_t, f_t, h_m_t, c_m_t, h_w_t, c_w_t, last_goal,real_goal, t, temperature)
            if t % step_size == 0:
                if t > 0:
                    real_goal = last_goal
                    last_goal = Variable(torch.zeros(batch_size, goal_out_size))
                if use_cuda:
                    last_goal = last_goal.cuda(non_blocking=True)
            gen_token_list.append(x_t)
            t = t_
        gen_token = torch.stack(gen_token_list).permute(1,0)
        return gen_token
    return func

This code defines an input variable, "cur_sen", which is initialized as a zero-filled tensor of dimensions, [batch_size, seq_len, vector_size], and then is passed to the forward function in the l2 discriminator in the line f_t = discriminator(cur_sen)["feature"]. The return value of the forward function is a 3-valued dictionary, the "feature" value of which is assigned to f_t. However, that's not important right now, because inside the forward function of the discriminator call, we get the following exception:
Traceback (most recent call last):   File "/mnt/tmp/ReLeakGan/./aux.py", line 48, in <module>
    model_dict_l2, optimizer_dict_l2, scheduler_dict_l2 = pretrain_discriminator_l2(model_dict_l2,   File "/mnt/tmp/ReLeakGan/main.py", line 367, in pretrain_discriminator_l2
    generate_samples(model_dict_l2, neg_l2_fd, batch_size, "l2", use_cuda, temperature)   File "/mnt/tmp/ReLeakGan/main.py", line 279, in generate_samples
    sample = model_dict["discriminator"].get_sample(model_dict, use_cuda, temperature)   File "/mnt/tmp/ReLeakGan/Discriminator.py", line 117, in get_sample
    f_t = discriminator(cur_sen)["feature"]   File "/usr/lib64/python3.9/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)   File "/mnt/tmp/ReLeakGan/Discriminator.py", line 71, in forward
    convs = [F.relu(conv(x)).squeeze(3) for conv in self.convs] # [batch_size * num_filter * seq_len] --> seq_length: Number of sentences in padded paragraph.   File "/mnt/tmp/ReLeakGan/Discriminator.py", line 71, in <listcomp>
    convs = [F.relu(conv(x)).squeeze(3) for conv in self.convs] # [batch_size * num_filter * seq_len] --> seq_length: Number of sentences in padded paragraph.   File "/usr/lib64/python3.9/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)   File "/usr/lib64/python3.9/site-packages/torch/nn/modules/conv.py", line 423, in forward
    return self._conv_forward(input, self.weight)   File "/usr/lib64/python3.9/site-packages/torch/nn/modules/conv.py", line 419, in _conv_forward
    return F.conv2d(input, weight, self.bias, self.stride, RuntimeError: Expected 4-dimensional input for 4-dimensional weight [100, 1, 1, 10], but got 3-dimensional input of size [10, 30, 2000] instead

Now, there are three likely factors. First, the 2nd level discriminator does not embed its input data inside of an nn.Embedding vector, because the data is already in float format, while the l1 discriminator does. Further, in the line torch.zeros(batch_size, seq_len, vector_size), vocab_size),  an extra dimension is added to the zeroed tensor, 'vector_size`. Finally, the variable vocab_size should not really be relevant, as there is no such thing as counting the vocabulary of possible inputs for the 2nd level discriminator, which is essentially uncountably infinite. Though I did not change or remove vocab_size because I don't know what I would put in its place. It's only leftover from the 1st level discriminator.
The code for the 2st level discriminator has been cloned from the 1st level discriminator and modified slightly, BUT IT'S CONSTRUCTOR STILL INITIALIZES THE LIST OF CONV2D MODULES IN EXACTLY THE SAME WAY AS THE 1ST LEVEL DISCRIMINATOR.
The way the 1st level (and currently, the second level) discriminator initializes its conv2d modules is as follows:
self.convs = nn.ModuleList([
    nn.Conv2d(1, num_f, (f_size, self.dis_emb_dim)) for f_size, num_f in zip(self.filter_sizes, self.num_filters)
])

Where self.filter_sizes and self.num_filters are each a list of numbers describing a set of filter sizes and the number of filters of that size for each conv2d created. Both thel1 and l2 discriminators should properly have the values for both.
How would I change the initialization of the modulelist of nn.Conv2d modules so that the list of modules in the 2nd level discriminator takes the 3-dimensional tensor for input instead of the original one?
The pytorch for nn.conv2d documentation, found here is not very helpful.
Thank you for your time.


title: Different outputs with LSTM in PyTorch vs Tensorflow

body:
I am trying to convert a Tensorflow(1.15) model to PyTorch model. Since I was getting very different loss values, I tried comparing the output of the LSTM in the forward pass for the same input. The declaration and initialization of the LSTM is given below:
Tensorflow Code
rnn_cell_video_fw = tf.contrib.rnn.LSTMCell(
            num_units=self.options['rnn_size'],
            state_is_tuple=True,
            initializer=tf.orthogonal_initializer()
        )

rnn_cell_video_fw = tf.contrib.rnn.DropoutWrapper(
            rnn_cell_video_fw,
            input_keep_prob=1.0 - rnn_drop,
            output_keep_prob=1.0 - rnn_drop
        )

sequence_length = tf.expand_dims(tf.shape(video_feat_fw)[1], axis=0)
initial_state = rnn_cell_video_fw.zero_state(batch_size=batch_size, dtype=tf.float32)

rnn_outputs_fw, _ = tf.nn.dynamic_rnn(
                    cell=rnn_cell_video_fw,
                    inputs=video_feat_fw,
                    sequence_length=sequence_length,
                    initial_state=initial_state,
                    dtype=tf.float32
                )

PyTorch code
self.rnn_video_fw = nn.LSTM(self.options['video_feat_dim'], self.options['rnn_size'], dropout = self.options['rnn_drop'])

rnn_outputs_fw, _ = self.rnn_video_fw(video_feat_fw)

Initialization for LSTM in train.py
def init_weight(m):
        if type(m) in [nn.LSTM]:
            for param in m.parameters():
                    nn.init.orthogonal_(m.weight_hh_l0)
                    nn.init.orthogonal_(m.weight_ih_l0)

The output for tensorflow
The output for pytorch
The same is pretty much the case for every data item and my PyTorch model isn't converging. Is my suspicion of difference in output  LSTM being the reason for it correct? If so, where am I going wrong?
Link to the paper
Link to TF code
let me know if anything else is required.


title: How to Fine-tune GPT2 language model with unlabelled data

body:
I am creating a Language Model (i.e., next word predictor) using GPT2. I have followed this blog https://www.analyticsvidhya.com/blog/2019/08/comprehensive-guide-language-model-nlp-python-code/
and used this code to create the language model.
# Import required libraries
import torch
from pytorch_transformers import GPT2Tokenizer, GPT2LMHeadModel

# Load pre-trained model tokenizer (vocabulary)
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')

# Encode a text inputs
text = "What is the fastest car in the"
indexed_tokens = tokenizer.encode(text)

# Convert indexed tokens in a PyTorch tensor
tokens_tensor = torch.tensor([indexed_tokens])

# Load pre-trained model (weights)
model = GPT2LMHeadModel.from_pretrained('gpt2')

# Set the model in evaluation mode to deactivate the DropOut modules
model.eval()

# If you have a GPU, put everything on cuda
tokens_tensor = tokens_tensor.to('cuda')
model.to('cuda')

# Predict all tokens
with torch.no_grad():
outputs = model(tokens_tensor)
predictions = outputs[0]

# Get the predicted next sub-word
predicted_index = torch.argmax(predictions[0, -1, :]).item()
predicted_text = tokenizer.decode(indexed_tokens + [predicted_index])

# Print the predicted word
print(predicted_text)

Now I want to know how I can fine-tune the model or do the transfer learning to train the model on my own dataset. I have a dataset which is related to medicines and healthcare. So, I want my model to do the next word prediction according to my dataset.
Hope you understand what I wanted to do. Any kind of help would be greatly appreciated. Thanks in advance.


title: Evidence lower bound loss function and its gradient in pytorch or tensorflow

body:
How to implement evidence lower bound ELBO loss function and its gradient in pytorch. I have been using KL divergence as following:
# KL Divergence loss function
loss       = nn.KLDivLoss(size_average=False, log_target=True)
out        = loss(Gaussian.log_prob(x), Gaussian.log_prob(xtrue))
gr         = autograd.grad(outputs=[out], inputs=[x])[0]

How can I implement ELBO/gradient?


title: How to achieve better inference latency without GPU acceleration with PyTorch on AWS Lambda?

body:
I am currently using the CycleGAN (https://junyanz.github.io/CycleGAN/) + PyTorch image-to-image translation machine learning framework to convert images into paintings. I am hoping to make a website that can convert a users photos into paintings, and I have successfully trained a model with CycleGAN utilizing a resnet9 generator to make visually appealing outputs. I am learning a lot in this project as I'm relatively new to machine learning and web development. Right now I am able to run the generator network on an AWS lambda and achieve an inference time of around 15 seconds; alternatively I am able to run the network using AWS elastic inference and achieve inference speeds of around 2 seconds (but this is more expensive as I have to pay for the accelerator).
For the last couple of weeks I have been exploring ways to improve the speed of the inference while maintaining the cost effectiveness + scalability of the Lambda infrastructure or elastic inference infrastructure. I know an obvious way to improve inference speed alone is to rent a P3 instance (V100 GPU), but that is way too expensive as I don't imagine the project will generate much revenue, at-least during the beginning phases of the project. For this reason, the AWS Lambda is really appealing to me - I only would have to pay for inferences that are required and there is less of a fixed cost associated with this. My problem is that I am a little out of my element and am not entirely sure where to start without wasting too much of my time. I have converted my model to ONNX and have tried to optimize it using this framework through their tools and also quantization. The quantization however seems to slow down my model for whatever reason (the CPU supports AVX2 instructions). I have tried Intel's OpenVino with the same result - but havent tested this on AWS Lambda yet. I have tried AWS Neo but haven't been able to get that to work correctly. I have tried converting to Tensorflow to run with Tensorflow Lite to no avail.
All of these options I have tried/looked into seem promising to me, but yet I have no idea which choice is optimal. I was hoping someone here has been in a similar situation and have solved it successfully in the past and are willing to point me in the right direction.
Thanks so much.


title: Tracing Tensor Sizes in TorchScript

body:
I'm exporting a PyTorch model via TorchScript tracing, but I'm facing issues. Specifically, I have to perform some operations on tensor sizes, but the JIT compilers hardcodes the variable shapes as constants, braking compatibility with tensor of different sizes.
For example, create the class:
class Foo(nn.Module):
    """Toy class that plays with tensor shape to showcase tracing issue.

    It creates a new tensor with the same shape as the input one, except
    for the last dimension, which is doubled. This new tensor is filled
    based on the values of the input.
    """
    def __init__(self):
        nn.Module.__init__(self)

    def forward(self, x):
        new_shape = (x.shape[0], 2*x.shape[1])  # incriminated instruction
        x2 = torch.empty(size=new_shape)
        x2[:, ::2] = x
        x2[:, 1::2] = x + 1
        return x2

and run the test code:
x = torch.randn((3, 5))  # create example input

foo = Foo()
traced_foo = torch.jit.trace(foo, x)  # trace
print(traced_foo(x).shape)  # obviously this works
print(traced_foo(x[:, :4]).shape)  # but fails with a different shape!

I could solve the issue by scripting, but in this case I really need to use tracing. Moreover, I think that tracing should be able to handle tensor size manipulations correctly.


title: Why is accuracy droping in the last batch?

body:
I am using PyTorch for a classification task. For some reason, the accuracy drops in the last iteration, I would like to know why? any answer is appreciated.
Hers is the code
class Classifier(nn.Module):
    def __init__(self):
        super(Classifier, self).__init__()
        self.layers = nn.Sequential(nn.Linear(89, 128),
                                    nn.ReLU(),
                                    nn.Linear(128, 64),
                                    nn.ReLU(),
                                    nn.Linear(64, 2))
    def forward(self, x):
        return self.layers(x)

def train(train_dl, model, epochs):
    loss_function = nn.CrossEntropyLoss()
    optimizer = optim.Adam(model.parameters(), lr=0.1)
    for epoch in range(epochs):
        for (features, target) in train_dl:
            optimizer.zero_grad()
            features, target = features.to(device), target.to(device)
            output = model(features.float())
            target = target.view(-1)
            loss = loss_function(output, target)
            loss.backward()
            optimizer.step()
            output = torch.argmax(output, dim=1)
            correct = (output == target).float().sum()
            accuracy = correct / 512
            print(accuracy, loss)
        break

model = Classifier().to(device)
train(train_dl, model, 10)

and the last part of the output
tensor(0.6465, device='cuda:0') tensor(0.6498, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(0.6348, device='cuda:0') tensor(0.6574, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(0.6582, device='cuda:0') tensor(0.6423, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(0.6660, device='cuda:0') tensor(0.6375, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(0.6719, device='cuda:0') tensor(0.6338, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(0.6426, device='cuda:0') tensor(0.6523, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(0.6699, device='cuda:0') tensor(0.6347, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(0.6582, device='cuda:0') tensor(0.6422, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(0.6543, device='cuda:0') tensor(0.6449, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(0.6465, device='cuda:0') tensor(0.6502, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(0.6992, device='cuda:0') tensor(0.6147, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(0.6777, device='cuda:0') tensor(0.6289, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(0.6836, device='cuda:0') tensor(0.6244, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(0.6738, device='cuda:0') tensor(0.6315, device='cuda:0', grad_fn=<NllLossBackward>)
tensor(0.1387, device='cuda:0') tensor(0.5749, device='cuda:0', grad_fn=<NllLossBackward>)


title: Python multiprocessing problem using Pytorch BrokenPipeError: [Errno 32] Broken pipe on Windows 7

body:
error
"File “C:\Users\max\anaconda3\lib\multiprocessing\reduction.py”,
 line 60, in dump ForkingPickler(file, protocol).dump(obj)
 BrokenPipeError: [Errno 32] Broken pipe.

not i use win7. when i use dataloader with number of worker any number greater than 0 i receive the error shown above.
when i set number of worker=0 the some of codes executed successfully when the codes reach training, the spyder automatically exit. the same problem appears when i use different versions of Spyder how i can fix this problem please. here some of my codes

    trainer = Trainer(net=model, lr=1e-3, batch_size=96, num_epochs=10)#100)
trainer.run()
train_logs = trainer.train_df_logs
train_logs.columns = ["train_"+ colname for colname in train_logs.columns]
val_logs = trainer.val_df_logs
val_logs.columns = ["val_"+ colname for colname in val_logs.columns]
test_dataset = ECGDataset(test_df)
test_dataloader = DataLoader(dataset=test_dataset, batch_size=96, num_workers=0, shuffle=False)
def make_test_stage(dataloader, model, probs=False):
    cls_predictions = []
    cls_ground_truths = []


title: How does a transposed convolutional layer apply its kernels to the inputs?

body:
I know how a convolutional layer apply its kernels to an input but I don't understand how transposed convolutional layer apply its kernels to an input. My main question is below.
Question:
If a convolutional layer takes in a three channel input and outputs a two channel output and the shape of each kernels are 2x2, I know surely that it will have 6 kernels divided into two groups because it is to give out a 2 channel output so therefore two groups has shown below.
>>> from torch import nn
>>> conv1 = nn.Conv2d(3,2,2,1,0)
>>> conv1.weight.data.numpy()

    array([[[[-0.09294177,  0.19253497],
         [-0.27820718, -0.07189114]],

        [[-0.27682984, -0.05606458],           <---- one group
         [ 0.01134909, -0.21749675]],

        [[-0.2351923 ,  0.2374857 ],
         [-0.0346033 , -0.26447   ]]],

       [[[ 0.2467276 ,  0.06628369],
         [ 0.26501465,  0.11644475]],
                                               <----- one group making two
                                                      groups of three
        [[-0.09835644, -0.06396657],
         [-0.05590855,  0.06890304]],

        [[ 0.22788118,  0.22287966],
         [-0.20899878, -0.03188486]]]], dtype=float32)

You can see that it has six kernels divided into two groups of 3 which makes sense but if a transposed convolutional layer were to take in a three channel input and outputs a two channel output and the shape of each kernels being 2x2, why is it that instead of it having 6 kernels divided into two groups of three, it has 6 kernels divided into 3 groups of two as shown below?
>>> conv2 = nn.ConvTranspose2d(3,2,2,1,0)
>>> conv2.weight.data.numpy()

    array([[[[ 0.28577724, -0.29587495],
         [-0.24003945, -0.3524448 ]],
                                        <---- one group
        [[-0.15984103,  0.22188954],
         [-0.10990701, -0.20565327]]],

       [[[ 0.17101079, -0.17623127],
         [-0.12097928, -0.0211492 ]],
                                         <----- one group
        [[-0.21161021, -0.33530322],
         [-0.16497111,  0.19984488]]],

       [[[-0.05084743, -0.2563213 ],
         [-0.28287342, -0.30839682]],
                                         <----- one group making three groups
                                                two
        [[-0.330719  ,  0.07809895],
         [-0.16823643, -0.34404978]]]], dtype=float32)

You can see above that it has 6 kernels divide into 3 groups of 2. Isn't it meant to be divided into 2 groups of three because it is to give out two channel thats why it should be divided into two groups like the convolutional layer? Why is the arrangement this way and how does 6 kernels divided into 3 groups of 2 turn a 3 channel input into two?


title: Runtime error: cuda extensions compiled with non-matching version used to compile pytorch - Conda environment

body:
I am trying to install this repo Scene-Graph-Benchmark according to the instructions but get this error when I try to set up apex with the command python setup.py build_ext install:

I followed the instructions in the repo exactly, including the specific versions within the command conda install pytorch==1.4.0 torchvision==0.5.0 cudatoolkit=10.1 -c pytorch. I thought I should use cudatoolkit because running Nvidia-smi gives me

I verified that pytorch is installed and confirmed that torch.cuda.is_available() == True
I also tried not specifying the versions, by doing conda install pytorch torchvision cudatoolkit -c pytorch but also get an error. Any help would be greatly appreciated
For sake of completeness, this is what the environment looks like after following the setup instructions in the repo


How can I add a module into a network without change the main body of the network? For example, I want to add an attention module after the layer1 of Resnet. I have tried to use forward_hook to add the new module, but after reading the source code of nn.Module:
def __call__(self, *input, **kwargs):
   result = self.forward(*input, **kwargs)
   for hook in self._forward_hooks.values():
       hook_result = hook(self, input, result)
   ...
   return result

I found that hooks are done after forward, so I can't use hook to add new modules. Are there any other way to realize this function?


title: Passing Parameters in Python/Pytorch Classmethod

body:
I am confused while passing the parameters in classmethod.
The code is shown below:
def gather_feature(fmap, index):
    # fmap.shape(B, k1, 1)  index.shape(B, k2)
    dim = fmap.size(-1)
    index = index.unsqueeze(len(index.shape)).expand(*index.shape, dim)  # this works

    fmap = fmap.gather(dim=1, index=index)  # out[i][j][k] = input[i][index[i][j][k]][k]  # if dim == 1.
    return fmap

def gather_feature(fmap, index):
    # fmap.shape(B, k1, 1)  index.shape(B, k2)
    dim = fmap.size(-1)
    index = index.unsqueeze(len(index.shape))
    index = index.expand(*index.shape, dim)  # raise error

    fmap = fmap.gather(dim=1, index=index)  # out[i][j][k] = input[i][index[i][j][k]][k]  # if dim == 1.
    return fmap

Once index.unsqueeze() has done, the shape of index would be changed to (B, k2, 1).
If the index.shape that pass to expand() classmethod is (B, k2, 1), an error has raised.
However, if writing these tow classmethod in one line, namely index.unsqueeze().expand(), the index.shape passing to expand() classmethod seems to be (B, k2).
Has the index.shape been computed and stored before performing .unsqueeze()?
Therefore, the .unsqueeze() won't affect the index.shape which pass to .expand().
That is my guess, but I cannot figure out another one.
Thank you for your time.


title: In pytorch, which part cost most time? Loss calculation or backward+step?

body:
In pytorch, which part cost most of time?

Loss calculation(no backward, just calculating)
backward
step
Please explain a bit if you can.


title: Pytorch map tensor row-wise

body:
I have a tensor, like this, where each line corresponds to one prediction of a probability distribution.
output = [0.42 0.32 0.000 .0123 ...]
         [0.12 0.352 0.000 .013 ...]
         [          ...            ]
         [0.22 0.32 0.000 .0143 ...]

I have another tensor with the references and I want to calculate a custom loss.
For that, I need to iterate each line row-wise to use the first row of each tensor to calculate one loss.
example:
loss = 0
for i in range (0, output.shape[0]):
    custom_loss += loss(output[i,:], reference[i,:])
return loss

Is there any method available to do this? Any code suggestions to keep it simple?


title: How to train pytorch model using batches with partial information

body:
In this PyTorch model I have two streams data with two different modalities that are input into the model at the same time. The streams of data are blocks of sequential data. So I have modality one M1 = [[1,1,1,1],[2,2,2,2],[3,3,3,3],[4,4,4,4],[5,5,5,5]] and modality two M2 = [[11,11,11,11],[22,22,22,22],[33,33,33,33],[44,44,44,44]]. I want to train this model with a system that during training, batches of sequential data will randomly have partial or full information.
So there will be three possibilities during training: M1 and M2 will have its full sequential data, or M1 will be clipped meaning that the sequential data information will be set to zero (e.g. M1 = [[0,0,0,0],[0,0,0,0],[0,0,0,0],[0,0,0,0],[0,0,0,0]] ), while M2 will have its full sequential data info, or M2 will be clipped meaning that the sequential data information in M2 will be set to zero, while M1 will have its full sequential data info.
Are there any PyTorch functions that will do that automatically for me, or does anyone know what would be a good way to implement this?


title: Mean squared logarithmic error using pytorch

body: hello I'm new with PyTorch and i would like to use Mean squared logarithmic error as a loss function in my neural  network for training my DQN agent but i can't find the MSLE in the nn.functional in PyTorch what the best way to implement it ?


title: Adding predicted test labels to original test data frame in PyTorch LSTM

body:
I have run an LSTM model in PyTorch on text data. My original data frame (both test and train) contains 3 columns. Below is the tail end of my test data frame, which I put into my model.
    TEXT    ICD_25000   HADM_ID
23995   s brother diabetes deceased brother mastoid ca...   0   115229.0
23996   x to be used with insulin pen three times a da...   0   170587.0
23997   lung biopsy in borderline diabetes diagnosed y...   0   174893.0
23998   have nv she has been unable to eat today past ...   0   108008.0
23999   one brother had a stroke at age several member...   0   151301.0

I run my model and get ~82% accuracy, but need help getting the predicted results (From this LSTM model) back into the original test data frame above (which contains the actual labels, (ICD9_2500)). I need to keep the HADM_ID codes in the output test data frame with the predicted value (0 or 1), actual value (0 or 1) and TEXT.
My code is as follow:

fields = [('TEXT', TEXT), ('ICD_25000', LABEL)]

train_df = data.TabularDataset(
    path="window_train_with_HADM.csv", format='csv',
    skip_header=True, fields=fields)

test_df = data.TabularDataset(
    path="window_test_with_HADM.csv", format='csv',
    skip_header=True, fields=fields)

TEXT.build_vocab(train_df, max_size=VOCABULARY_SIZE)
LABEL.build_vocab(train_df)

print(f'Vocabulary size: {len(TEXT.vocab)}')
print(f'Number of classes: {len(LABEL.vocab)}')

train_loader,test_loader = data.BucketIterator.splits(
    (train_df, test_df),
    batch_size=BATCH_SIZE,
    sort_within_batch=True, # necessary for packed_padded_sequence
    sort_key=lambda x: len(x.TEXT),
    device=DEVICE)

import torch.nn as nn

class RNN(nn.Module):
    def __init__(self, input_dim, embedding_dim, hidden_dim, output_dim):

        super().__init__()
        #Here is a preliminary model using LSTM cell
        #The primary goal of this lab is to vary the dimensions of the embeddings and see the results
        #The second task is to use a another RNN cell such as GRU and perform parameter tuning and report the results.
        self.embedding = nn.Embedding(input_dim, embedding_dim)
        self.rnn = nn.LSTM(embedding_dim, hidden_dim)
        self.fc = nn.Linear(hidden_dim, output_dim)

        # Define proportion or neurons to dropout

    def forward(self, text, text_length):

        #[sentence len, batch size] => [sentence len, batch size, embedding size]
        embedded = self.embedding(text)

        packed = torch.nn.utils.rnn.pack_padded_sequence(embedded, text_length)

        #[sentence len, batch size, embedding size] =>
        #  output: [sentence len, batch size, hidden size]
        #  hidden: [1, batch size, hidden size]
        packed_output, (hidden, cell) = self.rnn(packed)
         # Apply dropout

        return self.fc(hidden.squeeze(0)).view(-1)

This is the  code I have used in my training to calculate the training and test accuracy.
def compute_binary_accuracy(model, data_loader, device):
    model.eval()
    correct_pred, num_examples = 0, 0
    with torch.no_grad():
        for batch_idx, batch_data in enumerate(data_loader):
            text, text_lengths = batch_data.TEXT
            logits = model(text, text_lengths.cpu())
            predicted_labels = (torch.sigmoid(logits) > 0.5).long()
            num_examples += batch_data.ICD_25000.size(0)
            correct_pred += (predicted_labels.long() == batch_data.ICD_25000.long()).sum()
        return correct_pred.float()/num_examples * 100

Can anyone help me come up with a function that takes the test loader data and outputs a data frame with all the original test data, PLUS the predictions from my model? I do not understand  where the results from the model are being stored for the test data, and need to input that into a data frame with the original test data.


title: What is the best way to make the number of MRI slices of the patients the same when we are loading them as one dimension to train the model

body:
I have 100 patients, each has one exam which includes a number of MRI slices. For training the model I decided to have the number of slices as a channel. So if each image is 256x256 greyscale, then I have (number_of_slices x 256 x 256) as the size of the input of the CNN model. The problem is that we have a different number of slices for each patient, a number between 21 and 28.
What is the correct way to make the number of slices the same across the patients' exams?  For instance, have 24 slices for all of the exams.
My solution for that was using interpolation in PyTorch which I wasn't able to get it working.
sourceSize = [sourceImg.shape[0], sourceImg.shape[1], sourceImg.shape[2]]
targetSize = [24, 256, 256]

c = np.linspace(0, sourceSize[0]-1, targetSize[0])
w = np.linspace(0, sourceSize[1]-1, targetSize[1])
h = np.linspace(0, sourceSize[2]-1, targetSize[2])

C, H, W = np.meshgrid(c,w,h, indexing='ij')

interpn((c, w, h), C, sourceImg)


title: Difference between Tensor and tensor

body:
I am working with Pytorch. But when I use tensor I got an error while I don't get any error when I use Tensor.
What is the difference between Tensor and tensor?
This is my example:
tns = torch.tensor([91,21,34,56])

tns.mean()

I got this error:

RuntimeError: Can only calculate the mean of floating types. Got Long
instead.

Thanks in advance.


title: How to drop the nodes in specified positions of the layer in PyTorch

body:
I’m trying to implement a fully connected network to do a
    classification task, while my training data contains some missing
    values and the positions of missingness are not fixed.

For example:

[x1, x2, NaN, x4, x5]  [y]
[x1, x2, x3, x4, NaN]  [y]
[NaN, x2, x3, NaN, x5] [y]

I have tried to implement some methods to fill in the missing values
but it will introduce bias after all.

To solve the bias problem, I’m planning to keep the missingness but
drop the corresponding nodes that the missing values appear in the
input layer so that the NaNs will not be fed into the network and cause
errors, and the following layers of the network stay the same. I wonder
if there are specific methods to achieve it in PyTorch, either by:

Drop the node in a specified position of the layer, or
Enable variable input size of the neural network.

However, these are just two possible solutions I can come up with, I
would be appreciated if you could share with me other useful ways.


title: Best output format for storing outputs colllected after batch

body:
I have a pre-trained segmentation network which takes in a (batch_size,5,224,224) Tensor and outputs a (batch_size,224,224) feature vector, say batch_features, originally a Torch tensor but can be converted to numpy array.

There are multiple ways to collect the torch tensor.

Option A : Use a list

val_features = []
val_features.extend(list(batch_features)

***Option B : Use a numpy array*

val_features = np.empty()
val_features = [*val_features, *batch_features]

At the end, I want to iterate through all these arrays and do some post-processing. Which data format would work best in that case?


title: Can we change the contents of the tensors once it is used?

body:
I am wondering if we are allowed to change the contents of the tensors after they are sent to the Loss Functions in pytorch. For example:

x = torch.zeros(1000)
y = torch.zeros(1000)
output = net(x)

loss = criterion(oytput, y)
loss.backward()
optimizer.step()

After we have done this, can we change the contents of y and output without ill-effect? For example:

y[0] = 990
output[0] = 1000

After I have done this after a mini-batch, but keep feeding it with more mini-batches, will this cause issues?

I am not so sure, because maybe the nodes are still referenced internally by the computational graph.


title: I load a trained model, the loss is quite large

body:
I have trained a translation model:

When  step % 1000 == 0, I evaluate the loss on validation.
If the loss is small，I save the state.
When I finish training, I load the model and the loss is quite large.

    if step % 1000 ==0:
        model.eval()
        val_loss = eval(model,val_loader)
        print("epoch:",epoch,"step:",step,"loss:",loss,"val_loss:",val_loss)

        if val_loss.item() < best_loss:
            best_loss = val_loss.item()
            best_epoch = epoch
            torch.save(model.state_dict(),'best_1.m')

when I finish the training, the loss is about 0.665 but when I load the same model the loss is about 300.

model.load_state_dict(torch.load('best_1.m'))
model.eval()
val_loss = eval(model,val_loader)
print('val_loss:',val_loss)


title: Pytorch tensor data is disturbed when I convert data loader tensor to numpy array

body:
I am using a simple train loop for a regression task. To make sure that regression ground-truth values are the same as what I expect in the training loop, I decided to plot each batch of data.
However, I see that when I convert the data loader’s tensor to numpy array and plot it, it is disturbed. I am using myTensor.data.cpu().numpy() for the conversion.

My code is as below:

train_ds = TensorDataset(x_train, y_train)
train_dl = DataLoader(train_ds, batch_size = 32, shuffle = True, num_workers = 0, drop_last = True)

for epoch in range(epochs):
  model.train()
for i, (x, y) in enumerate(train_dl):
  x = x.cuda()
  y = y.cuda()
  yy = y.data.cpu().numpy()
  pyplot.plot(yy[0: 32, 0])
  pyplot.show()


title: L1 regulariser Pytorch acting opposite to what I expect

body:
I'm trying to add an L1 penalty to a specific layer of a neural network, and I have the code below (in which I attempt to add l1 penalty to the first layer). If I run it for lambda = 0 (i.e. no penalty), the output gets very close to the expected weights those being [10, 12, 2, 11, -0.25]) and if I run for enough epochs or reduce batch size it will get it exactly, as in the output below:

  mlp.0.weight

  Parameter containing:

  tensor([[  9.8657, -11.8305,   2.0242,  10.8913,  -0.1978]],
         requires_grad=True)

Then, when I run it for a large lambda, say 1000, I would expect these weights to shrink towards zero as there is a large penalty being added to the loss that we are trying to minimise. However, the opposite happens and the weights explode, as in the output below (for lam = 1000)

  mlp.0.weight

  Parameter containing:

  tensor([[-13.9368,   9.9072,   2.2447, -11.6870,  26.7293]],
         requires_grad=True)

If anyone could help me, that'd be great. I'm new to pytorch (but not the idea of regularisation), so I'm guessing it's something in my code that is the problem.

Thanks

import torch
import torch.nn as nn
from torch.utils.data import Dataset, DataLoader
import numpy as np
from sklearn.linear_model import LinearRegression

class TrainDataset(Dataset):
    def __init__(self, data):
        self.data = data
    def __len__(self):
        return self.data.shape[0]
    def __getitem__(self, ind):
        x = self.data[ind][1:]
        y = self.data[ind][0]
        return x, y

class TestDataset(TrainDataset):
    def __getitem__(self, ind):
        x = self.data[ind]
        return x

torch.manual_seed(94)

x_train = np.random.rand(1000, 5)
y_train = x_train[:, 0] * 10 - x_train[:, 1] * 12 + x_train[:, 2] * 2 + x_train[:, 3] * 11 - x_train[:, 4] * 0.25
y_train = y_train.reshape(1000, 1)
x_train.shape
y_train.shape
train_data = np.concatenate((y_train, x_train), axis=1)

train_set = TrainDataset(train_data)

batch_size = 100
train_loader = DataLoader(train_set, batch_size=batch_size, shuffle=True)

class MLP(nn.Module):
    def __init__(self):
        super(MLP, self).__init__()
        self.mlp = nn.Sequential(nn.Linear(5, 1, bias=False))

    def forward(self, x_mlp):
        out = self.mlp(x_mlp)
        return out

device = 'cpu'

model = MLP()

optimizer = torch.optim.SGD(model.parameters(), lr=0.02, momentum=0.82)

criterion = nn.MSELoss()

epochs = 5
lam = 0
model.train()

for epoch in range(epochs):
    losses = []
    for batch_num, input_data in enumerate(train_loader):
        optimizer.zero_grad()

        x, y = input_data
        x = x.to(device).float()
        y = y.reshape(batch_size, 1)
        y = y.to(device)

        output = model(x)
        for name, param in model.named_parameters():
            if name == 'mlp.0.weight':
                l1_norm = torch.norm(param, 1)

        loss = criterion(output, y) + lam * l1_norm
        loss.backward()

        optimizer.step()
        print('\tEpoch %d | Batch %d | Loss %6.2f' % (epoch, batch_num, loss.item()))

for name, param in model.named_parameters():
    if param.requires_grad:
        print(name)
        print(param)


title: Trying to use Cuda in pyTorch

body:
I've been working on PyTorch and wanted to use Cuda tensors but I've been having trouble getting it to work.

When I use the line torch.cuda.is_available(), it returns false.

But when I use the same line on the anaconda command prompt, it returns true. What gives? Do I need to set the device somehow? Or maybe have the interpreter include my GPU? All I want is my GPU to be recognized as CUDA usable and can use in code.

I use:

python 3.7

CUDA 10.1

GPU: GTX 1070


Is there a convenient method to only save model architecture information in Pytorch to a protobuf ruled file?

I know how to use pytorch.save to save both weights and net at the same time, to a dictionary structured data. But if I'd like to save the data to an isolated file which only contain the net architecture, like what Caffe did train from initial status, is that possible? The file may used in somewhere else. Does ONNX can do something sort of like that?


title: Can we make use of variable inside method definition within a class without defining it outside the method and inside class apriori?

body:
I'm beginner in pytorch and python, I was looking as some examples of codes and

I'm wondering the use of d in the following codes:

def train_and_eval(self):
        self.entity_idxs = {d.entities[i]:i for i in range(len(d.entities))}
        self.relation_idxs = {d.relations[i]:i for i in range(len(d.relations))}

in the d.entities[i], I didn't see anywhere before the code where d is defined.

So what could this mean? can you explain the codes? I don't know if my question is clear or not, please ask if you need extra backgroud.

I can also show the beginning of code:

class Experiment:

    def __init__(self, learning_rate=0.0005, ent_vec_dim=200, rel_vec_dim=200,
                 num_iterations=500, batch_size=128, decay_rate=0., cuda=False,
                 input_dropout=0.3, hidden_dropout1=0.4, hidden_dropout2=0.5,
                 label_smoothing=0.):
        self.learning_rate = learning_rate
        self.ent_vec_dim = ent_vec_dim
        self.rel_vec_dim = rel_vec_dim
        self.num_iterations = num_iterations
        self.batch_size = batch_size
        self.decay_rate = decay_rate
        self.label_smoothing = label_smoothing
        self.cuda = cuda
        self.kwargs = {"input_dropout": input_dropout, "hidden_dropout1": hidden_dropout1,
                       "hidden_dropout2": hidden_dropout2}

    def get_data_idxs(self, data):
        data_idxs = [(self.entity_idxs[data[i][0]], self.relation_idxs[data[i][1]], \
                      self.entity_idxs[data[i][2]]) for i in range(len(data))]
        return data_idxs

You see that here they were using self.entity_idxs in the first method of the class and it was not defined anywhere eiter. and this was used again later on in the class which is the first snipped of code i provided.


title: Save the output of a trained pytorch neural network in .ark format

body: I am training my acoustic model which is multilayer DNN written in PyTorch. I want to save the output of the network as (.ark) file to use it in the decoding stage for automatic speech recognition. I searched a lot but can't find a function to do this. Any Help !!!


title: Epoch time increasing when running Pytorch on AML docker

body:
The epoch time keeps increasing when running Pytorch training on AML and a custom docker image.
When the same code is running locally the epoch time is constant (the difference here is that there is no docker image involved and the training dataset exists locally, so not a mounted blob storage by AML and the machine is different)

Any suggestions on how I can figure out what's happening?
For example how can I log useful memory consumption? GPU, Pytorch, disk access, etc.


title: How to create a tensor index that will have multiple indices?

body:
I am new to PyTorch. I need to store id for my validation set in GPU. For example, I will have one id say 1012, now I need to create another set like:

tensor results1=(
[102(another id), 12(rank) ], //(another id) and (rank) will not be there
[103(another id),  1(rank) ],
[104(another id),  5(rank) ],
[104(another id) , 1(rank) ]
)
...

What I want is that I can have a tensor variable in GPU like :

tensor [1012][results1]
tensor [1013][results2]

And also how to retrieve for each index.
Any help will be appreciated.


title: how to apply model developped in fast.ai / pytorch?

body:
I've trained a model which i'm trying to apply onto new data. I'm totally new to fast.ai
i'm creating my databunch as below (ds being the data i want to score):

bs = 64
data_lm = (TextList.from_df(df, path, cols='comment_text')
            .split_by_rand_pct(0.1)
            .label_for_lm()
            .databunch(bs=bs))

The problem being that I cannot ommit the .split_by_rand_pct(0.1), so I cannot score the whole data

I then go and load/apply the model as below

data_clas = load_data(path, 'data_clas.pkl', bs=bs)
learn = text_classifier_learner(data_clas, AWD_LSTM, drop_mult=0.5)
learn.load_encoder('fine_tuned_enc')
learn.load('third');
preds, target = learn.get_preds(DatasetType.Test, ordered=True)
labels = preds.numpy()

But the problem is i'm only scoring 0.1 pct of my data as the first piece of code when I create the databunch is not correct...i'm wanting to apply the saved/loaded model onto the overall DF.

Many thanks in advance


title: I modified a few layers to an example of a neural network just to see if I could. What's wrong with it?

body:
A simple neural network I found had the layers w1, Relu, and w2. I tried to add a new weight layer in the middle and a second Relu after it. So, the layers are as follows w1, Relu, w_mid, Relu, and w2.
It is much much slower than the original 3 layer network if it works at all. I'm not sure if everything is getting a forward pass and if back prop is working across every part it is supposed to.
The neural network is from this link. It is the third block of code down the page.

This is the code I changed.
Below it is the original.

    import torch
    dtype = torch.float
    device = torch.device("cpu")
    #device = torch.device("cuda:0") # Uncomment this to run on GPU

    # N is batch size; D_in is input dimension;
    # H is hidden dimension; D_out is output dimension.
    N, D_in, H, D_out = 64, 250, 250, 10

    # Create random input and output data
    x = torch.randn(N, D_in, device=device, dtype=dtype)
    y = torch.randn(N, D_out, device=device, dtype=dtype)

    # Randomly initialize weights
    w1 = torch.randn(D_in, H, device=device, dtype=dtype)
    w_mid = torch.randn(H, H, device=device, dtype=dtype)
    w2 = torch.randn(H, D_out, device=device, dtype=dtype)

    learning_rate = 1e-5
    for t in range(5000):
        # Forward pass: compute predicted y
        h = x.mm(w1)
        h_relu = h.clamp(min=0)
        k = h_relu.mm(w_mid)
        k_relu = k.clamp(min=0)
        y_pred = k_relu.mm(w2)

        # Compute and print loss
        loss = (y_pred - y).pow(2).sum().item()
        if t % 1000 == 0:
            print(t, loss)

        # Backprop to compute gradients of w1, mid, and w2 with respect to loss
        grad_y_pred = (y_pred - y) * 2
        grad_w2 = k_relu.t().mm(grad_y_pred)
        grad_k_relu = grad_y_pred.mm(w2.t())
        grad_k = grad_k_relu.clone()
        grad_k[k < 0] = 0
        grad_mid = h_relu.t().mm(grad_k)
        grad_h_relu = grad_k.mm(w1.t())
        grad_h = grad_h_relu.clone()
        grad_h[h < 0] = 0
        grad_w1 = x.t().mm(grad_h)

        # Update weights
        w1 -= learning_rate * grad_w1
        w_mid -= learning_rate * grad_mid
        w2 -= learning_rate * grad_w2

The loss is ..
0 1904074240.0
1000 639.4848022460938
2000 639.4848022460938
3000 639.4848022460938
4000 639.4848022460938

This is the original code from the Pytorch website.

    import torch

    dtype = torch.float
    #device = torch.device("cpu")
    device = torch.device("cuda:0") # Uncomment this to run on GPU

    # N is batch size; D_in is input dimension;
    # H is hidden dimension; D_out is output dimension.
    N, D_in, H, D_out = 64, 1000, 100, 10

    # Create random input and output data
    x = torch.randn(N, D_in, device=device, dtype=dtype)
    y = torch.randn(N, D_out, device=device, dtype=dtype)

    # Randomly initialize weights
    w1 = torch.randn(D_in, H, device=device, dtype=dtype)
    w2 = torch.randn(H, D_out, device=device, dtype=dtype)

    learning_rate = 1e-6
    for t in range(500):
        # Forward pass: compute predicted y
        h = x.mm(w1)
        h_relu = h.clamp(min=0)
        y_pred = h_relu.mm(w2)

        # Compute and print loss
        loss = (y_pred - y).pow(2).sum().item()
        if t % 100 == 99:
            print(t, loss)

        # Backprop to compute gradients of w1 and w2 with respect to loss
        grad_y_pred = 2.0 * (y_pred - y)
        grad_w2 = h_relu.t().mm(grad_y_pred)
        grad_h_relu = grad_y_pred.mm(w2.t())
        grad_h = grad_h_relu.clone()
        grad_h[h < 0] = 0
        grad_w1 = x.t().mm(grad_h)

        # Update weights using gradient descent
        w1 -= learning_rate * grad_w1
        w2 -= learning_rate * grad_w2


title: Constructing pytorch tensor from existing cuda memory

body:
I am working on speeding up a AI to detect the presence and pose of robots from a video stream.
Video Stream -> Java Cuda GPU Decoder -> Video File -> Decoder -> Load GPU Tensor -> Run AI Model

I am wondering if it is possible remove the file/io, redundant decoding, and GPU-CPU-GPU data tranfer by changing the workflow to.
Video Stream -> Java Cuda GPU Decoder -> Format to Tensor -> Run AI Model

I think have seen some examples similar to this when a GPU decoder in C is able to convert the GPU object directly to pytorch in the DALI Library. But I am really clueless to how to proceed about this. If anyone can point me to some directions I would greatly appreciate it.


title: Pytorch Problem with Custom Dataset Class

body:
First, I made a custom dataset to load in images from my dataframe (containing the image filepath and corresponding int label):
class Dataset(torch.utils.data.Dataset):

    def __init__(self, dataframe, transform=None):
        self.frame = dataframe
        self.transform = transform

    def __len__(self):
        return len(self.frame)

    def __getitem__(self, idx):
        if torch.is_tensor(idx):
            idx = idx.tolist()

        filename = self.frame.iloc[idx, 0]
        image = torch.from_numpy(io.imread(filename).transpose((2, 0, 1))).float()
        label = self.frame.iloc[idx, 1]
        sample = {'image': image, 'label': label}
        if self.transform:
            sample = self.transform(sample)
        return sample

Then, I use pre-existing model architecture like so:
model = models.densenet161()
num_ftrs = model.classifier.in_features
model.classifier = nn.Linear(num_ftrs, 10)  # where 10 is my number of classes

criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)

Finally, for training, I do the following:
model.train()  # switch to train mode

for epoch in range(5):
    for i, sample in enumerate(train_set):  # where train_set is an instance of my Dataset class
        optimizer.zero_grad()
        image, label = sample['image'].unsqueeze(0), torch.Tensor(sample['label']).long()
        output = model(image)

        loss = criterion(output, label)
        loss.backward()
        optimizer.step()

However, I am experiencing errors with loss = criterion(output, label). It tells me that ValueError: Expected input batch_size (1) to match target batch_size (2).. Can someone teach me how to properly use a custom dataset, especially with loading in batches of data? Also, why am I experiencing that ValueError? Thank you!


title: Finding the mean per class in Pytorch

body:
I am naively iterating over each and every sample of the dataset. Is there anyway to calculate the mean efficiently ?
my_root = '/mini_imagenet_full_size/train/'
Clmean = []
dir_list = os.listdir(my_root)
print(len(dir_list))
miniImagenet_dataset = datasets.ImageFolder(root=my_root, transform=data_transform)
Clmean=torch.zeros([64,3,224,224])
for t,c in miniImagenet_dataset:
   print(c)
   Clmean[c,:,:,:]+=t

print(Clmean)


title: pytorch cnn Test result is sticked

body:
TRAINING CODE
if os.path.isfile(PATH):
      print("checkpoint training '{}' ...".format(PATH))
      checkpoint = torch.load(PATH)
      start_epoch = checkpoint['epoch']
      start_i = checkpoint['i']
      net.load_state_dict(checkpoint['state_dict'])
      print("=> loaded checkpoint '{}' (trained for {} epochs, {} i)".format(PATH, checkpoint['epoch'],
                                                                           checkpoint['i']))
    else:
        print('new training')

for epoch in range(num_epochs):  # loop over the dataset multiple times
    running_loss = 0.0
    for i in range(len(train_folder_list2)):
        # get the inputs; data is a list of [inputs, labels]
        # net.train()
        inputs, labels = train_input[i], train_list[i]
        inputs = torch.as_tensor(inputs).cuda()
        inputs = inputs.transpose(1, 3)
        labels = torch.as_tensor(labels).cuda()
        # forward + backward + optimize
        outputs = net(inputs)
        loss = criterion(outputs, labels)

        # zero the parameter gradients
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        running_loss += loss.item()
        if i % 100 == 1:
            save_checkpoint({
               'epoch': start_epoch + epoch + 1,
               'i': start_i + i + 1,
               'state_dict': net.state_dict(),
        })

TEST CODE
PATH = './checkpoint.pth'
model = Net().cuda()
if os.path.isfile(PATH):
    print('checkpoint check!')
    checkpoint = torch.load(PATH)
    model.load_state_dict(checkpoint['state_dict'])
    model.eval()

for k in range(len(train_folder_list2)):
    inputs = train_input[k]
    inputs = torch.as_tensor(inputs).cuda()
    inputs = inputs.transpose(1, 3)
    outputs = model(inputs)
    result = outputs.cpu().detach().numpy()

This is the code to find the edges of the image.
If I run the training code, train it, and test it with the test code, it doesn't seem to find any edges in the image. The edges are on the same side, whatever image i put.
**ADD
CNN CODE
In addition, we added cnn code to give you information. Data input was put in the list separately from the image and label.
class Net(nn.Module):
def __init__(self):
    super(Net, self).__init__()
    self.conv1 = nn.Conv2d(3, 6, 5)
    self.pool = nn.MaxPool2d(2, 2)
    self.conv2 = nn.Conv2d(6, 16, 5)
    self.fc1 = nn.Linear(293904, 120)
    self.fc2 = nn.Linear(120, 84)
    self.fc3 = nn.Linear(84, 18)

def forward(self, x):
    x = self.pool(F.relu(self.conv1(x)))
    x = self.pool(F.relu(self.conv2(x)))
    x = x.view(-1, 293904)
    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(x))
    x = self.fc3(x)
    x = x.view(18)

    return x


title: LSTM - Predicting the same constant values after some rows

body:
I build up a module using nn.LSTM to predicate same thing but it predict same values after 5-10 predictions:
    def __init__(self, input_size=len(train_cols), hidden_size=40, output_size=2, num_layer=10):
        super(NET, self).__init__()
        self.rnn = nn.LSTM(input_size, hidden_size, num_layer)
        self.out = nn.Linear(hidden_size, output_size)

    def forward(self, x):
        out, _ = self.rnn(x)
        out = self.out(out[:, -1, :])
        return out
net = NET(output_size=1)
optimizer = torch.optim.Adam(net.parameters(), lr=0.08, betas=(0.9, 0.999), eps=1e-08, weight_decay=0)
loss_func = torch.nn.MSELoss()
test_x = test_x.reshape(-1, 1, len(train_cols))
test_x = torch.from_numpy(test_x)
test_xv = Variable(test_x)
for epoch in range(100):
    var_x = Variable(train_x).type(torch.FloatTensor)
    var_y = Variable(train_y).type(torch.FloatTensor)
    out = net(var_x)
    loss = loss_func(out.squeeze(), var_y.squeeze())
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    if (epoch + 1) % 5 == 0:
        print('Epoch: {}, Loss: {:.5f}'.format(epoch + 1, loss.data.numpy()))
        pred_test = net(test_xv)
        pred_test = pred_test.view(-1).data.numpy()
        viz.text(str(np.dstack((pred_test,test_y))), win='compare',
                 opts=dict(title='pred'))

The prediction is:

[[[0.90698224 0.97657084] [1.3066663 0.98598385] [1.4000531 1.003238 ]
[1.4139748 1.018212 ] [1.4166933 1.0115894 ] [1.4169703 0.99727374]
[1.4168639 0.9989065 ] [1.4166994 0.9569661 ] [1.4165486 0.96585363]
[1.4164208 0.96672124] [1.416315 0.9726626 ] [1.4162277 0.9950739 ]
[1.4161557 1.0281057 ] [1.4160964 1.0196408 ] [1.4160475 1.0191873 ]
[1.4160074 1.0168635 ] [1.4159743 1.0126513 ] [1.4159468 1.004374 ]
[1.4159242 1.0187122 ] [1.4159056 1.0370985 ] [1.4158903 1.0679934 ]
[1.4158777 1.0439979 ] [1.4158672 1.044638 ] [1.4158586 1.0534846 ]
[1.4158515 1.0213561 ] [1.4158456 0.97619045] [1.4158409 0.97762746]
[1.4158368 0.9567483 ] [1.4158336 0.93333334] [1.4158307 0.9560899 ]]]

Who can tell me why?


title: Can anyone help me understand why my keras to pytorch conversion isn't working?

body:
I am trying to migrate from keras to pytorch and I am getting some inconsistent behaviors. First of all I'm noticing that I have a different number of parameters but the other thing I'm noticing is that the keras training converges a lot faster and more smoothly than the pytorch. Have I done something wrong? Can anyone help? The relevant code is below.
I have the following simple architecture in keras:
def identity_block(input_tensor, units):
   x = layers.Dense(units, kernel_regularizer=reg)(input_tensor)
   x = layers.BatchNormalization()(x)
   x = layers.Activation('relu')(x)

   x = layers.Dense(units, kernel_regularizer=reg)(x)
   x = layers.BatchNormalization()(x)
   x = layers.Activation('relu')(x)

   x = layers.Dense(units, kernel_regularizer=reg)(x)
   x = layers.BatchNormalization()(x)
   x = layers.add([x, input_tensor])
   x = layers.Activation('relu')(x)
   return x

def dens_block(input_tensor, units, reps=2):
    x = input_tensor

    for _ in range(reps):
        x = layers.Dense(units, kernel_regularizer=reg)(x)
        x = layers.BatchNormalization()(x)
        x = layers.Activation('relu')(x)

    x = layers.Dense(units, kernel_regularizer=reg)(x)
    x = layers.BatchNormalization()(x)
    shortcut = layers.Dense(units, kernel_regularizer=reg)(input_tensor)
    shortcut = layers.BatchNormalization()(shortcut)
    x = layers.add([x, shortcut])
    x = layers.Activation('relu')(x)
    return x

def resnet_block(input_tensor, width=16):
    x = dens_block(input_tensor, width)
    x = identity_block(x, width)
    x = identity_block(x, width)
    return x

def RegResNet(input_size=8, reps=3, initial_weights=None, width=16, num_gpus=0, lr=1e-4):
    input_layer = layers.Input(shape=(input_size,))
    x = input_layer
    for _ in range(reps):
        x = resnet_block(x, width)

    x = layers.BatchNormalization()(x)
    x = layers.Dense(1, activation=None)(x)

    model = models.Model(inputs=input_layer, outputs=x)
    return model

Which yields a summary I couldn't fit and the following parameter count.
==================================================================================================
Total params: 9,905
Trainable params: 8,913
Non-trainable params: 992
==================================================================================================

Below is my pytorch translation of the same architecture:
class _DenseBlock(nn.Module):
        def __init__(self, input_size, output_size):
            super(_DenseBlock, self).__init__()
            self.input_size = input_size
            self.output_size = output_size
            self.linear = nn.Linear(self.input_size, self.output_size)
            self.bnorm = nn.BatchNorm1d(self.output_size)

    def forward(self, x):
        x = self.linear(x)
        x = self.bnorm(x)
        x = nn.ReLU()(x)
        return x

class DenseBlock(nn.Module):
    def __init__(self, input_size, output_size, repetitions=2):
        super(DenseBlock, self).__init__()
        self.input_size = input_size
        self.output_size = output_size
        self.repetitions = repetitions
        self.dense_blocks = nn.ModuleList([_DenseBlock(self.input_size, self.output_size)]
                      + [_DenseBlock(self.output_size, self.output_size)
                         for _ in range(self.repetitions - 1)])
        self.b_norm1 = nn.BatchNorm1d(self.output_size)
        self.b_norm2 = nn.BatchNorm1d(self.output_size)
        self.linear_1 = nn.Linear(self.output_size, self.output_size)
        self.linear_2 = nn.Linear(self.input_size, self.output_size)

    def forward(self, x):
        identity = x

        for l in self.dense_blocks:
            x = l(x)

        x = self.linear_1(x)
        x = self.b_norm1(x)

        shortcut = self.linear_2(identity)
        shortcut = self.b_norm2(shortcut)

        x = shortcut + x
        x = nn.ReLU()(x)
        return x

class IdentityBlock(nn.Module):
    def __init__(self, input_size, output_size):
        super(IdentityBlock, self).__init__()
        self.input_size = input_size
        self.output_size = output_size
        self.linear_1 = nn.Linear(self.input_size, self.output_size)
        self.linear_2 = nn.Linear(self.input_size, self.output_size)
        self.linear_3 = nn.Linear(self.input_size, self.output_size)
        self.bnorm_1 = nn.BatchNorm1d(self.input_size)
        self.bnorm_2 = nn.BatchNorm1d(self.output_size)
        self.bnorm_3 = nn.BatchNorm1d(self.output_size)

    def forward(self, x):
        input_tensor = x
        x = self.linear_1(x)
        x = self.bnorm_1(x)
        x = nn.ReLU()(x)

        x = self.linear_2(x)
        x = self.bnorm_2(x)
        x = nn.ReLU()(x)

        x = self.linear_3(x)
        x = self.bnorm_3(x)
        x = x + input_tensor
        x = nn.ReLU()(x)
        return x

class ResnetBlock(nn.Module):
    def __init__(self, input_size, output_size):
        super(ResnetBlock, self).__init__()
        self.input_size = input_size
        self.output_size = output_size
        self.dense_block = DenseBlock(self.input_size, self.output_size)
        self.identity_block_1 = IdentityBlock(self.output_size, self.output_size)
        self.identity_block_2 = IdentityBlock(self.output_size, self.output_size)

    def forward(self, x):
        x = self.dense_block(x)
        x = self.identity_block_1(x)
        x = self.identity_block_2(x)
        return x

class RegResNet(nn.Module):

    def __init__(self, input_size, block_width, repititions=3):
        super(RegResNet, self).__init__()
        self.input_size = input_size
        self.repititions = repititions
        self.block_width = block_width
        self.resnet_blocks = nn.ModuleList([ResnetBlock(self.input_size, self.block_width)]
                                           + [ResnetBlock(self.block_width, self.block_width)
                                              for _ in range(self.repititions - 1)])
        self.bnorm = nn.BatchNorm1d(self.block_width)
        self.out = nn.Linear(self.block_width, 1)

    def forward(self, x):
        for layer in self.resnet_blocks:
            x = layer(x)

        x = self.bnorm(x)
        x = self.out(x)
        return x

Using the torchsummary library yields I got the summary (which I couldnt fit) and the following parameter count
================================================================
    Total params: 8,913
    Trainable params: 8,913
    Non-trainable params: 0
================================================================


title: Policy loss in DDPG

body:
I am trying to implement DDPG algorithm.
However I have a query that why actor loss is calculated as negative mean of the model predicted Q values in the states we are in.
Shouldn't it be like the difference of the Q values when random action is taken and Q values of the model predicted actions in that state.
Basically if a random action is generating a better Q value...only then the actor Q value should be updated else we already have the value and no need to propagate actor loss separately..it will be taken care in value loss.
I know this post ought to be in AI forum but stackoverflow seems to provide good and quick responses that is why posting here


title: How to use apply DataFrame method with PyTorch library

body:
There are many useful methods in DataFrame that you would like to use when works with PyTorch library. Ir order to speedup the working with DataFrame.apply(for columns) method I decided to use PyTorch library.
My first impression that I cannot use this library for my task - PyTorch works with data as with usual numpy array.It means that I cannot speedup 'apply' method using PyTorch. I quess that it is true and for other nice DataFrame features. If I'm wrong please explain me how I could solve my problem...


title: Fast save/load list of lists of unsigned integers

body:
I have a list of lists of unsigned integers where every list represent a sentence from a text and every interger represent a unique id associated to a word.
From those "sentences" I need to create (target, context) or (target, [c1, c2, ..., cN]) couples inside a pytorch Dataset.
For now I store the sentences with the pickle module e read all of them when the Dataset instance is created (bad for memory if I have tons of them).
I've also already tried to save and read them sequentially, one "sentence" at a time to the same file (bad due to the continuosly reading from disk).
I've also tried to save them to a txt file and load them one sentence at a time and recreate the list with map(int, line) (but i think is bad due to both the reading from disk and the processing time).
Is there a way to save and read them as efficiently as possible, wrt memory consumption and loading speed?
Or, is there a way to partially load a pickle file, for example loading the first N lines and convert them back to the list object they represent?
Thank you all in advance
Edit
Answering questions from @Jerome:

Do you have Python 2 or Python 3? Python 3
Do you need the file to be portable across multiple machines or OS? No
What do you want precisely load (eg. extract one list or items of all lists or only one item, etc.)? I need to fetch one list at a time, and from it construct a batch of examples of (context, target) words to be fed to a PyTorch model, so I need all items in the list, and I need all of them
How big is N and the average size of each lists? N depends on the window size hyperparameter, it can vary from 2 to 10 or 20. The average size of each lists is approximately of 1000 integers, but even this number depends on multiple factors (words frequency and so on)


title: Inputting data with multiple variable dimensions to an LSTM

body:
My data is of shape batch_size * num_paths * num_edges * emb_dim. Here:

batch_size: refers to batch size (32)
num_paths: refers to the no of paths between two given terms in a dependency tree
num_edges: refers to the no of edges in a specific path
emb_dim: refers to the embedding dimension (300)

Both num_paths and num_edges are variable for every training sample. In other words, for every training sample, there may be variable number of paths, each with variable number of edges.
(Note that data is an n-d list of lists in native Python, since two dimensions (num_paths and num_edges) are variable.)
I want to pass each path of each training instance as input to an LSTM, since a path is a sequence of edges and I want the resultant path representation after passing it through the LSTM. After obtaining the representation for each path in a training example, I want to take the sum of all these representations.
I know I can pack variable number of edges through pack_padded_sequence. But what about variable number of paths? How do I account for that? Is there any way to do it in native Pytorch, without going for messy solutions like iterating through loops?


title: Back propagate through embedding layer

body:
I have a dataset of sentences which belong to 2 categories. I have trained a classifier on this data to classify a given sentence into one of the 2 categories. Next, I want to train a Pytorch generator (encoder-decoder) model to convert a given sentence from class-1 to class-2 using the pre-trained classifier. So basically it’s a style transfer in NLP. Here is a skeleton of my models:
Encoder:
  Embedding layer --> LSTM [outputs= o, h]

Decoder:
  Embedding layer --> LSTM --> Linear --> Relu --> log_softmax [output= probability for each word in vocab]

Classifier:
  Encoder --> Linear layer1 --> Linear2 --> sigmoid [output = class probability]

Generator:
  Encoder --> Decoder --> topK(1) [outputs = token for each word in generated sentence as floats]

What I am trying to do is, train the generator using error signals from pre-trained Classifier model. Whether this model would work or not is a separate question (and I would definitely love to hear feedbacks from more experienced members here). But, the major concern here is that Generator returns an array of sentence tokens (words) as floats which should then be passed to a (freezed) Classifier model which contains Embedding layer as first layer, which only accepts Long datatype. But converting float to long tensor would destroy the gradient history. From what I understand from similar embedding related questions, it’s not possible to retain gradients with type conversion. So what are my options here? Any workarounds? For example, astute readers would have noticed that even “topK/argmax” operation would break the gradients history, for which I am planning to use a Linear layer to find argmax while training. Any similar solutions for “embedding” problem too? I am quite sure people would have tried similar things but I can’t find any resources on seq2seq + classifier.
Note: I am not posting code to keep the post clean for better understanding. If required I can provide related sections.


title: BERT zero layer fixed word embeddings

body:
I want to do an experiment with bert zero-layer vectors (input vectors), which I understand are of dimension 128.
I can not find where I can get a file with the tokens and their vectors.
Is there such a thing?
Is there a file in the Glove/word2vec format but of BERT's input vectors?
I'd appreciate any help with this!!
Thanks,


title: compute the mutual information of two models for evaluation of one dataset

body:
my work :I am evaluating two models (for example vgg16) on imagenet.
my goal :is to compute the mutual information between feature maps of each vgg16.
question:how to do it in python (pytorch, keras) ?
thank you


title: Average pooling by desired output shape

body:
I have a logic question.
I'm trying to create some sort of version of PyTorch's nn.functional.avg_pool function with a slight modification: I want the padding/stride to be chosen dynamically based on the desired size of the output. I have successfully worked this out for a single 1d tensor:
def ave_pool(tnsr, reduction, pad_val=0, dim = 1):

    size = len(tnsr)
    if size % reduction != 0:
        padlen = np.ceil(size / reduction) * reduction - size
        fpad, bpad = np.ceil(padlen / 2), np.floor(padlen/2)
        tnsr = F.pad(tnsr, (int(fpad), int(bpad)), "constant", pad_val)
        size += padlen
    group = int(size / reduction)
    return tnsr.reshape((-1, group)).mean(dim = 1)

This really works great for a single 1d Tensor. However, I want to apply this idea to "batch" of 1d tensors as they pass through my neural network (essentially 2d input at this point).
# This works as expected:
x = torch.tensor([2,3,4,8,9,10]).float()
ave_pool(x, 3)
>>>
tensor([2.5000, 6.0000, 9.5000])

# But what's the logic to up the dimensions to two?
x = torch.tensor([[2,3,4,8,9,10],
                 [6,7,8,9,19,12]]).float()
ave_pool(x, 3)
>>>
tensor([ 0.,  2.,  3.,  4.,  8.,  9., 10.,  0.,  6.,  7.,  8.,  9., 19., 12.])

So, how do I extend the logic of my currently working code to 2d tensors so that the output of the previous code block would be:
tensor([[2.5000, 6.0000, 9.5000],
        [ 6.5000,  8.5000, 15.5000]])

Obviously this question could apply to TensorFlow tensors or NumPy ndarrays in addition to PyTorch Tensors. I work with all three and the logic would probably be about the same.


title: Inconsitencies/Inaccuracies between torchvision.datasets.MNIST and Michael Nielsens neuralnetworksanddeeplearning

body:
I printed the vectorized form of the first training image of the mnist dataset of pytorch and https://github.com/mnielsen/neural-networks-and-deep-learning/tree/master/data .
The difference seems too big for just floating point precision error.
Full Diff of first mnist train image:
https://www.diffchecker.com/6y6YTFiN

Code to reproduce:
import mnist_loader
import numpy
numpy.set_printoptions(formatter={'float': '{: 0.4f}'.format})

d = mnist_loader.load_data()
print numpy.array(d[0][0][0]).reshape(784,1)

pytorch:
import torch
import torchvision
from torchvision import transforms as tf

dtype = torch.float32

transforms = tf.Compose([tf.ToTensor()])

mnist_data = torchvision.datasets.MNIST("./mnist", transform=transforms, download=True)
data_loader = torch.utils.data.DataLoader(mnist_data,
                                          batch_size=500,
                                          shuffle=False,
                                          num_workers=1)

data, target = next(data_loader.__iter__())
print(data[0].reshape(784,1))

Anyone an idea what's causing this ?


title: Retrieving original data from PyTorch nn.Embedding

body:
I'm passing a dataframe with 5 categories (ex. car, bus, ...) into nn.Embedding.
When I do embedding.parameters(), I can see that there are 5tensors but how do I know which index corresponds to the original input (ex. car, bus, ...)?


title: Kaggle CSV files

body:
I have trained a pre-trained Resnet50 classifier on the kaggle images dataset and label them as 0 for No-Covid 1 for Thorax diseases and 2 for covid.
now for the competition, I need to save these labels in the CSV files.
when I run the following code for evaluation of the test images,
# Read test images from a covid folder and perform predictions
total_imgs = 0
covid_positive = 0
for root, dirs, files in os.walk("/content/gdrive/My Drive/dlai3-phase3/VALIDATE/VALIDATE", topdown=False):
    for name in files:
        total_imgs += 1
        #im_covid = Image.open("data/test/covid/03BF7561-A9BA-4C3C-B8A0-D3E585F73F3C.jpeg")
        img = Image.open(os.path.join(root, name))
        img = img.convert('RGB')
        pred, probs = predict_covid(img)
        print(pred)
print("Accuracy on covid positive test set: {}".format(covid_positive/total_imgs))

but it prints the string labels, not the number.
Could anyone please tell me how I can get the numeric labels?


title: multiply many matrices and many vectors pytorch

body:
I am trying to multiply the following:
A batch of matrices N x M x D
A batch of vectors  N x D x 1
To get a result: N x M x 1
as if I were doing N dot products on M x D D x 1.
I cant seem to find the correct function in PyTorch.
torch.bmm as far as I can tell only works for a batch of vectors and a single matrix. If I have to use torch.einsum then so be it but id rather not!


title: Pytorch: computing the eigenvector of data point

body:
Lets say we already have estimated the principal directions and eigenvalues of a dataset X using pytorch:
U, S, V = torch.svd(X)

Now, given a new data point x, I'd like to estimate its eigenvector v representation given the already estimated principal directions:
U*S*v = x

Immedtiatly I'd think one would use torch.solve(U*S, x) for this, but it appears to only support square matrices. As a reminder, U*S is generaly not a square matrix, while x and v are vectors.


title: Is there a built in way to ensure image size match per batch?

body:
I have a torch.utills.data.Dataset that returns images which slightly vary in size.
I produce my training batches using torch.utils.data.DataLoader. Is there an inbuilt way to resize the images such that they can be stacked into a batch on the data loader end?
If there's not, I'll probably write a collate_fn to do it. Or is there a better way?
PS: I don't want to resize my whole dataset to one size. That's why I'm specifically asking about doing it at a batch level.


title: Create a tensor with ones where another tensor has non-zero elements in Pytorch

body:
Say I have a tensor A with any shape. It has a number k of non-zero elements. I want to build another tensor B, with 1s where A is non zero and 0s where A is zero.
For example:
A = [[1,2,0],
     [0,3,0],
     [0,0,5]]

then B will be :
B = [[1,1,0],
     [0,1,0],
     [0,0,1]]

Is there a simple way to implement this in Pytorch?


title: pytorch quantization during training

body: I am fairly new to pytorch and have a question regarding the possibilities and best practices in pytorch. I want to change the data type of the weights during training e.g. from float32 to float16. I already know that I have to readjust than the inputs and output of this layer again. Do you think this is possible in pytorch and if yes is there an elegant way to adapt the datatype than again for the next layer?


title: Why Fixed Point Mutiplication is much slower than the Float Point?

body:
Recently, when benchmarking the "matmul" op in Pytorch, and I realize the fixed point(when the dytpe is int) multiplication is much slower, this looks counter intuitive.
 n = 128 m = 1024 p = 512
 t = []
ring_size=(2 ** 64)
for split in np.arange(2,64,2):
    mm =  m // split
    a = torch.randint(-(ring_size // 2), (ring_size - 1) // 2, (n, mm), dtype=torch.int64)
    b = torch.randint(-(ring_size // 2), (ring_size - 1) // 2, (mm, p), dtype=torch.int64)
    init_time = time.time()
    c  = getattr(torch, 'matmul')(a, b)
    t.append(time.time() - init_time)
plt.plot(t)

The following is the results
n = 128 m = 1024 p = 512
t = []
for split in np.arange(2,64,2):
            mm =  m // split
            a = torch.tensor(np.random.normal(loc=100, scale=100, size=(n,mm)))
            b = torch.tensor(np.random.normal(loc=100, scale=100, size=(mm,p)))
            init_time = time.time()
            c  = getattr(torch, 'matmul')(a, b)
        t.append(time.time() - init_time) plt.plot(t)

The following is the result


title: RNN With or Without Loop : Meaning of each output?

body:
For some tasks I have seen two different ways of using RNNS:

In the First Method, Input [batch_size, input_size, sequence_size] is directly passed and the outputs, hidden are returned by it.
In Second Method, Each timestamp [input_size, sequence_size] of the Input is passed and the outputs, hidden are received on the other end, this process happens in a loop(for i in range(batch_size)).

My main doubt is what's really the difference between these two methods?
To support this, I have written a small piece of code, which might help you understand where I went wrong.
This is my basic RNN Class,
class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers, num_classes, device):
        super(RNN, self).__init__()
        self.num_layers = num_layers
        self.hidden_size = hidden_size
        self.device = device

        #nn.RNN and nn.GRU work same way
        self.rnn = nn.GRU(input_size, hidden_size, num_layers, batch_first = True)
        # input : batch_size, sequence_len, input_size
        self.fc = nn.Linear(hidden_size, num_classes)

    def forward(self,x,  h0 = None):
        if h0 is None:
          h0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size).to(self.device)
        out, ht = self.rnn(x, h0)

        return out, ht

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
rnn = RNN(28, 128, 1 , 10, device)

Below I try to get outputs and hidden directly without looping through each timestamp
#Without Loop
inps = torch.randn((32,100,28))
out_wl,ht_wl = rnn(inps)
print("Output shape : ", out_wl.shape) #Output shape :  torch.Size([32, 100, 128])
print("Hidden shape : ", ht_wl.shape) #Hidden shape :  torch.Size([1, 32, 128])

Here I am trying to send one timestamp(row) at a time and record all of the outputs.
#With Loop
outs = []
hns = []
hn_temp = torch.zeros(1, 1, 128).to(device)
for inp in inps:
  inp = inp.unsqueeze(0)
  out_temp,hn_temp = rnn(inp, hn_temp)
  outs.append(out_temp)
  hns.append(hn_temp)

outs = torch.stack(outs)
hns = torch.stack(hns)

print(outs.shape) #torch.Size([32, 1, 100, 128])
print(hns.shape) #torch.Size([32, 1, 1, 128])

outs = outs.squeeze(1)
hns = hns.squeeze(1)

print(outs.shape) #torch.Size([32, 100, 128])
print(hns.shape) #torch.Size([32, 1, 128])

The issue here is when I do
out_wl == outs

The output tensor is not True everywhere.
This all boils down to few questions,

Both methods I talked about, are they same or not?
If they are same then the resulting output doesn't match, do mind model has not done any backward pass and the same randn input is used in both with and without loop snippets.
Is last value of outs same as hns, does that mean outs is just a collection of hns for all s


title: How much should the GPU is needed?

body:
I am using V100 that has 32GB GRAM and 128GB RAM.
I made a network size of 2.5M weights, which takes 2 x (3,144,288) + 2 x (3,288,576) images as inputs. The batch size is 2. only 2.
This network keeps printing the error message : RuntimeError: CUDA out of memory. Tried to allocate 1.90 GiB (GPU 0; 31.88 GiB total capacity; 24.91 GiB already allocated; 1.28 GiB free; 26.73 GiB reserved in total by PyTorch)
I don't understand why it keeps saying that I'm lack of memory, since I used 15M-weights model with 16x(6,1000,1200) without any problem.
Or is there any way to load the inputs in RAM and model in GRAM but I am not aware of it ? If there is, please tell me how and thank you very much.
My pseudo-code for loading model is :
train_load, val_load = SRD.load_sr_st_dataset(route_img,route_dpt,bs=batch_size)
#print(train_load.dataset.__sizeof__())
sr_stereo = SRD.sr_stereo(max_d=192,bs=batch_size).cuda()

And for loading data is :
def parsedata(loaded_data,split):
    l10,l20,r10,r20,depth = loaded_data['l10'],loaded_data['l20'],loaded_data['r10'],loaded_data['r20'],loaded_data['depth']
    #print('l10',l10.shape)
    l10_v = A.Variable(l10.cuda())
    l20_v = A.Variable(l20.cuda())
    r10_v = A.Variable(r10.cuda())
    r20_v = A.Variable(r20.cuda())
    depth_v = A.Variable(depth.cuda())
    return l10_v,l20_v,r10_v,r20_v,depth_v


title: Assign loss calculated outside of graph to Pytorch model

body: I need to find a way to use a non-pytorch loss function on a pytorch model.  Short of rewriting the pytorch loss system, is there a way to assign the calculated loss to the model graph so it may be used with .backwards? Would it be feasible to create an artificial computational graph?


title: How to sort a 3d tensor by coordinates in last dimension (pytorch)

body:
I have a tensor with shape [bn, k, 2]. The last dimension are coordinates and I want each batch to be sorted independently depending on the y coordinate ([:, :, 0]). My approach looks something like this:
import torch
a = torch.randn(2, 5, 2)
indices = a[:, :, 0].sort()[1]
a_sorted = a[:, indices]

print(a)
print(a_sorted)

So far so good, but I now it sorts both batches according to both index lists, so I get 4 batches in total:
a
tensor([[[ 0.5160,  0.3257],
         [-1.2410, -0.8361],
         [ 1.3826, -1.1308],
         [ 0.0338,  0.1665],
         [-0.9375, -0.3081]],

        [[ 0.4140, -1.0962],
         [ 0.9847, -0.7231],
         [-0.0110,  0.6437],
         [-0.4914,  0.2473],
         [-0.0938, -0.0722]]])

a_sorted
tensor([[[[-1.2410, -0.8361],
          [-0.9375, -0.3081],
          [ 0.0338,  0.1665],
          [ 0.5160,  0.3257],
          [ 1.3826, -1.1308]],

         [[ 0.0338,  0.1665],
          [-0.9375, -0.3081],
          [ 1.3826, -1.1308],
          [ 0.5160,  0.3257],
          [-1.2410, -0.8361]]],

        [[[ 0.9847, -0.7231],
          [-0.0938, -0.0722],
          [-0.4914,  0.2473],
          [ 0.4140, -1.0962],
          [-0.0110,  0.6437]],

         [[-0.4914,  0.2473],
          [-0.0938, -0.0722],
          [-0.0110,  0.6437],
          [ 0.4140, -1.0962],
          [ 0.9847, -0.7231]]]])

As you can see, I want only the 1st and the 4th batch to be returned. How do I do that?


title: Loss oscillate instead of decreasing seq2seq gru pytorch

body:
I'm trying to build seq2seq model with attention on Pytorch gru for sentiment extraction.
Example of sentiment extraction:
Input1: 'sooo sad i will miss you here in san diego!!!'
Target1: 'sooo sad'
Input2: 'my boss is bullying me...'
Target2: 'bullying me'
My model:
class Encoder(nn.Module):
    def __init__(self, vocab_size, embedding_size, hidden_size, dropout_odds=0.2):
        super(Encoder, self).__init__()
        self.dropout = nn.Dropout(dropout_odds)
        self.embedding = nn.Embedding(vocab_size, embedding_size)
        self.gru = nn.GRU(embedding_size, hidden_size, bidirectional=True)
        self.fc_hidden = nn.Linear(hidden_size*2, hidden_size)

    def forward(self, x):
        embedded = self.dropout(self.embedding(x)).view(1, 1, -1)
        output, hidden = self.gru(embedded)
        hidden = self.fc_hidden(torch.cat((hidden[0:1],hidden[1:2]),dim=2))

        return output, hidden

class Decoder(nn.Module):
    def __init__(self, vocab_size, embedding_size, hidden_size, dropout_odds=0.2):
        super(Decoder, self).__init__()

        self.dropout = nn.Dropout(dropout_odds)
        self.embedding = nn.Embedding(vocab_size, embedding_size)
        self.energy = nn.Linear(hidden_size*3, 1)
        self.gru = nn.GRU(hidden_size*2+embedding_size, hidden_size)
        self.out = nn.Linear(hidden_size, vocab_size)
        self.relu = nn.ReLU()
        self.softmax_en = nn.Softmax(dim=0)
        self.softmax_out = nn.Softmax(dim=2)

    def forward(self, x, hidden, encoder_outputs):
        embedded = self.dropout(self.embedding(x)).view(1, 1, -1)
        sequence_length = encoder_outputs.shape[0]
        h_reshaped = hidden[0].repeat(sequence_length, 1)
        energy_ = self.relu(self.energy(torch.cat((h_reshaped, encoder_outputs), dim=1)))
        attention = self.softmax_en(energy_)
        context_vector = torch.einsum("st,sh->th", attention, encoder_outputs)
        output = torch.cat((context_vector.unsqueeze(0), embedded),dim=2)
        output, hidden = self.gru(output, hidden)
        output = self.out(output)
        return output, hidden

class Seq2seq(nn.Module):
    def __init__(self, encoder, decoder):
        super(Seq2seq, self).__init__()
        self.encoder = encoder
        self.decoder = decoder

    def forward(self, source, target, teacher_force_ratio=0.5):
        sourse_len = source.shape[0]
        target_len = target.shape[0]
        target_vocab_size = vocab.n_words
        encoder_outputs =  torch.zeros(sourse_len, hidden_size*2, device=device)
        decoder_outputs = torch.zeros(target_len, 1, target_vocab_size, device=device)
        for ei in range(sourse_len):
            encoder_states, hidden = self.encoder(source[ei])
            encoder_outputs[ei] = encoder_states
        x = target[0]

        for t in range(1,target_len):
            output, hidden = self.decoder(x, hidden, encoder_outputs)
            decoder_outputs[t] = output
            best_guess = output.argmax()
            x = best_guess

        return decoder_outputs

hidden_size = 1024
embedding_size = 1024
epochs=100
encoder1 = Encoder(vocab.n_words, embedding_size, hidden_size).to(device)
attn_decoder1 = Decoder(vocab.n_words, embedding_size, hidden_size).to(device)
model = Seq2seq(encoder1,attn_decoder1).to(device)
# optimizer = optim.SGD(model.parameters(),lr=0.05)
optimizer = optim.Adam(model.parameters(), lr=0.1)
criterion = nn.CrossEntropyLoss()
training_pairs = [tensorsFromPair(pair) for pair in pairs]

model.train()
for epoch in range(1,epochs+1):
    print(f"[Epoch {epoch} / {epochs}]")

    indx = 0
    for tp in training_pairs[:100]:

        inp_data = tp[0]
        target = tp[1]

        output = model(inp_data, target)
        optimizer.zero_grad()
        loss = criterion(output.squeeze(1), target)
        loss.backward()
        optimizer.step()
        indx+=1
        if indx%100==0:
            print('loss: ',loss)

training pairs are list of tensors, each tensor contains a pair of tensors each element are index of word in vocab:
training_pairs[0] = (tensor([18, 19, 20, 21, 22,  1]), tensor([21, 22,  1])), where 1 EOS_token

For some reason when the model learns loss does not decrease, but oscillates:
[Epoch 1 / 100]
loss:  tensor(974.1525, device='cuda:0', grad_fn=<NllLossBackward>)
[Epoch 2 / 100]
loss:  tensor(5.4103, device='cuda:0', grad_fn=<NllLossBackward>)
[Epoch 3 / 100]
loss:  tensor(3404.2073, device='cuda:0', grad_fn=<NllLossBackward>)
[Epoch 4 / 100]
loss:  tensor(5.4103, device='cuda:0', grad_fn=<NllLossBackward>)
[Epoch 5 / 100]
loss:  tensor(2543.9885, device='cuda:0', grad_fn=<NllLossBackward>)
[Epoch 6 / 100]
loss:  tensor(28.9650, device='cuda:0', grad_fn=<NllLossBackward>)
[Epoch 7 / 100]
loss:  tensor(2998.9417, device='cuda:0', grad_fn=<NllLossBackward>)
[Epoch 8 / 100]
loss:  tensor(5.4103, device='cuda:0', grad_fn=<NllLossBackward>)
[Epoch 9 / 100]
loss:  tensor(5.4103, device='cuda:0', grad_fn=<NllLossBackward>)
[Epoch 10 / 100]
loss:  tensor(5.4103, device='cuda:0', grad_fn=<NllLossBackward>)

The situation repeats if you change input for encoder for SOS token.
I have also tried to calculate loss without an EOS token it does not change anything.
I can`t understand why the loss is calculated incorrectly.


title: PyTorch wheel not supported on this platform error while deploying application

body:
I am trying to deploy my pytorch application on zeet and my requirments.txt file contains this-
https://download.pytorch.org/whl/cpu/torch-1.3.1%2Bcpu-cp36-cp36m-linux_x86_64.whl
pickle-mixin
flask
simpletransformers
selenium
gunicorn
cloudpickle

When I try to build my application, I get the following error :
ERROR: torch-1.3.1+cpu-cp36-cp36m-linux_x86_64.whl is not a supported wheel on this platform.
Build failed

I tried using multiple different versions of the wheel, but none to be working, I even tried the windows build but that isnt working too and I tried all the solutions I could find online but none seem to be working.
EDIT : My build command is pip install -r requirements.txt and my python version is python 3.8
Any Help Would Be Appreciated!!


title: Tensorboard: All experiments were written as one (without provided tags)

body:
I wanted to compare several runs that I did in the loop creating new SummaryWriter instances like this:
for experiment_name in experiments:
    logger = SummaryWriter(self._log_path, comment=experiment_name)
    ...
    for epoch in range(5):
        ...
        logger.add_scalar("Epoch Loss", loss, epoch)
        ...
    logger.close()

In the log path I got several files like this:
events.out.tfevents.1609675249.nlp-vm.13735.0
events.out.tfevents.1609679736.nlp-vm.13735.1
events.out.tfevents.1609687200.nlp-vm.13735.2
events.out.tfevents.1609691662.nlp-vm.13735.3
events.out.tfevents.1609699158.nlp-vm.13735.4
events.out.tfevents.1609703743.nlp-vm.13735.5
events.out.tfevents.1609711308.nlp-vm.13735.6
events.out.tfevents.1609716054.nlp-vm.13735.7

But the Tensorboard displays all runs as one:

example

example

Could you say what I should do to fix it and can I do it without re-running all experiments?