title: VGG16 Model Outputs Incorrect dimension - Transfer Learning

body:
I am trying to use a pre-trained VGG16 model to classify CIFAR10 on pyTorch. The model was originally trained on ImageNet.
Here is how I imported and modified the model:
from torchvision import models

model = models.vgg16(pretrained=True).cuda()
model.classifier[6].out_features = 10

and this is the summary of the model
print(model)

VGG(
  (features): Sequential(
    (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (1): ReLU(inplace=True)
    (2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (3): ReLU(inplace=True)
    (4): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (5): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (6): ReLU(inplace=True)
    (7): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (8): ReLU(inplace=True)
    (9): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (10): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (11): ReLU(inplace=True)
    (12): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (13): ReLU(inplace=True)
    (14): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (15): ReLU(inplace=True)
    (16): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (17): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (18): ReLU(inplace=True)
    (19): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (20): ReLU(inplace=True)
    (21): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (22): ReLU(inplace=True)
    (23): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (24): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (25): ReLU(inplace=True)
    (26): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (27): ReLU(inplace=True)
    (28): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (29): ReLU(inplace=True)
    (30): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  )
  (avgpool): AdaptiveAvgPool2d(output_size=(7, 7))
  (classifier): Sequential(
    (0): Linear(in_features=25088, out_features=4096, bias=True)
    (1): ReLU(inplace=True)
    (2): Dropout(p=0.5, inplace=False)
    (3): Linear(in_features=4096, out_features=4096, bias=True)
    (4): ReLU(inplace=True)
    (5): Dropout(p=0.5, inplace=False)
    (6): Linear(in_features=4096, out_features=10, bias=True)
  )
)

Now I wanted to train the model on CIFAR10, I created the train loader with batch size = 128.
Below is the training function and I added some printing statements to check if everything is working fine or not. The problem is the model outputs for each data point a 1000 prediction (as the original - unmodified version).
def train(model, optimizer, train_loader, epoch=5):
    """
    This function updates/trains client model on client data
    """
    model.train()
    for e in range(epoch):
        for batch_idx, (data, target) in enumerate(train_loader):
            data, target = data.to(device), target.to(device)
            optimizer.zero_grad()
            output = model(data)
            print("shape output: ", output.shape)
            probs = F.softmax(output, dim=1) #get the entropy
            print("entropy: ", probs)
            _, predicted = torch.max(output.data, 1)
            loss = criterion(output, target)
            loss.backward()
            if batch_idx % 100 == 0:    # print every 100 mini-batches
                print('[%d, %5d] loss: %.3f' %
                  (e + 1, batch_idx + 1, loss.item()))
            optimizer.step()
    return loss.item()

Here is part of the the output which shows the output shape:
shape output:  torch.Size([128, 1000])

I don't understand why the model outputs the results in this way. Is there something that I am missing?


title: Distributed evaluation of validation dataset in pytorch

body: I'm using some ready-made scripts for distributed training of my model, and not understanding very well the mechanics behind it. Basically, it uses torch.distributed and the master script spawns multiple processes, but updating happens in the same model (script). However, validation on the validation dataset is done separately on the different processes, and I don't get an "overall" validation loss (see here). How can I combine the validation results from the different processes to get an overall result?


title: How do I resolve python build error?(pytorch ATSS model)

body:
source code : https://github.com/sfzhang15/ATSS
I want to use it in pytorch 1.7 version. But I got the following error.
My environment is nvidia docker 20.09.
ubuntu :18.04
python : 3.6
pytorch : 1.7
torchvision : 0.4
cuda : 11.0
Please help me
Traceback (most recent call last):
  File "setup.py", line 76, in <module>
    include_package_data=True,
  File "/opt/conda/lib/python3.6/site-packages/setuptools/__init__.py", line 163, in setup
    return distutils.core.setup(**attrs)
  File "/opt/conda/lib/python3.6/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/opt/conda/lib/python3.6/distutils/dist.py", line 955, in run_commands
    self.run_command(cmd)
  File "/opt/conda/lib/python3.6/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/opt/conda/lib/python3.6/distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
  File "/opt/conda/lib/python3.6/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/opt/conda/lib/python3.6/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/opt/conda/lib/python3.6/site-packages/setuptools/command/build_ext.py", line 87, in run
    _build_ext.run(self)
  File "/opt/conda/lib/python3.6/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run
    _build_ext.build_ext.run(self)enter code here
  File "/opt/conda/lib/python3.6/distutils/command/build_ext.py", line 339, in run
    self.build_extensions()
  File "/opt/conda/lib/python3.6/site-packages/torch/utils/cpp_extension.py", line 649, in build_extensions
    build_ext.build_extensions(self)
  File "/opt/conda/lib/python3.6/site-packages/Cython/Distutils/old_build_ext.py", line 194, in build_extensions
    self.build_extension(ext)
  File "/opt/conda/lib/python3.6/site-packages/setuptools/command/build_ext.py", line 208, in build_extension
    _build_ext.build_extension(self, ext)
  File "/opt/conda/lib/python3.6/distutils/command/build_ext.py", line 533, in build_extension
    depends=ext.depends)
  File "/opt/conda/lib/python3.6/site-packages/torch/utils/cpp_extension.py", line 478, in unix_wrap_ninja_compile
    with_cuda=with_cuda)
  File "/opt/conda/lib/python3.6/site-packages/torch/utils/cpp_extension.py", line 1233, in _write_ninja_file_and_compile_objects
    error_prefix='Error compiling objects for extension')
  File "/opt/conda/lib/python3.6/site-packages/torch/utils/cpp_extension.py", line 1529, in _run_ninja_build
    raise RuntimeError(message)
RuntimeError: Error compiling objects for extension


title: Using pytorch Autograd on both images axis

body:
I would like to use autodiff since I have heard many great stories about it. However, I want to check if
it is even possible what I want.
Given a function that returns an image X (generated by one single input image).. I want to compute the
x- and y-directional derivative. Normally I would use a kernel method, but I want to see if it is possible with this autodiff.
I understand that autodif is based on the Jacobian, and since I only have one input for my function.. it will only calculate
the derivative based on that variable. Hence, I expect that it will not be able to do these directional derivative.
Is this true? Or is it possible in another way?


title: Google Colab -- MLP execution time not constant, but decreasing?

body:
I am currently working a lot with Google Colab, and wanted to time the execution time of a small MLP for inference. However, when executing the below code in Google Colab, the reported runtime decreases with the number of executions of the notebook (i.e., repeatedly pressing the play-button after the notebook terminates).
I am getting values around
1st execution: 0.03   s
2nd execution: 0.005  s
3rd execution: 0.0007 s

tested on different machines and with different browsers. Note that I'm aware that time.time() has a precision limit of 1ms on Unix systems, however, this does not explain the behaviour.
Is there some sort of caching going on on the GPU / in PyTorch? If so, why, and can I expect such a speed increase in the final, deployed application as well?
Code to replicate the behaviour:
import time
import torch
import torch.nn.functional as F

class Model(torch.nn.Module):
  def __init__(self):
    super(Model, self).__init__()

    self.fc1 = torch.nn.Linear(in_features=3, out_features=512)
    self.fc2 = torch.nn.Linear(in_features=512, out_features=512)
    self.fc3 = torch.nn.Linear(in_features=512, out_features=512)
    self.fc4 = torch.nn.Linear(in_features=512, out_features=512)
    self.fc5 = torch.nn.Linear(in_features=512, out_features=512)
    self.fc6 = torch.nn.Linear(in_features=512, out_features=512)
    self.fc7 = torch.nn.Linear(in_features=512, out_features=1)

  def forward(self, x):
    in_dim = x.shape[0]
    x = F.relu(self.fc1(x.reshape(in_dim**2, 3)))
    x = F.relu(self.fc2(x))
    x = F.relu(self.fc3(x))
    x = F.relu(self.fc4(x))
    x = F.relu(self.fc5(x))
    x = F.relu(self.fc6(x))
    x = F.relu(self.fc7(x)).reshape((in_dim, in_dim, 1))
    return x

device = 'cuda'
mlp = Model().to(device)

dim = 1024
input_tensor = torch.rand((dim, dim, 3), device=device)

with torch.no_grad():
  start_time = time.time()
  out = mlp(input_tensor)
  end_time = time.time()
  print("Model FW pass {}p: {} seconds".format(input_tensor.shape[0],
                                               end_time-start_time))


title: how to access value of a learned parameter of an activation function in a sequential network

body:
I am implementing my custom activation function with learnable parameters. For example this can be similar to PReLu https://pytorch.org/docs/stable/generated/torch.nn.PReLU.html which a learnable parameter a.
How can I access/view the a parameter value after training ?
Solution the issue was related to the fact that it was a sequential network and this helped solve the issue https://discuss.pytorch.org/t/access-weights-of-a-specific-module-in-nn-sequential/3627


title: Why does the gradient free SVGD algorithm not converge?

body:
I am trying to implement the following algorithm given in this paper:

Here is my code:
import numpy as np
import torch
import sklearn
from torch.autograd import Variable
from torch.optim import RMSprop, Adadelta, Adam, Adagrad, SGD
from torch.distributions import Normal, Categorical, Poisson, Bernoulli, ContinuousBernoulli, Uniform
from torch.distributions.mixture_same_family import MixtureSameFamily
from torch.distributions import MultivariateNormal
from scipy.stats import multivariate_normal

def get_median(tensor):
    """
    torch.median works differently than np.median
    see https://stackoverflow.com/questions/54310861/pytorch-median-is-it-bug-or-am-i-using-it-wrong
    """
    tensor = tensor.detach().flatten()
    tensor_max = torch.tensor([tensor.max()])
    return (torch.cat((tensor, tensor_max)).median() + tensor.median()) / 2.

def kernel_rbf(x, y, h=-1):
    """
    returns the RBF kernel matrix k(x, y) = exp(-(1/h^2)|x-y|^2)
    x: [num_samples, dimension]
    y: [num_samples, dimension]
    h: bandwidth parameter
    """
    if x.dim() == 1:
        x = x.unsqueeze(1)
    if y.dim() == 1:
        y = y.unsqueeze(1)
    n = x.size()[0]
    x_i = x.unsqueeze(1)
    y_j = y.unsqueeze(0)
    dnorm2 = ((x_i-y_j)**2).sum(2)
    if h < 0:
        h = get_median(dnorm2)
        h = torch.sqrt(h**2/(2*np.log(n+1)))

    kernel_matrix = torch.exp(-dnorm2 / h)
    return kernel_matrix

def get_gf_optimal_direction(target_model, surrogate_model, inputs, h=-1, debug=False):
    """
    returns phi(x) the optimal direction of perturbation for gradient descent
    model: target model distribution (torch object)
    inputs: [num_samples, dimension]
    """

    n = inputs.shape[0]
    inputs = inputs.detach().requires_grad_(True)

    # compute the score function of the surrogate model
    log_prob = surrogate_model.log_prob(inputs)
    log_prob_grad = torch.autograd.grad(log_prob.sum(), inputs)[0]

    # compute the gradient of the kernel
    kernel = kernel_rbf(inputs, inputs, h=h)
    kernel_grad = -0.5 * torch.autograd.grad(kernel.sum(), inputs)[0]

    # compute weights
    dummy_input = inputs.clone().detach().squeeze().numpy()

    # evaluate rho(x)
    surrogate_temp = multivariate_normal(mean=surrogate_model.mean.numpy(), cov=surrogate_model.variance.numpy())
    weights_top = torch.tensor(surrogate_temp.pdf(dummy_input))
    # evaluate p(x)
    target_temp = multivariate_normal(mean=target_model.mean.numpy(), cov=target_model.variance.numpy())
    weights_bottom = torch.tensor(target_temp.pdf(dummy_input))

    weights = torch.unsqueeze(torch.div(weights_top, weights_bottom), 1)
    weight_sum = weights.sum()

    # optimal pertubation direction for discrete distribution
    phi_x = (weights/weight_sum) * (kernel.mm(log_prob_grad) + kernel_grad)

    return phi_x

def update_gf_Adam(target_model, surrogate_model, x0, n_iter=10):
    optimizer = Adam([x0], lr=0.0001)
    for iter in range(1, n_iter + 1):
        optimizer.zero_grad()
        x0.grad = (-1.) * get_gf_optimal_direction(target_model, surrogate_model,
                                                   x0, h=-1, debug=True)
        optimizer.step()
    return x0

# target distribution
d = 2
number_of_particles = 5

mu = torch.tensor(np.zeros(d))
A = torch.tensor(np.eye(d))
target_model = MultivariateNormal(mu, 2 * A)

# surrogate distribution
surrogate_model = MultivariateNormal(mu + 5, 6 * A)

# initial distribution
initial_model = MultivariateNormal(mu + 2, 10*A)
x0 = initial_model.sample([number_of_particles])

theta = x0.clone()
theta = update_gf_Adam(target_model, surrogate_model, theta)

expected_mean = torch.mean(theta, dim=0)
expected_covariance = get_covariance(theta)

print("expected mean = {}\n expected covariance = {}".format(expected_mean, expected_covariance))
print("actual mean = {}\n actual covariance = {}".format(target_model.mean, target_model.covariance_matrix))

I am trying to make the algorithm work for the following simple case:

The target distribution $p$ is the two-dimensional multivariate normal distribution, $p(x)=\mathcal{N}(x;\mu, 2*A)$ where the mean $\mu=(0,0)$ and
the matrix $A=\operatorname{Id}.$
The surrogate density function $\rho(x) = \mathcal{N}(x;\mu+5, 6*A).$
The initial particles ${x_i^0}_{i=1}^{n}$ are initialized from distribution $\mathcal{N}(x;\mu+2, 10*A).$
The kernel $k$ is chosen to be the RBF kernel with the form $k(x,x')=\exp(-\frac{1}{h^2}||x-x'||)$ where $h$ is set according the median heuristic, i.e. $h=\operatorname{med}^2/(2\log(n+1))$ where $n$ is the number of particles.

The update rule is implemented in the function get_gf_optimal_direction, and I think I have computed the quantities correctly in PyTorch, however, the algorithm does not converge. Any advice/suggestions will be much appreciated.


title: Difference between Library and framework in deep learning

body: I am having a big time difficulty understanding whether TensorFlow/Pytorch is a framework or a library, because they say its an AI/ML opensource framework but in some of the blogs I read it is a library. Could someone please help me with this and also what is the basic difference between framework and library?


title: How to calculate perplexity on very long documents in pytorch?

body:
I have a code snippet working perfectly on short sentences (<1000 tokens). But I wonder how to turn this snippet to support longer sentences. I was thinking about batch processing, however it is not a classification task, and there are no labels for the sentences. Thus most online posts using torchtext with don't work for my case.
Is there any efficient way to calculate overall perplexity for document which has very long sentences?
Here is my code snippet:
encodings = tokenizer(input_text, return_tensors="pt",
                      return_attention_mask=True,
                      add_special_tokens=True,
                      truncation=True,
                      max_length=1024)
input_ids = encodings["input_ids"]
att_mask = encodings["attention_mask"]
input_ids = input_ids.to(DEVICE)
att_mask = att_mask.to(DEVICE)
outputs = model(input_ids, attention_mask=att_mask, labels=input_ids)
perp = math.exp(outputs[0].item())

where input_text is the document with longer sentences, tokenizer and model are instances from huggingface's transformer packages, i.e., BertTokenizer, BertModel.
Thanks!


title: Slow inference for HelsinkiNLP translation model

body: I'm trying to use Helsenki translation model (en->de and de->en) for back-translation implementation. However, when I tested it on VM for a short paragraph, it took me around 23 seconds to translate. I'm not sure if it's supposed to take that long or I'm doing something wrong here. Given a big dataset, how can I translate all of them. THank you!


title: Broadcasting tuple of pytroch tensors for all process

body:
Usually we know how to broadcast a tensor for all process, but when i get a return from PrecisionRecallCurve we get a tuple of numpy arrays. I change their types to torch.Tensor then when i try to broadcast them this doesn't go well, How can i do that?
the error i get when i try to do that
def broadcast(tensor,
                  src,
                  group=group.WORLD,
                  async_op=False):
        """
        Broadcasts the tensor to the whole group.

        ``tensor`` must have the same number of elements in all processes
        participating in the collective.

        Arguments:
            tensor (Tensor): Data to be sent if ``src`` is the rank of current
                process, and tensor to be used to save received data otherwise.
            src (int): Source rank.
            group (ProcessGroup, optional): The process group to work on
            async_op (bool, optional): Whether this op should be an async op

        Returns:
            Async work handle, if async_op is set to True.
            None, if not async_op or if not part of the group

        """
        _check_single_tensor(tensor, "tensor")
        if _rank_not_in_group(group):
            return

        opts = BroadcastOptions()
        opts.rootRank = src
        opts.rootTensor = 0

        if group == GroupMember.WORLD:
            _check_default_pg()
            work = _default_pg.broadcast([tensor], opts)
        else:
            group_src_rank = _get_group_rank(group, src)
            opts.rootRank = group_src_rank
            work = group.broadcast([tensor], opts)
        if async_op:
            return work
        else:
>           work.wait()
E           RuntimeError: [/pytorch/third_party/gloo/gloo/transport/tcp/pair.cc:566] Read error [127.0.0.1]:60647: Connection reset by peer

Solutions i have tried:

Obviously it doesn't work if i tried to broadcast the tuple.
I tried to loop through the tuple and broadcast tensor by tensor but also this didn't work and the Error shown above is related to this approach

for tens in tensors:
          tens = idist.broadcast(tens, src=0)

It works fine for the first tensor, then it breaks in the next one.


title: Pytorch training loop for pix2code autoencoder model

body:
I am trying to implement the pix2code Paper according to this blogpost using Pytorch instead of Tensorflow. I want to focus only on the HTML version from the post.
In short terms: a pretrained cnn is used to extract a featuremap from the screenshot. Then the feature map is given to an encoder. Another encoder is used to extract features from an given sequence, that is already preprocessed (tokenized and numericalized text). The features extracted by the cnn and rnn will be concatenated and given to an LSTM Decoder, which then generates HTML token by token.
I already build the model and everything else, but now I am stuck at implementing the training loop. Thanks to the tensorflow fit function, which is mysterious to me.
I would really appreciate if somebody out there could give me some hints!! I am wondering about how to init and use the hidden states. If I am missing some important information just let me know and I will provide.
class VM(nn.Module):
    def __init__(self, nfeat = 128):
        super(VM, self).__init__()
        # Linear input size of feature map extracted by resnet
        self.fc = nn.Linear(256*14*14, nfeat)

    def forward(self, images):
        out = images.reshape(images.size(0), -1)
        out = self.fc(out)
        out = F.relu(out)
        return out

    class LM(nn.Module):
    def __init__(self, ntoken, ninp=200, nhidden = 256, nfeat = 128, nlayers=2):
        super(LM, self).__init__()
        self.nlayers = nlayers
        self.nhid = nhidden

        self.encoder = nn.Embedding(ntoken, ninp)
        self.lstm = nn.LSTM(ninp, nhidden, nlayers)
        self.decoder = nn.Linear(nhidden, nfeat)

    def forward(self, seq, hidden):
        embed = self.encoder(seq)
        out, hidden = self.lstm(embed, hidden)
        out = self.decoder(out)
        out = F.relu(out)
        return out, hidden

    def init_hidden(self, bsz):
        weight = next(self.parameters())
        return (weight.new_zeros(self.nlayers, bsz, self.nhid), weight.new_zeros(self.nlayers, bsz, self.nhid))

class Decoder(nn.Module):
    def __init__(self, ntoken, ninp, nhidden=512, nlayers=1):
        super(Decoder, self).__init__()
        self.nlayers = nlayers
        self.nhid = nhidden

        self.lstm = nn.LSTM(ninp, nhidden, nlayers)
        self.fc = nn.Linear(nhidden, ntoken)

    def forward(self, x, hidden):
        out, hidden = self.lstm(x, hidden)
        out = self.fc(out[:, -1, :])
        return F.log_softmax(out, dim=1), hidden

    def init_hidden(self, bsz):
        weight = next(self.parameters())
        return (weight.new_zeros(self.nlayers, bsz, self.nhid), weight.new_zeros(self.nlayers, bsz, self.nhid))

    class Pix2Code(nn.Module):
    def __init__(self, ntoken): # , ninp, nhidden=512, nlayers=1
        super(Pix2Code, self).__init__()
        self.vm = VM(nfeat=128)
        self.lm = LM(ntoken=ntoken, ninp=200, nhidden = 256, nfeat = 128, nlayers=2)
        self.decoder = Decoder(ntoken=ntoken, ninp=256)

    def forward(self, imgs, seqs, hidden):
        img_feats = vm(imgs)
        seq_feats = lm(seqs)
        features = torch.cat((vm_out, lm_out), 2)
        return decoder(features), hidden

    def init_hidden(self, bsz):
        weight = next(self.parameters())
        return (weight.new_zeros(self.nlayers, bsz, self.nhid), weight.new_zeros(self.nlayers, bsz, self.nhid))


title: Converting 1D tensor into a 1D array using Fastai

body:
I have a question, about hoy to convert a 1D Tensor to a 1D Array, using Fastai.
For example, I have this tensor:
tensor([2.2097e-05, 2.6679e-04, 4.6098e-05, 5.5458e-01, 4.4509e-01])


title: understanding the Average Precision& Recall for Pytorch object detection model

body:
I was going through the PyTorch object detection documentation tutorial pytorch object-detection.
While training the model, we are evaluating the Average Precision & Recall at different IoU(Intersection over Union).
Since we are evaluating AP/AR at different IoU. Which IOU (any specific IOU) we should focus on to validate that the model is learning and not going through overfitting or some other issue?
Does each IoU has specific significance and need to be addressed as per the requirement?
I have attached the below screenshot for reference.


title: Torch select index for mult dimensional vector

body:
Suppose I have the following data:
import torch
torch.manual_seed(42)
logits = torch.randn(2, 5, 3)
idx = torch.randint(0, 3, (2, 5))

What I want to do is the following:
[[logits[i,j,idx[i][j]] for j in range(len(idx[i]))] for i in range(len(idx))]

However, this is clearly inefficient.
The closest that I came to this was doing this, but this again looks pretty ugly:
new_idx = torch.stack([idx]*logits.shape[-1], dim=-1)
logits.gather(dim=-1, index=new_idx)

where in the above case the desired output is replicated 3 times.
In terms of practical use case, I'm thinking of a language model where the logits are of shape (batch_size, sequence_len, vocabulary), and the indices are simply (batch_size, sequence_len). Apologies if this has been asked before but I couldn't find anything.


title: Fill tensor with another tensor where mask is true

body:
I need to insert elements of tensor new into a tensor old with a certain probability, let's say that it is 0.8 for simplicity.
Substantially this is what masked_fill would do, but it only works with monodimensional tensor.
Actually I am doing
    prob = torch.rand(trgs.shape, dtype=torch.float32).to(trgs.device)
    mask = prob < 0.8

    dim1, dim2, dim3, dim4 = new.shape
    for a in range(dim1):
        for b in range(dim2):
            for c in range(dim3):
                for d in range(dim4):
                    old[a][b][c][d] = old[a][b][c][d] if mask[a][b][c][d] else new[a][b][c][d]

which is awful. I would like something like
    prob = torch.rand(trgs.shape, dtype=torch.float32).to(trgs.device)
    mask = prob < 0.8

    old = trgs.multidimensional_masked_fill(mask, new)


title: Pytorch add hyperparameters for 3x3,32 conv2d layer and 2x2 maxpool layer

body:
I am trying to create a conv2d layer below using pytorch. The hyperparameters are given in the image below. I am unsure how to implement the hyperparameters (3x3,32) for the first conv2d layer. I want to know how to use this using torch.nn.Conv2d.
Thank you very much.
Conv2d with hyperparameters


title: Why do "score.backward()" return a gradient tensor with all zeros?

body:
When I tried to modify a CNN visualization method named gradient CAM to work for my yolo v3 model, I met a problem that the gradient calculated contains all zeros which cannot be used for following process after calling score.backward(). The score is the bounding box's confidence regarding to that class and this value has been processed by NMS.
Any hint or help would be appreciated!
class GradCAM(object):
    """
    1: the network does not update gradient, input requires the update
    2: use targeted class's score to do backward propagation
    """

    def __init__(self, net, layer_name):
        self.net = net
        self.layer_name = layer_name
        self.feature = None
        self.gradient = None
        self.net.eval()
        self.handlers = []
        self._register_hook()

    def _get_features_hook(self, module, input, output):
        self.feature = output
        print("feature shape:{}".format(output.size()))

    def _get_grads_hook(self, module, input_grad, output_grad):
        """
        :param input_grad: tuple, input_grad[0]: None
                                   input_grad[1]: weight
                                   input_grad[2]: bias
        :param output_grad:tuple,len = 1
        :return:
        """
        #print(input_grad)
        #print(output_grad)
        self.gradient = output_grad[0]

    def _register_hook(self):
        for i, module in enumerate(self.net.module_list):
            if module == self.layer_name:
                self.handlers.append(module.register_forward_hook(self._get_features_hook))
                self.handlers.append(module.register_backward_hook(self._get_grads_hook))

    def remove_handlers(self):
        for handle in self.handlers:
            handle.remove()

    def __call__(self, inputs, index=0):
        """
        :param inputs: {"image": [C,H,W], "height": height, "width": width}
        :param index: which bbx
        :return:
        """
        self.net.zero_grad()
        output = self.net(inputs['image'])[0]
        output_nonmax = utils.non_max_suppression(output, conf_thres=0.25, iou_thres=0.45, multi_label=True)[0]
        print(output_nonmax)
        score = output_nonmax[index][4]
        print(score)
        score.retain_grad()
        #score = output[0]['instances'].scores[index]
        #proposal_idx = output[0]['instances'].indices[index]
        proposal_idx = index
        **score.backward()**
        print(score.is_leaf)

        print(score.grad)
        gradient = self.gradient[proposal_idx].cpu().data.numpy()  # [C,H,W]
        #print(gradient)
        weight = np.mean(gradient, axis=(1, 2))  # [C]
        #print(weight)
        feature = self.feature[proposal_idx].cpu().data.numpy()  # [C,H,W]

        cam = feature * weight[:, np.newaxis, np.newaxis]  # [C,H,W]
        cam = np.sum(cam, axis=0)  # [H,W]
        cam = np.maximum(cam, 0)  # ReLU
        # normalize
        cam = cam - np.min(cam)
        cam = cam / np.max(cam)

        # resize to 224*224
        box = output[index][:4].detach().numpy().astype(np.int32)
        x1, y1, x2, y2 = box
        cam = cv2.resize(cam, (x2 - x1, y2 - y1))

        class_id = output[index][-1].detach().numpy()
        return cam, box, class_id


title: Pytorch random tensor generation with one fix value and rest of random value

body:
In my testing dataset, I have to always include one specific image(image at position 0) in each batch but others values can be randomly selected. So I am making a tensor which will have 1st value 0 (for 1st image) and the rest of others can be anything other than 0. My code snippet is below.
a= torch.randperm(len(l-1)) #where l is total no of testing image in dataset, code output->tensor([10, 0, 1, 2, 4, 5])
b=torch.tensor([0]) # code output-> tensor([0])
c=torch.cat((b.view(1),a))# gives output as -> tensor([0, 10, 0, 1, 2, 4, 5]) and 0 is used twice so repeated test image

However, above approach can include 0 twice as torch.randperm many times includes 0. Is there a way in torch to generate random number skipping one specific value. Or if you think another approach will be better please comment.


title: model.eval for class with field of network - pytorch

body:
I have a class model with field of pre-trained resnet
something like:
class A(nn.Module):
    def __init__(self, **kwargs):
        super(A, self).__init__()
        self.resnet = get_resnet()

    def forward(self, x):
        return self.resnet(x)

...

now Im doing
model = A()
...
model.eval()

Is it ok or shuld I overwrite the eval, train functions?


title: Some tensors are not moved to the device I want when doing .to(device)

body:
The Problem
I have a model that requires different linear models for different time steps:
class CPC(nn.Module):
    def __init__(self, batch_size, timestep):

        super().__init__()
        self.batch_size = batch_size

        self.timestep = timestep
        self.Wk  = nn.ModuleList([nn.Linear(77, 55) for i in range(timestep)])

        def _weights_init(m):
            if isinstance(m, nn.Linear):
                nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')
            if isinstance(m, nn.Conv1d):
                nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')
            elif isinstance(m, nn.BatchNorm1d):
                nn.init.constant_(m.weight, 1)
                nn.init.constant_(m.bias, 0)

    def forward(self, x):
        pred = torch.empty((self.batch_size,self.timestep,55)).float()
        for i in np.arange(0, self.timestep):
            linear = self.Wk[i]
            pred[:,i] = linear(x[:,i]) # Wk*c_t e.g. size 8*512

        return pred

When doing model=CPC(batch_size, timestep).to(device), pred is still in cpu instead of the device I want.
My attempt to solve it
Then I modified my code to be something like this:
class CPC(nn.Module):
    def __init__(self, batch_size, timestep):

        super().__init__()

        self.timestep = timestep
        pred = torch.nn.Parameter(torch.empty((batch_size,self.timestep,55)).float())
        self.register_parameter('pred', pred)
        self.Wk  = nn.ModuleList([nn.Linear(77, 55) for i in range(timestep)])

        def _weights_init(m):
            if isinstance(m, nn.Linear):
                nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')
            if isinstance(m, nn.Conv1d):
                nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')
            elif isinstance(m, nn.BatchNorm1d):
                nn.init.constant_(m.weight, 1)
                nn.init.constant_(m.bias, 0)

    def forward(self, x):
        for i in np.arange(0, self.timestep):
            linear = self.Wk[i]
            self.pred[:,i] = linear(x[:,i]) # Wk*c_t e.g. size 8*512

        return self.pred

Now, model.to(device) moves everything to device, but I get the following error when training.

RuntimeError: leaf variable has been moved into the graph interior

What should I do such that model.to(device) moves everything to the device I want, and still make the model trainable?
Dummy code for debugging
Here's my full code for training:
import torch
import torch.nn as nn
import numpy as np
from torchvision import transforms, datasets

device='cuda:0'

class CPC(nn.Module):
    def __init__(self, batch_size, timestep):

        super().__init__()

        self.timestep = timestep
        pred = torch.nn.Parameter(torch.empty((batch_size,self.timestep,55)).float())
        self.register_parameter('pred', pred)
        self.Wk  = nn.ModuleList([nn.Linear(77, 55) for i in range(timestep)])

        def _weights_init(m):
            if isinstance(m, nn.Linear):
                nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')
            if isinstance(m, nn.Conv1d):
                nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')
            elif isinstance(m, nn.BatchNorm1d):
                nn.init.constant_(m.weight, 1)
                nn.init.constant_(m.bias, 0)

    def forward(self, x):
        for i in np.arange(0, self.timestep):
            linear = self.Wk[i]
            self.pred[:,i] = linear(x[:,i]) # Wk*c_t e.g. size 8*512

        return self.pred

x = torch.randn(16, 12, 77)
A = torch.randn(77,55)
y = torch.matmul(x, A)

model = CPC(x.size(0), x.size(1))
model.to(device)

optimizer = torch.optim.Adam(model.parameters(), lr=1e-4)
loss_fn = torch.nn.MSELoss()

x = x.to(device)
y = y.to(device)
for i in range(10):
    optimizer.zero_grad()
    pred = model(x)
#     print(f'model.pred={model.pred}')
    loss = loss_fn(pred, y)
    loss.backward()
    optimizer.step()


title: PyTorch looking up indices (tracing a path) as tensor operations

body:
How could the following be written as a Tensor operation without a loop?
a = torch.Tensor([[0, 1, 1],
                  [2, 1, 0],
                  [1, 0, 1]]).long()
trace = torch.zeros(4, dtype=torch.long)

start_idx = 0
trace[0] = start_idx

for i in range(a.shape[0]):
    trace[i + 1] = a[i, trace[i]]

print(trace)
# tensor([0, 0, 2, 1])

The code basically traces a given tensor. Starting with start_idx we are iteratively looking up the next index in the tensor to add to our trace.


title: Tensorflow tf.ragged.constant stuck when initializing from numpy.ndarray or list

body:
I am trying to load directories of images to a tensor array and nest this in an array containing these folders. For this task it seems tf.ragged.constant is appropriate since it can accept input dimensions of varying size.
I have 30 directories with around 50-100 images in each.
For some reason, I cannot initialize the tf.ragged.constant. When I try, it never passes the initialization. I have tried converting the array to a NumPy array or list before initializing but to no avail.
import numpy as np
from PIL import Image
import glob
import os
import tensorflow as tf

# Points to folders with images that needs to be converted
filelist2 = glob.glob('dataset_videoes\\train\\01April_2010_Thursday_heute-6695\\*.png')

train_dir = "dataset_videoes\\train"
path, dirs, files = next(os.walk(train_dir))

np_array_imgs_all_dirs = []

# Converts each image to a numpy array and stores it in the np_array_imgs_all_dirs list
for dir_path in dirs:
    filelist = glob.glob('dataset_videoes\\train\\' + dir_path + '\\*.png')
    x = np.array([np.array(Image.open(fname)) for fname in filelist])
    np_array_imgs_all_dirs.append(x)

print(x.shape) # (53, 224, 224, 3)

# Creates a second array with the images in a folder
x2 = np.array([np.array(Image.open(fname)) for fname in filelist2])

# Adds x1 and x2 to the same array. This tests that the errors are still present with only two directories, and also that it is not due to a error from casting to np.array from list.
two_directories_as_numpy = np.array([x, x2])

# Converts a single directory of numpy image arrays to a tensor. This works.
tf.convert_to_tensor(x)

# Tries to converts two of the image folders that were stored in a array to a tensor.
# This does not work, due to the tensor having varying input size in the first dimensionm, i suspect
# Gives following errors when passed list or numpy array:
# Can't convert non-rectangular Python sequence to Tensor.
# Failed to convert a NumPy array to a Tensor (Unsupported object type numpy.ndarray).

#tf.convert_to_tensor(two_directories_as_numpy)
print(two_directories_as_numpy.shape) # (2,)

# To solve this I convert them to tf.ragged.constant, but initalizing makes
# it stuck regardless of the datatype i test with.
print("Before initalization")
data_tensor = tf.ragged.constant(two_directories_as_numpy)
print("Initalization completed")
data_tensor = tf.ragged.constant(np_array_imgs_all_dirs)
data_tensor = tf.ragged.constant(x)


title: The PyTorch TabularDataset split is showing wrong results

body:
I've been following this tutorial on sentiment analysis. The author has a notebook on how we can use own datasets instead of the PyTorch built in ones. So I followed his footsteps, Split my dataset to train, test and validation csv files after cleaning it up and removing punctuation and unnecessary columns and finally trimming the dataset to only 500000 rows.
So after doing all of that and following the author's footsteps on how to create my fields, creating the train_data, valid_data and test_data and finally creating the vocab, I thought I'd check what is in my train_data variable I got this: {'text': ['0'], 'polarity': '0'}. Why did it show this?
Here's my code:
TEXT = data.Field()
POLARITY = data.LabelField()
fields = [('text', TEXT), ('polarity', POLARITY)]
train_data, valid_data, test_data = data.TabularDataset.splits(
                                        path = 'data',
                                        train = 'train.csv',
                                        validation = 'valid.csv',
                                        test = 'test.csv',
                                        format = 'csv',
                                        fields = fields,
                                        skip_header = True
)
print(vars(train_data[0]))


title: What target format and loss functions to use while predicting chords?

body:
I'm building lstm neural network, which is predicting chords based on the melody.
The melody (input) is in one hot format (chromagram):
tensor([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.]],
       dtype=torch.float64)

meaning that tone C is followed by D.
Targets are chromagrams of chords playing at the moment, in other words, multi hot with three 1s:
tensor([[1., 0., 0., 0., 1., 0., 0., 1., 0., 0., 0., 0.],
        [0., 0., 1., 0., 0., 0., 1., 0., 0., 1., 0., 0.]],
       dtype=torch.float64)

meaning that Cmaj chord is followed by Dmaj.
From that, I can compute dictionary of all chords in the training set. And then somehow predicting the key(index). ...For example:
1:Cmaj
2:Dmaj
...

Furthermore, keys(or indices) in dictionary can by converted to one hots. In dataset, there is 26 different chords.
I already tried chromagram multi-hot targets and one-hot indices, both with torch.nn.BCEWithLogitsLoss() loss function but I'm not sure if it is good aproach.
My question is what target x loss function would be best? Could you also provide example on how to feed the loss function?


title: Customized NN Layers each one having its own forward function PyTorch

body:
I would like to create a NN using PyTorch having different custom layers. What I mean is this:
I want several layers made this way, each one having its own forward function, because I have specific needs of different behaviours.
class myCustomLayer1(nn.Module):
    def __init__(self):
       super().__init__()
       self.layer = nn.Linear(1,2)

    def forward(self, input):
        #do something
        return output

So this structure is replicated:
class myCustomLayer2(nn.Module):
    def __init__(self):
       super().__init__()
       self.layer = #another different type of layer

    def forward(self, input):
        #do something
        return output

and so on. Problem is: what happens when I define the container for all those layer? I would like to have something similar to this
class myNN(nn.Module):
       def __init__(self):
           super().__init__()
           self.layer1 = myCustomLayer1()
           self.layer2 = myCustomLayer2()
           ...
       def forward(self, input):
           ##here code from forward functions of different layers is called

and
model = myNN(input)

where input is of proper shape.
I know forward function is called in __call__ method. So I wonder if there's a way to combine multiple forward functions in one. I hope I was clear. Thanks to all


title: Runtime connect() timed out error when using multiple node with distributedDataParallel in pytorch

body:
I am having problem running training on Multiple GPUs on multiple node using DistributedDataParallel. I get RuntimeError: connect() timed out on Node 2. The code works fine when I am using just one Node and multiple GPUs on that Node. I am running my code in the docker image. I have pasted my code below and also the steps I use to run the training.
batch_loader.py:
from torch.utils import data
import random
import os
import numpy as np
import torch

class TrainFolder(data.Dataset):
    def __init__(self, file):
        super(TrainFolder, self).__init__()

        self.images = []
        fid = file
        for x in fid:
            if x == '':
                continue
            labelfile = x.replace("nonPR", "PR")
            info = (x, labelfile)
            self.images.append(info)

        random.shuffle(self.images)

    def __len__(self):
        return len(self.images)

    def __getitem__(self, index):
        image_file, label_file = self.images[index]
        img = np.load(image_file)
        lab = np.load(label_file)

        img = np.rollaxis(img, 2, 0)
        lab = np.rollaxis(lab, 2, 0)

        im = img.copy()
        lb = lab.copy()
        img = torch.from_numpy(im[:, :, :])
        lab = torch.from_numpy(lb[:, :, :])
        return img, lab

gan_network.py:
import math
import torch
import torch.nn as nn

def gen_initialization(m):
    if type(m) == nn.Conv2d:
        sh = m.weight.shape
        nn.init.normal_(m.weight, std=math.sqrt(2.0 / (sh[0]*sh[2]*sh[3])))
        nn.init.constant_(m.bias, 0)
    elif type(m) == nn.BatchNorm2d:
        nn.init.normal_(m.weight)
        nn.init.normal_(m.bias)

class TripleConv(nn.Module):
    def __init__(self, in_ch, out_ch):
        super(TripleConv, self).__init__()
        mid_ch = (in_ch + out_ch) // 2
        self.conv = nn.Sequential(
            nn.Conv2d(in_ch, mid_ch, kernel_size=3, stride=1, padding=1, bias=True),
            nn.ReLU(),
            nn.Conv2d(mid_ch, mid_ch, kernel_size=3, stride=1, padding=1, bias=True),
            nn.ReLU(),
            nn.Conv2d(mid_ch, out_ch, kernel_size=3, stride=1, padding=1, bias=True),
            nn.ReLU()
        )
        self.conv.apply(gen_initialization)

    def forward(self, x):
        return self.conv(x)

class Down(nn.Module):
    def __init__(self, in_ch, out_ch):
        super(Down, self).__init__()
        self.triple_conv = TripleConv(in_ch, out_ch)
        self.avg_pool_conv = nn.AvgPool2d(2, 2)
        self.in_ch = in_ch
        self.out_ch = out_ch

    def forward(self, x):
        self.cache = self.triple_conv(x)
        pad = torch.zeros(x.shape[0], self.out_ch - self.in_ch, x.shape[2], x.shape[3], device=x.device)
        x = torch.cat((x, pad), dim=1)
        self.cache += x
        return self.avg_pool_conv(self.cache)

class Center(nn.Module):
    def __init__(self, in_ch, out_ch):
        super(Center, self).__init__()

        self.conv = nn.Sequential(
            nn.Conv2d(in_ch, out_ch, kernel_size=3, stride=1, padding=1, bias=True),
            nn.ReLU()
        )
        self.conv.apply(gen_initialization)

    def forward(self, x):
        return self.conv(x)

class Up(nn.Module):
    def __init__(self, in_ch, out_ch):
        super(Up, self).__init__()
        self.upsample = nn.Upsample(scale_factor=2, mode='bilinear',
                                    align_corners=True)
        self.triple_conv = TripleConv(in_ch, out_ch)

    def forward(self, x, cache):
        x = self.upsample(x)
        x = torch.cat((x, cache), dim=1)
        x = self.triple_conv(x)
        return x

class UNet(nn.Module):
    def __init__(self, in_ch, first_ch=None):
        super(UNet, self).__init__()

        if not first_ch:
            first_ch = 32

        self.down1 = Down(in_ch, first_ch)
        self.down2 = Down(first_ch, first_ch*2)
        self.down3 = Down(first_ch*2, first_ch*4)
        self.down4 = Down(first_ch*4, first_ch*8)
        self.center = Center(first_ch*8, first_ch*8)
        self.up4 = Up(first_ch*8*2, first_ch*4)
        self.up3 = Up(first_ch*4*2, first_ch*2)
        self.up2 = Up(first_ch*2*2, first_ch)
        self.up1 = Up(first_ch*2, first_ch)
        self.output = nn.Conv2d(first_ch, in_ch, kernel_size=3, stride=1,
                                padding=1, bias=True)
        self.output.apply(gen_initialization)

    def forward(self, x):

        x = self.down1(x)
        x = self.down2(x)
        x = self.down3(x)
        x = self.down4(x)
        x = self.center(x)
        x = self.up4(x, self.down4.cache)
        x = self.up3(x, self.down3.cache)
        x = self.up2(x, self.down2.cache)
        x = self.up1(x, self.down1.cache)
        x = self.output(x)

        return x

pr_train_mp.py:
from configobj import ConfigObj
from tqdm import tqdm
import os
import gan_network
import glob
import random
import torch
from torch.utils.data import DataLoader
from torch.utils.data.dataset import random_split
from tensorboardX import SummaryWriter
from batch_loader import TrainFolder
import torch.nn as nn
import torch.optim as optim
import torch.multiprocessing as mp
import argparse
import torch.distributed as dist

os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
os.environ["CUDA_VISIBLE_DEVICES"] = "0"

def init_parameters():
    tc, vc = ConfigObj(), ConfigObj()
    tc.batch_size, vc.batch_size = 1, 1
    tc.n_channels, vc.n_channels = 6, 6
    tc.image_size, vc.image_size = 1024, 1024
    return tc, vc

def train(gpu, args):
    ############################################################
    rank = args.nr * args.gpus + gpu
    dist.init_process_group(
        backend='nccl',
        init_method='env://',
        world_size=args.world_size,
        rank=rank
    )
    ############################################################
    torch.manual_seed(47)
    torch.backends.cudnn.benchmark = True
    netG = gan_network.UNet(6, first_ch=32)
    torch.cuda.set_device(gpu)
    netG.cuda(gpu)

    optimizerG = optim.Adam(netG.parameters(), lr=1e-4, betas=(0.9, 0.999))

    # Initialize BCELoss function
    criterion = nn.MSELoss().cuda(gpu)

    ###############################################################
    # Wrap the model
    netG = nn.parallel.DistributedDataParallel(netG, device_ids=[gpu])
    ###############################################################

    # Data loading code
    train_samples = glob.glob('/home/data/nas/Processed_Data/training_data/phase_recovery/110920/npyfiles/size_1024/train/*_nonPR.npy')
    valid_samples = glob.glob('/home/data/nas/Processed_Data/training_data/phase_recovery/110920/npyfiles/size_1024/valid/*_nonPR.npy')

    random.shuffle(train_samples)

    trainData = TrainFolder(train_samples)
    validData = TrainFolder(valid_samples)

    ################################################################
    train_sampler = torch.utils.data.distributed.DistributedSampler(
        trainData,  num_replicas=args.world_size, rank=rank)

    valid_sampler = torch.utils.data.distributed.DistributedSampler(
        validData,  num_replicas=args.world_size, rank=rank)
    ################################################################
    train_config, valid_config = init_parameters()
    train_data_loader = DataLoader(dataset=trainData, num_workers=0, batch_size=train_config.batch_size,
                                    drop_last=False, pin_memory=True, sampler=train_sampler)
    valid_data_loader = DataLoader(dataset=validData, num_workers=0, batch_size=valid_config.batch_size,
                                    drop_last=False, pin_memory=True, sampler=valid_sampler)

    niter = 10000
    for epoch in range(niter):
        train_sampler.set_epoch(epoch)
        valid_sampler.set_epoch(epoch)
        netG.train()
        for i, (images, labels) in enumerate(tqdm(train_data_loader)):
            images = images.to(non_blocking=True)
            labels = labels.cuda(non_blocking=True)
            images = images.float()
            labels = labels.float()

            netG.zero_grad()
            optimizerG.zero_grad()

            output = netG(images)
            errG_mse = criterion(output, labels)
            errG_mse.backward()

            optimizerG.step()

        netG.eval()
        with torch.no_grad():
            for i, (images, labels) in enumerate(tqdm(valid_data_loader)):
                images = images.cuda(non_blocking=True)
                labels = labels.cuda(non_blocking=True)
                images = images.float()
                labels = labels.float()

                G_output = netG(images)
                valid_errG_mse = criterion(G_output, labels)

        if epoch % 3 == 0 and gpu == 0:
            torch.save(netG.state_dict(), f'model/network_epoch{epoch}.pth')

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('-n', '--nodes', default=1, type=int, metavar='N')
    parser.add_argument('-g', '--gpus', default=1, type=int,
                        help='number of gpus per node')
    parser.add_argument('-nr', '--nr', default=0, type=int,
                        help='ranking within the nodes')
    args = parser.parse_args()
    #########################################################
    args.world_size = args.gpus * args.nodes                #
    os.environ['MASTER_ADDR'] = 'a.b.c.d'             #
    os.environ['MASTER_PORT'] = '110'                       #
    mp.spawn(train, nprocs=args.gpus, args=(args,))         #
    #########################################################

if __name__ == '__main__':
    main()

Steps followed to train:

Launch my docker image on Node 1 and Node 2 using this command: (docker image name is aqusens-train
sudo docker run --network=host -it --shm-size=32G --rm --runtime=nvidia -v /home:/home aqusens-train

On node 1 run the following command:
 python3 pr_train_mp.py -n 2 -g 1 -nr 0

On node 2  run the following command:
 python3 pr_train_mp.py -n 2 -g 1 -nr 1

Node 1 is the root machine. I can run the code individually on each node.
Input Image SIze: (NCHW) = (1,6,1024,1024)
Node 1 Environment::
Pytorch: 1.7.1                                                                                Ubuntu: '18.04'                                                                                cudnn: 7605                                                                                cuda: 10.2
Node 2 Environment::
Pytorch: 1.7.0                                                                                Ubuntu: '18.04'                                                                                cudnn: 7605                                                                                cuda: 10.2


title: Ray Tune get summary table after tune.run() has finished?

body:
When tune.run() has finished running, is it possible to grab a DataFrame or table of the table that is frequently printed out during training?
This is an example of the table containing the hyperparameters of all the trials and some metrics.
+-------------------+------------+-------+-----------+-----------------+--------+------------------+------------+
| Trial name        | status     | loc   |   dropout |   learning_rate |   iter |   total time (s) |       loss |
|-------------------+------------+-------+-----------+-----------------+--------+------------------+------------+
| train_d48b7_00000 | TERMINATED |       |       0.5 |          5e-05  |     39 |          349.52  |   0.112351 |
| train_d48b7_00001 | TERMINATED |       |       0.6 |          5e-05  |     39 |          351.183 |   0.121403 |
| train_d48b7_00002 | TERMINATED |       |       0.7 |          5e-05  |     39 |          351.692 |   0.141085 |
+-------------------+------------+-------+-----------+-----------------+--------+------------------+------------+

Or another summary table that shows some overview of the entire Tune run. I will be using this summary data for automatically sending a Slack notification.


title: [Pytorch][RNN] basic using of RNN and fully connected layer (Inconsistency of matrix shape)

body:
I'm studying pyTorch, and I have a question.
I try to design a simple binary classifier using LSTM.
I met a situation, cannot connect between output layer and fully connected layer.
The error message was there is a inconsistency of shape.
My code is as bellows.
# define layers
define __init__(self, num_embeddings, padding_idx, embedding_dim,
                 hidden_dim, num_layers, dropout, bidirectional):
    self.embedding = nn.Embedding(num_embeddings, embedding_dim, padding_idx = padding_idx)
    self.dropout = nn.Dropout(dropout)
    self.rnn: nn.Module = nn.LSTM(input_size=embedding_dim, hidden_size= hidden_dim,
                                      num_layers=num_layers, batch_first=True, bidirectional=bidirectional)
    self.fc: nn.Module = nn.Linear(2*hidden_dim, 1)

#define forward
def forward(self, text, text_lengths):
    embedded = self.dropout(self.embedding(text))

    #isomg pack_padded_sequence - cause, sequential data
    packed_embedded = pack_padded_sequence(embedded, text_lengths.to('cpu'), batch_first = True)
    packed_output, _ = self.rnn(packed_embedded)

    outputs, _ = pad_packed_sequence(packed_output, batch_first=True)
    print ("HERE", outputs.shape)
    print ("JERE",outputs[:,-1,].shape)

    # using [:,-1,] to pick sequentially last data in out
    ret = self.fc(outputs[:,-1,])

    output: torch.Tensor = self.fc(ret)

At the line, the output of LSTM passing fully connected layer, Error message is printed.
RuntimeError: mat1 and mat2 shapes cannot be multiplied (32x1 and 64x1)

Lines printing the shape of Tesor printed shows as bellows
HERE torch.Size([32, 907, 64]) //shape of output of lstm
JERE torch.Size([32, 64]) // shape of every last sequence of output of lstm

My question is

Why the shape of LSTM's output is [32,907,64]?? not [32,907,32]?? LSTM's hidden node size is 32.
Why cannot multiply [32,64] x [64,1], my fully connected layer?


title: What is the most appropriate way for masking in loss functions in PyTorch

body:
The target values look something like this:
y = tensor([[10.3, -1, -1, 4.5, 6.7, -1]])

and the predicted values from the network look something like this:
y_hat = tensor([[13.1, 12, 4.2, 6.2, 3.2, 10.1]], dtype=torch.float64, grad_fn=<ReluBackward0>)

I would like to evaluate the mean square error only for those elements which are not equal to -1 in tensor y without losing track of the gradients.
In essence, I would like the loss to be:
loss = (1/3)*sqrt((13.1-10.3)^2 + (6.2-4.5)^2 + (3.2-6.7)^2)

What is the most appropriate way of evaluating a loss like this?


title: How to make the backword when some loss come from the eigenvector in PyTorch?

body:
my loss seems like
e1, v1 = torch.eig(A)
e2, v2 = torch.eig(B)
sim = torch.matmul(v1, v2.permute(0, 2, 1))
loss_sim = torch.sum(sim)
loss_sim.backward()

where A and B is the intermediate tensors with shape SEQUENCE_LENGH * SEQUENCE_LENGTH
and my code meet the error:
 File "/usr/local/Anaconda3/lib/python3.8/site-packages/torch/tensor.py", line 185, in backward  torch.autograd.backward(self, gradient, retain_graph, create_graph) File "/usr/local/Anaconda3/lib/python3.8/site-packages/torch/autograd/__init__.py", line 125, in backward Variable._execution_engine.run_backward( RuntimeError: eig_backward: Backward calculation does not support complex eigenvalues at the moment. Exception raised from eig_backward at /opt/conda/conda-bld/pytorch_1595629395347/work/torch/csrc/autograd/generated/Functions.cpp:1877 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x4d (0x7f74932e077d in /usr/local/Anaconda3/lib/python3.8/site-packages/torch/lib/libc10.so) frame #1: torch::autograd::generated::EigBackward::apply(std::vector<at::Tensor, std::allocator<at::Tensor> >&&) + 0xb5d (0x7f74cc4b894d in /usr/local/Anaconda3/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so) frame #2: <unknown function> + 0x30d1017 (0x7f74ccb0c017 in /usr/local/Anaconda3/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so) frame #3: torch::autograd::Engine::evaluate_function(std::shared_ptr<torch::autograd::GraphTask>&, torch::autograd::Node*, torch::autograd::InputBuffer&, std::shared_ptr<torch::autograd::ReadyQueue> const&) + 0x1400 (0x7f74ccb07860 in /usr/local/Anaconda3/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so) frame #4: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x451 (0x7f74ccb08401 in /usr/local/Anaconda3/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so) frame #5: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x89 (0x7f74ccb00579 in /usr/local/Anaconda3/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so) frame #6: torch::autograd::python::PythonEngine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x4a (0x7f74d0e2a1ba in /usr/local/Anaconda3/lib/python3.8/site-packages/torch/lib/libtorch_python.so) frame #7: <unknown function> + 0xc819d (0x7f74d393719d in /usr/local/Anaconda3/lib/python3.8/site-packages/torch/lib/../../../.././libstdc++.so.6) frame #8: <unknown function> + 0x76ba (0x7f74ed28c6ba in /lib/x86_64-linux-gnu/libpthread.so.0) frame #9: clone + 0x6d (0x7f74ecfc241d in /lib/x86_64-linux-gnu/libc.so.6)
How do I accomplish the backword with torch.eig()?


title: MADDPG: RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation

body:
I have a problem with the inplace operation in my code. I was figuring that it may be a difference between the torch version, where I am using it torch=1.8.1 and torch=1.4.0 works fine for my codes. I also use torch.autograd.detect_anomaly() to find some reasons.
where my torch=1.8.1, python=1.7.5
There's the traceback:
Warnning: at step 0 , agent 22372852612 switch to phase 7 . Maybe empty road
maddpg.py:96: UserWarning: Anomaly Detection has been enabled. This mode will increase the runtime and should only be enabled for debugging.
  with torch.autograd.detect_anomaly():
maddpg.py:103: UserWarning: Anomaly Detection has been enabled. This mode will increase the runtime and should only be enabled for debugging.
  with torch.autograd.detect_anomaly():
/usr/local/lib/python3.7/dist-packages/torch/autograd/__init__.py:147: UserWarning: Error detected in AddmmBackward. No forward pass information available. Enable detect anomaly during forward pass for more information. (Triggered internally at  /pytorch/torch/csrc/autograd/python_anomaly_mode.cpp:85.)
  allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag
Traceback (most recent call last):
  File "main.py", line 101, in <module>
    maddpg_agent.learn(memory)
  File "maddpg.py", line 97, in learn
    critic_loss.backward(retain_graph=True)
  File "/usr/local/lib/python3.7/dist-packages/torch/tensor.py", line 245, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/usr/local/lib/python3.7/dist-packages/torch/autograd/__init__.py", line 147, in backward
    allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [64, 8]], which is output 0 of TBackward, is at version 3; expected version 2 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

There are the following codes:
main.py
raw_obs_, raw_reward, raw_done, info = env.step(actions)
memory.store_transition(obs, state, raw_actions, reward, obs_, state_, done)

        # if total_step % 100 == 0 and not evaluate:
maddpg_agent.learn(memory)
obs_ = obs

maddpg.py
def learn(self, memory):
    if not memory.ready():
        return
    actor_states, states, actions, rewards, \
    actor_new_states, states_, dones = memory.sample_buffer()

    device = self.agents[0].actor.device

    states = torch.tensor(states, dtype=torch.float).to(device)
    actions = torch.tensor(actions, dtype=torch.float).to(device)
    rewards = torch.tensor(rewards, dtype=torch.float).to(device)
    states_ = torch.tensor(states_, dtype=torch.float).to(device)
    dones = torch.tensor(dones).to(device)

    all_agents_new_actions = []
    all_agents_new_mu_actions = []
    old_agents_actions = []

    for agent_idx, agent in enumerate(self.agents):
        new_states = torch.tensor(actor_new_states[agent_idx],
                             dtype=torch.float).to(device)

        new_pi = agent.target_actor.forward(new_states)

        all_agents_new_actions.append(new_pi)
        mu_states = torch.tensor(actor_states[agent_idx],
                             dtype=torch.float).to(device)
        pi = agent.actor.forward(mu_states)
        all_agents_new_mu_actions.append(pi)
        old_agents_actions.append(actions[agent_idx])
    new_actions = torch.cat([acts for acts in all_agents_new_actions], dim=1)
    mu = torch.cat([acts for acts in all_agents_new_mu_actions], dim=1)
    old_actions = torch.cat([acts for acts in old_agents_actions],dim=1)

    for agent_idx, agent in enumerate(self.agents):
        critic_value_ = agent.target_critic.forward(states_, new_actions).flatten()
        critic_value_[dones[:, 0]] = 0.0
        critic_value = agent.critic.forward(states, old_actions).flatten()

        target = rewards[:, agent_idx] + agent.gamma * critic_value_
        critic_loss = F.mse_loss(target, critic_value)
        agent.critic.optimizer.zero_grad()
        with torch.autograd.detect_anomaly():
            critic_loss.backward(retain_graph=True)
            agent.critic.optimizer.step()
        actor_loss = agent.critic.forward(states, mu).flatten()
        actor_loss = torch.mean(actor_loss)
        agent.actor.optimizer.zero_grad()
        with torch.autograd.detect_anomaly():
            actor_loss.backward(retain_graph=True)
            agent.actor.optimizer.step()

        agent.update_network_parameters()

and my netwrok.py is:
class CriticNetwork(nn.Module):
    def __init__(self, beta, input_dim, fc1_dims, fc2_dims,
                 n_agents, n_actions, name, chkpt_dir):
        super(CriticNetwork, self).__init__()

        if not chkpt_dir:
            os.mkdir(chkpt_dir)
        self.chkpt_file = os.path.join(chkpt_dir, name)

        self.input_dim = input_dim
        self.n_agents = n_agents
        self.n_actions = n_actions

        self.fc1 = nn.Linear(input_dim + n_agents * n_actions, fc1_dims)
        self.fc2 = nn.Linear(fc1_dims, fc2_dims)
        self.q = nn.Linear(fc2_dims, 1)

        self.optimizer = optim.Adam(self.fc1.parameters(), lr=beta)
        self.device = torch.device('cuda:0' if torch.cuda.is_available() else "cpu")
        self.to(self.device)

    def forward(self, states, actions):
        x = F.relu(self.fc1(torch.cat([states, actions], dim=1)))
        x = F.relu(self.fc2(x))
        q = self.q(x)
        return q


title: How to efficiently perform random array assignments and stack the results in PyTorch?

body:
Basic requirement: suppose I have an array of [9, 8, 7], and I can generate two random assignment indices arrays, suppose they are [1, 1, 0] and [0, 2, 1], so the assignment results are [8, 8, 9] and [9, 7, 8], and finally I want to get the stacked result: [[8, 8, 9], [9, 7, 8]].
I can achieve the above function by the following code with a for-loop:
import torch
my_array = torch.Tensor([9, 8, 7])
for i in range(2):  # for each layer
    my_assignment_indices = torch.randint(0, 3, size=(3,))  # for each location, randomly choose a source index
    current_assignment_array = my_array[my_assignment_indices].unsqueeze(0)  # move (assign)
    if i == 0:
        result = current_assignment_array  # the first layer, create result
    else:
        result = torch.cat([result, current_assignment_array])  # the subsequent layers, stack them together
print(result)  # output my result

In practice, I have two parameters: N (array length), K (layer number).
Notice that each index is randomly chosen from [0, ..., N-1].
Now I am confused by the following two problems:

How can I perform the above process efficiently with very large N and K on CPU or GPU?
How can I randomly assign K different elements to each location? The above code does not guarantee that each two indices for a location must be different.

I'm waiting for a better solution, thank you for watching my problems :).


title: PyTorch ValueError: Expected target size (2, 33), got torch.Size([2, 73])

body:
I'm quite new to Pytorch.
I want to compute the loss of a batch in a Transformer. In this case my 'batch' has only two replicas.
The batch outputs a Tensor of the shape 2,73,33:
output
tensor([[[ 21.1355,  -7.5047,   2.8138,  ..., -14.1462, -15.1999,  -7.2595],...
output.size()
>>> torch.Size([2, 73, 33])

The target has the categorical solution for each position. It has the shape 2,73:
target.size()
>>> torch.Size([2, 73])

When I compute the loss, I get a value in the case I only compare the first replica:
loss = torch.nn.CrossEntropyLoss(ignore_index=1)
loss(output[0], target[0])
tensor(0.1967)

But it errors when I do it all at once:
loss = torch.nn.CrossEntropyLoss(ignore_index=1)
loss(output, target)
ValueError: Expected target size (2, 33), got torch.Size([2, 73])

Do I have to loop over the replicas and average them?
Any help is much appreciated.


title: Failed to convert a Pytorch model to CoreML

body:
I tried to convert a Pytorch model (like DeepLab) into CoreMl. However, I always faced an error at the end of the conversion:
"Error compiling model: "Error reading protobuf spec. validator error: Layer 'input' produces an output named 'input' which is also an output produced by the layer '__input'."."
I checked the naming of the input/inputs and didn't see any issue at all. Could anyone help to shed some light on how to resolve this? Many thanks!


title: Pytorch: create a mask that is larger than the n-th quantile of each 2D tensor in a batch

body:
I have a torch.Tensor of shape (2, 2, 2) (can be bigger), where the values are normalized within range [0, 1].
Now I am given a positive integer K, which tells me that I need to create a mask where for each 2D tensor inside the batch, values are 1 if it is larger than 1/k of all the values, and 0 elsewhere. The return mask also has shape (2, 2, 2).
For example, if I have a batch like this:
tensor([[[1., 3.],
         [2., 4.]],
        [[5., 7.],
         [9., 8.]]])

and let K=2, it means that I must mask the values where they are greater than 50% of all the values inside each 2D tensor.
In the example, the 0.5 quantile is 2.5 and 7.5, so this is the desired output:
tensor([[[0, 1],
         [0, 1]],
        [[0, 0],
         [1, 1]]])

I tried:
a = torch.tensor([[[0, 1],
                   [0, 1]],
                  [[0, 0],
                   [1, 1]]])
quantile = torch.tensor([torch.quantile(x, 1/K) for x in a])
torch.where(a > val, 1, 0)

But this is the result:
tensor([[[0, 0],
         [0, 0]],
        [[1, 0],
         [1, 1]]])


title: How to get modules of 3d tensors in 4d torch tensor?

body:
I have a torch tensor with 4 dimensions.
It's shape is [50, 1, 1, 200]. I have to get a list with modules of 200 hundred 3d tensors. Which is the easiest way to do so?


title: How to name an unnamed parameter of a model in pytorch?

body:
How can one name an unnamed parameter of a model in Pytorch?
My end goal is to find out if a specific parameter is being optimized or not, but since it does not have a name, it makes it hard to find it out when listing the parameters. I thought putting a name on that variable would make life easier.


title: PyTorch Convolutional Accuracy frozen

body:
I'm having problems making my PyTorch program's accuracy go higher than 0.  The loss that I'm getting also doesn't change.
The code:
train_loader = DataLoader(dataset=train_dataset, batch_size=batch_size, shuffle=True)
test_loader = DataLoader(dataset=test_dataset, batch_size=batch_size, shuffle=False)

class ConvNet(nn.Module):
    def __init__(self, numclasses):
        """
        args
        """
        super(ConvNet, self).__init__()
        self.layer1 = nn.Sequential(
            nn.Conv2d(1, 64, 3, 1, 1),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2)
        )
        self.layer2 = nn.Sequential(
            nn.Conv2d(64, 256, 3, 1, 1),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2)
        )
        self.layer3 = nn.Sequential(
            nn.Conv2d(256, 512, 3, 1, 1),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2)
        )
        self.fc = nn.Sequential(
            nn.Linear(512*3*3, 512),
            nn.Linear(512, numclasses)
        )

    def forward(self, net):
        net = self.layer1(net)
        net = self.layer2(net)
        net = self.layer3(net)
        net = net.view(net.size(0), -1)
        net = self.fc(net)
        return net
model = ConvNet(100)
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=.001)

total_step = len(train_loader)
loss_list = []
acc_list = []
num_epochs=3
for epoch in range(num_epochs):
    for i, (images, labels) in enumerate(train_loader):
        # Run the forward pass
        outputs = model(images)
        loss = criterion(outputs, labels)
        loss_list.append(loss.item())

        # Backprop and perform Adam optimisation
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        # Track the accuracy
        total = labels.size(0)
        _, predicted = torch.max(outputs.data, 1)
        correct = (predicted == labels).sum().item()
        acc_list.append(correct / total)

        if (i + 1) % 20 == 0:
            print('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}, Accuracy: {:.2f}%'
                  .format(epoch + 1, num_epochs, i + 1, total_step, loss.item(),
                          (correct / total) * 100))

I'm not sure if my train_dataset is correct.  It's made up of 10000 tuples of images and labels.  The images have shapes like:
 tensor([[[0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,
          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,
          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,
          0.0000, 0.0000, 0.0000, 0.0000],
         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,
          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.1098, 0.2667, 0.4196,
          0.1686, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,
          0.0000, 0.0000, 0.0000, 0.0000],
        ...
         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,
          0.0000, 0.0000, 0.0000, 0.6510, 0.9725, 1.0000, 1.0000, 1.0000,
          0.6510, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,
          0.0000, 0.0000, 0.0000, 0.0000]]])

and each label has a shape like:
tensor(78)
From what I can tell, when the model is running the images are unzipped properly along with the labels.  Do I need to change the images' format, or add a setting to the optimizer?  Thanks for any help provided.


title: How to convert this nested JSON file in pandas df?

body:
I have to deal with this database for a project:
In particular, I need to obtain a pandas df to formatting these data as input for a neural network in an NLP task. The Json format is the following:
json file
├── "data"
│   └── [i]
│       ├── "paragraphs"
│       │   └── [j]
│       │       ├── "context": "paragraph text"
│       │       └── "qas"
│       │           └── [k]
│       │               ├── "answers"
│       │               │   └── [l]
│       │               │       ├── "answer_start": N
│       │               │       └── "text": "answer"
│       │               ├── "id": "&lt;uuid&gt;"
│       │               └── "question": "paragraph question?"
│       └── "title": "document id"
└── "version": 1.1

I tried hard using .json_normalize method but I can't get any results. I've noticed that most of my attempts (those which don't end with an error) end with recognizing as indexes just "data" and "version" and as the only object the rest of the text, as here:
f = open("SQuAD_it-test.json", "r",encoding="Latin-1" )

data = json.load(f)

df = pd.json_normalize(data)
df.sample(1)

                                         data         version
0   [{'paragraphs': [{'qas': [{'question': 'Quando...   1.1

And if I try to visualize more samples an error occurs to me which tells the population is only 1.
My desired output is something like this, selecting the indexes to use which can be at different level of the tree:
df.sample(5)

        title    context   question  text  answer_start
        str1     str6      str11     str16    N1
        str2     str7      str12     str17    N2
        str3     str8      str13     str13    N3
        str4     str9      str14     str18    N4
        str5     str10     str15     str19    N5

I've also worked on the argument of .json_normalize.
But I'm not able to fully comprehend the explanation. Could you help me?


title: only first gpu is allocated (eventhough I make other gpus visible, in pytorch cuda framework)

body:
I am using cuda in pytorch framwework in linux server with multiple cuda devices.
The problem is that
eventhough I specified certain gpus that can be shown,
the program keeps using only first gpu.
(But other program works fine and other specified gpus are allocated well.
because of that, I think it is not nvidia or system problem.
nvidia-smi shows all gpus well and there's no problem.
I didn't have problem with allocating gpus with below codes before (except when the system is not working)
)
os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID"
os.environ["CUDA_VISIBILE_DEVICES"] = str(args.gpu)

I wrote that before running main function.
and it works fine for other programs in same system.
I printed that args.gpu variable, and could see that the value is not "0".


title: Pytorch Custom C++ and Cuda Extensions RunTime error

body:
I am trying to implement custom C++ and Cuda Extensions for Pytorch. What I have is two separate modules, one for CPU implementation and another for GPU implementation.

As per the tutorial, in the pytorch/ folder I created two sub-folders module_c and module_cuda_c

pytorch/
    module_c/
        module.c
        module.h
        setup.py
    module_cuda_c/
        module_cuda.c
        module_cuda.h
        module_kernel.cu
        module_kernel.h
        setup.py

For module_c, setup.py is:
from setuptools import setup, Extension
from torch.utils import cpp_extension
from torch.utils.cpp_extension import BuildExtension, CUDAExtension
import os
setup(name='module_c',
      ext_modules=[cpp_extension.CppExtension('module_c',
                    [os.path.abspath('module.c')])],
                    cmdclass={'build_module_ext': cpp_extension.BuildExtension})

And module.c is as follows:
#include <torch/extension.h>
#include <TH/TH.h>
#include <math.h>

int module_forward(...)
{
   ...
}
PYBIND11_MODULE(module_c, m) {
  m.def("forward", &module_forward, "module_forward");
}

For module_cuda_c, setup.py is:
from setuptools import setup, Extension
from torch.utils import cpp_extension
from torch.utils.cpp_extension import BuildExtension, CUDAExtension
import os
setup(
    name='module_cuda_c',
    ext_modules=[
        CUDAExtension('module_cuda_c', [
            os.path.abspath('module_cuda.c'),
            os.path.abspath('module_kernel.cu'),
        ])
    ],
    cmdclass={
        'build_roi_temporal_pooling_cuda_ext': BuildExtension
    })

And module_cuda.c is as follows:
#include <torch/extension.h>
#include <THC/THC.h>
#include <math.h>
#include "module_kernel.h"

int module_forward_cuda(...)
{
   ...
}
int module_backward_cuda(...)
{
   ...
}
PYBIND11_MODULE(module_cuda_c, m) {
    m.def("forward_cuda", &module_forward_cuda, "module_forward_cuda");
    m.def("backward_cuda", &module_backward_cuda, "module_backward_cuda");
}

Finally, the Python file that uses these C-functions is as follows:
myPathC = "../../../../pytorch/module_c"
myPathCuda = "../../../../pytorch/module_cuda_c"
from torch.utils.cpp_extension import load
module_cMod = load(name="module_c",
                        sources=[os.path.abspath(myPathC + "/" + "module.c")],
                        verbose=True)
print (help(module_cMod.forward))
module_cudaMod = load(name="module_cuda_c",
                      sources=[os.path.abspath(myPathCuda + "/" + "module_cuda.c"),
                               os.path.abspath(myPathCuda + "/" + "module_kernel.cu")],
                      verbose=True)

The first load works and "help" shows the signature of forward function. But the second load fails. The error is as follows:
Using /home/user/.cache/torch_extensions as PyTorch extensions root...
Emitting ninja build file /home/user/.cache/torch_extensions/moduleing_c/build.ninja...
Building extension module moduleing_c...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module moduleing_c...

None
Using /home/user/.cache/torch_extensions as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file /home/user/.cache/torch_extensions/moduleing_cuda_c/build.ninja...
Building extension module moduleing_cuda_c...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/2] c++ -MMD -MF moduleing_cuda.o.d -DTORCH_EXTENSION_NAME=moduleing_cuda_c -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /home/user/anaconda3/envs/pytorchvideo/lib/python3.7/site-packages/torch/include -isystem /home/user/anaconda3/envs/pytorchvideo/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -isystem /home/user/anaconda3/envs/pytorchvideo/lib/python3.7/site-packages/torch/include/TH -isystem /home/user/anaconda3/envs/pytorchvideo/lib/python3.7/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /home/user/anaconda3/envs/pytorchvideo/include/python3.7m -D_GLIBCXX_USE_CXX11_ABI=1 -fPIC -std=c++14 -c /home/user/workspace/pytorch/moduleing_cuda_c/moduleing_cuda.c -o moduleing_cuda.o
FAILED: moduleing_cuda.o
c++ -MMD -MF moduleing_cuda.o.d -DTORCH_EXTENSION_NAME=moduleing_cuda_c -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /home/user/anaconda3/envs/pytorchvideo/lib/python3.7/site-packages/torch/include -isystem /home/user/anaconda3/envs/pytorchvideo/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -isystem /home/user/anaconda3/envs/pytorchvideo/lib/python3.7/site-packages/torch/include/TH -isystem /home/user/anaconda3/envs/pytorchvideo/lib/python3.7/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /home/user/anaconda3/envs/pytorchvideo/include/python3.7m -D_GLIBCXX_USE_CXX11_ABI=1 -fPIC -std=c++14 -c /home/user/workspace/pytorch/moduleing_cuda_c/moduleing_cuda.c -o moduleing_cuda.o
/home/user/workspace/pytorch/moduleing_cuda_c/moduleing_cuda.c: In function ‘int moduleing_forward_cuda(int, int, int, float, float, c10::TensorImpl*, c10::TensorImpl*, c10::TensorImpl*, c10::TensorImpl*)’:
/home/user/workspace/pytorch/moduleing_cuda_c/moduleing_cuda.c:41:27: error: ‘THCState_getCurrentStream’ was not declared in this scope
     cudaStream_t stream = THCState_getCurrentStream(state);
                           ^~~~~~~~~~~~~~~~~~~~~~~~~
/home/user/workspace/pytorch/moduleing_cuda_c/moduleing_cuda.c:41:27: note: suggested alternative: ‘THCState_getCudaHostAllocator’
     cudaStream_t stream = THCState_getCurrentStream(state);
                           ^~~~~~~~~~~~~~~~~~~~~~~~~
                           THCState_getCudaHostAllocator
/home/user/workspace/pytorch/moduleing_cuda_c/moduleing_cuda.c: In function ‘int moduleing_backward_cuda(int, int, int, float, float, c10::TensorImpl*, c10::TensorImpl*, c10::TensorImpl*, c10::TensorImpl*)’:
/home/user/workspace/pytorch/moduleing_cuda_c/moduleing_cuda.c:85:27: error: ‘THCState_getCurrentStream’ was not declared in this scope
     cudaStream_t stream = THCState_getCurrentStream(state);
                           ^~~~~~~~~~~~~~~~~~~~~~~~~
/home/user/workspace/pytorch/moduleing_cuda_c/moduleing_cuda.c:85:27: note: suggested alternative: ‘THCState_getCudaHostAllocator’
     cudaStream_t stream = THCState_getCurrentStream(state);
                           ^~~~~~~~~~~~~~~~~~~~~~~~~
                           THCState_getCudaHostAllocator
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
  File "/home/user/anaconda3/envs/pytorchvideo/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1672, in _run_ninja_build
    env=env)
  File "/home/user/anaconda3/envs/pytorchvideo/lib/python3.7/subprocess.py", line 512, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "module.py", line 27, in <module>
    verbose=True)
  File "/home/user/anaconda3/envs/pytorchvideo/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1092, in load
    keep_intermediates=keep_intermediates)
  File "/home/user/anaconda3/envs/pytorchvideo/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1303, in _jit_compile
    is_standalone=is_standalone)
  File "/home/user/anaconda3/envs/pytorchvideo/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1408, in _write_ninja_file_and_build_library
    error_prefix=f"Error building extension '{name}'")
  File "/home/user/anaconda3/envs/pytorchvideo/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1682, in _run_ninja_build
    raise RuntimeError(message) from e
RuntimeError: Error building extension 'moduleing_cuda_c'

Any leads on why the first load is okay, but second is not, would be very helpful.
EDIT: Another thing is that

Running CC=g++ CXX=g++ python setup.py install in /pytorch/module_c  prints "Finished processing dependencies for module-c==0.0.0" at the end. But,
Running CC=g++ CXX=g++ python setup.py install in /pytorch/module_cuda_c prints

running install
running bdist_egg
running egg_info
creating module_cuda_c.egg-info
writing module_cuda_c.egg-info/PKG-INFO
writing dependency_links to module_cuda_c.egg-info/dependency_links.txt
writing top-level names to module_cuda_c.egg-info/top_level.txt
writing manifest file 'module_cuda_c.egg-info/SOURCES.txt'
reading manifest file 'module_cuda_c.egg-info/SOURCES.txt'
writing manifest file 'module_cuda_c.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_ext
building 'module_cuda_c' extension
error: unknown file type '.cu' (from '/home/user/pytorch/module_cuda_c/module_kernel.cu')


title: BATCHNORM toward an axis in pytorch

body:
what is the equivalence keras batch normalization layer of
model.add(BatchNormalization(axis=1, input_shape=(28,28,1))

in pytorch?


title: Attention weighted aggregation

body:
Let the tensor shown below be the representation of two sentences (batch_size = 2) composed with 3 words (max_lenght = 3) and each word being represented by vectors of dimension equal to 5 (hidden_size = 5) obtained as output from a neural network:
net_output
# tensor([[[0.7718, 0.3856, 0.2545, 0.7502, 0.5844],
#          [0.4400, 0.3753, 0.4840, 0.2483, 0.4751],
#          [0.4927, 0.7380, 0.1502, 0.5222, 0.0093]],

#         [[0.5859, 0.0010, 0.2261, 0.6318, 0.5636],
#          [0.0996, 0.2178, 0.9003, 0.4708, 0.7501],
#          [0.4244, 0.7947, 0.5711, 0.0720, 0.1106]]])

Also consider the following attention scores:
att_scores
# tensor([[0.2425, 0.5279, 0.2295],
#         [0.2461, 0.4789, 0.2751]])

Which efficient approach allows obtaining the aggregation of vectors in net_output weighted by att_scores resulting in a vector of shape (2, 5)?


title: How to change the value of a torch tensor with requires_grad=True so that the backpropogation can start again?

body:
a=torch.tensor([1,2],dtype=dtype,requires_grad=True)
b=a[0]*a[1]
b.backward()
print(a.grad)

but when I use a+=1 and want to give a another value and do another round of backpropogation, it shows that a leaf Variable that requires grad is being used in an in-place operation.
However a=a+1 seems right. What's the difference between a=a+1 and a+=1?
(solved)
I find my doubt about torch actually lies in the following small example,
a=torch.tensor([1,2],dtype=dtype,requires_grad=True)
b=a[0]*a[1]
b.backward()
print(a.grad)

print(a.requires_grad)
c=2*a[0]*a[1]
c.backward()
print(a.grad)

When I do the first round of backpropogation, I get a.grad=[2,1].
Then I want to make a fresh start and calculate the differential of c with respect to a, however, the gradient seems to accumulate. How can I eliminate this effect?


title: Add to Weight Matrix Pytorch

body:
I want to add the attention_matrix to my model's weight every time during the training. What's the grammaly correct way of extracting the weight and adding it to the weight matrix? I am new to pytorch. Thanks!
    model = linearRegression(inputDim, outputDim)

    criterion = torch.nn.MSELoss()
    optimizer = torch.optim.SGD(model.parameters(), lr=learningRate)

    for epoch in range(epochs):
        # Converting inputs and labels to Variable
        inputs = torch.tensor(x_train, dtype=torch.float)
        labels = torch.tensor(y_train, dtype=torch.float).unsqueeze(-1)

        optimizer.zero_grad()

        attention_matrix = torch.rand(4)

        outputs = model(inputs)
        loss = criterion(outputs, labels)

        loss.backward()
        optimizer.step()


title: Custom Dataset not accepting argument in PyTorch

body:
I am trying to create a custom Dataset in PyTorch using this dataset. It is of the shape (X, 785), X being number of samples and each row containing the label at index 0 and 784 pixel values. This is my code :
from torch.utils.data import Dataset
def SignMNISTDataset(Dataset):

  def __init__(self, csv_file_path, mode='Train'):
    self.labels = []
    self.pixels = []
    self.mode = mode

    data = pd.read_csv(csv_file_path).values
    if self.mode == 'Train':
      self.labels = data[:,0].tolist()
      print("Training labels acquired")

    for idx in range(len(self.labels)):
      self.pixels.append(data[idx][1:].tolist())

  def __len__(self):
    return len(self.labels)

  def __getitem__(self, idx):
    pixels = self.pixels[idx]
    if self.mode == 'Train':
      labels = self.labels[idx]
      content = {"pixels":pixels, "label":labels}
    else:
      content = {"pixels":pixels}
    return content

training_data = SignMNISTDataset('sign_mnist_train/sign_mnist_train.csv', 'Train')

On running, I get the following error :
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-46-0173199f8794> in <module>()
     27     return content
     28
---> 29 training_data = SignMNISTDataset('sign_mnist_train/sign_mnist_train.csv', 'Train')
     30 from torch.utils.data import DataLoader
     31

TypeError: SignMNISTDataset() takes 1 positional argument but 2 were given

Where exactly is this coming from? Is the mode argument somehow not being read during the object creation?
My end goal is to create a neural network for classifying sign characters, following this tutorial.
I tried explicitly mentioning the keyword mode during the object creation. This is what I got -
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-48-fd796c48dc67> in <module>()
     27     return content
     28
---> 29 training_data = SignMNISTDataset('sign_mnist_train/sign_mnist_train.csv', mode='Train')

TypeError: SignMNISTDataset() got an unexpected keyword argument 'mode'


title: from keras.preprocessing.text import one_hot equivalent in pytorch?

body:
I just started using pytorch for NLP. I found a tutorial that uses from keras.preprocessing.text import one_hot and converts text to one_hot representation given a vocabulary size.
For example:
The input is
vocab_size = 10000
sentence = ['the glass of milk',
            'the cup of tea',
            'I am a good boy']

onehot_repr = [one_hot(words, vocab_size) for words in sentence]

The output is"
[[6654, 998, 8896, 1609], [6654, 998, 1345, 879], [123, 7653, 1, 5678,7890]]

how can i perform the same procedure in pytorch and get the output like above.


title: Are variables sent to GPU automatically removed from GPU memory when they become out of scope?

body: If I send some variable x.to(DEVICE) within a function, will "x" remain in GPU memory after the function exits? What if I interrupt the function?


title: model.to(torch.device("cuda")) Error - Torch not compiled with CUDA enabled

body:
I have been trying to run the code from here in Visual Studio Code
https://www.kaggle.com/abhishek/melanoma-detection-with-pytorch?scriptVersionId=35193166
and I get an error for  model.to(torch.device("cuda"))
and it also says AssertionError: Torch not compiled with CUDA enabled
I am very sure Cuda is enabled for Torch (I have NVIDIA GeForce GTX 1650)
cuda installed
Here is a part of the code as well
training_data_path = ".."
model_path = ".."
df = pd.read_csv("..")
device = "cuda"
epochs = 50
train_bs = 32
valid_bs = 16

df_train = df[df.kfold != fold].reset_index(drop=True)
df_valid = df[df.kfold == fold].reset_index(drop=True)

model = SEResNext50_32x4d(pretrained = "imagenet")
model.to(torch.device("cuda"))

I do not understand why it doesn't recognize it.
Please help.


title: How to generate an attention mask after padding?

body:
After padding a tensor A which has 4 dimensions [4, 5, 129, 24] in second and third dimension to [4, 6, 136, 24], how can I generate its' attention mask?
I have figured out two solutions:

The first one is to create a zero-tensor A_attention that likes A_pad，and then traverse A to fill 1 to relevant position in A_attention.
The second one is to create attention mask in the padding procedure. But it seems a little troublesome because the initial tensor A has 4 dimensions.

Are there any ways to generate attention mask more efficiently after padding? Is there an API?
Thank you very much.


title: How targets should look like for UNet

body:
I have implemented UNet from scratch for medical image segmentation. Everything was working fine for testing data, where targets for 6 classes where saved on 6 channels, so mask shape was (H,W,6) and target.min(), target.max(): 0.0, 1.0
For loss function I was using torch.nn.BCEWithLogitsLoss(), and at the end I was using sigmoid to get the output:
pred = torch.sigmoid(pred)
pred = pred.data.cpu().numpy()

Now I jumped to different data available on Kaggle. The label masks are stored in an RGB format and the label information is stored in the red (R) channel. So it's basically (H,W) with 6 classes (0 is the background):
np.unique(target): [0 1 2 3 5]
np.unique(target1): [0 1 2]
np.unuque(target2): [0 1 4]
etc.

How do I proceed? I could probably create one hot encoding for target, but it would be 6 additional channels for images around 1024x1024. Is there maybe different way, maybe another loss function?


title: How can I pick a subset of a dataset in Pytorch?

body: I'm trying to run https://github.com/menardai/FashionGenAttnGAN in Google Colab on GPU with the disk size of 30 GB. The code file with its dataset files are about 15 GB. After I extract this code, the disk remains is about 14 GB. When I try to run the Pretrain.py, I can see the captions are loading but suddenly I got the "Assertion Error". Since I've not got any proper answer for the cause of this error, I think that it is because of my lack of space in Colab environment. The solution that came to my mind is that write some code to tell the model to select just 30% of the train and test datasets to load. But I don't know how to do this. Can anyone help me please?


title: Phase unwrap on GPU using pytorch

body: I wonder if there is any function like skimage.restoration.unwrap_phase in pytorch. I want to use it on GPU.


title: The file saved has nothing changed after the average calculated

body:
to get the parameters from training:
 unchanged_model_dict = train()  #unchanged_model_dict is the result after training
 new_state_dict = OrderedDict()  #initialization a dict to store the averaged result

get the parameters from file:
 net.load_state_dict(torch.load('./save_parameters_B'))  #load the file to be averaged
 model_dict_B = net.state_dict()  #get the parameters from the file

start to average the two parts:
 for key in unchanged_model_dict.keys():
     new_state_dict[key] = (model_dict_B[key] + unchanged_model_dict[key])/2

save the result calculated:
 torch.save(new_state_dict,('./save_parameters_B'))

Every time I load the file again and continue to train, can only get the same loss of the parameters in file './save_parameters_B' unchanged but not the result of new_state_dict.  I don't know why.


title: compiling tensorflow operations using specific cuda version

body:
I am having trouble with compiling .cuda and .cpp operations for tensorflow using g++.
Situation
this is the link that i wanted to compile.
This repository claims that i need to set my CUDA version 9.0.
So, i created a virtual environment using conda(:Python 2.7, TensorFlow 1.11.0, CUDA 9.0 and cuDNN 7.3)
But as i compiled it, I got warnings and error, failed compiling (below is the message).
tf_interpolate.cpp: In lambda function:
tf_interpolate.cpp:29:44: warning: ignoring return value of ‘tensorflow::Status tensorflow::shape_inference::InferenceContext::WithRank(tensorflow::shape_inference::ShapeHandle, tensorflow::int64, tensorflow::shape_inference::ShapeHandle*)’, declared with attribute warn_unused_result [-Wunused-result]
         c->WithRank(c->input(0), 3, &dims1);
                                            ^
In file included from tf_interpolate.cpp:8:0:
/home/ygkim/anaconda3/envs/pointconv_tf2/lib/python2.7/site-packages/tensorflow/include/tensorflow/core/framework/shape_inference.h:394:10: note: declared here
   Status WithRank(ShapeHandle shape, int64 rank,
          ^~~~~~~~
tf_interpolate.cpp:31:44: warning: ignoring return value of ‘tensorflow::Status tensorflow::shape_inference::InferenceContext::WithRank(tensorflow::shape_inference::ShapeHandle, tensorflow::int64, tensorflow::shape_inference::ShapeHandle*)’, declared with attribute warn_unused_result [-Wunused-result]
         c->WithRank(c->input(1), 3, &dims2);
                                            ^
In file included from tf_interpolate.cpp:8:0:
/home/ygkim/anaconda3/envs/pointconv_tf2/lib/python2.7/site-packages/tensorflow/include/tensorflow/core/framework/shape_inference.h:394:10: note: declared here
   Status WithRank(ShapeHandle shape, int64 rank,
          ^~~~~~~~
/usr/bin/ld: cannot find -lcudart
collect2: error: ld returned 1 exit status

Even though i activated my virtual environment, i still have CUDA version 10.0 which is originally set on my gpu. (I have checked it by typing 'nvcc -V')
So, I edited 'CUDA_PATH' to 'CUDA_PATH=/usr/local/cuda-10.0' in here. This time, i still got warnings, but the error disappeared. But the operations are not compiled properly so that i cannot use them (below is the message).
tf_interpolate.cpp: In lambda function:
tf_interpolate.cpp:29:44: warning: ignoring return value of ‘tensorflow::Status tensorflow::shape_inference::InferenceContext::WithRank(tensorflow::shape_inference::ShapeHandle, tensorflow::int64, tensorflow::shape_inference::ShapeHandle*)’, declared with attribute warn_unused_result [-Wunused-result]
         c->WithRank(c->input(0), 3, &dims1);
                                            ^
In file included from tf_interpolate.cpp:8:0:
/home/ygkim/anaconda3/envs/pointconv_tf2/lib/python2.7/site-packages/tensorflow/include/tensorflow/core/framework/shape_inference.h:394:10: note: declared here
   Status WithRank(ShapeHandle shape, int64 rank,
          ^~~~~~~~
tf_interpolate.cpp:31:44: warning: ignoring return value of ‘tensorflow::Status tensorflow::shape_inference::InferenceContext::WithRank(tensorflow::shape_inference::ShapeHandle, tensorflow::int64, tensorflow::shape_inference::ShapeHandle*)’, declared with attribute warn_unused_result [-Wunused-result]
         c->WithRank(c->input(1), 3, &dims2);
                                            ^
In file included from tf_interpolate.cpp:8:0:
/home/ygkim/anaconda3/envs/pointconv_tf2/lib/python2.7/site-packages/tensorflow/include/tensorflow/core/framework/shape_inference.h:394:10: note: declared here
   Status WithRank(ShapeHandle shape, int64 rank,
          ^~~~~~~~

Questions
Do i have to download the cuda 9.0 driver or any other setting went wrong?