title: Pinned memory in LibTorch

body:
I might be missing something really fundamental but I couldn't find any explanation in the documentation or online
I'm trying to copy a GPU at::Tensor to a pinned tensor on the CPU, but once I copy it, the CPU tensor is no longer pinned. I assume it just creates a new copy of the GPU tensor and assigns it, but if that's the case, how do you copy to a pre-allocated pinned memory?
My testing code:
    at::Tensor gpu = at::randn({1025,1025}, device(at::kCUDA));
    at::Tensor pinned = at::empty(gpu.sizes(), device(at::kCPU).pinned_memory(true));
    std::cout << "Is Pinned: " << std::boolalpha << pinned.is_pinned() << std::endl;
    pinned = gpu.to(at::kCPU);
    std::cout << "Is Pinned: " << std::boolalpha << pinned.is_pinned() << std::endl;

The output is
Is Pinned: true
Is Pinned: false

This also happens with torch:: instead of at::
Tested on Ubuntu 16.04 using LibTorch 1.5.0 compiled from sources


title: Loss going to NaN after few iterations

body:
In my model, the input is a graph data in the form of edge-index and the node features. After a few iterations of training on graph data, loss (EDIT: which is a combination of MSELoss function and a negative loss function i.e., L1 + (-L2)) becomes NaN. Both L1 and -L2 become NaN after around 40 iterations.
Learning rate = 0.00001. I also checked for invalid input data also, but found none.
from torch.nn.parameter import Parameter
from torch.nn.modules.module import Module
import torch.optim as optim
import torch.nn.functional as F
import torch.nn as nn
import networkx as nx
from torch_geometric.nn import GCNConv
from torch_geometric.data import Data

class Model(nn.Module):
    def __init__(self, nin, nhid1, nout, inp_l, hid_l, out_l=1):
        super(Model, self).__init__()

        self.g1 = GCNConv(in_channels= nin, out_channels= nhid1)
        self.g2 = GCNConv(in_channels= nhid1, out_channels= nout)
        self.dropout = 0.5
        self.lay1 = nn.Linear(inp_l ,hid_l)
        self.lay2 = nn.Linear(hid_l ,out_l)

    def forward(self, x, adj):
        x = F.relu(self.g1(x, adj))
        x = F.dropout(x, self.dropout, training=self.training)
        x = self.g2(x, adj)

        x = self.lay1(x)
        x = F.relu(x)
        x = self.lay2(x)
        x = F.relu(x)

        return x

The inputs to the model:
x (Tensor , optional ) – Node feature matrix with shape [num_nodes, num_node_features].
edge_index (LongTensor , optional ) – Graph connectivity in COO format with shape [2, num_edges]
Here num_nodes=1000 ; num_node_features=1 ; num_edges = 5000
GCNConv is a graph embedder returns a [num_nodes, dim] matrix. It takes in the edge-list and the features to return a matrix.
EDIT 2: Added how the loss is calculated
def train_model(epoch):
    model= Model(nin = 1, nhid1=128, nout=128, inp_l=128, hid_l=64, out_l=1).to(device)
    optimizer = optim.Adam(model.parameters(), lr=0.00001)

    model.train()
    t = time.time()
    optimizer.zero_grad()
    Y = model(features, adjacency_list)

    Y1 = func(Y) #Y1 values are calculated from Y by passing through a function func to obtain a same sized vector as Y

    loss1 = ((Y1-Y)**2).mean()  #MSE Loss function

    loss2 = -Y.abs().mean() # This loss is implemented to prevent Y values going to 0. Notice the "-" sign

    loss_train = loss1 + loss2
    loss_train.backward(retain_graph=True)
    nn.utils.clip_grad_norm_(model.parameters(), 0.5)

    optimizer.step()

    if epoch%20==0:
        print("MSE loss = ",loss1,"\t","Mean Loss = ",loss2)
        print('Epoch: {:04d}'.format(epoch+1),
                  'loss_train: {:.4f}'.format(loss_train.item()),
                  'time: {:.4f}s'.format(time.time() - t))
        print("\n\n")

    return Y


title: Why does loss continue decreasing but performance keep unchanged?

body:
I am using bert-lstm-crf model, with bert model from https://github.com/huggingface/pytorch-pretrained-BERT/ and lstm crf models are written by myself.

After training bert-lstm-crf model for 25 epochs, the performance on training set, dev set and test set keep unchanged but the loss continue decreasing. Where should I make a change?

Here is performance:

25th epoch:

tensor(10267.6279, device='cuda:0')
(0.42706720346856614, 0.4595134955014995, 0.4426966292134832)
(0.43147208121827413, 0.4271356783919598, 0.42929292929292934)
(0.4460093896713615, 0.4668304668304668, 0.4561824729891957)

26th epoch:

tensor(10219.3398, device='cuda:0')
(0.44544364508393286, 0.4951682772409197, 0.46899163642101943)
(0.4469135802469136, 0.4547738693467337, 0.45080946450809467)
(0.45871559633027525, 0.4914004914004914, 0.4744958481613286)

27th epoch:

tensor(10169.0742, device='cuda:0')
(0.44544364508393286, 0.4951682772409197, 0.46899163642101943)
(0.4469135802469136, 0.4547738693467337, 0.45080946450809467)
(0.45871559633027525, 0.4914004914004914, 0.4744958481613286)

more epochs:
lower loss with same performance:

(0.44544364508393286, 0.4951682772409197, 0.46899163642101943)
(0.4469135802469136, 0.4547738693467337, 0.45080946450809467)
(0.45871559633027525, 0.4914004914004914, 0.4744958481613286)

It is really a weird problem, I have no idea how to handle this. Any suggestion would be of great help.

Here is related code:

  for epoch in tqdm(range(200)):

  {loss = train_one_epoch(dataloader=source_train_dataloader,
  model=model, optimizer=optimizer)

  train_perf = test_one_epoch(dataloader=source_train_dataloader_for_test,
  model=model)

  dev_perf = test_one_epoch(dataloader=source_dev_dataloader, model=model)

  test_perf = test_one_epoch(dataloader=source_test_dataloader,
  model=model)

  base_result_loc = "bert_char_ps/bert_char_result"

  # store performance result

  add_model_result(
    base_result_loc,
    epoch,
    loss,
    train_perf,
    dev_perf,
    test_perf)
  }

The performance should change with loss, but now it does not


title: PyTorch LSTMCell Teacher Forcing

body:
I'm fairly new to PyTorch and I'm trying to design an 18 node LSTM using LSTMCell with Teacher Forcing. I have quite a few difficulties.

Here's my model:

class tryLSTM(nn.moduleList):
    def __init__(self, input_size, hidden_size, batch_size):
        super(tryLSTM, self).__init__()

       self.input_size = input_size
       self.hidden_size = hidden_size
       self.batch_size = batch_size

       self.lstm0 = nn.LSTMCell(input_size, hidden_size, bias=True)
       self.lstm1 = nn.LSTMCell(input_size, hidden_size, bias=True)
       self.lstm2 = nn.LSTMCell(input_size, hidden_size, bias=True)
       .........
       self.lstm17 = nn.LSTMCell(input_size, hidden_size, bias=True)
   def init_hidden(self):
       # initialize the hidden state and the cell state to zeros
       hidden = torch.zeros(self.batch_size, self.hidden_size)
       cell = torch.zeros(self.batch_size, self.hidden_size)
           return hidden, cell
   def forward(self, x, hc):
       out = []
       h_0, c_0 = hc

       h_1, c_1 = self.lstm1(x[0], h_0, c_0)
       out[0] = h_1

       h_2, c_2 = self.lstm2(x[1], h_1, c_1)
       out[1] = h_2
       ......
       h_17, c_17 = self.lstm17(x[16], h_16, c_16)
       out[16] = h_17

model = tryLSTM(input_size=128, hidden_size=128, batch_size=18)

if gpu: model.cuda()

optimizer = optim.Adam(model.parameters(), lr=0.0001)

criterion = nn.BCELoss(weight=None, reduction='mean')

here's the training loop:

def train(epoch):
model.train()
# initialize hidden and cell state
hc = model.init_hidden()
for batch_idx, (data, target) in enumerate(train_loader):
    # Zero out the gradients
    optimizer.zero_grad()
    target = data[1:]
    print(target.size())
    # Put data on GPU
    if gpu:
        data = data.cuda()
        target = target.cuda()
    # Get outputs of LSTM
    output = model(data, hc)
    print(output.size)
    # Calculate loss
    loss = criterion(output, target)
    # Calculate gradients
    loss.backward()
    # Update model parameters
    optimizer.step()
    train_loss.append(loss.item())

Q.1 I'm getting the following error:

  TypeError: forward() takes from 2 to 3 positional arguments but 4 were given

Please help, Thanks!


title: Pytorch: How to access CrossEntropyLoss() gradient?

body:
I want to modify the tensor that stores the CrossEntropyLoss() gradient, that is, P(i)-T(i). Where is it stored and how do I access it?

code:

input = torch.randn(3, 5, requires_grad=True)
input.register_hook(lambda x: print(" \n input hook: ",x))
print(input)
target = torch.empty(3, dtype=torch.long).random_(5)
print(target)

criterion = nn.CrossEntropyLoss()
criterion.requires_grad = True
loss0 = criterion(input,target)
loss0.register_hook(lambda x: print(" \n loss0 hook: ",x))
print("before backward loss0.grad :",loss0.grad)
print("loss0 :",loss0)
loss0.backward()
print("after backward loss0.grad :",loss0.grad)

output:

tensor([[-0.6149, -0.8179,  0.6084, -0.2837, -0.5316],
        [ 1.7246,  0.5348,  1.3646, -0.7148, -0.3421],
        [-0.3478, -0.6732, -0.7610, -1.0381, -0.5570]], requires_grad=True)
tensor([4, 1, 0])
before backward loss0.grad : None
loss0 : tensor(1.7500, grad_fn=<NllLossBackward>)

 loss0 hook:  tensor(1.)

 input hook:  tensor([[ 0.0433,  0.0354,  0.1472,  0.0603, -0.2862],
        [ 0.1504, -0.2876,  0.1050,  0.0131,  0.0190],
        [-0.2432,  0.0651,  0.0597,  0.0452,  0.0732]])
after backward loss0.grad : None


title: Implementation of the Dense Synthesizer

body:
I’m trying to understand the Synthesizer paper (https://arxiv.org/pdf/2005.00743.pdf 1) and there’s a description of the dense synthesizer mechanism that should replace the traditional attention model as described in the Transformer architecture.

The Dense Synthesizer is described as such:

So I tried to implement the layer and it looks like this but I’m not sure whether I’m getting it right:

class DenseSynthesizer(nn.Module):
    def __init__(self, l, d):
        super(DenseSynthesizer, self).__init__()
        self.linear1 = nn.Linear(d, l)
        self.linear2 = nn.Linear(l, l)

    def forward(self, x, v):
        # Equation (1) and (2)
        # Shape: l x l
        b = self.linear2(F.relu(self.linear1(x)))
        # Equation (3)
        # [l x l] x [l x d] -> [l x d]
        return torch.matmul(F.softmax(b), v)

Usage:

l, d = 4, 5

x, v =  torch.rand(l, d), torch.rand(l, d)

synthesis = DenseSynthesizer(l, d)
synthesis(x, v)

Example:

x and v are tensors:

x = tensor([[0.0844, 0.2683, 0.4299, 0.1827, 0.1188],
         [0.2793, 0.0389, 0.3834, 0.9897, 0.4197],
         [0.1420, 0.8051, 0.1601, 0.3299, 0.3340],
         [0.8908, 0.1066, 0.1140, 0.7145, 0.3619]])

v = tensor([[0.3806, 0.1775, 0.5457, 0.6746, 0.4505],
         [0.6309, 0.2790, 0.7215, 0.4283, 0.5853],
         [0.7548, 0.6887, 0.0426, 0.1057, 0.7895],
         [0.1881, 0.5334, 0.6834, 0.4845, 0.1960]])

And passing through a forward pass through the dense synthesis, it returns:

>>> synthesis = DenseSynthesizer(l, d)
>>> synthesis(x, v)

tensor([[0.5371, 0.4528, 0.4560, 0.3735, 0.5492],
        [0.5426, 0.4434, 0.4625, 0.3770, 0.5536],
        [0.5362, 0.4477, 0.4658, 0.3769, 0.5468],
        [0.5430, 0.4461, 0.4559, 0.3755, 0.5551]], grad_fn=<MmBackward>)

Is the implementation and understanding of the dense synthesizer correct?

Theoretically, how is that different from a multi-layered perceptron that takes in two different inputs and makes uses of it at different point in the forward propagation?


title: Pytorch: Dimensions for cross entropy is correct but somehow wrong for MSE?

body:
I was creating a program that would take in as input the Fashion MNIST set and I was tweaking around with my model to see how different parameters would change the accuracy.

One of the tweaks I made to my model was to change my model's loss function from cross entropy to MSE.

# The code above is miscellaneous training data import code

trainloader = torch.utils.data.DataLoader(trainset, batch_size = 64, shuffle = True, num_workers=4)
testloader = torch.utils.data.DataLoader(testset, batch_size = 64, shuffle = True, num_workers=4)

dataiter = iter(trainloader)
images, labels = dataiter.next()
from torch import nn, optim
import torch.nn.functional as F

model = nn.Sequential(nn.Linear(784, 128),
                      nn.ReLU(),
                      nn.Linear(128, 10),
                      nn.LogSoftmax(dim = 1)
                     )
model.to(device)

# Define the loss
criterion = nn.MSELoss()

# Define the optimizer
optimizer = optim.Adam(model.parameters(), lr = 0.001)

# Define the epochs
epochs = 5

train_losses, test_losses = [], []

for e in range(epochs):
  running_loss = 0
  for images, labels in trainloader:
    # Flatten Fashion-MNIST images into a 784 long vector
    images = images.to(device)
    labels = labels.to(device)
    images = images.view(images.shape[0], -1)

    # Training pass
    optimizer.zero_grad()

    output = model.forward(images)

    loss = criterion(output, labels)
    loss.backward()
    optimizer.step()

My model worked without any problems when using cross entropy loss, but when I changed to MSE loss, the interpreter complained and said that my tensors were different sizes and thus could not be computed.

<class 'torch.Tensor'>
torch.Size([64, 1, 28, 28])
torch.Size([64])
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-62-ec6942122f02> in <module>
     44     output = model.forward(images)
     45
---> 46     loss = criterion(output, labels)
     47     loss.backward()
     48     optimizer.step()

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--> 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/loss.py in forward(self, input, target)
    429
    430     def forward(self, input, target):
--> 431         return F.mse_loss(input, target, reduction=self.reduction)
    432
    433

/opt/conda/lib/python3.7/site-packages/torch/nn/functional.py in mse_loss(input, target, size_average, reduce, reduction)
   2213             ret = torch.mean(ret) if reduction == 'mean' else torch.sum(ret)
   2214     else:
-> 2215         expanded_input, expanded_target = torch.broadcast_tensors(input, target)
   2216         ret = torch._C._nn.mse_loss(expanded_input, expanded_target, _Reduction.get_enum(reduction))
   2217     return ret

/opt/conda/lib/python3.7/site-packages/torch/functional.py in broadcast_tensors(*tensors)
     50                 [0, 1, 2]])
     51     """
---> 52     return torch._C._VariableFunctions.broadcast_tensors(tensors)
     53
     54

RuntimeError: The size of tensor a (10) must match the size of tensor b (64) at non-singleton dimension 1

I tried reshaping my tensors and creating new arrays as placeholders for my output array, yet seem to be getting nowhere.

Why cross entropy loss works without any errors yet MSE does not?


title: What should be the size of input image for training a YOLOv3 Model Architecture CNN.?

body:
I've implemented a YOLOv3 from scratch and I plan to fine-tune using MS-COCO weights for some different data.
The dataset I've chosen has images of 720*1280 size.
When I go through the YOLOv3 paper, 1st CONV2d layer is there with filter_size =3 and stride = 1, and output size is 256*256....
Can someone give me a walkthrough for how YOLO training part works in here?


title: How to add hidden neurons to a Pytorch RNN

body:
How can I add hidden neurons to a Recurrent Neural Network in pytorch? In my understanding, torch.nn.RNN has n neurons with inputs being the input and the hidden state, where n is equal to the size of the hidden state.
How can I add additional layers before the neurons go back to the hidden state? E.g. If i only have 1 input and 1 output, but want to be able to model more complex functions?
I tried using the num_layers parameter but this just adds more layers of single neurons. I also tried using torch.nn.Sequential to stack individual RNNs with different sized inputs/outputs but this didnt work as Sequential objects dont seem to pass through additional parameters (h0, the initial hidden state).
Im trying to model f(x)=sin(x) with the initial hidden state beign the initial value of the sinewave (sin(x_0)), the inputs being x and the outputs being sin(x).


title: How to prepare the training dataset for a trainer of BERT in pytorch?

body:
The task is to detect whether a blood label is present in a text sequence using BERT for sequence classification pre-trained model.
class BloodDataset(Dataset):
    """MIMIC Blood dataset."""

    def __init__(self, arff_file):
        """
        Args:
            arff_file (string): Path to the arff file with annotations.
        """
        self.indices, self.contents, self.labels = read_arff(arff_file)
        self.labels = torch.as_tensor(self.labels)
        self.inputs = encode(self.contents)
        self.input_ids = (self.inputs['input_ids'])
        self.attention_mask = (self.inputs['attention_mask'])

    def __len__(self):
        return len(self.indices)

    def __getitem__(self, idx):
        if torch.is_tensor(idx):
            idx = idx.tolist()
        if idx in self.indices:
            sample_index = self.indices.index(idx)
            sample = {'index': idx,
                      'content': self.contents[sample_index],
                      'label': self.labels[sample_index],
                      'input_ids': self.input_ids[sample_index],
                      'attention_mask': self.attention_mask[sample_index]
                      }
            return sample
        else:
            return "Sample not found!"

Tutorial from Huggingface proposes a trainer solution:
    model = BertForSequenceClassification.from_pretrained(model_type)
    training_args = TrainingArguments(
        output_dir='./results',          # output directory
        logging_dir='./logs',            # directory for storing logs
    )
    trainer = Trainer(
        # the instantiated  Transformers model to be trained
        model=model,
        args=training_args,
        train_dataset=train_dataset,         # training dataset
        eval_dataset=test_dataset            # evaluation dataset
    )
    return trainer

https://huggingface.co/transformers/training.html#trainer
train_dataset = preprocess.BloodDataset("test_blood.arff")
trainer = train.run(train_dataset, train_dataset)
trainer.train()

yields error:
/.local/lib/python3.6/site-packages/transformers/data/data_collator.py", line 38, in <listcomp>
    features = [vars(f) for f in features]
TypeError: vars() argument must have __dict__ attribute

What's a proper dataset input for the trainer?


title: Extending the vocabulary on deployment

body:
When doing training, I initialize my embedding matrix, using the pretrained embeddings picked for words in training set vocabulary.

import torchtext as tt
contexts = tt.data.Field(lower=True, sequential=True, tokenize=tokenizer, use_vocab=True)
contexts.build_vocab(data, vectors="fasttext.en.300d",
                         vectors_cache=config["vectors_cache"])

In my model I pass contexts.vocab as parameter and initialize embeddings:

embedding_dim = vocab.vectors.shape[1]
self.embeddings = nn.Embedding(len(vocab), embedding_dim)
self.embeddings.weight.data.copy_(vocab.vectors)
self.embeddings.weight.requires_grad=False

I train my model and during training I save its 'best' state via torch.save(model, f).

Then I want to test/create demo for model in separate file for evaluation. I load the model via torch.load. How do I extend the embedding matrix to contain test vocabulary? I tried to replace embedding matrix

# data is TabularDataset with test data
contexts.build_vocab(data, vectors="fasttext.en.300d",
                     vectors_cache=config["vectors_cache"])

model.embeddings = torch.nn.Embedding(len(contexts.vocab), contexts.vocab.vectors.shape[1])
model.embeddings.weight.data.copy_(contexts.vocab.vectors)
model.embeddings.weight.requires_grad = False

But the results are terrible (almost 0 accuracy). Model was doing good during training. What is the 'correct way' of doing this?


title: Get PyTorch tensor values into python set

body:
I've a set of pytorch tensors of integers representing image masks (shape is (1, x, x)), I don't have the labels and I need to grab them manually from this pytorch tensor.

Is there an effective way to get these values into a set without having to iterate over every cell.


title: Converting spinning up policy gradient to pytorch

body:
I'm trying to learn deep reinforcement learning through OpenAI spinning up. To do this, I want to rewrite some of their code using pytorch instead of tensorflow.
Currently I'm trying to convert the code for basic policy gradient (link with explanations) and this is my code so far:

import torch
import torch.nn as nn
from torch.nn.functional import log_softmax
from torch.distributions import Categorical
import torch.optim as optim
import numpy as np
import gym
from gym.spaces import Discrete, Box

class Policy(nn.Module):

    def __init__(self, sizes, activation=nn.Tanh(), output_activation=None):
        # Build a feedforward neural network.
        super(Policy, self).__init__()
        self.layers=nn.ModuleList([nn.Linear(sizes[i],sizes[i+1]) for i in
        range(len(sizes)-1)])
        self.activation=activation
        self.output_activation=output_activation
        self.returns=[] # for R(tau) weighting in policy gradient
        self.rewards=[] # list for rewards accrued throughout ep
        self.logits=[]  # for measuring episode logits

    def forward(self,x):

        for layer in self.layers[:-1]:
            x=self.activation(layer(x))

        x=self.layers[-1](x)
        if not self.output_activation==None:
            x=self.output_activation(self.layers[-1](x))
        return x

# make action selection op (outputs int actions, sampled from policy)
def select_action(logits):
    return Categorical(logits=logits).sample()

# make loss function whose gradient, for the right data, is policy gradient
def loss(action_logits,tau_rets):
    return torch.sum(torch.dot(log_softmax(action_logits),tau_rets))

def train(env_name='CartPole-v0', hidden_sizes=[32], lr=1e-2,
      epochs=50, batch_size=5000, render=False):

# make environment, check spaces, get obs / act dims
env = gym.make(env_name)
assert isinstance(env.observation_space, Box), \
    "This example only works for envs with continuous state spaces."
assert isinstance(env.action_space, Discrete), \
    "This example only works for envs with discrete action spaces."

obs_dim = env.observation_space.shape[0]
n_acts = env.action_space.n

# make core of policy network
policy = Policy(sizes=[obs_dim]+hidden_sizes+[n_acts])

# make train op
train_op = optim.Adam(policy.parameters(), lr=lr)

# for training policy
def train_one_epoch():
    # make some empty lists for logging.
    batch_returns = []         # for measuring episode returns
    batch_lens = []            # for measuring episode lengths

    # reset episode-specific variables
    obs = torch.from_numpy(env.reset()).type(torch.FloatTensor)       # first obs comes from starting distribution
    done = False                                                      # signal from environment that episode is over
    num_obs=0                                                         # to measure the number of observations

    # render first episode of each epoch
    finished_rendering_this_epoch = False

    # collect experience by acting in the environment with current policy
    while True:
        # rendering
        if (not finished_rendering_this_epoch) and render:
            env.render()

        # act in the environment
        act_logit=policy.forward(obs)
        act = select_action(act_logit)
        tmp, reward, done, _ = env.step(act.numpy())
        obs=torch.from_numpy(tmp).type(torch.FloatTensor)
        num_obs+=1

        # save logit, reward
        policy.rewards.append(reward)
        policy.logits.append(act_logit[act].item())

        if done:
            # if episode is over, record info about episode
            ep_ret, ep_len = sum(policy.rewards), len(policy.rewards)
            batch_returns.append(ep_ret)
            batch_lens.append(ep_len)

            # the weight for each logprob(a|s) is R(tau)
            policy.returns+= [ep_ret] * ep_len

            # reset episode-specific variables
            tmp, done, policy.rewards = env.reset(), False, []
            obs=torch.from_numpy(tmp).type(torch.FloatTensor)

            # won't render again this epoch
            finished_rendering_this_epoch = True

            # end experience loop if we have enough of it
            if num_obs > batch_size:
                break

    # take a single policy gradient update step
    print (len(policy.returns),len(policy.rewards),len(policy.logits))
    batch_loss = loss(torch.tensor(policy.logits),torch.tensor(policy.returns))
    batch_loss.backward()
    return batch_loss, batch_returns, batch_lens

# training loop
for i in range(epochs):
    batch_loss, batch_rets, batch_lens = train_one_epoch()
    print('epoch: %3d \t loss: %.3f \t return: %.3f \t ep_len: %.3f'%
            (i, batch_loss, np.mean(batch_rets), np.mean(batch_lens)))

When I run train(), I get the following error:

RuntimeError                              Traceback (most recent call last)
<ipython-input-163-2da0ffaf5447> in <module>()
----> 1 train()

<ipython-input-162-560e772be08b> in train(env_name, hidden_sizes, lr, epochs,
batch_size, render)
     114     # training loop
     115     for i in range(epochs):
 --> 116         batch_loss, batch_rets, batch_lens = train_one_epoch()
     117         print('epoch: %3d \t loss: %.3f \t return: %.3f \t ep_len: %.3f'%
     118                 (i, batch_loss, np.mean(batch_rets), np.mean(batch_lens)))

<ipython-input-162-560e772be08b> in train_one_epoch()
     109         print (len(policy.returns),len(policy.rewards),len(policy.logits))
     110         batch_loss = loss(torch.tensor(policy.logits),torch.tensor(policy.returns))
 --> 111         batch_loss.backward()
     112         return batch_loss, batch_returns, batch_lens
     113

~\Anaconda3\lib\site-packages\torch\tensor.py in backward(self, gradient,
retain_graph, create_graph)
     91                 products. Defaults to ``False``.
     92         """
---> 93         torch.autograd.backward(self, gradient, retain_graph, create_graph)
     94
     95     def register_hook(self, hook):

~\Anaconda3\lib\site-packages\torch\autograd\__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
     88     Variable._execution_engine.run_backward(
     89         tensors, grad_tensors, retain_graph, create_graph,
---> 90         allow_unreachable=True)  # allow_unreachable flag
     91
     92

RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

I don't understand why this happens since my code is similar to other rl pytorch code such as this.


title: How to get unique value along 1 dim of a 2d array in pytorch?

body:
I have a 2d tensor now, which may have repeated elements along a dim, like

tmp = torch.tensor([[1,2,3,2,4],[0,5,6,7,2],[3,4,5,3,5],[7,5,6,7,7]])

I hope to get unique elements along dim=1, the result should be like this

result = [[1,2,3,4],[0,5,6,7,2],[3,4,5],[5,6,7]]

Is there a way could get the result without using for- loop?

I tried using torch.unique,  like this,

result=[]
for i in range(tmp.shape[0]):
    t = tmp[i,:]
    result.append(torch.unique(t))

It works, but time-consuming.


title: Graph input (networkx) for Dataloader in Pytorch

body:
I am newbie in Graph using Deep learning.
I'm trying to create DataLoader for Pytorch to training.

I using networkx to save my graph, then get dataset from util of  torch_geometric to have dataset for DataLoader. But I can not train my model. My code:

#import...
G = nx.karate_club_graph()
dataset = torch_geometric.utils.from_networkx(G)
dataloader = DataLoader(dataset=dataset, batch_size=32, shuffle=True)

for data in dataloader:
   print(data) # Error here: TypeError: getattr(): attribute name must be string

So how do I put Graph Input for Pytorch to train?


title: Adding a second input LSTM time series using pytorch

body:
I'm trying to follow a simple LSTM Time series prediction (https://stackabuse.com/time-series-prediction-using-lstm-with-pytorch-in-python/) tutorial using Pytorch, currently i'm using only one time series as input and it works fine, but i want to add another Time series and use it as a second input ( i want to use 2 features to predict one output) and i don't know how to do it

import torch
import torch.nn as nn

import seaborn as sns
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from pandas import read_csv, read_excel
from sklearn.preprocessing import MinMaxScaler
from torch.utils.tensorboard import SummaryWriter

fig_size = plt.rcParams["figure.figsize"]
fig_size[0] = 15
fig_size[1] = 5

#temp = read_csv('D:/downloads/t.csv')
#temp = temp.iloc[:, 0].values.astype(float)

all_pma = read_excel('D:/downloads/p.xlsx')
all_pma = all_pma.iloc[:, 0].values.astype(float)

#data = np.stack([all_pma, temp], 1)
#data = torch.tensor(data)

test_data_size = 24

train_data = all_pma[:-test_data_size]
test_data = all_pma[-test_data_size:]

scaler = MinMaxScaler(feature_range=(-1, 1))

train_data_normalized = scaler.fit_transform(train_data.reshape(-1, 1))

train_data_normalized = torch.FloatTensor(train_data_normalized).view(-1)

#data_norm = torch.FloatTensor(data_norm).view(-1)

train_window = 24

def create_inout_sequences(input_data, tw):
    inout_seq = []
    L = len(input_data)
    for i in range(L - tw):
        train_seq = input_data[i:i + tw]
        train_label = input_data[i + tw:i + tw + 1]
        inout_seq.append((train_seq, train_label))
    return inout_seq

train_inout_seq = create_inout_sequences(train_data_normalized, train_window)

class LSTM(nn.Module):
    def __init__(self, input_size=1, hidden_layer_size=100, output_size=1):
        super().__init__()
        self.hidden_layer_size = hidden_layer_size

        self.lstm = nn.LSTM(input_size, hidden_layer_size)

        self.linear = nn.Linear(hidden_layer_size, output_size)

        self.hidden_cell = (torch.zeros(1, 1, self.hidden_layer_size),
                            torch.zeros(1, 1, self.hidden_layer_size))

    def forward(self, input_seq):
        lstm_out, self.hidden_cell = self.lstm(input_seq.view(len(input_seq), 1, -1),
        self.hidden_cell)
        predictions = self.linear(lstm_out.view(len(input_seq), -1))
        return predictions[-1]

model = LSTM()
loss_function = nn.L1Loss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
print(model)

epochs = 1

for i in range(epochs):
    for seq, labels in train_inout_seq:
        optimizer.zero_grad()
        model.hidden_cell = (torch.zeros(1, 1, model.hidden_layer_size),
                             torch.zeros(1, 1, model.hidden_layer_size))

        y_pred = model(seq)

        single_loss = loss_function(y_pred, labels)
        single_loss.backward()
        optimizer.step()

    if i % 25 == 1:
        print(f'epoch: {i:3} loss: {single_loss.item():10.8f}')

print(f'epoch: {i:3} loss: {single_loss.item():10.10f}')

fut_pred = 24

test_inputs = train_data_normalized[-train_window:].tolist()
print(test_inputs)

model.eval()

for i in range(fut_pred):
    seq = torch.FloatTensor(test_inputs[-train_window:])
    with torch.no_grad():
        model.hidden = (torch.zeros(1, 1, model.hidden_layer_size),
                        torch.zeros(1, 1, model.hidden_layer_size))
        test_inputs.append(model(seq).item())

actual_predictions = scaler.inverse_transform(np.array(test_inputs[train_window:]).reshape(-1, 1))
print('actual predictions :', actual_predictions)

x = np.arange(8759, 8783, 1)

real_values = all_pma[8759:]
rv = np.array(real_values.reshape(-1, 1))

plt.title('Power Consumption')
plt.ylabel('Power Consumption')
plt.grid(True)
plt.autoscale(axis='x', tight=False)
plt.plot(x, rv, color='red')
plt.plot(x, actual_predictions, color='black')
plt.show()


title: Tensorboard add_image shows green image red

body:
I encountered a quite weird problem. After running some images through my neural network and trying to display the segmentation as follows:

print(label.shape)

  torch.Size([1, 3, 321, 321])

Now displaying the image with matplotlib shows everything correctly:

plt.imshow(fake_seg_tb[0].permute(1,2,0))

When I throw the image into tensorboard (pytorch API), I get a red version of the image, which is not what I have as pixel values.

writer_semisuper = SummaryWriter()
writer_semisuper.add_image('My Label: ', torchvision.utils.make_grid(label), some_step)

The library I am importing is (PyTorch 1.5.0):

from torch.utils.tensorboard import SummaryWriter

With the tensorboard --version beeing 2.1.0.

I have no clue why in matplotlib and in the tensor the RGB values are correct but when i display it, it is just not right.


title: RuntimeError: CUDA out of memory. Tried to allocate... but memory is empty

body:
I'm trying to run the train file from this Unet with their default hyperparameters, batch size = 1.
I have a GTX970 with 4GB and made Windows use the integrated graphics.
When I run nvidia-smi, it says that the memory of the GPU is almost free (52MiB /  4096MiB), "No running processes found " and pytorch uses the GPU not the integrated graphics
I do not understand what is using the memory:

RuntimeError: CUDA out of memory. Tried to allocate 150.00 MiB (GPU 0; 4.00 GiB total capacity; 2.77 GiB already allocated; 72.46 MiB free; 2.82 GiB reserved in total by PyTorch).


title: CUDNN error using pretrained vgg16 model

body:
I am trying to extract the activations of the last layer in a VGG16 model.
For that end I used a decorator over the model as shown below.

When I pass a cuda tensor to the model I get a CUDNN_STATUS_INTERNAL_ERROR with the following traceback.

Anyone knows where I went wrong?

traceback:

  File "/media/data1/iftachg/frame_glimpses/parse_files_to_vgg.py", line 80, in get_activation
    return model(image)
  File "/media/data1/iftachg/miniconda2/lib/python2.7/site-packages/torch/nn/modules/module.py", line 206, in __call__
    result = self.forward(*input, **kwargs)
  File "/media/data1/iftachg/frame_glimpses/partial_vgg.py", line 24, in forward
    x = self.vgg16.features(x)
  File "/media/data1/iftachg/miniconda2/lib/python2.7/site-packages/torch/nn/modules/module.py", line 206, in __call__
    result = self.forward(*input, **kwargs)
  File "/media/data1/iftachg/miniconda2/lib/python2.7/site-packages/torch/nn/modules/container.py", line 64, in forward
    input = module(input)
  File "/media/data1/iftachg/miniconda2/lib/python2.7/site-packages/torch/nn/modules/module.py", line 206, in __call__
    result = self.forward(*input, **kwargs)
  File "/media/data1/iftachg/miniconda2/lib/python2.7/site-packages/torch/nn/modules/conv.py", line 237, in forward
    self.padding, self.dilation, self.groups)
  File "/media/data1/iftachg/miniconda2/lib/python2.7/site-packages/torch/nn/functional.py", line 39, in conv2d
    return f(input, weight, bias)
RuntimeError: CUDNN_STATUS_INTERNAL_ERROR

Class:

class partial_vgg(nn.Module):

    def __init__(self):
        super(partial_vgg, self).__init__()
        self.vgg16 = models.vgg16(pretrained=True).cuda()
        for param in self.vgg16.parameters():
            param.requires_grad = False

    def forward(self, x):

        x = self.vgg16.features(x)
        x = x.view(x.size(0), -1)
        for l in list(self.vgg16.classifier.children())[:-3]:
            x = l(x)
        return x


title: Seq2seq encoder returns same hidden state for all sentences in pytorch

body:
I'm trying to implement a seq2seq encoder-decoder that from an English sentence reconstructs the same English sentence. The decoder works fine (trained with teacher forcing it reconstructs valid sentences), but the encoder returns the same hidden encoding for any sentence.

I'm using pytorch and I liked to know if is this a common problem. What is the probable cause of this?

hidden_size = 300
output_size = vocabularySize

class EncoderLSTM(nn.Module):
def __init__(self,emb_dim=vocabularySize):
    super(EncoderLSTM,self).__init__()
    self.lstm = nn.LSTM(emb_dim,hidden_size)

def forward(self,X,h):
    X=X.view(1,1,-1)
    #print(X.shape,h[0].shape,h[1].shape)
    st,h2=self.lstm(X,h)
    #print(st.shape)
    return h2

def initHidden(self):
    result = (Variable(torch.zeros(1, 1,hidden_size)),Variable(torch.zeros(1, 1, hidden_size)))
    return result

...

hidden = encoder.initHidden()
for i in range(target_length):
    encoder_input=target_variable[0][target_length-1-i]
    hidden = encoder.forward(encoder_input,hidden)

...

for i in range(1,target_length):

    pred,hidden = decoder.forward(decoder_input,hidden)

    target = target_variable[0][i].unsqueeze(0)
    values,ids = target.data.topk(1)
    target_id = Variable(torch.from_numpy(np.array([ids[0][0]])).long())

    loss+= criterion(pred[0],target_id)

...

loss/=target_length
encoder_optimizer.zero_grad()
decoder_optimizer.zero_grad()
loss.backward()
encoder_optimizer.step()
decoder_optimizer.step()


title: Can anyone solve this?

body:
torch.tensor(2,require_grad=True)

      Traceback (most recent call last):

      File "", line 1, in
       torch.tensor(2,require_grad=True)

      TypeError: tensor() got an unexpected keyword argument 'require_grad'


title: How to feed the output of a finetuned bert model as inpunt to another finetuned bert model?

body:
I finetuned two separate bert model (bert-base-uncased) on sentiment analysis and pos tagging tasks. Now, I want to feed the output of the pos tagger (batch, seqlength, hiddensize) as input to the sentiment model.The original bert-base-uncased model is in 'bertModel/' folder which contains 'model.bin' and 'config.json'. Here is my code:

class DeepSequentialModel(nn.Module):
def __init__(self, sentiment_model_file, postag_model_file, device):
    super(DeepSequentialModel, self).__init__()

    self.sentiment_model = SentimentModel().to(device)
    self.sentiment_model.load_state_dict(torch.load(sentiment_model_file, map_location=device))
    self.postag_model = PosTagModel().to(device)
    self.postag_model.load_state_dict(torch.load(postag_model_file, map_location=device))

    self.classificationLayer = nn.Linear(768, 1)

def forward(self, seq, attn_masks):
    postag_context = self.postag_model(seq, attn_masks)
    sent_context = self.sentiment_model(postag_context, attn_masks)
    logits = self.classificationLayer(sent_context)
    return logits

class PosTagModel(nn.Module):
def __init__(self,):
    super(PosTagModel, self).__init__()
    self.bert_layer = BertModel.from_pretrained('bertModel/')
    self.classificationLayer = nn.Linear(768, 43)

def forward(self, seq, attn_masks):
    cont_reps, _ = self.bert_layer(seq, attention_mask=attn_masks)
    return cont_reps

class SentimentModel(nn.Module):
def __init__(self,):
    super(SentimentModel, self).__init__()
    self.bert_layer = BertModel.from_pretrained('bertModel/')
    self.cls_layer = nn.Linear(768, 1)

def forward(self, input, attn_masks):
    cont_reps, _ = self.bert_layer(encoder_hidden_states=input, encoder_attention_mask=attn_masks)
    cls_rep = cont_reps[:, 0]
    return cls_rep

But I get the below error. I appreciate it if someone could help me. Thanks!

    cont_reps, _ = self.bert_layer(encoder_hidden_states=input, encoder_attention_mask=attn_masks)
    result = self.forward(*input, **kwargs)
    TypeError: forward() got an unexpected keyword argument 'encoder_hidden_states'


title: PyTorch: Index high dimensional tensor with two dimensional tensor

body:
Suppose I have the following tensors:

N = 2
k = 3
d = 2

L = torch.arange(N * k * d * d).view(N, k, d, d)
L
tensor([[[[ 0,  1],
          [ 2,  3]],

         [[ 4,  5],
          [ 6,  7]],

         [[ 8,  9],
          [10, 11]]],

        [[[12, 13],
          [14, 15]],

         [[16, 17],
          [18, 19]],

         [[20, 21],
          [22, 23]]]])

index = torch.Tensor([0,1,0,0]).view(N,-1)
index
tensor([[0., 1.],
        [0., 0.]])

I now would like to use the index tensor to pick out the corresponding matrices on the second dimension, i.e. I would like to get something like:

tensor([[[[ 0,  1],
          [ 2,  3]],

         [[ 4,  5],
          [ 6,  7]]],

        [[[12, 13],
          [14, 15]],

         [[[[12, 13],
          [14, 15]]])

Any idea how I could achieve this?
Thank you so much!


title: PyTorch: in which cases register_hook can return None gradient?

body:
Here is an example of using register_hook with nn.Module, also I double check gradient shape using .grad on weights after loss.backward():
class MyModule(nn.Module):
    def __init__(self):
        super().__init__()

        self.layers = nn.Sequential(
            nn.Linear(5, 6)
        )

    def forward(self, x):
        out = self.layers(x)

        h = self.layers[0].weight.register_hook(lambda grad: print(f'DEBUG: grad hook {grad.size()}'))

        return out

a = torch.ones(5)
a.requires_grad = True
print('a.size()', a.size())
m = MyModule()
b = m(a)
print('b.size()', b.size())
loss = b.mean() # Fake loss
loss.backward()
print('grad.size():', m.layers[0].weight.grad.size())

Output:
a.size() torch.Size([5])
b.size() torch.Size([6])
DEBUG: grad hook torch.Size([6, 5])
grad.size(): torch.Size([6, 5])

Here is more complex example with torch.no_grad() (so hook doesn't called):
class MyModule(nn.Module):
    def __init__(self):
        super().__init__()

        self.layers = nn.Sequential(
            nn.Linear(5, 5)
        )

    def break_grad_flow(self, x):
        with torch.no_grad():
            x = x + 1
        return x

    def forward(self, x):
        out = self.layers(x)
        out = self.break_grad_flow(out)
        out = out + x*0 # To overcome RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

        h = self.layers[0].weight.register_hook(lambda grad: print(f'DEBUG: grad hook {grad.size()}'))

        return out

def get_grad_size(grad):
    if grad is None:
        return None
    else:
        return grad.size()

a = torch.ones(5)
a.requires_grad = True
print('a.size()', a.size())
m = MyModule()
b = m(a)
print('b.size()', b.size())
loss = b.mean() # Fake loss
loss.backward()
print('grad.size():', get_grad_size(m.layers[0].weight.grad))

Output:
a.size() torch.Size([5])
b.size() torch.Size([5])
grad.size(): None

These outputs are valid and in second example hook is not called when gradient is None (https://discuss.pytorch.org/t/register-hook-not-always-working-saving-gradients/56381/6)
But in real big project where I also have torch.no_grad() somewhere in the middle of the architecture (which I can't reproduce in simple example) I get None shape from hook and valid shape from .grad after loss.backward() which looks weird (I expect hook don't called and None shape from .grad after loss.backward()), so is there any reason why hook can return None shape?
Also if I just want to check if gradient is not None what are pros and cons of using register_hook instead of .grad on weights after loss.backward()?


title: Pytorch cuda error happend(cublas_status_execution_failed)

body:
I'm training a python code.
i'm using rtx 3080, wsl2 and docker.
when i check if docker find a gpu, it does.
the code requirements are pytorch 1.2.0 and cuda 10.0
is there any way that i can run this code or fix the error?
Traceback (most recent call last):
  File "oscar/run_captioning.py", line 884, in <module>
    main()
  File "oscar/run_captioning.py", line 863, in main
    global_step, avg_loss = train(args, train_dataset, val_dataset, model, tokenizer)
  File "oscar/run_captioning.py", line 434, in train
    outputs = model(**inputs)
  File "/home/apple/anaconda3/envs/py37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/mnt/d/oscar/oscar/modeling/modeling_bert.py", line 440, in forward
    return self.encode_forward(*args, **kwargs)
  File "/mnt/d/oscar/oscar/modeling/modeling_bert.py", line 448, in encode_forward
    encoder_history_states=encoder_history_states)
  File "/home/apple/anaconda3/envs/py37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/mnt/d/oscar/oscar/modeling/modeling_bert.py", line 271, in forward
    encoder_history_states=encoder_history_states)
  File "/home/apple/anaconda3/envs/py37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/mnt/d/oscar/oscar/modeling/modeling_bert.py", line 109, in forward
    history_state)
  File "/home/apple/anaconda3/envs/py37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/mnt/d/oscar/oscar/modeling/modeling_bert.py", line 140, in forward
    head_mask, history_state)
  File "/home/apple/anaconda3/envs/py37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/mnt/d/oscar/oscar/modeling/modeling_bert.py", line 82, in forward
    self_outputs = self.self(input_tensor, attention_mask, head_mask, history_state)
  File "/home/apple/anaconda3/envs/py37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/mnt/d/oscar/oscar/modeling/modeling_bert.py", line 36, in forward
    mixed_query_layer = self.query(hidden_states)
  File "/home/apple/anaconda3/envs/py37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/apple/anaconda3/envs/py37/lib/python3.7/site-packages/torch/nn/modules/linear.py", line 87, in forward
    return F.linear(input, self.weight, self.bias)
  File "/home/apple/anaconda3/envs/py37/lib/python3.7/site-packages/torch/nn/functional.py", line 1371, in linear
    output = input.matmul(weight.t())
RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)`


title: torchfile.T7ReaderException

body:
I'm working on Windows 10. I get this issue:

 DATAPATH:  ../data/coco/test/val_captions.t7
Traceback (most recent call last):
  File "main.py", line 77, in <module>
    algo.sample(datapath, cfg.STAGE)
  File "D:\documenti\Monica\StackGAN-Pytorch\code\trainer.py", line 243, in sample
    t_file = torchfile.load(datapath)
  File "C:\Users\Utente\venv\lib\site-packages\torchfile.py", line 424, in load
    return reader.read_obj()
  File "C:\Users\Utente\venv\lib\site-packages\torchfile.py", line 386, in read_obj
    v = self.read_obj()
  File "C:\Users\Utente\venv\lib\site-packages\torchfile.py", line 386, in read_obj
    v = self.read_obj()
  File "C:\Users\Utente\venv\lib\site-packages\torchfile.py", line 414, in read_obj
    "unknown object type / typeidx: {}".format(typeidx))
torchfile.T7ReaderException: unknown object type / typeidx: -1112529805

Can anyone help me?
I installed python 3.6.8, pytorch, torchfile, torchvision, etc., and I'm working on virtualenv.


title: Calculation of activation maps spatial gradients in PyTorch

body:
I have a 4D tensor of activation maps, i.e., X of size (bs, channels, dim, dim); e.g.,

import torch
bs = 3
channels = 512
dim = 64
X = torch.rand(bs, channels, dim, dim)

I want to calculate the (x, y)-gradients of the activation maps (which are roughly seen as "images"). I think that this can be done using a 2D convolution with fixed weights. For the x-gradient, for instance,

import numpy as np
import torch.nn as nn

grad_x_weights = np.array([[1, 0, -1],
                           [2, 0, -2],
                           [1, 0, -1]])
conv_x = nn.Conv2d(512, 512, kernel_size=3, stride=1, padding=1, bias=False)
conv_x.weight = nn.Parameter(torch.from_numpy(grad_x_weights).float().expand(1, 512, 3,3))

grad_x = conv_x(X)

This will (as expected) give an output of size (3, 1, 64, 64), but what I would like to have is the gradient of the activation maps, for each activation map, so something of size (3, 512, 64, 64) in a way that grad_x[i, j, :, :] will have the x-gradient of the j-th activation map of the i-th input.

Finally, I would like to set grad_x_weights as non-learnable, so that conv_x will always calculates the gradients of the activation maps.

Thank you!


title: Attention Text Generation in Character-by-Character fashion

body:
I am searching the web for a couple of days for any text generation model that would use only attention mechanisms.

The Transformer architecture that made waves in the context of Seq-to-Seq models is actually based solely on Attention mechanisms but is mainly designed and used for translation or chat bot tasks so it doesn't fit to the purpose, but the principle does.

My question is:

Does anyone knows or heard of a text generation model based solely on Attention without any recurrence?

Thanks a lot!

P.S. I'm familiar with PyTorch.


title: Pytorch Tensorboard SummaryWriter.add_video() Produces Bad Videos

body:
I’m trying to create a video by generating a sequence of 500 matplotlib plots, converting each to a numpy array, stacking them and then passing them to a SummaryWriter()'s add_video(). When I do this, the colorbar is converted from colored to black & white, and only a small number (~3-4) of the matplotlib plots are repeated. I confirmed that my numpy arrays are correct by using them to recreate a matplotlib figure.

My input tensor has shape (B,C,T,H,W), dtype np.uint8, and values between [0, 255].

Minimal working example below. To be clear, the code runs without any errors. My problem is that the resulting video is wrong.

import matplotlib.pyplot as plt
import numpy as np
import torch
from torch.utils.tensorboard import SummaryWriter

tensorboard_writer = SummaryWriter()
print(tensorboard_writer.get_logdir())

def fig2data(fig):

    # draw the renderer
    fig.canvas.draw()

    # Get the RGB buffer from the figure
    data = np.frombuffer(fig.canvas.tostring_rgb(), dtype=np.uint8)
    data = data.reshape(fig.canvas.get_width_height()[::-1] + (3,))
    return data

size = 500
x = np.random.uniform(0, 2., size=500)
y = np.random.uniform(0, 2., size=500)
trajectory_len = len(x)
trajectory_indices = np.arange(trajectory_len)
width, height = 3, 2

# tensorboard takes video of shape (B,C,T,H,W)
video_array = np.zeros(
    shape=(1, 3, trajectory_len, height*100, width*100),
    dtype=np.uint8)

for trajectory_idx in trajectory_indices:

    fig, axes = plt.subplots(
        1,
        2,
        figsize=(width, height),
        gridspec_kw={'width_ratios': [1, 0.05]})
    fig.suptitle('Example Trajectory')
    # plot the first trajectory
    sc = axes[0].scatter(
        x=[x[trajectory_idx]],
        y=[y[trajectory_idx]],
        c=[trajectory_indices[trajectory_idx]],
        s=4,
        vmin=0,
        vmax=trajectory_len,
        cmap=plt.cm.jet)

    axes[0].set_xlim(-0.25, 2.25)
    axes[0].set_ylim(-0.25, 2.25)

    colorbar = fig.colorbar(sc, cax=axes[1])
    colorbar.set_label('Trajectory Index Number')

    # extract numpy array of figure
    data = fig2data(fig)

    # UNCOMMENT IF YOU WANT TO VERIFY THAT THE NUMPY ARRAY WAS CORRECTLY EXTRACTED
    # plt.show()
    # fig2 = plt.figure()
    # ax2 = fig2.add_subplot(111, frameon=False)
    # ax2.imshow(data)
    # plt.show()

    # close figure to save memory
    plt.close(fig=fig)

    video_array[0, :, trajectory_idx, :, :] = np.transpose(data, (2, 0, 1))

# tensorboard takes video_array of shape (B,C,T,H,W)
tensorboard_writer.add_video(
    tag='sampled_trajectory',
    vid_tensor=torch.from_numpy(video_array),
    global_step=0,
    fps=4)

print('Added video')

tensorboard_writer.close()


title: nn.LSTM() received an invalid combination of arguments

body:
I use lstm from pytorch in my code to predict time series. while i write this code
class LSTM_model(nn.Module):
    def __init__(self, input_size, output_size, hidden_size,num_layers,dropout):
        super(LSTM_model,self).__init__()

        self.input_size = input_size
        self.hidden_size = hidden_size
        self.seq_len = seq_len
        self.num_layers = num_layers
        self.dropout = dropout
        self.output_size = output_size

        # self.lstm=nn.LSTM(self.input_dim, self.hidden_dim, self.num_layers)
        self.lstm = nn.LSTM(self.input_size, self.hidden_size,self.num_layers , self.dropout, batch_first=True)
        self.fc= nn.Linear(self.hidden_size , self.output_size)

    def forward(self, x , hidden):
        x, hidden= self.lstm(x,hidden)
        x = self.fc(x)
        return x,hidden

but when I use the class, I get an error with the x,hidden=self.lstm(x,hidden) line about the internal nn.LSTM() function from PyTorch.
<ipython-input-63-211c1442b5a7> in forward(self, x, hidden)
     15
     16     def forward(self, x , hidden):
---> 17         x, hidden= self.lstm(x,hidden)
     18         x = self.fc(x)
     19         return x,hidden

D:\Anaconda\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

D:\Anaconda\lib\site-packages\torch\nn\modules\rnn.py in forward(self, input, hx)
    232         _impl = _rnn_impls[self.mode]
    233         if batch_sizes is None:
--> 234             result = _impl(input, hx, self._flat_weights, self.bias, self.num_layers,
    235                            self.dropout, self.training, self.bidirectional, self.batch_first)
    236         else:

TypeError: rnn_tanh() received an invalid combination of arguments - got (Tensor, Tensor, list, int, int, float, bool, bool, bool), but expected one of:
 * (Tensor data, Tensor batch_sizes, Tensor hx, tuple of Tensors params, bool has_biases, int num_layers, float dropout, bool train, bool bidirectional)
      didn't match because some of the arguments have invalid types: (Tensor, Tensor, !list!, !int!, !int!, !float!, !bool!, bool, bool)
 * (Tensor input, Tensor hx, tuple of Tensors params, bool has_biases, int num_layers, float dropout, bool train, bool bidirectional, bool batch_first)
      didn't match because some of the arguments have invalid types: (Tensor, Tensor, !list!, !int!, int, float, bool, bool, bool)

i called the function with this line
model = LSTM_model(input_size=1, output_size=1, hidden_size=128, num_layers=2, dropout=0).to(device)

and its called here
from tqdm.auto import tqdm
def loop_fn(mode, dataset, dataloader, model, criterion, optimizer,device):
    if mode =="train":
        model.train()
    elif mode =="test":
        model.eval()
    cost = 0
    for feature, target in tqdm(dataloader, desc=mode.title()):
        feature, target = feature.to(device), target.to(device)
        output , hidden = model(feature,None)
        loss = criterion(output,target)

        if mode =="train":
            loss.backward()
            optimizer.step()
            optimizer.zero_grad()

        cost += loss.item() * feature.shape[0]
    cost = cost / len(dataset)
    return cost

Thank You in advance


title: Tips or patterns for reshaping 4D/5D arrays, (videos to frames)

body:
I find it really hard to visualize reshaping 4D 5D arrays in numpy/pytorch. (I assume both reshape in similar patter, I am using pytorch currently!).

Like suppose I have videos with dimension [N x C x D x H x W]

(num videos x channels video x frames video x height video x width video)

Suppose I want to reshape video into frames as [N x C x H x W], how should I proceed in reshape.

Simple applying x = x.reshape(N*D, C, H, W) doesn't actually do it, it gives wrong order of elements.

Can you help me with how to do this, and any slight of intuition of pattern you used?

  On a sidenote, if i have one video (i.e suppose 1x3x100x256x256 I use :

  the following code approach:

  x = x.squeeze(0).T.reshape((100,3,256,256))[:,:,None,:,:] and it works

  great. Couldnt figure out for more than 1 video.

Thanks!

As per the request :

input = np.random.randn(N,C,D,H,W)
output = np.zeros((N*D,C,H,W))

As per the request, a for loop based code to show what I want

for h in range(N):
    for i in range(D):
        for j in range(C):
            for k in range(H):
                for l in range(W):
                    output[h*D + i,j,k,l] = input[h,j,i,k,l]


title: VS Code v.1.31: Pytorch auto-filling issue

body: VS Code v.1.31: Doesn't work auto-filling for "torch." class like: torch.randn() etc. Showed "No such module error". Although everything executes just fine without any issues. Is there any solution to fix the error without disabling it to be displayed in future? Thanks for your help in advance!


title: How to do random search for hyperparameters on different GPUs in parallel?

body:
Assuming my model uses only one GPU but Virtual Machine has 4.

How to leverage all GPUs for this code?

channel_1_range = [8, 16, 32, 64]
channel_2_range = [8, 16, 32, 64]
kernel_size_1_range = [3, 5, 7]
kernel_size_2_range = [3, 5, 7]
max_count = 40
for count in range(max_count):
    reg = 10**np.random.uniform(-3, 0)
    learning_rate = 10**np.random.uniform(-6, -3)
    channel_1 = channel_1_range[np.random.randint(low=0, high=len(channel_1_range))]
    channel_2 = channel_2_range[np.random.randint(low=0, high=len(channel_2_range))]
    kernel_size_1 = kernel_size_1_range[np.random.randint(low=0, high=len(kernel_size_1_range))]
    kernel_size_2 = kernel_size_2_range[np.random.randint(low=0, high=len(kernel_size_2_range))]

    model = ThreeLayerConvNet(in_channel=3, channel_1=channel_1, kernel_size_1=kernel_size_1, \
        channel_2=channel_2, kernel_size_2=kernel_size_2, num_classes=10)
    optimizer = optim.Adam(model.parameters(), lr=learning_rate)
    engine = Engine(loader_train=loader_train, loader_val=loader_val, device=device, dtype=dtype, print_every=100, \
        verbose=False)
    engine.train(model, optimizer, epochs=1, reg=reg)

    print("Reg: {0:.2E}, LR: {1:.2E}, Ch_1: {2:2} [{4}], Ch_2: {3:2} [{5}], Acc: {6:.2f} [{7:.2f}], {8:.2f} secs". \
         format(reg, learning_rate, channel_1, channel_2, kernel_size_1, kernel_size_2, \
               engine.accuracy, engine.accuracy_train, engine.duration))

One option is to move this to standalone console app, start N instances (N == number of GPUs) and aggregate results (one output file).

Is it possible to do it directly in Python so I could continue to use jupyter notebook?


title: Converting normalised CIELAB image tensor to RGB image

body:
I trained an image to image translation model on pytorch and the input and output images are in CIELAB color space. How do I convert this to an RGB image? Simply converting the image causes some sort of clipping and produces white patches.

out=model.forward(x)
out=torch.squeeze(out)
out=out.permute(1,2,0)
out=torch.from_numpy(out.data.numpy())
plt.imshow(out)

This doesn't produce white patches however I cant use OpenCV and convert it to RGB as the values are in range 0-1.

Now if I convert the tensor to a PIL image and then convert to RGB(0-255) some sort of clipping occurs and produces white patches which are even visible before converting to RGB

out=model.forward(x)
out=torch.squeeze(out)
out=np.asarray(transforms.ToPILImage()(out))
plt.imshow(out)

The white patches after using out=cv2.cvtColor(out, cv2.COLOR_LAB2RGB) to convert

How can I properly convert the CIELAB image to RGB?


title: How to convert RGB image to single channel image(but no grayscale) image

body:
I am having a 3 channel image, in which each channel has some information encoded in the form of colors.
I want to convert it to a single channel image with all the information retained. When I convert it into 1 channel (using grayscale) I loose all that color information and get a tensor with zero values and visualizing this image show a total black image.
So, is there any way to change the 3 channel image to 1 channel image but not grayscale ?


title: AttributeError: 'PngImageFile' object has no attribute 'apply_tfms'

body:
I am getting this error when I am predicting image class using learn.predict(img)
the image file format is png
AttributeError: PngImageFile object has no attribute apply_tfms


title: PyTorch: Dropout (?) causes different model convergence for training+validation V. training-only

body:
We are facing a very strange issue. We tested the exact same model into two different “execution” settings. In the first case, given a certain amount of epochs, we train using mini-batches for one epoch, and thereafter we test on the validation set following the same criteria. Then, we go for the next epoch. Clearly, before each training epoch, we use model.train(), and before validation we turn on model.eval().
Then we take the exact same model (same init, same dataset, same epochs, etc.) and we just train it without validation after each epoch.
Just looking at performance on training set, we observed that, even if we fixed all seeds, the two training procedures evolve differently and produce quite different metrics results (losses, accuracy, and so on). Specifically, the training-only procedure is less performing.
We also observe the following things:

It is not a reproducibility issue, because multiple executions of the
same procedure produce exactly the same results (and this is
intended);
Removing the dropout, it appears that the problem vanishes;
Batchnorm1d layer, that still has different behaviours between
training and evaluation, seems to work properly;
The issue still happens if we move from training onto TPUs to CPUs.
We are working and tried Pythorch 1.6, Pythorch nightly, XLA 1.6.

We quite lost one full day in trying to tackle this issue (and no, we cannot avoid using dropout). Does anyone have any idea about how to solve this fact?
Thank you very much!
p.s. Here the code employed for the training (on CPU).
def sigmoid(x):
    return 1 / (1 + torch.exp(-x))

def _run(model, EPOCHS, training_data_in, validation_data_in=None):

    def train_fn(train_dataloader, model, optimizer, criterion):

        running_loss = 0.
        running_accuracy = 0.
        running_tp = 0.
        running_tn = 0.
        running_fp = 0.
        running_fn = 0.

        model.train()

        for batch_idx, (ecg, spo2, labels) in enumerate(train_dataloader, 1):

            optimizer.zero_grad()

            outputs = model(ecg)

            loss = criterion(outputs, labels)

            loss.backward() # calculate the gradients
            torch.nn.utils.clip_grad_norm_(model.parameters(), 0.5)
            optimizer.step() # update the network weights

            running_loss += loss.item()
            predicted = torch.round(sigmoid(outputs.data)) # here determining the sigmoid, not included in the model

            running_accuracy += (predicted == labels).sum().item() / labels.size(0)

            fp = ((predicted - labels) == 1.).sum().item()
            fn = ((predicted - labels) == -1.).sum().item()
            tp = ((predicted + labels) == 2.).sum().item()
            tn = ((predicted + labels) == 0.).sum().item()
            running_tp += tp
            running_fp += fp
            running_tn += tn
            running_fn += fn

        retval = {'loss':running_loss / batch_idx,
                'accuracy':running_accuracy / batch_idx,
                'tp':running_tp,
                'tn':running_tn,
                'fp':running_fp,
                'fn':running_fn
                }

        return retval

    def valid_fn(valid_dataloader, model, criterion):

        running_loss = 0.
        running_accuracy = 0.
        running_tp = 0.
        running_tn = 0.
        running_fp = 0.
        running_fn = 0.

        model.eval()

        for batch_idx, (ecg, spo2, labels) in enumerate(valid_dataloader, 1):

            outputs = model(ecg)

            loss = criterion(outputs, labels)

            running_loss += loss.item()
            predicted = torch.round(sigmoid(outputs.data)) # here determining the sigmoid, not included in the model

            running_accuracy += (predicted == labels).sum().item() / labels.size(0)

            fp = ((predicted - labels) == 1.).sum().item()
            fn = ((predicted - labels) == -1.).sum().item()
            tp = ((predicted + labels) == 2.).sum().item()
            tn = ((predicted + labels) == 0.).sum().item()
            running_tp += tp
            running_fp += fp
            running_tn += tn
            running_fn += fn

        retval = {'loss':running_loss / batch_idx,
                'accuracy':running_accuracy / batch_idx,
                'tp':running_tp,
                'tn':running_tn,
                'fp':running_fp,
                'fn':running_fn
                }

        return retval

    # Defining data loaders

    train_dataloader = torch.utils.data.DataLoader(training_data_in, batch_size=BATCH_SIZE, shuffle=True, num_workers=1)

    if validation_data_in != None:
        validation_dataloader = torch.utils.data.DataLoader(validation_data_in, batch_size=BATCH_SIZE, shuffle=False, num_workers=1)

    # Defining the loss function
    criterion = nn.BCEWithLogitsLoss()

    # Defining the optimizer
    import torch.optim as optim
    optimizer = optim.AdamW(model.parameters(), lr=3e-4, amsgrad=False, eps=1e-07)

    # Training code

    metrics_history = {"loss":[], "accuracy":[], "precision":[], "recall":[], "f1":[], "specificity":[], "accuracy_bis":[], "tp":[], "tn":[], "fp":[], "fn":[],
                    "val_loss":[], "val_accuracy":[], "val_precision":[], "val_recall":[], "val_f1":[], "val_specificity":[], "val_accuracy_bis":[], "val_tp":[], "val_tn":[], "val_fp":[], "val_fn":[],}

    train_begin = time.time()
    for epoch in range(EPOCHS):
        start = time.time()

        print("EPOCH:", epoch+1)

        train_metrics = train_fn(train_dataloader=train_dataloader,
                                model=model,
                                optimizer=optimizer,
                                criterion=criterion)

        metrics_history["loss"].append(train_metrics["loss"])
        metrics_history["accuracy"].append(train_metrics["accuracy"])
        metrics_history["tp"].append(train_metrics["tp"])
        metrics_history["tn"].append(train_metrics["tn"])
        metrics_history["fp"].append(train_metrics["fp"])
        metrics_history["fn"].append(train_metrics["fn"])

        precision = train_metrics["tp"] / (train_metrics["tp"] + train_metrics["fp"]) if train_metrics["tp"] > 0 else 0
        recall = train_metrics["tp"] / (train_metrics["tp"] + train_metrics["fn"]) if train_metrics["tp"] > 0 else 0
        specificity = train_metrics["tn"] / (train_metrics["tn"] + train_metrics["fp"]) if train_metrics["tn"] > 0 else 0
        f1 = 2*precision*recall / (precision + recall) if precision*recall > 0 else 0
        metrics_history["precision"].append(precision)
        metrics_history["recall"].append(recall)
        metrics_history["f1"].append(f1)
        metrics_history["specificity"].append(specificity)

        if validation_data_in != None:
            # Calculate the metrics on the validation data, in the same way as done for training
            with torch.no_grad(): # don't keep track of the info necessary to calculate the gradients

                val_metrics = valid_fn(valid_dataloader=validation_dataloader,
                                    model=model,
                                    criterion=criterion)

                metrics_history["val_loss"].append(val_metrics["loss"])
                metrics_history["val_accuracy"].append(val_metrics["accuracy"])
                metrics_history["val_tp"].append(val_metrics["tp"])
                metrics_history["val_tn"].append(val_metrics["tn"])
                metrics_history["val_fp"].append(val_metrics["fp"])
                metrics_history["val_fn"].append(val_metrics["fn"])

                val_precision = val_metrics["tp"] / (val_metrics["tp"] + val_metrics["fp"]) if val_metrics["tp"] > 0 else 0
                val_recall = val_metrics["tp"] / (val_metrics["tp"] + val_metrics["fn"]) if val_metrics["tp"] > 0 else 0
                val_specificity = val_metrics["tn"] / (val_metrics["tn"] + val_metrics["fp"]) if val_metrics["tn"] > 0 else 0
                val_f1 = 2*val_precision*val_recall / (val_precision + val_recall) if val_precision*val_recall > 0 else 0
                metrics_history["val_precision"].append(val_precision)
                metrics_history["val_recall"].append(val_recall)
                metrics_history["val_f1"].append(val_f1)
                metrics_history["val_specificity"].append(val_specificity)

            print("  > Training/validation loss:", round(train_metrics['loss'], 4), round(val_metrics['loss'], 4))
            print("  > Training/validation accuracy:", round(train_metrics['accuracy'], 4), round(val_metrics['accuracy'], 4))
            print("  > Training/validation precision:", round(precision, 4), round(val_precision, 4))
            print("  > Training/validation recall:", round(recall, 4), round(val_recall, 4))
            print("  > Training/validation f1:", round(f1, 4), round(val_f1, 4))
            print("  > Training/validation specificity:", round(specificity, 4), round(val_specificity, 4))
        else:
            print("  > Training loss:", round(train_metrics['loss'], 4))
            print("  > Training accuracy:", round(train_metrics['accuracy'], 4))
            print("  > Training precision:", round(precision, 4))
            print("  > Training recall:", round(recall, 4))
            print("  > Training f1:", round(f1, 4))
            print("  > Training specificity:", round(specificity, 4))

        print("Completed in:", round(time.time() - start, 1), "seconds \n")

    print("Training completed in:", round((time.time()- train_begin)/60, 1), "minutes")

    # Save the model weights
    torch.save(model.state_dict(), './nnet_model.pt')

    # Save the metrics history
    torch.save(metrics_history, 'training_history')

And here is the function that initializes the model and set the seeds, called before each execution of the code of "_run":
def reinit_model():
    torch.manual_seed(42)
    np.random.seed(42)
    random.seed(42)
    net = Net() # the model
    return net


title: Force TensorRT to run on CPU, or convert trt model back to onnx?

body:
I realize this is not the intended usage of TensorRT, but I am a bit stuck so maybe there are some ideas out there.  Currently I have been provided some neural network models as TensorRT serialized engines, so-called .trt files.  These are basically models compiled and optimized from PyTorch to run on a specific GPU.
Now, this works fine since I do have a compatible GPU for development, however, for setting up CI/CD, I am having some trouble because the cloud servers on which it will be running for testing purposes only do not have adequate GPUs for this CUDA-compiled "engine".
So, I would like to force these models to run on CPU, or otherwise find some other way to make them run.  On CPU would be just fine, because I just need to run handful of inferences to check the output, it is fine if it's slow.  Again, I know this is not the intended usage of TensorRT, but I need some output from the models for integration testing.
Alternative approach
The other idea I had was maybe to convert the .trt files back to .onnx or another format that I could load into another runtime engine, or just into PyTorch or TensorFlow, but I cannot find any TensorRT tools that load an engine and write a model file.  Presumably because it is "compiled" and no longer convertible; yet, the model parameters must be in there, so does anyone know how to do such a thing?


title: Pytorch - Token embeddings using Character level LSTM

body:
I'm trying to train a neural network that classifies a sequence of words. Based on a paper I'm trying to replicate, I'd need to have both token-level embeddings and character-level embeddings of tokens.
For example, take this sentence:
The shop is open

I need 2 embeddings - one is the normal nn.Embedding layer for the token-level embedding (very simplified!):
[The, shop, is, open] -> nn.Embedding -> [4,3,7,2]

the other is a BiLSTM embedding on the character-level:
[[T,h,e], [s,h,o,p], [i,s], [o,p,e,n]] -> nn.LSTM -> [9,10,23,5]

Both of them produce word-level embeddings but on a different scale. I tried working out how to do this in PyTorch but I can't seem to do it. The only time I can do them both at the same time is if I pass the characters as one long sequence ([t,h,e,s,h,o,p,i,s,o,p,e,n]), but that will only produce one embedding.
If anyone could help that would be appreciated.


title: with torch.no_grad: AttributeError: __enter__

body:
with torch.no_grad:AttributeError: __enter__

I got this error while running pytorch code.
I have torch==0.4.1 torchvision==0.3.0, I run the code in google colab.


title: AttentionDecoderRNN without MAX_LENGTH

body:
From the PyTorch Seq2Seq tutorial, http://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#attention-decoder

We see that the attention mechanism is heavily reliant on the MAX_LENGTH parameter to determine the output dimensions of the attn -> attn_softmax -> attn_weights, i.e.

class AttnDecoderRNN(nn.Module):
    def __init__(self, hidden_size, output_size, dropout_p=0.1, max_length=MAX_LENGTH):
        super(AttnDecoderRNN, self).__init__()
        self.hidden_size = hidden_size
        self.output_size = output_size
        self.dropout_p = dropout_p
        self.max_length = max_length

        self.embedding = nn.Embedding(self.output_size, self.hidden_size)
        self.attn = nn.Linear(self.hidden_size * 2, self.max_length)
        self.attn_combine = nn.Linear(self.hidden_size * 2, self.hidden_size)
        self.dropout = nn.Dropout(self.dropout_p)
        self.gru = nn.GRU(self.hidden_size, self.hidden_size)
        self.out = nn.Linear(self.hidden_size, self.output_size)

More specifically

self.attn = nn.Linear(self.hidden_size * 2, self.max_length)

I understand that the MAX_LENGTH variable is the mechanism to reduce the no. of parameters that needs to be trained in the AttentionDecoderRNN.

If we don't have a MAX_LENGTH pre-determined. What values should we initialize the attn layer with?

Would it be the output_size? If so, then that'll be learning the attention with respect to the full vocabulary in the target language. Isn't that the real intention of the Bahdanau (2015) attention paper?


title: How can I assign pytorch tensor a matrix from numpy?

body:
Create WxW tensor:

x = Variable(torch.FloatTensor(W,W).zero_(), requires_grad=True)

Do some calculations:

x_copy = x0=np.copy(x.data.numpy())
x_upd = handleArray(x_copy)

How can I assign x data from x_upd ?


title: Pytorch tracing - How do you trace a saved weight?

body: I am trying to trace a saved weight I have so that I can load it with Catalyst. I checked the documentation on PyTorch, but I only found tracing on functions.


title: Pytorch:Network not learning at all + Weights are too low

body:
About the input. Sorry for the bad formatting. The for each two rows first row is the key and second row is the value. 18~20_ride is the label and is not included in the input. Below is one input. And train set consists of 400000 of these.

bus_route_id    station_code    latitude    longitude   6~7_ride
0               4270000         344         33.48990    126.49373
7~8_ride    8~9_ride    9~10_ride   10~11_ride  11~12_ride  6~7_takeoff
0.0         1.0         2.0         5.0         2.0         6.0
7~8_takeoff 8~9_takeoff 9~10_takeoff    10~11_takeoff   11~12_takeoff
0.0         0.0         0.0             0.0             0.0
18~20_ride  weekday dis_jejusi  dis_seoquipo
0.0         6       2.954920    26.256744

Example weights: Captured at 4th epoch. After 20 epochs of training I got much smaller values (ex. -7e-44 or 1e-55)

 2.3937e-11, -2.6920e-12, -1.0445e-11,  ..., -1.0754e-11, 1.1128e-11, -1.4814e-11

The model's prediction and target

#Target
[2.],
[0.],
[0.]

#Prediction
[1.4187],
[1.4187],
[1.4187]

MyDataset.py

from torch.utils.data import Dataset, DataLoader
import numpy as np
import pandas as pd
import torch
import os

class MyDataset(Dataset):
  def __init__(self, csv_filename):
    self.dataset = pd.read_csv(csv_filename, index_col=0)
    self.labels = self.dataset.pop("18~20_ride")
    self.dataset = self.dataset.values
    self.labels = np.reshape(self.labels.values,(-1,1))

  def __len__(self):
    return len(self.dataset)

  def __getitem__(self, idx):
    return self.dataset[idx], self.labels[idx]

Model

class Network(nn.Module):
    def __init__(self, input_num):
        super(Network, self).__init__()
        self.fc1 = nn.Sequential(
          nn.Linear(input_num, 64),
          nn.BatchNorm1d(64),
          GELU()
        )

        self.fc2 = nn.Sequential(
          nn.Linear(64, 64),
          nn.BatchNorm1d(64),
          GELU()
        )
        self.fc3 = nn.Sequential(
          nn.Linear(64, 64),
          nn.BatchNorm1d(64),
          GELU()
        )
        self.fc4 = nn.Sequential(
          nn.Linear(64, 64),
          nn.BatchNorm1d(64),
          GELU()
        )
        self.fc5 = nn.Sequential(
          nn.Linear(64, 64),
          nn.BatchNorm1d(64),
          GELU()
        )
        self.fc6 = nn.Sequential(
          nn.Linear(64, 64),
          nn.BatchNorm1d(64),
          GELU)
        )
        self.fc7 = nn.Sequential(
          nn.Linear(64, 64),
          nn.BatchNorm1d(64),
          GELU()
        )
        self.fc8 = nn.Sequential(
          nn.Linear(64, 64),
          nn.BatchNorm1d(64),
          GELU())
        )
        self.fc9 = nn.Linear(64, 1)

The training and validation

def train(model, device, train_loader, optimizer, loss_fn, log_interval, epoch):
  print("Training")
  model.train()
  for batch_idx, (data, target) in enumerate(train_loader):
    data, target = data.float().to(device), target.float().to(device)
    optimizer.zero_grad()
    output = model(data)
    loss = loss_fn(output, target)
    loss.backward()
    optimizer.step()
    if batch_idx % log_interval == 0:
        print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
            epoch, (batch_idx+1) * len(data), len(train_loader.dataset),
            100. * batch_idx / len(train_loader), loss.item()))

def validate(model, device, loader, loss_fn):
  print("\nValidating")
  model.eval()
  test_loss = 0
  correct = 0
  with torch.no_grad():
    for batch_idx, (data, target) in enumerate(loader):
      data, target = data.float().to(device), target.float().to(device)
      output = model(data)
      test_loss += loss_fn(output, target).item()  # sum up batch loss

  test_loss /= len(loader)

  print('Validation average loss: {:.4f}\n'.format(
      test_loss))
  return test_loss

Entire process of training and validation

from MyDataset import MyDataset
from torch.utils.data import DataLoader
from torch.optim.lr_scheduler import StepLR
from datetime import datetime

train_dataset_path = "/content/drive/My Drive/root/bus/dataset/train.csv"
val_dataset_path = "/content/drive/My Drive/root/bus/dataset/val.csv"
model_base_path = "/content/drive/My Drive/root/bus/models/"

model_file = "/content/drive/My Drive/root/bus/models/checkpoints/1574427776.202017.pt"

"""
Training Config
"""
epochs = 10
batch_size = 32
learning_rate = 0.5

check_interval = 4

log_interval = int(40000/batch_size)
gamma = 0.1

load_model = False
save_model = True
make_checkpoint = True
"""
End of config
"""

# Read test set
train_set = MyDataset(train_dataset_path)
train_loader = DataLoader(train_set, batch_size=batch_size, shuffle=True)
val_set = MyDataset(val_dataset_path)
val_loader = DataLoader(val_set, batch_size=1)
print("Data READY")

device = torch.device("cuda")
net = Network(19).float().to(device)
if load_model:
  net.load_state_dict(torch.load(model_file))
loss_fn = torch.nn.MSELoss()
optimizer = optim.AdamW(net.parameters(), lr=learning_rate)

best_loss = float('inf')
isAbort = False
for epoch in range(1, epochs+1):
  train(net, device, train_loader, optimizer, loss_fn, log_interval, epoch)
  val_loss = validate(net, device, val_loader, loss_fn)
  if epoch%check_interval==0:
    if make_checkpoint:
      print("Saving new checkpoint")
      torch.save(net.state_dict(), model_base_path+"checkpoints/"+str(datetime.today().timestamp())+".pt")
      """
  if val_loss < best_loss and epoch%check_interval==0:
    best_loss = val_loss
    if make_checkpoint:
      print("Saving new checkpoint")
      torch.save(net.state_dict(), model_base_path+"checkpoints/"+str(datetime.today().timestamp())+".pt")
  else:
    print("Model overfit detected. Aborting training")
    isAbort = True
    break
    """
if save_model and not isAbort:
    torch.save(net.state_dict(), model_base_path+"finals/"+str(datetime.today().timestamp())+".pt")

So I tried to train a fully connected model for a regression problem, with google colab. But it did not get trained well; The loss absolutely did not decrease. So I dug down and found out that the weights were really small. Any idea why this is happening and how I could avoid this? Thank you
I used MSE for loss and used ADaW optimizer. Below are the things I have tried

Tried other architectures (Changing number of layers sizes, Changed activation function ReLU, GELU)but the loss did not decrease
Tried changing the learning rate from 3e-1~1e-3, even tried 1
Tried other pre-processing(Used day/month/year instead of weekday) for the data
Given the label in the input data but loss did not decrease
Tried different batch_sizes(4, 10, 32, 64)
Removed batch_normalization
Other kinds of optimizer such as SGD, Adam
Training 20 epochs but loss did not decrease at all
The weights do change at loss.backward()


title: Using fastai (pytorch) in c# code, how to normalize Bitmap with mean and std?

body:
A few days ago i switched from tensorflow to fastai for my c# Project. But now i am facing a problem with my normalisation. For both i use an onnx pipeline to load the model and the data.

var onnxPipeline = mLContext.Transforms.ResizeImages(resizing: ImageResizingEstimator.ResizingKind.Fill, outputColumnName: inputName,
                                                                 imageWidth: ImageSettings.imageWidth, imageHeight: ImageSettings.imageHeight,
                                                                 inputColumnName: nameof(ImageInputData.Image))
                .Append(mLContext.Transforms.ExtractPixels(outputColumnName: inputName, interleavePixelColors: true, scaleImage: 1 / 255f))
                .Append(mLContext.Transforms.ApplyOnnxModel(outputColumnName: outputName, inputColumnName: inputName, modelFile: onnxModelPath));

var emptyData = mLContext.Data.LoadFromEnumerable(new List<ImageInputData>());
var onnxModel = onnxPipeline.Fit(emptyData);

with

    class ImageInputData
    {
        [ImageType(ImageSettings.imageHeight, ImageSettings.imageWidth)]
        public Bitmap Image { get; set; }

        public ImageInputData(byte[] image)
        {
            using (var ms = new MemoryStream(image))
            {
                Image = new Bitmap(ms);
            }
        }
        public ImageInputData(Bitmap image)
        {
            Image = image;
        }
    }

After using fastai i learned, that the models get better accuracy if the data is normalized with a specific mean and standard deviation (because i used the resnet34 model it should be means { 0.485, 0.456, 0.406 } stds = { 0.229, 0.224, 0.225 } respectively).
So the pixelvalues (for each color ofc.) have to be transformed with those values to match the trainings images. But how can i achive this in C#?
What i tried so far is:

int imageSize = 256;
double[] means = new double[] { 0.485, 0.456, 0.406 }; // used in fastai model
double[] stds = new double[] { 0.229, 0.224, 0.225 };
Bitmap bitmapImage = inputBitmap;
Image image = bitmapImage;

Color[] pixels = new Color[imageSize * imageSize];
for (int x = 0; x < bitmapImage.Width; x++)
{
   for (int y = 0; y < bitmapImage.Height; y++)
   {
      Color pixel = bitmapImage.GetPixel(x, y);
      pixels[x + y] = pixel;
      double red = (pixel.R - (means[0] * 255)) / (stds[0] * 255); // *255 to scale the mean and std values to the Bitmap
      double gre = (pixel.G - (means[1] * 255)) / (stds[1] * 255);
      double blu = (pixel.B - (means[2] * 255)) / (stds[2] * 255);
      Color pixel_n = Color.FromArgb(pixel.A, (int)red, (int)gre, (int)blu);
      bitmapImage.SetPixel(x, y, pixel_n);

   }
}

Ofcourse its not working, because the Colorvalues can`t be negative (which i realised only later).
But how can i achive this normalisation between -1 and 1 for my model in C# with the onnx-model?

Is there a different way to feed the model or to handle the normalisation?

Any help would be appreciated!


title: How to convert a string of pixels into an image with python?

body:
I am having a string of pixels of a B/W image in this manner:-
48 34 21 18 16 21 26 36 40...44 53 57 64 82 95 9

of string length 547, and I want these pixels to be converted into an image of 48*48 pixel B/W image.
Is there any python library or PyTorch function to make it possible?
Link to the Notebook


title: Using sigmoid output for cross entropy loss on Pytorch

body:
I’m trying to modify Yolo v1 to work with my task which each object has only 1 class. (e.g: an obj cannot be both cat and dog)
Due to the architecture (other outputs like localization prediction must be used regression) so sigmoid was applied to the last output of the model (f.sigmoid(nearly_last_output)). And for classification, yolo 1 also use MSE as loss. But as far as I know that MSE sometimes not going well compared to cross entropy for one-hot like what I want.
And specific: GT like this: 0 0 0 0 1 (let say we have only 5 classes in total, each only has 1 class so only one number 1 in them, of course this is class 5th in this example)
and output model at classification part: 0.1 0.1 0.9 0.2 0.1
I found some suggestion use nn.BCE / nn.BCEWithLogitsLoss but I think I should ask here for more correct since I’m not good at math and maybe I’m wrong somewhere so just ask to learn more and for sure what should I use correctly?


title: PyTorch not updating weights when using autograd in loss function

body:
I am trying to use the gradient of a network with respect to its inputs as part of my loss function. However, whenever I try to calculate it, the training proceeds but the weights do not update
import torch
import torch.optim as optim
import torch.autograd as autograd

ic = torch.rand((25, 3))
ic = torch.tensor(ic, requires_grad=True)
optimizer = optim.RMSprop([ic], lr=1e-2)

for itr in range(1, 50):
    optimizer.zero_grad()
    sol = torch.tanh(.5*torch.stack(100*[ic])) # simplified for minimal working example

    dx = sol[-1, :, 0]
    dxdxy, = autograd.grad(dx,
                           inputs=ic,
                           grad_outputs = torch.ones(ic.shape[0]), # batchwise
                           retain_graph=True
                          )
    dxdxy = torch.tensor(dxdxy, requires_grad=True)
    loss = torch.sum(dxdxy)

    loss.backward()
    optimizer.step()

    if itr % 5 == 0:
        print(loss)

What am I doing wrong?


There is 6x7 numpy array:
<class 'numpy.ndarray'>
[[[0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0]]]

Model is training normally, when it is passed to this network:
class Net(BaseFeaturesExtractor):
    def __init__(self, observation_space: gym.spaces.Box, features_dim: int = 256):
        super(Net, self).__init__(observation_space, features_dim)
        # We assume CxHxW images (channels first)
        # Re-ordering will be done by pre-preprocessing or wrapper
        # n_input_channels = observation_space.shape[0]
        n_input_channels = 1
        print("Input channels:", n_input_channels)
        self.cnn = nn.Sequential(
            nn.Conv2d(n_input_channels, 32, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.Conv2d(32, 64, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.Conv2d(64, 128, kernel_size=3, stride=1, padding=0),
            nn.ReLU(),
            nn.Flatten(),
        )

        # Compute shape by doing one forward pass
        with th.no_grad():
            n_flatten = self.cnn(
                th.as_tensor(observation_space.sample()[None]).float()
            ).shape[1]

        self.linear = nn.Sequential(nn.Linear(n_flatten, features_dim), nn.ReLU())

    def forward(self, observations: th.Tensor) -> th.Tensor:
        return self.linear(self.cnn(observations))

6x7 numpy array is modified to 3x6x7 numpy array:
<class 'numpy.ndarray'>

[[[0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0]]

 [[0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0]]

 [[0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0]
  [1 1 1 1 1 1 1]]]

After modifying the array, it is giving this error:

RuntimeError: Given groups=1, weight of size [32, 1, 3, 3], expected
input[1, 3, 6, 7] to have 1 channels, but got 3 channels instead

In order to solve this problem, I have tried to change the number of channels:
n_input_channels = 3

However, now it is showing this error:

RuntimeError: Given groups=1, weight of size [32, 3, 3, 3], expected
input[1, 1, 6, 7] to have 3 channels, but got 1 channels instead

How can I make network accept 3x6x7 array?
Update:
I provide more code to make my case clear:
6x7 input array case:
...
board = np.array(self.obs['board']).reshape(1, self.rows, self.columns)
# board = board_3layers(self.obs.mark, board)
print(type(board))
print(board)
return board

Output:
<class 'numpy.ndarray'>
[[[0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0]]]

Number of channels is 3:
n_input_channels = 1

It is working.
I am trying to modify array to 3x6x7:
board = np.array(self.obs['board']).reshape(1, self.rows, self.columns)
board = board_3layers(self.obs.mark, board)
print(type(board))
print(board)
return board

Output:
<class 'numpy.ndarray'>
[[[0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0]]

 [[0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0]]

 [[0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0]
  [1 1 1 1 1 1 1]]]

Number of channels is 3:
    n_input_channels = 3

I do not understand why it is showing this error:
RuntimeError: Given groups=1, weight of size [32, 3, 3, 3], expected input[1, 1, 6, 7] to have 3 channels, but got 1 channels instead


title: Trying to understand PyTorch SmoothL1Loss Implementation

body:
I have been trying to go through all of the loss functions in PyTorch and build them from scratch to gain a better understanding of them and I’ve run into what is either an issue with my recreation, or an issue with PyTorch’s implementation.
According to Pytorch’s documentation for SmoothL1Loss it simply states that if the absolute value of the prediction minus the ground truth is less than beta, we use the top equation. Otherwise, we use the bottom one. Please see documentation for the equations.
Below is my implementation of this in the form of a minimum test:
import torch
import torch.nn as nn
import numpy as np

predictions = torch.randn(3, 5, requires_grad=True)
target = torch.randn(3, 5)

def l1_loss_smooth(predictions, targets, beta = 1.0):

    loss = 0
    for x, y in zip(predictions, targets):
        if abs(x-y).mean() < beta:
            loss += (0.5*(x-y)**2 / beta).mean()
        else:
            loss += (abs(x-y) - 0.5 * beta).mean()

    loss = loss/predictions.shape[0]

output = l1_loss_smooth(predictions, target)
print(output)
Gives an output of:
tensor(0.7475, grad_fn=<DivBackward0>)

Now the Pytorch implementation:
loss = nn.SmoothL1Loss(beta=1.0)
output = loss(predictions, target)
Gives an output of:
tensor(0.7603, grad_fn=<SmoothL1LossBackward>)

I can’t figure out where the error in implementation lies.
Upon looking a little deeper into the smooth_l1_loss function in the _C module (file: smooth_c_loss_op.cc) I noticed that the doc string mentions that it’s a variation on Huber Loss but the documentation for SmoothL1Loss says it is Huber Loss.
So overall, just confused on how it’s implemented and whether it’s a combo of SmoothL1Loss and Huber Loss, Just Huber Loss, or something else.


title: Segmentation fault (core dumped) and how to setup new python environments

body:
How to manage the new environment when I run many projects? For instance, I was usually got the errors during training where I updated the libraries such as matplotlib and skimage and my system oupute the error like : Segmentation fault (core dumped).

It's probably to handle new environments to new project that will not conflict with my current projects. Any tutorial or ideas about this. I currently working on Ubuntu 17.10, Python 3.6, Tensorflow, and Pytorch.


title: Indices out of range for MaxUnpool2d

body:
I am trying to understand unpooling in Pytorch because I want to build a convolutional auto-encoder.

I have the following code

from torch.autograd import Variable

data = Variable(torch.rand(1, 73, 480))
pool_t = nn.MaxPool2d(2, 2, return_indices=True)
unpool_t = nn.MaxUnpool2d(2, 2)
out, indices1 = pool_t(data)
out = unpool_t(out, indices1)

But I am constantly getting this error on the last line (unpooling).

IndexError: tuple index out of range

Although the data is simulated in this example, the input has to be of that shape because of the preprocessing that has to be done.

I am fairly new to convolutional networks, but I have even tried using a ReLU and convolutional 2D layer before the pooling however, the indices always seem to be incorrect when unpooling for this shape.


title: How can I change part of a PyTorch tensor based on the values of another tensor?

body:
This question may not be clear, so please ask for clarification in the comments and I will expand.

I have the following tensors of the following shape:

mask.size() == torch.Size([1, 400])
clean_input_spectrogram.size() == torch.Size([1, 400, 161])
output.size() == torch.Size([1, 400, 161])

mask is comprised only of 0 and 1. Since it's a mask, I want to set the elements of output equal to clean_input_spectrogram where that relevant mask value is 1.

How would I do that?


title: Not compile with GPU support in detectron2

body:
enter image description here

Detectron2 ran faster- RCNN when the error, from the error, should be the network RPN part caused the error.

The GPU should be running because the backbone part did not report an error.

How can I solve this problem？


title: Pytorch version of numpy.histogram

body:
I want to convert the output label into a distribution and force the distribution into my desired values through KL divergence loss. Although I can do it through for loop or numpy.histogram, the gradient is lost and I have the following error:

RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

Is there an alternative of numpy.histogram in pytorch? Or any similar approach workable when doing loss.backward()?


I've been trying to visualize a function within my neural network using Tensorboard. However, I get this error that I can't find elsewhere on the web.

AttributeError: 'function' object has no attribute 'param_init_net'

To give a brief summary of what my model is, it is an invertible neural network so it has a forward pass and an inverse pass. I would like to visualise the forward pass and have tried to do so with this code.

import torch
from torch.utils.tensorboard import SummaryWriter

model = networks.INN(*args, **kwargs)

tb = SummaryWriter()
z0 = model.latent.sample((1000,))
tb.add_graph(model.Z_to_X, z0)
tb.close()

The full traceback is,

Traceback (most recent call last):
  File "script.py", line 135, in <module>
    tb.add_graph(model.Z_to_X, z0)
  File "~/anaconda3/lib/python3.7/site-packages/torch/utils/tensorboard/writer.py", line 552, in add_graph
    model, **kwargs)
  File "~/anaconda3/lib/python3.7/site-packages/torch/utils/tensorboard/_caffe2_graph.py", line 774, in model_to_graph_def
    nets = [model.param_init_net, model.net]
AttributeError: 'function' object has no attribute 'param_init_net'

I assume Tensorboard only works when called from the model itself rather than a function of the model? Is there a solution to this or would I have to create a forward method then wrap it around the function I want to visualize?

Thanks in advance!