title: PyTorch "NCCL error: unhandled system error" during backprop

body:
I am trying to do distributed training with PyTorch and encountered a problem.
This runtime error occurs during backprop (first batch, first epoch).

  File "/home/user/anaconda3/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/user/anaconda3/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/user/anaconda3/lib/python3.7/site-packages/mpi4py/__main__.py", line 7, in <module>
    main()
  File "/home/user/anaconda3/lib/python3.7/site-packages/mpi4py/run.py", line 196, in main
    run_command_line(args)
  File "/home/user/anaconda3/lib/python3.7/site-packages/mpi4py/run.py", line 47, in run_command_line
    run_path(sys.argv[0], run_name='__main__')
  File "/home/user/anaconda3/lib/python3.7/runpy.py", line 263, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "/home/user/anaconda3/lib/python3.7/runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "/home/user/anaconda3/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "project/main.py", line 115, in <module>
    trainer.run(config["epochs"])
  File "/home/user/project/trainer/trainer.py", line 107, in run
    self.run_epoch()
  File "/home/user/project/trainer/trainer.py", line 70, in run_epoch
    loss.backward()
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/tensor.py", line 107, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/autograd/__init__.py", line 93, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:272, unhandled system error

I use MPI for automatic rank assignment and NCCL as main back-end.
Initialization is done through file on a shared file system.
Each process uses 2 GPUs, processes run on different nodes.
Environment variable NCCL_SOCKET_IFNAME is set.
Previously, this code was working fine :(

Does anyone know why this error may occur?

UPD 13/05/19: There has been has progress.
Initially error occurred on model initialization, now on first backward pass.


title: How can I parallelize a for loop for use in PyTorch?

body:
I realize that for loops are slow with Python in general. I have some code that messes around with some tensors:

            for batch_index, mask_batch in enumerate(mask):
                mask_len = torch.sum(mask_batch).int()

                if mask_len == 0:
                    side_input = torch.zeros((max_inp_len, side_input.shape[1])).to(mask.device)
                else:

                    m_nonzero = mask_batch.nonzero().flatten()
                    first_nonzero = m_nonzero[0]
                    last_nonzero = m_nonzero[-1]

                    if side == 'left':
                        end_index = first_nonzero - 1
                        start_index = 0
                    elif side == 'right':
                        start_index = last_nonzero + 1
                        end_index = inputs[batch_index].size(1)

                    side_input = inputs[batch_index][start_index:end_index]

                    if end_index - start_index < max_inp_len:
                        pad_zeros = torch.zeros(
                            (max_inp_len - side_input.shape[0], side_input.shape[1])).to(mask.device)
                        if side == 'left':
                            side_input = torch.cat((pad_zeros, side_input), 0)
                        elif side == 'right':
                            side_input = torch.cat((side_input, pad_zeros), 0)

                side_inputs.append(side_input)

        return torch.stack(side_inputs)

I feel like this loop is REALLY slowing things down. Is there some way for me to do it without the loop?


title: Masking diagonal to a specific value with PyTorch tensors

body:
How do I fill the diagonal with a value in torch? In numpy you can do:
a = np.zeros((3, 3), int)
np.fill_diagonal(a, 5)

array([[5, 0, 0],
       [0, 5, 0],
       [0, 0, 5]])

I know that torch.diag() returns the diagonal, but how to use this as a mask to assign new values is beyond me. I haven't been able to find the answer here or in the PyTorch documentation.


title: How to Train Word Embedding With Pytorch

body:
I am trying learn and practice how to train the embedding of Vocabulary set using pytorch.

https://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html

loss = loss_function(log_probs, torch.tensor([word_to_ix[target]], dtype=torch.long))

At the above tutorial example, the loss was calculated between log_probs which is 4 x 10 tensor (4 for context words number and 10 is embedding_dimension) and the word index of target which is integer ranged from 0 to 49.

I can't understand why the code doesn't compare between its context embedding with its target embedding but just compare with its class index, which is a mere integer holding no information..

I think one must go back to the embedding parameter then call it, and compare it with its context I guess.

Is it just because it's only for the tutorial or I am misinterpreting some point?

Thanks for your help in advance.


title: Simple L1 loss in PyTorch

body:
I want to calculate L1 loss in a neural network, I came across this example at https://discuss.pytorch.org/t/simple-l2-regularization/139/2, but there are some errors in this code.

Is this really how to calculate L1 Loss in a NN or is there a simpler way?

l1_crit = nn.L1Loss()
reg_loss = 0
for param in model.parameters():
    reg_loss += l1_crit(param)

factor = 0.0005
loss += factor * reg_loss

Is this equivalent in any way to simple doing:

loss = torch.nn.L1Loss()

I assume not, because I am not passing along any network parameters. Just checking if there isn existing function to do this.


title: Training and testing CNN with pytorch. With and without model.eval()

body:
I have two questions:-

I am trying to train a convolution neural network initialized with some pre trained weights (Netwrok contains batch normalization layers as well) (taking reference from here). Before training I want to calculate a validation error using loss_fn = torch.nn.MSELoss().cuda().
And in the reference, the author is using model.eval() before calculating  the validation error. But with that result, the CNN model is off from what it should be however when I comment out model.eval(), the output is good (what it should be with pre-trained weights). What could be reason behind it as I have read on many posts that model.eval should be used before testing the model and model.train() before training it.
While calculating the validation error with pre-trained weights and above mentioned loss function what should be the batch size. Shouldn't it be 1 as i want output on each of my input, calculate error with ground truth and in the end take average of all results. If i use higher batch size error is increased. So question is can i use higher batch size if yes what should be the right way. In given code i have given err = float(loss_local) / num_samples but i observed without averaging i.e err = float(loss_local). Error is different for different batch size. I am doing this without model.eval right now.

    batch_size = 1
    data_path = 'path_to_data'
    dtype = torch.FloatTensor
    weight_file = 'path_to_weight_file'
    val_loader = torch.utils.data.DataLoader(NyuDepthLoader(data_path, val_lists),batch_size=batch_size, shuffle=True, drop_last=True)
    model = Model(batch_size)
    model.load_state_dict(load_weights(model, weight_file, dtype))
    loss_fn = torch.nn.MSELoss().cuda()
    # model.eval()

    with torch.no_grad():
        for input, depth in val_loader:
            input_var = Variable(input.type(dtype))
            depth_var = Variable(depth.type(dtype))

            output = model(input_var)

            input_rgb_image = input_var[0].data.permute(1, 2, 0).cpu().numpy().astype(np.uint8)
            input_gt_depth_image = depth_var[0][0].data.cpu().numpy().astype(np.float32)
            pred_depth_image = output[0].data.squeeze().cpu().numpy().astype(np.float32)
            print (format(type(depth_var)))
            pred_depth_image_resize = cv2.resize(pred_depth_image, dsize=(608, 456), interpolation=cv2.INTER_LINEAR)
            target_depth_transform = transforms.Compose([flow_transforms.ArrayToTensor()])
            pred_depth_image_tensor = target_depth_transform(pred_depth_image_resize)
            #both inputs to loss_fn are 'torch.Tensor'
            loss_local += loss_fn(pred_depth_image_tensor, depth_var)

            num_samples += 1
            print ('num_samples {}'.format(num_samples))

    err = float(loss_local) / num_samples
    print('val_error before train:', err)


title: How to implement contractive autoencoder in Pytorch?

body:
I'm trying to create a contractive autoencoder in Pytorch. I found this thread and tried according to that. This is the snippet I wrote based on the mentioned thread:
import datetime
import numpy as np
import torch
import torchvision
from torchvision import datasets, transforms
from torchvision.utils import save_image, make_grid
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import matplotlib.pyplot as plt
%matplotlib inline

dataset_train = datasets.MNIST(root='MNIST',
                               train=True,
                               transform = transforms.ToTensor(),
                               download=True)
dataset_test  = datasets.MNIST(root='MNIST',
                               train=False,
                               transform = transforms.ToTensor(),
                               download=True)
batch_size = 128
num_workers = 2
dataloader_train = torch.utils.data.DataLoader(dataset_train,
                                               batch_size = batch_size,
                                               shuffle=True,
                                               num_workers = num_workers,
                                               pin_memory=True)

dataloader_test = torch.utils.data.DataLoader(dataset_test,
                                               batch_size = batch_size,
                                               num_workers = num_workers,
                                               pin_memory=True)

def view_images(imgs, labels, rows = 4, cols =11):
    imgs = imgs.detach().cpu().numpy().transpose(0,2,3,1)
    fig = plt.figure(figsize=(8,4))
    for i in range(imgs.shape[0]):
        ax = fig.add_subplot(rows, cols, i+1, xticks=[], yticks=[])
        ax.imshow(imgs[i].squeeze(), cmap='Greys_r')
        ax.set_title(labels[i].item())

# now let's view some
imgs, labels = next(iter(dataloader_train))
view_images(imgs, labels,13,10)

class Contractive_AutoEncoder(nn.Module):
    def __init__(self):
        super().__init__()
        self.encoder = nn.Linear(784, 512)
        self.decoder = nn.Linear(512, 784)

    def forward(self, input):
        # flatten the input
        shape = input.shape
        input = input.view(input.size(0), -1)
        output_e = F.relu(self.encoder(input))
        output = F.sigmoid(self.decoder(output_e))
        output = output.view(*shape)
        return output_e, output

def loss_function(output_e, outputs, imgs, device):
    output_e.backward(torch.ones(output_e.size()).to(device), retain_graph=True)
    criterion = nn.MSELoss()
    assert outputs.shape == imgs.shape ,f'outputs.shape : {outputs.shape} != imgs.shape : {imgs.shape}'

    imgs.grad.requires_grad = True
    loss1 = criterion(outputs, imgs)
    print(imgs.grad)
    loss2 = torch.mean(pow(imgs.grad,2))
    loss = loss1 + loss2
    return loss

epochs = 50
interval = 2000
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = Contractive_AutoEncoder().to(device)
optimizer = optim.Adam(model.parameters(), lr =0.001)

for e in range(epochs):
    for i, (imgs, labels) in enumerate(dataloader_train):
        imgs = imgs.to(device)
        labels = labels.to(device)

        outputs_e, outputs = model(imgs)
        loss = loss_function(outputs_e, outputs, imgs,device)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        if i%interval:
            print('')

    print(f'epoch/epoechs: {e}/{epochs} loss : {loss.item():.4f} ')

For the sake of brevity I just used one layer for the encoder and the decoder. It should work regardless of number of layers in either of them obviously!
But the catch here is, aside from the fact that I don't know if this is the correct way of doing this, (calculating gradients with respect to the input), I get an error which makes the former solution wrong/not applicable.
That is:
    imgs.grad.requires_grad = True
produces the error :

AttributeError : 'NoneType' object has no attribute 'requires_grad'

I also tried the second method suggested in that thread which is as follows:
class Contractive_Encoder(nn.Module):
    def __init__(self):
        super().__init__()
        self.encoder = nn.Linear(784, 512)

    def forward(self, input):
        # flatten the input
        input = input.view(input.size(0), -1)
        output_e = F.relu(self.encoder(input))
        return output_e

class Contractive_Decoder(nn.Module):
    def __init__(self):
        super().__init__()
        self.decoder = nn.Linear(512, 784)

    def forward(self, input):
        # flatten the input
        output = F.sigmoid(self.decoder(input))
        output = output.view(-1,1,28,28)
        return output

epochs = 50
interval = 2000
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

model_enc = Contractive_Encoder().to(device)
model_dec = Contractive_Decoder().to(device)

optimizer = optim.Adam([{"params":model_enc.parameters()},
                        {"params":model_dec.parameters()}], lr =0.001)

optimizer_cond = optim.Adam(model_enc.parameters(), lr = 0.001)

criterion = nn.MSELoss()

for e in range(epochs):
    for i, (imgs, labels) in enumerate(dataloader_train):
        imgs = imgs.to(device)
        labels = labels.to(device)

        outputs_e = model_enc(imgs)
        outputs = model_dec(outputs_e)
        loss_rec = criterion(outputs, imgs)
        optimizer.zero_grad()
        loss_rec.backward()
        optimizer.step()

        imgs.requires_grad_(True)
        y = model_enc(imgs)
        optimizer_cond.zero_grad()
        y.backward(torch.ones(imgs.view(-1,28*28).size()))

        imgs.grad.requires_grad = True
        loss = torch.mean([pow(imgs.grad,2)])
        optimizer_cond.zero_grad()
        loss.backward()
        optimizer_cond.step()

        if i%interval:
            print('')

    print(f'epoch/epoechs: {e}/{epochs} loss : {loss.item():.4f} ')

but I face the error :
RuntimeError: invalid gradient at index 0 - got [128, 784] but expected shape compatible with [128, 512]

How should I go about this in Pytorch?


title: Is there function in pytorch similar to tf.contrib.distributions.percentile of tensorflow?

body: Is there a function in PyTorch that does the same as tf.contrib.distributions.percentile of Tensorflow?


****I set my model and data to the same device, but always raise the error like this:
RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same**
The following is training code**

total_epoch = 1
best_epoch = 0
training_losses = []
val_losses = []

for epoch in range(total_epoch):

    epoch_train_loss = 0

    for X, y in train_loader:

    X, y = X.cuda(), y.cuda()

    optimizer.zero_grad()
    result = model(X)
    loss = criterion(result, y).
    epoch_train_loss += loss.item()
    loss.backward()
    optimizer.step()

training_losses.append(epoch_train_loss)

epoch_val_loss = 0
correct = 0
total = 0

with torch.no_grad():
    for X, y in val_loader:

        X, y = X.cuda(), y.cuda()

        result = model(X)
        loss = criterion(result, y)
        epoch_val_loss += loss.item()
        _, maximum = torch.max(result.data, 1)
        total += y.size(0)
        correct += (maximum == y).sum().item()

val_losses.append(epoch_val_loss)
accuracy = correct/total
print("EPOCH:", epoch, ", Training Loss:", epoch_train_loss, ", Validation Loss:", epoch_val_loss, ", Accuracy: ", accuracy)

if min(val_losses) == val_losses[-1]:
    best_epoch = epoch
    checkpoint = {'model': model,
                        'state_dict': model.state_dict(),
                        'optimizer' : optimizer.state_dict()}

    torch.save(checkpoint, models_dir + '{}.pth'.format(epoch))
    print("Model saved")

when i Run the following code for detection using cv2.capture(0) .

import cvlib as cv
from PIL import Image

cap = cv2.VideoCapture(0)

font_scale=1
thickness = 2
red = (0,0,255)
green = (0,255,0)
blue = (255,0,0)
font=cv2.FONT_HERSHEY_SIMPLEX

face_cascade = cv2.CascadeClassifier( cv2.data.haarcascades +'haarcascade_frontalface_default.xml')

while(cap.isOpened()):
    ret, frame = cap.read()
    if ret == True:

    gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
    faces = face_cascade.detectMultiScale(gray, 1.4, 4)

    for (x, y, w, h) in faces:

        cv2.rectangle(frame, (x, y), (x+w, y+h), blue, 2)

        croped_img = frame[y:y+h, x:x+w]
        pil_image = Image.fromarray(croped_img, mode = "RGB")
        pil_image = train_transforms(pil_image)
        image = pil_image.unsqueeze(0)

        result = loaded_model(image)
        _, maximum = torch.max(result.data, 1)
        prediction = maximum.item()

        if prediction == 0:
            cv2.putText(frame, "Masked", (x,y - 10), font, font_scale, green, thickness)
            cv2.rectangle(frame, (x, y), (x+w, y+h), green, 2)
        elif prediction == 1:
            cv2.putText(frame, "No Mask", (x,y - 10), font, font_scale, red, thickness)
            cv2.rectangle(frame, (x, y), (x+w, y+h), red, 2)

    cv2.imshow('frame',frame)

    if (cv2.waitKey(1) & 0xFF) == ord('q'):
        break
else:
    break

cap.release()
cv2.destroyAllWindows()

about the function loaded_model declared as below

def load_checkpoint(filepath):
checkpoint = torch.load(filepath)
model = checkpoint['model']
model.load_state_dict(checkpoint['state_dict'])
for parameter in model.parameters():
parameter.requires_grad = False

return model.eval()

filepath = models_dir + str(best_epoch) + ".pth"
loaded_model = load_checkpoint(filepath)

ERROR:

---------------------------------------------------------------------------

RuntimeError                              Traceback (most recent call last)

<ipython-input-29-b3a630684f44> in <module>()
     43
     44
---> 45             result = loaded_model(image)
     46             _, maximum = torch.max(result.data, 1)
     47             prediction = maximum.item()

5 frames

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py in _conv_forward(self, input, weight)
    344                             _pair(0), self.dilation, self.groups)
    345         return F.conv2d(input, weight, self.bias, self.stride,
--> 346                         self.padding, self.dilation, self.groups)
    347
    348     def forward(self, input):

RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same

**I hope you can answer it.Thanks!**


title: What are the default values of kernel size and padding in pytorch?

body:
For an instance, take this piece of code :

conv = conv2d(in_channels = 3, out_channels = 64)

What can I expect the padding and kernel size to be, by default?


title: Pytorch Not Updating Variables in .step()

body:
I'm attempting to convert old code to PyTorch code as an experiment. Ultimately, I will be doing regression on a 10,000+ x 100 Matrix, updating weights and whatnot appropriately.

Trying to learn, I'm slowly scaling up on toy examples. I'm hitting a wall with the following sample code.

import torch
import torch.nn as nn
import torch.nn.functional as funct
from torch.autograd import Variable

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

x_data = Variable( torch.Tensor( [ [1.0, 2.0], [2.0, 3.0], [3.0, 4.0] ] ),
requires_grad=True )
y_data = Variable( torch.Tensor( [ [2.0], [4.0], [6.0] ] ) )

w = Variable( torch.randn( 2, 1, requires_grad=True ) )

b = Variable( torch.randn( 1, 1, requires_grad=True ) )

class Model(torch.nn.Module) :
    def __init__(self) :
        super( Model, self).__init__()
        self.linear = torch.nn.Linear(2,1) ## 2 features per entry. 1 output
    def forward(self, x2, w2, b2) :
        y_pred = x2 @ w2 + b2
        return y_pred

model = Model()

criterion = torch.nn.MSELoss( size_average=False )
optimizer = torch.optim.SGD( model.parameters(), lr=0.01 )

for epoch in range(10) :
    y_pred = model( x_data,w,b ) # Get prediction
    loss = criterion( y_pred, y_data ) # Calc loss
    print( epoch, loss.data.item() ) # Print loss
    optimizer.zero_grad() # Zero gradient
    loss.backward() # Calculate gradients
    optimizer.step() # Update w, b

However, doing so, my loss is always the same, and investigating shows my w and b never actually change.  I'm a bit lost at what's going on here.

Ultimately, I'd like to be able to store the results of the "new" w and b to compare across iterations and datasets.


title: How to fix "TypeError: data type not understood" in numpy when creating transformer no peak mask

body:
I'm trying to implement and train a transformer for NMT via a blog post, everything works except I can't create the no peaking mask as I get this error: "TypeError: data type not understood"

Code:

target_seq = batch.Python.transpose(0,1)
target_pad = PY_TEXT.vocab.stoi['<pad>']
target_msk = (target_seq != target_pad).unsqueeze(1)
size = target_seq.size(1) # get seq_len for matrix
nopeak_mask = np.triu(np.ones(1, size, size),
k=1).astype('uint8')
nopeak_mask = Variable(torch.from_numpy(nopeak_mask) == 0)
target_msk = target_msk & nopeak_mask

Error message:

TypeError                                 Traceback (most recent call last)
<ipython-input-36-e19167b74ba0> in <module>()
      4 target_msk = (target_seq != target_pad).unsqueeze(1)
      5 size = target_seq.size(1) # get seq_len for matrix
----> 6 nopeak_mask = np.triu(np.ones(1, size, size),
      7 k=1).astype('uint8')
      8 nopeak_mask = Variable(torch.from_numpy(nopeak_mask) == 0)

~/.local/lib/python3.6/site-packages/numpy/core/numeric.py in ones(shape, dtype, order)
    201
    202     """
--> 203     a = empty(shape, dtype, order)
    204     multiarray.copyto(a, 1, casting='unsafe')
    205     return a

TypeError: data type not understood


title: Pytorch - Distributed Data Parallel Confusion

body:
I was just looking at the DDP Tutorial:

https://pytorch.org/tutorials/intermediate/ddp_tutorial.html

According to this:

  It’s common to use torch.save and torch.load to checkpoint modules
  during training and recover from checkpoints. See SAVING AND LOADING
  MODELS for more details. When using DDP, one optimization is to save
  the model in only one process and then load it to all processes,
  reducing write overhead. This is correct because all processes start
  from the same parameters and gradients are synchronized in backward
  passes, and hence optimizers should keep setting parameters to the
  same values. If you use this optimization, make sure all processes do
  not start loading before the saving is finished. Besides, when loading
  the module, you need to provide an appropriate map_location argument
  to prevent a process to step into others’ devices. If map_location is
  missing, torch.load will first load the module to CPU and then copy
  each parameter to where it was saved, which would result in all
  processes on the same machine using the same set of devices. For more
  advanced failure recovery and elasticity support, please refer to
  TorchElastic.

I dont understand what this means. Shouldn't only one process/first GPU be saving the model? Is saving and loading how weights are shared across the processes/GPUs?


title: How to train Actor-Critic (A2C) reinforcement learning

body:
I am currently been able to train a system using Q-Learning. I will to move it to Actor_Critic (A2C) method. Please don't ask me why for this move, I have to.

I am currently borrowing the implementation from https://github.com/higgsfield/RL-Adventure-2/blob/master/1.actor-critic.ipynb

The thing is, I am keep getting a success rate of approx ~ 50% (which is basically random behavior). My game is a long episode (50 steps). Should I print out the reward, the value, or what? How should I debug this?

Here are some log:

simulation episode 2: Success, turn_count =20
loss = tensor(1763.7875)

simulation episode 3: Fail,  turn_count= 42
loss = tensor(44.6923)

simulation episode 4: Fail,  turn_count= 42
loss = tensor(173.5872)

simulation episode 5: Fail,  turn_count= 42
loss = tensor(4034.0889)

simulation episode 6: Fail,  turn_count= 42
loss = tensor(132.7567)

loss = simulation episode 7: Success, turn_count =22
loss = tensor(2099.5344)

As a general trend, I have observed that for Success episodes, the loss tends to be huge, where as for Fail episode, the loss function output tends to be small.


title: how to load images data into pytorch dataLoader?

body:
i am new to deep learning I want to use an algorithm written by pytorch, the example in pytorch tutorial is very specific . i have dataset in my Pc and i want to preprocess them .
thanks


title: PyTorch: Row-wise Dot Product

body:
Suppose I have two tensors:

a = torch.randn(10, 1000, 1, 4)
b = torch.randn(10, 1000, 6, 4)

Where the third index is the index of a vector.

I want to take the dot product between each vector in b with respect to the vector in a.

To illustrate, this is what I mean:

dots = torch.Tensor(10, 1000, 6, 1)
for b in range(10):
     for c in range(1000):
           for v in range(6):
            dots[b,c,v] = torch.dot(b[b,c,v], a[b,c,0])

How would I achieve this using torch functions?


title: Explanation behind actor-critic algorithm in pytorch example?

body:
Pytorch provides a good example of using actor-critic to play Cartpole in the OpenAI gym environment.

I'm confused about several of their equations in the code snippet found at https://github.com/pytorch/examples/blob/master/reinforcement_learning/actor_critic.py#L67-L79:

saved_actions = model.saved_actions
value_loss = 0
rewards = []
for r in model.rewards[::-1]:
    R = r + args.gamma * R
    rewards.insert(0, R)
rewards = torch.Tensor(rewards)
rewards = (rewards - rewards.mean()) / (rewards.std() + np.finfo(np.float32).eps)
for (action, value), r in zip(saved_actions, rewards):
    action.reinforce(r - value.data.squeeze())
    value_loss += F.smooth_l1_loss(value, Variable(torch.Tensor([r])))
optimizer.zero_grad()
final_nodes = [value_loss] + list(map(lambda p: p.action, saved_actions))
gradients = [torch.ones(1)] + [None] * len(saved_actions)
autograd.backward(final_nodes, gradients)
optimizer.step()

What do r and value mean in this case? Why do they run REINFORCE on the action space with the reward equal to r - value? And why do they try to set the value so that it matches r?

Thanks for your help!


title: PyTorch: How to parallelize over multiple GPU using multiprocessing.pool

body:
I have the following code which I am trying to parallelize over multiple GPUs in PyTorch:
import numpy as np
import torch
from torch.multiprocessing import Pool

X = np.array([[1, 3, 2, 3], [2, 3, 5, 6], [1, 2, 3, 4]])
X = torch.DoubleTensor(X).cuda()

def X_power_func(j):
    X_power = X**j
    return X_power

if __name__ == '__main__':
  with Pool(processes = 2) as p:   # Parallelizing over 2 GPUs
    results = p.map(X_power_func, range(4))

results

But when I ran the code, I am getting this error:
---------------------------------------------------------------------------
RemoteTraceback                           Traceback (most recent call last)
RemoteTraceback:
"""
Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
  File "<ipython-input-35-6529ab6dac60>", line 11, in X_power_func
    X_power = X**j
RuntimeError: CUDA error: initialization error
"""

The above exception was the direct cause of the following exception:

RuntimeError                              Traceback (most recent call last)
<ipython-input-35-6529ab6dac60> in <module>()
     14 if __name__ == '__main__':
     15   with Pool(processes = 1) as p:
---> 16     results = p.map(X_power_func, range(8))
     17
     18 results

1 frames
/usr/lib/python3.6/multiprocessing/pool.py in get(self, timeout)
    642             return self._value
    643         else:
--> 644             raise self._value
    645
    646     def _set(self, i, obj):

RuntimeError: CUDA error: initialization error

Where have I gone wrong? Any help would really be appreciated.


title: python child process exit unexpectedly with exit code -9

body:
I have a PyTorch script with 16 processes. Following is a code snippet from the main process:

procs = [mp.Process(target=self.worker_wrapper, args=(i, )) for i in range(self.n_workers)]
for p in procs: p.start()
while True:
    time.sleep(60)
    for i, p in enumerate(procs):
        self.logger.info('Check: id %d, exitcode %s, alive %s' % (
            i, str(p.exitcode), str(p.is_alive())))

the worker_wrapper is this:

def worker_wrapper(self, id):
    try:
        self.worker(id)
    except Exception as e:
        self.logger.info(e)
        self.logger.error(traceback.format_exc())

The worker will do some stuff and should never stop. It works well at beginning, but usually after 1 days running, some child processes (not all) exit unexpectedly, I get the output like this:

[2017-06-15 08:45:21,540] Check: id 13, exitcode -9, alive False

And it didn't catch any exception. I tried to search exitcode -9 but didn't get anything useful. Does anyone meet this before? Or do you have any idea about how to debug this? I'm sorry I can't provide a minimal reproducible code snippet, because it usually takes one day to reproduce. And I'm using python2.7


title: Using Softmax Activation function after calculating loss from BCEWithLogitLoss (Binary Cross Entropy + Sigmoid activation)

body:
I am going through a Binary Classification tutorial using PyTorch and here, the last layer of the network is torch.Linear()  with just one neuron. (Makes Sense) which will give us a single neuron.  as pred=network(input_batch)

After that the choice of Loss function is loss_fn=BCEWithLogitsLoss()  (which is numerically stable than using the softmax first and then calculating loss) which will apply Softmax function to the output of last layer to give us a probability.  so after that, it'll calculate the binary cross entropy to minimize the loss.

loss=loss_fn(pred,true)

My concern is that after all this, the author used torch.round(torch.sigmoid(pred))

Why would that be? I mean I know it'll get the prediction probabilities in the range [0,1] and then round of the values with default threshold of 0.5.

Isn't it better to use the sigmoid once after the last layer within the network rather using a softmax and a sigmoid at 2 different places given it's a binary classification??

Wouldn't it be better to just

out = self.linear(batch_tensor)
return self.sigmoid(out)

and then calculate the BCE loss and use the argmax() for checking accuracy??

I am just curious that can it be a valid strategy?


title: Pytorch - Efficient Elementwise Multiply?

body:
I have a tensor of 3D Points of [100x3]

I have a vector of weights of [100x1], which needs to be element wise multiplied into the X,Y,Z coordinates.

Currently, I am creating a new vector W where I stack the [100x3] element with repetition into a [100x3] tensor, before i do an element wise multiply.

I need to do this many times, and this is way too slow and memory intensive. Is there a better way?


title: How to add augmented images to original dataset using Pytorch?

body:
From my understanding, RandomHorizontalFlip etc. replace image rather than adding new images to dataset. How do I increase my dataset size by adding augmented images to dataset using PyTorch?

I have gone through the links posted & haven't found a solution. I want to increase the data size by adding flipped/rotated images - but the post addresses the in-place processing of images.

Thanks.


title: How can I give a name to each module in ModuleList?

body:
I have the following component in my model:

feedfnn = []
for task_name, num_class in self.tasks:
    if self.config.nonlinear_fc:
        ffnn = nn.Sequential(OrderedDict([
            ('dropout1', nn.Dropout(self.config.dropout_fc)),
            ('dense1', nn.Linear(self.config.nhid * self.num_directions * 8, self.config.fc_dim)),
            ('tanh', nn.Tanh()),
            ('dropout2', nn.Dropout(self.config.dropout_fc)),
            ('dense2', nn.Linear(self.config.fc_dim, self.config.fc_dim)),
            ('tanh', nn.Tanh()),
            ('dropout3', nn.Dropout(self.config.dropout_fc)),
            ('dense3', nn.Linear(self.config.fc_dim, num_class))
        ]))
    else:
        ffnn = nn.Sequential(OrderedDict([
            ('dropout1', nn.Dropout(self.config.dropout_fc)),
            ('dense1', nn.Linear(self.config.nhid * self.num_directions * 8, self.config.fc_dim)),
            ('dropout2', nn.Dropout(self.config.dropout_fc)),
            ('dense2', nn.Linear(self.config.fc_dim, self.config.fc_dim)),
            ('dropout3', nn.Dropout(self.config.dropout_fc)),
            ('dense3', nn.Linear(self.config.fc_dim, num_class))
        ]))
    feedfnn.append(ffnn)
self.ffnn = nn.ModuleList(feedfnn)

When I print my model, I get the description of the above component as:

(ffnn): ModuleList (
(0): Sequential (
  (dropout1): Dropout (p = 0)
  (dense1): Linear (4096 -> 512)
  (dropout2): Dropout (p = 0)
  (dense2): Linear (512 -> 512)
  (dropout3): Dropout (p = 0)
  (dense3): Linear (512 -> 2)
)
(1): Sequential (
  (dropout1): Dropout (p = 0)
  (dense1): Linear (4096 -> 512)
  (dropout2): Dropout (p = 0)
  (dense2): Linear (512 -> 512)
  (dropout3): Dropout (p = 0)
  (dense3): Linear (512 -> 3)
)
(2): Sequential (
  (dropout1): Dropout (p = 0)
  (dense1): Linear (4096 -> 512)
  (dropout2): Dropout (p = 0)
  (dense2): Linear (512 -> 512)
  (dropout3): Dropout (p = 0)
  (dense3): Linear (512 -> 3)
)
)

Can I put a specific name like (task1): Sequential, (task2): Sequential instead of (0): Sequential, (1): Sequential?


title: cannot install pip install torchvision

body:
I am using python 3.8.3 version, I tried to install torchvision and torch module and faced this error, none of them are installed. The error comes as ERROR:could not find a version that satisfies the requirement torch==1.4.0(from versions:0.1.2,0.1.2,post1,0.1.2.post2)
Error:No matching distribution found for torch==1.4.0


title: RNN is not training (PyTorch)

body:
I can't get what I am doing wrong when training RNN. I am trying to train RNN for AND operation on sequences (to learn how it works on simple task).
But my network is not learning, loss stays the same and it can't event overfit the model.
Can you please help me to find the problem?

Data I am using:

data = [
    [1, 1, 1, 1, 0, 0, 1, 1, 1],
    [1, 1, 1, 1],
    [0, 0, 1, 1],
    [0, 0, 0, 0, 0, 0, 0],
    [1, 1, 1, 1, 1, 1, 1],
    [1, 1],
    [0],
    [1],
    [1, 0]]
labels = [
    0,
    1,
    0,
    0,
    1,
    1,
    0,
    1,
    0
]

Code for NN:

class AndRNN(nn.Module):
def __init__(self):
    super(AndRNN, self).__init__()
    self.rnn = nn.RNN(1, 10, 5)
    self.fc = nn.Sequential(
        nn.Linear(10, 30),
        nn.Linear(30, 2)
    )

def forward(self, input, hidden):
    x, hidden = self.rnn(input, hidden)
    x = self.fc(x[-1])
    return x, hidden

def initHidden(self):
    return Variable(torch.zeros((5, 1, 10)))

Training loop:

criterion = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(net.parameters(), lr=0.001, momentum=0.9)

correct = 0
for e in range(20):
for i in range(len(data)):
    tensor = torch.FloatTensor(data[i]).view(-1, 1, 1)
    label = torch.LongTensor([labels[i]])
    hidden = net.initHidden()
    optimizer.zero_grad()

    out, hidden = net(Variable(tensor), Variable(hidden.data))

    _, l = torch.topk(out, 1)
    if label[0] == l[0].data[0]:
        correct += 1

    loss = criterion(out, Variable(label))
    loss.backward()
    optimizer.step()

    print("Loss ", loss.data[0], "Accuracy ", (correct / (i + 1)))

Shape for tensor will be (sequence_len, 1 (which is batch size), 1), that is correct according to the PyTorch docs for RNN


title: PyTorch Binary Classification - same network structure, 'simpler' data, but worse performance?

body:
To get to grips with PyTorch (and deep learning in general) I started by working through some basic classification examples. One such example was classifying a non-linear dataset created using sklearn (full code available as notebook here)

n_pts = 500
X, y = datasets.make_circles(n_samples=n_pts, random_state=123, noise=0.1, factor=0.2)
x_data = torch.FloatTensor(X)
y_data = torch.FloatTensor(y.reshape(500, 1))

This is then accurately classified using a pretty basic neural net

class Model(nn.Module):
    def __init__(self, input_size, H1, output_size):
        super().__init__()
        self.linear = nn.Linear(input_size, H1)
        self.linear2 = nn.Linear(H1, output_size)

    def forward(self, x):
        x = torch.sigmoid(self.linear(x))
        x = torch.sigmoid(self.linear2(x))
        return x

    def predict(self, x):
        pred = self.forward(x)
        if pred >= 0.5:
            return 1
        else:
            return 0

As I have an interest in health data I then decided to try and use the same network structure to classify some a basic real-world dataset. I took heart rate data for one patient from here, and altered it so all values > 91 would be labelled as anomalies (e.g. a 1 and everything <= 91 labelled a 0). This is completely arbitrary, but I just wanted to see how the classification would work. The complete notebook for this example is here.

What is not intuitive to me is why the first example reaches a loss of 0.0016 after 1,000 epochs, whereas the second example only reaches a loss of 0.4296 after 10,000 epochs

Perhaps I am being naive in thinking that the heart rate example would be much easier to classify. Any insights to help me understand why this is not what I am seeing would be great!


title: How to save LambdaLR scheduler in pytorch with lambda function?

body:
Running pytorch 0.4.1 with python 3.6 I encountered this problem:
I cannot torch.save my learning rate scheduler because python won't pickle a lambda function:

lambda1 = lambda epoch: epoch // 30
scheduler = LambdaLR(optimizer, lr_lambda=lambda1)
torch.save(scheduler.state_dict(), 'scheduler.pth.tar')

results with an error

PicklingError: Can't pickle <function <lambda> at 0x7f7583fe92f0>:
attribute lookup <lambda> on __main__ failed

How can I save my scheduler?

I know that using a proper function instead of a lambda function for lambda1 can be saved, but I need a lambda function because I want to be able to control this function when it is defined (for instance I want to be able to change the fixed 30 in the denominator).
How can this be done and still allows me to save the scheduler?


title: How do I use tensorboard with pytorch?

body:
I was following the pytorch tensorboard tutorial: https://pytorch.org/tutorials/intermediate/tensorboard_tutorial.html.

But I can't even start because of the following error:

from torch.utils.tensorboard import SummaryWriter

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
~/apps/anaconda3/envs/torch/lib/python3.7/site-packages/torch/utils/tensorboard/__init__.py in <module>
      1 try:
----> 2     from tensorboard.summary.writer.record_writer import RecordWriter  # noqa F401
      3 except ImportError:

ModuleNotFoundError: No module named 'tensorboard'

During handling of the above exception, another exception occurred:

ImportError                               Traceback (most recent call last)
<ipython-input-4-c8ffdef1cfab> in <module>
----> 1 from torch.utils.tensorboard import SummaryWriter
      2
      3 # default `log_dir` is "runs" - we'll be more specific here
      4 writer = SummaryWriter('runs/fashion_mnist_experiment_1')

~/apps/anaconda3/envs/torch/lib/python3.7/site-packages/torch/utils/tensorboard/__init__.py in <module>
      2     from tensorboard.summary.writer.record_writer import RecordWriter  # noqa F401
      3 except ImportError:
----> 4     raise ImportError('TensorBoard logging requires TensorBoard with Python summary writer installed. '
      5                       'This should be available in 1.14 or above.')
      6 from .writer import FileWriter, SummaryWriter  # noqa F401

ImportError: TensorBoard logging requires TensorBoard with Python summary writer installed. This should be available in 1.14 or above.

I installed pytorch 1.14 via conda. Am I supposed to install something else?


title: Pytorch: Convert 2D-CNN model to tflite

body:
I'd like to convert a model (eg Mobilenet V2) from pytorch to tflite in order to run it on a mobile device.

Has anyone managed to do so?

All I found, was a method that uses ONNX to convert the model into an inbetween state. However, this seems not to work properly, as Tensorflow expects a NHWC-channel order whereas onnx and pytorch work with NCHW channel order.

There is a discussion on github, however in my case the conversion worked without complaints until a "frozen tensorflow graph model", after trying to convert the model further to tflite, it complains about the channel order being wrong...

Here is my code so far:

import torch
import torch.onnx
import onnx
from onnx_tf.backend import prepare

# Create random input
input_data = torch.randn(1,3,224,224)

# Create network
model = torch.hub.load('pytorch/vision:v0.6.0', 'mobilenet_v2', pretrained=True)
model.eval()

# Forward Pass
output = model(input_data)

# Export model to onnx
filename_onnx = "mobilenet_v2.onnx"
filename_tf = "mobilenet_v2.pb"

torch.onnx.export(model, input_data, filename_onnx)

# Export model to tensorflow
onnx_model = onnx.load(filename_onnx)
tf_rep = prepare(onnx_model)
tf_rep.export_graph(filename_tf)

All working without errors until here (ignoring many tf warnings). Then I look up the names of the input and output tensors using netron ("input.1" and "473").

Finally I apply my usual tf-graph to tf-lite conversion script from bash:

tflite_convert \
    --output_file=mobilenet_v2.tflite \
    --graph_def_file=mobilenet_v2.pb \
    --input_arrays=input.1 \
    --output_arrays=473

My configuration:

torch                1.6.0.dev20200508 (needs pytorch-nightly to work with mobilenet V2 from torch.hub)
tensorflow-gpu       1.14.0
onnx                 1.6.0
onnx-tf              1.5.0

Here is the exact error message I'm getting from tflite:

Unexpected value for attribute 'data_format'. Expected 'NHWC'
Fatal Python error: Aborted

UPDATE:
Updating my configuration:

torch                1.6.0.dev20200508
tensorflow-gpu       2.2.0
onnx                 1.7.0
onnx-tf              1.5.0

using

tflite_convert \
    --output_file=mobilenet_v2.tflite \
    --graph_def_file=mobilenet_v2.pb \
    --input_arrays=input.1 \
    --output_arrays=473 \
    --enable_v1_converter  # <-- needed for conversion of frozen graphs

leading to another error:

Exception: <unknown>:0: error: loc("convolution"): 'tf.Conv2D' op is neither a custom op nor a flex op

Update:
Here is an onnx model of mobilenet v2 loaded via netron:

Here  is a gdrive link to my converted onnx and pb file


title: Batchnorm2d Pytorch - Why pass number of channels to batchnorm?

body: Why do I need to pass the previous nummber of channels to the batchnorm? The batchnorm should normalize over each datapoint in the batch, why does it need to have the number of channels then ?


title: How to load Omniglot on Pytorch

body:
I'm trying to do some experiments on the Omniglot dataset, and I saw that Pytorch implemented it. I've run the command

from torchvision.datasets import Omniglot

but I have no idea on how to actually load the dataset. Is there a way to open it equivalent to how we open MNIST? Something like the following:

train_dataset = dsets.MNIST(root='./data',
                            train=True,
                            transform=transforms.ToTensor(),
                            download=True)

The final goal is to be able to open training and test set separately and run experiments on it.


title: How is cross entropy loss work in pytorch?

body:
I am experimenting with some of the pytorch codes. With cross entropy loss I found some interesting results and I have used both binary cross entropy loss and cross entropy loss of pytorch.
import torch
import torch.nn as nn

X = torch.tensor([[1,0],[1,0],[0,1],[0,1]],dtype=torch.float)
softmax = nn.Softmax(dim=1)

bce_loss = nn.BCELoss()
ce_loss= nn.CrossEntropyLoss()

pred = softmax(X)

bce_loss(X,X) # tensor(0.)
bce_loss(pred,X) # tensor(0.3133)
bce_loss(pred,pred) # tensor(0.5822)

ce_loss(X,torch.argmax(X,dim=1)) # tensor(0.3133)

I expected the cross entropy loss for the same input and output to be zero. Here X, pred and torch.argmax(X,dim=1) are same/similar with some transformations. This reasoning only worked for bce_loss(X,X) # tensor(0.) where-else all other resulted in a loss greater than zero. I speculated the output for bce_loss(pred,X), bce_loss(pred,pred) and ce_loss(X,torch.argmax(X,dim=1)) should be zero.
What is the mistake here?


title: Giving a time series input to Pytorch-LSTM using a Batch size of 128

body:
I have a dataset of n training samples (Where X matrix = n * 30 * 20) where 30 is the number of sequences, 20 is the number of features. I am using categorical cross-entropy loss since this is a classification task. My input matrix is of shape 10000 *30 * 20 and target matrix is 1000 * 1. I was wondering what is the correct way to give the input to the LSTM. My entire code is attached below.

import torch.nn as nn
import torch.nn.functional as F
import torch.utils.data as data
import torch.backends.cudnn as cudnn
import torch.optim as optim
import torchvision.transforms as transforms
import torch.utils.data as utils
from torch.utils.data import DataLoader
import torch
from torch.autograd import Variable

class MyData(data.Dataset):
    def __init__(self,X,y,transform=None, target_transform=None):
        self.data = X
        self.targets = y
        self.transform = transform
        self.target_transform= target_transform
    def __getitem__(self, index):
        item, target = self.data[index], self.targets[index]
        item = torch.FloatTensor(item)
        target = torch.LongTensor([target])
        return item, target

    def __len__(self):
        return len(self.data)

class MyPredictor(nn.Module):
    def __init__(self, num_blocks=1, input_dim=22, batch_size=128, hidden_dim1=64, hidden_dim2=128, GPU=True):
        super(MyPredictor, self).__init__()
        self.batch_size= batch_size
        self.hidden_dim1=hidden_dim1
        self.hidden_dim2=hidden_dim2
        self.GPU=GPU

        self.lstm1 = nn.LSTM(input_dim,self.hidden_dim1,num_layers=2)
        self.lstm2 = nn.LSTM(self.hidden_dim1, self.hidden_dim2)
        self.fc1 = nn.Linear(self.hidden_dim2, 256)
        self.fc2=nn.Linear(256,1)

    if self.GPU:
        self.hx1 = Variable(torch.zeros(30, self.batch_size, self.hidden_dim1).cuda())
        self.cx1 = Variable(torch.zeros(30, self.batch_size, self.hidden_dim1).cuda())
        self.hx2 = Variable(torch.zeros(30, self.batch_size, self.hidden_dim2).cuda())
        self.cx2 = Variable(torch.zeros(30, self.batch_size, self.hidden_dim2).cuda())
    else:
        self.hx1 = Variable(torch.zeros(30, self.batch_size, self.hidden_dim1))
        self.cx1 = Variable(torch.zeros(30, self.batch_size, self.hidden_dim1))
        self.hx2 = Variable(torch.zeros(30, self.batch_size, self.hidden_dim2))
        self.cx2 = Variable(torch.zeros(30, self.batch_size, self.hidden_dim2))

        self.hidden1=(self.hx1,self.cx1)
        self.hidden2=(self.hx2,self.cx2)

    def forward(self, x):

        out1, self.hidden1 = self.lstm1(x, self.hidden1)
        out2, self.hidden2 = self.lstm1(x, self.hidden2)
        out3=self.fc1(out2)
        out3= F.relu(out3)
        out4=self.fc2(out3)
        out=F.relu(out4)

        return out

Moreover, I am initializing the hidden and cell states as 1 * Batch_size * hidden size. However since I have 30 as sequence length, should it be 30 * batch_size * hidden size?

def train(epoch):
    print('\nEpoch: %d' % epoch)
    net.train()
    train_loss = 0
    correct = 0
    total = 0
    store=[]
    for batch_idx, (inputs, targets) in enumerate(trainloader):
        inputs, targets = inputs.to(device), targets.to(device)
        optimizer.zero_grad()
        outputs = net(inputs)
        loss = criterion(outputs, targets)
        loss.backward()
        optimizer.step()

        train_loss += loss.item()
        _, predicted = outputs.max(1)
        total += targets.size(0)
        correct += predicted.eq(targets).sum().item()

#         progress_bar(batch_idx, len(trainloader), 'Loss: %.3f | Acc: %.3f%% (%d/%d)'
#             % (train_loss/(batch_idx+1), 100.*correct/total, correct, total))
    print("Train->Epoch: ",epoch," Accuracy: ",100.*correct/total," Loss ",train_loss/len(trainloader))
    return[100.*correct/total,train_loss/len(trainloader)]

def test(epoch):
    global best_acc
    criterion = nn.CrossEntropyLoss()
    net.eval()
    test_loss = 0
    correct = 0
    total = 0
    with torch.no_grad():
        for batch_idx, (inputs, targets) in enumerate(testloader):
            inputs, targets = inputs.to(device), targets.to(device)
            outputs = net(inputs)
            loss = criterion(outputs, targets)

            test_loss += loss.item()
            _, predicted = outputs.max(1)
            total += targets.size(0)
            correct += predicted.eq(targets).sum().item()

#             progress_bar(batch_idx, len(testloader), 'Loss: %.3f | Acc: %.3f%% (%d/%d)'
#                 % (test_loss/(batch_idx+1), 100.*correct/total, correct, total))

    print("Test->Epoch: ",epoch," Accuracy: ",100.*correct/total," Loss ",test_loss/len(testloader))

I am loading my inputs as below:

from sklearn.externals import joblib
import numpy as np

#See if Gpu is available
device = 'cuda' if torch.cuda.is_available() else 'cpu'
net=MyPredictor()
net = net.to(device)
GPU=False
if device == 'cuda':
    net = torch.nn.DataParallel(net)
    cudnn.benchmark = True
    GPU= True

#optimizer = optim.SGD(model.parameters(), lr=0.1)
optimizer = optim.Adam(net.parameters(), lr=0.001)
criterion = nn.CrossEntropyLoss()

#Load the datset

X, y = joblib.load("/app/dhanush/lstm_training_data.pkl")
X_train=np.array(X[0:30000])
X_test=np.array(X[30000:])
y_train= np.array(y[0:30000])
y_test= np.array(y[30000:])

print (X_train.shape)
print (y_train.shape)

trainset = MyData(X_train,y_train)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=128, shuffle=False, num_workers=2)

testset = MyData(X_test, y_test)
testloader = torch.utils.data.DataLoader(testset, batch_size=128, shuffle=False, num_workers=2)

for epoch in range(0, 21):
    train(epoch)
    test(epoch)

I get the below shape mismatch error. I definitely think it has something to do with my input tensors or the way I am initializing my hidden states. Also, I am not sure if my input to the first fully connected layer is correct. Should I give only the last hidden state output from the LSTM 2 (self.hx2[-1]?)

On reading other posts, I see that for 2D inputs, you are sometimes asked to reshape to (sequence * batch size * feature) for direct input to LSTM. How does this work ?

And most important I get the below error :

RuntimeError: Expected hidden[0] size (1, 30, 64), got (30, 128, 64)

I understand this is all a lot but my goal here is to provide as much information as possible. I have read other posts, sample GitHub codes but most of them deal with using an embedding later which I don't want to use.


title: Changing thresholds in the Sigmoid Activation in Neural Networks

body:
Hi I am new to machine learning and I have a query about changing thresholds for sigmoid function.

I know Sigmoid function's value is in the range [0;1], 0.5 is taken as a threshold, if h(theta) < 0.5 we assume that it's value is 0, if h(theta) >= 0.5 then it's 1.

Thresholds are used only on the output layer of the network and it's only when classifying. So, if you're trying to classify between 3 classes can you give different thresholds for each class (0.2,0.4,0.4 - for each class)? Or can you specify a different threshold overall, like 0.8? I am unsure how to define this in the code below. Any guidance is appreciated.

# Hyper Parameters
input_size = 14
hidden_size = 40
hidden_size2 = 30
num_classes = 3
num_epochs = 600
batch_size = 34
learning_rate = 0.01

class Net(torch.nn.Module):
    def __init__(self, n_input, n_hidden, n_hidden2, n_output):
        super(Net, self).__init__()
        # define linear hidden layer output
        self.hidden = torch.nn.Linear(n_input, n_hidden)
        self.hidden2 = torch.nn.Linear(n_hidden, n_hidden2)
        # define linear output layer output
        self.out = torch.nn.Linear(n_hidden, n_output)

    def forward(self, x):
        """
            In the forward function we define the process of performing
            forward pass, that is to accept a Variable of input
            data, x, and return a Variable of output data, y_pred.
        """
        # get hidden layer input
        h_input1 = self.hidden(x)
        # define activation function for hidden layer
        h_output1 = torch.sigmoid(h_input1)

        # get hidden layer input
        h_input2 = self.hidden2(h_output1)
        # define activation function for hidden layer
        h_output2 = torch.sigmoid(h_input2)

        # get output layer output
        out = self.out(h_output2)

        return out

net = Net(input_size, hidden_size, hidden_size, num_classes)

criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(net.parameters(), lr=learning_rate)

all_losses = []

for epoch in range(num_epochs):
    total = 0
    correct = 0
    total_loss = 0
    for step, (batch_x, batch_y) in enumerate(train_loader):
        X = batch_x
        Y = batch_y.long()

    # Forward + Backward + Optimize
    optimizer.zero_grad()  # zero the gradient buffer
    outputs = net(X)
    loss = criterion(outputs, Y)
    all_losses.append(loss.item())
    loss.backward()
    optimizer.step()

    if epoch % 50 == 0:
        _, predicted = torch.max(outputs, 1)
        # calculate and print accuracy
        total = total + predicted.size(0)
        correct = correct + sum(predicted.data.numpy() == Y.data.numpy())
        total_loss = total_loss + loss
    if epoch % 50 == 0:
        print(
            "Epoch [%d/%d], Loss: %.4f, Accuracy: %.2f %%"
            % (epoch + 1, num_epochs, total_loss, 100 * correct / total)
        )

train_input = train_data.iloc[:, :input_size]
train_target = train_data.iloc[:, input_size]

inputs = torch.Tensor(train_input.values).float()
targets = torch.Tensor(train_target.values - 1).long()

outputs = net(inputs)
_, predicted = torch.max(outputs, 1)


title: Best way to save many tensors of different shapes?

body:
I would like to store thousands to millions of tensors with different shapes to disk. The goal is to use them as a time series dataset. The dataset will probably not fit into memory and I will have to load samples or ranges of samples from disk.
What is the best way to accomplish this while keeping storage and access time low?


title: How to connect the input to the output directly using single fully connected layer in PyTorch?

body:
I am new to deep learning and cnn and trying to get familiar with that field using CIFAR10 tutorial code from PyTorch website. So, in that code I was playing with removing/adding layers to better understand the effect of them and I tried to connect the input(which is the initial data with the batch of 4 images) to the output directly with using only single fully connected layer. I know that does not make much sense, but I do it only for the sake of experiment. So, when I tried to do it, I faced with some errors, which are as follows:

First, here is the code snippet:

########################################################################
# 2. Define a Convolution Neural Network
# ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
# Copy the neural network from the Neural Networks section before and modify it to
# take 3-channel images (instead of 1-channel images as it was defined).

import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        #self.conv1 = nn.Conv2d(3, 6, 5)
        #self.pool = nn.MaxPool2d(2, 2)
        #self.conv2 = nn.Conv2d(6, 16, 5)
        #self.fc1 = nn.Linear(16 * 5 * 5, 120)
        #self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(768 * 4 * 4, 10)

    def forward(self, x):
        #x = self.pool(F.relu(self.conv1(x)))
        #x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 768 * 4 * 4)
        #x = F.relu(self.fc1(x))
        #x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

net = Net()

#######################################################################
# 3. Define a Loss function and optimizer
# ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
# Let's use a Classification Cross-Entropy loss and SGD with momentum.

import torch.optim as optim

criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)

########################################################################
# 4. Train the network
# ^^^^^^^^^^^^^^^^^^^^
#
# This is when things start to get interesting.
# We simply have to loop over our data iterator, and feed the inputs to the
# network and optimize.

for epoch in range(4):  # loop over the dataset multiple times

    running_loss = 0.0
    for i, data in enumerate(trainloader, 0):
        # get the inputs
        inputs, labels = data

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = net(inputs)
        print(len(outputs))
        print(len(labels))
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()
        if i % 2000 == 1999:    # print every 2000 mini-batches
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 2000))
            running_loss = 0.0

print('Finished Training')

So, when I run the code, I get the following error:

Traceback (most recent call last):
      File "C:\Users\Andrey\Desktop\Machine_learning_Danila\Homework 3\cifar10_tutorial1.py", line 180, in <module>
        loss = criterion(outputs, labels)
      File "C:\Program Files\Python36\lib\site-packages\torch\nn\modules\module.py", line 477, in __call__
        result = self.forward(*input, **kwargs)
      File "C:\Program Files\Python36\lib\site-packages\torch\nn\modules\loss.py", line 862, in forward
        ignore_index=self.ignore_index, reduction=self.reduction)
      File "C:\Program Files\Python36\lib\site-packages\torch\nn\functional.py", line 1550, in cross_entropy
        return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
      File "C:\Program Files\Python36\lib\site-packages\torch\nn\functional.py", line 1405, in nll_loss
        .format(input.size(0), target.size(0)))
    ValueError: Expected input batch_size (1) to match target batch_size (4).

I was trying to check the length of x and it turns out, that it is 4 initially but it becomes 1 after the line

x = x.view(-1, 768 * 4 * 4)

I think my numbers are correct, but it seems like I am having only 1 tensor instead of 4 as I supposed to have, and I feel like that is what causes that error.
I am wondering, why is that and what is the best way to fix that?
Also, what would be the best optimal number for output dimension output in  nn.Linear(Fully connected Layer) in this case?


title: Pytorch median - is it bug or am I using it wrong

body:
I am trying to get median of each row of 2D torch.tensor. But the result is not what I expect when compared to working with standard array or numpy

import torch
import numpy as np
from statistics import median

print(torch.__version__)
>>> 0.4.1

y = [[1, 2, 3, 5, 9, 1],[1, 2, 3, 5, 9, 1]]
median(y[0])
>>> 2.5

np.median(y,axis=1)
>>> array([2.5, 2.5])

yt = torch.tensor(y,dtype=torch.float32)
yt.median(1)[0]
>>> tensor([2., 2.])


title: Pytorch: is there a function similar to torch.argmax which can really keep the dimension of the original data?

body:
For example,
the code is

input = torch.randn(3, 10)
result = torch.argmax(input, dim=0, keepdim=True)

input is

tensor([[ 1.5742,  0.8183, -2.3005, -1.1650, -0.2451],
       [ 1.0553,  0.6021, -0.4938, -1.5379, -1.2054],
       [-0.1728,  0.8372, -1.9181, -0.9110,  0.2422]])

and result is

tensor([[ 0,  2,  1,  2,  2]])

However, I want a result like this

tensor([[ 1,  0,  0,  0,  0],
        [ 0,  0,  1,  0,  0],
        [ 0,  1,  0,  1,  1]])


title: Using autograd to compute Jacobian matrix of outputs with respect to inputs

body:
I apologize if this question is obvious or trivial. I am very new to pytorch and I am trying to understand the autograd.grad function in pytorch. I have a neural network G that takes in inputs (x,t) and outputs (u,v). Here is the code for G:

class GeneratorNet(torch.nn.Module):
    """
    A three hidden-layer generative neural network
    """

    def __init__(self):
        super(GeneratorNet, self).__init__()
        self.hidden0 = nn.Sequential(
            nn.Linear(2, 100),
            nn.LeakyReLU(0.2)
            )

        self.hidden1 = nn.Sequential(
            nn.Linear(100, 100),
            nn.LeakyReLU(0.2)
            )

        self.hidden2 = nn.Sequential(
            nn.Linear(100, 100),
            nn.LeakyReLU(0.2)
            )

        self.out = nn.Sequential(
            nn.Linear(100, 2),
            nn.Tanh()
            )

    def forward(self, x):
        x = self.hidden0(x)
        x = self.hidden1(x)
        x = self.hidden2(x)
        x = self.out(x)
        return x

Or simply G(x,t) = (u(x,t), v(x,t)) where u(x,t) and v(x,t) are scalar valued. Goal: Compute $\frac{\partial u(x,t)}{\partial x}$ and $\frac{\partial u(x,t)}{\partial t}$. At every training step, I have a minibatch of size $100$ so u(x,t) is a [100,1] tensor. Here is my attempt to compute the partial derivatives, where coords is the input (x,t) and just like below I added the requires_grad_(True) flag to the coords as well:

tensor = GeneratorNet(coords)
tensor.requires_grad_(True)
u, v = torch.split(tensor, 1, dim=1)
du = autograd.grad(u, coords, grad_outputs=torch.ones_like(u), create_graph=True,
                   retain_graph=True, only_inputs=True, allow_unused=True)[0]

du is now a [100,2] tensor.
Question: Is this the tensor of the partials for the 100 input points of the minibatch?

There are similar questions like computing derivatives of the output with respect to inputs but I could not really figure out what's going on. I apologize once again if this is already answered or trivial. Thank you very much.


I am trying to run a sequence of bigrams into an LSTM. This involves:

2.Padding the sequences using pad_sequence

3.Inputting the padded sequences in the embedding layer

4.Packing the output of the embedding layer

5.Inserting the pack into the LSTM.

class LSTMClassifier(nn.Module):
    # LSTM initialization
    def __init__(self, embedding_dim=32, hidden_dim=50, vocab_size=7138, label_size=2, static_size, batch_size=32):
        super(LSTMClassifier, self).__init__()
        # Initializing batch size
        self.batch_size = batch_size
        # Setting the hidden layer dimension of the LSTM
        self.hidden_dim = hidden_dim
        # Initializing the embedding layer
        self.embeddings = nn.Embedding(vocab_size, embedding_dim-2)
        # Initializing the LSTM layer with one hidden layer
        self.lstm = nn.LSTM(((embedding_dim*vocab_size)+static_size), hidden_dim, num_layers=1, batch_first=True)
        # Initializing linear linear that takes the hidden layer output
        self.hidden2label = nn.Linear(hidden_dim, label_size)
        # Initializing the hidden layer
        self.hidden = self.init_hidden()

    # Defining the hidding state of the LSTM
    def init_hidden(self):
        # the first is the hidden h
        # the second is the cell  c
        return (autograd.Variable(torch.zeros(1, self.batch_size, self.hidden_dim).cuda()),
                autograd.Variable(torch.zeros(1, self.batch_size, self.hidden_dim).cuda()))

    # Defining the feed forward logic of the LSTM. It contains:
    # 1. The embedding layer
    # 2. The LSTM layer with one hidden layer
    # 3. The software layer
    def forward(self, seq, freq, time, static):
        # reset the LSTM hidden state. Must be done before you run a new batch. Otherwise the LSTM will treat
        # a new batch as a continuation of a sequence
        self.hidden = self.init_hidden()

        # Get sequence lengths
        seq_lengths = torch.LongTensor(list(map(len, seq))) # length of 59

        # Pad the sequences
        seq = rnn_utils.pad_sequence(seq, batch_first = True)
        freq = rnn_utils.pad_sequence(freq, batch_first = True)
        time = rnn_utils.pad_sequence(time, batch_first = True)
        static = rnn_utils.pad_sequence(static, batch_first = True)

        seq = autograd.Variable(seq)
        freq = autograd.Variable(freq)
        time = autograd.Variable(time)
        static = autograd.Variable(static)

        # This is the pass to the embedding layer.
        # The sequence is of dimension N and the output is N x Demb
        embeds = self.embeddings(seq)
        embeds = torch.cat((embeds,freq), dim=3)
        embeds = torch.cat((embeds,time), dim=3)
        print(embeds.size()) #torch.Size([32, 59, 7138, 32])

        x = embeds.view(self.batch_size, seq.size()[1], 1,-1)
        print(x.size()) #torch.Size([32, 59, 1, 228416])

        static = static.view(self.batch_size, -1,1,3)
        x = torch.cat([x, static], dim=3)
        print(x.size()) #torch.Size([32, 59, 1, 228419])

        # pack the padded sequence so that paddings are ignored
        x = torch.nn.utils.rnn.pack_padded_sequence(x, seq_lengths, batch_first=True)

        lstm_out, self.hidden = self.lstm(x, self.hidden)

        # unpack the packed padded sequence so that it is ready for prediction
        lstm_out = torch.nn.utils.rnn.pad_packed_sequence(lstm_out, batch_first=True)

        y = self.hidden2label(lstm_out[-1])
        log_probs = F.log_softmax(y)
        return log_probs

However, I get an error that is the following:

---> 66         lstm_out, self.hidden = self.lstm(x, self.hidden)
     67
     68         # unpack the packed padded sequence so that it is ready for prediction

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    487             result = self._slow_forward(*input, **kwargs)
    488         else:
--> 489             result = self.forward(*input, **kwargs)
    490         for hook in self._forward_hooks.values():
    491             hook_result = hook(self, input, result)

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/modules/rnn.py in forward(self, input, hx)
    173                 hx = (hx, hx)
    174
--> 175         self.check_forward_args(input, hx, batch_sizes)
    176         _impl = _rnn_impls[self.mode]
    177         if batch_sizes is None:

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/modules/rnn.py in check_forward_args(self, input, hidden, batch_sizes)
    129             raise RuntimeError(
    130                 'input must have {} dimensions, got {}'.format(
--> 131                     expected_input_dim, input.dim()))
    132         if self.input_size != input.size(-1):
    133             raise RuntimeError(

RuntimeError: input must have 2 dimensions, got 3
x = rnn_utils.pad_sequence(x, batch_first=True)
import torch

I thought LSTM model required 3 dimensional input? I am very confused as to why it requires 2 dimensions. How should I fix it?


title: Keras vs PyTorch LSTM different results

body:
Trying to get similar results on same dataset with Keras and PyTorch.
Data
from numpy import array
from numpy import hstack

from sklearn.model_selection import train_test_split

# split a multivariate sequence into samples
def split_sequences(sequences, n_steps):
    X, y = list(), list()
    for i in range(len(sequences)):
        # find the end of this pattern
        end_ix = i + n_steps
        # check if we are beyond the dataset
        if end_ix > len(sequences):
            break
        # gather input and output parts of the pattern
        seq_x, seq_y = sequences[i:end_ix, :-1], sequences[end_ix-1, -1]
        X.append(seq_x)
        y.append(seq_y)
    return array(X), array(y)

def get_data():
    # define input sequence
    in_seq1 = array([x for x in range(0,500,10)])/1
    in_seq2 = array([x for x in range(5,505,10)])/1
    out_seq = array([in_seq1[i]+in_seq2[i] for i in range(len(in_seq1))])
    # convert to [rows, columns] structure
    in_seq1 = in_seq1.reshape((len(in_seq1), 1))
    in_seq2 = in_seq2.reshape((len(in_seq2), 1))
    out_seq = out_seq.reshape((len(out_seq), 1))
    # horizontally stack columns
    dataset = hstack((in_seq1, in_seq2, out_seq))

    n_features = 2 # this is number of parallel inputs
    n_timesteps = 3 # this is number of timesteps

    # convert into input/output
    X, y = split_sequences(dataset, n_timesteps)
    print(X.shape, y.shape)
    X_train,x_test,Y_train, y_test = train_test_split(X,y,test_size = 0.2,shuffle=False)

    return X_train,x_test,Y_train, y_test

Keras
from keras.models import Sequential
from keras.layers import LSTM
from keras.layers import Dense

from sklearn.metrics import mean_squared_error

import testing.TimeSeries.datacreator as dc # !!!!change this!!!!
X_train,x_test,Y_train, y_test = dc.get_data()

n_features = 2 # this is number of parallel inputs
n_timesteps = 3 # this is number of timesteps

# define model
model = Sequential()
model.add(LSTM(1024, activation='relu',
               input_shape=(n_timesteps, n_features),
               kernel_initializer='uniform',
               recurrent_initializer='uniform'))
model.add(Dense(512, activation='relu'))
model.add(Dense(1))
opt = keras.optimizers.Adam(lr=0.001,
                      beta_1=0.9,
                      beta_2=0.999,
                      epsilon=keras.optimizers.K.epsilon(),
                      decay=0.0,
                      amsgrad=False)
model.compile(optimizer=opt, loss='mse')
# fit model
model.fit(X_train, Y_train, epochs=200, verbose=1,validation_data=(x_test,y_test))

yhat = model.predict(x_test, verbose=0)

mean_squared_error(y_test, yhat)

PyTorch - module class
import numpy as np
import torch
import torch.nn.functional as F

from sklearn.metrics import mean_squared_error

import testing.TimeSeries.datacreator as dc # !!!! change this !!!!
X_train,x_test,Y_train, y_test =   dc.get_data()
n_features = 2 # this is number of parallel inputs
n_timesteps = 3 # this is number of timesteps

class MV_LSTM(torch.nn.Module):
    def __init__(self,n_features,seq_length):
        super(MV_LSTM, self).__init__()
        self.n_features = n_features # number of parallel inputs
        self.seq_len = seq_length # number of timesteps
        self.n_hidden = 1024 # number of hidden states
        self.n_layers = 1 # number of LSTM layers (stacked)

        self.l_lstm = torch.nn.LSTM(input_size = n_features,
                                 hidden_size = self.n_hidden,
                                 num_layers = self.n_layers,
                                 batch_first = True)
        # according to pytorch docs LSTM output is
        # (batch_size,seq_len, num_directions * hidden_size)
        # when considering batch_first = True
        self.l_linear = torch.nn.Linear(self.n_hidden*self.seq_len, 512)
#        self.l_linear1 = torch.nn.Linear(512, 512)
        self.l_linear2 = torch.nn.Linear(512, 1)

    def init_hidden(self, batch_size):
        # even with batch_first = True this remains same as docs
        hidden_state = torch.zeros(self.n_layers,batch_size,self.n_hidden).to(next(self.parameters()).device)
        cell_state = torch.zeros(self.n_layers,batch_size,self.n_hidden).to(next(self.parameters()).device)
        self.hidden = (hidden_state, cell_state)

    def forward(self, x):
        batch_size, seq_len, _ = x.size()

        lstm_out, self.hidden = self.l_lstm(x,self.hidden)
        # lstm_out(with batch_first = True) is
        # (batch_size,seq_len,num_directions * hidden_size)
        # for following linear layer we want to keep batch_size dimension and merge rest
        # .contiguous() -> solves tensor compatibility error
        x = lstm_out.contiguous().view(batch_size,-1)
        x = F.relu(x)
        x = F.relu(self.l_linear(x))
#        x = F.relu(self.l_linear1(x))
        x = self.l_linear2(x)
        return x

PyTorch - init and train
# create NN
mv_net = MV_LSTM(n_features,n_timesteps)
criterion = torch.nn.MSELoss()
import keras # for epsilon constant
optimizer = torch.optim.Adam(mv_net.parameters(),
                             lr=1e-3,
                             betas=[0.9,0.999],
                             eps=keras.optimizers.K.epsilon(),
                             weight_decay=0,
                             amsgrad=False)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
mv_net.to(device)

train_episodes = 200
batch_size = 32
eval_batch_size = 32

for t in range(train_episodes):
    # TRAIN
    mv_net.train()
    for b in range(0,len(X_train),batch_size):
        inpt = X_train[b:b+batch_size,:,:]
        target = Y_train[b:b+batch_size]

        x_batch = torch.tensor(inpt,dtype=torch.float32).to(device)
        y_batch = torch.tensor(target,dtype=torch.float32).to(device)

        mv_net.init_hidden(x_batch.size(0))

        output = mv_net(x_batch)
        loss = criterion(output.view(-1), y_batch)

        loss.backward()
        optimizer.step()
        optimizer.zero_grad()

    # EVAL
    mv_net.eval()
    mv_net.init_hidden(eval_batch_size)
    acc = 0
    for b in range(0,len(x_test),eval_batch_size):
        inpt = x_test[b:b+eval_batch_size,:,:]
        target = y_test[b:b+eval_batch_size]

        x_batch = torch.tensor(inpt,dtype=torch.float32).to(device)
        y_batch = torch.tensor(target,dtype=torch.float32).to(device)
        mv_net.init_hidden(x_batch.size(0))

        output = mv_net(x_batch)
        acc += mean_squared_error(y_batch.cpu().detach().numpy(), output.view(-1).cpu().detach().numpy())
    print('step:' , t , 'train loss:' , round(loss.item(),3),'eval acc:',round(acc/len(x_test),3))

mv_net.init_hidden(len(x_test))
val = torch.tensor(x_test,dtype=torch.float32).to(device)
otp = mv_net(val)
print(mean_squared_error(y_test, otp.view(-1).cpu().detach().numpy()))

Results
Keras produces test MSE almost 0, but PyTorch about 6000, which is way too different
I have tried couple tweaks in PyTorch code, but none got me anywhere close to similar keras, even with identical optim params
I cant see what is wrong with (kinda tutorialic) PyTorch code


title: How to understand hidden_states of the returns in BertModel?(huggingface-transformers)

body:
Returns last_hidden_state (torch.FloatTensor of shape (batch_size,
  sequence_length, hidden_size)): Sequence of hidden-states at the
  output of the last layer of the model.

  pooler_output (torch.FloatTensor: of shape (batch_size, hidden_size)):
  Last layer hidden-state of the first token of the sequence
  (classification token) further processed by a Linear layer and a Tanh
  activation function. The Linear layer weights are trained from the
  next sentence prediction (classification) objective during
  pre-training.

  This output is usually not a good summary of the semantic content of
  the input, you’re often better with averaging or pooling the sequence
  of hidden-states for the whole input sequence.

  hidden_states (tuple(torch.FloatTensor), optional, returned when
  config.output_hidden_states=True): Tuple of torch.FloatTensor (one for
  the output of the embeddings + one for the output of each layer) of
  shape (batch_size, sequence_length, hidden_size).

  Hidden-states of the model at the output of each layer plus the
  initial embedding outputs.

  attentions (tuple(torch.FloatTensor), optional, returned when
  config.output_attentions=True): Tuple of torch.FloatTensor (one for
  each layer) of shape (batch_size, num_heads, sequence_length,
  sequence_length).

  Attentions weights after the attention softmax, used to compute the
  weighted average in the self-attention heads.

This is from https://huggingface.co/transformers/model_doc/bert.html#bertmodel. Although the description in the document is clear, I still don't understand the hidden_states of returns. There is a tuple, one for the output of the embeddings, and the other for the output of each layer.
Please tell me how to distinguish them, or what is the meaning of them? Thanks very much!![wink~


title: My Google Colab keeps crashing because of something in this code. Not sure what it is

body:
My google colab keeps crashing at train, even though RAM and disk are plenty. The problem is somewhere in this code, I think, but I don't know what it is. I am doing LSTM. I'd appreciate any help. I am using PyTorch.

This code is followed by generate function, encoder decoder class, etc. It is when I get to train that it crashes (for "unknown reason")

class LSTMLM(torch.nn.Module):
  def __init__(self,
              vocab_size,
              embedding_size,
              hidden_size,
              num_layers=1,
              dropout=0.1):
    super().__init__()
    self.vocab_size = vocab_size
    self.embedding_size = embedding_size
    self.hidden_size = hidden_size
    self.num_layers = num_layers
    self.dropout = torch.nn.Dropout(dropout)

    self.embedding = torch.nn.Embedding(vocab_size, embedding_size)

    self.lstm = torch.nn.LSTM(input_size=hidden_size, hidden_size=hidden_size, num_layers=num_layers, batch_first=True)

    self.logistl = torch.nn.Linear(hidden_size, vocab_size)

    pass

  def forward(self, x, init_hidden_state=None):
    assert x.shape[0] == 1

    emb = self.embedding(x)

    emb = self.dropout(emb)

    if init_hidden_state is None:
      h0 = torch.zeros(self.num_layers, 1, self.hidden_size)
      c0 = torch.zeros(self.num_layers, 1, self.hidden_size)
    else:
      h0, c0 = init_hidden_state
    output, (hn, hc) = self.lstm(emb, (h0, c0))

    hidden_states = output
    final_hidden_state = hn
    final_cell_state = cn
    final_state = [final_hidden__state, final_cell_state]

    hidden_states = self.dropout(hidden_states)

    output_dist = self.logistl(hidden_states)

    return output_dist, hidden_states, final_state


The following is the error information with pytorch:
TypeError: only integer tensors of a single element can be converted to an index
Traceback (most recent call last):

  File "Main_mine.py", line 377, in <module>
    best_model, [losses_train, losses_valid, losses_epochs_train, losses_epochs_valid] = Train_Model(grud, train_dataloader, valid_dataloader , lr, epochs,patience)
  File "Main_mine.py", line 206, in Train_Model
    outputs = model(inputs)
  File "/home/alaa/.local/lib/python3.5/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/media/alaa/New Drive2/PHD/PHD3-Missing Values/Codes/archive(1)/run/GRU-D_my_version/LSTMD_mine.py", line 168, in forward
    , torch.squeeze(Delta[:,i:i+1,:]))
  File "/media/alaa/New Drive2/PHD/PHD3-Missing Values/Codes/archive(1)/run/GRU-D_my_version/LSTMD_mine.py", line 135, in step
    h = ((delta_h * h))
TypeError: only integer tensors of a single element can be converted to an index.

The following is the code:
def step(self, x, x_last_obsv, x_mean, h, ct, mask, delta):
        batch_size = x.shape[0]
        dim_size = x.shape[1]

        delta_x = torch.exp(-torch.max(self.zeros_x, self.gamma_x_l(delta)))
        delta_h = torch.exp(-torch.max(self.zeros_h, self.gamma_h_l(delta)))

        x = mask * x + (1 - mask) * (delta_x * x_last_obsv + (1 - delta_x) * x_mean)
        h = ((delta_h * h))

        combined = torch.cat((x, h, mask), 1)
        f = F.sigmoid(self.fl(combined))
        i = F.sigmoid(self.il(combined))
        o = F.sigmoid(self.fl(combined))
        c = F.tanh(self.cl(combined))
        ct=f*ct + i*c
        h= (torch.round(o* torch.tanh(ct)))
        return h,ct


title: How to take and restore snapshots of model training on another VM in Google Colab?

body: There is a 12 hour time limit for training DL models on GPU, according to google colab. Other people have had similar questions in the past, but there has been no clear answer on how to save and load models halfway through training when the 12 hour limits get exceeded, including saving the number of epochs that has been completed/saving other parameters. Is there an automated script for me to save the relevant parameters and resume operations on another VM? I am a complete noob; clear cut answers will be much appreciated.


title: non-linear regression by Pytorch

body:
I am trying to implement a non-linear regression task using PyTorch framework. The inputs are sample sentences and the targets are their scores (these scores are some float numbers). In order for the computer to understand the sentences, I convert each sentence to a 50 dimensional real vector (embedding vectors) using a torch.nn.RNN (Recurrent Neural Network) layer. Then I use a linear layer, torch.nn.Linear, to predict the score.

This is my model:

import torch
import torch.nn as nn
from torch.nn.utils.rnn import pack_padded_sequence
import text_processing as tp
import torch.nn.functional as F

class RNNModel(nn.Module):
##################        __init__          ####################
    def __init__(self, vocab_size, input_size,hidden_size, output_size,my_embeddings,n_layers,bidirectional=True):
        super(RNNModel, self).__init__()
        self.hidden_size = hidden_size
        self.n_layers = n_layers
        self.n_directions = int(bidirectional) + 1

        self.embedding = nn.Embedding(vocab_size, hidden_size)
        self.rnn = nn.RNN(input_size,hidden_size,n_layers,bidirectional=bidirectional)
        self.fc = nn.Linear(hidden_size, output_size)
####################          forward          #################
    def forward(self, batch_of_sentences, batch_of_len):
        """
        input shape: B x S ([num of sentences in this batch] x [max sent len in this batch] )
        transpose to make S x B
        """
        input=batch_of_sentences
        input = input.t()
        batch_size = input.size(1)

        # Make a hidden
        hidden = self._init_hidden(batch_size)

        # Embedding S x B -> S x B x I (embedding size)
        embedded = self.embedding(input)
        # Pack them up nicely
        rnn_input = pack_padded_sequence(embedded, batch_of_len.data.cpu().numpy())

        # To compact weights again call flatten_parameters().
        self.rnn.flatten_parameters()
        output, hidden = self.rnn(rnn_input, hidden)
        fc_output = self.fc(hidden[-1])
        return fc_output

##################          _init_hidden          ##################
    def _init_hidden(self, batch_size):
        hidden = torch.zeros(self.n_layers * self.n_directions,
                             batch_size, self.hidden_size)
        return tp.create_variable(hidden)

I have called the model in the main program:

my_rnn_model = RNNModel(VOCAB_SIZE,INPUT_SIZE, HIDDEN_SIZE,OUTPUT_SIZE, tp.my_embeddings_train_dev,N_LAYERS, bidirectional=True)
criterion = torch.nn.MSELoss(size_average=False)
LR = 0.001
optimizer = torch.optim.Adam(my_rnn_model.parameters(), lr=LR, betas=(0.9, 0.99)
for epoch in range(1, N_EPOCHS + 1):
   loss=train()

Here is train() function:

def train():

    #Loading Data From TrainLoader
    for i, (batch_sents, batch_target_scores) in enumerate(train_loader, 1):
        #Zero Padding
        zero_padded_batch_sent, sorted_batch_len, sorted_batch_target_scores = tp.make_variables(batch_sents, batch_target_scores,"train")

        #Compute Output for Each Batch of Sentences
        output = my_rnn_model(zero_padded_batch_sent, sorted_batch_len)

        #Compute Loss
        loss = criterion(output, sorted_batch_target_scores)

        #Updating Parameters by Backward Propagation
        my_rnn_model.zero_grad()
        loss.backward()
        optimizer.step()

But I think that my model can not learn the data, because after training the model for 30 epochs, the loss plot for train and test data is fluctuating and looks unstable, whereas, I expect the error to converge over time and not to have oscillating behavior.

Here is my loss vs. training plot for train and test data during 30 epochs:

Which part of my code is wrong?


title: Removing last 2 layers from a BERT classifier results in " 'tuple' object has no attribute 'dim' " error. Why?

body:
I fine tuned a huggingface transformer using Keras (with ktrain) and then reloaded the model in Pytorch.

I want to access the third to last layer (pre_classifier), so I removed the two last layers:

BERT2 = torch.nn.Sequential(*(list(BERT.children())[:-2]))

Running an encoded sentence through this yields the following error message:

AttributeError                            Traceback (most recent call last)
<ipython-input-38-640702475573> in <module>
----> 1 ans2=BERT2(torch.tensor([e1]))
      2 print (ans2)

C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    539             result = self._slow_forward(*input, **kwargs)
    540         else:
--> 541             result = self.forward(*input, **kwargs)
    542         for hook in self._forward_hooks.values():
    543             hook_result = hook(self, input, result)

C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\container.py in forward(self, input)
     90     def forward(self, input):
     91         for module in self._modules.values():
---> 92             input = module(input)
     93         return input
     94

C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    539             result = self._slow_forward(*input, **kwargs)
    540         else:
--> 541             result = self.forward(*input, **kwargs)
    542         for hook in self._forward_hooks.values():
    543             hook_result = hook(self, input, result)

C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\linear.py in forward(self, input)
     85
     86     def forward(self, input):
---> 87         return F.linear(input, self.weight, self.bias)
     88
     89     def extra_repr(self):

C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\functional.py in linear(input, weight, bias)
   1366         - Output: :math:`(N, *, out\_features)`
   1367     """
-> 1368     if input.dim() == 2 and bias is not None:
   1369         # fused op is marginally faster
   1370         ret = torch.addmm(bias, input, weight.t())

AttributeError: 'tuple' object has no attribute 'dim'

Meanwhile deleting the classifier entirely (all three layers)

BERT3 = torch.nn.Sequential(*(list(BERT.children())[:-3]))

Yields the expected tensor (within a size 1 tuple) with the expected shape ([sentence_num,token_num,768]).

Why does the removal of two (but not three) layers breaks the model?
And how can I access the pre_classifier results?

It is not accessible by setting config with output_hidden_states=True as this flag returns the hidden values of the BERT transformer stack, not those of the classifier layers downstream to it.

--

PS

The code used to initialize the BERT model:

def collect_data_for_FT():

    from sklearn.datasets import fetch_20newsgroups
    train_data = fetch_20newsgroups(subset='train', shuffle=True, random_state=42)
    test_data =  fetch_20newsgroups(subset='test', shuffle=True, random_state=42)

    print('size of training set: %s' % (len(train_b['data'])))
    print('size of validation set: %s' % (len(test_b['data'])))
    print('classes: %s' % (train_b.target_names))

    x_train = train_data.data
    y_train = train_data.target
    x_test = test_data.data
    y_test = test_data.target

    return(x_train,y_train,x_test,y_test)

 bert_name = 'distilbert-base-uncased'
    from transformers import DistilBertForSequenceClassification,AutoConfig,AutoTokenizer
import os
dir_path = os.getcwd()
dir_path=os.path.join(dir_path,'models')

config = AutoConfig.from_pretrained(bert_name,num_labels=20) # change model configuration to access hidden values.

try:
    BERT = DistilBertForSequenceClassification.from_pretrained(dir_path,config=config)
    print ("Finetuned predictor loaded")
except:
    import tensorflow.keras as keras
    print ("No finetuned predictor found.\nTraining.")
    (x_train,y_train,x_test,y_test)=collect_data_for_FT()
    ####
    # prework:
    import ktrain
    from ktrain import text
    t = text.Transformer(bert_name, maxlen=500, classes=train_b.target_names)
    trn = t.preprocess_train(x_train, y_train)
    val = t.preprocess_test(x_test, y_test)
    pre_trained_model = t.get_classifier()
    learner = ktrain.get_learner(pre_trained_model, train_data=trn, val_data=val, batch_size=6)
    ####

    ####
    # Find best learning rate
    learner.lr_find()
    learner.lr_plot()
    ####

    learner.fit_onecycle(2e-4, 4) # choosen based on the learning rate/loss plot.

    ####
    # prepare and save:
    predictor = ktrain.get_predictor(learner.model, preproc=t)
    predictor.save('my_distilbertbase_predictor')
    predictor.model.save_pretrained(dir_path)
    ####
    BERT = DistilBertForSequenceClassification.from_pretrained(os.path.join(dir_path), from_tf=True,config=config) # re-load tensorflow to pytorch
    BERT.save_pretrained(dir_path) # save as a "full blooded" pytorch model
    BERT = DistilBertForSequenceClassification.from_pretrained(dir_path,config=config)  # re-load
    from tensorflow.keras import backend as K
    K.clear_session() # loading from tensorflow takes up space and the GPU. This releases it/


title: Pytorch transformation on MNIST dataset

body:
I currently have a project with Weak Supervision where I need to put a "masking" in front of a dataset. My issue right now is that I don't exactly know how to do it. Let me explain further with some code and images.

I am using the MNIST dataset that I have to edit in this way. As you can see a middle square is cut out. The code below is used to edit the MNIST using a for loop.

for i in range(int(image_size/2-5),int(image_size/2+3)):
   for j in range(int(image_size/2-5),int(image_size/2+3)):
      image[i][j] = 0

However, I am currently not sure how I should use this in a dataloader transform. The code for the dataloader and transform is shown here:

transform = torchvision.transforms.Compose([torchvision.transforms.ToTensor()])

train_dataset = torchvision.datasets.MNIST(
    root="~/torch_datasets", train=True, transform=transform, download=True
)

test_dataset = torchvision.datasets.MNIST(
    root="~/torch_datasets", train=False, transform=transform, download=True
)

train_loader = torch.utils.data.DataLoader(
    train_dataset, batch_size=128, shuffle=True, num_workers=4, pin_memory=True
)

test_loader = torch.utils.data.DataLoader(
    test_dataset, batch_size=32, shuffle=False, num_workers=4
)

def imshow(img):
    #img = img / 2 + 0.5     # unnormalize
    npimg = img.numpy()
    plt.imshow(np.transpose(npimg, (1, 2, 0)))

dataiter = iter(train_loader)
images, labels = dataiter.next()
imshow(torchvision.utils.make_grid(images))

So is there a straightforward way to apply the transform to the full dataset in the torchvision.transforms.Compose?


title: Preprocessing image for Tensorflow model instead Pytorch preprocessing

body:
I had a pytorch resnet101 encoder model, when input image got this preprocessing:
import torchvision as tv
from PIL import Image

data_transforms = tv.transforms.Compose([
    tv.transforms.Resize((224, 224)),
    tv.transforms.ToTensor(),
    tv.transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])])
img = Image.open(img_path)
img = img.convert('RGB')
img = data_transforms(img)
img = torch.FloatTensor(img)
img = img.unsqueeze(0)
print(img)

pytorch image tensor
Shape of input for encoder in this case is [1, 3, 224, 224], and this picture is normalized with the mean and std of ImageNet.
Now I'm export this model to tensorflow, so, how to make the same image preprocessing for tf-model?
I tried to do something like this:
from PIL import Image

img = Image.open(img_path)
img = img.convert('RGB')
img = tf.keras.preprocessing.image.img_to_array(img)
img = tf.image.resize(tf_img, (224, 224))
img = tf.keras.applications.resnet.preprocess_input(img)# now shape is [224, 224, 3]
img = tf.reshape(img, [1, 3, 224, 224])
print(img)

tensorflow image tensor
but I'm sure that I did something wrong, becouse torch and tf tensors looks very different for one image and give completely different output results from one encoder model.
Can anyone helps, what should I fix in tf preprocessing?


title: Installing Anaconda package behind firewall

body:
I am behind various firewalls at work and trying to install pytorch. Normally I can just download the tar.gz file and install it directly, but when I do this now I get this error message:

---
CondaHTTPError: HTTP None None for url <https://repo.continuum.io/pkgs/free/win-64/repodata.json.bz2>

An HTTP errors occurred when trying to retrieve this URL.
----

Are there any way to avoid it going to any websites to download and just install from the locally installed tar.gz file?

I can't use conda install or any of the other standard installation methods.


title: Two-Layer Neural Network in PyTorch does not Converge

body:
Problem

I am trying to implement 2-layer neural network using different methods (TensorFlow, PyTorch and from scratch) and then compare their performance based on MNIST dataset.

I am not sure what mistakes I have made, but the accuracy in PyTorch is only about 10%, which is basically random guess. I think probably the weights does not get updated at all.

Note that I intentionally use the dataset provided by TensorFlow to keep the data I use through 3 different methods consistent for accurate comparison.

from tensorflow.examples.tutorials.mnist import input_data
import torch

class Net(torch.nn.Module):
    def __init__(self):
      super(Net, self).__init__()
      self.fc1 =  torch.nn.Linear(784, 100)
      self.fc2 =  torch.nn.Linear(100, 10)

    def forward(self, x):
      # x -> (batch_size, 784)
      x = torch.relu(x)
      # x -> (batch_size, 10)
      x = torch.softmax(x, dim=1)
      return x

net = Net()
net.zero_grad()
Loss = torch.nn.CrossEntropyLoss()
optimizer =  torch.optim.SGD(net.parameters(), lr=0.01)

for epoch in range(1000):  # loop over the dataset multiple times

    batch_xs, batch_ys = mnist_m.train.next_batch(100)
    # convert to appropriate settins
    # note the input to the linear layer should be (n_sample, n_features)
    batch_xs = torch.tensor(batch_xs, requires_grad=True)
    # batch_ys -> (batch_size,)
    batch_ys = torch.tensor(batch_ys, dtype=torch.int64)

    # forward
    # output -> (batch_size, 10)
    output = net(batch_xs)
    # result -> (batch_size,)
    result = torch.argmax(output, dim=1)
    loss = Loss(output, batch_ys)

    # backward
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()


title: Unsupported Wheel Error when pip installing PyTorch without Conda

body:
I have been trying to install PyTorch in Windows 10 for Python 3.7.1

I do not have Anaconda on my machine, and do not wish to install it. I believe I have already satisfied all the necessary prerequisites (CUDA v10.0, NumPy). When I run the following installation command in the admin command line, (found on the PyTorch webpage):

pip3 install https://download.pytorch.org/whl/cu100/torch-1.0.0-cp37-cp37m-win_amd64.whl

I received the following error:

torch-1.0.0-cp37-cp37m-win_amd64.whl is not a supported wheel on this platform.

I tried downloading the wheel file in my browser, then running a modified command in my downloads directory.

pip install torch-1.0.0-cp37-cp37m-win_amd64.whl

I received the same error message. My pip version is up to date and I am attempting to install the appropriate wheel file for my Python version. This problem is unique to others, as I do not want to use Conda to install PyTorch. What is causing  this problem?


title: Getting gradient of vectorized function in pytorch

body:
I am brand new to PyTorch and want to do what I assume is a very simple thing but am having a lot of difficulty.

I have the function sin(x) * cos(x) + x^2 and I want to get the derivative of that function at any point.

If I do this with one point it works perfectly as

x = torch.autograd.Variable(torch.Tensor([4]),requires_grad=True)
y = torch.sin(x)*torch.cos(x)+torch.pow(x,2)
y.backward()
print(x.grad) # outputs tensor([7.8545])

However, I want to be able to pass in a vector as x and for it to evaluate the derivative element-wise. For example:

Input: [4., 4., 4.,]
Output: tensor([7.8545, 7.8545, 7.8545])

But I can't seem to get this working.

I tried simply doing

x = torch.tensor([4., 4., 4., 4.], requires_grad=True)
out = torch.sin(x)*torch.cos(x)+x.pow(2)
out.backward()
print(x.grad)

But I get the error "RuntimeError: grad can be implicitly created only for scalar outputs"

How do I adjust this code for vectors?

Thanks in advance,


title: How forward() method is used when it have more than one two input parameters in pytorch

body:
Can someone tell me the concept behind the multiple parameters in forward() method?
Generally, the implementation of forward() method has two parameters

self
input

if a forward method has more than these parameters how PyTorch is using the forward method.

Let's consider this codebase:
https://github.com/bamps53/kaggle-autonomous-driving2019/blob/master/models/centernet.py
here online 236 authors have used forward method with two more parameters:

centers
return_embeddings

I am unable to find a single article that can answer my query on what condition Line 254(return_embeddings:) and Line 257(if centers is not None:) will execute. As per my knowledge forward, the method is internally called by nn module. Can someone please put some lights on this?