title: Convert n-dimensional numpy array into torch tensor

body:
I have 4-dimensional numpy array (type(x) is ndarray, type(x[i]) is ndarray, type(x[i][j]) is ndarray, and type(x[i][j][k]) is float32). I need to convert it to torch tensor to use it as part of dataset. I tried to stack torch.Tensor like that:

def convert_to_tensor(ndarr):
    for i in ndarr:
        for j in ndarr[i]:
            tensor_ndarr = torch.stack([torch.Tensor(k) for k in ndarr[i][j]])
    return tensor_ndarr

but not sure if that gonna work.
Is there more simple or convenient way to do that?

P.S. X looks like that:

[array([[0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ..., 9.1042261e-07,
        4.2489510e-06, 1.0829879e-07],
       [0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ..., 4.5446509e-06,
        1.0727225e-05, 7.5697239e-06],
       [0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ..., 1.4139889e-06,
        8.1363060e-06, 5.8842788e-06],
       ...,
       [0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ..., 1.2733157e-07,
        3.6470925e-07, 2.5473477e-07],
       [0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ..., 7.2166131e-08,
        2.0409759e-07, 2.3035474e-07],
       [0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ..., 5.7966993e-08,
        1.7371592e-07, 1.5426794e-07]], dtype=float32)
...
 array([[0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ..., 9.1042261e-07,
        4.2489510e-06, 1.0829879e-07],
       [0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ..., 4.5446509e-06,
        1.0727225e-05, 7.5697239e-06],
       [0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ..., 1.4139889e-06,
        8.1363060e-06, 5.8842788e-06],
       ...,
       [0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ..., 1.2733157e-07,
        3.6470925e-07, 2.5473477e-07],
       [0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ..., 7.2166131e-08,
        2.0409759e-07, 2.3035474e-07],
       [0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ..., 5.7966993e-08,
        1.7371592e-07, 1.5426794e-07]], dtype=float32)]


title: Convert a list of numpy array to torch tensor list

body:
The numpy arrays in the list are 2D array that have different sizes, let's say:
1x1, 4x4, 8x8, etc
about 7 arrays in total.
I know how to convert each on of them, by:
torch.from_numpy(a1by1).type(torch.FloatTensor)
torch.from_numpy(a4by4).type(torch.FloatTensor)
etc..

Is there a way to convert the entire list in one command?
I found these 2 question:
How to convert a list or numpy array to a 1d torch tensor?

How to convert a list of tensors into a torch::Tensor?

but it's not what Im' looking for


title: Pytorch What's the difference between define layer in __init__() and directly use in forward()?

body:
What is the difference between the method that define layers in __init__() function, call layer in forward later and the method that directly use layer in forward() function ?
Should I define every layer in my compute graph in constructed function(eg. __init__) before I write my compute graph?
Could I direct define and use them in forward()?


title: How to create a custom PyTorch dataset when the order and the total number of training samples is not known in advance?

body:
I have a 42 GB jsonl file. Every element of this file is a json object. I create training samples from every json object. But the number of training samples from every json object that I extract can vary between 0 to 5 samples. What is the best way to create a custom PyTorch dataset without reading the entire jsonl file in memory?

This is the dataset I am talking about - Google Natural Questions.


title: Torchtext TabularDataset: data.Field doesn't contain actual imported data?

body:
I learned from the Torchtext documentation that the way to import csv files is through TabularDataset. I did it like this:

train = data.TabularDataset(path='./data.csv',
                        format='csv',
                        fields=[("label",data.Field(use_vocab=True,include_lengths=False)),
                                ("statement",data.Field(use_vocab=True,include_lengths=True))],
                        skip_header=True)

"label" and "statement" are the header names of the 2 columns in my csv file. I defined them as data.Field, but "label" and "statement" don't seem to actually contain the data from my csv file, despite being recognized as data field objects by the console with no problem. I found out this issue when I tried to build a vocab list with statement.build_vocab(train, max_size=25000). I printed len(statement.vocab), the return is "2", which obviously doesn't reflect the actual data in the csv file. Did I do something wrong when importing the csv data or is my vocab building done wrong? Is there a separate method to put the data in the field objects? Thanks!!


title: pytorch dataloader stucked if using opencv resize method

body:
I can run all the cells of the tutorial notebook of Pytorch about dataloading (pytorch tutorial).
But when I use OpenCV in place of Skimage to resize the image, the dataloader gets stuck, i.e nothing happens.

In the Rescale class:

class Rescale(object):
    .....
    def __call__(self, sample):
       ....
       #img = transform.resize(image, (new_h, new_w))
       img = cv2.resize(image, (new_h, new_w))
       .....

The dataloader and the for loop are defined with:

dataloader = DataLoader(transformed_dataset, batch_size=4,
                        shuffle=True, num_workers=4)

for i_batch, sample_batched in enumerate(dataloader):
    print(i_batch, sample_batched['image'].size(),
          sample_batched['landmarks'].size())

I can get the iterator to print something if num_workers=0. It looks like opencv does not play well with the Multiprocessing of pytorch.
I would really prefer to use same package to transform the images at train time and test time (and I am already using OpenCV for the image rescale at test time).
Any suggestions would be greatly appreciated.


title: What is the difference between MLP implementation from scratch and in PyTorch?

body:
Following up the question from How to update the learning rate in a two layered multi-layered perceptron?

Given the XOR problem:

X = xor_input = np.array([[0,0], [0,1], [1,0], [1,1]])
Y = xor_output = np.array([[0,1,1,0]]).T

And a simple

two layered Multi-Layered Perceptron (MLP) with
sigmoid activations between them and
Mean Square Error (MSE) as the loss function/optimization criterion

If we train the model from scratch as such:

from itertools import chain
import matplotlib.pyplot as plt
import numpy as np
np.random.seed(0)

def sigmoid(x): # Returns values that sums to one.
    return 1 / (1 + np.exp(-x))

def sigmoid_derivative(sx):
    # See https://math.stackexchange.com/a/1225116
    return sx * (1 - sx)

# Cost functions.
def mse(predicted, truth):
    return 0.5 * np.mean(np.square(predicted - truth))

def mse_derivative(predicted, truth):
    return predicted - truth

X = xor_input = np.array([[0,0], [0,1], [1,0], [1,1]])
Y = xor_output = np.array([[0,1,1,0]]).T

# Define the shape of the weight vector.
num_data, input_dim = X.shape
# Lets set the dimensions for the intermediate layer.
hidden_dim = 5
# Initialize weights between the input layers and the hidden layer.
W1 = np.random.random((input_dim, hidden_dim))

# Define the shape of the output vector.
output_dim = len(Y.T)
# Initialize weights between the hidden layers and the output layer.
W2 = np.random.random((hidden_dim, output_dim))

# Initialize weigh
num_epochs = 5000
learning_rate = 0.3

losses = []

for epoch_n in range(num_epochs):
    layer0 = X
    # Forward propagation.

    # Inside the perceptron, Step 2.
    layer1 = sigmoid(np.dot(layer0, W1))
    layer2 = sigmoid(np.dot(layer1, W2))

    # Back propagation (Y -> layer2)

    # How much did we miss in the predictions?
    cost_error = mse(layer2, Y)
    cost_delta = mse_derivative(layer2, Y)

    #print(layer2_error)
    # In what direction is the target value?
    # Were we really close? If so, don't change too much.
    layer2_error = np.dot(cost_delta, cost_error)
    layer2_delta = cost_delta *  sigmoid_derivative(layer2)

    # Back propagation (layer2 -> layer1)
    # How much did each layer1 value contribute to the layer2 error (according to the weights)?
    layer1_error = np.dot(layer2_delta, W2.T)
    layer1_delta = layer1_error * sigmoid_derivative(layer1)

    # update weights
    W2 += - learning_rate * np.dot(layer1.T, layer2_delta)
    W1 += - learning_rate * np.dot(layer0.T, layer1_delta)
    #print(np.dot(layer0.T, layer1_delta))
    #print(epoch_n, list((layer2)))

    # Log the loss value as we proceed through the epochs.
    losses.append(layer2_error.mean())
    #print(cost_delta)

# Visualize the losses
plt.plot(losses)
plt.show()

We get a sharp dive in the loss from epoch 0 and then saturates quickly:

But if we train a similar model with pytorch, the training curve has a gradual drop in losses before saturating:

What is the difference between the MLP from scratch and the PyTorch code?

Why is it achieving convergence at different point?

Other than the weights initialization, np.random.rand() in the code from scratch and the default torch initialization, I can't seem to see a difference in the model.

Code for PyTorch:

from tqdm import tqdm
import numpy as np

import torch
from torch import nn
from torch import tensor
from torch import optim

import matplotlib.pyplot as plt

torch.manual_seed(0)
device = 'gpu' if torch.cuda.is_available() else 'cpu'

# XOR gate inputs and outputs.
X = xor_input = tensor([[0,0], [0,1], [1,0], [1,1]]).float().to(device)
Y = xor_output = tensor([[0],[1],[1],[0]]).float().to(device)

# Use tensor.shape to get the shape of the matrix/tensor.
num_data, input_dim = X.shape
print('Inputs Dim:', input_dim) # i.e. n=2

num_data, output_dim = Y.shape
print('Output Dim:', output_dim)
print('No. of Data:', num_data) # i.e. n=4

# Step 1: Initialization.

# Initialize the model.
# Set the hidden dimension size.
hidden_dim = 5
# Use Sequential to define a simple feed-forward network.
model = nn.Sequential(
            # Use nn.Linear to get our simple perceptron.
            nn.Linear(input_dim, hidden_dim),
            # Use nn.Sigmoid to get our sigmoid non-linearity.
            nn.Sigmoid(),
            # Second layer neurons.
            nn.Linear(hidden_dim, output_dim),
            nn.Sigmoid()
        )
model

# Initialize the optimizer
learning_rate = 0.3
optimizer = optim.SGD(model.parameters(), lr=learning_rate)

# Initialize the loss function.
criterion = nn.MSELoss()

# Initialize the stopping criteria
# For simplicity, just stop training after certain no. of epochs.
num_epochs = 5000

losses = [] # Keeps track of the loses.

# Step 2-4 of training routine.

for _e in tqdm(range(num_epochs)):
    # Reset the gradient after every epoch.
    optimizer.zero_grad()
    # Step 2: Foward Propagation
    predictions = model(X)

    # Step 3: Back Propagation
    # Calculate the cost between the predictions and the truth.
    loss = criterion(predictions, Y)
    # Remember to back propagate the loss you've computed above.
    loss.backward()

    # Step 4: Optimizer take a step and update the weights.
    optimizer.step()

    # Log the loss value as we proceed through the epochs.
    losses.append(loss.data.item())

plt.plot(losses)


title: ResNet50 torchvision implementation gives low accuracy on CIFAR-10

body:
I am new to Deep Learning and PyTorch. I am using the resnet-50 model in the torchvision module on cifar10. I have imported the CIFAR-10 dataset from torchvision. The accuracy is very low on testing and I have tried configuring the classification layers but there is no change in the accuracy. Is there something wrong with my code? Am I making a mistake in calculating the accuracy?

import torchvision
import torch
import torch.nn as nn
from torch import optim
import os
import torchvision.transforms as transforms
from torch.utils.data import DataLoader
import numpy as np
from collections import OrderedDict
import matplotlib.pyplot as plt

transformations=transforms.Compose([transforms.ToTensor(),transforms.Normalize([0.485, 0.456, 0.406],[0.229, 0.224, 0.225])])
trainset=torchvision.datasets.CIFAR10(root='./CIFAR10',download=True,transform=transformations,train=True)

testset=torchvision.datasets.CIFAR10(root='./CIFAR10',download=True,transform=transformations,train=False)

trainloader=DataLoader(dataset=trainset,batch_size=4)
testloader=DataLoader(dataset=testset,batch_size=4)

inputs,labels=next(iter(trainloader))
labels=labels.float()
inputs.size()

print(labels.type())
resnet=torchvision.models.resnet50(pretrained=True)

if torch.cuda.is_available():
  resnet=resnet.cuda()
  inputs,labels=inputs.cuda(),torch.Tensor(labels).cuda()

outputs=resnet(inputs)
outputs.size()

for param in resnet.parameters():
  param.requires_grad=False

numft=resnet.fc.in_features
print(numft)
resnet.fc=torch.nn.Sequential(nn.Linear(numft,1000),nn.ReLU(),nn.Linear(1000,10))
resnet.cuda()
resnet.train(True)
optimizer=torch.optim.SGD(resnet.parameters(),lr=0.001,momentum=0.9)
criterion=nn.CrossEntropyLoss()

for epoch in range(5):
    resnet.train(True)

    trainloss=0
    correct=0
    for x,y in trainloader:
        x,y=x.cuda(),y.cuda()
        optimizer.zero_grad()

        yhat=resnet(x)
        loss=criterion(yhat,y)

        loss.backward()
        optimizer.step()
        trainloss+=loss.item()

    print('Epoch: {} Loss: {}'.format(epoch,(trainloss/len(trainloader))))

    accuracy=[]
    running_corrects=0.0
    for x_test,y_test in testloader:

        x_test,y_test=x_test.cuda(),y_test.cuda()
        yhat=resnet(x_test)
        _,z=yhat.max(1)
        running_corrects += torch.sum(y_test == z)

    accuracy.append(running_corrects/len(testloader))

print(running_corrects/len(testloader))
accuracy=max(accuracy)
print(accuracy)

OUTPUT AFTER TRAINING/TESTING

Epoch: 0 Loss: 1.9808503997325897
Epoch: 1 Loss: 1.7917569598436356
Epoch: 2 Loss: 1.624434965057373
Epoch: 3 Loss: 1.4082191940283775
Epoch: 4 Loss: 1.1343850775527955
tensor(1.1404, device='cuda:0')
tensor(1.1404, device='cuda:0')


title: Conda Pytorch on Windows results in HTTP Connection Error

body:
In general, my package manager and connection work fine. I just ran a conda update --all command that worked and updated all of my packages. However, when I run conda install -c pytorch pytorch as specified on anaconda's website https://anaconda.org/pytorch/pytorch it results in the following error.

CondaHTTPError: HTTP 000 CONNECTION FAILED for url <https://conda.anaconda.org/pytorch/win-64/pytorch-0.4.0-py36_cuda80_cudnn7he774522_1.tar.bz2>
Elapsed: -An HTTP error occurred when trying to retrieve this URL. HTTP errors are often intermittent, and a simple retry will get you on your way.

I was curious if anyone else is experiencing this problem, as you may notice the pytorch upload for windows was only made on conda 3 days ago. Otherwise, if someone has a fix that will be compatible with conda, your input is appreciated.

UPDATE [SOLUTION]: I went to work and found an additional conda upload command for pytorch has been recently updated for windows. That is conda install -c soumith pytorch as specified https://anaconda.org/soumith/pytorch

I found this command to work on both my home and work laptops.


title: PyTorch Object Detection with GPU on Ubuntu 18.04 - RuntimeError: CUDA out of memory. Tried to allocate xx.xx MiB

body:
I'm attempting to get this PyTorch person detection example:

https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html

running locally with a GPU, either in a Jupyter Notebook or a regular python file.  I get the error in the title either way.

I'm using Ubuntu 18.04.  Here is a summary of the steps I've performed:

1) Stock Ubuntu 18.04 install on a Lenovo ThinkPad X1 Extreme Gen 2 with a GTX 1650 GPU.

2) Perform a standard CUDA 10.0 / cuDNN 7.4 install.  I'd rather not restate all the steps as this post is going to be more than long enough already.  This is a standard procedure, pretty much any link found via googling is what I followed.

3) Install torch and torchvision

pip3 install torch torchvision

4) From this link on the PyTorch site:

https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html

I've both saved the linked notebook:

https://colab.research.google.com/github/pytorch/vision/blob/temp-tutorial/tutorials/torchvision_finetuning_instance_segmentation.ipynb

And Also tried the link at the bottom that has the regular Python file:

https://pytorch.org/tutorials/_static/tv-training-code.py

5) Before running either the notebook or the regular Python way, I did the following (found at the top of the above linked notebook):

Install the CoCo API into Python:

cd ~
git clone https://github.com/cocodataset/cocoapi.git
cd cocoapi/PythonAPI

open Makefile in gedit, change the two instances of "python" to "python3", then:

python3 setup.py build_ext --inplace
sudo python3 setup.py install

Get the necessary files the above linked files need to run:

cd ~
git clone https://github.com/pytorch/vision.git
cd vision
git checkout v0.5.0

from ~/vision/references/detection, copy coco_eval.py, coco_utils.py, engine.py, transforms.py, and utils.py to whichever directory the above linked notebook or tv-training-code.py file are being ran from.

6) Download the Penn Fudan Pedestrian dataset from the link on the above page:

https://www.cis.upenn.edu/~jshi/ped_html/PennFudanPed.zip

then unzip and put in the same directory as the notebook or tv-training-code.py

In case the above link ever breaks or just for easier reference, here is tv-training-code.py as I have downloaded it at this time:

# Sample code from the TorchVision 0.3 Object Detection Finetuning Tutorial
# http://pytorch.org/tutorials/intermediate/torchvision_tutorial.html

import os
import numpy as np
import torch
from PIL import Image

import torchvision
from torchvision.models.detection.faster_rcnn import FastRCNNPredictor
from torchvision.models.detection.mask_rcnn import MaskRCNNPredictor

from engine import train_one_epoch, evaluate
import utils
import transforms as T

class PennFudanDataset(object):
    def __init__(self, root, transforms):
        self.root = root
        self.transforms = transforms
        # load all image files, sorting them to
        # ensure that they are aligned
        self.imgs = list(sorted(os.listdir(os.path.join(root, "PNGImages"))))
        self.masks = list(sorted(os.listdir(os.path.join(root, "PedMasks"))))

    def __getitem__(self, idx):
        # load images ad masks
        img_path = os.path.join(self.root, "PNGImages", self.imgs[idx])
        mask_path = os.path.join(self.root, "PedMasks", self.masks[idx])
        img = Image.open(img_path).convert("RGB")
        # note that we haven't converted the mask to RGB,
        # because each color corresponds to a different instance
        # with 0 being background
        mask = Image.open(mask_path)

        mask = np.array(mask)
        # instances are encoded as different colors
        obj_ids = np.unique(mask)
        # first id is the background, so remove it
        obj_ids = obj_ids[1:]

        # split the color-encoded mask into a set
        # of binary masks
        masks = mask == obj_ids[:, None, None]

        # get bounding box coordinates for each mask
        num_objs = len(obj_ids)
        boxes = []
        for i in range(num_objs):
            pos = np.where(masks[i])
            xmin = np.min(pos[1])
            xmax = np.max(pos[1])
            ymin = np.min(pos[0])
            ymax = np.max(pos[0])
            boxes.append([xmin, ymin, xmax, ymax])

        boxes = torch.as_tensor(boxes, dtype=torch.float32)
        # there is only one class
        labels = torch.ones((num_objs,), dtype=torch.int64)
        masks = torch.as_tensor(masks, dtype=torch.uint8)

        image_id = torch.tensor([idx])
        area = (boxes[:, 3] - boxes[:, 1]) * (boxes[:, 2] - boxes[:, 0])
        # suppose all instances are not crowd
        iscrowd = torch.zeros((num_objs,), dtype=torch.int64)

        target = {}
        target["boxes"] = boxes
        target["labels"] = labels
        target["masks"] = masks
        target["image_id"] = image_id
        target["area"] = area
        target["iscrowd"] = iscrowd

        if self.transforms is not None:
            img, target = self.transforms(img, target)

        return img, target

    def __len__(self):
        return len(self.imgs)

def get_model_instance_segmentation(num_classes):
    # load an instance segmentation model pre-trained pre-trained on COCO
    model = torchvision.models.detection.maskrcnn_resnet50_fpn(pretrained=True)

    # get number of input features for the classifier
    in_features = model.roi_heads.box_predictor.cls_score.in_features
    # replace the pre-trained head with a new one
    model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)

    # now get the number of input features for the mask classifier
    in_features_mask = model.roi_heads.mask_predictor.conv5_mask.in_channels
    hidden_layer = 256
    # and replace the mask predictor with a new one
    model.roi_heads.mask_predictor = MaskRCNNPredictor(in_features_mask,
                                                       hidden_layer,
                                                       num_classes)

    return model

def get_transform(train):
    transforms = []
    transforms.append(T.ToTensor())
    if train:
        transforms.append(T.RandomHorizontalFlip(0.5))
    return T.Compose(transforms)

def main():
    # train on the GPU or on the CPU, if a GPU is not available
    device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')

    # our dataset has two classes only - background and person
    num_classes = 2
    # use our dataset and defined transformations
    dataset = PennFudanDataset('PennFudanPed', get_transform(train=True))
    dataset_test = PennFudanDataset('PennFudanPed', get_transform(train=False))

    # split the dataset in train and test set
    indices = torch.randperm(len(dataset)).tolist()
    dataset = torch.utils.data.Subset(dataset, indices[:-50])
    dataset_test = torch.utils.data.Subset(dataset_test, indices[-50:])

    # define training and validation data loaders
    data_loader = torch.utils.data.DataLoader(
        dataset, batch_size=2, shuffle=True, num_workers=4,
        collate_fn=utils.collate_fn)

    data_loader_test = torch.utils.data.DataLoader(
        dataset_test, batch_size=1, shuffle=False, num_workers=4,
        collate_fn=utils.collate_fn)

    # get the model using our helper function
    model = get_model_instance_segmentation(num_classes)

    # move model to the right device
    model.to(device)

    # construct an optimizer
    params = [p for p in model.parameters() if p.requires_grad]
    optimizer = torch.optim.SGD(params, lr=0.005,
                                momentum=0.9, weight_decay=0.0005)
    # and a learning rate scheduler
    lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer,
                                                   step_size=3,
                                                   gamma=0.1)

    # let's train it for 10 epochs
    num_epochs = 10

    for epoch in range(num_epochs):
        # train for one epoch, printing every 10 iterations
        train_one_epoch(model, optimizer, data_loader, device, epoch, print_freq=10)
        # update the learning rate
        lr_scheduler.step()
        # evaluate on the test dataset
        evaluate(model, data_loader_test, device=device)

    print("That's it!")

if __name__ == "__main__":
    main()

Here is an exmaple run of tv-training-code.py

$ python3 tv-training-code.py
Epoch: [0]  [ 0/60]  eta: 0:01:17  lr: 0.000090  loss: 4.1717 (4.1717)  loss_classifier: 0.8903 (0.8903)  loss_box_reg: 0.1379 (0.1379)  loss_mask: 3.0632 (3.0632)  loss_objectness: 0.0700 (0.0700)  loss_rpn_box_reg: 0.0104 (0.0104)  time: 1.2864  data: 0.1173  max mem: 1865
Traceback (most recent call last):
  File "tv-training-code.py", line 165, in <module>
    main()
  File "tv-training-code.py", line 156, in main
    train_one_epoch(model, optimizer, data_loader, device, epoch, print_freq=10)
  File "/xxx/PennFudanExample/engine.py", line 46, in train_one_epoch
    losses.backward()
  File "/usr/local/lib/python3.6/dist-packages/torch/tensor.py", line 166, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/usr/local/lib/python3.6/dist-packages/torch/autograd/__init__.py", line 99, in backward
    allow_unreachable=True)  # allow_unreachable flag
  File "/usr/local/lib/python3.6/dist-packages/torch/autograd/function.py", line 77, in apply
    return self._forward_cls.backward(self, *args)
  File "/usr/local/lib/python3.6/dist-packages/torch/autograd/function.py", line 189, in wrapper
    outputs = fn(ctx, *args)
  File "/usr/local/lib/python3.6/dist-packages/torchvision/ops/roi_align.py", line 38, in backward
    output_size[0], output_size[1], bs, ch, h, w, sampling_ratio)
RuntimeError: CUDA out of memory. Tried to allocate 132.00 MiB (GPU 0; 3.81 GiB total capacity; 2.36 GiB already allocated; 132.69 MiB free; 310.59 MiB cached) (malloc at /pytorch/c10/cuda/CUDACachingAllocator.cpp:267)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x33 (0x7fdfb6c9b813 in /usr/local/lib/python3.6/dist-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x1ce68 (0x7fdfb6edce68 in /usr/local/lib/python3.6/dist-packages/torch/lib/libc10_cuda.so)
frame #2: <unknown function> + 0x1de6e (0x7fdfb6edde6e in /usr/local/lib/python3.6/dist-packages/torch/lib/libc10_cuda.so)
frame #3: at::native::empty_cuda(c10::ArrayRef<long>, c10::TensorOptions const&, c10::optional<c10::MemoryFormat>) + 0x279 (0x7fdf59472789 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch.so)
[many more frame lines omitted]

Clearly the line:

RuntimeError: CUDA out of memory. Tried to allocate 132.00 MiB (GPU 0; 3.81 GiB total capacity; 2.36 GiB already allocated; 132.69 MiB free; 310.59 MiB cached) (malloc at /pytorch/c10/cuda/CUDACachingAllocator.cpp:267)

is the critical error.

If I run an nvidia-smi before a run:

$ nvidia-smi
Tue Dec 24 14:32:49 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.44       Driver Version: 440.44       CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 1650    Off  | 00000000:01:00.0  On |                  N/A |
| N/A   47C    P8     5W /  N/A |    296MiB /  3903MiB |      3%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      1190      G   /usr/lib/xorg/Xorg                           142MiB |
|    0      1830      G   /usr/bin/gnome-shell                          72MiB |
|    0      3711      G   ...uest-channel-token=14371934934688572948    78MiB |
+-----------------------------------------------------------------------------+

It seems pretty clear there is plenty of GPU memory available (this GPU is 4GB).

Moreover, I'm confident my CUDA/cuDNN install and GPU hardware are good b/c I train and inference the TensorFlow object detection API on this computer frequently, and as long as I use the allow_growth option I never have GPU related errors.

From Googling on this error it seems to be relatively common.  The most common solutions are:

1) Try a smaller batch size (not really applicable in this case since the training and testing batch sizes are 2 and 1 respectively, and I tried with 1 and 1 and still got the same error)

2) Update to the latest version of PyTorch (but I'm already at the latest version).

Some other suggestions involve reworking the training script.  I'm very familiar with TensorFlow but I'm new to PyTorch so I'm not sure how to go about that.  Also, most of the rework suggestions I can find for this error do not pertain to object detection and therefore I'm not able to relate them to this training script specifically.

Has anybody else gotten this script to run locally with an NVIDIA GPU?  Do you suspect a OS/CUDA/PyTorch configuration concern, or is there someway the script can be reworked to prevent this error?  Any assistance would be greatly appreciated.


title: How to get the probability of a particular token(word) in a sentence given the context

body:
I'm trying to calculate the probability or any type of score for words in a sentence using NLP. I've tried this approach with GPT2 model using Huggingface Transformers library, but, I couldn't get satisfactory results due to the model's unidirectional nature which for me didn't seem to predict within context. So I was wondering whether there is a way, to calculate the above said using BERT since it's Bidirectional.

I've found this post relatable, which I randomly saw the other day but didn't see any answer which would be useful for me as well.

Hope I will be able to receive ideas or a solution for this. Any help is appreciated. Thank you.


title: How can I save my training progress in PyTorch for a certain batch no.?

body:
I'm simply trying to train a ResNet18 model using PyTorch library. The training dataset consists of 25,000 images. Therefore, it is taking a lot of time for even the first epoch to complete. Therefore, I want to save the progress after a certain no. of batch iteration is completed. But I can't figure out how to modify my code and how to use the torch.save() and torch.load() functions in my code to save the periodic progress.
My code is given below:

                # BUILD THE NETWORK
import torch
import torch.nn as nn
import torch.optim as optim
import torch.utils.data
import torch.nn.functional as F
import torchvision
import torchvision.models as models
from torchvision import transforms
from PIL import Image
import matplotlib.pyplot as plt

                # DOWNLOAD PRETRAINED MODELS ON ImageNet

model_resnet18 = torch.hub.load('pytorch/vision', 'resnet18', pretrained = True)
model_resnet34 = torch.hub.load('pytorch/vision', 'resnet34', pretrained = True)

for name, param in model_resnet18.named_parameters():
    if('bn' not in name):
        param.requires_grad = False

for name, param in model_resnet34.named_parameters():
    if('bn' not in name):
        param.requires_grad = False

num_classes = 2

model_resnet18.fc = nn.Sequential(nn.Linear(model_resnet18.fc.in_features, 512),
                                  nn.ReLU(),
                                  nn.Dropout(),
                                  nn.Linear(512, num_classes))

model_resnet34.fc = nn.Sequential(nn.Linear(model_resnet34.fc.in_features, 512),
                                  nn.ReLU(),
                                  nn.Dropout(),
                                  nn.Linear(512, num_classes))

                    # FUNCTIONS FOR TRAINING AND LOADING DATA

def train(model, optimizer, loss_fn, train_loader, val_loader, epochs = 5, device = "cuda"):
    print("Inside Train Function\n")
    for epoch in range(epochs):
        print("Epoch : {} running".format(epoch))
        training_loss = 0.0
        valid_loss = 0.0
        model.train()
        k = 0
        for batch in train_loader:
            optimizer.zero_grad()
            inputs, targets = batch
            inputs = inputs.to(device)
            output = model(inputs)
            loss = loss_fn(output, targets)
            loss.backward()
            optimizer.step()
            training_loss += loss.data.item() * inputs.size(0)
            print("End of batch loop iteration {} \n".format(k))
            k = k + 1
        training_loss /= len(train_loader.dataset)

        model.eval()
        num_correct = 0
        num_examples = 0
        for batch in val_loader:
            inputs, targets = batch
            inputs.to(device)
            output = model(inputs)
            targets = targets.to(device)
            loss = loss_fn(output, targets)
            valid_loss += loss.data.item() * inputs.size(0)

            correct = torch.eq(torch.max(F.softmax(output, dim = 1), dim = 1)[1], targets).view(-1)
            num_correct += torch.sum(correct).item()
            num_examples += correct.shape[0]
        valid_loss /= len(val_loader.dataset)

        print('Epoch: {}, Training Loss: {:.4f}, Validation Loss: {:.4f}, accuracy = {:.4f}'.format(epoch, training_loss, valid_loss, num_correct / num_examples))

batch_size = 32
img_dimensions = 224

img_transforms = transforms.Compose([ transforms.Resize((img_dimensions, img_dimensions)),
                                      transforms.ToTensor(),
                                      transforms.Normalize(mean = [0.485, 0.456, 0.406], std = [0.229, 0.224, 0.225])
])

img_test_transforms = transforms.Compose([ transforms.Resize((img_dimensions, img_dimensions)),
                                           transforms.ToTensor(),
                                           transforms.Normalize(mean = [0.485, 0.456, 0.406], std = [0.229, 0.224, 0.225])
])

def check_image(path):
    try:
        im = Image.open(path)
        return True
    except:
        return False

train_data_path = "E:\Image Recognition\dogsandcats\\train\\"
train_data = torchvision.datasets.ImageFolder(root=train_data_path,transform=img_transforms, is_valid_file=check_image)

validation_data_path = "E:\\Image Recognition\\dogsandcats\\validation\\"
validation_data = torchvision.datasets.ImageFolder(root=validation_data_path,transform=img_test_transforms, is_valid_file=check_image)

test_data_path = "E:\\Image Recognition\\dogsandcats\\test\\"
test_data = torchvision.datasets.ImageFolder(root=test_data_path,transform=img_test_transforms, is_valid_file=check_image)

num_workers = 6
train_data_loader      = torch.utils.data.DataLoader(train_data, batch_size=batch_size, shuffle=True, num_workers=num_workers)
validation_data_loader = torch.utils.data.DataLoader(validation_data, batch_size=batch_size, shuffle=False, num_workers=num_workers)
test_data_loader       = torch.utils.data.DataLoader(test_data, batch_size=batch_size, shuffle=False, num_workers=num_workers)

print(torch.cuda.is_available(), "\n")

if torch.cuda.is_available():
    device = torch.device("cuda")
else:
    device = torch.device("cpu")

print(f'Num training images: {len(train_data_loader.dataset)}')
print(f'Num validation images: {len(validation_data_loader.dataset)}')
print(f'Num test images: {len(test_data_loader.dataset)}')

def test_model(model):
    print("Inside Test Model Function\n")
    correct = 0
    total = 0
    with torch.no_grad():
        for data in test_data_loader:
            images, labels = data[0].to(device), data[1].to(device)
            outputs = model(images)
            _, predicted = torch.max(outputs.data, 1)
            total += labels.size(0)
            correct += (predicted == labels).sum().item()
    print('correct: {:d}  total: {:d}'.format(correct, total))
    print('accuracy = {:f}'.format(correct / total))

model_resnet18.to(device)
optimizer = optim.Adam(model_resnet18.parameters(), lr=0.001)
if __name__ == "__main__":
    train(model_resnet18, optimizer, torch.nn.CrossEntropyLoss(), train_data_loader, validation_data_loader, epochs=2, device=device)
    test_model(model_resnet18)

model_resnet34.to(device)
optimizer = optim.Adam(model_resnet34.parameters(), lr=0.001)
if __name__ == "__main__":
    train(model_resnet34, optimizer, torch.nn.CrossEntropyLoss(), train_data_loader, validation_data_loader, epochs=2, device=device)
    test_model(model_resnet34)

import os
def find_classes(dir):
    classes = os.listdir(dir)
    classes.sort()
    class_to_idx = {classes[i]: i for i in range(len(classes))}
    return classes, class_to_idx

def make_prediction(model, filename):
    labels, _ = find_classes('E:\\Image Recognition\\dogsandcats\\test\\test')
    img = Image.open(filename)
    img = img_test_transforms(img)
    img = img.unsqueeze(0)
    prediction = model(img.to(device))
    prediction = prediction.argmax()
    print(labels[prediction])

make_prediction(model_resnet34, 'E:\\Image Recognition\\dogsandcats\\test\\test\\3.jpg') #dog
make_prediction(model_resnet34, 'E:\\Image Recognition\\dogsandcats\\test\\test\\5.jpg') #cat

torch.save(model_resnet18.state_dict(), "./model_resnet18.pth")
torch.save(model_resnet34.state_dict(), "./model_resnet34.pth")

# Remember that you must call model.eval() to set dropout and batch normalization layers to
# evaluation mode before running inference. Failing to do this will yield inconsistent inference results.

resnet18 = torch.hub.load('pytorch/vision', 'resnet18')
resnet18.fc = nn.Sequential(nn.Linear(resnet18.fc.in_features,512),nn.ReLU(), nn.Dropout(), nn.Linear(512, num_classes))
resnet18.load_state_dict(torch.load('./model_resnet18.pth'))
resnet18.eval()

resnet34 = torch.hub.load('pytorch/vision', 'resnet34')
resnet34.fc = nn.Sequential(nn.Linear(resnet34.fc.in_features,512),nn.ReLU(), nn.Dropout(), nn.Linear(512, num_classes))
resnet34.load_state_dict(torch.load('./model_resnet34.pth'))
resnet34.eval()

# Test against the average of each prediction from the two models
models_ensemble = [resnet18.to(device), resnet34.to(device)]

correct = 0
total = 0

if __name__ == '__main__':
    with torch.no_grad():
        for data in test_data_loader:
            images, labels = data[0].to(device), data[1].to(device)
            predictions = [i(images).data for i in models_ensemble]
            avg_predictions = torch.mean(torch.stack(predictions), dim=0)
            _, predicted = torch.max(avg_predictions, 1)
            total += labels.size(0)
            correct += (predicted == labels).sum().item()

if total != 0:
    print('accuracy = {:f}'.format(correct / total))
print('correct: {:d}  total: {:d}'.format(correct, total))

To be very precise, I want to save my progress at the end of  for batch in train_loader: loop, for say k = 1500.
If anyone can guide me about modifying my code so that I can save my progress and resume it later, then it will be a great and highly appreciated.


title: Is there a way to use conda to install libraries in a Docker image?

body:
I'm trying to install some libraries (specifically pytorch) in my docker image.  I have a Dockerfile that installs anaconda properly, but now I'd like to use conda to install a few other things in the image.  Is there a way to do this?  I've tried

RUN /opt/anaconda/bin/conda install -y pytorch torchvision

And anaconda is installed in the right location.  Am I doing this the right way?


title: Loss is not decreasing for convolutional autoencoder

body:
I'm trying to train a convolutional autoencoder to encode and decode a piano roll representation of monophonic midi clips. I reduced the note range to 3 octaves, divide songs into 100 time step pieces (where 1 time step = 1/100th of a second), and train the net in batches of 3 pieces.

I'm using Adagrad as my optimizer, and MSE as my loss function. The loss is huge, and I see no decrease in average loss even after hundreds of training examples are fed in.

Here's my code:

"""
Most absolutely simple assumptions:
  - not changing the key of any of the files
  - not changing the tempo of any of the files

- take blocks of 36 by 100
- divide up all songs by this amount, cutting off any excess from the
end, train
"""
from __future__ import print_function
import cPickle as pickle
import numpy as np
import torch
from torch.autograd import Variable
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

from reverse_pianoroll import piano_roll_to_pretty_midi as pr2pm

N = 1000
# load a NxMxC dataset
    # N: Number of clips
    # M: Piano roll size, the number of midi notes that could possibly be 'on'
    # C: Clip length, in 100ths of a second
dataset = pickle.load(open('mh-midi-data.pickle', 'rb'))
######## take a subset of the data for training ######
# based on the mean and standard deviation of non zero entries in the data, I've
# found that the most populous, and thus best range of notes to take is from
# 48 to 84 (C2 - C5); this is 3 octaves, which is much less than the original
# 10 and a half. Additionally, we're going to take a subsample of 1000 because
# i'm training on my macbook and the network is pretty simple
######################################################
dataset = dataset[:, :, 48:84, :]
dataset = dataset[:N]
######################################################

midi_dim, clip_len = dataset.shape[2:]

class Autoencoder(nn.Module):
    def __init__(self, **kwargs):
        super(Autoencoder, self).__init__(**kwargs)
        # input is 3 x 1 x 36 x 100
        self.conv1 = nn.Conv2d(in_channels=1, out_channels=14, kernel_size=(midi_dim, 2))
        # now transformed to 3 x 14 x 1 x 99
        self.conv2 = nn.Conv2d(in_channels=14, out_channels=77, kernel_size=(1, 4))
        # now transformed to 3 x 77 x 1 x 96
        input_size = 3*77*1*96
        self.fc1 = nn.Linear(input_size, input_size/2)
        self.fc2 = nn.Linear(input_size/2, input_size/4)
        self.fc3 = nn.Linear(input_size/4, input_size/2)
        self.fc4 = nn.Linear(input_size/2, input_size)
        self.tconv2 = nn.ConvTranspose2d(in_channels=77, out_channels=14, kernel_size=(1, 4))
        self.tconv1 = nn.ConvTranspose2d(in_channels=14, out_channels=1, kernel_size=(midi_dim, 2))
        self.sigmoid = nn.Sigmoid()
        return

    def forward(self, x):
        # print("1: {}".format(x.size()))
        x = F.relu(self.conv1(x))
        # print("2: {}".format(x.size()))
        x = F.relu(self.conv2(x))
        # print("3: {}".format(x.size()))
        x = x.view(-1, np.prod(x.size()[:]))
        # print("4: {}".format(x.size()))
        x = F.relu(self.fc1(x))
        # print("5: {}".format(x.size()))
        h = F.relu(self.fc2(x))
        # print("6: {}".format(h.size()))
        d = F.relu(self.fc3(h))
        # print("7: {}".format(d.size()))
        d = F.relu(self.fc4(d))
        # print("8: {}".format(d.size()))
        d = d.view(3, 77, 1, 96)
        # print("9: {}".format(d.size()))
        d = F.relu(self.tconv2(d))
        # print("10: {}".format(d.size()))
        d = self.tconv1(d)
        d = self.sigmoid(d)
        # print("11: {}".format(d.size()))
        return d

net = Autoencoder()
loss_fn = nn.MSELoss()
# optimizer = optim.SGD(net.parameters(), lr=1e-3, momentum=0.9)
optimizer = optim.Adagrad(net.parameters(), lr=1e-3)

batch_count = 0
avg_loss = 0.0
print_every = 3
print("Beginning Training")
for epoch in xrange(2):
    # for i, clip in enumerate(dataset):
    for i in xrange(len(dataset)/3):
        batch = dataset[(3*i):(3*i + 3), :, :]
        # get the input, wrap it in a Variable
        inpt = Variable(torch.from_numpy(batch).type(torch.FloatTensor))

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outpt = net(inpt)
        loss = loss_fn(outpt, inpt)
        loss.backward()
        optimizer.step()

        # print stats out
        avg_loss += loss.data[0]
        if batch_count % print_every == print_every - 1:
           print('epoch: %d, batch_count: %d, loss: %.3f'%(
                epoch + 1, batch_count + 1, avg_loss / print_every))
           avg_loss = 0.0
        batch_count += 1

print('Finished Training')

I'm really a beginner with this stuff, so any advice would be greatly appreciated.


title: Pytorch LSTM grad only on last output

body:
I'm working with sequences of different lengths. But I would only want to grad them based on the output computed at the end of the sequence.

The samples are ordered so that they are decreasing in length and they are zero-padded. For 5 1D samples it looks like this (omitting width dimension for visibility):

array([[5, 7, 7, 4, 5, 8, 6, 9, 7, 9],
       [6, 4, 2, 2, 6, 5, 4, 2, 2, 0],
       [4, 6, 2, 4, 5, 1, 3, 1, 0, 0],
       [8, 8, 3, 7, 7, 7, 9, 0, 0, 0],
       [3, 2, 7, 5, 7, 0, 0, 0, 0, 0]])

For the LSTM I'm using nn.utils.rnn.pack_padded_sequence with the individual sequence lengths:

x = nn.utils.rnn.pack_padded_sequence(x, [10, 9, 8, 7, 5], batch_first=True)

The initialization of LSTM in the Model constructor:

self.lstm = nn.LSTM(width, n_hidden, 2)

Then I call the LSTM and unpack the values:

x, _ = self.lstm(x)
x = nn.utils.rnn.pad_packed_sequence(x1, batch_first=True)

Then I'm applying a fully connected layer and a softmax

x = x.contiguous()
x = x.view(-1, n_hidden)
x = self.linear(x)
x = x.reshape(batch_size, n_labels, 10) # 10 is the sample height
return F.softmax(x, dim=1)

This gives me an output of shape batch x n_labels x height (5x12x10).

For each sample, I would only want to use a single score, for the last output batch x n_labels (5*12). My question is How can I achieve this?

One idea is to apply tanh on the last hidden layer returned from the model but I'm not quite sure if that would give the same results. Is it possible to efficiently extract the output computed at the end of the sequence eg using the same lengths sequence used for pack_padded_sequence?


title: how can I remove layer in Pytorch?

body:
I want to remove the decoder portion of the Autoencoder.

and I want to put FC in the removed part.

In addition, the encoder parts will not train with pre-learned weights.

    self.encoder = nn.Sequential(
        nn.Conv2d(1, 16, 3, padding=1),
        nn.ReLU(True),
        nn.MaxPool2d(kernel_size=2, stride=2),
        nn.Conv2d(16, 8, 3, padding=1),
        nn.ReLU(True),
        nn.MaxPool2d(kernel_size=2, stride=2),
        nn.Conv2d(8, 8, 3, padding=1),
        nn.ReLU(True),
        nn.MaxPool2d(kernel_size=4, stride=1),

    )
    self.decoder = nn.Sequential(
        nn.Conv2d(8, 8, 3, padding=1),
        nn.ReLU(True),
        nn.ConvTranspose2d(8, 8, kernel_size=2, stride=2),
        nn.Conv2d(8, 8, 3, padding=1),
        nn.ReLU(True),
        nn.ConvTranspose2d(8, 8, kernel_size=2, stride=2),
        nn.Conv2d(8, 16, 3),
        nn.ReLU(True),
        nn.ConvTranspose2d(16, 16, kernel_size=2, stride=2),
        nn.Conv2d(16, 1, 3, padding=1)
    )
    def forward(self, x):
       if self.training :
           x = self.encoder(x)
           x = self.decoder(x)
           return x
       else:
           x = classifier(x)
           return x

is this possible?
help me...


title: Finetuning GPT-2 in huggingface's pytorch-transformers library

body:
I am looking to finetune the huggingface's GPT-2 PyTorch model.
https://github.com/huggingface/pytorch-transformers/tree/master

I read the 'Examples' section in their documentation where they have shown an example of training the old GPT model on RocStories dataset.
https://huggingface.co/pytorch-transformers/examples.html#fine-tuning

I would like to know how to perform this kind of training for GPT-2. Also, my dataset doesn't have an 'Evaluation Set'. I simply want to finetune GPT-2 on my dataset and then use it to generate new text which is 'similar' to the dataset it was finetuned on.

I am pretty sure I am missing something somewhere in the documentation.
I would be glad if anyone can point me in the right direction.


title: inconsistent error of : TypeError: super(type, obj): obj must be an instance or subtype of type

body:
I have a python code with the following two classes.

import torch
import torch.nn as nn
import torch.nn.functional as F

class QNet_baseline(nn.Module):

    """
        A MLP with 2 hidden layer

        observation_dim (int): number of observation features
        action_dim (int): Dimension of each action
        seed (int): Random seed
    """

    def __init__(self, observation_dim, action_dim, seed):
        super(QNet_baseline, self).__init__()
        self.seed = torch.manual_seed(seed)
        self.fc1 = nn.Linear(observation_dim, 128)
        self.fc2 = nn.Linear(128, 64)
        self.fc3 = nn.Linear(64, action_dim)

    def forward(self, observations):
        """
           Forward propagation of neural network

        """

        x = F.relu(self.fc1(observations))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

class QNet_3hidden(nn.Module):

    """
        A MLP with 3 hidden layer

        observation_dim (int): number of observation features
        action_dim (int): Dimension of each action
        seed (int): Random seed
    """

    def __init__(self, observation_dim, action_dim, seed):
        super(QNet_3hidden, self).__init__()
        self.seed = torch.manual_seed(seed)
        self.fc1 = nn.Linear(observation_dim, 128)
        self.fc2 = nn.Linear(128, 64)
        self.fc3 = nn.Linear(64, 64)
        self.fc4 = nn.Linear(64, action_dim)

    def forward(self, observations):
        """
           Forward propagation of neural network

        """

        x = F.relu(self.fc1(observations))
        x = F.relu(self.fc2(x))
        x = F.relu(self.fc3(x))
        x = self.fc4(x)
        return x

I used the same code to instantiate both classes. QNet_baseline works fine, but I got the following error for QNet_3hidden. Why would QNet_baseline works but QNet_3hidden has an error? What did I miss here? Thanks!

/home/workspace/QNetworks.py in __init__(self, observation_dim, action_dim, seed)
     44
     45     def __init__(self, observation_dim, action_dim, seed):
---> 46         super(QNet_3hidden, self).__init__()
     47         self.seed = torch.manual_seed(seed)
     48         self.fc1 = nn.Linear(observation_dim, 128)

TypeError: super(type, obj): obj must be an instance or subtype of type

Also, below is how the two classes instantiated:

class DDQN_Agent():
    """Interacts with and learns from the environment.

    Attributes:
        state_size (int): dimension of each state
        action_size (int): dimension of each action
        seed (int): random seed
    """

    def __init__(self, state_size, action_size, seed, qnet="baseline", filename=None):
        """Initialize an Agent object.

        Args:
            filename: path of .pth file with trained weights
        """
        self.state_size = state_size
        self.action_size = action_size
        self.seed = random.seed(seed)

        # Q-Network
        if qnet=="3hidden":
            self.qnetwork_local = QNet_3hidden(state_size, action_size, seed).to(device)
            self.qnetwork_target = QNet_3hidden(state_size, action_size, seed).to(device)
            self.optimizer = optim.Adam(self.qnetwork_local.parameters(), lr=LR)
        else:
            self.qnetwork_local = QNet_baseline(state_size, action_size, seed).to(device)
            self.qnetwork_target = QNet_baseline(state_size, action_size, seed).to(device)
            self.optimizer = optim.Adam(self.qnetwork_local.parameters(), lr=LR)

        if filename:
            weights = torch.load(filename)
            self.qnetwork_local.load_state_dict(weights)
            self.qnetwork_target.load_state_dict(weights)

        # Replay memory
        self.memory = ReplayBuffer(action_size, BUFFER_SIZE, BATCH_SIZE, seed)
        # Initialize time step (for updating every UPDATE_EVERY steps)
        self.t_step = 0


title: How can I get the MSE of a tensor across a specific dimension?

body:
I have 2 tensors with .size of torch.Size([2272, 161]). I want to get mean-squared-error between them. However, I want it along each of the 161 channels, so that my error tensor has a .size of torch.Size([161]). How can I accomplish this?

It seems that torch.nn.MSELoss doesn't let me specify a dimension.


title: Munging PyTorch's tensor shape from (C, B, H) to (B, C*H)

body:
Given an input tensor of shape (C, B, H) torch.Size([2, 5, 32]) of some neural net layers, where

channels = 2
batch_size = 5
hidden_size = 32

The goal is to flatten the channels and manipulate the input tensor to the shape (B, C*H) torch.Size([5, 2 * 32]), where:

batch_size = 5
hidden_size = 32 * 2

I've tried to do the following:

import torch

t = torch.rand([2, 5, 32])

# Changed from (channels, batch_size, hidden_size)
# -> (batch_size, channels, hidden_size)
t = t.permute(1, 0, 2)

# Reshape using view(), where batch_size is t.size(0)
# and -1 is to flatten the left over values to the other dimension.
z = t.contiguous().view(t.size(0), -1)

print(z.shape)
print(z)

[out]:

torch.Size([5, 64])
tensor([[0.3911, 0.9586, 0.2104, 0.3937, 0.9976, 0.3378, 0.0630, 0.6676, 0.0806,
         0.9311, 0.5219, 0.1697, 0.7442, 0.5162, 0.2555, 0.0826, 0.5502, 0.9700,
         0.3375, 0.5012, 0.9025, 0.8176, 0.1465, 0.1848, 0.3460, 0.9999, 0.7892,
         0.7577, 0.6615, 0.2620, 0.6868, 0.2003, 0.4840, 0.8354, 0.9253, 0.3172,
         0.9516, 0.8962, 0.1272, 0.2268, 0.6510, 0.5166, 0.6772, 0.9616, 0.9826,
         0.5254, 0.9191, 0.4378, 0.7048, 0.8808, 0.0299, 0.1102, 0.9710, 0.8714,
         0.7256, 0.9684, 0.6117, 0.1957, 0.8663, 0.4742, 0.2843, 0.6548, 0.9592,
         0.1559],
        [0.2333, 0.0858, 0.5284, 0.2965, 0.3863, 0.3370, 0.6940, 0.3387, 0.3513,
         0.1022, 0.3731, 0.3575, 0.7095, 0.0053, 0.7024, 0.4091, 0.3289, 0.5808,
         0.5640, 0.8847, 0.7584, 0.8878, 0.9873, 0.0525, 0.7731, 0.2501, 0.9926,
         0.5226, 0.0925, 0.0300, 0.4176, 0.0456, 0.4643, 0.4497, 0.5920, 0.9519,
         0.6647, 0.2379, 0.4927, 0.9666, 0.1675, 0.9887, 0.7741, 0.5668, 0.7376,
         0.4452, 0.7449, 0.1298, 0.9065, 0.3561, 0.5813, 0.1439, 0.2115, 0.5874,
         0.2038, 0.1066, 0.3843, 0.6179, 0.8321, 0.9428, 0.1067, 0.5045, 0.9324,
         0.3326],
        [0.6556, 0.1479, 0.9288, 0.9238, 0.1324, 0.0718, 0.6620, 0.2659, 0.7162,
         0.7559, 0.7564, 0.2120, 0.3943, 0.9497, 0.7520, 0.8455, 0.4444, 0.4708,
         0.8371, 0.6365, 0.3616, 0.0326, 0.1581, 0.4973, 0.6701, 0.9245, 0.8274,
         0.3464, 0.7044, 0.5376, 0.0441, 0.5210, 0.8603, 0.7396, 0.2544, 0.3514,
         0.5686, 0.3283, 0.7248, 0.4303, 0.9531, 0.5587, 0.8703, 0.1585, 0.9161,
         0.9043, 0.9778, 0.4489, 0.9463, 0.8655, 0.5576, 0.1135, 0.1268, 0.3424,
         0.1504, 0.2265, 0.1734, 0.1872, 0.3995, 0.1191, 0.0532, 0.6109, 0.1662,
         0.6937],
        [0.6342, 0.1922, 0.1758, 0.4625, 0.7654, 0.6509, 0.2908, 0.1546, 0.4768,
         0.3779, 0.2490, 0.0086, 0.6170, 0.5425, 0.6953, 0.4730, 0.5834, 0.8326,
         0.0165, 0.8236, 0.0023, 0.7479, 0.5621, 0.9894, 0.5957, 0.0857, 0.6087,
         0.5667, 0.5478, 0.8197, 0.9228, 0.7329, 0.4434, 0.5894, 0.9860, 0.6133,
         0.2395, 0.4718, 0.8830, 0.6361, 0.6104, 0.6630, 0.5084, 0.7604, 0.7591,
         0.3601, 0.6888, 0.6767, 0.9178, 0.5291, 0.0591, 0.4320, 0.7875, 0.5038,
         0.4419, 0.0319, 0.3719, 0.5843, 0.0334, 0.3525, 0.0023, 0.1205, 0.4040,
         0.7908],
        [0.0989, 0.8436, 0.0425, 0.6247, 0.6091, 0.4778, 0.2692, 0.4785, 0.9217,
         0.9604, 0.6355, 0.4686, 0.9414, 0.7722, 0.8013, 0.1660, 0.6578, 0.6414,
         0.6814, 0.6212, 0.4124, 0.7102, 0.7416, 0.7404, 0.9842, 0.6542, 0.0106,
         0.3826, 0.5529, 0.8079, 0.9855, 0.3012, 0.2341, 0.9353, 0.6597, 0.7177,
         0.8214, 0.1438, 0.4729, 0.6747, 0.9310, 0.4167, 0.3689, 0.8464, 0.9395,
         0.9407, 0.8419, 0.5486, 0.1786, 0.1423, 0.9900, 0.9365, 0.3996, 0.1862,
         0.6232, 0.7547, 0.7779, 0.4767, 0.6218, 0.9079, 0.6153, 0.1488, 0.5960,
         0.4015]])

Although the permute() + view() achieve the desired output, are there other ways to perform the same operation? Is there a better way that can directly rehape without first permutating the order of the shape?


title: Pytorch:Correct way to use custom weight maps in unet architecture

body:
There is a famous trick in u-net architecture to use custom weight maps to increase accuracy.Below are the details of it-

Now, by asking here and at multiple other place, I get to know about 2 approaches.I want to know which one is correct or is there any other right approach which is more correct?

1)First is to use torch.nn.Functional method in the training loop-

loss = torch.nn.functional.cross_entropy(output, target, w)
where w will be the calculated custom weight.

2)Second is to use reduction='none' in the calling of loss function outside the training loop
criterion = torch.nn.CrossEntropy(reduction='none')

and then in the training loop multiplying with the custom weight-

gt # Ground truth, format torch.long
pd # Network output
W # per-element weighting based on the distance map from UNet
loss = criterion(pd, gt)
loss = W*loss # Ensure that weights are scaled appropriately
loss = torch.sum(loss.flatten(start_dim=1), axis=0) # Sums the loss per image
loss = torch.mean(loss) # Average across a batch

Now, I am kinda confused which one is right or is there any other way, or both are right?


title: fastai error predicting with exported/reloaded model: "Input type and weight type should be the same"

body:
Whenever I export a fastai model and reload it, I get this error (or a very similar one) when I try and use the reloaded model to generate predictions on a new test set:

RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.cuda.HalfTensor) should be the same

Minimal reprodudeable code example below, you just need to update your FILES_DIR variable to where the MNIST data gets deposited on your system:

from fastai import *
from fastai.vision import *

# download data for reproduceable example
untar_data(URLs.MNIST_SAMPLE)
FILES_DIR = '/home/mepstein/.fastai/data/mnist_sample'  # this is where command above deposits the MNIST data for me

# Create FastAI databunch for model training
tfms = get_transforms()
tr_val_databunch = ImageDataBunch.from_folder(path=FILES_DIR,  # location of downloaded data shown in log of prev command
                                train = 'train',
                                valid_pct = 0.2,
                                ds_tfms = tfms).normalize()

# Create Model
conv_learner = cnn_learner(tr_val_databunch,
                           models.resnet34,
                           metrics=[error_rate]).to_fp16()

# Train Model
conv_learner.fit_one_cycle(4)

# Export Model
conv_learner.export()  # saves model as 'export.pkl' in path associated with the learner

# Reload Model and use it for inference on new hold-out set
reloaded_model = load_learner(path = FILES_DIR,
                              test = ImageList.from_folder(path = f'{FILES_DIR}/valid'))

preds = reloaded_model.get_preds(ds_type=DatasetType.Test)

Output:

  "RuntimeError: Input type (torch.cuda.FloatTensor) and weight type
  (torch.cuda.HalfTensor) should be the same"

Stepping through the code statement by statement, everything works fine until the last line pred = ... which is where the torch error above pops up.

Relevant software versions:

Python 3.7.3
fastai           1.0.57
torch            1.2.0
torchvision      0.4.0


title: PyTorch why does the forward function run multiple times and can I change the input shape?

body:
import torch
import torch.nn as nn
import torchvision.datasets as dsets
from skimage import transform
import torchvision.transforms as transforms
from torch.autograd import Variable
import pandas as pd;
import numpy as np;
from torch.utils.data import Dataset, DataLoader
import statistics

import random
import math

class FashionMNISTDataset(Dataset):
  '''Fashion MNIST Dataset'''
  def __init__(self, csv_file, transform=None):
    """
    Args:
        csv_file (string): Path to the csv file
        transform (callable): Optional transform to apply to sample
    """

    data = pd.read_csv(csv_file)
    self.X = np.array(data.iloc[:, 1:]).reshape(-1, 1, 28, 28)
    self.Y = np.array(data.iloc[:, 0])

    del data
    self.transform = transform

  def __len__(self):
    return len(self.X)

  def __getitem__(self, idx):
    item = self.X[idx]
    label = self.Y[idx]

    if self.transform:
        item = self.transform(item)

    return (item, label)

class CNN(nn.Module):
  def __init__(self):
    super(CNN, self).__init__()
    self.layer1 = nn.Sequential(
      nn.Linear(616,300),
      nn.ReLU())
    self.layer2 = nn.Sequential(
      nn.Linear(300,100),
      nn.ReLU())
    self.fc = nn.Linear(100, 10)

  def forward(self, x):
    print("x shape",x.shape)
    out = self.layer1(x)
    out = self.layer2(out)
    out = self.fc(out)
    return out

def run():

  device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
  num_epochs = 15
  batch_size = 100
  learning_rate = 0.0001

  train_dataset = FashionMNISTDataset(csv_file='fashion-mnist_train.csv')
  test_dataset = FashionMNISTDataset(csv_file='fashion-mnist_test.csv')

  train_loader = torch.utils.data.DataLoader(dataset=train_dataset,batch_size=batch_size,shuffle=True)
  test_loader = torch.utils.data.DataLoader(dataset=test_dataset,batch_size=batch_size,shuffle=True)

  #instance of the Conv Net
  cnn = CNN()
  cnn.to(device)

  #loss function and optimizer
  criterion = nn.CrossEntropyLoss()
  optimizer = torch.optim.Adam(cnn.parameters(), lr=learning_rate)

  losses = []
  for epoch in range(num_epochs):
    l = 0
    for i, (images, labels) in enumerate(train_loader):

      images = Variable(images.float())
      labels = Variable(labels)
      #print(images[0])
      images = images.to(device)
      labels = labels.to(device)
      print("img shape=",images.shape, "label shape=",labels.shape)
      images = images.resize_((100,616))

      print("img shape=",images.shape, "label shape=",labels.shape)
      # Forward + Backward + Optimize
      optimizer.zero_grad()
      outputs = cnn(images)

      loss = criterion(outputs, labels)
      #print(loss)
      loss.backward()
      optimizer.step()
      #print(loss.item())
      losses.append(loss.item())
      l = loss.item()

      cnn.eval()
      with torch.no_grad():
        val_loss = []
        for images, labels in test_loader:
          images = Variable(images.float()).to(device)
          labels = labels.to(device)
          outputs = cnn.forward(images)
          batch_loss = criterion(outputs, labels)
          val_loss.append(batch_loss.item())
        avgloss = statistics.mean(val_loss)
        if avgloss < min(losses):
          torch.save(cnn.state_dict(), 'model')
        cnn.train()
      if (i+1) % 100 == 0:
        print ('Epoch : %d/%d, Iter : %d/%d,  Loss: %.4f'
              %(epoch+1, num_epochs, i+1, len(train_dataset)//batch_size, loss.item()))
    print(l)

  final_model = CNN()
  final_model.load_state_dict(torch.load('model'))
  final_model.eval()
  correct = 0
  total = 0
  for images, labels in test_loader:
      images = Variable(images.float()).to(device)
      outputs = final_model(images).to(device)
      labels.to(device)
      _, predicted = torch.max(outputs.data, 1)
      total += labels.size(0)
      correct += (predicted == labels).sum()
  print('Test Accuracy of the model on the 10000 test images: %.4f %%' % (100 * correct / total))

if __name__ == '__main__':
  run()

I have enclosed all the code for testing purposes. But Here is the error I get

  img shape= torch.Size([100, 1, 28, 28]) label shape= torch.Size([100])
  img shape= torch.Size([100, 616]) label shape= torch.Size([100]) x
  shape torch.Size([100, 616]) x shape torch.Size([100, 1, 28, 28])
  Traceback (most recent call last):   File "test.py", line 145, in

      run()   File "test.py", line 115, in run
      outputs = cnn.forward(images)   File "test.py", line 56, in forward
      out = self.layer1(x)   File "/usr/share/anaconda3/envs/DL/lib/python3.6/site-packages/torch/nn/modules/module.py",
  line 489, in call
      result = self.forward(*input, **kwargs)   File "/usr/share/anaconda3/envs/DL/lib/python3.6/site-packages/torch/nn/modules/container.py",
  line 92, in forward
      input = module(input)   File "/usr/share/anaconda3/envs/DL/lib/python3.6/site-packages/torch/nn/modules/module.py",
  line 489, in call
      result = self.forward(*input, **kwargs)   File "/usr/share/anaconda3/envs/DL/lib/python3.6/site-packages/torch/nn/modules/linear.py",
  line 67, in forward
      return F.linear(input, self.weight, self.bias)   File "/usr/share/anaconda3/envs/DL/lib/python3.6/site-packages/torch/nn/functional.py",
  line 1354, in linear
      output = input.matmul(weight.t()) RuntimeError: size mismatch, m1: [2800 x 28], m2: [616 x 300] at
  /opt/conda/conda-bld/pytorch_1549630534704/work/aten/src/THC/generic/THCTensorMathBlas.cu:266

The problem here is that I want all 616 pixels to feed as input into the neural network but I dont know how to do so. I tried to reshape the input to solve the problem but it ran model.forward twice, once with the correct shape and then the wrong shape.


title: Multiply rows of matrix by vector elementwise in pytorch?

body:
I would like to do the below but using PyTorch.

The below example and description is from this post.

  I have a numeric matrix with 25 columns and 23 rows, and a vector of
  length 25. How can I multiply each row of the matrix by the vector
  without using a for loop?

  The result should be a 25x23 matrix (the same size as the input), but
  each row has been multiplied by the vector.

Example Code in R (source: reproducible example from @hatmatrix's answer):

matrix <- matrix(rep(1:3,each=5),nrow=3,ncol=5,byrow=TRUE)

     [,1] [,2] [,3] [,4] [,5]
[1,]    1    1    1    1    1
[2,]    2    2    2    2    2
[3,]    3    3    3    3    3

vector <- 1:5

Desired output:

     [,1] [,2] [,3] [,4] [,5]
[1,]    1    2    3    4    5
[2,]    2    4    6    8   10
[3,]    3    6    9   12   15

What is the best way of doing this using Pytorch?


title: Fine tuning BART to generate Summary

body:
I am trying to fine tune the BART model to generate news headlines.

I am taking the dataset from Kaggle News Summary

Fine tuning Colab notebook

However, in the validation section when trying to generate the text from encoded tokens, i am running into error

---------------------------------------------------------------------------
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-22-059727516ce4> in <module>()
      1 for epoch in range(1):
----> 2     predictions, actuals = validate(epoch)
      3     writer(predictions, actuals)
      4     print('Output Files generated for review')

4 frames
<ipython-input-21-6738e7a3f1f9> in validate(epoch)
     10             mask = data['source_mask'].to(device, dtype = torch.long)
     11
---> 12             generate_ids = model.generate(input_ids = ids,attention_mask = mask, num_beams=4,repetition_penalty=2.5,length_penalty=2.0,early_stopping=True)
     13             preds = [tokenizer.decode(g, skip_special_tokens=True, clean_up_tokenization_spaces=True) for g in generate_ids]
     14             target = [tokenizer.decode(t, skip_special_tokens=True, clean_up_tokenization_spaces=True)for t in y]

/usr/local/lib/python3.6/dist-packages/torch/autograd/grad_mode.py in decorate_context(*args, **kwargs)
     13         def decorate_context(*args, **kwargs):
     14             with self:
---> 15                 return func(*args, **kwargs)
     16         return decorate_context
     17

/usr/local/lib/python3.6/dist-packages/transformers/modeling_utils.py in generate(self, input_ids, max_length, min_length, do_sample, early_stopping, num_beams, temperature, top_k, top_p, repetition_penalty, bad_words_ids, bos_token_id, pad_token_id, eos_token_id, length_penalty, no_repeat_ngram_size, num_return_sequences, attention_mask, decoder_start_token_id, use_cache, **model_specific_kwargs)
    914
    915         # We cannot generate if the model does not have a LM head
--> 916         if self.get_output_embeddings() is None:
    917             raise AttributeError(
    918                 "You tried to generate sequences with a model that does not have a LM Head."

/usr/local/lib/python3.6/dist-packages/transformers/modeling_bart.py in get_output_embeddings(self)
   1021
   1022     def get_output_embeddings(self):
-> 1023         return _make_linear_from_emb(self.model.shared)  # make it on the fly
   1024
   1025

/usr/local/lib/python3.6/dist-packages/transformers/modeling_bart.py in _make_linear_from_emb(emb)
    148     vocab_size, emb_size = emb.weight.shape
    149     lin_layer = nn.Linear(vocab_size, emb_size, bias=False)
--> 150     lin_layer.weight.data = emb.weight.data
    151     return lin_layer
    152

RuntimeError: Attempted to call `variable.set_data(tensor)`, but `variable` and `tensor` have incompatible tensor type.

It will be great if i could get some guidance from the group. Thanks!


title: Deep learning: save and load a universal machine model through different libraries

body:
My questions can be divided into two parts.

Is there a format of machine learning model file that can be used through different libraries? For example, I saved a model by pytorch, then load it using tensorflow?
If not, is there a library that can help transfer the formats so that a pytorch machine learning model can be used directly in keras?

The reason why I ask this question is that recently I need to adjust some of my previous trained models in tensorflow to pytorch.

An update for this question:
Facebook and Microsoft are going to launch a model standard called ONNX, which is used for transferring models between different framworks, for example between Pytorch to Caffe2. Link in the following:

https://research.fb.com/facebook-and-microsoft-introduce-new-open-ecosystem-for-interchangeable-ai-frameworks/

An further update for this question:
Tensorflow itself use Protocol Buffer format to store model file, which can be used for transfer between different models. Link in the following:

https://www.tensorflow.org/extend/tool_developers/


I've been struggling to understand the differences between .clone(), .detach() and copy.deepcopy when using Pytorch. In particular with Pytorch tensors.
I tried writing all my question about their differences and uses cases and became overwhelmed quickly and realized that perhaps have the 4 main properties of Pytorch tensors would clarify much better which one to use that going through every small question. The 4 main properties I realized one needs keep track are:

if one has a new pointer/reference to a tensor
if one has a new tensor object instance (and thus most likely this new instance has it's own meta-data like require_grads, shape, is_leaf, etc.)
if it has allocated a new memory for the tensor data (i.e. if this new tensor is a view of a different tensor)
if it's tracking the history of operations or not (or even if it's tracking a completely new history of operations or the same old one in the case of deep copy)

According to what mined out from the Pytorch forums and the documentation this is my current distinctions for each when used on tensors:
Clone
For clone:
x_cloned = x.clone()

I believe this is how it behaves according to the main 4 properties:

the cloned x_cloned  has it's own python reference/pointer to the new object
it has created it's own new tensor object instance (with it's separate meta-data)
it has allocated a new memory for x_new with the same data as x
it is keeping track of the original history of operations and in addition included this clone operation as .grad_fn=<CloneBackward>

it seems that the main use of this as I understand is to create copies of things so that inplace_ operations are safe. In addition coupled with .detach as .detach().clone() (the "better" order to do it btw) it creates a completely new tensor that has been detached with the old history and thus stops gradient flow through that path.
Detach
x_detached = x.detach()

creates a new python reference (the only one that does not is doing x_new = x of course). One can use id for this one I believe
it has created it's own new tensor object instance (with it's separate meta-data)
it has NOT allocated a new memory for x_detached with the same data as x
it cuts the history of the gradients and does not allow it to flow through it. I think it's right to think of it as having no history, as a brand new tensor.

I believe the only sensible use I know of is of creating new copies with it's own memory when coupled with .clone() as .detach().clone(). Otherwise, I am not sure what the use it. Since it points to the original data, doing in place ops might be potentially dangerous (since it changes the old data but the change to the old data is NOT known by autograd in the earlier computation graph).
copy.deepcopy
x_deepcopy = copy.deepcopy(x)

if one has a new pointer/reference to a tensor
it creates a new tensor instance with it's own meta-data (all of the meta-data should point to deep copies, so new objects if it's implemented as one would expect I hope).
it has it's own memory allocated for the tensor data
If it truly is a deep copy, I would expect a deep copy of the history. So it should do a deep replication of the history. Though this seems really expensive but at least semantically consistent with what deep copy should be.

I don't really see a use case for this. I assume anyone trying to use this really meant 1) .detach().clone() or just 2) .clone() by itself, depending if one wants to stop gradient flows to the earlier graph with 1 or if they want just to replicate the data with a new memory 2).
So this is the best way I have to understand the differences as of now rather than ask all the different scenarios that one might use them.
So is this right? Does anyone see any major flaw that needs to be correct?
My own worry is about the semantics I gave to deep copy and wonder if it's correct wrt the deep copying the history.
I think a list of common use cases for each would be wonderful.

Resources
these are all the resources I've read and participated to arrive at the conclusions in this question:

Migration guide to 0.4.0 https://pytorch.org/blog/pytorch-0_4_0-migration-guide/
Confusion about using clone: https://discuss.pytorch.org/t/confusion-about-using-clone/39673/3
Clone and detach in v0.4.0: https://discuss.pytorch.org/t/clone-and-detach-in-v0-4-0/16861/2
Docs for clone:

https://pytorch.org/docs/stable/tensors.html#torch.Tensor.clone

Docs for detach (search for the word detach in your browser there is no direct link):

https://pytorch.org/docs/stable/tensors.html#torch.Tensor

Difference between detach().clone() and clone().detach(): https://discuss.pytorch.org/t/difference-between-detach-clone-and-clone-detach/34173
Why am I able to change the value of a tensor without the computation graph knowing about it in Pytorch with detach? Why am I able to change the value of a tensor without the computation graph knowing about it in Pytorch with detach?
What is the difference between detach, clone and deepcopy in Pytorch tensors in detail? What is the difference between detach, clone and deepcopy in Pytorch tensors in detail?
Copy.deepcopy() vs clone() https://discuss.pytorch.org/t/copy-deepcopy-vs-clone/55022/10


title: Implementing RNN and LSTM into DQN Pytorch code

body:
I have some troubles finding some example on the great www to how i implement a recurrent neural network with LSTM layer into my current Deep q-network in Pytorch so it become a DRQN.. Bear with me i am just getting started..
Futhermore, I am NOT working with images processing, thereby CNN so do not worry about this. My states are purely temperatures values.

Here is my code that i am currently train my DQN with:

# Importing the libraries

import numpy as np
import random # random samples from different batches (experience replay)
import os # For loading and saving brain
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim # for using stochastic gradient descent
import torch.autograd as autograd # Conversion from tensor (advanced arrays) to avoid all that contains a gradient
# We want to put the tensor into a varaible taht will also contain a
# gradient and to this we need:
from torch.autograd import Variable
# to convert this tensor into a variable containing the tensor and the gradient

# Creating the architecture of the Neural Network
class Network(nn.Module): #inherinting from nn.Module

    #Self - refers to the object that will be created from this class
    #     - self here to specify that we're referring to the object
    def __init__(self, input_size, nb_action): #[self,input neuroner, output neuroner]
        super(Network, self).__init__() #inorder to use modules in torch.nn
        # Input and output neurons
        self.input_size = input_size
        self.nb_action = nb_action
        # Full connection between different layers of NN
        # In this example its one input layer, one hidden layer and one output layer
        # Using self here to specify that fc1 is a variable of my object
        self.fc1 = nn.Linear(input_size, 40)
        self.fc2 = nn.Linear(40, 30)
        #Example of adding a hiddenlayer
        # self.fcX = nn.Linear(30,30)
        self.fc3 = nn.Linear(30, nb_action) # 30 neurons in hidden layer

    # For function that will activate neurons and perform forward propagation
    def forward(self, state):
        # rectifier function
        x = F.relu(self.fc1(state))
        x = F.relu(self.fc2(x))
        q_values = self.fc3(x)
        return q_values

# Implementing Experience Replay
# We know that RL is based on MDP
# So going from one state(s_t) to the next state(s_t+1)
# We gonna put 100 transition between state into what we call the memory
# So we can use the distribution of experience to make a decision
class ReplayMemory(object):

    def __init__(self, capacity):
        self.capacity = capacity #100 transitions
        self.memory = [] #memory to save transitions

    # pushing transitions into memory with append
    #event=transition
    def push(self, event):
        self.memory.append(event)
        if len(self.memory) > self.capacity: #memory only contain 100 events
            del self.memory[0] #delete first transition from memory if there is more that 100

    # taking random sample
    def sample(self, batch_size):
        #Creating variable that will contain the samples of memory
        #zip =reshape function if list = ((1,2,3),(4,5,6)) zip(*list)= (1,4),(2,5),(3,6)
        #                (state,action,reward),(state,action,reward)
        samples = zip(*random.sample(self.memory, batch_size))
        #This is to be able to differentiate with respect to a tensor
        #and this will then contain the tensor and gradient
        #so for state,action and reward we will store the seperately into some
        #bytes which each one will get a gradient
        #so that eventually we'll be able to differentiate each one of them
        return map(lambda x: Variable(torch.cat(x, 0)), samples)

# Implementing Deep Q Learning

class Dqn():

    def __init__(self, input_size, nb_action, gamma, lrate, T):
        self.gamma = gamma #self.gamma gets assigned to input argument
        self.T = T
        # Sliding window of the evolving mean of the last 100 events/transitions
        self.reward_window = []
        #Creating network with network class
        self.model = Network(input_size, nb_action)
        #creating memory with memory class
        #We gonna take 100000 samples into memory and then we will sample from this memory to
        #to get a snakk number of random transitions
        self.memory = ReplayMemory(100000)
        #creating optimizer (stochastic gradient descent)
        self.optimizer = optim.Adam(self.model.parameters(), lr = lrate) #learning rate
        #input vector which is batch of input observations
        #by unsqeeze we create a fake dimension to this is
        #what the network expect for its inputs
        #have to be the first dimension of the last_state
        self.last_state = torch.Tensor(input_size).unsqueeze(0)
        #Inilizing
        self.last_action = 0
        self.last_reward = 0

    def select_action(self, state):
        #Q value depends on state
        #Temperature parameter T will be a positive number and the closer
        #it is to ze the less sure the NN will when taking an action
        #forexample
        #softmax((1,2,3))={0.04,0.11,0.85} ==> softmax((1,2,3)*3)={0,0.02,0.98}
        #to deactivate brain then set T=0, thereby it is full random
        probs = F.softmax((self.model(Variable(state, volatile = True))*self.T),dim=1) # T=100
        #create a random draw from the probability distribution created from softmax
        action = probs.multinomial()
        print(probs.multinomial())
        return action.data[0,0]

    # See section 5.3 in AI handbook
    def learn(self, batch_state, batch_next_state, batch_reward, batch_action):
        outputs = self.model(batch_state).gather(1, batch_action.unsqueeze(1)).squeeze(1)
        #next input for target see page 7 in attached AI handbook
        next_outputs = self.model(batch_next_state).detach().max(1)[0]
        target = self.gamma*next_outputs + batch_reward
        #Using hubble loss inorder to obtain loss
        td_loss = F.smooth_l1_loss(outputs, target)
        #using  lass loss/error to perform stochastic gradient descent and update weights
        self.optimizer.zero_grad() #reintialize the optimizer at each iteration of the loop
        #This line of code that backward propagates the error into the NN
        #td_loss.backward(retain_variables = True) #userwarning
        td_loss.backward(retain_graph = True)
        #And this line of code uses the optimizer to update the weights
        self.optimizer.step()

    def update(self, reward, new_signal):
        #Updated one transition and we have dated the last element of the transition
        #which is the new state
        new_state = torch.Tensor(new_signal).float().unsqueeze(0)
        self.memory.push((self.last_state, new_state, torch.LongTensor([int(self.last_action)]), torch.Tensor([self.last_reward])))
        #After ending in a state its time to play a action
        action = self.select_action(new_state)
        if len(self.memory.memory) > 100:
            batch_state, batch_next_state, batch_action, batch_reward = self.memory.sample(100)
            self.learn(batch_state, batch_next_state, batch_reward, batch_action)
        self.last_action = action
        self.last_state = new_state
        self.last_reward = reward
        self.reward_window.append(reward)
        if len(self.reward_window) > 1000:
            del self.reward_window[0]
        return action

    def score(self):
        return sum(self.reward_window)/(len(self.reward_window)+1.)

    def save(self):
        torch.save({'state_dict': self.model.state_dict(),
                    'optimizer' : self.optimizer.state_dict(),
                   }, 'last_brain.pth')

    def load(self):
        if os.path.isfile('last_brain.pth'):
            print("=> loading checkpoint... ")
            checkpoint = torch.load('last_brain.pth')
            self.model.load_state_dict(checkpoint['state_dict'])
            self.optimizer.load_state_dict(checkpoint['optimizer'])
            print("done !")
        else:
            print("no checkpoint found...")

I hope there is someone out there that can help me and could implement a RNN and a LSTM layer into my code! I believe in you stackflow!

Best regards Søren Koch


title: Does PyTorch's nn.DataParallel load the same model in each GPU?

body: The only way it would seem to work (logically) is if the model was loaded in each of the GPUs. This would mean that when weights are updated, each GPU would need to update the weights as well, increasing the workload compared to a single GPU. Is this line of reasoning correct?


title: Pytorch installation could not find a version that satisfies the requirement

body:
When i tried to install Pytorch in the way they suggest on their website:
pip install torch===1.7.0 torchvision===0.8.1 torchaudio===0.7.0 -f https://download.pytorch.org/whl/torch_stable.html

this is the error that appear:
ERROR: Could not find a version that satisfies the requirement torch===1.7.0 (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)
ERROR: No matching distribution found for torch===1.7.0

How can i solve it?


title: why "RuntimeError CUDA out of memory" in testing?

body:
The same model ran fine for training with batch-size=5. I reduced the batch size from 80 to 5 during training because of the same error. I am using a GPU with 11GB of memory instead of Titan X (12GB memory), the one used by the author in actual experiment.

However, now in testing, which only has batch-size=1, it is not running.

The issue is in I-frame model testing phase, the other two models have successfully produced results for testing.

Following is my testing command:

time python test.py --arch resnet152 --data-name ucf101 --representation iframe --data-root data/ucf101/mpeg4_videos --test-list data/datalists/ucf101_split1_test.txt --weights ucf101_iframe_model_iframe_model_best.pth.tar --save-scores iframe_score_file

I have used nvidia-smi to make sure nothing else is running on the GPU.

Following is the actual error message:

RuntimeError: CUDA out of memory. Tried to allocate 384.00 MiB (GPU 0; 10.92 GiB total capacity; 10.12 GiB already allocated; 245.50 MiB free; 21.69 MiB cached)

What could be the issue and how it can be fixed?

EDIT: By removing the following two lines from test.py, it starts running without an memeory issue, but it is taking ages to process:

net = torch.nn.DataParallel(net.cuda(devices[0]), device_ids=devices)
net.eval()

Yes, the above lines are for GPU based parallel processing.

Still, is there a solution to my problem?


title: Pytorch Grayscale input to Vgg

body:
I am new to pytorch and I want to use Vgg for transfer learning.
I want to delete the fully connected layers and add some new fully connected layers. Also rather than RGB input I want to use grayscale input. For this I will add the weights of the input layer and get a single weight. So the three channel's weights will be added.

I achieved deleting the fully connected layers but I am having trouble with grayscale part. I add the three weights together and form a new weight. Then I try to change the state dict of the vgg model but this gives me error. The networks code is below:

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
    vgg=models.vgg16(pretrained = True).features[:30]

    w1=vgg.state_dict()['0.weight'][:,0,:,:] #first channel of first input layer's weight
    w2=vgg.state_dict()['0.weight'][:,1,:,:]
    w3=vgg.state_dict()['0.weight'][:,2,:,:]
    w4=w1+w2+w3 # add the three weigths of the channels
    w4=w4.unsqueeze(1) # make it 4 dimensional

    a=vgg.state_dict()#create a new statedict
    a['0.weight']=w4 #replace the new state dict's weigt

    vgg.load_state_dict(a) # this line gives the error,load the new state dict

    self.vgg =nn.Sequential(vgg)
    self.fc1 = nn.Linear(14*14*512, 1000)
    self.fc2 = nn.Linear(1000, 2)

def forward(self, x):
    x = self.vgg(x)
    x = x.view(-1, 14 * 14 * 512)
    x = F.relu(self.fc1(x))
    x = self.fc2(x)
    return x

This gives an error of:

  RuntimeError: Error(s) in loading state_dict for Sequential:  size
  mismatch for 0.weight: copying a param with shape torch.Size([64, 1,
  3, 3]) from checkpoint, the shape in current model is torch.Size([64,
  3, 3, 3]).

So it doesn't allow me to replace the weight with a different sized weight. Is there a solution for this problem or is there anything other that I can try. All I want to do is use the vgg's layers up to fully connected layers and change the first layers weights.


title: NOT using multiprocessing but get CUDA error on google colab while using PyTorch DataLoader

body:
I've cloned my GitHub repo into google colab and trying to load data using PyTorch's DataLoader.

global gpu, device
if torch.cuda.is_available():
    gpu = True
    device = 'cuda:0'
    torch.set_default_tensor_type('torch.cuda.FloatTensor')
    print("Using GPU")
else:
    gpu = False
    device = 'cpu'
    print("Using CPU")

data_transforms = transforms.Compose([
    #transforms.Resize(224),
    transforms.ToTensor(),
    transforms.Normalize([0.3112, 0.2636, 0.2047], [0.2419, 0.1972, 0.1554])
    ])
train_path = '/content/convLSTM/code/data/train/'
val_path = '/content/convLSTM/code/data/val/'
test_path = '/content/convLSTM/code/data/test/'

train_data = datasets.ImageFolder(root=train_path, transform=data_transforms)
val_data = datasets.ImageFolder(root=val_path, transform=data_transforms)
test_data = datasets.ImageFolder(root=test_path, transform=data_transforms)

train_loader = torch.utils.data.DataLoader(
    train_data,
    batch_size=18,
    num_workers=4,
    shuffle=False,
    pin_memory=True
    )

val_loader = torch.utils.data.DataLoader(
    val_data,
    batch_size=18,
    shuffle=False,
    num_workers=4,
    pin_memory=True
    )

test_loader = torch.utils.data.DataLoader(
    test_data,
    batch_size=18,
    shuffle=False,
    num_workers=4,
    pin_memory=True
    )
for batch_idx, (data, target) in enumerate(train_loader):
  print(batch_idx)
  if batch_idx==3:
    break

I'm getting the following error when I run the last for loop:

RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method

I tried num_workers = 1 instead of 4 but the error persists. I'm not using any multiprocessing.

I also tried without setting torch.set_default_tensor_type('torch.cuda.FloatTensor') but the error persists.

Python  : 3.6.8 | PyTorch : 1.3.1

What seems to be the problem?


PyTorch Dataloader hangs when num_workers > 0. The code hangs with only about 500 M GPU memory usage.
System info: NVIDIA-SMI 418.56 Driver Version: 418.56  CUDA Version: 10.1.
The same issue appears with pytorch1.5 or pytorch1.6, codes are run in anaconda envs.
Note that this error appears when I run my script in terminal as  python main.py, but when I
debug the same code on Pycharm or VScode, or when I run the same code (in terminal) on
other machines, everything goes fine.
Any idea about the reason for this?
Here is the trace when I ctrl c the code in terminal:
  File "train.py", line 226, in main
    train_domain_adaptation(model, source_loader, target_loader, val_loader,
  File "/home/zhangyu/codes/person_seg/IntraDA/ADVENT/advent/domain_adaptation/train_UDA.py", line 326, in train_domain_adaptation
    train_advent(model, trainloader, targetloader, val_loader, cfg, group=group, fk_loader=fk_loader)
  File "/home/zhangyu/codes/person_seg/IntraDA/ADVENT/advent/domain_adaptation/train_UDA.py", line 114, in train_advent
    _, (images_source, labels, src_names, voc_ids, _) = trainloader_iter.__next__()
  File "/home/zhangyu/anaconda3/envs/pt16/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 363, in __next__
    data = self._next_data()
  File "/home/zhangyu/anaconda3/envs/pt16/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 974, in _next_data
    idx, data = self._get_data()
  File "/home/zhangyu/anaconda3/envs/pt16/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 931, in _get_data
    success, data = self._try_get_data()
  File "/home/zhangyu/anaconda3/envs/pt16/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 779, in _try_get_data
    data = self._data_queue.get(timeout=timeout)
  File "/home/zhangyu/anaconda3/envs/pt16/lib/python3.8/queue.py", line 179, in get
    self.not_empty.wait(remaining)
  File "/home/zhangyu/anaconda3/envs/pt16/lib/python3.8/threading.py", line 306, in wait
    gotit = waiter.acquire(True, timeout)
KeyboardInterrupt
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/home/zhangyu/anaconda3/envs/pt16/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/home/zhangyu/anaconda3/envs/pt16/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/home/zhangyu/anaconda3/envs/pt16/lib/python3.8/site-packages/torch/utils/data/_utils/pin_memory.py", line 25, in _pin_memory_loop
    r = in_queue.get(timeout=MP_STATUS_CHECK_INTERVAL)
  File "/home/zhangyu/anaconda3/envs/pt16/lib/python3.8/multiprocessing/queues.py", line 116, in get
    return _ForkingPickler.loads(res)
  File "/home/zhangyu/anaconda3/envs/pt16/lib/python3.8/site-packages/torch/multiprocessing/reductions.py", line 282, in rebuild_storage_fd
    fd = df.detach()
  File "/home/zhangyu/anaconda3/envs/pt16/lib/python3.8/multiprocessing/resource_sharer.py", line 57, in detach
    with _resource_sharer.get_connection(self._id) as conn:
  File "/home/zhangyu/anaconda3/envs/pt16/lib/python3.8/multiprocessing/resource_sharer.py", line 87, in get_connection
    c = Client(address, authkey=process.current_process().authkey)
  File "/home/zhangyu/anaconda3/envs/pt16/lib/python3.8/multiprocessing/connection.py", line 508, in Client
    answer_challenge(c, authkey)
  File "/home/zhangyu/anaconda3/envs/pt16/lib/python3.8/multiprocessing/connection.py", line 752, in answer_challenge
    message = connection.recv_bytes(256)         # reject large message
  File "/home/zhangyu/anaconda3/envs/pt16/lib/python3.8/multiprocessing/connection.py", line 216, in recv_bytes
    buf = self._recv_bytes(maxlength)
  File "/home/zhangyu/anaconda3/envs/pt16/lib/python3.8/multiprocessing/connection.py", line 414, in _recv_bytes
    buf = self._recv(4)
  File "/home/zhangyu/anaconda3/envs/pt16/lib/python3.8/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
ConnectionResetError: [Errno 104] Connection reset by peer


title: How to freeze selected layers of a model in Pytorch?

body:
I am using the mobileNetV2  and I only want to freeze part of the model. I know I can use the following code to freeze the entire model
MobileNet = models.mobilenet_v2(pretrained = True)

    for param in MobileNet.parameters():
        param.requires_grad = False

but I want everything from (15) onward to remain unfrozen. How can I selectively freeze everything before the desired layer is frozen?
    (15): InvertedResidual(
      (conv): Sequential(
        (0): ConvBNReLU(
          (0): Conv2d(160, 960, kernel_size=(1, 1), stride=(1, 1), bias=False)
          (1): BatchNorm2d(960, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (2): ReLU6(inplace=True)
        )
        (1): ConvBNReLU(
          (0): Conv2d(960, 960, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=960, bias=False)
          (1): BatchNorm2d(960, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (2): ReLU6(inplace=True)
        )
        (2): Conv2d(960, 160, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (3): BatchNorm2d(160, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    (16): InvertedResidual(
      (conv): Sequential(
        (0): ConvBNReLU(
          (0): Conv2d(160, 960, kernel_size=(1, 1), stride=(1, 1), bias=False)
          (1): BatchNorm2d(960, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (2): ReLU6(inplace=True)
        )
        (1): ConvBNReLU(
          (0): Conv2d(960, 960, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=960, bias=False)
          (1): BatchNorm2d(960, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (2): ReLU6(inplace=True)
        )
        (2): Conv2d(960, 160, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (3): BatchNorm2d(160, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    (17): InvertedResidual(
      (conv): Sequential(
        (0): ConvBNReLU(
          (0): Conv2d(160, 960, kernel_size=(1, 1), stride=(1, 1), bias=False)
          (1): BatchNorm2d(960, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (2): ReLU6(inplace=True)
        )
        (1): ConvBNReLU(
          (0): Conv2d(960, 960, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=960, bias=False)
          (1): BatchNorm2d(960, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (2): ReLU6(inplace=True)
        )
        (2): Conv2d(960, 320, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (3): BatchNorm2d(320, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    (18): ConvBNReLU(
      (0): Conv2d(320, 1280, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (1): BatchNorm2d(1280, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (2): ReLU6(inplace=True)
    )   )   (classifier): Sequential(
    (0): Dropout(p=0.2, inplace=False)
    (1): Linear(in_features=1280, out_features=1000, bias=True)   ) )


title: How to normalize set of images between (-1,1)

body:
i have a dataset on images and i would like to normalize them betwwen (-1,1) before feeding them to NN how can i do that ?

x=sample
#Normalized Data
normalized = (x-min(x))/(max(x)-min(x))

# Histogram of example data and normalized data
par(mfrow=c(1,2))
hist(x,          breaks=10, xlab="Data",            col="lightblue", main="")
hist(normalized, breaks=10, xlab="Normalized Data", col="lightblue", main="")

i found this code online but it did not solve my problem since  i have image dataset