title: How to convert multiclass data into binary class in python for image classification?

body:
I am trying to do binary class image classification using PyTorch.I have a csv file in where my dataset classes (V00XRKL) are stored. The dataset has five classes but I would like to convert it into binary class using the condition like if V00XRKL<2 then the Class =1 ,otherwise Class= 2.
Here is the example of my dataset
.

For this purpose I wrote my code in this way

# read the csv file

classfile = pd.read_csv(csvroot)
print('df length orig:', len(classfile))

# remove the NA value from the file
df_NA = classfile.dropna(how='any', axis=0)
print('df length no NA:', len(df_NA))

# Dropout the duplicates of number from ID and Side
df_drop = df_NA.drop_duplicates(subset=['ID', 'SIDE'])
print('df length no dup:', len(df_drop))

# reset the index
df_in = df_drop.reset_index(drop=True)
df_in.loc[df_in['V00XRKL'] < 2, 'V00XRKL'] = 1
df_in.loc[df_in['V00XRKL'] >= 2, 'V00XRKL'] = 2
#df_in.to_clipboard(sep=',')

After running this code I seen that my data has converted in to binary class,here is the screenshot of converted data .

But the problem is that whenever I try to train my cnn model I am getting this error messages.

Traceback (most recent call last):
  File "/home/Downloads/test to con.py", line 379, in <module>
    model, train_loss, valid_loss,val_acc = train_model(net,  patience, n_epochs,costFunction)
  File "/home/Downloads/test to con.py", line 165, in train_model
    loss = criterion(output, labels)
  File "/home/miniconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/miniconda3/lib/python3.6/site-packages/torch/nn/modules/loss.py", line 942, in forward
    ignore_index=self.ignore_index, reduction=self.reduction)
  File "/home/miniconda3/lib/python3.6/site-packages/torch/nn/functional.py", line 2056, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
  File "/home/miniconda3/lib/python3.6/site-packages/torch/nn/functional.py", line 1871, in nll_loss
    ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
RuntimeError: Assertion `cur_target >= 0 && cur_target < n_classes' failed.  at /opt/conda/conda-bld/pytorch_1556653099582/work/aten/src/THNN/generic/ClassNLLCriterion.c:92

Here is my code that I used to make dataset class

#make a class for data
class MyDataset(Dataset):
    def __init__(self, csv_file,root_dir, transform=None):
       # self.data = pd.read_csv(csv_file, header=None)
       self.df_data= csv_file
       self.root_dir = root_dir

       self.transform = transform

    def __len__(self):
       return len(self.df_data)

    def __getitem__(self,idx):
        img_name = os.path.join(self.root_dir, str(self.df_data['ID'].iloc[idx])+'.npy')

        patches, p_id = np.load(img_name, allow_pickle= True)

        img_class = int(self.df_data.iloc[idx,2])

        side = self.df_data.iloc[idx,1]

        if side ==1:
            image = Image.fromarray(patches['R'].astype('uint8'), 'L')                                              #[image['R':side]|image['L':side]]
        else:
            image = Image.fromarray(patches['L'].astype('uint8'), 'L')

        if self.transform is not None:
             image = self.transform(image)

        sample = {'image': image, 'grade':img_class}

        return sample

And this is the CNN model

class ConvNet(nn.Module):
    def __init__(self):
            super(ConvNet, self).__init__()
          # Convolution 1
            self.layer1 = nn.Sequential(
                nn.Conv2d(1, 32, kernel_size=5, stride=1, padding=2),
                nn.BatchNorm2d(32),
                nn.ReLU(),
                nn.MaxPool2d(kernel_size=2, stride=2)) # maxpolling layer 1
          # Convolution 1
            self.layer2 = nn.Sequential(
                nn.Conv2d(32, 64, kernel_size=5, stride=1, padding=2),
                nn.BatchNorm2d(64),
                nn.ReLU(),
                nn.MaxPool2d(kernel_size=2, stride=2))
            self.drop_out = nn.Dropout(0.2)
          # Fully connected
            self.fc1 = nn.Linear(16 * 16 * 64, 1000)
            self.fc2 = nn.Linear(1000,2)

    def forward(self, x):
           out = self.layer1(x)
           out = self.layer2(out)
           out = out.reshape(out.size(0), -1)
           #print(out.shape)
           out = self.drop_out(out)
           out = self.fc1(out)
           out = self.fc2(out)
           return out

I would appreciate  any kind of help in this regard.


title: Pytorch customize weight

body:
I have a network

class Net(nn.Module)

and two different weights w0 and w1 (concatenate weights of all layers into a vector). Now I want to optimize the network on the line connecting w0 and w1, which means that the weight will have the form theta  * w0 + (1-theta) * w1. So now the parameter I want to optimize is no longer the weight itself, but the theta.

How can I implement this? In Pytorch, how can I define the parameter to be theta, and set the weight to be form I want. To be specific, if I create a new class

NetOnLine(nn.Module)

how should I write the forward(self, X) function?


title: How does the reshape work before the fully connected layer in the following CNN model?

body:
Consider the convolutional neural network (two convolutional layers):

class ConvNet(nn.Module):
    def __init__(self, num_classes=10):
        super(ConvNet, self).__init__()
        self.layer1 = nn.Sequential(
            nn.Conv2d(1, 16, kernel_size=5, stride=1, padding=2),
            nn.BatchNorm2d(16),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2))
        self.layer2 = nn.Sequential(
            nn.Conv2d(16, 32, kernel_size=5, stride=1, padding=2),
            nn.BatchNorm2d(32),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2))
        self.fc = nn.Linear(7*7*32, num_classes)

    def forward(self, x):
        out = self.layer1(x)
        out = self.layer2(out)
        out = out.reshape(out.size(0), -1)
        out = self.fc(out)
        return out

The fully connected layer fc is to have 7*7*32 inputs coming in. The above:

out = out.reshape(out.size(0), -1) leads to a tensor with size of (32, 49).
This doesn't seem right as the dimensions of input for the dense layer is different. What am I missing here?

[Note that in Pytorch the input is in the following format: [N, C, W, H] so no. of channels comes before the width and height of image]

source: https://github.com/yunjey/pytorch-tutorial/blob/master/tutorials/02-intermediate/convolutional_neural_network/main.py#L35-L56


title: Unable to create custom dataset and dataloader using torchtext

body:
I have questions regarding building custom dataset and iterator using torchtext. I used the following code found in this post and modified based on my case:
tokenizer = XLNetTokenizer.from_pretrained("xlnet-base-cased")
text_field = Field(sequential=True, eos_token="[CLS]", tokenize=tokenizer)
label_field = Field(sequential=False, use_vocab=False)
data_fields = [("file", None),
               ("text", text_field),
               ("label", label_field)]
train, val = train_test_split(input_dt, test_size=0.1)
train.to_csv("train_output_path", index=False)
val.to_csv("val_output_path", index=False)
train, val = TabularDataset(path="path", train="train.csv", validation="val.csv",
                            format="csv", skip_header=True, fields=data_fields)

When it comes to text_field.build_vocab(train), I got this error: TypeError: '<' not supported between instances of 'list' and 'int'.
The only difference between my code and the post is the pre-trained word embeddings. In the post, the author used glove, which I use XLNetTokenizer from transformers package. I also searched for other posts who used the similar method, but they all used the pre-trained word embeddings, therefore they did have such an issue.
Does anyone know how to fix this issue? Many thanks!


title: why is my simple feedforward neural network diverging (pytorch)?

body:
I am experimenting with a simple 2 layer neural network with pytorch, feeding in only three inputs of size 10 each, with a single value as output. I have normalised inputs and lowered learning rate. It is my understanding that a two layer fully connected neural network should be able to trivially fit to this data

Features:

0.8138  1.2342  0.4419  0.8273  0.0728  2.4576  0.3800  0.0512  0.6872  0.5201
1.5666  1.3955  1.0436  0.1602  0.1688  0.2074  0.8810  0.9155  0.9641  1.3668
1.7091  0.9091  0.5058  0.6149  0.3669  0.1365  0.3442  0.9482  1.2550  1.6950
[torch.FloatTensor of size 3x10]

Targets
[124, 125, 122]
[torch.FloatTensor of size 3]

The code is adapted from a simple example and I am using MSELoss as the loss function. The loss diverges to infinity after just a few iterations:

features = torch.from_numpy(np.array(features))

x_data = Variable(torch.Tensor(features))
y_data = Variable(torch.Tensor(targets))

class Model(torch.nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.linear = torch.nn.Linear(10,5)
        self.linear2 = torch.nn.Linear(5,1)

    def forward(self, x):
        l_out1 = self.linear(x)
        y_pred = self.linear2(l_out1)
        return y_pred

model = Model()

criterion = torch.nn.MSELoss(size_average = False)
optim = torch.optim.SGD(model.parameters(), lr = 0.001)

def main():
    for iteration in range(1000):
        y_pred = model(x_data)
        loss = criterion(y_pred, y_data)

        print(iteration, loss.data[0])
        optim.zero_grad()

        loss.backward()
        optim.step()

Any help would be appreciated. Thanks

EDIT:

Indeed it seems that this was simply due to the learning rate being too high. Setting to 0.00001 fixes convergence issues, albeit giving very slow convergence.


title: How to convert from a Pytroch Tensor from mathematically a column vector to a column matrix?

body:
I am working with tensors in pytorch.
How can I convert a tensor corresponding to a column vector to a tensor corresponding to its transpose?

import numpy as np
coef = torch.from_numpy(np.arange(1.0, 5.0)).float()
print(coef)
print(coef.size())

Currently the size of coef is [4] but I want it to be [4, 1] with the same content.


title: Using Augmented Data Images in Testing

body:
I am working on a Person Re-Identification problem and am showing the results using a CMC curve.
I used augmented data/Image along with the normal images (Currently training on CUHK01) in the training set. While testing if I don't use the augmented data along with my normal test images for Calculating Rank let's say Rank_1 I get Rank_1 of ~30% on the other hand on using augmented data gives me a Rank_1 of ~65-70% (which is weirdly high regarding the current Rank_1 accuracy in the world).

So my questions are

a) How does augmented data affect the testing set especially in my case.

b) Am I over-fitting or something of that sort.

c) Is it a general rule to avoid usage of augmented images in the test case.


title: Pytorch: List of layers returns 'optimizer got an empty parameter list'

body:
I have a defined model and defined layer, I add n instances of my defined layer to a list in the init function of my model as follow:

    self.layers = []
    for i in range(len(nhid)-1):
        self.layers.append(MyLayer(nhid[i], nhid[i+1]))

but when I create optimizer by

optim.Adam(model.parameters(),lr=args.lr, weight_decay=args.weight_decay)

it says:

  ValueError: optimizer got an empty parameter list

But when I write it for two layers as follow I got no error:

self.layer1 = MyLayer[nhid[0],nhid[1]]
self.layer2 = MyLayer[nhid[1],nhid[2]]


title: Balancing an Unbalanced Dataset with K-Fold Cross Validation

body:
I'm trying to train/validate a CNN using Pytorch on an unbalanced image dataset (class 1:250 images, class 0: 4000ish images), and right now, I've tried augmentation solely on my training set (thanks @jodag). However, my model is still learning to favor the class with significantly more images.
I want to find ways to compensate for my unbalanced data set.
I thought about using oversampling/undersampling using the imbalanced data sampler (https://github.com/ufoym/imbalanced-dataset-sampler), but I already use a sampler to select indices for my 5-fold validation. Is there a way I could implement cross-validation using the code below and also add this sampler? Similarly, is there a way to augment one label more frequently than the other? Along the lines of these questions, are there any alternative easier ways that I could address my unbalanced dataset that I haven't looked into yet?
Here's an example of what I have so far
total_set = datasets.ImageFolder(PATH)
KF_splits = KFold(n_splits= 5, shuffle = True, random_state = 42)

for train_idx, valid_idx in KF_splits.split(total_set):
    #sampler to get indices for cross validation
    train_sampler = SubsetRandomSampler(train_idx)
    valid_sampler = SubsetRandomSampler(valid_idx)

    #Use a wrapper to apply augmentation only to training set
    #These are dataloaders that pull images from the same folder but sort into validation and training sets
    #Though transforms augment only the training set, it doesn't address
    #the underlying issue of a heavily unbalanced dataset

    train_loader = torch.utils.data.DataLoader(
        WrapperDataset(total_set, transform=data_transforms['train']),
        batch_size=32, sampler=ImbalancedDatasetSampler(total_set))
    valid_loader = torch.utils.data.DataLoader(
        WrapperDataset(total_set, transform=data_transforms['val']),
        batch_size=32)

    print("Fold:" + str(i))

    for epoch in range(epochs):
        #Train/validate model below

`
Thank you for your time and help!


title: Perform a batch matrix - multiple weight matrices multiplications in pytorch

body:
I have a batch of matrices A with size torch.Size([batch_size, 9, 5]) and weight matrices B with size torch.Size([3, 5, 6]). In Keras, a simple K.dot(A, B) is able to handle the matrix multiplication to give an output with size (batch_size, 9, 3, 6). Here, each row in A is multiplied to the 3 matrices in B to form a (3x6) matrix.

How do you perform a similar operation in torch. From the documentation, torch.bmm requires that A and B must have the same batch size, so I tried this:

B = B.unsqueeze(0).repeat((batch_size, 1, 1, 1))
B.size() # torch.Size([batch_size, 3, 5, 6])
torch.bmm(A,B) # gives an error

  RuntimeError: invalid argument 2: expected 3D tensor, got 4D

Well, the error is expected but how do I perform such an operation?


title: Problem with updating running_mean and running_var in a custom Batchnorm built in Pytorch?

body:
I have been trying to implement a custom batch normalization function such that it can be extended to the Multi GPU version, in particular, the DataParallel module in Pytorch.The custom batchnorm works alright when using 1 GPU, but, when extended to 2 or more, the running mean and variance work in the forward function, but when it returns back from the network, the mean and variance are reinitialized to 0 and 1.

The torch.nn.DataParallel mentions in the warning section that " In each forward, module is replicated on each device, so any updates to the running module in forward will be lost. For example, if module has a counter attribute that is incremented in each forward, it will always stay at the initial value because the update is done on the replicas which are destroyed after forward." But I am not really sure how to retain the mean and variance from the default device.

I have provided code with the result obtained during multi GPU training. This code utilizes the Batchnorm provided here.

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import torch.backends.cudnn as cudnn
import torchvision
import torchvision.transforms as transforms
from torch.nn.parameter import Parameter

class ptrblck_BatchNorm2d(nn.BatchNorm2d):
    def __init__(self, num_features, eps=1e-5, momentum=0.1,
                 affine=True, track_running_stats=True):
        super(ptrblck_BatchNorm2d, self).__init__(
            num_features, eps, momentum, affine, track_running_stats)

    def forward(self, input):
        self._check_input_dim(input)

        exponential_average_factor = 0.0

        if self.training and self.track_running_stats:
            if self.num_batches_tracked is not None:
                self.num_batches_tracked += 1
                if self.momentum is None:  # use cumulative moving average
                    exponential_average_factor = 1.0 / float(self.num_batches_tracked)
                else:  # use exponential moving average
                    exponential_average_factor = self.momentum

        # calculate running estimates
        if self.training:
            mean = input.mean([0, 2, 3])
            # use biased var in train
            var = input.var([0, 2, 3], unbiased=False)
            n = input.numel() / input.size(1)
            with torch.no_grad():
                self.running_mean = exponential_average_factor * mean\
                    + (1 - exponential_average_factor) * self.running_mean
                # update running_var with unbiased var
                self.running_var = exponential_average_factor * var * n / (n - 1)\
                    + (1 - exponential_average_factor) * self.running_var
        else:
            mean = self.running_mean
            var = self.running_var

        input = (input - mean[None, :, None, None]) / (torch.sqrt(var[None, :, None, None] + self.eps))
        if self.affine:
            input = input * self.weight[None, :, None, None] + self.bias[None, :, None, None]

        return input

class net(nn.Module):
    def __init__(self):
        super(net, self).__init__()
        self.conv1 = nn.Conv2d(3, 64, kernel_size=3, padding=1)
        self.bn1 = ptrblck_BatchNorm2d(64)
        print("==> printing bn1 mean when init")
        print(self.bn1.running_mean)
        print("==> printing bn1 when init")
        print(self.bn1.running_mean)
        self.avgpool = nn.AdaptiveAvgPool2d((1, 1))
        self.pool = nn.MaxPool2d(kernel_size=2, stride=2)
        self.classifier = nn.Linear(64, 10)

    def forward(self, x):
        x = self.conv1(x)
        x = self.bn1(x)
        x = F.relu(x)
        x = self.pool(x)
        x = self.avgpool(x)

        x = x.view(x.size(0), -1)
        x = self.classifier(x)
        print("======================================================")
        print("==> printing bn1 running mean from NET during forward")
        print(net.module.bn1.running_mean)
        print("==> printing bn1 running mean from SELF. during forward")
        print(self.bn1.running_mean)
        print("==> printing bn1 running var from NET during forward")
        print(net.module.bn1.running_var)
        print("==> printing bn1 running mean from SELF. during forward")
        print(self.bn1.running_var)
        return x

# Data
print('==> Preparing data..')
transform_train = transforms.Compose([
    transforms.RandomCrop(32, padding=4),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(),
    transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010))])

transform_test = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010))])

trainset = torchvision.datasets.CIFAR10(root='./data', train=True, download=True, transform=transform_train)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=64, shuffle=True, num_workers=2)
testset = torchvision.datasets.CIFAR10(root='./data', train=False, download=True, transform=transform_test)
testloader = torch.utils.data.DataLoader(testset, batch_size=64, shuffle=False, num_workers=2)
classes = ('plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck')

# Model
print('==> Building model..')
net = net()
net = torch.nn.DataParallel(net).cuda()
print('Number of GPU {}'.format(torch.cuda.device_count()))

criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=0.1, momentum=0.9, weight_decay=5e-4)

# Training
def train(epoch):
    print('\nEpoch: %d' % epoch)
    net.train()
    train_loss = 0
    correct = 0
    total = 0

    for batch_idx, (inputs, targets) in enumerate(trainloader):
        inputs, targets = inputs.cuda(), targets.cuda()
        outputs = net(inputs)
        loss = criterion(outputs, targets)
        print("====================================================")
        print("==> printing bn1 running mean FROM net after forward")
        print(net.module.bn1.running_mean)
        print("==> printing bn1 running var FROM net after forward")
        print(net.module.bn1.running_var)

        break
        # optimizer.zero_grad()
        # loss.backward()
        # optimizer.step()

        # train_loss += loss.item()
        # _, predicted = outputs.max(1)
        # total += targets.size(0)
        # correct += predicted.eq(targets).sum().item()

        # break

for epoch in range(0, 1):
    train(epoch)

Result:

==> Preparing data..
Files already downloaded and verified
Files already downloaded and verified
==> Building model..
==> printing bn1 mean when init
tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])
==> printing bn1 when init
tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])
Number of GPU 2

Epoch: 0
======================================================
==> printing bn1 running mean from NET during forward
tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       device='cuda:0')
==> printing bn1 running mean from SELF. during forward
tensor([ 0.0053,  0.0010, -0.0077, -0.0290,  0.0241,  0.0258, -0.0048,  0.0151,
        -0.0133,  0.0080,  0.0197, -0.0042, -0.0188,  0.0233,  0.0310, -0.0230,
        -0.0133,  0.0222,  0.0119, -0.0042, -0.0220, -0.0169, -0.0342, -0.0025,
         0.0338, -0.0070,  0.0202,  0.0050,  0.0108,  0.0008,  0.0363,  0.0347,
        -0.0106,  0.0082,  0.0128,  0.0074,  0.0111, -0.0030, -0.0089,  0.0070,
        -0.0262, -0.0029,  0.0053, -0.0136, -0.0183,  0.0045, -0.0014, -0.0221,
         0.0132,  0.0064,  0.0388, -0.0220, -0.0008,  0.0400, -0.0187,  0.0397,
        -0.0131, -0.0176,  0.0035,  0.0055, -0.0270,  0.0066, -0.0149,  0.0135],
       device='cuda:0')
==> printing bn1 running var from NET during forward
tensor([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
        1., 1., 1., 1., 1., 1., 1., 1., 1., 1.], device='cuda:0')
==> printing bn1 running mean from SELF. during forward
tensor([0.9665, 0.9073, 0.9220, 1.0947, 1.0687, 0.9624, 0.9252, 0.9131, 0.9066,
        0.9536, 0.9258, 0.9203, 1.0359, 0.9690, 1.1066, 1.0636, 0.9135, 0.9644,
        0.9373, 0.9846, 0.9696, 0.9454, 1.0459, 0.9245, 0.9778, 0.9709, 0.9352,
        0.9995, 0.9657, 0.9510, 1.0943, 1.0171, 0.9298, 1.0747, 0.9341, 0.9635,
        0.9978, 0.9303, 0.9261, 0.9137, 0.9569, 1.0066, 1.0463, 0.9955, 0.9621,
        0.9172, 0.9836, 0.9817, 0.9086, 0.9576, 1.0905, 0.9861, 0.9661, 1.1773,
        0.9345, 1.0904, 0.9133, 1.0660, 0.9164, 0.9058, 0.9446, 0.9225, 1.0914,
        0.9292], device='cuda:0')
======================================================
==> printing bn1 running mean from NET during forward
tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       device='cuda:0')
==> printing bn1 running mean from SELF. during forward
tensor([-0.0020,  0.0002, -0.0103, -0.0426,  0.0386,  0.0311, -0.0059,  0.0151,
        -0.0140,  0.0145,  0.0218, -0.0029, -0.0281,  0.0284,  0.0449, -0.0329,
        -0.0107,  0.0278,  0.0135, -0.0123, -0.0260, -0.0214, -0.0423, -0.0035,
         0.0410, -0.0097,  0.0276,  0.0102,  0.0197, -0.0001,  0.0483,  0.0451,
        -0.0078,  0.0190,  0.0135, -0.0004,  0.0196, -0.0028, -0.0140,  0.0070,
        -0.0332, -0.0110,  0.0151, -0.0210, -0.0226,  0.0074, -0.0088, -0.0314,
         0.0125, -0.0003,  0.0505, -0.0312,  0.0086,  0.0544, -0.0245,  0.0528,
        -0.0086, -0.0290,  0.0063,  0.0042, -0.0339,  0.0061, -0.0277,  0.0092],
       device='cuda:1')
==> printing bn1 running var from NET during forward
tensor([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
        1., 1., 1., 1., 1., 1., 1., 1., 1., 1.], device='cuda:0')
==> printing bn1 running mean from SELF. during forward
tensor([0.9665, 0.9072, 0.9211, 1.0999, 1.0714, 0.9610, 0.9209, 0.9125, 0.9063,
        0.9553, 0.9260, 0.9189, 1.0386, 0.9706, 1.1139, 1.0610, 0.9121, 0.9660,
        0.9366, 0.9886, 0.9683, 0.9454, 1.0511, 0.9227, 0.9792, 0.9704, 0.9330,
        0.9989, 0.9657, 0.9476, 1.1008, 1.0191, 0.9294, 1.0814, 0.9320, 0.9642,
        1.0006, 0.9287, 0.9254, 0.9128, 0.9559, 1.0100, 1.0521, 0.9972, 0.9621,
        0.9168, 0.9849, 0.9803, 0.9083, 0.9556, 1.0946, 0.9865, 0.9651, 1.1880,
        0.9330, 1.0959, 0.9116, 1.0706, 0.9149, 0.9057, 0.9450, 0.9215, 1.0972,
        0.9261], device='cuda:1')
====================================================
==> printing bn1 running mean FROM net after forward
tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       device='cuda:0')
==> printing bn1 running var FROM net after forward
tensor([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
        1., 1., 1., 1., 1., 1., 1., 1., 1., 1.], device='cuda:0')

How can I make sure that the running estimates of the default device be used? Currently, I am not working towards synchronized Batchnorm.


title: PyTorch RoI pooling implementation deviation

body:
I am trying to implement ROI pooling by PyTorch. Here's the minimal demo.

import torch
import torch.nn as nn
import torch.nn.functional as F

def roi_pooling(feature_map, rois, size=(7, 7)):
    """
    :param feature_map: (1, C, H, W)
    :param rois: (1, N, 4) N refers to bbox num, 4 represent (ltx, lty, w, h)
    :param size: output size
    :return: (1, C, size[0], size[1])
    """
    output = []
    rois_num = rois.size(1)

    for i in range(rois_num):
        roi = rois[0][i]
        x, y, w, h = roi
        output.append(F.adaptive_max_pool2d(feature_map[:, :, y:y+h, x:x+w], size))

    return torch.cat(output)

if __name__ == '__main__':
    test_tensor = torch.tensor([
        [0.88, 0.44, 0.14, 0.16, 0.37, 0.77, 0.96, 0.27],
        [0.19, 0.45, 0.57, 0.16, 0.63, 0.29, 0.71, 0.70],
        [0.66, 0.26, 0.82, 0.64, 0.54, 0.73, 0.59, 0.26],
        [0.85, 0.34, 0.76, 0.84, 0.29, 0.75, 0.62, 0.25],
        [0.32, 0.74, 0.21, 0.39, 0.34, 0.03, 0.33, 0.48],
        [0.20, 0.14, 0.16, 0.13, 0.73, 0.65, 0.96, 0.32],
        [0.19, 0.69, 0.09, 0.86, 0.88, 0.07, 0.01, 0.48],
        [0.83, 0.24, 0.97, 0.04, 0.24, 0.35, 0.50, 0.91]
    ])
    test_tensor = test_tensor.view(1, 1, 8, 8)
    rois = torch.tensor([[0, 3, 7, 5]])
    rois = rois.view(1, -1, 4)
    output = roi_pooling(test_tensor, rois, (2, 2))
    print(output)

I implement this by referencing this website: https://deepsense.ai/region-of-interest-pooling-explained/. And the test_tensor and RoI also comes from the website's example. However, as the website display, the output should be

[ [0.85, 0.84], [0.97, 0.96] ]

instead of my demo's output:

[ [0.85, 0.96], [0.97, 0.96] ]

So, what's the exact problem of my code? Is the coordinate split phase wrong?


title: PyTorch weak_script_method decorator

body:
I came across some code in an introduction to Word2Vec and PyTorch that I'm not quite familiar with. I haven't seen this type of code structure before.

>>> import torch
>>> from torch import nn

>>> # an Embedding module containing 10 tensors of size 3
>>> embedding = nn.Embedding(10, 3)
>>> # a batch of 2 samples of 4 indices each
>>> input = torch.LongTensor([[1,2,4,5],[4,3,2,9]])
>>> embedding(input)

tensor([[[-0.0251, -1.6902,  0.7172],
         [-0.6431,  0.0748,  0.6969],
         [ 1.4970,  1.3448, -0.9685],
         [-0.3677, -2.7265, -0.1685]],

        [[ 1.4970,  1.3448, -0.9685],
         [ 0.4362, -0.4004,  0.9400],
         [-0.6431,  0.0748,  0.6969],
         [ 0.9124, -2.3616,  1.1151]]])

I'm a little confused about the following line of code.

>>> embedding(input)

I may have inadvertently ignored this syntax in the past, but I don't recall seeing a variable being passed to a class instance before? Referring to the PyTorch documentation where Class Embedding() is defined, is this behaviour enabled with decorator @weak_script_method wrapping def forward()? The code below suggests this may be the case?

>>> torch.manual_seed(2)
>>> torch.eq(embedding(input), embedding.forward(input)).all()

tensor(1, dtype=torch.uint8)

Why is the use of decorator @weak_script_method preferable in this case?


title: copy construct from a tensor: USER WARNING

body:
I am creating a random tensor from normal distribution and since this tensor is served as the weight in the NN, to add requires_grad attributes, I use torch.tensor() as below:

import torch

input_dim, hidden_dim = 3, 5

norm = torch.distributions.normal.Normal(loc=0, scale=0.01)
W = norm.sample((input_dim, hidden_dim))
W = torch.tensor(W, requires_grad=True)

I am getting user warning error as below:

    UserWarning: To copy construct from a tensor,
    it is recommended to use sourceTensor.clone().detach() or
sourceTensor.clone().detach().requires_grad_(True),
rather than torch.tensor(sourceTensor).

Is there an alternative way to achieve the above? Thanks


title: How to load large multi file parquet files for tensorflow/pytorch

body:
I am trying to load a few parquet files from a directory into Python for tensorflow/pytorch.

The files are too large to be loaded through the pyarrow.parquet functions

import pyarrow.parquet as pq
dataset = pq.ParquetDataset('dir')
table = dataset.read()

This gives out of memory error.

I have also tried using petastorm, but that doesn't work for make_reader() because it isn't of the petastorm type.

with make_batch_reader('dir') as reader:
  dataset = make_petastorm_dataset(reader)

When I used the make_batch_reader() and then the make_petastorm_dataset(reader), it again gave an zip not iterable error or something along those lines.

I am not sure how to load the file into Python for ML training.
Some quick help would be greatly appreciated.

Thanks
Zash


title: Tensorflow vs PyTorch: convolution doesn't work

body:
I'm trying to test if Tensorflow convolution output matches PyTorch convolution output with the same weights.

Here's my code in which I copy the weights from Tensorflow to Torch, convolve and compare outputs:

import tensorflow as tf
import numpy as np
import math
from math import floor, ceil
import os
import math
import datetime
from scipy import misc
from PIL import Image
import model
import torch
from torch import nn
import common
import torch.nn.functional as F

sess = tf.Session()
np.random.seed(1)
tf.set_random_seed(1)

#parameters
kernel_size = 3
input_feat = 4
output_feat = 4

#inputs
npo = np.random.random((1,5,5, input_feat))
x = tf.convert_to_tensor(npo, tf.float32)
x2 = torch.tensor(np.transpose(npo, [0, 3, 1, 2])).double()

#the same weights
weights = np.random.random((kernel_size,kernel_size,input_feat,output_feat))
weights_torch = np.transpose(weights, [3, 2, 1, 0])

#convolving with tensorflow
w = tf.Variable(weights, name="testconv_W", dtype=tf.float32)
res = tf.nn.conv2d(x, w, strides=[1, 1, 1, 1], padding="VALID")

sess.run(tf.global_variables_initializer())

#convolving with torch
torchres = F.conv2d(x2, torch.tensor(weights_torch), padding=0, bias=torch.zeros((output_feat)).double())

#comparing the results
print(np.mean(np.transpose(sess.run(res), [0, 3, 1, 2])) - torch.mean(torchres).detach().numpy())

It outputs

0.15440369065716908

Why? Why is there such a big difference? Is the Tensorflow conv2d implementation incorrect? Why doesn't it match PyTorch? Am I doing something wrong? On kernel size 1 everything works fine.
Please help.


title: How to do backprop in Pytorch (autograd.backward(loss) vs loss.backward()) and where to set requires_grad=True?

body:
I have been using Pytorch for a while now. One question I had regarding backprop is as follows:

let's say we have a loss function for a neural network. For doing backprop, I have seen two different versions. One like:

optimizer.zero_grad()
autograd.backward(loss)
optimizer.step()

and the other one like:

optimizer.zero_grad()
loss.backward()
optimizer.step()

Which one should I use? Is there any difference between these two versions?

As a last question, do we need to specify the requires_grad=True for the parameters of every layer of our network to make sure their gradients is being computed in the backprop?

For example do I need to specify it for the layer nn.Linear(hidden_size, output_size) inside my network or it is automatically being set to True by default?


title: Loading pytorch model in C++, problem with libtorch.dll

body:
I'm trying to load neural network model trained with pytorch to C++ program.
There is a tutorial how to do it, but can't get it working.

The console appears, and then I get,

  "The Code execution cannot proceed, because the object xxx.dll was not found".

Sometimes it is c10.dll, other time torch.dll, or caffe2.dll.

I checked my C/C++ and Linker properties like 50 times.
I checked it with using debug and release version of libtorch.
I ran it on Debug x86, Debug x64, Release x86, Release x64.
And I added those files manually to Debug folder (where .exe file is)
None of it worked.

This is my code

#include "pch.h"
#include <torch/script.h>
#include <memory>
#include <iostream>

int main()
{
    std::cout << "Hello World!\n";
}

This is how I set directory for header files:

 $(SolutionDir)libtorch\include

...and linker directory for additional libraries:

$(SolutionDir)libtorch\lib

...and all .lib files I added:

torch.lib;onnxifi_loader.lib;onnxifi_dummy.lib;onnx_proto.lib;onnx.lib;libprotoc.lib;libprotobuf-lite.lib;libprotobuf.lib;foxi_loader.lib;foxi_dummy.lib;cpuinfo.lib;clog.lib;caffe2_module_test_dynamic.lib;caffe2_detectron_ops.lib;caffe2.lib;c10.lib;

It's not that some function or classes from this library are not working. I cannot compile a simple "Hello World" program.

I even downloaded some random .dll file to check if it is this particular library problem, and other .dll worked with no problems.

I set up the project manually, and followed the instruction from the link I sent (with creating project with CMAKE), and still I have this error.

I'm working on it for few hours and I'm pretty annoyed about this.I ran out of ideas. I really don't know what else I could miss in this situation.

I'm running VS 2017 Community, version 15.9.14 on Windows 10.


title: Prevent GPU usage in SLURM when --gpus is not set

body:
We're using SLURM to manage a small on-premise cluster.  A key resource we are managing is GPUs.  When a user requests GPUs via --gpus=2 the CUDA_VISIBLE_DEVICES environment variable is set with the GPUs SLURM allocates to the user.

$ srun --gpus=2 bash -c 'echo $CUDA_VISIBLE_DEVICES'
0,1

We have a small team and can trust our users to not abuse the system (they could easily overwrite the environment variable) so this works great.  However, it's a bit too easy to bypass this accidentally because when --gpus isn't specified $CUDA_VISIBLE_DEVICES is left unset so the user can use any GPU (we're typically using PyTorch).

In other words, the following command works fine (so long as it lands on a GPU node) but I would prefer that it fails (because no GPU was requested).

srun sudo docker run -e CUDA_VISIBLE_DEVICES --runtime=nvidia pytorch/pytorch:1.1.0-cuda10.0-cudnn7.5-runtime python -c 'import torch; print(torch.tensor([1., 2.], device=torch.device("cuda:0")))'

It would fail if $CUDA_VISIBLE_DEVICES were set to -1.

$ CUDA_VISIBLE_DEVICES=-1 sudo docker run -e CUDA_VISIBLE_DEVICES --runtime=nvidia pytorch/pytorch:1.1.0-cuda10.0-cudnn7.5-runtime python -c 'import torch; print(torch.tensor([1., 2.], device=torch.device("cuda:0")))'
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1556653099582/work/aten/src/THC/THCGeneral.cpp line=51 error=38 : no CUDA-capable device is detected
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/conda/lib/python3.6/site-packages/torch/cuda/__init__.py", line 163, in _lazy_init
    torch._C._cuda_init()
RuntimeError: cuda runtime error (38) : no CUDA-capable device is detected at /opt/conda/conda-bld/pytorch_1556653099582/work/aten/src/THC/THCGeneral.cpp:51

How can I configure SLURM to set CUDA_VISIBLE_DEVICES to -1 when --gpus is not specified?


title: Training Error in PyTorch - RuntimeError: Expected object of type FloatTensor vs ByteTensor

body:
A minimal working sample will be difficult to post here but basically I am trying to modify this project http://torch.ch/blog/2015/09/21/rmva.html which works smoothly with MNIST. I am trying to run it with my own dataset with a custom dataloader.py as below:

from __future__ import print_function, division #ds
import numpy as np
from utils import plot_images

import os #ds
import pandas as pd #ds
from skimage import io, transform #ds
import torch
from torchvision import datasets
from torch.utils.data import Dataset, DataLoader #ds
from torchvision import transforms
from torchvision import utils #ds
from torch.utils.data.sampler import SubsetRandomSampler

class CDataset(Dataset):

    def __init__(self, csv_file, root_dir, transform=None):
        """
        Args:
            csv_file (string): Path to the csv file with annotations.
            root_dir (string): Directory with all the images.
            transform (callable, optional): Optional transform to be applied
                on a sample.
        """
        self.frame = pd.read_csv(csv_file)
        self.root_dir = root_dir
        self.transform = transform

    def __len__(self):
        return len(self.frame)

    def __getitem__(self, idx):
        img_name = os.path.join(self.root_dir,
                                self.frame.iloc[idx, 0]+'.jpg')
        image = io.imread(img_name)
#       image = image.transpose((2, 0, 1))
        labels = np.array(self.frame.iloc[idx, 1])#.as_matrix() #ds
        #landmarks = landmarks.astype('float').reshape(-1, 2)
        #print(image.shape)
        #print(img_name,labels)
        sample = {'image': image, 'labels': labels}

        if self.transform:
            sample = self.transform(sample)

        return sample

class ToTensor(object):
    """Convert ndarrays in sample to Tensors."""

    def __call__(self, sample):
        image, labels = sample['image'], sample['labels']
        #print(image)
        #print(labels)
        # swap color axis because
        # numpy image: H x W x C
        # torch image: C X H X W
        image = image.transpose((2, 0, 1))
        #print(image.shape)
        #print((torch.from_numpy(image)))
        #print((torch.from_numpy(labels)))
        return {'image': torch.from_numpy(image),
                'labels': torch.from_numpy(labels)}

def get_train_valid_loader(data_dir,
                           batch_size,
                           random_seed,
                           #valid_size=0.1, #ds
                           #shuffle=True,
                           show_sample=False,
                           num_workers=4,
                           pin_memory=False):
    """
    Utility function for loading and returning train and valid
    multi-process iterators over the MNIST dataset. A sample
    9x9 grid of the images can be optionally displayed.

    If using CUDA, num_workers should be set to 1 and pin_memory to True.

    Args
    ----
    - data_dir: path directory to the dataset.
    - batch_size: how many samples per batch to load.
    - random_seed: fix seed for reproducibility.
    - #ds valid_size: percentage split of the training set used for
      the validation set. Should be a float in the range [0, 1].
      In the paper, this number is set to 0.1.
    - shuffle: whether to shuffle the train/validation indices.
    - show_sample: plot 9x9 sample grid of the dataset.
    - num_workers: number of subprocesses to use when loading the dataset.
    - pin_memory: whether to copy tensors into CUDA pinned memory. Set it to
      True if using GPU.

    Returns
    -------
    - train_loader: training set iterator.
    - valid_loader: validation set iterator.
    """
    #ds
    #error_msg = "[!] valid_size should be in the range [0, 1]."
    #assert ((valid_size >= 0) and (valid_size <= 1)), error_msg
    #ds

    # define transforms
    #normalize = transforms.Normalize((0.1307,), (0.3081,))
    trans = transforms.Compose([
        ToTensor(), #normalize,
    ])

    # load train dataset
    #train_dataset = datasets.MNIST(
    #    data_dir, train=True, download=True, transform=trans
    #)

    train_dataset = CDataset(csv_file='/home/Desktop/6June17/util/train.csv',
                                    root_dir='/home/caffe/data/images/',transform=trans)

    # load validation dataset
    #valid_dataset = datasets.MNIST( #ds
    #    data_dir, train=True, download=True, transform=trans #ds
    #)

    valid_dataset = CDataset(csv_file='/home/Desktop/6June17/util/eval.csv',
                                    root_dir='/home/caffe/data/images/',transform=trans)

    num_train = len(train_dataset)
    train_indices = list(range(num_train))
    #ds split = int(np.floor(valid_size * num_train))

    num_valid = len(valid_dataset) #ds
    valid_indices = list(range(num_valid)) #ds

    #if shuffle:
    #    np.random.seed(random_seed)
    #    np.random.shuffle(indices)

    #ds train_idx, valid_idx = indices[split:], indices[:split]
    train_idx = train_indices #ds
    valid_idx = valid_indices #ds

    train_sampler = SubsetRandomSampler(train_idx)
    valid_sampler = SubsetRandomSampler(valid_idx)

    train_loader = torch.utils.data.DataLoader(
        train_dataset, batch_size=batch_size, sampler=train_sampler,
        num_workers=num_workers, pin_memory=pin_memory,
    )

    print(train_loader)

    valid_loader = torch.utils.data.DataLoader(
        valid_dataset, batch_size=batch_size, sampler=valid_sampler,
        num_workers=num_workers, pin_memory=pin_memory,
    )

    # visualize some images
    if show_sample:
        sample_loader = torch.utils.data.DataLoader(
            dataset, batch_size=9, #shuffle=shuffle,
            num_workers=num_workers, pin_memory=pin_memory
        )
        data_iter = iter(sample_loader)
        images, labels = data_iter.next()
        X = images.numpy()
        X = np.transpose(X, [0, 2, 3, 1])
        plot_images(X, labels)

    return (train_loader, valid_loader)

def get_test_loader(data_dir,
                    batch_size,
                    num_workers=4,
                    pin_memory=False):
    """
    Utility function for loading and returning a multi-process
    test iterator over the MNIST dataset.

    If using CUDA, num_workers should be set to 1 and pin_memory to True.

    Args
    ----
    - data_dir: path directory to the dataset.
    - batch_size: how many samples per batch to load.
    - num_workers: number of subprocesses to use when loading the dataset.
    - pin_memory: whether to copy tensors into CUDA pinned memory. Set it to
      True if using GPU.

    Returns
    -------
    - data_loader: test set iterator.
    """
    # define transforms
    #normalize = transforms.Normalize((0.1307,), (0.3081,))
    trans = transforms.Compose([
        ToTensor(), #normalize,
    ])

    # load dataset
    #dataset = datasets.MNIST(
    #    data_dir, train=False, download=True, transform=trans
    #)

    test_dataset = CDataset(csv_file='/home/Desktop/6June17/util/test.csv',
                                    root_dir='/home/caffe/data/images/',transform=trans)

    test_loader = torch.utils.data.DataLoader(
        test_dataset, batch_size=batch_size, shuffle=False,
        num_workers=num_workers, pin_memory=pin_memory,
    )

    return test_loader

#for i_batch, sample_batched in enumerate(dataloader):
#    print(i_batch, sample_batched['image'].size(),
#          sample_batched['landmarks'].size())

#    # observe 4th batch and stop.
#    if i_batch == 3:
#        plt.figure()
#        show_landmarks_batch(sample_batched)
#        plt.axis('off')
#        plt.ioff()
#        plt.show()
#        break

Other main change I have made is closing off the parameter intake for validation size and shuffling (as I am using a pre-existing train, validation and test split and I have already shuffled these splits)

And my last change is in train_one_epoch(self, epoch) function, while iterating in trainer.py. I have changed this part because formerly the x, y were being returned as strings of "image" and "labels" - headers of the python dictionary rather than the values in batches.

for i, batch in enumerate(self.train_loader):
     x, y = batch["image"], batch["labels"]

But now I get errors with the network training that I can not figure out as I am new to pytorch:

[*] Train on 64034 samples, validate on 18951 samples Epoch: 1/200 - LR: 0.000300 <torch.utils.data.dataloader.DataLoader object at 0x7fe065fd4f60>   0%|                                                  | 0/64034 [00:00<?, ?it/s]/home/duygu/recurrent-visual-attention-master/modules.py:106: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number   from_x, to_x = from_x.data[0], to_x.data[0] /home/duygu/recurrent-visual-attention-master/modules.py:107: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number   from_y, to_y = from_y.data[0], to_y.data[0]

Traceback (most recent call last):   File "main.py", line 49, in <module>
    main(config)   File "main.py", line 40, in main
    trainer.train()   File "/home/duygu/recurrent-visual-attention-master/trainer.py", line 168, in train
    train_loss, train_acc = self.train_one_epoch(epoch)   File "/home/duygu/recurrent-visual-attention-master/trainer.py", line 252, in train_one_epoch
    h_t, l_t, b_t, p = self.model(x, l_t, h_t)   File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)   File "/home/duygu/recurrent-visual-attention-master/model.py", line 101, in forward
    g_t = self.sensor(x, l_t_prev)   File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)   File "/home/duygu/recurrent-visual-attention-master/modules.py", line 214, in forward
    phi_out = F.relu(self.fc1(phi))   File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)   File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/linear.py", line 55, in forward
    return F.linear(input, self.weight, self.bias)   File "/usr/local/lib/python3.5/dist-packages/torch/nn/functional.py", line 992, in linear
    return torch.addmm(bias, input, weight.t()) RuntimeError: Expected object of type torch.FloatTensor but found type torch.ByteTensor for argument #4 'mat1'

I am seeking recommendations on how to fix this error and to understand what is causing it.I get this error even when I run it without GPU support on. I wonder if somehow my parameters are passed empty by looking at the initial warning.


title: how do you generate an ONNX representation of a pytorch bert pretrained neural network?

body:
I am trying to generate an ONNX file for the pytorch-pretrained-bert run_classifier.py example.

In this case, I'm running it with the following parameters as per the main README.md:

export GLUE_DIR=/tmp/glue_data

python run_classifier.py \
  --task_name MRPC \
  --do_train \
  --do_eval \
  --do_lower_case \
  --data_dir $GLUE_DIR/MRPC/ \
  --bert_model bert-base-uncased \
  --max_seq_length 128 \
  --train_batch_size 32 \
  --learning_rate 2e-5 \
  --num_train_epochs 3.0 \
  --output_dir /tmp/mrpc_output/

The following code is modified/added at line 552:

    # Save a trained model
    model_to_save = model.module if hasattr(model, 'module') else model  # Only save the model it-self
    output_model_file = os.path.join(args.output_dir, "pytorch_model.bin")
    if args.do_train:
        torch.save(model_to_save.state_dict(), output_model_file)

    # Save ONNX
    msl = args.max_seq_length
    dummy_input = torch.randn(1, msl, msl, msl, num_labels, device="cpu")
    output_onnx_file = os.path.join(args.output_dir, "classifier.onnx")
    torch.onnx.export(model, dummy_input, output_onnx_file)

The dummy_input is supposed to correspond to the bert pretrained model inputs. A sample_batch_size of 1 seems suitable for my needs, I think.

A few suggest that the arguments should match the arguments to the models forward() method. In this case:

class BertForSequenceClassification(PreTrainedBertModel):
    def forward(self, input_ids, token_type_ids=None, attention_mask=None, labels=None):

In this case, I believe the ranks are:

input_ids: 1 x 128 <the max_seq_length specified in the args>
token_type_ids: 1 x max_seq_length
attention_mask: 1 x max_seq_length
labels: 1 x 2 <the number of labels for MRPC>

so the effective call is:

    dummy_input = torch.randn(1, 128, 128, 128, 2, device="cpu")

Unfortunately this produces an error:

Exception has occurred: RuntimeError
The expanded size of the tensor (2) must match the existing size (128) at non-singleton dimension 4.  Target sizes: [1, 128, 128, 128, 2].  Tensor sizes: [1, 128]

It seems likely that it's something fairly straightforward. Suggestions appreciated!


title: How to handle large JSON file in Pytorch?

body: I am working on a time series problem. Different training time series data is stored in a large JSON file with the size of 30GB. In tensorflow I know how to use TF records. Is there a similar way in pytorch?


title: How to compute gradient of the error with respect to the model input?

body:
Given a simple 2 layer neural network, the traditional idea is to compute the gradient w.r.t. the weights/model parameters. For an experiment, I want to compute the gradient of the error w.r.t the input. Are there existing Pytorch methods that can allow me to do this?

More concretely, consider the following neural network:

import torch.nn as nn
import torch.nn.functional as F

class NeuralNet(nn.Module):
    def __init__(self, n_features, n_hidden, n_classes, dropout):
        super(NeuralNet, self).__init__()

        self.fc1 = nn.Linear(n_features, n_hidden)
        self.sigmoid = nn.Sigmoid()
        self.fc2 = nn.Linear(n_hidden, n_classes)
        self.dropout = dropout

    def forward(self, x):
        x = self.sigmoid(self.fc1(x))
        x = F.dropout(x, self.dropout, training=self.training)
        x = self.fc2(x)
        return F.log_softmax(x, dim=1)

I instantiate the model and an optimizer for the weights as follows:

import torch.optim as optim
model = NeuralNet(n_features=args.n_features,
            n_hidden=args.n_hidden,
            n_classes=args.n_classes,
            dropout=args.dropout)
optimizer_w = optim.SGD(model.parameters(), lr=0.001)

While training, I update the weights as usual. Now, given that I have values for the weights, I should be able to use them to compute the gradient w.r.t. the input. I am unable to figure out how.

def train(epoch):
    t = time.time()
    model.train()
    optimizer.zero_grad()
    output = model(features)
    loss_train = F.nll_loss(output[idx_train], labels[idx_train])
    acc_train = accuracy(output[idx_train], labels[idx_train])
    loss_train.backward()
    optimizer_w.step()

    # grad_features = loss_train.backward() w.r.t to features
    # features -= 0.001 * grad_features

for epoch in range(args.epochs):
    train(epoch)


title: What is the benefit of random image crop on Convolutional Network?

body:
I`m studying about transfer learning with the pytorch tutorial. I found pytorch tutorial author uses the different approach to train set and validation set.

data_transforms = {
    'train': transforms.Compose([
        transforms.RandomResizedCrop(224),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
    'val': transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
}

As above, training session uses the Randomly cropped image on training. But I found that after that transformation, some of the images are totally cut off the target object and remains no useful feature for detection. I am curious why the author has chosen that method, even if it is more difficult to define different pipeline to train&validation dataset. Is there any benefit of the random cropping?


title: PyTorch: RuntimeError: element 0 of variables tuple is volatile

body:
I'm training a LSTM based model in PyTorch 0.3.1.

My problem is that after increasing the learning rate I always get a RuntimeError saying: element 0 of variables tuple is volatile.

This does not happen at the beginning, but after some training, like in epoch 3, 4, 5 .. etc.

When looking after this error I found this Question on Stackoverflow suggesting to use zero_grad(). But this was already in use when the error occurred.

So my questions are:

What does it mean that an element of a variable is volatile?
And what are possible causes for a variables element to get "volatile"?
Is there a way to test which variable contains the volatile element, so that I can backtrace the problem?

Thanks a lot in advance for any help!

Here is the code of the training step I'm using:

for epoch in range(num_epochs):
    states = (Var(torch.zeros(num_layers, batch_size, hidden_size)),
              Var(torch.zeros(num_layers, batch_size, hidden_size)))
    new_batch = True
    step = 0
    epoch_loss = []
    print('Epoch: ', epoch+1)
    for i in range(0, token_ids.size(1) - seq_length, seq_length):
        #print(i)
        input_sequence = Var(token_ids[:,i:i+seq_length])
        target_sequence = Var(token_ids[:,(i+1):(i+1)+seq_length])
        entity_target_sequence = Var(entety_targets[:,(i+1):(i+1)+seq_length]).contiguous()

        if int(input_sequence )== 0:
            states = (Var(torch.zeros(num_layers, batch_size, hidden_size)),
                      Var(torch.zeros(num_layers, batch_size, hidden_size)))
            print('New Document')
        model.zero_grad()
        states = detach(states)
        out, states, z = model(input_sequence, states)

        if new_batch:
            loss = loss_func(out, target_sequence.view(-1)) + bce_loss(z, entity_target_sequence)
            new_batch = False
        else:
            loss += loss_func(out, target_sequence.view(-1)) + bce_loss(z, entity_target_sequence)
        if (i+1) % wbatch_size == 0:
            step += 1 // seq_length
            if step % 10 == 0:
                epoch_loss.append((loss.data[0]/wbatch_size))
                print ('Epoch [%d/%d], Step[%d/%d], Loss: %.3f' % (epoch+1, num_epochs, step, num_wbatches, (loss.data[0]/wbatch_size)))
                sys.stdout.flush()

            loss.backward(retain_graph=True)
            torch.nn.utils.clip_grad_norm(model.parameters(), 0.5)
            optimizer.step()
            new_batch = True

(I left out the model itself to avoid a wall of code here and to keep this readable, but if this helps to solve the problem I can of course add the code.)

Traceback:

Traceback (most recent call last):
  File "ent_lm.py", line 223, in <module>
    loss.backward(retain_graph=True)
  File "/usr/local/lib/python3.6/site-packages/torch/autograd/variable.py", line 167, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, retain_variables)
  File "/usr/local/lib/python3.6/site-packages/torch/autograd/__init__.py", line 99, in backward
    variables, grad_variables, retain_graph)


title: Fast way to multiple 3D tensors of shape (1, 1, 256) and (10, 1, 256) in PyTorch and Numpy

body:
I am trying to adapt the seq2seq model for my own task, https://github.com/spro/practical-pytorch/blob/master/seq2seq-translation/seq2seq-translation.ipynb

I have two tensors at the decoder stage

rnn_output: (1, 1, 256)       # time_step x batch_size x hidden_dimension
encoder_inputs: (10, 1, 256)  # seq_len   x batch_size x hidden_dimension

They should be multipied to get attention score (before softmax) of shape

attn_score: (10, 1, 1)

What's the best way to do so? The notebook seems to use a for loop, is there are better via a matrix multiplication kind of operation?


title: conda list returning run-time error Path not Found after installing PyTortch

body:
Recently I tried to install Pytrotch using the command
    conda install pytorch torchvision cuda100 -c pytorch

To verity the package installed correctly I ran conda list in the anaconda prompt and got the following error:

RuntimeError: Path not found: C:\Users\[name]\AppData\Local\Continuum\Anaconda3\Lib\site-packages\Sphinx-1.5.6-py3.6.egg\EGG-INFO

I'm currently running conda version 4.6.1 and python version 3.6.7 on windows 10, I'd appreciate any  help in determining what caused this error and how it can be fixed so I can properly manage my anaconda packages.

Full stack trace:

Traceback (most recent call last):
      File "C:\Users\[name]\AppData\Local\Continuum\Anaconda3\lib\site-packages\conda\exceptions.py", line 1001, in __call__
        return func(*args, **kwargs)
      File "C:\Users\[name]\AppData\Local\Continuum\Anaconda3\lib\site-packages\conda\cli\main.py", line 84, in _main
        exit_code = do_call(args, p)
      File "C:\Users\[name]\AppData\Local\Continuum\Anaconda3\lib\site-packages\conda\cli\conda_argparse.py", line 81, in do_call
        exit_code = getattr(module, func_name)(args, parser)
      File "C:\Users\[name]\AppData\Local\Continuum\Anaconda3\lib\site-packages\conda\cli\main_list.py", line 142, in execute
        show_channel_urls=context.show_channel_urls)
      File "C:\Users\[name]\AppData\Local\Continuum\Anaconda3\lib\site-packages\conda\cli\main_list.py", line 80, in print_packages
        show_channel_urls=show_channel_urls)
      File "C:\Users\[name]\AppData\Local\Continuum\Anaconda3\lib\site-packages\conda\cli\main_list.py", line 45, in list_packages
        installed = sorted(PrefixData(prefix, pip_interop_enabled=True).iter_records(),
      File "C:\Users\[name]\AppData\Local\Continuum\Anaconda3\lib\site-packages\conda\core\prefix_data.py", line 116, in iter_records
        return itervalues(self._prefix_records)
      File "C:\Users\[name]\AppData\Local\Continuum\Anaconda3\lib\site-packages\conda\core\prefix_data.py", line 145, in _prefix_records
        return self.__prefix_records or self.load() or self.__prefix_records
      File "C:\Users\[name]\AppData\Local\Continuum\Anaconda3\lib\site-packages\conda\core\prefix_data.py", line 69, in load
        self._load_site_packages()
      File "C:\Users\[name]\AppData\Local\Continuum\Anaconda3\lib\site-packages\conda\core\prefix_data.py", line 258, in _load_site_packages
        python_record = read_python_record(self.prefix_path, af, python_pkg_record.version)
      File "C:\Users\[name]\AppData\Local\Continuum\Anaconda3\lib\site-packages\conda\gateways\disk\read.py", line 245, in read_python_record
        pydist = PythonDistribution.init(prefix_path, anchor_file, python_version)
      File "C:\Users\[name]\AppData\Local\Continuum\Anaconda3\lib\site-packages\conda\common\pkg_formats\python.py", line 90, in init
        return PythonEggInfoDistribution(anchor_full_path, python_version, sp_reference)
      File "C:\Users\[name]\AppData\Local\Continuum\Anaconda3\lib\site-packages\conda\common\pkg_formats\python.py", line 400, in __init__
        super(PythonEggInfoDistribution, self).__init__(anchor_full_path, python_version)
      File "C:\Users\[name]\AppData\Local\Continuum\Anaconda3\lib\site-packages\conda\common\pkg_formats\python.py", line 104, in __init__
        raise RuntimeError("Path not found: %s" % anchor_full_path)
    RuntimeError: Path not found: C:\Users\[name]\AppData\Local\Continuum\Anaconda3\Lib\site-packages\Sphinx-1.5.6-py3.6.egg\EGG-INFO

any help would be appreciated.


title: Faster pytorch dataset file

body:
I have the following problem, I have many files of 3D volumes that I open to extract a bunch of numpy arrays.
I want to get those arrays randomly, i.e. in the worst case I open as many 3D volumes as numpy arrays I want to get, if all those arrays are in separate files.
The IO here isn't great, I open a big file only to get a small numpy array from it.
Any idea how I can store all these arrays so that the IO is better?
I can't pre-read all the arrays and save them all in one file because then that file would be too big to open for RAM.

I looked up LMDB but it all seems to be about Caffe.
Any idea how I can achieve this?


I have a dataloader as below which load data with a batch size (default to 8):

class DataLoaderStego(DataLoader):
    def __init__(self, cover_dir, stego_dir, embedding_otf=False,
                 shuffle=False, pair_constraint=False, batch_size=1,
                 transform=None, num_workers=0, pin_memory=False):
        self.pair_constraint = pair_constraint
        self.embedding_otf = embedding_otf
        if pair_constraint and batch_size % 2 == 0:
            dataset = DatasetPair(cover_dir, stego_dir, embedding_otf,
                                  transform)
            _batch_size = int(batch_size / 2)
        else:
            dataset = DatasetNoPair(cover_dir, stego_dir, embedding_otf,
                                    transform)
            _batch_size = batch_size
        if pair_constraint:
            if shuffle:
                sampler = RandomSampler(dataset)
            else:
                sampler = SequentialSampler(dataset)
        else:
            sampler = RandomBalancedSampler(dataset)
        super(DataLoaderStego, self). \
            __init__(dataset, _batch_size, None, sampler,
                     None, num_workers, pin_memory=pin_memory, drop_last=True)
        self.shuffle = shuffle

    def __iter__(self):
        return DataLoaderIterWithReshape(self)
        # if self.pair_constraint:
        # return DataLoaderIterWithReshape(self)
        # else:
        #     return DataLoaderIter(self)

This class make the dataset using DatasetPair class:

class DatasetPair(Dataset):
    def __init__(self, cover_dir, stego_dir, embedding_otf=False,
                 transform=None):
        self.cover_dir = cover_dir
        self.stego_dir = stego_dir
        self.cover_list = [x.split('/')[-1]
                           for x in glob(cover_dir + '/*')]
        self.transform = transform
        self.embedding_otf = embedding_otf
        assert len(self.cover_list) != 0, "cover_dir is empty"
        # stego_list = ['.'.join(x.split('/')[-1].split('.')[:-1])
        #               for x in glob(stego_dir + '/*')]

    def __getitem__(self, idx):
        idx = int(idx)
        labels = np.array([0, 1], dtype='int32')
        cover_path = os.path.join(self.cover_dir,
                                  self.cover_list[idx])
        print("cover path", cover_path)
        # cover = Image.open(cover_path)
        cover = Image.open(cover_path)
        # print("@@@", cover)
        images = np.empty((2, cover.size[0], cover.size[1], 1),
                          dtype='uint8')
        print("images", images.shape) # Print image
        images[0, :, :, 0] = np.array(cover)
        if self.embedding_otf:
            images[1, :, :, 0] = np.copy(images[0, :, :, 0])
            beta_path = os.path.join(self.stego_dir,
                                     '.'.join(self.cover_list[idx].
                                              split('.')[:-1]) + '.mat')
            beta_map = io.loadmat(beta_path)['pChange']
            rand_arr = np.random.rand(cover.size[0], cover.size[1])
            inf_map = rand_arr < (beta_map / 2.)
            images[1, np.logical_and(
                images[0, :, :, 0] != 255, inf_map), 0] += 1
            inf_map[:, :] = rand_arr > 1 - (beta_map / 2.)
            images[1, np.logical_and(
                images[0, :, :, 0] != 0, inf_map), 0] -= 1
        else:
            stego_path = os.path.join(self.stego_dir,
                                      self.cover_list[idx])
            print("stego path", stego_path)
            images[1, :, :, 0] = Image.open(stego_path)
        samples = {'images': images, 'labels': labels}
        print(images[0].shape)
        if self.transform:
            samples = self.transform(samples)
        return samples

All of my images are with size of 256*256 when I print cover:

@@@ <PIL.PpmImagePlugin.PpmImageFile image mode=L size=256x256 at 0x7F96631B8748>

When I create my network model with these configurations:

class MyNet(nn.Module):
    def __init__(self, with_bn=False, threshold=3):
        super(YeNet, self).__init__()
        self.with_bn = with_bn
        self.preprocessing = SRM_conv2d(1, 0)
        self.TLU = nn.Hardtanh(-threshold, threshold, True)
        ...

    def forward(self, x):
        x = x.float()
        print("########", x.shape)
        x = self.preprocessing(x)
        ...

And run my network, I get this error:

Traceback (most recent call last):
  File "main.py", line 169, in <module>
    train(epoch)
  File "main.py", line 121, in train
    outputs = net(images)
  File "/home/emad/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/emad/myworks/stego/codes/analysis/YeNet-Pytorch/YeNet.py", line 110, in forward
    x = self.preprocessing(x)
  File "/home/emad/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/emad/myworks/stego/codes/analysis/YeNet-Pytorch/YeNet.py", line 53, in forward
    self.dilation, self.groups
RuntimeError: Expected 4-dimensional input for 4-dimensional weight 30 1 5 5 0, but got 5-dimensional input of size [4, 2, 1, 256, 256] instead

This is my network call:

for batch_idx, data in enumerate(train_loader):
        images, labels = Variable(
            data['images']), Variable(data['labels'])
        if args.cuda:
            images, labels = images.cuda(), labels.cuda()
        optimizer.zero_grad()
        outputs = net(images)

And this is self.preprocessing function:

class SRM_conv2d(nn.Module):
    def __init__(self, stride=1, padding=0):
        super(SRM_conv2d, self).__init__()
        self.in_channels = 1
        self.out_channels = 30
        self.kernel_size = (5, 5)
        if isinstance(stride, int):
            self.stride = (stride, stride)
        else:
            self.stride = stride
        if isinstance(padding, int):
            self.padding = (padding, padding)
        else:
            self.padding = padding
        self.dilation = (1, 1)
        self.transpose = False
        self.output_padding = (0,)
        self.groups = 1
        print(1)
        self.weight = Parameter(torch.Tensor(30, 1, 5, 5),
                                requires_grad=True)
        print(2)
        self.bias = Parameter(torch.Tensor(30),
                              requires_grad=True)
        self.reset_parameters()

    def reset_parameters(self):
        self.weight.data.numpy()[:] = SRM_npy
        self.bias.data.zero_()

    def forward(self, input):
        # print(
        #     "3333", input, self.weight, self.bias,
        #     self.stride, self.padding,
        #     self.dilation, self.groups
        # )
        print("###")
        return F.conv2d(
            input, self.weight, self.bias,
            self.stride, self.padding,
            self.dilation, self.groups
        )
        print("$$$")

I want to enable batch input for my network. But also when I run my code with batch_size=1 get the same error.

Extra info:
These are my three transforms which I use:

class ToTensor(object):
    def __call__(self, samples):
        images, labels = samples['images'], samples['labels']
        print("EEE", images.shape)
        images = images.transpose((0, 3, 1, 2))
        # images = (images.transpose((0,3,1,2)).astype('float32') / 127.5) - 1.
        return {'images': torch.from_numpy(images),
                'labels': torch.from_numpy(labels).long()}

class RandomRot(object):
    def __call__(self, samples):
        images = samples['images']
        rot = random.randint(0, 3)
        return {'images': np.rot90(images, rot, axes=[1, 2]).copy(),
                'labels': samples['labels']}

class RandomFlip(object):
    def __call__(self, samples):
        if random.random() < 0.5:
            images = samples['images']
            return {'images': np.flip(images, axis=2).copy(),
                    'labels': samples['labels']}
        else:
            return samples


title: Why should the training label for Generator in GAN be always True?

body:
I am currently learning deep learning especially GAN.
I found a simple code of GAN from a web site below.
https://medium.com/@devnag/generative-adversarial-networks-gans-in-50-lines-of-code-pytorch-e81b79659e3f

However, in the code, I don't understand why we always need to give true label to Generator as below.

        for g_index in range(g_steps):
            # 2. Train G on D's response (but DO NOT train D on these labels)
            G.zero_grad()

            gen_input = Variable(gi_sampler(minibatch_size, g_input_size))
            g_fake_data = G(gen_input)
            dg_fake_decision = D(preprocess(g_fake_data.t()))
            g_error = criterion(dg_fake_decision, Variable(torch.ones(1)))  # we want to fool, so pretend it's all genuine

            g_error.backward()
            g_optimizer.step()  # Only optimizes G's parameters

Specifically, on this line.

            g_error = criterion(dg_fake_decision, Variable(torch.ones(1)))  # we want to fool, so pretend it's all genuine

Input data for Generator is fake data(includes noise), so if we assign True labels on those input data, I think Generator ends up creating data which is similar to fake data(doesn't look like genuine). Is my understanding wrong? Sorry for the silly question, but if you have knowledge, plz help me out.
I'll put a whole code below.

    #!/usr/bin/env python

    # Generative Adversarial Networks (GAN) example in PyTorch.
    # See related blog post at https://medium.com/@devnag/generative-adversarial-networks-gans-in-50-lines-of-code-pytorch-e81b79659e3f#.sch4xgsa9
    import numpy as np
    import torch
    import torch.nn as nn
    import torch.nn.functional as F
    import torch.optim as optim
    from torch.autograd import Variable

    # Data params
    data_mean = 4
    data_stddev = 1.25

    # Model params
    g_input_size = 1     # Random noise dimension coming into generator, per output vector
    g_hidden_size = 50   # Generator complexity
    g_output_size = 1    # size of generated output vector
    d_input_size = 100   # Minibatch size - cardinality of distributions
    d_hidden_size = 50   # Discriminator complexity
    d_output_size = 1    # Single dimension for 'real' vs. 'fake'
    minibatch_size = d_input_size

    d_learning_rate = 2e-4  # 2e-4
    g_learning_rate = 2e-4
    optim_betas = (0.9, 0.999)
    num_epochs = 30000
    print_interval = 200
    d_steps = 1  # 'k' steps in the original GAN paper. Can put the discriminator on higher training freq than generator
    g_steps = 1

    # ### Uncomment only one of these
    #(name, preprocess, d_input_func) = ("Raw data", lambda data: data, lambda x: x)
    (name, preprocess, d_input_func) = ("Data and variances", lambda data: decorate_with_diffs(data, 2.0), lambda x: x * 2)

    print("Using data [%s]" % (name))

    # ##### DATA: Target data and generator input data

    def get_distribution_sampler(mu, sigma):
        return lambda n: torch.Tensor(np.random.normal(mu, sigma, (1, n)))  # Gaussian

    def get_generator_input_sampler():
        return lambda m, n: torch.rand(m, n)  # Uniform-dist data into generator, _NOT_ Gaussian

    # ##### MODELS: Generator model and discriminator model

    class Generator(nn.Module):
        def __init__(self, input_size, hidden_size, output_size):
            super(Generator, self).__init__()
            self.map1 = nn.Linear(input_size, hidden_size)
            self.map2 = nn.Linear(hidden_size, hidden_size)
            self.map3 = nn.Linear(hidden_size, output_size)

        def forward(self, x):
            x = F.elu(self.map1(x))
            x = F.sigmoid(self.map2(x))
            return self.map3(x)

    class Discriminator(nn.Module):
        def __init__(self, input_size, hidden_size, output_size):
            super(Discriminator, self).__init__()
            self.map1 = nn.Linear(input_size, hidden_size)
            self.map2 = nn.Linear(hidden_size, hidden_size)
            self.map3 = nn.Linear(hidden_size, output_size)

        def forward(self, x):
            x = F.elu(self.map1(x))
            x = F.elu(self.map2(x))
            return F.sigmoid(self.map3(x))

    def extract(v):
        return v.data.storage().tolist()

    def stats(d):
        return [np.mean(d), np.std(d)]

    def decorate_with_diffs(data, exponent):
        mean = torch.mean(data.data, 1, keepdim=True)
        mean_broadcast = torch.mul(torch.ones(data.size()), mean.tolist()[0][0])
        diffs = torch.pow(data - Variable(mean_broadcast), exponent)
        return torch.cat([data, diffs], 1)

    d_sampler = get_distribution_sampler(data_mean, data_stddev)
    gi_sampler = get_generator_input_sampler()
    G = Generator(input_size=g_input_size, hidden_size=g_hidden_size, output_size=g_output_size)
    D = Discriminator(input_size=d_input_func(d_input_size), hidden_size=d_hidden_size, output_size=d_output_size)
    criterion = nn.BCELoss()  # Binary cross entropy: http://pytorch.org/docs/nn.html#bceloss
    d_optimizer = optim.Adam(D.parameters(), lr=d_learning_rate, betas=optim_betas)
    g_optimizer = optim.Adam(G.parameters(), lr=g_learning_rate, betas=optim_betas)

    for epoch in range(num_epochs):
        for d_index in range(d_steps):
            # 1. Train D on real+fake
            D.zero_grad()

            #  1A: Train D on real
            d_real_data = Variable(d_sampler(d_input_size))
            d_real_decision = D(preprocess(d_real_data))
            d_real_error = criterion(d_real_decision, Variable(torch.ones(1)))  # ones = true
            d_real_error.backward() # compute/store gradients, but don't change params

            #  1B: Train D on fake
            d_gen_input = Variable(gi_sampler(minibatch_size, g_input_size))
            d_fake_data = G(d_gen_input).detach()  # detach to avoid training G on these labels
            d_fake_decision = D(preprocess(d_fake_data.t()))
            d_fake_error = criterion(d_fake_decision, Variable(torch.zeros(1)))  # zeros = fake
            d_fake_error.backward()
            d_optimizer.step()     # Only optimizes D's parameters; changes based on stored gradients from backward()

        for g_index in range(g_steps):
            # 2. Train G on D's response (but DO NOT train D on these labels)
            G.zero_grad()

            gen_input = Variable(gi_sampler(minibatch_size, g_input_size))
            g_fake_data = G(gen_input)
            dg_fake_decision = D(preprocess(g_fake_data.t()))
            g_error = criterion(dg_fake_decision, Variable(torch.ones(1)))  # we want to fool, so pretend it's all genuine

            g_error.backward()
            g_optimizer.step()  # Only optimizes G's parameters

        if epoch % print_interval == 0:
            print("%s: D: %s/%s G: %s (Real: %s, Fake: %s) " % (epoch,
                                                                extract(d_real_error)[0],
                                                                extract(d_fake_error)[0],
                                                                extract(g_error)[0],
                                                                stats(extract(d_real_data)),
                                                                stats(extract(d_fake_data))))


title: Make personnal Dataloader with PYTORCH

body:
I'm searching to create a personnal dataloader with a specific format to use Pytorch library, someone have an idea how can I do it ? I have follow Pytorch Tutorial but I don't find my answer!

I need a DataLoader that yields the tuples of the following format:
        (Bx3xHxW FloatTensor x, BxHxW LongTensor y, BxN LongTensor y_cls) where
        x - batch of input images,
        y - batch of groung truth seg maps,
        y_cls - batch of 1D tensors of dimensionality N: N total number of classes,
        y_cls[i, T] = 1 if class T is present in image i, 0 otherwise

I hope that someone can unlock the problem .. :) Thanks !


title: PyTorch transfer learning with pre-trained ImageNet model

body:
I want to create an image classifier using transfer learning on a model already trained on ImageNet.

How do I replace the final layer of a torchvision.models ImageNet classifier with my own custom classifier?


title: Standard deviation in batches

body: I am using this function to compute the standard deviation of my output. I am evaluating my model on 2000 examples, in batches of 4. Can I just take the standard deviation per batch and then divide by 2000 ?


title: GPU driver and CUDA is not enabled and accessible by PyTorch on MacOS

body:
GPU driver and CUDA is not enabled and accessible by PyTorch.
torch.cuda.is_available() returns false

I am using macOS Mojave 10.14.6
I have installed Cuda 10.0 version of pytorch.
I tried verfication on https://pytorch.org/get-started/locally/ and constructing a randomly initialized tensor works just fine.

But when I tried

import torch
torch.cuda.is_availalbe()

it returns false.

Therefore, I followed instructions on Pytorch and installed Anaconda and Cuda.

Then tried this:

git clone --recursive https://github.com/pytorch/pytorch
cd pytorch
MACOSX_DEPLOYMENT_TARGET=10.9 CC=clang CXX=clang++ python setup.py install

In terminal, I got

fatal error: 'string.h' file not found
#include_next <string.h>

I searched on stackoverflow and came up with this. Build Pytorch from source . So I tried:

$ find /Library/Developer/CommandLineTools/usr -type f -name string.h

which returned /Library/Developer/CommandLineTools/usr/include/c++/v1/string.h

Doesn't this mean I already have string.h?

How can I solve this problem?


title: Why do we need pack_padded_sequence() when we have pack_sequence()?

body: After reading the answers to this question I'm still a bit confused about the whole PackedSequence object thing. As I understand it, this is an object optimized for parallel processing of variable sized sequences in recurrent models, a problem to which zero padding is one [imperfect] solution. It seems that given a PackedSequence object, a Pytorch RNN will process each sequence in the batch to its end, and not continue to process the padding. So why is padding needed here? Why are there both a pack_padded_sequence() and pack_sequence() methods?


title: Copy GpuMat to CUDA Tensor

body:
I am trying to run model inference in C++.
I succesfully traced model in Python with torch.jit.trace.
I am able to load model in C++ using torch::jit::load().
I was able to perform inference both on cpu and gpu, however the starting point was always torch::from_blob method which seems to be creating cpu-side tensor.
For efficiency, I would like to cast/copy cv::cuda::GpuMat directly to CUDA Tensor. I have been digging through pytorch tests and docs in search of convinient example, but was unable to find one.

Question:
How to create CUDA Tensor from cv::cuda::GpuMat?


title: How can I combine and overlap two tensors?

body:
I have two tensors that should together overlap each other to form a larger tensor. To illustrate:

a = torch.Tensor([[1, 2, 3], [1, 2, 3]])
b = torch.Tensor([[5, 6, 7], [5, 6, 7]])

a = [[1 2 3]    b = [[5 6 7]
     [1 2 3]]        [5 6 7]]

I want to combine the two tensors and have them partially overlap by a single column, with the average being taken for those elements that overlap.

e.g.

result = [[1 2 4 6 7]
          [1 2 4 6 7]]

The first two columns are the first two columns of  'a'. The last two columns are the last two columns of 'b'. The middle column is the average of 'a's last column and 'b's first column.

I know how to merge two tensors side by side or in a new dimension. But doing this eludes me.

Can anyone help?


title: torchvision MNIST HTTPError: HTTP Error 403: Forbidden

body:
I am trying to replicate this experiment presented in this webpage https://adversarial-ml-tutorial.org/adversarial_examples/
I got the jupyter notebook and loaded in my localhost and open it using Jupiter notebook. When I run the following code to get the dataset using the following code:
from torchvision import datasets, transforms
from torch.utils.data import DataLoader

mnist_train = datasets.MNIST("../data", train=True, download=True, transform=transforms.ToTensor())
mnist_test = datasets.MNIST("../data", train=False, download=True, transform=transforms.ToTensor())
train_loader = DataLoader(mnist_train, batch_size = 100, shuffle=True)
test_loader = DataLoader(mnist_test, batch_size = 100, shuffle=False)

and I get the following error:
Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz to ../data\MNIST\raw\train-images-idx3-ubyte.gz
0/? [00:00<?, ?it/s]
---------------------------------------------------------------------------
HTTPError                                 Traceback (most recent call last)
<ipython-input-15-e6f62798f426> in <module>
      2 from torch.utils.data import DataLoader
      3
----> 4 mnist_train = datasets.MNIST("../data", train=True, download=True, transform=transforms.ToTensor())
      5 mnist_test = datasets.MNIST("../data", train=False, download=True, transform=transforms.ToTensor())
      6 train_loader = DataLoader(mnist_train, batch_size = 100, shuffle=True)

~\Anaconda3\lib\site-packages\torchvision\datasets\mnist.py in __init__(self, root, train, transform, target_transform, download)
     77
     78         if download:
---> 79             self.download()
     80
     81         if not self._check_exists():

~\Anaconda3\lib\site-packages\torchvision\datasets\mnist.py in download(self)
    144         for url, md5 in self.resources:
    145             filename = url.rpartition('/')[2]
--> 146             download_and_extract_archive(url, download_root=self.raw_folder, filename=filename, md5=md5)
    147
    148         # process and save as torch files

~\Anaconda3\lib\site-packages\torchvision\datasets\utils.py in download_and_extract_archive(url, download_root, extract_root, filename, md5, remove_finished)
    254         filename = os.path.basename(url)
    255
--> 256     download_url(url, download_root, filename, md5)
    257
    258     archive = os.path.join(download_root, filename)

~\Anaconda3\lib\site-packages\torchvision\datasets\utils.py in download_url(url, root, filename, md5)
     82                 )
     83             else:
---> 84                 raise e
     85         # check integrity of downloaded file
     86         if not check_integrity(fpath, md5):

~\Anaconda3\lib\site-packages\torchvision\datasets\utils.py in download_url(url, root, filename, md5)
     70             urllib.request.urlretrieve(
     71                 url, fpath,
---> 72                 reporthook=gen_bar_updater()
     73             )
     74         except (urllib.error.URLError, IOError) as e:  # type: ignore[attr-defined]

~\Anaconda3\lib\urllib\request.py in urlretrieve(url, filename, reporthook, data)
    245     url_type, path = splittype(url)
    246
--> 247     with contextlib.closing(urlopen(url, data)) as fp:
    248         headers = fp.info()
    249

~\Anaconda3\lib\urllib\request.py in urlopen(url, data, timeout, cafile, capath, cadefault, context)
    220     else:
    221         opener = _opener
--> 222     return opener.open(url, data, timeout)
    223
    224 def install_opener(opener):

~\Anaconda3\lib\urllib\request.py in open(self, fullurl, data, timeout)
    529         for processor in self.process_response.get(protocol, []):
    530             meth = getattr(processor, meth_name)
--> 531             response = meth(req, response)
    532
    533         return response

~\Anaconda3\lib\urllib\request.py in http_response(self, request, response)
    639         if not (200 <= code < 300):
    640             response = self.parent.error(
--> 641                 'http', request, response, code, msg, hdrs)
    642
    643         return response

~\Anaconda3\lib\urllib\request.py in error(self, proto, *args)
    567         if http_err:
    568             args = (dict, 'default', 'http_error_default') + orig_args
--> 569             return self._call_chain(*args)
    570
    571 # XXX probably also want an abstract factory that knows when it makes

~\Anaconda3\lib\urllib\request.py in _call_chain(self, chain, kind, meth_name, *args)
    501         for handler in handlers:
    502             func = getattr(handler, meth_name)
--> 503             result = func(*args)
    504             if result is not None:
    505                 return result

~\Anaconda3\lib\urllib\request.py in http_error_default(self, req, fp, code, msg, hdrs)
    647 class HTTPDefaultErrorHandler(BaseHandler):
    648     def http_error_default(self, req, fp, code, msg, hdrs):
--> 649         raise HTTPError(req.full_url, code, msg, hdrs, fp)
    650
    651 class HTTPRedirectHandler(BaseHandler):

HTTPError: HTTP Error 403: Forbidden

Any help solving this issue is much appreciated.
I also can download the dataset directly from the link but then I don't know how to use that!


title: Using PyTorch with Celery

body:
I'm trying to run a PyTorch model in a Django app. As it is not recommended to execute the models (or any long-running task) in the views, I decided to run it in a Celery task. My model is quite big and it takes about 12 seconds to load and about 3 seconds to infer. That's why I decided that I couldn't afford to load it at every request. So I tried to load it at settings and save it there for the app to use it. So my final scheme is:

When the Django app starts, in the settings the PyTorch model is loaded and it's accessible from the app.
When views.py receives a request, it delays a celery task
The celery task uses the settings.model to infer the result

The problem here is that the celery task throws the following error when trying to use the model
[2020-08-29 09:03:04,015: ERROR/ForkPoolWorker-1] Task app.tasks.task[458934d4-ea03-4bc9-8dcd-77e4c3a9caec] raised unexpected: RuntimeError("Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method")
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/tensor/lib/python3.7/site-packages/celery/app/trace.py", line 412, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/tensor/lib/python3.7/site-packages/celery/app/trace.py", line 704, in __protected_call__
    return self.run(*args, **kwargs)
  /*...*/
  File "/home/ubuntu/anaconda3/envs/tensor/lib/python3.7/site-packages/torch/cuda/__init__.py", line 191, in _lazy_init
    "Cannot re-initialize CUDA in forked subprocess. " + msg)
RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method

Here's the code in my settings.py loading the model:
if sys.argv and sys.argv[0].endswith('celery') and 'worker' in sys.argv: #In order to load only for the celery worker
    import torch
    torch.cuda.init()
    torch.backends.cudnn.benchmark = True
    load_model_file()

And the task code
@task
def getResult(name):
    print("Executing on GPU:", torch.cuda.is_available())
    if os.path.isfile(name):
        try:
            outpath = model_inference(name)
            os.remove(name)
            return outpath
        except OSError as e:
            print("Error", name, "doesn't exist")
    return ""

The print in the task shows "Executing on GPU: true"
I've tried setting torch.multiprocessing.set_start_method('spawn') in the settings.py before and after the torch.cuda.init() but it gives the same error.


title: Implement SeparableConv2D in Pytorch

body:
Main objective
PyTorch equivalent for SeparableConv2D with padding = 'same':
from tensorflow.keras.layers import SeparableConv2D
x = SeparableConv2D(64, (1, 16), use_bias = False, padding = 'same')(x)

What is the PyTorch equivalent for SeparableConv2D?
This source says:

If groups = nInputPlane, kernel=(K, 1), (and before is a Conv2d layer with groups=1 and kernel=(1, K)), then it is separable.

While this source says:

Its core idea is to break down a complete convolutional acid into a two-step calculation, Depthwise Convolution and Pointwise.

This is my attempt:
class SeparableConv2d(nn.Module):
    def __init__(self, in_channels, out_channels, depth, kernel_size, bias=False):
        super(SeparableConv2d, self).__init__()
        self.depthwise = nn.Conv2d(in_channels, out_channels*depth, kernel_size=kernel_size, groups=in_channels, bias=bias)
        self.pointwise = nn.Conv2d(out_channels*depth, out_channels, kernel_size=1, bias=bias)

    def forward(self, x):
        out = self.depthwise(x)
        out = self.pointwise(out)
        return out

Is this correct? Is this equivalent to tensorflow.keras.layers.SeparableConv2D?
What about padding = 'same'?
How to ensure that my input and output size is the same while doing this?
My attempt:
x = F.pad(x, (8, 7, 0, 0), )

Because the kernel size is (1,16), I added left and right padding, 8 and 7 respectively. Is this the right way (and best way) to achieve padding = 'same'? How can I place this inside my SeparableConv2d class, and calculate on the fly given the input data dimension size?
All together
class SeparableConv2d(nn.Module):
    def __init__(self, in_channels, out_channels, depth, kernel_size, bias=False):
        super(SeparableConv2d, self).__init__()
        self.depthwise = nn.Conv2d(in_channels, out_channels*depth, kernel_size=kernel_size, groups=in_channels, bias=bias)
        self.pointwise = nn.Conv2d(out_channels*depth, out_channels, kernel_size=1, bias=bias)

    def forward(self, x):
        out = self.depthwise(x)
        out = self.pointwise(out)
        return out

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.separable_conv = SeparableConv2d(
            in_channels=32,
            out_channels=64,
            depth=1,
            kernel_size=(1,16)
        )

    def forward(self, x):
        x = F.pad(x, (8, 7, 0, 0), )
        x = self.separable_conv(x)
        return x

Any problem with these codes?


title: Periodic/Oscillating Loss Function for Pytorch CCNN

body:
I’m training a CNN on ImageNet, and I’m seeing some odd, periodic oscillations in my loss function. The curve looks generally correct, but you can see clear oscillations as the curve flattens out. I’m capturing the loss every 100 mini batches, and since my batch size is 128 and my training set is ~100k images, this means that it takes 10 steps to go through the data. So the periodicity I see here is that the loss increases over the course of 10 steps, and then drops suddenly at the start of a new epoch and then progressively increases until the next epoch is reached.

I'm showing most of my training code below (not showing it in its entirety for brevity). I’m resetting the “running_loss” to zero at the beginning of each epoch, and at the end of every set of 100 mini-batches. Does anyone have any perspective here? I don’t know whether I’m doing something wrong in the way I’m computing/collecting the loss, or if there’s something more fundamental wrong with my model.

train_data = ImageDataset(train_file_list, transform=transform)
train_data_loader = DataLoader(train_data, batch_size=128, shuffle=True, num_workers=4)

alexnet = AlexNetPyTorch(NUM_CLASSES)
alexnet = alexnet.to(device)
loss_fn = nn.CrossEntropyLoss()
optimizer = optim.Adam(params=alexnet.parameters(), lr=3.0e-4, weight_decay=5.0e-4)

history = {}
history['loss'] = []
history['val_loss'] = []
history['accuracy'] = []
history['val_accuracy'] = []
step_increment = 100
for epoch in range(30):

    running_loss = 0.0
    running_val_loss = 0.0

    for step, data in enumerate(train_data_loader):
        alexnet.train()
        X_train, y_train = data
        X_train = X_train.to(device)
        y_train = y_train.to(device)

        # forward + backward + optimize
        y_pred = alexnet(X_train)
        loss = loss_fn(y_pred, y_train)

        # zero the parameter gradients
        optimizer.zero_grad()

        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()

        if step % step_increment == step_increment-1:    # print every 100 mini-batches

            # Compute validation loss
            with torch.no_grad():
                alexnet.eval()

                _, preds = torch.max(y_pred,1)
                train_accuracy = torch.sum(preds == y_train).item()/len(y_train)

                X_val, y_val = validation_data
                X_val = X_val.to(device)
                y_val = y_val.to(device)
                y_pred = alexnet(X_val)
                val_loss = loss_fn(y_pred, y_val).item()

                _, preds = torch.max(y_pred,1)
                val_accuracy = torch.sum(preds == y_val).item()/len(y_val)

                history['loss'].append(np.mean(running_loss/step_increment))
                history['accuracy'].append(train_accuracy)
                history['val_accuracy'].append(val_accuracy)
                history['val_loss'].append(val_loss)

                print('[%d, %5d] loss: %.3f acc: %.3f val loss: %.3f val acc: %.3f' %
                      (epoch + 1, step+1, running_loss/step_increment, train_accuracy, val_loss, val_accuracy))

                running_loss = 0.0


title: How to understand creating leaf tensors in PyTorch?

body:
From PyTorch documentation:
b = torch.rand(10, requires_grad=True).cuda()
b.is_leaf
False
# b was created by the operation that cast a cpu Tensor into a cuda Tensor

e = torch.rand(10).cuda().requires_grad_()
e.is_leaf
True
# e requires gradients and has no operations creating it

f = torch.rand(10, requires_grad=True, device="cuda")
f.is_leaf
True
# f requires grad, has no operation creating it

But why are e and f leaf Tensors, when they both were also cast from a CPU Tensor, into a Cuda Tensor (an operation)?
Is it because Tensor e was cast into Cuda before the in-place operation requires_grad_()?
And because f was cast by assignment device="cuda" rather than by method .cuda()?


title: Pytorch nn.Linear RuntimeError: mat1 dim 1 must match mat2 dim 0

body:
class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 16, 3, padding=1)
        self.conv2 = nn.Conv2d(16, 32, 3, padding=1)
        self.max_pool = nn.MaxPool2d(2)
        self.fc1 = nn.Linear(7*7*32, 128)
        self.fc1 = nn.Linear(128, 10)

    def forward(self, x):
        x = self.max_pool(F.relu(self.conv1(x)))
        x = self.max_pool(F.relu(self.conv2(x)))
        x = torch.flatten(x, 1)
        x = self.fc1(x)
        x = F.relu(x)
        x = self.fc2(x)
        x = F.softmax(x, dim=1)
        return x

The above is the model. The shape of input image is 1×28×28 with batch_size 16. I tried to print x.shape after flatten, which is (16, 1568), completely corresponding to the input size of self.fc1(). Does anyone have any idea?


title: Seq2seq pytorch Inference slow

body:
I tried the seq2seq pytorch implementation available here seq2seq . After profiling the evaluation(evaluate.py) code, the piece of code taking longer time was the decode_minibatch method

def decode_minibatch(
    config,
    model,
    input_lines_src,
    input_lines_trg,
    output_lines_trg_gold
):
    """Decode a minibatch."""
    for i in xrange(config['data']['max_trg_length']):

        decoder_logit = model(input_lines_src, input_lines_trg)
        word_probs = model.decode(decoder_logit)
        decoder_argmax = word_probs.data.cpu().numpy().argmax(axis=-1)
        next_preds = Variable(
            torch.from_numpy(decoder_argmax[:, -1])
        ).cuda()

        input_lines_trg = torch.cat(
            (input_lines_trg, next_preds.unsqueeze(1)),
            1
        )

return input_lines_trg

Trained the model on GPU and loaded the model in CPU mode to make inference. But unfortunately, every sentence seems to take ~10sec. Is slow prediction expected on pytorch?

Any fixes, suggestions to speed up would be much appreciated. Thanks.


title: PyTorch Autograd automatic differentiation feature

body: I am just curious to know, how does PyTorch track operations on tensors (after the .requires_grad is set as True and how does it later calculate the gradients automatically. Please help me understand the idea behind autograd. Thanks.


title: Train SqueezeNet model using MNIST dataset Pytorch

body:
I want to train SqueezeNet 1.1 model using MNIST dataset instead of ImageNet dataset.
Can i have the same model as torchvision.models.squeezenet?
Thanks!