title: Custom loss function does not minimize in PyTorch

body:
I am using a PyTorch code to train over a custom loss function in an unsupervised setting.However, the loss doesn't go down and stays the same over may epochs during the training phase. Please see the training code snippet below:

X = np.load(<data path>) #Load dataset which is a numpy array of N points with some dimension each.
num_samples, num_features = X.shape

gmm = GaussianMixture(n_components=num_classes, covariance_type='spherical')
gmm.fit(X)
z_gmm = gmm.predict(X)

R_gmm = gmm.predict_proba(X)
pre_R = Variable(torch.log(torch.from_numpy(R_gmm + 1e-8)).type(dtype), requires_grad=True)
R = torch.nn.functional.softmax(pre_R)

F = torch.stack(Variable(torch.from_numpy(X).type(dtype), requires_grad=True))
U = Variable(torch.from_numpy(gmm.means_).type(dtype), requires_grad=False)

z_pred = torch.max(R, 1)[1]

distances = torch.sum(((F.unsqueeze(1) - U) ** 2), dim=2)
custom_loss = torch.sum(R * distances) / num_samples

learning_rate = 1e-3
opt_train= torch.optim.Adam([train_var], lr = learning_rate)
U = torch.div(torch.mm(torch.t(R), F), torch.sum(R, dim=0).unsqueeze(1)) #In place assignment with a formula over variables and hence no gradient update is needed.

for epoch in range(max_epochs+1):
    running_loss = 0.0
    for i in range(stepSize):
    # zero the parameter gradients
    opt_train.zero_grad()

    # forward + backward + optimize
    loss = custom_loss
    loss.backward(retain_graph=True)
    opt_train.step()
    running_loss += loss.data[0]

if epoch % 25 == 0:
    print(epoch, loss.data[0]) # OR running_loss also gives the same values.
    running_loss = 0.0

O/P:
0 5.8993988037109375
25 5.8993988037109375
50 5.8993988037109375
75 5.8993988037109375
100 5.8993988037109375

Am I missing something in the training? I followed this example/tutorial.
Any help and pointers in this regard will be much appreciated.


I was trying to do element-wise multiplication of two tensors using the example provided here.

My code:

import torch

x = torch.Tensor([2, 3])
y = torch.Tensor([2, 1])
z = torch.cmul(x, y)
print(z)

It is giving me the following error.

AttributeError: module 'torch' has no attribute 'cmul'

Can anyone tell me why I am getting this error?


title: pytorch collate_fn reject sample and yield another

body:
I have built a Dataset, where I'm doing various checks on the images I'm loading. I'm then passing this DataSet to a DataLoader.

In my DataSet class I'm returning the sample as None if a picture fails my checks and i have a custom collate_fn function which removes all Nones from the retrieved batch and returns the remaining valid samples.

However at this point the returned batch can be of varying size. Is there a way to tell the collate_fn to keep sourcing data until the batch size meets a certain length?

class DataSet():
     def __init__(self, example):
          # initialise dataset
          # load csv file and image directory
          self.example = example
     def __getitem__(self,idx):
          # load one sample
          # if image is too dark return None
          # else
          # return one image and its equivalent label

dataset = Dataset(csv_file='../', image_dir='../../')

dataloader = DataLoader(dataset , batch_size=4,
                        shuffle=True, num_workers=1, collate_fn = my_collate )

def my_collate(batch): # batch size 4 [{tensor image, tensor label},{},{},{}] could return something like G = [None, {},{},{}]
    batch = list(filter (lambda x:x is not None, batch)) # this gets rid of nones in batch. For example above it would result to G = [{},{},{}]
    # I want len(G) = 4
    # so how to sample another dataset entry?
    return torch.utils.data.dataloader.default_collate(batch)


title: Best way in Pytorch to upsample a Tensor and transform it to rgb?

body:
For a nice output in Tensorboard I want to show a batch of input images, corresponding target masks and output masks in a grid.
Input images have different size then the masks. Furthermore the images are obviously RGB.
From a batch of e.g. 32 or 64 I only want to show the first 4 images.

After some fiddling around I came up with the following example code. Good thing: It works.
But I am really not sure if I missed something in Pytorch. It just looks much longer then I expected. Especially the upsampling and transformation to RGB seems wild. But the other transformations I found would not work for a whole batch.

import torch
from torch.autograd import Variable
import torch.nn.functional as FN
import torchvision.utils as vutils
from tensorboardX import SummaryWriter
import time
batch = 32
i_size = 192
o_size = 112
nr_imgs = 4

# Tensorboard init
writer = SummaryWriter('runs/' + time.strftime('%Y%m%d_%H%M%S'))

input_image=Variable(torch.rand(batch,3,i_size,i_size))
target_mask=Variable(torch.rand(batch,o_size,o_size))
output_mask=Variable(torch.rand(batch,o_size,o_size))

# upsample target_mask, add dim to have gray2rgb
tm = FN.upsample(target_mask[:nr_imgs,None], size=[i_size, i_size], mode='bilinear')
tm = torch.cat( (tm,tm,tm), dim=1)  # grayscale plane to rgb

# upsample target_mask, add dim to have gray2rgb
om = FN.upsample(output_mask[:nr_imgs,None], size=[i_size, i_size], mode='bilinear')
om = torch.cat( (om,om,om), dim=1)  # grayscale plane to rgb

# add up all images and make grid
imgs = torch.cat( ( input_image[:nr_imgs].data, tm.data, om.data ) )
x = vutils.make_grid(imgs, nrow=nr_imgs, normalize=True, scale_each=True)

# Tensorboard img output
writer.add_image('Image', x, 0)

EDIT: Found this on Pytorchs Issues list. Its about Batch support for Transform. Seems there are no plans to add batch transforms in the future. So my current code might be the best solution for the time being, anyway?


title: I can't load my model because I can't put a PosixPath

body:
I'm setting up a script and I need to use some functions from fast-ai package. The fact is that I'm on Windows and when I define my paths, the function from fast-ai named load_learner can't load the model.

I've tried to change the function into the package as:

state = pickle.load(open(str(path) + '/' + str(fname), 'rb'))

instead of:

state = pickle.load(open(path/fname, 'rb'))

but I obtain this error:

 File "lib\site-packages\fastai\basic_train.py", line 462, in load_learner
    state = pickle.load(open(path/fname, 'rb'))
  File "\lib\pathlib.py", line 1006, in __new__
    % (cls.__name__,))
NotImplementedError: cannot instantiate 'PosixPath' on your system

My paths are defined as:

folder_path = './models/model1'
fname = 'model.pkl'

and I call the function as:
model = load_learner(folder_path, fname)

How can I use Windows paths in this function?

UPDATE 1

The answer posted was correct only on Linux. I still have the issue on Windows. I didn't find a way to pass through the PosixPath on Windows. The only solution that I found is to change the internal packages from my modules but it is not a secure way to solve this kind of issue.

Thanks in advance.


title: Jupyter Notebook/PyTorch, module 'torch' has no attribute 'BoolTensor'

body:
I am working in PyTorch and I need to use a BoolTensor, which is available according to the documentation https://pytorch.org/docs/stable/tensors.html

However, when I try to initialize the BoolTensor I get the error: "AttributeError: module 'torch' has no attribute 'BoolTensor'"

All other tensors do work normally, like LongTensor or DoubleTensor

I have no idea what could cause this particular error, since it seems pretty strange that only one type of tensor doesn't work to me...


title: PyTorch - RuntimeError: Assertion 'cur_target >= 0 && cur_target < n_classes' failed

body:
I'm trying to make a neural network with PyTorch to predict student's final exam grades. I've done it like this -

# Hyper Parameters
input_size = 2
hidden_size = 50
num_classes =21
num_epochs = 500
batch_size = 5
learning_rate = 0.1

# define a customise torch dataset
class DataFrameDataset(torch.utils.data.Dataset):
def __init__(self, df):
    self.data_tensor = torch.Tensor(df.as_matrix())

# a function to get items by index
def __getitem__(self, index):
    obj = self.data_tensor[index]
    input = self.data_tensor[index][0:-1]
    target = self.data_tensor[index][-1] - 1
    return input, target

# a function to count samples
def __len__(self):
    n, _ = self.data_tensor.shape
    return n

# load all data
data_i = pd.read_csv('dataset/student-mat.csv', header=None,delimiter=";")
data = data_i.iloc[:,30:33]

# normalise input data
for column in data:
# the last column is target
if column != data.shape[1] - 1:
    data[column] = data.loc[:, [column]].apply(lambda x: (x - x.mean()) / x.std())

# randomly split data into training set (80%) and testing set (20%)
msk = np.random.rand(len(data)) < 0.8
train_data = data[msk]
test_data = data[~msk]

# define train dataset and a data loader
train_dataset = DataFrameDataset(df=train_data)
train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=batch_size, shuffle=True)

# Neural Network
class Net(nn.Module):
def __init__(self, input_size, hidden_size, num_classes):
    super(Net, self).__init__()
    self.fc1 = nn.Linear(input_size, hidden_size)
    self.sigmoid = nn.Sigmoid()
    self.fc2 = nn.Linear(hidden_size, num_classes)

def forward(self, x):
    out = self.fc1(x)
    out = self.sigmoid(out)
    out = self.fc2(out)
    return out

net = Net(input_size, hidden_size, num_classes)

# Loss and Optimizer
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Rprop(net.parameters(), lr=learning_rate)

# store all losses for visualisation
all_losses = []

# train the model by batch
for epoch in range(num_epochs):
    for step, (batch_x, batch_y) in enumerate(train_loader):
        # convert torch tensor to Variable
        X = Variable(batch_x)
        Y = Variable(batch_y.long())

        # Forward + Backward + Optimize
        optimizer.zero_grad()  # zero the gradient buffer
        outputs = net(X)
        loss = criterion(outputs, Y)
        all_losses.append(loss.data[0])
        loss.backward()
        optimizer.step()

        if epoch % 50 == 0:
            _, predicted = torch.max(outputs, 1)
            # calculate and print accuracy
            total = predicted.size(0)
            correct = predicted.data.numpy() == Y.data.numpy()

            print('Epoch [%d/%d], Step [%d/%d], Loss: %.4f, Accuracy: %.2f %%'
                  % (epoch + 1, num_epochs, step + 1,
                      len(train_data) // batch_size + 1,
                      loss.data[0], 100 * sum(correct)/total))

I'm getting an error at line loss = criterion(outputs, Y) which says -
RuntimeError: Assertion 'cur_target >= 0 && cur_target < n_classes' failed.  at /pytorch/torch/lib/THNN/generic/ClassNLLCriterion.c:62

I'm not able to figure out what I'm doing wrong as I'm pretty new to this and I've already checked the other posts here but, they don't seem to help.
The data dataframe looks like -

     30  31  32

0     5   6   6
1     5   5   6
2     7   8  10
3    15  14  15
4     6  10  10
5    15  15  15

Can anyone tell me what I am doing wrong and how can I correct it. Any help is appreciated! Thanks! :)


title: Convert 'int' to pytorch 'Variable' makes problems

body:
First project with pytorch and I got stuck trying to convert an MNIST label 'int' into a torch 'Variable'. Debugger says it has no dimension?!

# numpy mnist data
X_train, Y_train = read_data("training")
X_test , Y_test  = read_data("testing")

arr = np.zeros(5)
for i in range(5):
    # in your training loop:
    costs_ = 0
    for k in range(10000):
        optimizer.zero_grad()                            # zero the gradient buffers
        a = torch.from_numpy(np.expand_dims(X_train[k].flatten(), axis=0)).float()
        b = torch.from_numpy(np.array(Y_train[k], dtype=np.float)).float()
        input = Variable(a)
        output = net(input)
        target = Variable(b)                             # PROBLEM!!
        loss = criterion(output, target)
        loss.backward()
        optimizer.step()                                 # Does the update

        costs_ += loss.data.numpy()
    arr[i] = costs_
    print(i)

Thrown error is: "RuntimeError: input and target have different number of elements: input[1 x 1] has 1 elements, while target[] has 0 elements at /b/wheel/pytorch-src/torch/lib/THNN/generic/MSECriterion.c:12"


title: What is the difference between view and view_as in PyTorch?

body: I am building neural networks in Pytorch, I see view and view_as used interchangeably in various implementation what is the difference between them?


title: cudnn error while running a pyorch code on gpu

body:
I have the following error:
Traceback (most recent call last):

  File "odenet_mnist.py", line 343, in <module>
    logits = model(x)
  File "/home/subhashnerella/.conda/envs/pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/subhashnerella/.conda/envs/pytorch/lib/python3.7/site-packages/torch/nn/modules/container.py", line 92, in forward
    input = module(input)
  File "/home/subhashnerella/.conda/envs/pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/subhashnerella/.conda/envs/pytorch/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 320, in forward
    self.padding, self.dilation, self.groups)
RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED

This is not my code. I am just trying to run the code of a recent paper "Neural Ordinary Differential Equations"-by Chen et al.
Here is the link to the code.

gpu: nvidia 2080Ti

pytorch version:'1.0.1.post2'

cuda 9.0

python 3.7.2

cudnn:

#define CUDNN_MAJOR 7
#define CUDNN_MINOR 4
#define CUDNN_PATCHLEVEL 2
--
#define CUDNN_VERSION (CUDNN_MAJOR * 1000 + CUDNN_MINOR * 100 + CUDNN_PATCHLEVEL)

I am new to pytorch. Why am i getting this error and how do i fix it?


title: What would be equivalent of pytorch's torch.nn.CosineEmbeddingLoss in tensorflow?

body: CosineEmbeddingLoss in Pytorch is the perfect function I am looking for in tensorflow, but I can only find tf.losses.cosine_distance. Is there a way or code that writes CosineEmbeddingLoss in tensorflow?


title: Windows installing pytorch 0.3

body: I need to install pytorch==0.3 (I'm using conda), but when I run the command line, conda says that the needed packages are not available. Is there a way I can install it (possibly without using ubuntu)?


title: Pytorch problem testing CNN: RuntimeError: shape '[64, 1]' is invalid for input of size 1920

body:
Testing network:

 def test(args, model, device, test_loader):
           model.eval()
           total = 0
           test_loss = 0
           correct = 0
           with torch.no_grad():
           for batch_idx, batch in enumerate(test_loader):
               data = batch['image']
               target = batch['key_points']
               data, target = data.to(device), target.to(device)
               # data, target = Variable(data), Variable(target)
           # for data, target in test_loader:
           #     data, target = data.to(device), target.to(device)
           #     data, target = Variable(data), Variable(target)
                 data = data.unsqueeze(1).float()

                 print('TESTING 1: Data and target shape: ', data.shape, '
                       ', target.shape)
                 output = model(data)
                 target = target.view(target.shape[0], -1)

                 print('TESTING 2: Data and target shape: ', data.shape, '
                       ', target.shape)
                 # test_loss += F.nll_loss(output, target,
                 reduction='sum').item()  # sum up batch loss orginal
                 test_loss = F.nll_loss(output, torch.max(target, 1)[1])
                  # test_loss += F.nll_loss(output,
                  torch.max(target.float(), 1)[1], reduction='sum').item()

                   print('TESTING 3: Data and target shape: ', data.shape,
                       ' ', target.shape)
                   # pred = output.max(1, keepdim=True)[1]
                   pred = output.argmax(dim=1, keepdim=True)  # original

                   print('TESTING 4: Data and target shape: ', data.shape,
                       ' ', target.shape)

                   correct += pred.eq(target.view_as(pred)).sum().item()
                   print('TESTING last: Data and target shape: ',
                   data.shape, ' ', target.shape)

                   test_loss /= len(test_loader.dataset)
                   print('\nTest set: Average loss: {:.4f}, Accuracy:
                        {}/{} ({:.0f}%)\n'.format(
                         test_loss , correct, len(test_loader.dataset),
                         100. * correct / len(test_loader.dataset)))

I can run the training loop which is fine however when I try to run testing loop as well the training will only run 1 epoch and I get the following message.

TESTING 1: Data and target shape:  torch.Size([64, 1, 96, 96])
torch.Size([64, 15, 2]) TESTING 2: Data and target shape:
torch.Size([64, 1, 96, 96])   torch.Size([64, 30])
TESTING 3: Data and target shape:  torch.Size([64, 1, 96, 96])
torch.Size([64, 30])
TESTING 4: Data and target shape:  torch.Size([64, 1, 96, 96])
 torch.Size([64, 30]) Traceback (most recent call last):   File
 "/home/keith/PycharmProjects/FacialLandMarks/WorkOut.py", line 468, in <module>
    main()   File "/home/keith/PycharmProjects/FacialLandMarks/WorkOut.py", line 463, in main
    test(args, model, device, test_loader)   File "/home/keith/PycharmProjects/FacialLandMarks/WorkOut.py", line 380, in test
    correct += pred.eq(target.view_as(pred)).sum().item()
 RuntimeError: shape '[64, 1]' is invalid for input of size 1920
 Process finished with exit code 1

I have not been able to come across any helpful resources on-line or in a number of books I've purchased, on this specific problem, facial key point detection testing phase. I think problem is shown in error message(marked with comment) or with my data.  I am using a separate set of test images with training key points csv file.  As you can see I have printed out shape of data throughout the testing phase.
Any help or useful links always appreciated. Thank you

When I use pycharm debugger and view breakpoints when I get to the line of code below:

pred = output.max(1, keepdim=True)[1]

Then pred tensor changes to this:

tensor([[29],
        [29],
        [29],
        [29],
        [29],
        [29],
        [29],
        [29],

         ...
        [29],
        [29],
        [29],
        [29],
        [29],
        [29],
        [29]], device='cuda:0')


title: How do I assign a numpy.int64 to a torch.cuda.FloatTensor?

body:
Im not able to assign int64 to torch tensor. I have got the following tensor

tempScale = torch.zeros((total, len(scale))).cuda() if useGpu else torch.zeros((nbPatchTotal, len(scale)))

In my code, when I'm using the following line, it's throwing an error message

tmpScale[:, j] = scale

The error message is

TypeError: can't assign a numpy.int64 to a torch.cuda.FloatTensor

what am I missing?


title: Downloading transformers models to use offline

body:
I have a trained transformers NER model that I want to use on a machine not connected to the internet. When loading such a model, currently it downloads cache files to the .cache folder.

To load and run the model offline, you need to copy the files in the .cache folder to the offline machine. However, these files have long, non-descriptive names, which makes it really hard to identify the correct files if you have multiple models you want to use. Any thoughts on this?

Example of model files


title: Pytorch : RuntimeError: mat1 dim 1 must match mat2 dim 0

body:
using resnet50 model. Customize the last layer and it showing runtime error..Im new to PyTorch and I keep getting the error mat1 dim1 must match mat1 dim0
this is my code for the network
from torchvision import models
model = models.resnet50(pretrained=True)

for param in model.parameters():
    param.requires_grad = False

class Identity(nn.Module):
    def __init__(self):
        super(Identity, self).__init__()
    def forward(self, x):
        return x

model.avgpool = Identity()
model.fc = nn.Linear(2048, 2, bias=True)

for param in model.fc.parameters():
    param.requires_grad = True
model = model.to(device)

criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.01)

def train(num_epoch, model):
    for epoch in range(0, 3):
        losses = []
        model.train()
        loop = tqdm(enumerate(train_loader), total=len(train_loader))
        for batch_idx, (data, targets) in loop:
            data = data.to(device=device)
            targets = targets.to(device=device)
            scores = model.forward(data)
            loss = criterion(scores, targets)

            optimizer.zero_grad()
            losses.append(loss)
            loss.backward()
            optimizer.step()

            loop.set_description(f"Epoch {epoch+1}/{num_epoch} process: {int((batch_idx / len(train_loader)) * 100)}")
            loop.set_postfix(loss=loss.data.item())

train(1, model)

RuntimeError: mat1 dim 1 must match mat2 dim 0


title: How to load 2D data into an LSTM in pytorch

body:
I have a series of sine waves that i have loaded in using a custom dataloader. The data is converted to a torch tensor using from_numpy. I then try to load the data using an enumerator over the train_loader. The iterator is shown below.

for epoch in range(epochs):
    for i, data in enumerate(train_loader):
        input = np.array(data)
        train(epoch)

The error i receive is:

RuntimeError: input must have 3 dimensions, got 2

I know i need to have my input data in [sequence length, batch_size, input_size] for an LSTM but i have no idea how to format my array data of 1000 sine waves of length 10000.

Below is my training method.

def train(epoch):
    model.train()
    train_loss = 0
    def closure():
        optimizer.zero_grad()
        print(input.shape)
        output = model(Variable(input))
        loss = loss_function(output)
        print('epoch: ', epoch.item(),'loss:', loss.item())
        loss.backward()
        return loss
    optimizer.step(closure)

I thought i would try add (seq_length, batch_size, input_size) in a tuple but this cant be fed into the network. Further to this my assumption was that the dataloader fed batch size into the system. Any help would be appreciated.

edit:

Here is my sample data:

T = 20
L = 1000
N = 100

x = np.empty((N, L), 'int64')
x[:] = np.array(range(L)) + np.random.randint(-4 * T, 4 * T, N).reshape(N, 1)
data = np.sin(x / 1.0 / T).astype('float64')
torch.save(data, open('traindata.pt', 'wb'))


title: Pytorch get "RuntimeError: CUDA error: device-side assert triggered"

body:
Snippet from my code :

max = torch.tensor([3])
if USE_CUDA: max = max.cuda()
max_embedding = self.max_embedding(max) # dim of max_embedding: 1*5

item_dict = {}
for item in item_list:
    item = torch.tensor(item)
    if USE_CUDA: item = item.cuda()
    item_embedding = self.item_embedding(item) # dim of item_embedding: 1*20

embedded = torch.cat((max_embedding, item_embedding), 1)

But I get error of "RuntimeError: CUDA error: device-side assert triggered".
The output by adding CUDA_LAUNCH_BLOCKING=1:

/pytorch/aten/src/THC/THCTensorIndex.cu:308: void indexSelectSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, long) [with T = float, IndexType = unsigned int, DstDim =2, SrcDim = 2, IdxDim = -2: block: [0,0,0], thread: [0,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:308: void indexSelectSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, long) [with T = float, IndexType = unsigned int, DstDim =2, SrcDim = 2, IdxDim = -2: block: [0,0,0], thread: [1,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:308: void indexSelectSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, long) [with T = float, IndexType = unsigned int, DstDim =2, SrcDim = 2, IdxDim = -2: block: [0,0,0], thread: [2,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:308: void indexSelectSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, long) [with T = float, IndexType = unsigned int, DstDim =2, SrcDim = 2, IdxDim = -2: block: [0,0,0], thread: [3,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:308: void indexSelectSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, long) [with T = float, IndexType = unsigned int, DstDim =2, SrcDim = 2, IdxDim = -2: block: [0,0,0], thread: [4,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
Traceback (most recent call last):
  File "mytest.py", line 33, in <module>
     if USE_CUDA: item = item.cuda()
RuntimeError: CUDA error: device-side assert triggered

How to fix it?


title: Choosing a margin for contrastive loss in a siamese network

body:
I'm building a siamese network for a metric-learning task, using a contrastive loss function, and I'm uncertain on how to set the 'margin' hyperparameter for the loss.

My inputs to the loss function are currently 1024-dimension dense embeddings from an RNN layer - Does the dimensionality of that input affect how I pick a margin? Should I use a dense layer to project it to a lower-dimensional space first? Any pointers on how to pick a specific margin value (or any relevant research) would be really appreciated! In case it matters, I'm using PyTorch.


title: Torch squeeze and the batch dimension

body:
does anyone here know if the torch.squeeze function respects the batch (e.g. first) dimension? From some inline code it seems it does not.. but maybe someone else knows the inner workings better than I do.
Btw, the underlying problem is that I have tensor of shape (n_batch, channel, x, y, 1). I want to remove the last dimension with a simple function, so that I end up with a shape of (n_batch, channel, x, y).
A reshape is of course possible, or even selecting the last axis. But I want to embed this functionality in a layer so that I can easily add it to a ModuleList or Sequence object.
EDIT: just found out that for Tensorflow (2.5.0) the function tf.linalg.diag DOES respect batch dimension. Just a FYI that it might differ per function you are using


title: Pytorch: How to find accuracy for Multi Label Classification?

body:
I am using vgg16, where number of classes is 3, and I can have multiple labels predicted for a data point.

vgg16 = models.vgg16(pretrained=True)
vgg16.classifier[6]= nn.Linear(4096, 3)

using loss function : nn.BCEWithLogitsLoss()

I am able to find find accuracy in case of a single label problem, as

 `images, labels = data
 images, labels = images.to(device), labels.to(device)
 labels = Encode(labels)
 outputs = vgg16(images)
 _, predicted = torch.max(outputs.data, 1)
 total += labels.size(0)
 correct += (predicted == labels).sum().item()
 acc = (100 * correct / total)`

How can I find accuracy for multi label classification?


title: TypeError: mul() argument 'other' (position 1) must be Tensor, not ReLU

body:
I would like to add a torch.nn.ReLU() layer between fc1 and fc2 layer.

Original code:

model:

# ...
self.fc1 = nn.Linear(4096, 256)
self.fc2 = nn.Linear(256, 4096)
# ...

def forward(...):
    # ...
    x = x.view(-1, 4096)
    x = self.fc1(x))
    if a7 is not None:
        x = x * a7.squeeze()
    # ...

I tried

# ...
x = x.view(-1, 4096)
x = nn.ReLU(self.fc1(x)))
if a7 is not None:
    x = x * a7.squeeze()
# ...

and this error pops out.


title: PyTorch FasterRCNN TypeError: forward() takes 2 positional arguments but 3 were given

body:
I am working on object detection and I have a dataset containing images and their corresponding bounding boxes (ground-truth values).

I actually have built my own feature extractor which takes an image as input and outputs a feature map(basically an encoder-decoder system where the final output of the decoder is the same as the image size and has 3 channels). Now, I want to feed this feature map as an input to a FasterRCNN model for detection instead of the original image. I am using the following code to add the feature map(using RTFNet to generate feature map -  code at this link) on top the FRCNN detection module

frcnn_model = fasterrcnn_resnet50_fpn(pretrained=True)
in_features = frcnn_model.roi_heads.box_predictor.cls_score.in_features
frcnn_model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)
fpn_block = frcnn_model.backbone.fpn
rpn_block = frcnn_model.rpn

backbone = RTFNet(num_classes) RTFNet is a feature extractor taking as input, an image having 4 channels(fused RGB and thermal image) ,
model = nn.Sequential(backbone, nn.ReLU(inplace=True))
model = nn.Sequential(model,fpn_block)
model = nn.Sequential(model,rpn_block)
model = nn.Sequential(model,FastRCNNPredictor(in_features, num_classes))

I am just trying to test and see if it is working by using the following code which generates random images and bounding boxes

images, boxes = torch.rand(1, 4, 512, 640), torch.rand(4, 11, 4)
labels = torch.randint(1, num_classes, (4, 11))
images = list(image for image in images)
targets = []
for i in range(len(images)):
  d = {}
  d['boxes'] = boxes[i]
  d['labels'] = labels[i]
  targets.append(d)
output = model(images, targets)

Running this gives me the following error

TypeError                                 Traceback (most recent call last)
<ipython-input-22-2637b8c27ad2> in <module>()
----> 1 output = model(images, targets)

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--> 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

TypeError: forward() takes 2 positional arguments but 3 were given

However, when I replace my model with a normal FasterRCNN Model with the following,

model = fasterrcnn_resnet50_fpn(pretrained=True)
in_features = model.roi_heads.box_predictor.cls_score.in_features
model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)

there is no error and it works fine

Can anyone let me know where I am going wrong? Thanks in advance


title: How to run Pytorch model in normal non-parallel way?

body:
I am going through this script, and there is a code block which takes 2 options into account, DataParallel and DistributedDataParallel here:

if not args.distributed:
    if args.arch.startswith('alexnet') or args.arch.startswith('vgg'):
        model.features = torch.nn.DataParallel(model.features)
        model.cuda()
    else:
        model = torch.nn.DataParallel(model).cuda()
else:
    model.cuda()
    model = torch.nn.parallel.DistributedDataParallel(model)

What if I don't want either of these options, and I want to run it without even DataParallel. How do I do it?

How do I define my model so that it runs as a plain nn and not parallelizing anything?


title: Arguments are located on different GPUs when using nn.DataParallel(model)

body:
Torch 0.4.1

Python 2.7.12

I was adapting NMP QC code (with some compatibility issues ironed out) to use multiple GPUs since my GPU couldn't handle the workload (crashed after running out of VRAM)

I'm new to pytorch, but I found a tutorial on using nn.DataParallel(model) to implement multi-gpu use

I modified main.py to use nn.DataParallel(model). Areas I changed have "#NEW" stuck to them.

The code runs fine even in multi-gpu mode if running on a single GPU, but gets a "arguments are located on different GPUs" error when running on 2 or more GPU

libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'.
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs3
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs2
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
Unexpected end of /proc/mounts line `overlay / overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/QKSBQ5PAFDDC3OMBEELQQETALQ:/var/lib/docker/overlay2/l/WWYI3IDQPNXGON7AHODBPSTVXL:/var/lib/docker/overlay2/l/Q54I2HYS4TKH4LDJKBTVTGWWO6:/var/lib/docker/overlay2/l/IUV2LFPNMPOS3MREOTT52TKL54:/var/lib/docker/overlay2/l/DB5GBUCI3DCBPX6TJG3O337YVB:/var/lib/docker/overlay2/l/DNYKXCZJH5FMFNJLNGYJJ2ITPI:/var/lib/docker/overlay2/l/7DZCTDVNSTPJISGW65UG7U3F75:/var/lib/docker/overlay2/l/VOEQO652VS63NLDLZZ4TCIJLO6:/var/lib/docker/overlay2/l/4SI6ZCRUIORG5'
Traceback (most recent call last):
  File "main.py", line 332, in <module>
    main()
  File "main.py", line 190, in main
    train(train_loader, model, criterion, optimizer, epoch, evaluation, logger)
  File "main.py", line 251, in train
    output = model(g, h, e)
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/parallel/data_parallel.py", line 123, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/parallel/data_parallel.py", line 133, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/usr/local/lib/python2.7/dist-packages/torch/nn/parallel/parallel_apply.py", line 77, in parallel_apply
    raise output
RuntimeError: arguments are located on different GPUs at /pytorch/aten/src/THC/generic/THCTensorMathBlas.cu:236

Since I was sending the inputs one at a time instead of at once like in the tutorial, I checked by using .get_device(), which confirmed all 4 arguments being sent (g, h, e, target) were on the same device (device 0)


title: Unable to load one-hot labels in a torchtext iterator ( ValueError: too many dimensions 'str')

body:
My "label" field is a one-hot vector of 201 length. I am however unable to create an iterator with this one-hot representation. I'm getting the below error if I try to iterate over the iterator.

from torchtext.data import Field
from torchtext.data import TabularDataset
from torchtext.data import Iterator, BucketIterator

tokenize = lambda x: x.split()

TEXT = Field(sequential=True, tokenize=tokenize, lower=True)

LABEL = Field(sequential=True, use_vocab=False)

datafields = [("text", TEXT), ("label", LABEL)]

train, test = TabularDataset.splits(
               path = '/home/karthik/Documents/Deep_Learning/73Strings/',
               train = "train.csv", validation="test.csv",
               format='csv',
               skip_header=True,
               fields=datafields)

train_iter, val_iter = BucketIterator.splits(
 (train, test), # we pass in the datasets we want the iterator to draw data from
 batch_sizes=(64, 64),
 device=device, # if you want to use the GPU, specify the GPU number here
 sort_key=lambda x: len(x.text), # the BucketIterator needs to be told what function it should use to group the data.
 sort_within_batch=False,
 repeat=False # we pass repeat=False because we want to wrap this Iterator layer.
)
test_iter = Iterator(test, batch_size=64, sort=False, sort_within_batch=False, repeat=False)

for batch in train_iter:
    print(batch)

ValueError                                Traceback (most recent call last)
 in ()
----> 1 for batch in train_iter:
      2     print(batch)

/usr/local/lib/python3.6/dist-packages/torchtext/data/iterator.py in iter(self)
    155                     else:
    156                         minibatch.sort(key=self.sort_key, reverse=True)
--> 157                 yield Batch(minibatch, self.dataset, self.device)
    158             if not self.repeat:
    159                 return

/usr/local/lib/python3.6/dist-packages/torchtext/data/batch.py in init(self, data, dataset, device)
     32                 if field is not None:
     33                     batch = [getattr(x, name) for x in data]
---> 34                     setattr(self, name, field.process(batch, device=device))
     35
     36     @classmethod

/usr/local/lib/python3.6/dist-packages/torchtext/data/field.py in process(self, batch, device)
    199         """
    200         padded = self.pad(batch)
--> 201         tensor = self.numericalize(padded, device=device)
    202         return tensor
    203

/usr/local/lib/python3.6/dist-packages/torchtext/data/field.py in numericalize(self, arr, device)
    321                 arr = self.postprocessing(arr, None)
    322
--> 323         var = torch.tensor(arr, dtype=self.dtype, device=device)
    324
    325         if self.sequential and not self.batch_first:

ValueError: too many dimensions 'str'


title: PyTorch expected CPU got CUDA tensor

body:
I've been struggling to find what's wrong in my code. I'm trying to implement DCGAN paper and from the past 1 hour, I'm going through these errors. Could anyone please help me fix this?
I'm training this on Google colab with GPU runtime but I'm getting this error. Yesterday, I implemented the first GAN paper by Ian Goodfellow and I did not got this error. I don't know what's happening any help would be appreciated. Also, please check whether the gen_input is correct or not.
Here is the code:
import torch
import numpy as np
import torch.nn as nn
import torch.nn.functional as F
import torchvision
import torchvision.transforms as transforms
from torchvision.utils import save_image
import torch.optim as optim

#---------------configuration part------------------#
lr = 0.00002 #learning rate
nc = 3 #color channels
nz = 100 #size of latent vector or size of generator input
ngf = 64 #size of feature maps in generator
ndf = 64 #size of feature maps in discriminator
height = 128 #height of the image
width = 128 #width of the image
num_epochs = 100 #the variable name tells everything
workers = 2 #number of workers to load the data in batches
batch_size = 64 #batch size
image_size = 128 #resizing parameter
root = '/content/gdrive/My Drive/sharingans/' #path to the training directory
beta1 = 0.4

#---------------------------------------------------#
#define the shape of the image
img_shape = (nc, height, width)

#---------------------------------------------------#
#define the weights initialization function
#in the DCGAN paper they state that all weights should be
#randomly initialize weights from normal distribution
#the following function does that

def weights_init(m):
    classname = m.__class__.__name__ #returns the class name(eg: Conv2d or ConvTranspose2d)
    if classname.find('Conv') != -1:
        nn.init.normal_(m.weight.data, 0.0, 0.02) #0.0 is mean and 0.02 is standard deviation
    elif classname.find('BatchNorm') != -1:
        nn.init.normal_(m.weight.data, 1, 0.02) #1 is mean and 0.02 is standard deviation
        nn.init.constant_(m.bias.data, 0.0)

#---------------------------------------------------#

#implement the data loader function to load images

def load_data(image_size, root):
    transform = transforms.Compose([
        transforms.Resize(image_size),
        transforms.ToTensor(),
        transforms.Normalize((0.486, 0.486, 0.486), (0.486, 0.486, 0.486))
        ])

    train_set = torchvision.datasets.ImageFolder(root = root, transform = transform)

    return train_set
train_set = load_data(128, root)
#getting the batches of data
train_data = torch.utils.data.DataLoader(train_set, batch_size = batch_size, shuffle = True, num_workers = workers)

#---------------------------------------------------#
#implement the generator network

class Generator(nn.Module):
    def __init__(self):
        super(Generator, self).__init__()
        self.convt1 = nn.ConvTranspose2d(in_channels = nz, out_channels = ngf*8, kernel_size = 4, stride = 1, padding = 0, bias = False)
        self.convt2 = nn.ConvTranspose2d(in_channels = ngf*8, out_channels = ngf*4, kernel_size = 4, stride = 2, padding = 1, bias = False)
        self.convt3 = nn.ConvTranspose2d(in_channels = ngf*4, out_channels = ngf*2, kernel_size = 4, stride = 2, padding = 1, bias = False)
        self.convt4 = nn.ConvTranspose2d(in_channels = ngf*2, out_channels = ngf, kernel_size = 4, stride = 2, padding = 1, bias = False)
        self.convt5 = nn.ConvTranspose2d(in_channels = ngf, out_channels = 3, kernel_size=4, stride = 2, padding = 1, bias = False)

    def forward(self, t):
        t = self.convt1(t)
        t = nn.BatchNorm2d(t)
        t = F.relu(t)

        t = self.convt2(t)
        t = nn.BatchNorm2d(t)
        t = F.relu(t)

        t = self.convt3(t)
        t = nn.BatchNorm2d(t)
        t = F.relu(t)

        t = self.convt4(t)
        t = nn.BatchNorm2d(t)
        t = F.relu(t)

        t = self.convt5(t)
        t = F.tanh(t)

        return t

#---------------------------------------------------#
#implement the discriminator network

class Discriminator(nn.Module):
    def __init__(self):
        super(Discriminator, self).__init__()
        self.conv1 = nn.Conv2d(in_channels = 3, out_channels = ndf, kernel_size = 4, stride = 2, padding = 1, bias = False)
        self.conv2 = nn.Conv2d(in_channels = ndf, out_channels = ndf*2, kernel_size = 4, stride = 2, padding = 1, bias = False)
        self.conv3 = nn.Conv2d(in_channels = ndf*2, out_channels = ndf*4, kernel_size = 4, stride = 2, padding = 1, bias = False)
        self.conv4 = nn.Conv2d(in_channels = ndf*4, out_channels = ndf*8, kernel_size = 4, stride = 2, padding = 1, bias = False)
        self.conv5 = nn.Conv2d(in_channels = ndf*8, out_channels = 1, kernel_size = 4, stride = 1, padding = 0, bias = False)

    def forward(self, t):
        t = self.conv1(t)
        t = F.leaky_relu(t, 0.2)

        t = self.conv2(t)
        t = nn.BatchNorm2d(t)
        t = F.leaky_relu(t, 0.2)

        t = self.conv3(t)
        t = nn.BatchNorm2d(t)
        t = F.leaky_relu(t, 0.2)

        t = self.conv4(t)
        t = nn.BatchNorm2d(t)
        t = F.leaky_relu(t, 0.2)

        t = self.conv5(t)
        t = F.sigmoid(t)

        return t

#---------------------------------------------------#
#create the instances of networks

generator = Generator()
discriminator = Discriminator()

#apply the weights_init function to randomly initialize weights to mean = 0 and std = 0.02
generator.apply(weights_init)
discriminator.apply(weights_init)

print(generator)
print(discriminator)

#---------------------------------------------------#
#define the loss function
criterion = nn.BCELoss()

#fixed noise
noise = torch.randn(64, nz, 1, 1).cuda()

#conventions for fake and real labels
real_label = 1
fake_label = 0

#create the optimizer instances
optimizer_d = optim.Adam(discriminator.parameters(), lr = lr, betas = (beta1, 0.999))
optimizer_g = optim.Adam(generator.parameters(), lr = lr, betas = (beta1, 0.999))

#---------------------------------------------------#
if torch.cuda.is_available():
    generator = generator.cuda()
    discriminator = discriminator.cuda()
    criterion = criterion.cuda()

Tensor = torch.cuda.FloatTensor if torch.cuda.is_available() else torch.FloatTensor
#---------------------------------------------------#

#Training loop
for epoch in range(num_epochs):
    for i, (images, labels) in enumerate(train_data):

        #ones is passed when the data is coming from original dataset
        #zeros is passed when the data is coming from generator
        ones = Tensor(images.size(0), 1).fill_(1.0)
        zeros = Tensor(images.size(0),1).fill_(0.0)

        real_images = images.cuda()

        optimizer_g.zero_grad()

        #following is the input to the generator
        #we create tensor with random noise of size 100
        gen_input = np.random.normal(0,3,(512,100,4,4))
        gen_input = torch.tensor(gen_input, dtype = torch.float32)
        gen_input = gen_input.cuda()
        #we then pass it to generator()
        gen = generator(gen_input) #this returns a image

        #now calculate the loss wrt to discriminator output
        g_loss = criterion(discriminator(gen), ones)

        #backpropagation
        g_loss.backward()
        #update weights
        optimizer_g.step()

        #above was for generator network

        #now for the discriminator network
        optimizer_d.zero_grad()

        #calculate the real loss
        real_loss = criterion(discriminator(real_images), ones)
        #calculate the fake loss from the generated image
        fake_loss = criterion(discriminator(gen.detach()),zeros)
        #average out the losses
        d_loss = (real_loss + fake_loss)/2

        #backpropagation
        d_loss.backward()
        #update weights
        optimizer_d.step()

        if i%100 == 0:
            print("[EPOCH %d/%d] [Batch %d/%d] [D loss: %f] [G loss: %f]"%(epoch, epochs, i, len(dataset), d_loss.item(), g_loss.item()))

        total_batch = epoch * len(dataset) + i
        if total_batch%20 == 0:
            save_image(gen.data[:5], '/content/gdrive/My Drive/tttt/%d.png' % total_batch, nrow=5)

And here's the error:
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-36-0af32f223344> in <module>()
     18         gen_input = gen_input.cuda()
     19         #we then pass it to generator()
---> 20         gen = generator(gen_input) #this returns a image
     21
     22         #now calculate the loss wrt to discriminator output

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/batchnorm.py in __init__(self, num_features, eps, momentum, affine, track_running_stats)
     40         self.track_running_stats = track_running_stats
     41         if self.affine:
---> 42             self.weight = Parameter(torch.Tensor(num_features))
     43             self.bias = Parameter(torch.Tensor(num_features))
     44         else:

TypeError: expected CPU (got CUDA)

Any help would be appreciated. Thank you!


title: Example CrossEntropyLoss for 3D semantic segmentation in pytorch

body:
I have a network performing 3D convolutions on a 5D input tensor. The output of my network if of size (1, 12, 60, 36, 60) corresponding to ( BatchSize, NumClasses, x-dim, y-dim, z-dim). I need to compute a voxel-wise cross entropy loss. However I keep on getting errors.

When trying to compute cross entropy loss using torch.nn.CrossEntropyLoss(), I keep on getting the following error message:

RuntimeError: multi-target not supported at .../src/THCUNN/generic/ClassNLLCriterion.cu:16

here is the extract of my code:

import torch
import torch.nn as nn
from torch.autograd import Variable
criterion = torch.nn.CrossEntropyLoss()
images = Variable(torch.randn(1, 12, 60, 36, 60)).cuda()
labels = Variable(torch.zeros(1, 12, 60, 36, 60).random_(2)).long().cuda()
loss = criterion(images.view(1,-1), labels.view(1,-1))

Same happens when I create a one-hot tensor for the labels:

nclasses = 12
labels = (np.random.randint(0,12,(1,60,36,60))) # Random labels with values between [0..11]
labels = (np.arange(nclasses) == labels[..., None] - 1).astype(int) # Converts labels to one_hot_tensor
a = np.transpose(labels,(0,4,3,2,1)) #  Reorder dimensions to match shape of "images" ([1, 12, 60, 36, 60])
b = Variable(torch.from_numpy(a)).cuda()
loss = criterion(images.view(1,-1), b.view(1,-1))

Any idea what I'm doing wrong?
Can someone provide an example of computing cross entropy on a 5D output tensor?


title: If we combine one trainable parameters with a non-trainable parameter, is the original trainable param trainable?

body:
I have two nets and I combine their parameters in some fancy way using only pytorch operations. I store the result in a third net which has its parameters set to non-trainable. Then I proceed and pass data through this new net. The new net is just a placeholder for:

placeholder_net.W = Op( not_trainable_net.W, trainable_net.W )

Then I pass data:

output = placeholder_net(input)

I am concerned that since the parameters of the placeholder net are set to non-trainable that it won’t actually train the variable that it should train. Will this happen? Or what is the result when you combine a trainable param with and non-trainable param (and then set that where the param is not trainable)?

Current solution:

del net3.conv0.weight
net3.conv0.weight = net.conv0.weight + net2.conv0.weight

import torch
from torch import nn
import torch.optim as optim

import torchvision
import torchvision.transforms as transforms

from collections import OrderedDict

import copy

def dont_train(net):
    '''
    set training parameters to false.
    '''
    for param in net.parameters():
        param.requires_grad = False
    return net

def get_cifar10():
    transform = transforms.Compose(
        [transforms.ToTensor(),
         transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])
    trainset = torchvision.datasets.CIFAR10(root='./data', train=True, download=True, transform=transform)
    trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,shuffle=True, num_workers=2)
    classes = ('plane', 'car', 'bird', 'cat','deer', 'dog', 'frog', 'horse', 'ship', 'truck')
    return trainloader,classes

def combine_nets(net_train, net_no_train, net_place_holder):
        '''
            Combine nets in a way train net is trainable
        '''
        params_train = net_train.named_parameters()
        dict_params_place_holder = dict(net_place_holder.named_parameters())
        dict_params_no_train = dict(net_no_train.named_parameters())
        for name, param_train in params_train:
            if name in dict_params_place_holder:
                layer_name, param_name = name.split('.')
                param_no_train = dict_params_no_train[name]
                ## get place holder layer
                layer_place_holder = getattr(net_place_holder, layer_name)
                delattr(layer_place_holder, param_name)
                ## get new param
                W_new = param_train + param_no_train  # notice addition is just chosen for the sake of an example
                ## store param in placehoder net
                setattr(layer_place_holder, param_name, W_new)
        return net_place_holder

def combining_nets_lead_to_error():
    '''
    Intention is to only train the net with trainable params.
    Placeholder rnet is a dummy net, it doesn't actually do anything except hold the combination of params and its the
    net that does the forward pass on the data.
    '''
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    ''' create three musketeers '''
    net_train = nn.Sequential(OrderedDict([
          ('conv1', nn.Conv2d(1,20,5)),
          ('relu1', nn.ReLU()),
          ('conv2', nn.Conv2d(20,64,5)),
          ('relu2', nn.ReLU())
        ])).to(device)
    net_no_train = copy.deepcopy(net_train).to(device)
    net_place_holder = copy.deepcopy(net_train).to(device)
    ''' prepare train, hyperparams '''
    trainloader,classes = get_cifar10()
    criterion = nn.CrossEntropyLoss()
    optimizer = optim.SGD(net_train.parameters(), lr=0.001, momentum=0.9)
    ''' train '''
    net_train.train()
    net_no_train.eval()
    net_place_holder.eval()
    for epoch in range(2):  # loop over the dataset multiple times
        running_loss = 0.0
        for i, (inputs, labels) in enumerate(trainloader, 0):
            optimizer.zero_grad() # zero the parameter gradients
            inputs, labels = inputs.to(device), labels.to(device)
            # combine nets
            net_place_holder = combine_nets(net_train,net_no_train,net_place_holder)
            #
            outputs = net_place_holder(inputs)
            loss = criterion(outputs, labels)
            loss.backward()
            optimizer.step()
            # print statistics
            running_loss += loss.item()
            if i % 2000 == 1999:  # print every 2000 mini-batches
                print('[%d, %5d] loss: %.3f' %
                      (epoch + 1, i + 1, running_loss / 2000))
                running_loss = 0.0
    ''' DONE '''
    print('Done \a')

if __name__ == '__main__':
    combining_nets_lead_to_error()


title: ImportError: cannot import name 'NoReturn' PyTorch on MacOS Catalina

body:
I'm having this weird problem in running torchvision in macOS Catalina. Here's the traceback
Traceback (most recent call last):
  File "/Users/xdriyad/PycharmProjects/Resnet/main.py", line 1, in <module>
    from model import ResNet50
  File "/Users/xdriyad/PycharmProjects/Resnet/model.py", line 1, in <module>
    import torch
  File "/Users/xdriyad/PycharmProjects/Resnet/env/lib/python3.6/site-packages/torch/__init__.py", line 573, in <module>
    import torch.quantization
  File "/Users/xdriyad/PycharmProjects/Resnet/env/lib/python3.6/site-packages/torch/quantization/__init__.py", line 9, in <module>
    from .quantize_fx import *
  File "/Users/xdriyad/PycharmProjects/Resnet/env/lib/python3.6/site-packages/torch/quantization/quantize_fx.py", line 1, in <module>
    from .fx import Fuser  # noqa: F401
  File "/Users/xdriyad/PycharmProjects/Resnet/env/lib/python3.6/site-packages/torch/quantization/fx/__init__.py", line 1, in <module>
    from .quantize import Quantizer
  File "/Users/xdriyad/PycharmProjects/Resnet/env/lib/python3.6/site-packages/torch/quantization/fx/quantize.py", line 2, in <module>
    from torch._fx import (
  File "/Users/xdriyad/PycharmProjects/Resnet/env/lib/python3.6/site-packages/torch/_fx/__init__.py", line 89, in <module>
    from .symbolic_trace import symbolic_trace, Tracer
  File "/Users/xdriyad/PycharmProjects/Resnet/env/lib/python3.6/site-packages/torch/_fx/symbolic_trace.py", line 9, in <module>
    from .proxy import Proxy, _create_proxy, TracerBase
  File "/Users/xdriyad/PycharmProjects/Resnet/env/lib/python3.6/site-packages/torch/_fx/proxy.py", line 7, in <module>
    from typing import Tuple, Dict, Optional, Iterable, NoReturn, Any, Union, Callable
ImportError: cannot import name 'NoReturn'


title: Optimizing a neural network with a multi-task objective in Pytorch

body:
In deep learning, you typically have an objective (say, image recognition), that you wish to optimize. In my field (natural language processing), though, we've seen a rise of multitask training. For instance, in next sentence prediction and sentence classification in a single system.

I understand how to build the forward pass, e.g. for a classification task (obj1) and a regression task (obj2)

class Net(nn.Module):
    def __init__():
        super().__init__()
        self.linear = Linear(300, 200)
        self.obj1 = Linear(200, 5)
        self.obj2 = Linear(200, 1)

    def forward(inputs):
        out = self.linear(inputs)
        out_obj1 = self.obj1(out)
        out_obj2 = self.obj2(out)
        return out_obj1, out_obj2

But the question then becomes, how does one optimize this. Do you call a backward pass over both losses separately? Or do you reduce them to a single loss (e.g. sum, average)? Is there an approach that is typically used for multi-task learning?

And to follow up on that, perhaps one could even argue that the parameters of the separate layers need different optimizers. In such case, the losses must be dealt with separately, I presume.


title: padding and attention mask does not work as intended in batch input in GPT language model

body:
The following code is without batch:

from transformers import GPT2LMHeadModel, GPT2Tokenizer
import torch

tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
model = GPT2LMHeadModel.from_pretrained('gpt2')
model.eval()
context=torch.tensor([tokenizer.encode("This is")])
output, past = model(context)
token = torch.argmax(output[..., -1, :])
print(tokenizer.decode(token.item()))

output: ' a'

This is working fine. Now, I extended this to batch setting:

from transformers import GPT2LMHeadModel, GPT2Tokenizer
import torch

tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
model = GPT2LMHeadModel.from_pretrained('gpt2')
model.eval()

context=[torch.tensor(tokenizer.encode("This is ")),torch.tensor(tokenizer.encode("Hello How are "))]
context=pad_sequence(context,batch_first=True)

mask=torch.tensor([[1,1,0],[1,1,1]])
output, past = model(context,attention_mask=mask)
token = torch.argmax(output[..., -1, :],dim=1)
tokenizer.decode(token)

output: '\n you'

Here \n is next token for the first context and you is next token for second context of the batch.
But The expected next token for the first context is a, since all the settings are same. Furthermore, if you reduce the second context to 2 token you will get a in this batch setting. So clearly, model can not understand the padding.
Also, the attention mask does not work. Because,
after padding the next token of sequence this is is 0 (zero). And according to the attention mask ([1,1,0]), this zero should be avoided and only the tokens this and is should be attended. The proofs that this attention masking is not working are:

Use attention mask [1,1,1], that means attend even on the padding zero, you get the same output
which is \n.
Use the the string this is!. Here ! has the zero index in the vocabulary matrix. Again you get the same output which is \n.

Only time, it is possible to get desirable output is without the batch settings and attention mask ( now it seems, it does not matter because it has no effect anyway)

Then I found this, which suggested to use pad_token. So I used like following:

from transformers import GPT2LMHeadModel, GPT2Tokenizer
import torch
from torch.nn.utils.rnn import pad_sequence

tokenizer = GPT2Tokenizer.from_pretrained("gpt2",pad_token="<PAD>")
model = GPT2LMHeadModel.from_pretrained('gpt2')
model.eval()

context=[torch.tensor(tokenizer.encode("This is <PAD> ")),torch.tensor(tokenizer.encode("Hello How are"))]
context=torch.stack(context)
print(context)
mask=torch.tensor([[1,1,0],[1,1,1]])

output, past = model(context,attention_mask=mask)
token = torch.argmax(output[..., -1, :],dim=1)
tokenizer.decode(token)

output: 'The you'

Here The is next token for the first context and you is next token for second context of the batch. This is also not working. Because The is not expected for the first context.

How do I use variable length sequence in batch setting in gpt/gpt2 model?


title: About cosine similarity, how to choose the loss function and the network(I have two plans)

body:
Sorry I have no clue, I don't know where to find a solution.
I'm using two networks to construct two embeddings，I have binary target to indicate whether embeddingA and embeddingB "match" or not(1 or -1).
The dataset like this:
embA0 embB0 1.0
embA1 embB1 -1.0
embA2 embB2 1.0
...

I hope to use cosine similarity to get classification results.
But I feel confused when choosing the loss function, the two networks that generate embeddings are trained separately, now I can think of two options as follows:
Plan 1:
Construct the 3rd network, use embeddingA and embeddingB as the input of nn.cosinesimilarity() to calculate the final result (should be probability in [-1,1] ), and then select a two-category loss function.
(Sorry, I dont know which loss function to choose.)
class cos_Similarity(nn.Module):
    def __init__(self):
        super(cos_Similarity,self).__init__()
        cos=nn.CosineSimilarity(dim=2)
        embA=generator_A()
        embB=generator_B()

    def forward(self,a,b):
        output_a=embA(a)
        output_b=embB(b)
        return cos(output_a,output_b)
loss_func=nn.CrossEntropyLoss()

y=cos_Similarity(a,b)
loss=loss_func(y,target)
acc=np.int64(y>0)

Plan 2:
The two Embeddings as the output, then use nn.CosineEmbeddingLoss() as loss function, when I calculate the accuracy, I use nn.Cosinesimilarity() to output the result(probability in [-1,1]).
output_a=embA(a)
output_b=embB(b)

cos=nn.CosineSimilarity(dim=2)
loss_function = torch.nn.CosineEmbeddingLoss()

loss=loss_function(output_a,output_b,target)
acc=cos(output_a,output_b)

I really need help. How do I make a choice? Why? Or I can only make a choice for me through experimental results.
Thank you very much!
###############################addition

def train_func(train_loss_list):

    train_data=load_data('train')
    trainloader = DataLoader(train_data, batch_size=BATCH_SIZE)

    cos_smi=nn.CosineSimilarity(dim=2)
    train_loss = 0

    for step,(a,b,target) in enumerate(trainloader):

        try:
            optimizer.zero_grad()

            output_a = model_A(a) #generate embA
            output_b = model_B(b) #generate embB

            acc=cos_smi(output_a,output_b)

            loss = loss_fn(output_a,output_b, target.unsqueeze(dim=1))

            train_loss += loss.item()

            loss.backward()

            optimizer.step()

            train_loss_list.append(loss)

            if step%10==0:
                print('train:',step,'step','loss:',loss,'acc',acc)

        except Exception as e:
            print('train:',step,'step')
            print(repr(e))

    return train_loss_list,train_loss/len(trainloader)


title: Pytorch can't convert np.ndarray of type numpy.object

body:
I am trying to create a PyTorch data-loader with variable image size. Here is a snippet of my code

def get_imgs(path_to_imgs):

    imgs = []
    for path in path_to_imgs:

        imgs.append(cv2.imread(path))

    imgs = np.asarray(imgs)

    return imgs

The function above takes a list of paths and loads the images from the path to the list 'imgs'. BTW the images are not equal-sized. The list looks like imgs = [NumPy array, NumPy array ....]. However, when I convert the list to np.asarray it turns the list into dtype = object.

This is my dataloader class

class Dataset(torch.utils.data.Dataset):

  def __init__(self, path_to_imgs, path_to_label):
        'Initialization'
        self.path_to_imgs = path_to_imgs
        self.path_to_label = path_to_label

        self.imgs = get_imgs(path_to_imgs)
        self.label = get_pts(path_to_label)

        self.imgs = torch.Tensor(self.imgs)             **Error here
        # self.imgs = torch.from_numpy(self.imgs)       ** I tried this as well. Same error

        self.label = torch.Tensor(self.label)

        self.len = len(self.imgs)

  def __len__(self):
        'Denotes the total number of samples'
        return self.len

  def __getitem__(self, index):

        return self.imgs, self.label

When I try to convert the list of images to tensor** it fails giving the following error

  can't convert np.ndarray of type numpy.object_. The only supported types are: float64, float32, float16, int64, int32, int16, int8, uint8, and bool.

I have looked similar questions here and here but they were not helpful.


title: How to transfer the following tensorflow code into pytorch

body:
I want to re-implement the word embedding here

here is the original tensorflow code  (version: 0.12.1)

import tensorflow as tf

class Network(object):
    def __init__(
        self, user_length,item_length, num_classes, user_vocab_size,item_vocab_size,fm_k,n_latent,user_num,item_num,
        embedding_size, filter_sizes, num_filters, l2_reg_lambda=0.0,l2_reg_V=0.0):

 # Skip the embedding
    pooled_outputs_u = []

    for i, filter_size in enumerate(filter_sizes):
        with tf.name_scope("user_conv-maxpool-%s" % filter_size):
            # Convolution Layer
            filter_shape = [filter_size, embedding_size, 1, num_filters]
            W = tf.Variable(tf.truncated_normal(filter_shape, stddev=0.1), name="W")
            b = tf.Variable(tf.constant(0.1, shape=[num_filters]), name="b")
            conv = tf.nn.conv2d(
                self.embedded_users,
                W,
                strides=[1, 1, 1, 1],
                padding="VALID",
                name="conv")
            # Apply nonlinearity
            h = tf.nn.relu(tf.nn.bias_add(conv, b), name="relu")
            # Maxpooling over the outputs
            pooled = tf.nn.max_pool(
                h,
                ksize=[1, user_length - filter_size + 1, 1, 1],
                strides=[1, 1, 1, 1],
                padding='VALID',
                name="pool")
            pooled_outputs_u.append(pooled)
    num_filters_total = num_filters * len(filter_sizes)
    self.h_pool_u = tf.concat(3,pooled_outputs_u)
    self.h_pool_flat_u = tf.reshape(self.h_pool_u, [-1, num_filters_total])

    with tf.name_scope("dropout"):
        self.h_drop_u = tf.nn.dropout(self.h_pool_flat_u, 1.0)
        self.h_drop_i= tf.nn.dropout(self.h_pool_flat_i, 1.0)
    with tf.name_scope("get_fea"):
        Wu = tf.get_variable(
            "Wu",
            shape=[num_filters_total, n_latent],
            initializer=tf.contrib.layers.xavier_initializer())
        bu = tf.Variable(tf.constant(0.1, shape=[n_latent]), name="bu")
        self.u_fea=tf.matmul(self.h_drop_u, Wu) + bu
        #self.u_fea = tf.nn.dropout(self.u_fea,self.dropout_keep_prob)
        Wi = tf.get_variable(
            "Wi",
            shape=[num_filters_total, n_latent],
            initializer=tf.contrib.layers.xavier_initializer())
        bi = tf.Variable(tf.constant(0.1, shape=[n_latent]), name="bi")
        self.i_fea = tf.matmul(self.h_drop_i, Wi) + bi
        #self.i_fea=tf.nn.dropout(self.i_fea,self.dropout_keep_prob)

    with tf.name_scope('fm'):
        self.z=tf.nn.relu(tf.concat(1,[self.u_fea,self.i_fea]))

        #self.z=tf.nn.dropout(self.z,self.dropout_keep_prob)

        WF1=tf.Variable(
                tf.random_uniform([n_latent*2, 1], -0.1, 0.1), name='fm1')
        Wf2=tf.Variable(
            tf.random_uniform([n_latent*2, fm_k], -0.1, 0.1), name='fm2')
        one=tf.matmul(self.z,WF1)

        inte1=tf.matmul(self.z,Wf2)
        inte2=tf.matmul(tf.square(self.z),tf.square(Wf2))

        inter=(tf.square(inte1)-inte2)*0.5

        inter=tf.nn.dropout(inter,self.dropout_keep_prob)

        inter=tf.reduce_sum(inter,1,keep_dims=True)
        print inter
        b=tf.Variable(tf.constant(0.1), name='bias')

And here is the pytorch version 1.0 that I try

class Network(torch.nn.Module):
    def __init__(
        self, user_length,item_length, num_classes, user_vocab_size,item_vocab_size,fm_k,n_latent,user_num,item_num,
        embedding_size, filter_sizes, num_filters, l2_reg_lambda=0.0,l2_reg_V=0.0):

        pooled_outputs_u = []

    def forward():

I mainly have the trouble with the convolutional layer tranforming.

Pytorch is much easy to use since we can skip lots of W and b definition. Is there some one can help me with the rest? Thanks


title: PyTorch Slow Batch matrix multiplication on GPU

body:
I am using Batch Matrix Multiplication on 2 3d tensors of sizes (100 , 128 , 128 ) each.

import torch
a = torch.randn(100,128,128)
b = torch.randn(100,128,128)

import time

t0 = time.time()
torch.bmm(a,b)
print(time.time() - t0)

  0.03233695030212402

Now if i do the same thing on GPU it takes a lot longer

a = a.cuda()
b = b.cuda()
t0 = time.time()
torch.bmm(a,b)
print(time.time() - t0)

  30.574532985687256

Why does it take so long to solve on GPU?

I have a GTX 1050 Ti 4GB

And processor core i3-6100 3.7Ghz


title: Why aren't torch.nn.Parameter listed when net is printed?

body:
I recently had to construct a module that required a tensor to be included. While back propagation worked perfectly using torch.nn.Parameter, it did not show up when printing the net object. Why isn't this parameter included in contrast to other modules like layer? (Shouldn't it behave just like layer?)

import torch
import torch.nn as nn

class MyNet(torch.nn.Module):
    def __init__(self):
        super(MyNet, self).__init__()
        self.layer = nn.Linear(10, 10)
        self.parameter = torch.nn.Parameter(torch.zeros(10,10, requires_grad=True))

net = MyNet()
print(net)

Output:

MyNet(
  (layer): Linear(in_features=10, out_features=10, bias=True)
)


title: Pytorch: How can I find indices of first nonzero element in each row of a 2D tensor?

body:
I have a 2D tensor with some nonzero element in each row like this:

import torch
tmp = torch.tensor([[0, 0, 1, 0, 1, 0, 0],
                    [0, 0, 0, 1, 1, 0, 0]], dtype=torch.float)

I want a tensor containing the index of first nonzero element in each row:

indices = tensor([2],
                 [3])

How can I calculate it in Pytorch?


title: Pytorch w/ GPU on Docker Container Error - no CUDA-capable device is detected

body:
I am trying to use Pytorch with a GPU on my Docker Container.

1. On the Host -
I have nvidia-docker installed, CUDA Driver etc

Here is the nvidia-smi output from host:

    Fri Mar 20 04:29:49 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.64.00    Driver Version: 440.64.00    CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K80           Off  | 00000000:00:04.0 Off |                    0 |
| N/A   33C    P8    28W / 149W |     16MiB / 11441MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      1860      G   /usr/lib/xorg/Xorg                            15MiB |
+-----------------------------------------------------------------------------+

2. On the Docker Container (Dockerfile for app - Docker Compose File below) -

FROM ubuntu:latest
FROM dsksd/pytorch:0.4
#FROM nvidia/cuda:10.1-base-ubuntu18.04
#FROM nablascom/cuda-pytorch
#FROM nvidia/cuda:10.0-base

RUN apt-get update -y --fix-missing
RUN apt-get install -y python3-pip python3-dev build-essential
RUN apt-get install -y sudo curl
#RUN sudo apt-get install -y nvidia-container-toolkit
#RUN apt-get install -y curl python3.7 python3-pip python3.7-dev python3.7-distutils build-essential
#RUN apt-get install -y curl
#RUN apt-get install -y sudo
#RUN curl -O http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/cuda-repo-ubuntu1604_10.0.130-1_amd64.deb
#RUN sudo dpkg -i cuda-repo-ubuntu1604_10.0.130-1_amd64.deb
#RUN sudo apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/7fa2af80.pub
#RUN sudo apt-get install cuda -y
#----------
# Add the package repositories
#RUN distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
#RUN curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
#RUN curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
#RUN sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
#RUN sudo systemctl restart docker
ENV NVIDIA_VISIBLE_DEVICES all
ENV NVIDIA_DRIVER_CAPABILITIES compute,utility
ENV LD_LIBRARY_PATH $LD_LIBRARY_PATH:/usr/local/cuda-10.1/compat/
ENV PYTHONPATH $PATH
#----------
ENV LC_ALL=mylocale.utf8
COPY . /app
WORKDIR /app
RUN pip3 install -r requirements.txt
ENTRYPOINT ["python3"]
EXPOSE 5000
CMD ["hook.py"]

When I try running my code on the GPU I run into:

>>> torch.cuda.current_device()
THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=50 error=100 : no CUDA-capable device is detected
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/dist-packages/torch/cuda/__init__.py", line 386, in current_device
    _lazy_init()
  File "/usr/local/lib/python3.6/dist-packages/torch/cuda/__init__.py", line 193, in _lazy_init
    torch._C._cuda_init()
RuntimeError: cuda runtime error (100) : no CUDA-capable device is detected at /pytorch/aten/src/THC/THCGeneral.cpp:50

I invoke the container using : docker-compose up --build

Here is my docker-compose.yaml file:

version: '3.6'
services:
  rdb:
    image: mysql:5.7
    #restart: always
    environment:
      MYSQL_DATABASE: 'c_rdb'
      MYSQL_USER: 'user'
      MYSQL_PASSWORD: 'password'
      MYSQL_ROOT_PASSWORD: '123123'
    #ports:
    #  - '3306:3306'
    #expose:
    #  - '3306'
    volumes:
      - rdb-data:/var/lib/mysql
      - ./init-db/init.sql:/docker-entrypoint-initdb.d/init.sql
  mongo:
    image: mongo
    #restart: always
    environment:
      MONGO_INITDB_ROOT_USERNAME: root
      MONGO_INITDB_ROOT_PASSWORD: 12312323
      MONGO_INITDB_DATABASE: chronicler_ndb
    volumes:
      - ndb-data:/data/db
      - ./init-db/init.js:/docker-entrypoint-initdb.d/init.js
    ports:
      - '27017-27019:27017-27019'
  mongo-express:
    image: mongo-express
    #restart: always
    depends_on:
        - mongo
        - backend
    ports:
      - 8081:8081
    environment:
      ME_CONFIG_MONGODB_ADMINUSERNAME: rooer
      ME_CONFIG_MONGODB_ADMINPASSWORD: 123123
  redis:
    image: redis:latest
    command: ["redis-server", "--appendonly", "yes"]
    hostname: redis
    #ports:
    #  - "6379:6379"
    volumes:
      - cache-data:/data
  backend:
    build: ./app
    ports:
     - "5000:5000"
    volumes:
     - backend-data:/code
    links:
     - rdb
     - redis

volumes:
  rdb-data:
    name: c-relational-data
  ndb-data:
    name: c-nosql-data
  cache-data:
    name: redis-data
  backend-data:
    name: backend-engine


title: Torch not compiled with CUDA enabled Windows 10

body:
I try to run my code with pytorch 10.2, but I get the assertion error described in the title. At first I installed cuda with version 11. Then I uninstalled it and installed version 10.2. Also, I used pytorch before without cuda and installed it using command: "conda install pytorch torchvision cpuonly -c pytorch"
Today after installing Cuda, I ran "conda install pytorch torchvision cudatoolkit=10.2 -c pytorch"
However,  when I run command: "torch.cuda.is_available()", the output is False.
After running "nvcc --version" I can see that Cuda 10.2 is installed: "Cuda compilation tools, release 10.2, V10.2.89". Any idea how to fix this assertion error?
Other answers describe that I should install pytorch with cuda, but I already did that and I still get the same error. Do I need to enable cuda with pytorch somehow manually?


title: .grad() returns None in pytorch

body:
I am trying to write a simple script for parameter estimation (where parameters are weights here). I am facing problem when .grad() returns None. I have gone through this  and this link also and understood the concept both theoretically and practically. For me following script should work but unfortunately, it is not working.

My 1st attempt: Following script is my first attempt

alpha_xy = torch.tensor(3.7, device=device, dtype=torch.float, requires_grad=True)
beta_y = torch.tensor(1.5, device=device, dtype=torch.float, requires_grad=True)
alpha0 = torch.tensor(1.1, device=device, dtype=torch.float, requires_grad=True)
alpha_y = torch.tensor(0.9, device=device, dtype=torch.float, requires_grad=True)
alpha1 = torch.tensor(0.1, device=device, dtype=torch.float, requires_grad=True)
alpha2 = torch.tensor(0.9, device=device, dtype=torch.float, requires_grad=True)
alpha3 = torch.tensor(0.001, device=device, dtype=torch.float, requires_grad=True)

learning_rate = 1e-4
total_loss = []

for epoch in tqdm(range(500)):
    loss_1 = 0
    for j in range(x_train.size(0)):
        input = x_train[j:j+1]
        target = y_train[j:j+1]
        input = input.to(device,non_blocking=True)
        target = target.to(device,non_blocking=True)
        x_dt = gamma*input[0][0] + \
               alpha_xy*input[0][0]*input[0][2] + \
               alpha1*input[0][0]

        y0_dt = beta_y*input[0][0] + \
                alpha2*input[0][1]

        y_dt = alpha0*input[0][1] + \
               alpha_y*input[0][2] + \
               alpha3*input[0][0]*input[0][2]

        pred = torch.tensor([[x_dt],
                             [y0_dt],
                             [y_dt]],device=device

                                   )
        loss = (pred - target).pow(2).sum()
        loss_1 += loss
        loss.backward()
        print(pred.grad, x_dt.grad, gamma.grad)

Above code throws an error message

element 0 of tensors does not require grad and does not have a grad_fn

at line loss.backward()

My Attempt 2: Improvement in 1st attempt is as follows:

gamma = torch.tensor(2.0, device=device, dtype=torch.float, requires_grad=True)
alpha_xy = torch.tensor(3.7, device=device, dtype=torch.float, requires_grad=True)
beta_y = torch.tensor(1.5, device=device, dtype=torch.float, requires_grad=True)
alpha0 = torch.tensor(1.1, device=device, dtype=torch.float, requires_grad=True)
alpha_y = torch.tensor(0.9, device=device, dtype=torch.float, requires_grad=True)
alpha1 = torch.tensor(0.1, device=device, dtype=torch.float, requires_grad=True)
alpha2 = torch.tensor(0.9, device=device, dtype=torch.float, requires_grad=True)
alpha3 = torch.tensor(0.001, device=device, dtype=torch.float, requires_grad=True)

learning_rate = 1e-4
total_loss = []

for epoch in tqdm(range(500)):
    loss_1 = 0
    for j in range(x_train.size(0)):
        input = x_train[j:j+1]
        target = y_train[j:j+1]
        input = input.to(device,non_blocking=True)
        target = target.to(device,non_blocking=True)
        x_dt = gamma*input[0][0] + \
               alpha_xy*input[0][0]*input[0][2] + \
               alpha1*input[0][0]

        y0_dt = beta_y*input[0][0] + \
                alpha2*input[0][1]

        y_dt = alpha0*input[0][1] + \
               alpha_y*input[0][2] + \
               alpha3*input[0][0]*input[0][2]

        pred = torch.tensor([[x_dt],
                             [y0_dt],
                             [y_dt]],device=device,
                                   dtype=torch.float,
                                   requires_grad=True)
        loss = (pred - target).pow(2).sum()
        loss_1 += loss
        loss.backward()
        print(pred.grad, x_dt.grad, gamma.grad)
#        with torch.no_grad():
#            gamma -= leraning_rate * gamma.grad

Now the script is working but except pred.gred other two return None.

I want to update all the parameters after computing loss.backward() and update them but it is not happening due to None. Can anyone suggest me how to improve this script? Thanks.


title: Visualizing the output of intermediate layers of cnn in pytorch

body:
I'm trying to visualize the output of the intermediate layers of the VGG19 network, from the torchvision module, specifically the layer, conv4_2.
I've extracted the output in a tensor of shape [1, 512, 50, 50].
But how do I visualize an image with 512 channels?


title: Pytorch DataLoader error when using num_worker larger than 0

body:
Unable to use Dataloader with setting num_worker larger than zero.

My torch version is 1.0.0.

My Code:

class InputData(Dataset):
    '''read data'''
    def __init__(self,train_serial_nums
                im_dir = TRAIN_PATH+'/img/', mask_dir = TRAIN_PATH+'/label/'):
        self.train_serial_nums = train_serial_nums
        self.im_dir = im_dir
        self.mask_dir = mask_dir

    def open_as_gray(self,serial_num):
        # open an image and convert to gray
        serial_num = str(serial_num)
        im_name = "img_"+serial_num+".jpg"
        mask_name = "label_"+serial_num+".png"

        im = np.asarray(Image.open(os.path.join(self.im_dir, im_name)))
        im = im/255

        _mask = Image.open(os.path.join(self.mask_dir, mask_name)).convert("L")
        _mask = np.expand_dims(_mask, axis=-1)
        mask_0 = np.where(np.logical_or(_mask == 60, _mask == 180), 1, 0)
        mask_1 = np.where(np.logical_or(_mask == 120, _mask == 180), 1, 0)
        mask = np.concatenate([mask_0, mask_1], axis=-1)

        return im.transpose(2,0,1), mask.transpose(2,0,1), im_name, mask_name

    def __len__(self):
        'Denotes the total number of samples'
        return len(self.train_serial_nums)

    def __getitem__(self, index):
        'Generates one sample of data'

        trainId = self.train_serial_nums[index]
        # Load data and get label
        trainImg, trainMask,_,_ = self.open_as_gray(trainId)
        return trainImg, trainMask

And I use DataLoader` for parallel computing.

train_serial_nums = np.arange(40000, 50000)
train_set = InputData(train_serial_nums)
    train_generator = DataLoader(
        train_set, batch_size=32,
        num_workers=1)

Error information:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<timed exec> in <module>

/data/anonym2/anaconda3/envs/tensorflow/lib/python3.6/site-packages/torch/utils/data/dataloader.py in __next__(self)
    629         while True:
    630             assert (not self.shutdown and self.batches_outstanding > 0)
--> 631             idx, batch = self._get_batch()
    632             self.batches_outstanding -= 1
    633             if idx != self.rcvd_idx:

/data/anonym2/anaconda3/envs/tensorflow/lib/python3.6/site-packages/torch/utils/data/dataloader.py in _get_batch(self)
    608             # need to call `.task_done()` because we don't use `.join()`.
    609         else:
--> 610             return self.data_queue.get()
    611
    612     def __next__(self):

/data/anonym2/anaconda3/envs/tensorflow/lib/python3.6/multiprocessing/queues.py in get(self, block, timeout)
     92         if block and timeout is None:
     93             with self._rlock:
---> 94                 res = self._recv_bytes()
     95             self._sem.release()
     96         else:

/data/anonym2/anaconda3/envs/tensorflow/lib/python3.6/multiprocessing/connection.py in recv_bytes(self, maxlength)
    214         if maxlength is not None and maxlength < 0:
    215             raise ValueError("negative maxlength")
--> 216         buf = self._recv_bytes(maxlength)
    217         if buf is None:
    218             self._bad_message_length()

/data/anonym2/anaconda3/envs/tensorflow/lib/python3.6/multiprocessing/connection.py in _recv_bytes(self, maxsize)
    405
    406     def _recv_bytes(self, maxsize=None):
--> 407         buf = self._recv(4)
    408         size, = struct.unpack("!i", buf.getvalue())
    409         if maxsize is not None and size > maxsize:

/data/anonym2/anaconda3/envs/tensorflow/lib/python3.6/multiprocessing/connection.py in _recv(self, size, read)
    377         remaining = size
    378         while remaining > 0:
--> 379             chunk = read(handle, remaining)
    380             n = len(chunk)
    381             if n == 0:

/data/anonym2/anaconda3/envs/tensorflow/lib/python3.6/site-packages/torch/utils/data/dataloader.py in handler(signum, frame)
    272         # This following call uses `waitid` with WNOHANG from C side. Therefore,
    273         # Python can still get and update the process status successfully.
--> 274         _error_if_any_worker_fails()
    275         if previous_handler is not None:
    276             previous_handler(signum, frame)

RuntimeError: DataLoader worker (pid 5235) exited unexpectedly with exit code 1. Details are lost due to multiprocessing.
Rerunning with num_workers=0 may give better error trace.

If I set num_workers=0 , the code runs well but very slowly.
(Delete No more detailsssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss)


title: CUDA error: CUBLAS_STATUS_ALLOC_FAILED when running loss.backward()

body:
I have a simple model, and I'm trying to run a classification task.
The following is my model
num_epochs = 10
batch_size = 64
learning_rate = 0.001

input_size =  44
hidden_size = 500
num_classes = 2

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

class Net(nn.Module):
    def __init__(self, input_size, hidden_size, num_classes):
        super(NeuralNet, self).__init__()
        self.input_size = input_size
        self.l1 = nn.Linear(input_size, hidden_size)
        self.relu = nn.ReLU()
        self.l2 = nn.Linear(hidden_size, num_classes)

    def forward(self, x):
        out = self.l1(x)
        out = self.relu(out)
        out = self.l2(out)
        return out

the error happens when running till loss.back()
    for epoch in range(num_epochs):
        for i, (features,labels) in enumerate(train_loader):

            features = features.view(-1, 44)

            labels = torch.tensor(labels, dtype=torch.long, device=device)

            outputs = model(features)

            print(outputs.shape)
            print(labels.shape)

            loss = criterion(outputs, labels)

            optimizer.zero_grad()
            loss.backward()  # Error happens
            optimizer.step()

the shape of output is (64,2)
the shape of labels is (64)
The error is like
 File "dl.py", line 218, in cnn_setup
    loss.backward()
  File "F:\Python3.7.7\lib\site-packages\torch\tensor.py", line 185, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "F:\Python3.7.7\lib\site-packages\torch\autograd\__init__.py", line 127, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: CUDA error: CUBLAS_STATUS_ALLOC_FAILED when calling `cublasCreate(handle)`
Exception raised from createCublasHandle at ..\aten\src\ATen\cuda\CublasHandlePool.cpp:8 (most recent call first):
00007FF8DB5E75A200007FF8DB5E7540 c10.dll!c10::Error::Error [<unknown file> @ <unknown line number>]
00007FF863A5AEA800007FF863A59E70 torch_cuda.dll!at::cuda::getCurrentCUDASparseHandle [<unknown file> @ <unknown line number>]
00007FF863A5A7D800007FF863A59E70 torch_cuda.dll!at::cuda::getCurrentCUDASparseHandle [<unknown file> @ <unknown line number>]
00007FF863A5B66700007FF863A5B1A0 torch_cuda.dll!at::cuda::getCurrentCUDABlasHandle [<unknown file> @ <unknown line number>]
00007FF863A5B24700007FF863A5B1A0 torch_cuda.dll!at::cuda::getCurrentCUDABlasHandle [<unknown file> @ <unknown line number>]

...

I found a similar question, but it seems doesn't apply to my situation,
what could be possible solutions to this problem?


title: Conv2d not accepting tensor as input, saying its not tensor

body:
I want to pass a tensor through a convolutional 2 layer. I am not able to execute it as I am getting a type error even though I have converted my numpy array to a tensor.

I tried using tf.convert_to_tensor() to solve this problem. Didn't work

import numpy as np
import tensorflow as tf

class Generator():

  def __init__(self):

    self.conv1 = nn.Conv2d(1, 28, kernel_size=3, stride=1, padding=1)
    self.pool1 = nn.MaxPool2d(kernel_size=3, stride=0, padding=1)

    self.fc1 = nn.Linear(100, 10)
    self.fc2 = nn.Linear(10, 5)

  def forward_pass(self, x):                                                                       #Why do we pass the object itself in every method?

    x = self.conv1(x)
    print(x)
    x = self.pool1(x)
    print(x)

    x = self.fc1(x)
    print(x)
    x = self.fc2(x)
    print(x)

    return x

arr = tf.convert_to_tensor(np.random.random((3,28,28)))

gen = Generator()
gen.forward_pass(arr)

Error message -

TypeError                                 Traceback (most recent call last)

<ipython-input-31-9fa8e764dcdb> in <module>()
      1 gen = Generator()
----> 2 gen.forward_pass(arr)

2 frames

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py in forward(self, input)
    336                             _pair(0), self.dilation, self.groups)
    337         return F.conv2d(input, self.weight, self.bias, self.stride,
--> 338                         self.padding, self.dilation, self.groups)
    339
    340

TypeError: conv2d(): argument 'input' (position 1) must be Tensor, not Tensor


title: Pytorch Implementation for ResNet Based UNet

body:
I want to implement a ResNet based UNet for segmentation (without pre-training). I have referred to this implementation using Keras but my project has been implemented using PyTorch that I am not sure if I have done the correct things.

Keras based implementation
U-net with simple Resnet Blocks

My PyTorch implementation (I am not sure if I am correct ...) Any suggestions will be highly appreciated.

def double_conv(in_channels, out_channels):
return nn.Sequential(
    nn.Conv2d(in_channels, out_channels, 3, padding=1),
    nn.BatchNorm2d(out_channels),
    nn.ReLU(inplace=True),
    nn.Conv2d(out_channels, out_channels, 3, padding=1),
    nn.BatchNorm2d(out_channels),
    nn.ReLU(inplace=True)
)

def root_block(in_channels, out_channels):
return nn.Sequential(
    nn.Conv2d(in_channels, out_channels, 3, padding=1),
    nn.BatchNorm2d(out_channels),
    nn.ReLU(inplace=True),
    nn.Conv2d(out_channels, out_channels, 3, padding=1),
    nn.BatchNorm2d(out_channels),
)

# Define the UNet architecture
class ResNetUNet(nn.Module):

def __init__(self, n_class):
    super().__init__()

    self.dconv_down1 = double_conv(3, 64)
    self.dconv_down11 = root_block(64, 64)
    self.dconv_down2 = double_conv(64, 128)
    self.dconv_down21 = root_block(128, 128)
    self.dconv_down3 = double_conv(128, 256)
    self.dconv_down31 = root_block(256, 256)
    self.dconv_down4 = double_conv(256, 512)
    self.dconv_down41 = root_block(512, 512)

    self.maxpool = nn.MaxPool2d(2)
    self.relu = nn.ReLU(inplace=True)

    self.dconv_up3 = double_conv(256 + 512, 256)
    self.dconv_up31 = root_block(256, 256)
    self.dconv_up2 = double_conv(128 + 256, 128)
    self.dconv_up21 = root_block(128, 128)
    self.dconv_up1 = double_conv(128 + 64, 64)
    self.dconv_up11 = root_block(64, 64)

    self.conv_last = nn.Conv2d(64, n_class, 1)

def forward(self, x):
    conv1 = self.dconv_down1(x)
    x = self.dconv_down11(conv1)
    x += conv1
    x = self.relu(x)
    x = self.maxpool(x)

    conv2 = self.dconv_down2(x)
    x = self.dconv_down21(conv2)
    x += conv2
    x = self.relu(x)
    x = self.maxpool(x)

    conv3 = self.dconv_down3(x)
    x = self.dconv_down31(conv3)
    x += conv3
    x = self.relu(x)
    x = self.maxpool(x)

    conv4 = self.dconv_down4(x)
    x = self.dconv_down41(conv4)
    x += conv4
    x = self.relu(x)

    deconv3 = nn.functional.interpolate(x, scale_factor=2, mode='bilinear', align_corners=True)
    deconv3 = torch.cat([deconv3, conv3], dim=1)
    uconv3 = self.dconv_up3(deconv3)
    x = self.dconv_up31(uconv3)
    x += uconv3
    x = self.relu(x)

    deconv2 = nn.functional.interpolate(x, scale_factor=2, mode='bilinear', align_corners=True)
    deconv2 = torch.cat([deconv2, conv2], dim=1)
    uconv2 = self.dconv_up2(deconv2)
    x = self.dconv_up21(uconv2)
    x += uconv2
    x = self.relu(x)

    deconv1 = nn.functional.interpolate(x, scale_factor=2, mode='bilinear', align_corners=True)
    deconv1 = torch.cat([deconv1, conv1], dim=1)
    uconv1 = self.dconv_up1(deconv1)
    x = self.dconv_up11(uconv1)
    x += uconv1
    x = self.relu(x)

    out = self.conv_last(x)

    return out

Keras based implementation extracted as below:

def convolution_block(x, filters, size, strides=(1,1), padding='same', activation=True):
x = Conv2D(filters, size, strides=strides, padding=padding)(x)
x = BatchNormalization()(x)
if activation == True:
    x = Activation('relu')(x)
return x

def residual_block(blockInput, num_filters=16):
x = Activation('relu')(blockInput)
x = BatchNormalization()(x)
x = convolution_block(x, num_filters, (3,3) )
x = convolution_block(x, num_filters, (3,3), activation=False)
x = Add()([x, blockInput])
return x

# Build model
def build_model(input_layer, start_neurons, DropoutRatio = 0.5):
# 101 -> 50
conv1 = Conv2D(start_neurons * 1, (3, 3), activation=None, padding="same")(input_layer)
conv1 = residual_block(conv1,start_neurons * 1)
conv1 = residual_block(conv1,start_neurons * 1)
conv1 = Activation('relu')(conv1)
pool1 = MaxPooling2D((2, 2))(conv1)
pool1 = Dropout(DropoutRatio/2)(pool1)

# 50 -> 25
conv2 = Conv2D(start_neurons * 2, (3, 3), activation=None, padding="same")(pool1)
conv2 = residual_block(conv2,start_neurons * 2)
conv2 = residual_block(conv2,start_neurons * 2)
conv2 = Activation('relu')(conv2)
pool2 = MaxPooling2D((2, 2))(conv2)
pool2 = Dropout(DropoutRatio)(pool2)

# 25 -> 12
conv3 = Conv2D(start_neurons * 4, (3, 3), activation=None, padding="same")(pool2)
conv3 = residual_block(conv3,start_neurons * 4)
conv3 = residual_block(conv3,start_neurons * 4)
conv3 = Activation('relu')(conv3)
pool3 = MaxPooling2D((2, 2))(conv3)
pool3 = Dropout(DropoutRatio)(pool3)

# 12 -> 6
conv4 = Conv2D(start_neurons * 8, (3, 3), activation=None, padding="same")(pool3)
conv4 = residual_block(conv4,start_neurons * 8)
conv4 = residual_block(conv4,start_neurons * 8)
conv4 = Activation('relu')(conv4)
pool4 = MaxPooling2D((2, 2))(conv4)
pool4 = Dropout(DropoutRatio)(pool4)

# Middle
convm = Conv2D(start_neurons * 16, (3, 3), activation=None, padding="same")(pool4)
convm = residual_block(convm,start_neurons * 16)
convm = residual_block(convm,start_neurons * 16)
convm = Activation('relu')(convm)

# 6 -> 12
deconv4 = Conv2DTranspose(start_neurons * 8, (3, 3), strides=(2, 2), padding="same")(convm)
uconv4 = concatenate([deconv4, conv4])
uconv4 = Dropout(DropoutRatio)(uconv4)

uconv4 = Conv2D(start_neurons * 8, (3, 3), activation=None, padding="same")(uconv4)
uconv4 = residual_block(uconv4,start_neurons * 8)
uconv4 = residual_block(uconv4,start_neurons * 8)
uconv4 = Activation('relu')(uconv4)

# 12 -> 25
#deconv3 = Conv2DTranspose(start_neurons * 4, (3, 3), strides=(2, 2), padding="same")(uconv4)
deconv3 = Conv2DTranspose(start_neurons * 4, (3, 3), strides=(2, 2), padding="valid")(uconv4)
uconv3 = concatenate([deconv3, conv3])
uconv3 = Dropout(DropoutRatio)(uconv3)

uconv3 = Conv2D(start_neurons * 4, (3, 3), activation=None, padding="same")(uconv3)
uconv3 = residual_block(uconv3,start_neurons * 4)
uconv3 = residual_block(uconv3,start_neurons * 4)
uconv3 = Activation('relu')(uconv3)

# 25 -> 50
deconv2 = Conv2DTranspose(start_neurons * 2, (3, 3), strides=(2, 2), padding="same")(uconv3)
uconv2 = concatenate([deconv2, conv2])

uconv2 = Dropout(DropoutRatio)(uconv2)
uconv2 = Conv2D(start_neurons * 2, (3, 3), activation=None, padding="same")(uconv2)
uconv2 = residual_block(uconv2,start_neurons * 2)
uconv2 = residual_block(uconv2,start_neurons * 2)
uconv2 = Activation('relu')(uconv2)

# 50 -> 101
#deconv1 = Conv2DTranspose(start_neurons * 1, (3, 3), strides=(2, 2), padding="same")(uconv2)
deconv1 = Conv2DTranspose(start_neurons * 1, (3, 3), strides=(2, 2), padding="valid")(uconv2)
uconv1 = concatenate([deconv1, conv1])

uconv1 = Dropout(DropoutRatio)(uconv1)
uconv1 = Conv2D(start_neurons * 1, (3, 3), activation=None, padding="same")(uconv1)
uconv1 = residual_block(uconv1,start_neurons * 1)
uconv1 = residual_block(uconv1,start_neurons * 1)
uconv1 = Activation('relu')(uconv1)

uconv1 = Dropout(DropoutRatio/2)(uconv1)
output_layer = Conv2D(1, (1,1), padding="same", activation="sigmoid")(uconv1)

return output_layer


title: How to use PyTorch to calculate the gradients of outputs w.r.t. the inputs in a neural network?

body:
I have a trained network. And I want to calculate the gradients of outputs w.r.t. the inputs. By querying the PyTorch Docs, torch.autograd.grad may be useful.
So, I use the following code:

    x_test = torch.randn(D_in,requires_grad=True)
    y_test = model(x_test)
    d = torch.autograd.grad(y_test, x_test)[0]

model is the neural network. x_test is the input of size D_in and y_test is a scalar output.
I want to compare the calculated result with numerical difference by scipy.misc.derivative.
So, I calculated the partial derivate by setting a index.

    idx = 3
    x_test = torch.randn(D_in,requires_grad=True)
    y_test = model(x_test)
    print(x_test[idx].item())
    d = torch.autograd.grad(y_test, x_test)[0]
    print(d[idx].item())
    def fun(x):
        x_input = x_test.detach()
        x_input[idx] = x
        with torch.no_grad():
            y = model(x_input)
        return y.item()
    x0 = x_test[idx].item()
    print(x0)
    print(derivative(fun, x0, dx=1e-6))

But I got totally different results.
The gradient calculated by torch.autograd.grad is -0.009522666223347187,
while that by scipy.misc.derivative is -0.014901161193847656.

Is there anything wrong about the calculation? Or I use torch.autograd.grad wrongly?


title: Masking and computing loss for a padded batch sent through an RNN with a linear output layer in pytorch

body:
Although a typical use case, I can't find one simple and clear guide on what is the canonical way to compute loss on a padded minibatch in pytorch, when sent through an RNN.

I think a canonical pipeline could be:

1) The pytorch RNN expects a padded batch tensor of shape: (max_seq_len, batch_size, emb_size)

2) So we give an Embedding layer for example this tensor:

tensor([[1, 1],
        [2, 2],
        [3, 9]])

9 is the padding index. Batch size is 2. The Embedding layer will make it to be of shape (max_seq_len, batch_size, emb_size). The sequences in the batch are in descending order, so we can pack it.

3) We apply pack_padded_sequence, we apply the RNN, finally we apply pad_packed_sequence. We have at this point (max_seq_len, batch_size, hidden_size)

4) Now we apply the linear output layer on the result and let's say the log_softmax. So at the end we have a tensor for a batch of scores of shape: (max_seq_len, batch_size, linear_out_size)

How should I compute the loss from here, masking out the padded part (with an arbitrary target)? Thanks!


title: from torchvision.datasets.vision import VisionDataset ImportError: No module named vision

body:
I am new to pytorch, and I am trying to follow guides (like this one) that start with:

from torchvision.datasets.vision import VisionDataset

or

from .vision import VisionDataset

When I try to run it I get an error:

ImportError: No module named vision

In addition, It seems that torchvision.datasets.vision is not mentioned in any documentation.
Please help.
Thanks.


title: How to adapt the gpu batch size during training?

body:
I found surprising that I could not find any resources online on how to dynamically adapt the GPU batch size without halting training.

The idea is the following:

1) Have a training script that is (almost) agnostic to the GPU in use. The batch size will dynamically adjust without interference of the user or need for tunning.

2) Still being able to specifying the desired training batch size, even if too big to fit in the biggest known GPU.

For instance, let's say I want to train a model using a batch size of 4096 images, each image 1024x1024. Let's also say that I have access to a server with different NVidea GPUs, but I don't know which one will be assigned to me in advance. (Or that everybody wants to use the biggest GPU and that I am left waiting a long time before it is my term).

I want my training script to find the max batch size (let's say it is 32 images per GPU batch),  and only update the optimizer when all 4096 images have been processed (one training batch = 128 GPU batches).