title: Freeing GPU Memory PyTorch

body:
So, my code is supposed to work as follows: import the images, get the embeddings from ResNet model, use those embeddings in a NN (user created) and run the analysis. ResNet imports the weights (pretrained=True) and requires grad has been set False for all parameters.
I have a lot of images to process, more than my RAM can handle. So, what I do is, I import batches of images, run them through the GPU, get the embeddings of the images, and save the results to some files (I have like 120k images, so I create a file for every 10k images, my RAM can handle that much). I then import each file in a loop and run them through my NN. The reason I am doing this is to save time when running for every epoch (So that I don't have to run ResNet each epoch). The issue I am facing however is that when I save the tensors (ResNet embeddings) to the file, I do it on the GPU since it is much much faster. However, when the saving is done, even though everything is done inside a function, the GPU RAM is not released and as such training cannot continue. I even tried deleting the tensors but it still didn't work.
My code is sth like this (overly simplified):
def save_to_file(model, imgs_batch):
  imgs_batch = model(imgs_batch.to('cuda'))
  torch.save(imgs_batch, file)
  del imgs_batch
model = resnet().to('cuda')
for each batch:
  imgs_batch = import_images(...)  # Imports images in a (batch size, 3, 224, 224) Tensor
  save_to_file(model, imgs_batch)
del model


title: How can I apply an image variable to a pytorch neural net trained on transformed data?

body:
The neural net was trained with inputs that have been imported and transformed in the following way:

transform = transforms.Compose(
    [transforms.ToTensor()])

data_path = '/home/tim/Documents/trainset/'
traindataset = torchvision.datasets.ImageFolder(
    root=data_path,
    transform=transform
)
trainloader = torch.utils.data.DataLoader(
    traindataset,
    batch_size=3,
    num_workers=1,
    shuffle=True
)

I now have a single numpy array called img that represents an image in RGB with pixel values ranging from 0 to 255. how can i apply trainloader and all the transforms to turn this array into a single element batch to be put through my neural network?


title: How to compare two different tensors through matplotlib or any other visualize tools?

body:
I have two tensors x and y, there both are the same dimension shape = (1, 64, 1, 1)

basically y is output from many functions, and x is the input

I want to compare these two tensor using visualize tool like matplotlib..etc

anyway to do ?

The below are x and y example, I only post 10 of 64 since the restriction

x

tensor([[[[-0.8467]],

         [[-0.0949]],

         [[-0.8253]],

         [[-0.1027]],

         [[ 0.0476]],

         [[-0.4173]],

         [[-0.0870]],

         [[ 0.0650]],

         [[ 0.3816]],

         [[ 0.2046]]]], grad_fn=<MulBackward0>)

y

tensor([[[[-2.0307]],

         [[-0.1594]],

         [[-1.5174]],

         [[-0.2767]],

         [[ 0.1049]],

         [[-0.9605]],

         [[-0.2127]],

         [[ 0.1342]],

         [[ 0.8275]],

         [[ 2.0508]],
]])


title: how to deal with uncertain amount of sentences in different instances within a batch?

body: I encountered a coding problem. In my dataset, an instance includes several sentences (different amounts in different instances). They can not be concatenated to serve as a single one. How can I effectively process this kind of data with PyTorch? Or I have to process instance one by one?


title: How to avoid create a network structure properly

body:
I tried to define a feedforwad function in my neural network model:

class FeedForward(nn.Module):
    def __init__(self):
        super(FeedForward,self).__init__()
        self.fc1 = nn.Linear(784, 256)
        self.fc2 = nn.Linear(256, 64)
        self.fc2 = nn.Linear(64, 10)

    def feedforward(self, x):
        x = x.view(x.shape[0], -1)  # make sure inputs are flattened

        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.relu(self.fc3(x))
        x= F.log_softmax(x, dim=1)  # preserve batch dim

        return x

The message says:

  NotImplementedError

I am not sure what I have missing.


title: Can I train two networks, one network which contains another netwrok, at the same time?

body:
I define two networks in my model.

def Net_One():
    conv2d
    conv2d
    ...
def Net_Two():
    Net_One(input)
    conv2d
    fc

So my question is: when I train Net_Two, using back propagation optimization, will pytorch automatically train Net_One or not? Why?


title: Best way to handle a data pipeline where training inputs are randomly sampled and retrieved from a 2nd dataset with torch.utils.data

body:
Here is an example what I am trying to do. Here's what my dataset may look like

2   | 34, 64, 2243, 55678, 323, 778, 4454, 23, 4433, 3445, 455, 32
23  | 343 ,56, 2, 5, 675, 34, 232, 677, 7, 54, 436, 77, 85, 33
592 | 343, 54, 4, 6
23  | 34
123 | 2, 4, 54, 38, 6643, 67, 3
...

Each of these numbers are indexes (not the actual model inputs/target data) which point to the actual data that will be fed into the model. The actual model data is in a separate dataset, organized by indexes (same number used in the first dataset), so it would look something like this

1 | data1
2 | data2
3 | data3

For the first set, on the left side are indexes to the targets, so 2, 23, 592, 23, 123. On the right side are indexes to the potential inputs, and for each target, 4 inputs are picked at random. For targets which have a number of inputs less than 4, inputs are repeated.

So taking the first training example in the first set, the model input would look like this

target = torch.tensor([data2])
input = torch.mean( torch.tensor([data34]), torch.tensor([data2243]), torch.tensor([data23]), torch.tensor([data32]) )

Both sets are too big to hold in memory at the sametime, and I don't have enough diskspace to save targets with every permutation of inputs.

I think the best strategy is to have X number of batches set up ahead of time, using multithreading to setup future batches, so that the data pipeline does't become the bottle neck in the training, but I am wondering how to go about doing that.

Edit:

I think my description is a bit hazy, so I made an analogy of the type of data pipeline I'm trying to develop.

An analogy would be some types of word embedding trainings, where the number of input words is smaller than the context window. So for a particular target word, the input words need to be chosen at random within the context window of that target word.

For example, take this sentence:

There once was a very fast dog who could out run any other animal in the city.

Say the target word was dog, and that the context window size was 6, but the input size was 2. So we have a choice of 'a', 'very', 'fast', 'who', 'could', 'out', of which we need to pick two inputs randomly. So an example training example would be the word embedding for 'dog' as the target, and the word embeddings of 'fast' and 'out' for the inputs.

So in word embedding trainings, many different inputs will be used in many different targets. But with word embedding trainings, all the word embeddings are able to be stored in live memory since the vocab is 6 figures. For my case, the inputs can't all be in live memory.

So an idea of what I'm looking to do is word2vec training but most of the word embeddings have to stay in disk space at any particular instance.


title: Return all dimensions if there are 3 channels, else return 0

body:
I want to plot some images that are sometimes grayscale, sometimes color.

Since I'm using pytorch, the transformed images are either (50, 100, 1) or (50, 100, 3) depending on if they're color or not. Since matplotlib cannot take the former for a picture, I need to return these shapes:

(100, 100, 3) # if it's a color picture
(100, 100) # if it's a grayscale picture

Those are the shapes matplotlib can deal with. Here is what my workflow looks like:

import numpy as np
import matplotlib.pyplot as plt

h, w = 50, 100

grey = np.random.randint(0, 256, (h, w, 1))
color = np.random.randint(0, 256, (h, w, 3)

With the same line, I need to be able to plot images that can either be color or greyscale.

plt.imshow(grey[:, :, 0 if grey.shape[-1] == 1 else :]
plt.imshow(color[:, :, 0 if color.shape[-1] == 1 else :]

But that's not correct python syntax.


title: How to use input in RNN model

body:
I have some rnn model

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        # some code #
    def forward(self, x):
        x = self.rnn(x)
        output = my_function(# here is a code to get input #)
        return x, output

And I need use input in my function in this model. Сan i do it somehow


title: Need to plot pytorch 2d tensor's [0, 0] object against loss value

body:
I need a plot like this:

Here is my Code

new_weights_tensor = w.clone()

weights_x = [new_weights_tensor[0, 0].item()]
loss_y = [float(loss.detach().numpy())]

for i in range(0, 10):
    new_weights_tensor = new_weights_tensor.clone()
    new_weights_tensor[0, 0] = (new_weights_tensor[0, 0] + 0.01)
    pred = model(inputs, new_weights_tensor)
    new_loss = mse(pred, targets)

    weights_x.append(new_weights_tensor[0, 0].item())
    loss_y.append(new_loss.item())

# print(weights_x)
# print(loss_y)

plt.plot(weights_x, loss_y)
plt.show()

But i m getting this instead:

This is the link for tutorial which i am following
https://jovian.ml/aakashns/02-linear-regression


title: Clearing GPU memory after interrupting the script

body: I noticed that when I break a program multiple times (written in Pytorch with cuda enabled), the GPU memory does not get flushed. Is there any way to do it in such instances when the script needs to be interrupted several times without finishing its execution?


title: What are possible reasons that validation error doesn't change but train loss decreases?

body:
I use pre-trained ResNet to extract 1000 dimensional features for each image, then put these images into my self-built net to do classification tasks and use triplet loss function.

There is a part of my code:

class Network(torch.nn.Module):
    def __init__(self,n_feature = 1000, n_hidden_1 = 200, n_hidden_2 = 100, n_hidden_3 = 50, n_hidden_4 = 20,n_output = 3):
        super(Network, self).__init__()
        self.net = torch.nn.Sequential(
                        torch.nn.Linear(n_feature, n_hidden_1),
                        torch.nn.BatchNorm1d(n_hidden_1),
                        torch.nn.ReLU(),
                        torch.nn.Linear(n_hidden_1, n_hidden_2),
                        torch.nn.BatchNorm1d(n_hidden_2),
                        torch.nn.ReLU(),
                        torch.nn.Linear(n_hidden_2, n_hidden_3),
                        torch.nn.BatchNorm1d(n_hidden_3),
                        torch.nn.ReLU(),
                        torch.nn.Linear(n_hidden_3, n_hidden_4),
                        torch.nn.BatchNorm1d(n_hidden_4),
                        torch.nn.ReLU(),
                        torch.nn.Linear(n_hidden_4, n_output),
                        torch.nn.Sigmoid()
                                )
    def forward(self, x):
        x = self.net(x)
        return x

and there are training and validation process:

    for epoch in tqdm(range(n_epoch)):
        model.train()
        for step, (batch_anchor, batch_positive, batch_negative )in enumerate(train_loader):

            anchor_out = model(batch_anchor)
            positive_out = model(batch_positive)
            negative_out = model(batch_negative)

            loss = loss_func(anchor_out, positive_out, negative_out)
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

        model.eval()
        for step, (batch_anchor_val, batch_positive_val, batch_negative_val) in enumerate(val_loader):

            anchor_out_val = model(batch_anchor_val)
            positive_out_val = model(batch_positive_val)
            negative_out_val = model(batch_negative_val)

            loss_val = loss_func(anchor_out_val, positive_out_val, negative_out_val)

where I define loss function and optimiser like following:

optimizer = optim.Adam(model.parameters(), lr=0.002)
loss_func = torch.nn.TripletMarginLoss(p=2, margin=1)

and some results:

Epoch: 1/50 - Loss: 0.8764 - Val_loss: 0.9920
Epoch: 2/50 - Loss: 0.7035 - Val_loss: 0.9897
Epoch: 3/50 - Loss: 0.6313 - Val_loss: 0.9972
Epoch: 4/50 - Loss: 0.5958 - Val_loss: 0.9980
Epoch: 5/50 - Loss: 0.5724 - Val_loss: 0.9930
Epoch: 6/50 - Loss: 0.5541 - Val_loss: 1.0123
...

While training, train loss always decrease, but val loss doesn't change, I don't know why, maybe someone know what are potential reasons? I have read some blogs, and check my data set, I am sure my data set are correctly split.


title: How to improve my image classifier to recognize real world images

body:
I would like to train a hand gesture classifier with pytorch.
  The dataset images looks like this.

I tried to use resnet34 and some kind of data augmentation. I got a high accuracy on test set, but low accuracy when trying to recognize my own gesture in the real world. It works fine when the background is white, goes crazy when other things(my face, chair, bed etc.) appeared in the background. Maybe that's because test images have a pure background, so how can I improve my classifier?
Also I want to add a 'non-gesture' category in my claasifier as well. How can I do that?
This is my data augmentation transforms:

transform = torchvision.transforms.Compose([
         torchvision.transforms.Grayscale(3),
         torchvision.transforms.RandomHorizontalFlip(),
         torchvision.transforms.RandomRotation(20),
         torchvision.transforms.RandomResizedCrop(64, (0.6, 1.2)),
         torchvision.transforms.ToTensor(),
         torchvision.transforms.Normalize((0.5, 0.5, 0.5), std=(0.5, 0.5, 0.5)),
    ])

Any advice is appreciated. Thanks!


title: DeepFix - Location Biased Convolution

body:
I got a question regarding this paper: https://arxiv.org/pdf/1510.02927.pdf
In the Network Architecture they implement something called location biased convolution.
Basically it is 16 2d-gausians appended to the 512 filters of the convolutional layer (See figure 5 from the paper)
Picture of Location Biased Convolution.
I want to implement this in PyTorch, but have no clue how to add fixed filters to a convolutional block. The weights should be trained as discussed in the paper.
Can anyone give a hint of what to do or has done this before?


title: Where can the symbol __Z10initModulev be found?

body:
While trying to build torch_glow from source under MacOS X 10.14.6 (Mojave), I'm getting:
dbanas@Davids-MBP:~/Documents/Projects/PyTorch-Glow/glow/torch_glow
$ CMAKE_ARGS="-DLLVM_DIR=/usr/local/opt/llvm/lib/cmake/llvm -DCMAKE_C_COMPILER=/usr/local/opt/llvm/bin/clang -DCMAKE_CXX_COMPILER=/usr/local/opt/llvm/
bin/clang++" python setup.py --run_cmake install
Unable to import torch. Error:
     dlopen(/usr/local/lib/python3.7/site-packages/torch/_C.cpython-37m-darwin.so, 2): Symbol not found: __Z10initModulev
  Referenced from: /usr/local/lib/python3.7/site-packages/torch/_C.cpython-37m-darwin.so
  Expected in: flat namespace
 in /usr/local/lib/python3.7/site-packages/torch/_C.cpython-37m-darwin.so
You need to install pytorch first.

Note that PyTorch was just installed, using a fairly recent nightly build and the following command:
$ CFLAGS=$CPPFLAGS CC=/usr/local/opt/llvm/bin/clang++ LD_LIBRARY_PATH="/usr/local/opt/openssl@1.1/lib:/usr/local/opt/libffi/lib:/usr/local/opt/readlin
e/lib:/usr/local/opt/sqlite/lib:/usr/local/opt/icu4c/lib:/usr/local/opt/libxml2/lib:/usr/local/opt/llvm/lib" LLVM_DIR=/usr/local/opt/llvm/lib/cmake/ll
vm CMAKE_C_COMPILER=/usr/local/opt/llvm/bin/clang CMAKE_CXX_COMPILER=/usr/local/opt/llvm/bin/clang++ python setup.py --cmake install -f
Building wheel torch-1.6.0a0+c73095e
-- Building version 1.6.0a0+c73095e

I can't find much on this mysterious symbol: __Z10initModulev.
Does anyone know where it lives?


title: Transfer learning by using vgg in pytorch

body:
I am using vgg16 for image classification. I want to test my transfered model with the following code:
classes = ['A', 'B', 'C']
len(classes) #3
len(test_data)#171
batch_size=10

# Testing
test_loss = 0.0
class_correct = list(0. for i in range(len(classes)))
class_total = list(0. for i in range(len(classes)))

vgg16.eval()

for data, target in test_loader:
    output = vgg16(data)
    loss = criterion(output, target)
    test_loss += loss.item()*data.size(0)
    _, pred = torch.max(output, 1)
    correct_tensor = pred.eq(target.data.view_as(pred))
    correct = np.squeeze(correct_tensor.numpy())
    for i in range(batch_size):
        label = target.data[i]
        class_correct[label] += correct[i].item()
        class_total[label] += 1

test_loss = test_loss/len(test_loader.dataset)
print('Test Loss: {:.6f}\n'.format(test_loss))

for i in range(len(classes)):
    if class_total[i] > 0:
        print('Test Accuracy of %5s: %2d%% (%2d/%2d)' % (
            classes[i], 100 * class_correct[i] / class_total[i],
            np.sum(class_correct[i]), np.sum(class_total[i])))
    else:
        print('Test Accuracy of %5s: N/A (no training examples)' % (classes[i]))

print('\nTest Accuracy (Overall): %2d%% (%2d/%2d)' % (
    100. * np.sum(class_correct) / np.sum(class_total),
    np.sum(class_correct), np.sum(class_total)))

I receive following error:
I receive following error:
     15     for i in range(batch_size):
     16         label = target.data[i]
---> 17         class_correct[label] += correct[i].item()
     18         class_total[label] += 1
     19

IndexError: too many indices for array

I do not know why I am getting this error and how I can solve it. I would be grateful if you could help me.


title: Why does my neural network that is supposed to be performing regression always return zeroes when predicting and the loss stays the same?

body:
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(df, y)
X_train = torch.from_numpy(X_train.to_numpy()).float()
X_test = torch.from_numpy(X_test.to_numpy()).float()
y_test = torch.squeeze(torch.from_numpy(y_test.to_numpy()).float())
y_train = torch.squeeze(torch.from_numpy(y_train.to_numpy()).float())

import torch.nn.functional as F
class Net(nn.Module):
  def __init__(self):
    super().__init__()
    self.linear1 = nn.Linear(51, 30)
    self.linear2 = nn.Linear(30, 30)
    self.linear3 = nn.Linear(30, 20)
    self.linear4 = nn.Linear(20, 10)
    self.linear5 = nn.Linear(10, 1)
  def forward(self, x):
    pred = F.relu(self.linear1(x))
    pred = F.relu(self.linear2(pred))
    pred = F.relu(self.linear3(pred))
    pred = F.relu(self.linear4(pred))
    pred = F.relu(self.linear5(pred))
    return pred

model = Net()
epochs = 20
criterion = nn.MSELoss()
optimizer = optim.Adam(model.parameters(), lr = 0.01)
losses = []
for epoch in range(epochs):
  for (data, target) in zip(X_train, y_train):
    y_pred = model(data.view(-1,51))
    optimizer.zero_grad()
    loss = criterion(y_pred, target.view(y_pred.shape))
    loss.backward()
    optimizer.step()
  losses.append(loss)

After I train this model and try to predict new results, the model is always predicting 0. Also, when I try to plot the losses, it is showing a straight line with 0 slope and no improvement in the loss. Why does this occur and how do I fix my model so that it will predict proper results and the loss will go down like a regular deep learning model?


title: Feature matching in LSTM Network Pytorch

body: When it comes to generating loss by feature matching it is pretty convenient on CNN but I want to know how can we do feature matching loss in an RNN?


additional_aug=[[zoom_crop(scale=(0.75,1.25), do_rand=False)],
                [brightness()],
                [contrast()]]

    ds_tfms = get_transforms(do_flip=True,
                      flip_vert=True,
                      max_lighting=0.2,
                      xtra_tfms=additional_aug)

    data = ImageDataBunch.from_folder(path_db, train=".", valid_pct=0.2,
        ds_tfms=get_transforms(), size=224,bs=32, num_workers=4).normalize(imagenet_stats)

Exception: It's not possible to apply those transforms to your dataset:
 'Image' object has no attribute '_maybe_add_crop_pad'


title: Train our FCN for depth map

body:
I want to train my own FCN model for training without any pretrained model and generate Depth map.
Data is being created but facing an issue on how to give image as labels and images as input.
Following this paper done with patch but got stuck in depth based CNN.
Any help would be really very helpful.
Thanks in advance.
https://www.semanticscholar.org/paper/Face-anti-spoofing-using-patch-and-depth-based-CNNs-Atoum-Liu/63ada149373f27d1ded6df24acaa5871d0c3e379


title: is there a way to shuffle multi dataloader in same order?

body:
Is there a way to shuffle each dataset equally when loading the dataset at once using each pytorch datataloder?
ex) facedata - face1, face2, face3, face4, maskdata - mask1, mask2, mask3, mask4
after call dataloader(facedata, shuffle=True), dataloader(maskdata, shuffle=True)
the result is same order face2, face3 and mask2, mask3


title: Given the forward optical flow between two frames, how to calculate the reverse optical flow between two frames？

body:
Given two frames img1 and img2. If we know the forward optical flow flo1(img1 + flo1 = img2) between two frames, how to calculate their reverse optical flow flo2(img2 + flo2 = img1)?
How can I do it with pytorch?


title: Getting runtime error for input dimension mismatch for inceptionV3

body:
I changed the dimension many way. But not a single change work for me. Here is my code
#transform for inceptionV3

train_transform = transforms.Compose([
                    transforms.Resize((299, 299)),
                    transforms.RandomHorizontalFlip(),
                    transforms.ToTensor(),
                    transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                     std=[0.229, 0.224, 0.225])])

val_transform = transforms.Compose([
                transforms.Resize((299, 299)),
                transforms.ToTensor(),
                transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                 std=[0.229, 0.224, 0.225])])
vocab = dataloader.get_vocab(dataset=args.dataset)()
# Data loader and validation split
#num_workers will be 4, I have used 0 for chittron
data_loader = dataloader.get_data_loader(dataset=args.dataset)(mode="train",
                                                               transform=train_transform,
                                                               vocab=vocab,
                                                               batch_size=args.batch_size,
                                                               shuffle=True,
                                                               num_workers=4,
                                                               data_dir=args.data_dir)

valid_data_loader = dataloader.get_data_loader(dataset=args.dataset)(mode="val",
                                                                    transform=val_transform,
                                                                     vocab=vocab,
                                                                 batch_size=args.batch_size,
                                                                     shuffle=False,
                                                                     num_workers=4,
                                                                     data_dir=args.data_dir)

And the main code is:
inception_v3 = getattr(models, dictionary['cnn_model'])(pretrained=True)
 # remove the last fc layer
self.inception_v3 = nn.Sequential(*list(inception_v3.children())[:-1])

for param in self.inception_v3.parameters():
     param.requires_grad = False
self.encoder_linear = nn.Linear(inception_v3.fc.in_features, dictionary['embed_size'])
self.bn = nn.BatchNorm1d(dictionary['embed_size'])

self.embedding = nn.Embedding(dictionary['vocab_size'], dictionary['embed_size'])
self.rnn = getattr(nn, dictionary['rnn_model'])(dictionary['embed_size'], dictionary['hidden_size'], dictionary['num_layers'], batch_first=True)
self.dropout = nn.Dropout(dictionary['dropout'])
self.decoder_linear = nn.Linear(dictionary['hidden_size'], dictionary['vocab_size'])

self.init_weights()

And i am getting this error.

self.padding, self.dilation, self.groups) RuntimeError: Expected
4-dimensional input for 4-dimensional weight 192 768 1 1, but got
2-dimensional input of size [8, 1000] instead

Any idea what should do. Is my code okay?


title: Tensorpack conv2d seems output different result with pytorch conv2d

body:
I'm trying to convert a resnet101 model checkpoint from tensorpack to pytorch. The checkpoint file has been converted but the inference results do not match. The key problem is that, from the very beginning, i.e. the first conv2d layer, the results become different.
The following is my tensorpack code (to make it short, I only post the key parts):
tensorpack model:
class Model(ImageNetModel):
    def __init__(self, depth, data_format='NCHW', mode='resnet'):
        super(Model, self).__init__(data_format)

        if mode == 'se':
            assert depth >= 50

        self.mode = mode
        basicblock = preresnet_basicblock if mode == 'preact' else resnet_basicblock
        bottleneck = {
            'resnet': resnet_bottleneck,
            'preact': preresnet_bottleneck,
            'se': se_resnet_bottleneck}[mode]
        self.num_blocks, self.block_func = {
            18: ([2, 2, 2, 2], basicblock),
            34: ([3, 4, 6, 3], basicblock),
            50: ([3, 4, 6, 3], bottleneck),
            101: ([3, 4, 23, 3], bottleneck),
            152: ([3, 8, 36, 3], bottleneck)
        }[depth]

    def get_logits(self, image):
        with argscope([Conv2D, MaxPooling, GlobalAvgPooling, BatchNorm], data_format=self.data_format):
            return resnet_backbone(
                image, self.num_blocks,
                preresnet_group if self.mode == 'preact' else resnet_group, self.block_func)

The resnet_backbone of the model is :
def resnet_backbone(image, num_blocks, group_func, block_func):
    with argscope(Conv2D, nl=tf.identity, use_bias=False,
                  W_init=tf.variance_scaling_initializer(scale=2.0, mode='fan_out')):
        l1 = (LinearWrap(image).Conv2D('conv0', 64, 7, stride=2, nl=tf.identity)())
        logits = (LinearWrap(l1)
            .BatchNorm('conv0/bn')
            .apply(tf.nn.relu)
            .MaxPooling('pool0', shape=3, stride=2, padding='SAME')
            .apply(group_func, 'group0', block_func, 64, num_blocks[0], 1)
            .apply(group_func, 'group1', block_func, 128, num_blocks[1], 2)
            .apply(group_func, 'group2', block_func, 256, num_blocks[2], 2)
            .apply(group_func, 'group3', block_func, 512, num_blocks[3], 2)
            .GlobalAvgPooling('gap')
            .FullyConnected('linear', NUM_CLASSES, nl=tf.identity)())
    return logits, l1

In particular, I return the l1 tensor, which is the output of the first conv layer of the resnet. Then I compare it with the output of the pytorch  conv layer.
The pytorch model is directly imported from the torchvision.models, and also I return the output of the first conv layer.
The input image is of size (1,3,224,224), the output shape is (1, 64, 112, 112). Both results match.
Then I made 3 tests:

The values of two results are quite different. The summation of absolute different is in scale of 1e6.

I implement a conv layer using numpy. Its result matches pretty well with pytorch, total difference < 0.1. And it's also 1e6 different from tensorpack.

In tensorpack code, I also tried to return the image (i.e. the input) directly to the output. And compare it with the input of pytorch. The two inputs match in both shape and value. So I'm quite sure the input and kernel values of the two conv2d are exactly the same.

What's wrong with my implementation?
Thank you all for helping me!


title: What is the difference between CharEmbeddings and CNNCharEmbeddings/Encoding?

body:
while studying an ML model I found two seemingly different modules to do CharEmbedding and CharEncoding, but it is not clear to me why both are needed and what their difference is.
The CharEmbedding is the following and is done through a LSTM, as I always believe:
class CharEmbeddings(nn.Module):
    def __init__(self, char_vocab, embedding_dim, hidden_size, which_cuda=0):
        super().__init__()

        self.embedding_dim = embedding_dim
        self.vocab = char_vocab
        self.hidden_size = hidden_size

        self.embeddings = nn.Embedding(
            num_embeddings = len(self.vocab),
            embedding_dim = embedding_dim,
            padding_idx = self.vocab.pad
        )

        self.bilstm = nn.LSTM(
            input_size = embedding_dim,
            hidden_size = self.hidden_size,
            num_layers = 1,
            bidirectional = True
        )

    def forward(self, sentence_batch):
        # char2index + padding
        words, lengths, unsort_idx = self.prepare(sentence_batch)
        # words -> (n_pad_words, max_word_length)
        # sort + remove 0 lenth words (pads)
        non_zero_words, non_zero_lengths = self.remove_pad_words(words, lengths)
        # non_zero_words -> (n_nonpad_words, max_word_length)

        embeddings = self.embeddings(non_zero_words).to(self.device)
        # embeddings -> (n_nonpad_words, max_word_length, embeddings_dim)
        # pack
        x = torch.nn.utils.rnn.pack_padded_sequence(embeddings, non_zero_lengths, batch_first=True)
        # pass through lstm
        output, hidden = self.bilstm(x)
        x, _ = torch.nn.utils.rnn.pad_packed_sequence(output, batch_first=True)
        # embeddings -> (n_nonpad_words, max_word_length, hidden_size*2)

        # take the output of the lstm correctly
        # filter idx is the id of the last character in each word
        # this aims to find the output of the lshtm on the last non pad char
        # ex: ['h', 'e', 'l', 'l', 'o', '<pad>']
        # instead of getting the output of '<pad>' we want the output of 'o'
        filter_idx = non_zero_lengths.long().view(-1, 1, 1).expand(-1, 1, self.hidden_size*2) -1
        # filter_idx -> (n_nonpad_words), max_word_length, hidden_size*2)
        forward_out = x.gather(1, filter_idx).squeeze(1)[:,:self.hidden_size]
        # filter_idx -> (n_nonpad_words, hidden_size)

        # get the output of the first character
        backward_out = x[:,0,self.hidden_size:]
        # concat first char's output last hidden state part with the last char's output first hidden state part
        x = torch.cat([forward_out, backward_out], 1)
        # x -> (n_nonpad_words, hidden_size*2)
        #
        x = torch.cat([x, torch.zeros(len(words)-len(non_zero_words), self.hidden_size*2).to(self.device)],0)
        # x -> (n_pad_words, hidden_size*2)
        # unsort
        x = x[unsort_idx]
        # reshape to sentence size
        x = x.view(len(sentence_batch), -1 , self.hidden_size*2)
        # x -> (batch_size, max_sentence_size, hidden_size*2)
        return x

    def prepare(self, sentence_batch):
    # receibes a batch of sentences and return a batch of words and their lengths sorted in decreasing order
        return padded_words, sort_word_len, unsort_idx

    def remove_pad_words(self, padded_words, word_lengths):
    # this function remove pad words from a batch of words
    # note that the index returned is the size of the batch before filtering pads
    # so, before unsorting, padded values must be re-added to the tensor
        return padded_words, sort_word_len

Leaving out the details of the code, this looks like a normal CharEmbedding to me.
Vice versa, I don't understand why you have to use a CNN to do a CharEncoding as shown below:
class CNNCharEmbeddings(CharEmbeddings):
    def __init__(self, char_vocab, cnn_embeddings_size, cnn_ce_kernel_size, cnn_ce_out_channels, which_cuda=0):
        torch.backends.cudnn.deterministic = True
        CharEmbeddings.__init__(self, char_vocab, cnn_embeddings_size, 1, which_cuda=which_cuda)

        self.embedding_dim = cnn_embeddings_size
        self.vocab = char_vocab

        self.cnn_ce_kernel_size = cnn_ce_kernel_size
        self.cnn_ce_out_channels = cnn_ce_out_channels

        self.cnn = nn.Conv1d(
            in_channels=self.embedding_dim,
            out_channels=self.cnn_ce_out_channels,
            kernel_size=cnn_ce_kernel_size,
            stride=1,
            padding=int((cnn_ce_kernel_size-1)/2),
        )

    def forward(self, sentence_batch):

        words, lengths, unsort_idx = self.prepare(sentence_batch)
        non_zero_words, non_zero_lengths = self.remove_pad_words(words, lengths)
        embeddings = self.embeddings(non_zero_words).to(self.device)

        x = self.cnn(embeddings.transpose(1,2))
        mask = (torch.arange(x.shape[2]).expand(x.shape).to(self.device) < non_zero_lengths.unsqueeze(1).unsqueeze(1).to(self.device)).float()
        x_min = (torch.arange(x.shape[2]).expand(x.shape).to(self.device) >= non_zero_lengths.unsqueeze(1).unsqueeze(1).to(self.device)).float() * torch.min(x)
        x_min = x_min.detach().float()
        x = x * mask + x_min
        x = F.relu(x)
        x = nn.MaxPool1d(
            kernel_size=x.shape[2]
        )(x)

        x=x.view([x.shape[0], x.shape[1]])
        x = torch.cat([x, torch.zeros(len(words)-len(non_zero_words), self.cnn_ce_out_channels).to(self.device)],0)

        x = x[unsort_idx]
        x = x.view(len(sentence_batch), -1 , self.cnn_ce_out_channels)

        return x

In general, using both of these modules improves the model more than it improves using only one of them. But I can't understand the usefulness of CNN and CharEncoding in general, if I already have a CharEmbedding.
It has something to do with encoder-decoder?
Thank you!


title: Pytorch rotaiton matrix

body:
I am trying to do a rotation matrix with pytorch and autograd but the result is not as it should be.
theta = tc.tensor([3.1415],requires_grad=True)
tx = tc.tensor([1.0],requires_grad=True)
ty = tc.tensor([1.0],requires_grad=True)

P = tc.tensor([[tc.cos(theta),-tc.sin(theta), tx],
               [tc.sin(theta), tc.cos(theta),ty],
               [0.0,0.0,0.0]],requires_grad=True)

I want to "learn" the parameters theta, tx, ty. After running i get a matrix of the type
 [-0.0506, -0.0973,  0.1012],
 [-0.0957,  0.1025, -0.0147],
 [ 0.0000,  0.0000,  0.0000]

which means that it learns each entry and not theta. We should have [0,0]==[1,1], [0,1]=-[1,0] which we do not. If i remove requires_grad=True from the matrix it want to backprop.
any idea how to make a simple matrix like above work?


title: Reinforcement Learning - early exploration problem in hierarchical setting

body:
Intro
Hi all. I am struggling to understand behavior with two implementations and I was hoping someone might have a clue.
So I have 2 algorithms: HER (RL with a bit of self-supervision) and HAC (hierarchical RL). The environment is picked and place.
HAC
A quick summary of how HAC works: two levels (high and low), high level generates a target (x,y,z) where the robot end-effector must be moved. the low level acts on the environment with those targets as goals. high level generates a new target every H steps.
Here's my problem
In HER, the first epochs renderings look smooths with slow movements (like deliberate exploration)
In HAC after only 50 iters (1 epoch = 2000 iters) the high-level model always outputs maximum actions (1,1,-1,1). Any idea what could cause this?
After some more training it starts outputting other actions, but most still maximum. It hinders exploration a lot and obviously fails to converge.
To give a comparison, HER converges to >80% accuracy in 40 mins. HAC goes up to 60% (sometimes) in over 48h.
Any help is greatly appreciated.
Architecture used is DDPG or TD3 (depending on version I want to use). Here's some code for selecting actions if it helps:
mu_v = self.actor(states, goals)
actions = mu_v.data.detach().cpu().numpy()
actions += np.random.normal(0, self.max_action * self.expl_noise, size=actions.shape)
actions = np.clip(actions, self.action_clips[0], self.action_clips[1])
return actions

Actor model definition
class Actor(nn.Module):
    def __init__(self, obs_size, goal_size, act_size, action_bounds, offset):
        super(Actor, self).__init__()

        self.net = nn.Sequential(
            # state + goal
            nn.Linear(obs_size + goal_size, HID_SIZE),
            nn.ReLU(),
            nn.Linear(HID_SIZE, HID_SIZE),
            nn.ReLU(),
            nn.Linear(HID_SIZE, HID_SIZE),
            nn.ReLU(),
            nn.Linear(HID_SIZE, act_size),
            nn.Tanh()
        )

        self.action_bounds = nn.Parameter(action_bounds, requires_grad=False)
        self.offset = nn.Parameter(offset, requires_grad=False)

    def forward(self, state, goal):
        return (self.net(torch.cat([state, goal], dim=1)) * self.action_bounds) + self.offset

Extra
I don't know what other information to add, but I will edit this section as people are asking questions. Thank you very much for your help!


title: Initiaized class member in pytorch module

body:
When declaring a model in pytorch, having a model class member variable declared and initialized mysteriously prevents it from being populated in the constructor. Is this expected? If so, why?
Testing code below, with example models with a component member variable.
The initialization value of the component (e.g. None, a number or a Tensor) does not change the behaviour.
import torch

class Lin1(torch.nn.Module):
    def __init__(self):
        super(Lin1, self).__init__()
        self.component = torch.nn.Linear(100,200)

class Lin2(torch.nn.Module):
    component = None
    def __init__(self):
        super(Lin2, self).__init__()
        self.component = torch.nn.Linear(100,200)

# instantiate and check member component
for cl in [Lin1, Lin2]:
    model = cl()
    print("\nModel:", cl)
    print("Component:", model.component)
    print("It's None?: ", model.component is None)

Model: <class '__main__.Lin1'>
Component: Linear(in_features=100, out_features=200, bias=True)
It's None?:  False

Model: <class '__main__.Lin2'>
Component: None
It's None?:  True


title: Why is MNIST showing as a list 4 levels deep?

body:
Experimenting with some simple code using PyTorch on MNIST, and I'm puzzled about an aspect of how it's representing the data; maybe I'm just overlooking something really obvious.
Given
train_loader = torch.utils.data.DataLoader(
    torchvision.datasets.MNIST(
        "data",
        train=True,
        download=True,
        transform=torchvision.transforms.Compose(
            [
                torchvision.transforms.ToTensor(),
                torchvision.transforms.Normalize((0.1307,), (0.3081,)),
            ]
        ),
    ),
    batch_size=batch_size_train,
    shuffle=True,
)

and
for batch_idx, (data, target) in enumerate(train_loader):
    print(data)

I get
tensor([[[[-0.4242, -0.4242, -0.4242,  ..., -0.4242, -0.4242, -0.4242],
          [-0.4242, -0.4242, -0.4242,  ..., -0.4242, -0.4242, -0.4242],
          [-0.4242, -0.4242, -0.4242,  ..., -0.4242, -0.4242, -0.4242],
          ...,
          [-0.4242, -0.4242, -0.4242,  ..., -0.4242, -0.4242, -0.4242],
          [-0.4242, -0.4242, -0.4242,  ..., -0.4242, -0.4242, -0.4242],
          [-0.4242, -0.4242, -0.4242,  ..., -0.4242, -0.4242, -0.4242]]],

        [[[-0.4242, -0.4242, -0.4242,  ..., -0.4242, -0.4242, -0.4242],
          [-0.4242, -0.4242, -0.4242,  ..., -0.4242, -0.4242, -0.4242],
          [-0.4242, -0.4242, -0.4242,  ..., -0.4242, -0.4242, -0.4242],
          ...,
          [-0.4242, -0.4242, -0.4242,  ..., -0.4242, -0.4242, -0.4242],
          [-0.4242, -0.4242, -0.4242,  ..., -0.4242, -0.4242, -0.4242],
          [-0.4242, -0.4242, -0.4242,  ..., -0.4242, -0.4242, -0.4242]]],

I was expecting a tensor corresponding to a list three levels deep: a list of images, each of which is a list of rows, each of which is a list of numbers. Or put another way, the innermost [] is a row, the next [] is an image, the outermost [] is the list of images.
But instead it is four levels deep.
Why the extra level?


title: How to use have batch norm not forget batch statistics it just used in Pytorch?

body:
I am in an unusual setting where I should not use running statistics (as that would be considered cheating e.g. meta-learning). However, I often run a forward pass on a set of points (5 in fact) and then I want to evaluate only on 1 point using the previous statistics but batch norm forgets the batch statistics it just uses. I've tried to hard code the value it should be but I get strange errors (even when I uncomment things like from the pytorch code itself like checking the dimension size).
How do I hardcode the previous batch statistics so that batch norm works on a new single data point and then reset them for a fresh new next batch?
note: I don't want to change the batch norm layer type.
Sample code I tried:
def set_tracking_running_stats(model):
    for attr in dir(model):
        if 'bn' in attr:
            target_attr = getattr(model, attr)
            target_attr.track_running_stats = True
            target_attr.running_mean = torch.nn.Parameter(torch.zeros(target_attr.num_features, requires_grad=False))
            target_attr.running_var = torch.nn.Parameter(torch.ones(target_attr.num_features, requires_grad=False))
            target_attr.num_batches_tracked = torch.nn.Parameter(torch.tensor(0, dtype=torch.long), requires_grad=False)
            # target_attr.reset_running_stats()
    return

my most comment errors:
    raise ValueError('expected 2D or 3D input (got {}D input)'
ValueError: expected 2D or 3D input (got 1D input)

and
IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

pytorch forum: https://discuss.pytorch.org/t/how-to-use-have-batch-norm-not-forget-batch-statistics-it-just-used/103437


title: Adversarial training with batch size > 1

body:
The tutorial notebook on Adverserial Training uses a batch size of 1. What code changes are needed if we want to train on a batch size of say 16. My understanding is, we only need to change the logic of
final_pred = output.max(1, keepdim=True)[1] # get the index of the max log-probability
# Now we have batch size > 1
final_pred.squeeze_()
indexes = final_pred == target
correct += torch.sum(indexes).item()

Is there something else needed? With this code change, I get values that are very similar to the case of batch_size=1 although not the same values.


title: Will it cause gradient vanishing by adding a constant positive and constant negative loss?

body:
I want to ask a question about using two losses to train one model.
I am going to generate some specific kinds of question sentences.
To achieve it, I use (1) a normal GAN to generate the normal question space. Then  (2) an auxiliary classifier to let the generator focus on generating that kind of questions.
By pre-experiment, as I use BCELoss() on Generator-Discriminator loss (loss_G_D), the loss value is around 3.
And as I use -Entropy on Generator-Classifier loss (loss_G_C), the loss value is always negative, and very big, e.g. -300.
To not let them affect each other's training procedure, I used this training method.
optimizer.zero_grad()
loss_G_D = BCELoss(discriminator(generated_data))
loss_G_D.backward()
optimizer.step()

optimizer.zero_grad()
loss_G_C = -Entropy(classifier(generated_data))
loss_G_C.backward()
optimizer.step()

However, this training procedure is a little bit slow and seems like they put the network training 'back and forth'.
So a pal suggested me this method:
optimizer.zero_grad()
loss_G_D = BCELoss(discriminator(generated_data))
loss_G_C = -Entropy(classifier(generated_data))
loss = loss_G_D+loss_G_C
# if you worry about the scale, give some weight, like
# loss = loss_G_D+0.01*loss_G_C
loss.backward()
optimizer.step()

I thought it makes sense. However, like the loss_G_D is 3, loss_G_C is -300, wouldn't it cause gradient vanishing by add them up? As using loss = loss_G_D+0.01*loss_G_C=0 ? Or can I say because they are different type of loss considering always positive and negative, we should not add them up?
(PS. I think maybe map the -Entropy loss into an always positive activation function, then add the two loss up will work?)


title: Bilstm self-attention output dim

body:
I have been trying to implement a self-attention mechanism for a bilstm. The code I wrote, looking for some resources on the web, for attention is the following:
class Attention(nn.Module):

    def __init__(self, hidden_size, batch_first=False):
        super(Attention, self).__init__()

        self.hidden_size = hidden_size
        self.batch_first = batch_first

        self.att_weights = nn.Parameter(torch.Tensor(1, hidden_size), requires_grad=True)

        stdv = 1.0 / np.sqrt(self.hidden_size)

        for weight in self.att_weights:
            nn.init.uniform_(weight, -stdv, stdv)

    def forward(self, inputs, lengths):
        batch_size, max_len = inputs.size()[:2]

        # matrix mult
        # apply attention layer
        weights = torch.bmm(inputs,
                            self.att_weights  # (1, hidden_size)
                            .permute(1, 0)  # (hidden_size, 1)
                            .unsqueeze(0)  # (1, hidden_size, 1)
                            .repeat(batch_size, 1, 1)  # (batch_size, hidden_size, 1)
                            )

        attentions = torch.softmax(F.relu(weights.squeeze(2)), dim=-1)

        # create mask based on the sentence lengths
        mask = torch.ones(attentions.size(), requires_grad=True).cuda()
        for i, l in enumerate(lengths):  # skip the first sentence
            if l < max_len:
                mask[i, l:] = 0

        # apply mask and renormalize attention scores (weights)
        masked = attentions * mask
        _sums = masked.sum(-1).unsqueeze(-1)  # sums per row

        attentions = masked.div(_sums)

        # apply attention weights
        weighted = torch.mul(inputs, attentions.unsqueeze(-1).expand_as(inputs))

        # get the final fixed vector representations of the sentences
        representations = weighted.sum(1)

        # (batch_size, n_lstm_unit) and (batch_size, sentence_len)
        return representations, attentions

class MyLSTM(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers, batch_first, bidirectional, dropout):
        super(MyLSTM, self).__init__()
        self.dropout = nn.Dropout(p=dropout)
        self.lstm1 = nn.LSTM(input_size=input_size,
                             hidden_size=hidden_size,
                             num_layers=num_layers,
                             batch_first=batch_first,
                             bidirectional=bidirectional,
                             dropout=dropout)
        self.atten1 = Attention(hidden_size * 2, batch_first=batch_first)  # 2 is bidrectional

    def forward(self, x, x_len):
        x = nn.utils.rnn.pack_padded_sequence(x, x_len, batch_first=True)
        out1, (h_n, c_n) = self.lstm1(x)
        x, lengths = nn.utils.rnn.pad_packed_sequence(out1, batch_first=True)
        x, att1 = self.atten1(x, lengths)  # skip connect

        tmp1 = torch.bmm(x.unsqueeze(2), att1.unsqueeze(1))
        tmpp1 = tmp1.transpose(1, 2)
        return tmpp1

What I get after the attention are two tensors of dimension, respectively, (batch_size, n_lstm_unit) and (batch_size, sentence_len). What I necessarily need is to have a single tensor of dimension (batch_size, sentence_len, n_lstm_unit).
What I could do is something like this:
tmp1 = torch.bmm(x.unsqueeze(2), att1.unsqueeze(1))
tmpp1 = tmp1.transpose(1, 2)

The problem is that in this way it seems that the performance of my parser, after inserting the attention, deteriorates.
I wanted to know if the performance hit could be caused by this  torch.bmm(x.unsqueeze(2), att1.unsqueeze(1)) operation to get a single vector of that size or if it is an attention implementation problem.


title: How to get index.cuda?

body:
I am reading a code about CMC downloaded from github, and it can't work on vscode.
Code:
if torch.cuda.is_available():
        index = index.cuda(async=True)
        inputs = inputs.cuda()

Error message is as follows:
  File "e:\CMC-master\train_CMC.py", line 219
    index = index.cuda(async=True)
                       ^
SyntaxError: invalid syntax

How to fix it?


title: cant make from data set DataLoader

body:
I have numpy array

> data[0]

array([array([[255, 255, 255, ..., 255, 255, 255],
       [255, 255, 255, ..., 255, 255, 255],
       [255, 255, 255, ..., 255, 255, 255],
       ...,
       [255, 255, 255, ..., 255, 255, 255],
       [255, 255, 255, ..., 255, 255, 255],
       [255, 255, 255, ..., 255, 255, 255]], dtype=uint8),
       62390], dtype=object)

And I need to make DataLoader from this data
But when I try
my_dataloader = DataLoader(data, batch_size= 64, shuffle= True, num_workers= 4)
I have error:  default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found object
.
But I can't change dtype.
How can I do it?


title: create a ones tensor according to another lengths tensor

body:
I have one tensor of the size of [batch_size,1] where each number for sample indicates an integer that is smaller than 5000. I'd like to create a new tensor of the size of [batch_size,5000] where the first numbers for each sample are ones, according to the first tensor, and the rest are zeros. For example:
t1=[[3],[5]]

so we will result with
t2=[[1,1,1,0,0,...],[1,1,1,1,1,0,0,...]]


title: Why Keras behave better than Pytorch under the same network?

body:
I'm practicing machine learning in kaggle. I want to use pytorch to imitate other people's network model of keras. But the MSE of pytorch is much larger than that of keras. I don't understand the reason. I need everybody help. thanks!
hear is keras:
model = Sequential()

model.add(Convolution2D(32, (3,3), padding='same', use_bias=False, input_shape=(96,96,1)))
model.add(LeakyReLU(alpha = 0.1))
model.add(BatchNormalization())

model.add(Convolution2D(32, (3,3), padding='same', use_bias=False))
model.add(LeakyReLU(alpha = 0.1))
model.add(BatchNormalization())
model.add(MaxPool2D(pool_size=(2, 2)))

model.add(Convolution2D(64, (3,3), padding='same', use_bias=False))
model.add(LeakyReLU(alpha = 0.1))
model.add(BatchNormalization())

model.add(Convolution2D(64, (3,3), padding='same', use_bias=False))
model.add(LeakyReLU(alpha = 0.1))
model.add(BatchNormalization())
model.add(MaxPool2D(pool_size=(2, 2)))

model.add(Convolution2D(96, (3,3), padding='same', use_bias=False))
model.add(LeakyReLU(alpha = 0.1))
model.add(BatchNormalization())

model.add(Convolution2D(96, (3,3), padding='same', use_bias=False))
model.add(LeakyReLU(alpha = 0.1))
model.add(BatchNormalization())
model.add(MaxPool2D(pool_size=(2, 2)))

model.add(Convolution2D(128, (3,3),padding='same', use_bias=False))
# model.add(BatchNormalization())
model.add(LeakyReLU(alpha = 0.1))
model.add(BatchNormalization())

model.add(Convolution2D(128, (3,3),padding='same', use_bias=False))
model.add(LeakyReLU(alpha = 0.1))
model.add(BatchNormalization())
model.add(MaxPool2D(pool_size=(2, 2)))

model.add(Convolution2D(256, (3,3),padding='same',use_bias=False))
model.add(LeakyReLU(alpha = 0.1))
model.add(BatchNormalization())

model.add(Convolution2D(256, (3,3),padding='same',use_bias=False))
model.add(LeakyReLU(alpha = 0.1))
model.add(BatchNormalization())
model.add(MaxPool2D(pool_size=(2, 2)))

model.add(Convolution2D(512, (3,3), padding='same', use_bias=False))
model.add(LeakyReLU(alpha = 0.1))
model.add(BatchNormalization())

model.add(Convolution2D(512, (3,3), padding='same', use_bias=False))
model.add(LeakyReLU(alpha = 0.1))
model.add(BatchNormalization())

model.add(Flatten())
model.add(Dense(512,activation='relu'))
model.add(Dropout(0.1))
model.add(Dense(30))
model.summary()
model.compile(optimizer='adam', loss='mean_squared_error',metrics=['mae'])
model.fit(X_train,y_train,epochs = 50,batch_size = 256,validation_split = 0.2)

and the result:
Epoch 45/50
5639/5639 [==============================] - 4s 713us/step - loss: 7.4105 - mean_absolute_error: 2.0711 - val_loss: 2.6523 - val_mean_absolute_error: 0.9367
Epoch 46/50
5639/5639 [==============================] - 4s 714us/step - loss: 6.6229 - mean_absolute_error: 1.9573 - val_loss: 3.4929 - val_mean_absolute_error: 1.2566
Epoch 47/50
5639/5639 [==============================] - 4s 714us/step - loss: 5.9132 - mean_absolute_error: 1.8473 - val_loss: 3.6761 - val_mean_absolute_error: 1.3301
Epoch 48/50
5639/5639 [==============================] - 4s 715us/step - loss: 5.6835 - mean_absolute_error: 1.8076 - val_loss: 10.4315 - val_mean_absolute_error: 2.8098
Epoch 49/50
5639/5639 [==============================] - 4s 714us/step - loss: 9.1544 - mean_absolute_error: 2.3395 - val_loss: 5.4645 - val_mean_absolute_error: 1.8136
Epoch 50/50
5639/5639 [==============================] - 4s 713us/step - loss: 7.2843 - mean_absolute_error: 2.0642 - val_loss: 4.6411 - val_mean_absolute_error: 1.1547

================================================================================
and,following is pytorch:
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import torch.utils.data as data_utils
from torch.utils.data.sampler import SubsetRandomSampler

class CNN(nn.Module):
    def __init__(self):
        super(CNN, self).__init__()
        self.conv1_1 = nn.Conv2d(in_channels=1, out_channels=32, kernel_size=3,padding=1) # (b,1,96,96) to (b,32,92,92)
        self.conv1_2 = nn.Conv2d(in_channels=32, out_channels=32, kernel_size=3,padding=1) # (b,32,96,96) to (b,32,92,92)
        self.conv1_bn = nn.BatchNorm2d(32)
        #maxpool
        self.conv2_1 = nn.Conv2d(in_channels=32, out_channels=64, kernel_size=3,padding=1) # (b,32,46,46) to (b,64,46,46)
        self.conv2_2 = nn.Conv2d(in_channels=64, out_channels=64, kernel_size=3,padding=1) # (b,64,48,48) to (b,64,48,48)
        self.conv2_bn = nn.BatchNorm2d(64)
        #maxpool
        self.conv3_1 = nn.Conv2d(in_channels=64, out_channels=96, kernel_size=3,padding=1) # (b,64,23,23) to (b,128,23,23)
        self.conv3_2 = nn.Conv2d(in_channels=96, out_channels=96, kernel_size=3,padding=1) # (b,128,24,24) to (b,128,24,24)
        self.conv3_bn = nn.BatchNorm2d(96)
        #maxpool
        self.conv4_1 = nn.Conv2d(in_channels=96, out_channels=128, kernel_size=3,padding=1) # (b,128,12,12) to (b,256,12,12)
        self.conv4_2 = nn.Conv2d(in_channels=128, out_channels=128, kernel_size=3,padding=1) # (b,256,12,12) to (b,256,12,12)
        self.conv4_bn = nn.BatchNorm2d(128)
        #maxpool
        self.conv5_1 = nn.Conv2d(in_channels=128, out_channels=256, kernel_size=3,padding=1) # (b,128,6,6) to (b,256,6,6)
        self.conv5_2 = nn.Conv2d(in_channels=256, out_channels=256, kernel_size=3,padding=1) # (b,256,6,6) to (b,256,6,6)
        self.conv5_bn = nn.BatchNorm2d(256)
        #maxpool
        self.conv6_1 = nn.Conv2d(in_channels=256, out_channels=512, kernel_size=3,padding=1) # (b,128,6,6) to (b,256,6,6)
        self.conv6_2 = nn.Conv2d(in_channels=512, out_channels=512, kernel_size=3,padding=1) # (b,256,6,6) to (b,256,6,6)
        self.conv6_bn = nn.BatchNorm2d(512)
        #! maxpool
        self.fc1 = nn.Linear(512*3*3, 512)#
        #self.fc2 = nn.Linear(512, 512)
        self.fc3 = nn.Linear(512, 30)
        self.dp = nn.Dropout(p=0.1)

    def forward(self, x, verbose=False):
        x = self.conv1_1(x)
        x = F.leaky_relu(x,negative_slope=0.1)
        x = self.conv1_bn(x)
        #x = F.leaky_relu(x,negative_slope=0.1)

        x = self.conv1_2(x)
        x = F.leaky_relu(x,negative_slope=0.1)
        x = self.conv1_bn(x)
        #x = F.leaky_relu(x,negative_slope=0.1)

        x = F.max_pool2d(x, kernel_size=2)

        x = self.conv2_1(x)
        x = F.leaky_relu(x,negative_slope=0.1)
        x = self.conv2_bn(x)
        #x = F.leaky_relu(x,negative_slope=0.1)

        x = self.conv2_2(x)
        x = F.leaky_relu(x,negative_slope=0.1)
        x = self.conv2_bn(x)
        #x = F.leaky_relu(x,negative_slope=0.1)

        x = F.max_pool2d(x, kernel_size=2)

        x = self.conv3_1(x)
        x = F.leaky_relu(x,negative_slope=0.1)
        x = self.conv3_bn(x)
        #x = F.leaky_relu(x,negative_slope=0.1)

        x = self.conv3_2(x)
        x = F.leaky_relu(x,negative_slope=0.1)
        x = self.conv3_bn(x)
        #x = F.leaky_relu(x,negative_slope=0.1)

        x = F.max_pool2d(x, kernel_size=2)

        x = self.conv4_1(x)
        x = F.leaky_relu(x,negative_slope=0.1)
        x = self.conv4_bn(x)
        #x = F.leaky_relu(x,negative_slope=0.1)

        x = self.conv4_2(x)
        x = F.leaky_relu(x,negative_slope=0.1)
        x = self.conv4_bn(x)
        #x = F.leaky_relu(x,negative_slope=0.1)

        x = F.max_pool2d(x, kernel_size=2)

        x = self.conv5_1(x)
        x = F.leaky_relu(x,negative_slope=0.1)
        x = self.conv5_bn(x)
        #x = F.leaky_relu(x,negative_slope=0.1)

        x = self.conv5_2(x)
        x = F.leaky_relu(x,negative_slope=0.1)
        x = self.conv5_bn(x)
        #x = F.leaky_relu(x,negative_slope=0.1)

        x = F.max_pool2d(x, kernel_size=2)

        x = self.conv6_1(x)
        x = F.leaky_relu(x,negative_slope=0.1)
        x = self.conv6_bn(x)
        #x = F.leaky_relu(x,negative_slope=0.1)

        x = self.conv6_2(x)
        x = F.leaky_relu(x,negative_slope=0.1)
        x = self.conv6_bn(x)
        #x = F.leaky_relu(x,negative_slope=0.1)

        x = x.view(-1, 512*3*3)

        # now use FC layer with relu
        x = self.fc1(x)
        x = F.relu(x)
        x = self.dp(x)

        #x = self.fc2(x)
        #x = F.relu(x)
        #x = self.dp(x)

        x = self.fc3(x)
        return x
        device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

def testing(model, device, valid_loader):
  model.eval()
  test_loss = 0
  for data, target in valid_loader:
    data, target = data.to(device), target.to(device)
    data = data.view(-1, 1, 96, 96)
    output = model(data)
    loss = criterion(output, target)
    test_loss += loss.item()

  test_loss /= len(valid_loader)
  return test_loss

def training(epochs, model, criterion, device, train_loader, valid_loader, optimizer):
  train_error_list = []
  val_error_list = []
  for epoch in range(epochs):
    model.train()
    train_loss = 0
    for batch_idx, (data, target) in enumerate(train_loader):
      data, target = data.to(device), target.to(device)
      data = data.view(-1, 1, 96, 96)
      optimizer.zero_grad()
      output = model(data)
      loss = criterion(output, target)
      train_loss += loss.item()
      loss.backward()
      optimizer.step()

    train_loss /= len(train_loader)
    eval_loss = testing(model, device, valid_loader)

    train_error_list.append(train_loss)
    val_error_list.append(eval_loss)
    if (epoch+1) % 1 == 0:
      print("End of epoch {}: \nTraining error = [{}]\tValidation error = [{}]".format(epoch+1, train_loss, eval_loss))
  return train_error_list, val_error_list

def train_test_split(X, validation_split):
  dataset_size = len(X)
  indices = list(range(dataset_size))
  val_num = int(np.floor(validation_split*dataset_size))
  np.random.shuffle(indices)
  train_indices, val_indices = indices[val_num:], indices[:val_num]

  train_sampler = SubsetRandomSampler(train_indices)
  valid_sampler = SubsetRandomSampler(val_indices)

  #TensorDataset(x,y)
  #loader_object[i][0]
  dataset = data_utils.TensorDataset(torch.from_numpy(X).float(), torch.from_numpy(y).float())
  train_loader = data_utils.DataLoader(dataset, batch_size=256, sampler=train_sampler)
  valid_loader = data_utils.DataLoader(dataset, batch_size=256, sampler=valid_sampler)
  return train_loader, valid_loader

train_loader, valid_loader = train_test_split(X, 0.2)
model = CNN()
model.to(device)
criterion = torch.nn.MSELoss()
optimizer = optim.Adam(model.parameters())

print('Number of parameters: {}'.format(get_n_params(model)))

train_error_list, valid_error_list = training(80, model, criterion, device, train_loader, valid_loader, optimizer)

result:
End of epoch 70:
Training error = [6.693267138107963]    Validation error = [23.338267008463543]
End of epoch 71:
Training error = [6.269878273424895]    Validation error = [26.488004684448242]
End of epoch 72:
Training error = [6.209241846333379]    Validation error = [17.319510142008465]
End of epoch 73:
Training error = [6.4750041754349414]   Validation error = [17.141239802042644]
End of epoch 74:
Training error = [6.196442832117495]    Validation error = [17.215673605600994]
End of epoch 75:
Training error = [6.49559790155162] Validation error = [26.468823432922363]
End of epoch 76:
Training error = [6.996155572974163]    Validation error = [17.23038117090861]
End of epoch 77:
Training error = [5.591304436973903]    Validation error = [16.537627855936687]
End of epoch 78:
Training error = [5.67340376066125] Validation error = [11.745362917582193]
End of epoch 79:
Training error = [5.76800128687983] Validation error = [9.491617997487387]
End of epoch 80:
Training error = [5.731863332831341]    Validation error = [13.719860871632894]


title: Add Samples after Partial Training in PyTorch

body:
I have trained a model in PyTorch - an RCNN for text classification. The model has very high precision and recall, but I may eventually receive new documents with text unlike what I used to train, validate, or test the model.
I would like to add new text samples to the model without retraining the model from the beginning. This is desirable because I may lose access to some of the text used for initial training.
If it is not possible to add samples (documents), is it possible to train a new model on only the new samples and then somehow combine the original model and the new model? How?
Here is what my model looks like.
RCNN(
  (embeddings): Embedding(10661, 300)
  (lstm): LSTM(300, 64, bidirectional=True)
  (dropout): Dropout(p=0.0, inplace=False)
  (W): Linear(in_features=428, out_features=64, bias=True)
  (tanh): Tanh()
  (fc): Linear(in_features=64, out_features=3, bias=True)
  (softmax): Softmax(dim=1)
  (loss_op): NLLLoss()
)

I am aware of techniques for saving the model and the corresponding load techniques.

State Dictionary:torch.save(model.state_dict(), PATH)
Model:torch.save(model, PATH)
Checkpoint:torch.save({'epoch': EPOCH, 'model_state_dict': net.state_dict(), 'optimizer_state_dict': optimizer.state_dict(), 'loss': LOSS,}, PATH)

I can continue training based on the original samples, but I do not know how to add samples.
If this is something TensorFlow can do by PyTorch cannot, I might switch to TensorFlow.


title: how to save the generated images from this code separated

body:
I have run StarGAN Code from github, this code generate all the generated images in one picture.
How I can save all the generated images separated to single folder? I do not want to save all the images in one picture.
this is how it generate the output (sample image
I want to save the generated images from the trained model, not as samples with all the images in one picture but just a file that contains all the generated images.
This is the part of the code i want to change it
# Translate fixed images for debugging.
if (i+1) % self.sample_step == 0:
    with torch.no_grad():
        x_fake_list = [x_fixed]
        for c_fixed in c_fixed_list:
            x_fake_list.append(self.G(x_fixed, c_fixed))
        x_concat = torch.cat(x_fake_list, dim=3)
        sample_path = os.path.join(self.sample_dir, '{}-images.jpg'.format(i+1))
        save_image(self.denorm(x_concat.data.cpu()), sample_path, nrow=1, padding=0)
        print('Saved real and fake images into {}...'.format(sample_path))


title: Computing the unscaled loss in two different ways yields different results?

body:
So I'm training several deep learning for time series forecasting models in PyTorch. Now most of the models use data that is scaled (e.g. StandardScaler) so that the loss doesn't explode. However, there are times when I want to inverse scale the data. Similarly I have a metric called unscaled_test_loss, however I seem to get two different values for it depending on how I compute it. So for instance if I compute the unscaled_test loss by first inverse scaling the data then computing the loss I get a different number than if I compute the loss then inverse scale it. Is this normal and if so which way is correct? For reference my standard function is this. In this case the data is first inverse scaled then the loss is calculated (if the validation_dataset which contains the original scaler is not none). However when I wrote a unit test that computing the scaled loss first then I inverse scaled it using the same StandardScaler it failed:
def compute_loss(labels, output, src, criterion, validation_dataset, probabilistic=None, output_std=None, m=1):
    # Warning this assumes src target is 1-D
    if not probabilistic and isinstance(output, torch.Tensor):
        if len(labels.shape) != len(output.shape):
            print(labels.shape)
            print(output.shape)
            if len(labels.shape) > 1:
                if labels.shape[1] == output.shape[1]:
                    labels = labels.unsqueeze(2)
                else:
                    labels = labels.unsqueeze(0)
    if probabilistic:
        if type(output_std) != torch.Tensor:
            print("Converted")
            output_std = torch.from_numpy(output_std)
        if type(output) != torch.Tensor:
            output = torch.from_numpy(output)
        output_dist = torch.distributions.Normal(output, output_std)
    if validation_dataset:
        if probabilistic:
            unscaled_out = validation_dataset.inverse_scale(output)
            try:
                output_std = numpy_to_tvar(output_std)
            except Exception:
                pass
            output_dist = torch.distributions.Normal(unscaled_out, output_std)
        else:
            output = validation_dataset.inverse_scale(output.cpu())
            labels = validation_dataset.inverse_scale(labels.cpu())
            src = validation_dataset.inverse_scale(src.cpu())
    if probabilistic:
        loss = -output_dist.log_prob(labels.float()).sum()  # FIX THIS
    elif isinstance(criterion, GaussianLoss):
        g_loss = GaussianLoss(output[0], output[1])
        loss = g_loss(labels)
    elif isinstance(criterion, MASELoss):
        assert len(labels.shape) == len(output.shape)
        loss = criterion(labels.float(), output, src, m)
    else:
        assert len(labels.shape) == len(output.shape)
        assert labels.shape[0] == output.shape[0]
        loss = criterion(output, labels.float())
    return loss

So the unit test essentially just called this function by running
 criterion = torch.nn.MSELoss()
 unscaled_loss = compute_loss(labels, output, criterion, validation_dataset)
 scaled_loss = compute_loss(labels, output, criterion, None) # We pass none here which results in no inverse scaling
 unscaled_loss_2 = validation_dataset.inverse_scale(scaled_loss)
 self.assertAlmostEqual(unscaled_loss_2, scaled_loss) # this test fails

Any help would be appreciated. Full context can be seen here


title: How to obtain the features of the masked regions in Detectron2 MaskRCNN?

body:
What I am trying to do is to extract the features of the predicted masked regions in Detectron2 MaskRCNN.
I have already explored this tutorial and followed the answer in the post. I have obtained the box features, but I want to match only the predicted masks obtained from the DefaultPredictor with their corresponding features.


title: Different behavior of tensors: debugging vs normal execution on gpu

body:
So I have a program (that I didn't write) running on a GPU that is being parallelized with CUDA v10.2. Now there's the following block of code:
1  ctrs = []
2  for icls, cls_id in enumerate(pred_cls_ids):
3      cls_msk = (mask == cls_id)
4      ms = MeanShiftTorch(bandwidth=radius)
5      ctr, ctr_labels = ms.fit(pred_ctr[cls_msk, :])
6      ctrs.append(ctr.detach().contiguous().cpu().numpy())
7      ctrs = torch.from_numpy(np.array(ctrs).astype(np.float32)).cuda()
8  print(ctrs.size())
9  n_ctrs, _ = ctrs.size()

When running this I get the following error:
  File "/media/datasets/benjamin/repos/PVN3D/pvn3d/lib/utils/pvn3d_eval_utils.py", line 60, in cal_frame_poses
    n_ctrs, _ = ctrs.size()
ValueError: not enough values to unpack (expected 2, got 1)

which makes sense with the output of the print statement in line 8 being:
torch.Size([0])

A few resources on the internet talking about tensors and tensor.shape/tensor.size() suggest casting the tensor.size() to a list, meaning in my case I do the following:
8  print(list(ctrs.size()))
9  n_ctrs, _ = list(ctrs.size())

This produces the same error though:
File "/media/datasets/benjamin/repos/PVN3D/pvn3d/lib/utils/pvn3d_eval_utils.py", line 60, in cal_frame_poses
    n_ctrs, _ = list(ctrs.size())
ValueError: not enough values to unpack (expected 2, got 1)

with the output of line 8 now being
[0]

If I understand that right the problem is not the type of my output, but I need to "assemble" my parallelized tensors to get the actual size of ctrs.
I come to this conclusion, because I can run this program while debugging without any value errors of that sort.
the output of the print statement while debugging is (for example):
[2, 3]

Is there any change that I can make to that code to get the correct size of the data without doing changes that potentially alter behavior of the code coming after? Something like putting together those parallelized tensors for one line of code to get the size and then immediately parallelizing them again.


title: Pytorch - mass classification

body:
I trained a model to classify picture into 2 classes.
I wanna use it to order my iPhone images, I have got a lot. So I moved the trained model on my MacBook and started some tests.
Everything works as expected as long a the image folder contains not more than 223 pictures.
It seems to have something to do with the size of the dataset, I can reproduce the issue with the following code.
I tried to split the pictures into batches but I could not figure out how. What is the common fix to that issue? Is it possible to use a Dataloader?
test_dataset = ImageFolderWithPaths('/Users/daniel/Gimp/test-ds/test/', test_trans)

# check images
for img, label, data in test_dataset:
     print(img.shape, label, data)
     data_path = '/Users/daniel/Gimp/test-ds/test/'
     data_path = '/Users/daniel/Pictures/Photos1/'
     data_file = data.split('/')[-1]
     data_file_new = 'class0_' + data.split('/')[-1]
     print(data_path + data_file_new)

Output
torch.Size([3, 64, 64]) 0 /Users/daniel/Gimp/test-ds/test/3/020B1786-E77D-47E8-B0F1-DD72313546B7.jpeg
/Users/daniel/Pictures/Photos1/class0_020B1786-E77D-47E8-B0F1-DD72313546B7.jpeg

torch.Size([3, 64, 64]) 0 /Users/daniel/Gimp/test-ds/test/3/020BC6D0-2AC0-4BF3-A540-6630BA7200ED.jpeg
/Users/daniel/Pictures/Photos1/class0_020BC6D0-2AC0-4BF3-A540-6630BA7200ED.jpeg
torch.Size([3, 64, 64]) 0 /Users/daniel/Gimp/test-ds/test/3/021453CC-CAC1-4546-8188-20739F3C778F.jpeg
/Users/daniel/Pictures/Photos1/class0_021453CC-CAC1-4546-8188-20739F3C778F.jpeg
torch.Size([3, 64, 64]) 0 /Users/daniel/Gimp/test-ds/test/3/0235EEF0-2831-46D3-8889-8E57CDF42205.jpeg
/Users/daniel/Pictures/Photos1/class0_0235EEF0-2831-46D3-8889-8E57CDF42205.jpeg
torch.Size([3, 64, 64]) 0 /Users/daniel/Gimp/test-ds/test/3/02398B72-6F73-49D6-843B-25E2A2593DA9.jpeg
/Users/daniel/Pictures/Photos1/class0_02398B72-6F73-49D6-843B-25E2A2593DA9.jpeg

Prediction Method
class pred01():
  def __init__(self, imgpath, model):
    cc = 0
    cc1 = 0
    cc0 = 0
    cc2 = 0
    """Predict image class"""
    self.ds = ImageFolderWithPaths(imgpath, transform=test_trans)
    for img, label, data in self.ds:
        self.xb = to_device(img.unsqueeze(0), device)
        self.yb = model(self.xb)
        _, self.preds = torch.max(self.yb, dim=1)
        self.preds1 = torch.softmax(self.yb, dim=1).max()
        cc += 1
        if dataset.classes[self.preds[0].item()] == '0' and self.preds1.item() > 0.80 or self.preds1.item() == 1.00:
            #print('normal pic')
            cc0 += 1
        #    pass
        elif dataset.classes[self.preds[0].item()] == '1' and self.preds1.item() > 0.80 or self.preds1.item() == 1.00:
            #print('class pic, new name: ' 'class1_' + data.split('/')[-1])
            #os.rename(data, '/Users/daniel/Pictures/Photos1/1/' + 'duolingo_' + data.split('/')[-1])
            cc1 += 1
        else:
            pass
            #print('not sure')
            cc2 += 1
    print(cc)
    print(f'cc0: {cc0}')
    print(f'cc1: {cc1}')
    print(f'cc2: {cc2}')

Results after 3 runs
365
cc0: 152
cc1: 195
cc2: 18

365
cc0: 154
cc1: 186
cc2: 25

365
cc0: 136
cc1: 206
cc2: 23


title: Question regarding focal loss function implementation

body:
In the paper introducing focal loss, they state that the loss function is formulated as such:

Where

I found an implementation of it on a Github page from another author who used it in their paper. I tried the function out on a segmentation problem dataset I have and it seems to work quite well, but the implementation seems odd to me.
Below is the implementation:
def binary_focal_loss(pred, truth, gamma=2., alpha=.25):
    eps = 1e-8
    pred = nn.Softmax(1)(pred)
    truth = F.one_hot(truth, num_classes = pred.shape[1]).permute(0,3,1,2).contiguous()

    pt_1 = torch.where(truth == 1, pred, torch.ones_like(pred))
    pt_0 = torch.where(truth == 0, pred, torch.zeros_like(pred))

    pt_1 = torch.clamp(pt_1, eps, 1. - eps)
    pt_0 = torch.clamp(pt_0, eps, 1. - eps)

    out1 = -torch.mean(alpha * torch.pow(1. - pt_1, gamma) * torch.log(pt_1))
    out0 = -torch.mean((1 - alpha) * torch.pow(pt_0, gamma) * torch.log(1. - pt_0))

    return out1 + out0

The part that I don't understand is the calculation of out0. Since the paper defines the Pt value when y =/= 1 as 1-p, I would have expected out0 to be written as this instead:
out0 = -torch.mean((1 - alpha) * torch.pow((1 - (1 - pt_0)), gamma) * torch.log(1. - pt_0))

instead of
out0 = -torch.mean((1 - alpha) * torch.pow(pt_0, gamma) * torch.log(1. - pt_0))

Can anyone help explain this to me?


title: Feed neural network: IndexError: index out of range

body:
I am new to PyTorch and try to feed the LSTM model of this tutorial:
https://blog.fastforwardlabs.com/2018/04/10/pytorch-for-recommenders-101.html
I copied the code for the model and the training but doesn't work for me now.
model:
class LSTMRating(nn.Module):

def __init__(self, num_items, embedding_dim=64, hidden_dim=128, num_output=1):
    super(LSTMRating, self).__init__()
    self.hidden_dim = hidden_dim
    self.item_embeddings = nn.Embedding(num_items, embedding_dim)
    self.lstm = nn.LSTM(embedding_dim, hidden_dim)
    self.linear = nn.Linear(hidden_dim, num_output)
    self.hidden = self.init_hidden()

def init_hidden(self):
    # initialize both hidden layers
    return (Variable(torch.zeros(1, 1, self.hidden_dim)),
            Variable(torch.zeros(1, 1, self.hidden_dim)))

def forward(self, sequence):
    embeddings = self.item_embeddings(sequence)
    output, self.hidden = self.lstm(embeddings.view(len(sequence), 1, -1),
                                    self.hidden)
    rating_scores = self.linear(output.view(len(sequence), -1))
    return rating_scores

def predict(self, sequence):
    rating_scores = self.forward(sequence)
    return rating_scores

training:
embedding_dim = 64
hidden_dim = 128
n_output = 1

# add one to represent padding when there is not enough history
model = LSTMRating(embedding_dim, hidden_dim, n_items+1, n_output)
loss_fn = nn.MSELoss()
optimizer = optim.SGD(model.parameters(), lr=0.1)

for sequence, target_ratings in training_data:
    model.zero_grad()
    # initialize hidden layers
    model.hidden = model.init_hidden()
    # convert sequence to PyTorch variables
    sequence_var = Variable(torch.LongTensor(sequence.astype('int64')))
    # forward pass
    ratings_scores = model(sequence_var)
    target_ratings_var = Variable(torch.FloatTensor(target_ratings.astype('float32')))
    # compute loss
    loss = loss_fn(ratings_scores, target_ratings_var)
    # backpropagate
    loss.backward()
    # update weights
    optimizer.step()

When i run the code, i get the following error:
Traceback (most recent call last):
    ratings_scores = local_model.predict(sequence_var)
  File "models.py", line 82, in predict
    rating_scores = self.forward(sequence)
  File "models.py", line 75, in forward
    embeddings = self.item_embeddings(sequence)
  File "\torch\nn\modules\module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "\torch\nn\modules\sparse.py", line 124, in forward
    return F.embedding(
  File "\torch\nn\functional.py", line 1852, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
IndexError: index out of range in self

I understand that something is wrong with the dimension of the tensor sequence_var. On the webpage, they say that the input to the model should be of dimension 10x1x20. I tried to reshape it to the dimension, but doesn't work - i can only cast it to a 2D tensor. I tried squeeze, view and reshape.
The relevant parameters look like follows:
training data:
[[array([209,  32, 189, 242, 171, 111, 256,   5,  74, 102]), array([4, 5, 3, 5, 5, 5, 4, 3, 1, 2])]]

tensor sequence_var:
tensor([209,  32, 189, 242, 171, 111, 256,   5,  74, 102])


title: model loss does not changing in pytorch

body:
Im dealing with titanic data with pytorch
these are my model & training code
import torch.nn.functional as F

class Net(torch.nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1_1=torch.nn.Linear(13, 512)
        self.fc1_2=torch.nn.Linear(512, 64)
        self.fc1_3=torch.nn.Linear(64, 10)

        self.fc2_1=torch.nn.Linear(13, 64)
        self.fc2_2=torch.nn.Linear(64, 512)
        self.fc2_3=torch.nn.Linear(512, 10)

        self.fc3_1=torch.nn.Linear(13, 128)
        self.fc3_2=torch.nn.Linear(128, 128)
        self.fc3_3=torch.nn.Linear(128, 10)

        self.fc_full_1=torch.nn.Linear(30, 64)
        self.fc_full_2=torch.nn.Linear(64, 128)
        self.fc_full_3=torch.nn.Linear(128, 2)

    def forward(self, x):
        x1=self.fc1_1(x)
        x1=F.relu(x1)
        x1=self.fc1_2(x1)
        x1=F.relu(x1)
        x1=self.fc1_3(x1)
        x1=F.relu(x1)

        x2=self.fc2_1(x)
        x2=F.relu(x2)
        x2=self.fc2_2(x2)
        x2=F.relu(x2)
        x2=self.fc2_3(x2)
        x2=F.relu(x2)

        x3=self.fc3_1(x)
        x3=F.relu(x3)
        x3=self.fc3_2(x3)
        x3=F.relu(x3)
        x3=self.fc3_3(x3)
        x3=F.relu(x3)

        x=torch.cat((x1, x2, x3), dim=1)
        x=self.fc_full_1(x)
        x=F.relu(x)
        x=self.fc_full_2(x)
        x=F.relu(x)
        x=self.fc_full_3(x)

        return x

model=Net()

as seen above, they are just fully connected layers
model loss function and optimization
cross ehtropy loss and adam
criterion = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model1.parameters(), lr=0.05)

these are training code
for epoch in range(100):
    model.train()
    x_var = Variable(torch.FloatTensor(x_train))
    y_var = Variable(torch.LongTensor(y_train))

    optimizer.zero_grad()
    train_pred = model(x_var)
    loss =criterion(train_pred, y_var)
    loss.backward()
    optimizer.step()

    train_acc=calc_accuracy(train_pred, y_var)
    loss=loss.data.numpy()

and finaly, the accuracy and loss printed
Epoch  0
0.6900209 0.531578947368421
valid:   0.692668 0.4621212121212121
Epoch  10
0.6900209 0.531578947368421
valid:   0.692668 0.4621212121212121
Epoch  20
0.6900209 0.531578947368421
valid:   0.692668 0.4621212121212121
Epoch  30
0.6900209 0.531578947368421
valid:   0.692668 0.4621212121212121
Epoch  40
0.6900209 0.531578947368421
valid:   0.692668 0.4621212121212121
Epoch  50
0.6900209 0.531578947368421
valid:   0.692668 0.4621212121212121
Epoch  60
0.6900209 0.531578947368421
valid:   0.692668 0.4621212121212121
Epoch  70
0.6900209 0.531578947368421
valid:   0.692668 0.4621212121212121
Epoch  80
0.6900209 0.531578947368421
valid:   0.692668 0.4621212121212121
Epoch  90
0.6900209 0.531578947368421
valid:   0.692668 0.4621212121212121

as seen above, model training loss and valid loss does not change at all.
what does seems to be the problem?


title: Broken For Loop when Training CNN

body:
I've built a relatively deep layered CNN which takes in 224x224 images. I have a trainset of ~5000 images which I'm using to train my CNN using the conventional for-loop method. However, I have an issue where the loop only prints the outcome of the first epoch and then just continues the run (i.e. the cell stays Executing forever).
My code with output:
#Model training

epochs = 2

for epoch in range(epochs):

    running_loss = 0

    #Train
    cnn.train()

    for images, labels in trainloader:
        if cuda:
            images, labels = images.to(device), labels.to(device)

        optimiser.zero_grad()
        outputs = cnn(images)
        loss = criterion(outputs, labels)
        running_loss += loss.item() * images.size(0)
        loss.backward()
        optimiser.step()

    epochs_train_loss = running_loss / len(trainloader.dataset)
    print ("Epoch {}, Training Loss: {}".format(epoch, epochs_train_loss))

Output:
Epoch 0, Training Loss: 0.3014729377767928


I'm having issues applying a Regex expression to a Split() operation found in the HuggingFace Library. The library requests the following input for Split().

pattern (str or Regex) – A pattern used to split the string. Usually a
string or a Regex

In my code I am applying the Split() operation like so:
tokenizer.pre_tokenizer = Split(pattern="[A-Z]+", behavior='isolated')

but it's not working because [A-Z]+ is being interpreted as  a string not a Regex expression. I've used the following to no avail:
pattern = re.compile("[A-Z]+")
tokenizer.pre_tokenizer = Split(pattern=pattern, behavior='isolated')

Getting the following error:

TypeError: Can't convert re.compile('[A-Z]+') (re.Pattern) to Union[str, tokenizers.Regex]


title: What's the best way to take a list of lists as input in deep Q-network algorithm?

body:
I have my own environment for the DQN algorithm. In my environment, the state space is represented by a list of lists, where each sublist can be of different lengths. In my case, the length of the global list is 300 and the length of each of the sublists varies from 0 to 10. What is the best way to use such state representation as a DQN input if I want to use the PyTorch platform?
#exapmle state with only 4 sublists
state = [[1,2,3,4], [1,20,20], [10], [20,4,5,6,7]]


title: How GNN layers can know which graph in a batch has the given node & node feature?

body:
When we pass input as node features (x) and edge index (edge_index) to pytorch_geometric layer (e.g. GATConv), I am worried whether the layer can differentiate which batch sample the given node elements belong to.
x follows the shape [num of nodes, feature size] and edge_index follows shape [2, num of edges]. However, these 2 do not have the given information to know which input graph of batch size 32 have given node feature in the x.
Anyone can clarify on this ?


title: Error in running backwards() function in PyTorch

body:
The code:
 import numpy as np
 predictors = np.array([[73,67,43],[91,88,64],[87,134,58],[102,43,37],[69,96,70]],dtype='float32')
 outputs = np.array([[56,70],[81,101],[119,133],[22,37],[103,119]],dtype='float32')

 inputs = torch.from_numpy(predictors)
 targets = torch.from_numpy(outputs)

 weights = torch.randn(2,3,requires_grad=True)
 biases = torch.randn(2,requires_grad=True)

 def loss_mse(x,y):
  d = x-y
  return torch.sum(d*d)/d.numel()

 def model(w,b,x):
  return x @ w.t() +b

 def train(x,y,w,b,lr,e):
  w = torch.tensor(w,requires_grad=True)
  b = torch.tensor(b,requires_grad=True)
  for epoch in range(e):
    preds = model(w,b,x)
    loss = loss_mse(y,preds)
    if epoch%5 == 0:
      print("Loss at Epoch [{}/{}] is {}".format(epoch,e,loss))
    #loss.requires_grad=True
    loss.backward()
    with torch.no_grad():
      w = w - lr*w.grad
      b = b - lr*b.grad
      w.grad.zero_()
      b.grad.zero_()

 train(inputs,targets,weights,biases,1e-5,100)

Running this gives different errors. Once it gave the error that loss was of size 0. Then it gave the error in the update line w = w-lr*w.grad that float can't be subtracted from NoneType.


title: Pytorch - Loss for Object Localization

body:
I am trying to perform an object localization task with MNIST based on Andrew Ng's lecture here. I am taking the MNIST digits and randomly placing them into a 90x90 shaped image and predicting the digit and it's center point. When I train, I am getting very poor results and my question is about whether or not my loss function is set up correctly. I basically just take the CrossEntropy for the digit, the MSE for the coordinates, and then add them all up. Is this correct? I don't get any errors, but the performance is just horrendous.
My dataset is defined as follows (which returns the label and the x y coordinates of the center of the digit):
class CustomMnistDataset_OL(Dataset):

def __init__(self, df, test=False):
    '''
    df is a pandas dataframe with 28x28 columns for each pixel value in MNIST
    '''
    self.df = df
    self.test = test

def __len__(self):
    return len(self.df)

def __getitem__(self, idx):
    if self.test:
        image = np.reshape(np.array(self.df.iloc[idx,:]), (28,28)) / 255.
    else:
        image = np.reshape(np.array(self.df.iloc[idx,1:]), (28,28)) / 255.

    # create the new image
    new_img = np.zeros((90, 90)) # images will be 90x90
    # randomly select a bottom left corner to use for img
    x_min, y_min = randrange(90 - image.shape[0]), randrange(90 - image.shape[0])
    x_max, y_max = x_min + image.shape[0], y_min + image.shape[0]

    x_center = x_min + (x_max-x_min)/2
    y_center = y_min + (y_max-x_min)/2

    new_img[x_min:x_max, y_min:y_max] = image

    label = [int(self.df.iloc[idx,0]), x_center, y_center] # the label consists of the digit and the center of the number
    sample = {"image": new_img, "label": label}

    return sample['image'], sample['label']

My training function is set up as follows:
loss_fn = nn.CrossEntropyLoss()
loss_mse = nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)

def train(dataloader, model, loss_fn, loss_mse, optimizer):
    model.train() # very important... This turns the model back to training mode
    size = len(train_dataloader.dataset)
    for batch, (X, y) in enumerate(dataloader):
        X, y0, y1, y2 = X.to(device), y[0].to(device), y[1].to(device), y[2].to(device)

        pred = model(X.float())
        # DEFINE LOSS HERE -------
        loss = loss_fn(pred[0], y0) + loss_mse(pred[1], y1.float()) + loss_mse(pred[2], y2.float())

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if batch % 100 == 0:
            loss, current = loss.item(), batch*len(X)
            print(f"loss: {loss:>7f}  [{current:>5d}/{size:>5d}]")


title: Pytorch: Parallel LSTMs with differing sequence lengths

body:
I'm building a deep model in PyTorch but I'm fairly new to it. I’ve built a couple models before in PyTorch / TF, but this particular question exceeded my googling capabilities. Apologies if the answer is obvious, I might not have known what to search for.
I'm trying to build a model with an array of RNNs (LSTMs, in particular) that operate in parallel. Here's a diagram:

Each RNN will have an input shape of (N_batch, seq_length, 2), so the the overall input tensor to the full stack of RNNs would be of shape (N_batch, seq_length, 2, N_RNNs). I want to run the RNNs all in parallel. I'm able to build up this portion of the model looping over the RNNs and then concatenating, them, but surely there's a better way.
An additional wrinkle: some elements of each RNN input sequence might be NaN, and which elements are NaN will not be consistent across a given batch or RNN time step. When any single RNN encounters a NaN, I just want it to skip that element of the input sequence (operate as if the element hadn’t been there at all).
Help? Please and thank you.


title: How to detect all GPU/CPU transfers in pytorch?

body:
I have a large pytorch project. How do I check if the project is performing any unexpected transfers of data from the GPU to the CPU or back?
As I understand it, GPU/CPU transfers are very costly for performance, and they can easily happen by accident if you are not very careful with your code. For example, calling .item() on a tensor on the GPU will cause that tensor to be transferred back to CPU, while blocking CPU execution.
Since there are a lot of ways to do this by accident, I need a reliable way to get all places in my code where data is transferred, so that I can prevent costly performance losses.
Ideally, I would be able to specify a section of code with a Context Manager that says "GPU/CPU transfers are allowed inside this block only. Any attempt to transfer data between GPU and CPU outside of this block causes an error."


title: Connection error when downloading EMNIST with pytorch on google colab

body:
The problem is very easy to re-create, when I open a new google colab notebook and write the following in the first cell:
from torchvision import datasets
d = datasets.EMNIST('data', split='digits', download=True)

I am getting the following error:
ConnectionError: HTTPConnectionPool(host='www.itl.nist.gov', port=80):
Max retries exceeded with url: /iaui/vip/cs_links/EMNIST/gzip.zip
(Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd2c4c6b110>: Failed to establish a new connection: [Errno 110] Connection timed out'))

My runtime type has no accelerator (GPU).
I read online a bit and saw some github discussions but I did not see any solution (Connecting from incognito does not help).
Please help me solve this connection error.

A small edit:
I tried to also download the MNIST dataset and I am now getting a 503 error or it runs for a very very long time (not forever though).
HTTPError: HTTP Error 503: Service Unavailable


title: PyTorch shows the error " 'NoneType' object has no attribute 'zero_' " when calling the zero_ method

body:
Can someone please answer why my code is showing an error.
Thanks in advance.
Code:
import torch
torch.manual_seed(0)

a = torch.rand((1, 3), requires_grad = True)
w1 = torch.rand((3, 3), requires_grad = True)
w2 = torch.rand((3, 1), requires_grad = True)

d = torch.matmul(torch.matmul(a, w1), w2)

L = (10 - d)

L.backward()

w1 = w1 - w1.grad*0.001
w1.grad.zero_()

Error:
AttributeError
'NoneType' object has no attribute 'zero_'


title: Is it possible to store PyTorch dataLoader object into a file or something?

body:
I'm working on a machine learning project, and the dataset I'm using is quite big. It takes me 20+ minutes everytime I run my getLoaders function (which just calls Imagefolder(), splits data and calls torch.utils.data.DataLoader())
loader = torch.utils.data.DataLoader(test_pair_list, batch_size=batchSize, sampler=valSampler)

So is there a way to store these objects into a file so that getLoader will only be called once? I went through PyTorch documentation but didn't have any luck there.


title: Validation accuracy does not increase for binary classification using GNN

body:
I am trying to perform graph classification with GINConv model of GNN. I have tried everything from varying dropouts to weight decay (for L2 regularization), learning rate (1e-6 to 1e-3), batch normalization to varying batch sets. It still doesn't show any change. My dataset is a set of 8000 graphs (with varying vertices and edges) that contain label 0 or 1. There are 4000 graphs with label 0 and 4000 with label 1. I am using Adam optimizer.There are 6 layers in my model. Here is my code :
def __init__(self):
    super(Net, self).__init__()

    self.dropout = 0.3
    self.first_h = []
    self.nns = []
    self.convs = []
    self.linears = []
    self.pooling = global_mean_pool

    #layer 0
    self.first_h = Sequential(Linear(60, 50), BatchNorm1d(50), ReLU(),
        Linear(50, 50), BatchNorm1d(50), ReLU())
    self.linears.append(Linear(50, 2))
    #rest
    i = 50
    while(i >= 20):
        j = i - 10
        self.nns.append(Sequential(Linear(i, j), BatchNorm1d(j), ReLU(),
        Linear(j, j), BatchNorm1d(j), ReLU()))
        self.convs.append(GINConv(self.nns[-1], train_eps=9))
        self.linears.append(Linear(j, 2))
        i = j

    self.nns.append(Sequential(Linear(10, 2), BatchNorm1d(2), ReLU(),
    Linear(2, 2), BatchNorm1d(2), ReLU()))
    self.convs.append(GINConv(self.nns[-1], train_eps=9))
    self.linears.append(Linear(2, 2))

    self.nns = torch.nn.ModuleList(self.nns)
    self.convs = torch.nn.ModuleList(self.convs)
    self.linears = torch.nn.ModuleList(self.linears)

def forward(self, data, flag):
    out = 0
    x, edge_index, batch = data.x, data.edge_index, data.batch
    for layer in range(6):
        if layer == 0:
            x = self.first_h(x)
            out += F.dropout(self.pooling(self.linears[layer](x), batch), p=self.dropout)
        else:
            x = self.convs[layer-1](x, edge_index)
            out += F.dropout(self.linears[layer](self.pooling(x, batch)), p=self.dropout, training=self.training)
    return out

Optimizer and train/valid/test splits :
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3, weight_decay=1e-5)
train_loader = DataLoader(data_list[0:8000], batch_size=8000, shuffle = True)
valid_loader = DataLoader(data_list[8000:9000], batch_size=1000, shuffle = True)
test_loader = DataLoader(data_list[9000:10000], batch_size=1000, shuffle = True)

Train code:
for epoch in range(1000):
    if(validVal < 0.4 and trainVal < 0.3):
        break
    trainVal = 0
    model.train()
    for batch,label in train_loader:
        optimizer.zero_grad()
        batch = batch.to(torch.device(device))
        out = model(batch,1)
        label = label.to(torch.device(device))
        loss = F.cross_entropy(out, label)
        trainVal += loss.cpu().detach().numpy()
        loss.backward()
        optimizer.step()
    model.eval()
    for batch,label in valid_loader:
        batch = batch.to(torch.device(device))
        out = model(batch,0)
        label = label.to(torch.device(device))
        loss = F.cross_entropy(out, label)
        validVal = loss.cpu().detach().numpy()
        if(minValid > loss.cpu().detach().numpy()):
            minValid = loss.cpu().detach().numpy()

I have tested by varying number of epochs as well. I have even tried mapping the problem into one class and using BCELosswithLogits. I have tried almost every possible scenario to prevent overfitting, but it does not seem to work. Any suggestions how to go about it?
This is my obtained curve :


title: Large datasets and Cuda memory Issue

body:
I was processing a large dataset and ran into this error: "RuntimeError: CUDA out of memory. Tried to allocate 1.35 GiB (GPU 0; 8.00 GiB total capacity; 3.45 GiB already allocated; 1.20 GiB free; 4.79 GiB reserved in total by PyTorch).
Any thought on how to solve this?


title: How to speed up the process of loading data while training my neural network?

body:
I am now training my LSTM neural network with my GPU.
The problem is:
I have 23,000 .csv files in my training set, which has a shape of (40,76). Each time I got a batch(64) to load my data, I found that it took about 1s to load the data(read 64 .csv files), and it took about 0.08s to compute the loss and update the parameters. When I checked the power and utilization of my GPU, I found it was of low efficiency. Therefore, how can I improve the organization of my training data?
Here is my own dataset class.enter image description here


title: AttributeError in torch_geometric.transforms

body:
I have a problem that I cannot understand: even though a module ‘torch_geometric.transforms’ has an attribute ‘AddTrainValTestMask’ according to documentation , I cannot import it. I keep receiving an error AttributeError: module 'torch_geometric.transforms' has no attribute 'AddTrainValTestMask
My Pytorch version is 1.7.1
I took the code from here
Minimum reproducible example:
import os.path as osp

import torch
import torch.nn.functional as F
from torch_geometric.datasets import Planetoid
import torch_geometric.transforms as T
from torch_geometric.nn import SplineConv

dataset = 'Cora'
transform = T.Compose([
    T.AddTrainValTestMask('train_rest', num_val=500, num_test=500),
    T.TargetIndegree(),
])
path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', dataset)
dataset = Planetoid(path, dataset, transform=transform)
data = dataset[0]

class Net(torch.nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = SplineConv(dataset.num_features, 16, dim=1, kernel_size=2)
        self.conv2 = SplineConv(16, dataset.num_classes, dim=1, kernel_size=2)

    def forward(self):
        x, edge_index, edge_attr = data.x, data.edge_index, data.edge_attr
        x = F.dropout(x, training=self.training)
        x = F.elu(self.conv1(x, edge_index, edge_attr))
        x = F.dropout(x, training=self.training)
        x = self.conv2(x, edge_index, edge_attr)
        return F.log_softmax(x, dim=1)

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model, data = Net().to(device), data.to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.01, weight_decay=5e-3)

def train():
    model.train()
    optimizer.zero_grad()
    F.nll_loss(model()[data.train_mask], data.y[data.train_mask]).backward()
    optimizer.step()

def test():
    model.eval()
    log_probs, accs = model(), []
    for _, mask in data('train_mask', 'test_mask'):
        pred = log_probs[mask].max(1)[1]
        acc = pred.eq(data.y[mask]).sum().item() / mask.sum().item()
        accs.append(acc)
    return accs

for epoch in range(1, 201):
    train()
    log = 'Epoch: {:03d}, Train: {:.4f}, Test: {:.4f}'
    print(log.format(epoch, *test()))

Can anybody explain to me the problem?


title: What is the default settings (e.g. hyperparameters) for MatLab's feedforwardnet?

body:
My code is very simple for one layer of 20 neurons:
FFNN = feedforwardnet(20);
FFNN_trained = train(FFNN,x,y);

This can result in a very good performance in a few hundreds of epochs. I want to reproduce it in Pytorch, so I need to know the details, e.g. learning rate, activation function, optimizer, batch size, when to stop, etc. Also, the data splitting for training/validation/testing seems to be random in feedforwardnet.
Where can I find these details for feedforwardnet? How to specify the training/validation/testing in feedforwardnet?
Thank you for the help. I realise that the levenberg-marquardt method is not available in Pytorch.