title: Back-Propagation of y = x / sum(x, dim=0) where size of tensor x is (H,W)

body:
Q1.
I'm trying to make my custom autograd function with pytorch.
But I had a problem with making analytical back propagation with  y = x / sum(x, dim=0)
where size of tensor x is (Height, Width) (x is 2-dimensional).
Here's my code
class MyFunc(torch.autograd.Function):
@staticmethod
def forward(ctx, input):
  ctx.save_for_backward(input)
  input = input / torch.sum(input, dim=0)

  return input

@staticmethod
def backward(ctx, grad_output):
  input = ctx.saved_tensors[0]
  H, W = input.size()
  sum = torch.sum(input, dim=0)
  grad_input = grad_output * (1/sum - input*1/sum**2)

  return grad_input

I used (torch.autograd import) gradcheck to compare Jacobian matrix,
from torch.autograd import gradcheck
func = MyFunc.apply
input = (torch.randn(3,3,dtype=torch.double,requires_grad=True))
test = gradcheck(func, input)

and the result was

Please someone help me to get correct back propagation result
Thanks!

Q2.
Thanks for answers!
Because of your help, I could implement back propagation in case of (H,W) tensor.
However, while I implemented back propagation in case of (N,H,W) tensor, I got a problem.
I think the problem would be initializing new tensor.
Here's my new code
import torch
import torch.nn as nn
import torch.nn.functional as F

class MyFunc(torch.autograd.Function):
  @staticmethod
  def forward(ctx, input):
    ctx.save_for_backward(input)

    N = input.size(0)
    for n in range(N):
      input[n] /= torch.sum(input[n], dim=0)

    return input

  @staticmethod
  def backward(ctx, grad_output):
    input = ctx.saved_tensors[0]
    N, H, W = input.size()
    I = torch.eye(H).unsqueeze(-1)
    sum = input.sum(1)

    grad_input = torch.zeros((N,H,W), dtype = torch.double, requires_grad=True)
    for n in range(N):
      grad_input[n] = ((sum[n] * I - input[n]) * grad_output[n] / sum[n]**2).sum(1)

    return grad_input

Gradcheck code is
from torch.autograd import gradcheck
func = MyFunc.apply
input = (torch.rand(2,2,2,dtype=torch.double,requires_grad=True))
test = gradcheck(func, input)
print(test)

and result is
enter image description here
I don't know why the error occurs...
Your help will be very helpful for me to implement my own convolutional network.
Thanks! Have a nice day.


title: how to solve Confusion matrix wrong predictions problem?

body:
I have trained a model using Pytorch where I have splitted the total training dataset for test and validation (70% of the total dataset for training, remaining 15 and 15 for test and validation). Also I have a separate folder where I have test images (having 27 images) which is used to test and predict individual images after training. The model gives accuracy of 75% while training and when I test the model with individual images that I have kept in test folder predicts the images accurately. I have tried plotting the confusion matrix, using the dataloaders of the test image from that test folder as well as from the test set that was created by splitting 15% of the total training set. Here is when the problem occurs. The image labels that were correctly predicted while testing individual images appears least accurate or not accurate and those that were incorrectly predicted while testing appears most accurate.
Here is the code to the dataloader after spliting,
from torch.utils.data.dataloader import DataLoader
batch_size = 64
train_dl = DataLoader(trainds, batch_size, shuffle=True, num_workers=4, pin_memory=True)
val_dl = DataLoader(val_ds, batch_size*2, shuffle=False, num_workers=4, pin_memory=True)
test_dl = DataLoader(testds, batch_size*2, shuffle=False, num_workers=4, pin_memory=True)

Code to test individual images:
def predict_image(img, model):
    #convert to a batch of i
    xb = to_device(img.unsqueeze(0), device)
    #get prediction from model
    yb = model(xb)
    #Pick index with hihest probability
    _, preds = torch.max(yb, dim=1)
    #Retrieve the class label
    return preds[0].item()

    img, label = test_ds[1]
    plt.imshow(img.permute(1,2,0))
    print('Label:',label, ",Predicted:", predict_image(img,model))
o/p -> label 0, predicted 4

 img, label = test_ds[5]
    plt.imshow(img.permute(1,2,0))
    print('Label:',label, ",Predicted:", predict_image(img,model))
o/p -> label 3, predicted 3

And here is the code to the confusion matrix

import sklearn.metrics as skmetrics
from sklearn.metrics import classification_report
import seaborn as sns
%matplotlib inline

def confusion_matrix(model, test_dl, normalize=True):
    model.eval()
    true_labels = []
    predictions = []
    with torch.no_grad():
        for images, labels in test_dl:
            images, labels = images.to(device), labels.to(device)
            outputs = model(images)
            predicted = torch.argmax(outputs.data,-1)
            true_labels.append(labels.cpu().numpy())

            predictions.append(predicted.cpu().numpy())
    true_labels = np.hstack(true_labels)
    predictions = np.hstack(predictions)

    return true_labels, predictions

true_labels, predictions = confusion_matrix(model, test_dl, normalize=True)
c_matrix = skmetrics.confusion_matrix(true_labels, predictions)
plt.figure(figsize=(6, 6))
plt.title("Confusion matrix")
sns.heatmap(c_matrix, cmap='Blues', annot=True, xticklabels=classes, yticklabels=classes, fmt='g', cbar=True)
plt.xlabel('predictions')
plt.ylabel('true labels')

The plot looks like this
All images of Rs. 20, Rs. 10 and Rs. 500 were predicted exactly while testing with each individual images of these classes, but it shows that Rs. 20, 500 are less accurate, while Rs. 100 and fake 100 were not predicted correctly but the plot shows that they are accurate..Is something wrong with the code, or have I mistaken somewhere?


title: Select specific indexes of 3D Pytorch Tensor using a 1D long tensor that represents indexes

body:
So I have a tensor that is M x B x C, where M is the number of models, B is the batch and C is the classes and each cell is the probability of a class for a given model and batch. Then I have a tensor of the correct answers which is just a 1D of size B we'll call "t". How do I use the 1D of size B to just return a M x B x 1, where the returned tensor is just the value at the correct class? Say the M x B x C tensor is called "blah" I've tried
blah[:, :, C]

for i in range(M):
    blah[i, :, C]

blah[:, C, :]

The top 2 just return the values of indexes t in the 3rd dimension of every slice. The last one returns the values at t indexes in the 2nd dimension. How do I do this?


title: Can the verification performance be output from MNIST training data and test data?

body:
import torchvision.datasets as dsets
import torchvision.transforms as transforms
import torch.nn.init
import torch.nn.functional as F
# from sklearn.linear_model import SGDClassifier # Test i did
# from sklearn.model_selection import cross_val_score

device = 'cuda' if torch.cuda.is_available() else 'cpu'
print(device)

learning_rate = 0.001
training_epochs = 15
batch_size = 100

mnist_train = dsets.MNIST(root='MNIST_data/', # Specify download path
                          train=True, # Specify True to download as training data
                          transform=transforms.ToTensor(), # Convert to tensor
                          download=True)

mnist_test = dsets.MNIST(root='MNIST_data/', # Specify download path
                         train=False, # If false is specified, download as test data
                         transform=transforms.ToTensor(), # Convert to tensor
                         download=True)

data_loader = torch.utils.data.DataLoader(dataset=mnist_train,
                                          batch_size=batch_size,
                                          shuffle=True,
                                          drop_last=True)

class CNN(torch.nn.Module):

    def __init__(self):
        super(CNN, self).__init__()
        # l layer
        # ImgIn shape=(?, 28, 28, 1)

        self.layer1 = torch.nn.Sequential(
            torch.nn.Conv2d(1, 32, kernel_size=3, stride=1, padding=1),
            torch.nn.ReLU(),
            torch.nn.MaxPool2d(kernel_size=2, stride=2))

        # 2 layer

        self.layer2 = torch.nn.Sequential(
            torch.nn.Conv2d(32, 64, kernel_size=3, stride=1, padding=1),
            torch.nn.ReLU(),
            torch.nn.MaxPool2d(kernel_size=2, stride=2))

        # 7x7x64 inputs -> 10 outputs
        self.fc1 = torch.nn.Linear(7 * 7 * 64, 100, bias=True)
        self.fc2 = torch.nn.Linear(100, 10, bias=True)

    def forward(self, x):
        out = self.layer1(x)
        out = self.layer2(out)
        out = out.view(out.size(0), -1)   # Flatten for total bonding layer
        out = F.relu(self.fc1(out))
        out = self.fc2(out)
        return out

# CNN model definition
model = CNN().to(device)

criterion = torch.nn.CrossEntropyLoss().to(device)    # Softmax function included in cost function.
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

total_batch = len(data_loader)
print('Number of batches : {}'.format(total_batch))

def epoch_acc():
    X_test = mnist_test.test_data.view(len(mnist_test), 1, 28, 28).float().to(device)
    Y_test = mnist_test.test_labels.to(device)

    prediction = model(X_test)
    correct_prediction = torch.argmax(prediction, 1) == Y_test
    accuracy = correct_prediction.float().mean()
    acc = accuracy.item()

    return acc

for epoch in range(training_epochs):
    avg_cost = 0

    for X, Y in data_loader: # I take it out in a mini-batch unit
        # image is already size of (28x28), no reshape
        # label is not one-hot encoded
        X = X.to(device)
        Y = Y.to(device)

        optimizer.zero_grad()
        hypothesis = model(X)
        cost = criterion(hypothesis, Y)
        cost.backward()
        optimizer.step()

        avg_cost += cost / total_batch

    print('[Epoch: {:>4}] cost = {:>.9}'.format(epoch + 1, avg_cost) + " " + "ACC= ", epoch_acc())

# I will not proceed with learning, so torch.no_grad()
with torch.no_grad():
    X_test = mnist_test.test_data.view(len(mnist_test), 1, 28, 28).float().to(device)
    Y_test = mnist_test.test_labels.to(device)

    prediction = model(X_test)
    correct_prediction = torch.argmax(prediction, 1) == Y_test
    accuracy = correct_prediction.float().mean()
    print('Accuracy:', accuracy.item())

This code is the data separation of MNIST using CNN.
If you run this
During 15 epochs, the cost and learning performance (acc) are displayed.
However, I would like to output validation performance as shown in the figure here

The picture is also a code that uses this code.
Currently, in this code, training data is set as 60000 and test data is 10000. How do you output the validation performance here?
Can i find the validation performance( VallACC ) in the code itself?
Or Using a sklearn should I create a new validation set and use the same method as cross validation?

2021-04-26 This code was modified by adding the code you uploaded.
import torchvision.transforms as transforms
import torch.nn.init
import torch.nn.functional as F
import numpy as np
from torch.utils.data import (
    DataLoader,
    random_split,
    SubsetRandomSampler,
    WeightedRandomSampler,
)

device = "cuda" if torch.cuda.is_available() else "cpu"
print(device)

learning_rate = 0.001
training_epochs = 15
batch_size = 100

mnist_train = dsets.MNIST(
    root="MNIST_data/",  # Specify download path
    train=True,  # Specify True to download as training data
    transform=transforms.ToTensor(),  # Convert to tensor
    download=True,
)

mnist_test = dsets.MNIST(
    root="MNIST_data/",  # Specify download path
    train=False,  # If false is specified, download as test data
    transform=transforms.ToTensor(),  # Convert to tensor
    download=True,
)

data_loader = torch.utils.data.DataLoader(
    dataset=mnist_train, batch_size=batch_size, shuffle=True, drop_last=True
)

class CNN(torch.nn.Module):
    def __init__(self):
        super(CNN, self).__init__()
        # l layer
        # ImgIn shape=(?, 28, 28, 1)

        self.layer1 = torch.nn.Sequential(
            torch.nn.Conv2d(1, 32, kernel_size=3, stride=1, padding=1),
            torch.nn.ReLU(),
            torch.nn.MaxPool2d(kernel_size=2, stride=2),
        )

        # 2 layer

        self.layer2 = torch.nn.Sequential(
            torch.nn.Conv2d(32, 64, kernel_size=3, stride=1, padding=1),
            torch.nn.ReLU(),
            torch.nn.MaxPool2d(kernel_size=2, stride=2),
        )

        # 7x7x64 inputs -> 10 outputs
        self.fc1 = torch.nn.Linear(7 * 7 * 64, 100, bias=True)
        self.fc2 = torch.nn.Linear(100, 10, bias=True)

    def forward(self, x):
        out = self.layer1(x)
        out = self.layer2(out)
        out = out.view(out.size(0), -1)  # Flatten for total bonding layer
        out = F.relu(self.fc1(out))
        out = self.fc2(out)
        return out

# CNN model definition
model = CNN().to(device)

criterion = torch.nn.CrossEntropyLoss().to(device)  # Softmax function included in cost function.
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

total_batch = len(data_loader)
print("Number of batches : {}".format(total_batch))

valid_size = 0.2
num_train = len(mnist_train)
indices = list(range(num_train))
np.random.shuffle(indices)
split = int(np.floor(valid_size * num_train))
train_idx, valid_idx = indices[split:], indices[:split]

train_sampler = SubsetRandomSampler(train_idx)
valid_sampler = SubsetRandomSampler(valid_idx)

train_loader = torch.utils.data.DataLoader(
    mnist_train, batch_size=64, sampler=train_sampler, num_workers=2
)
valid_loader = torch.utils.data.DataLoader(
    mnist_train, batch_size=30, sampler=valid_sampler, num_workers=2
)

for i, (data, target) in enumerate(train_loader):
    # move tensor to gpu if cuda is available

    data, target = data.to(device), target.to(device)
    # clear the gradiant of all optimizer variable
    optimizer.zero_grad()
    # forward pass: compute pradictions by passing inputs
    output = model(data)
    # calculate batch loss
    loss = criterion(output, target)
    # backward pass: compute gradiant of the loss with respect to the parameters
    loss.backward()
    # update parameters by optimizing single step
    optimizer.step()
    # update training loss
    train_loss += loss.item() * data.size(0)

# validate the model

model.eval()
for batch_idx, (data, target) in enumerate(valid_loader):
    # move tensor to gpu

    data, target = data.to(device), target.to(device)
    # forward pass: compute the validation predictions
    output = model(data)
    # calculate the loss
    loss = criterion(output, target)
    # update the validation loss
    valid_loss += loss.item() * data.size(0)

# calculate average loss
train_losses = train_loss / len(train_loader.sampler)
valid_losses = valid_loss / len(valid_loader.sampler)
scheduler.step()
# Print the train and validation loss statistic
print(
    "Epoch: {} \t Training Loss: {:.3f} \t Validation Loss: {:.3f}".format(
        epoch, train_losses, valid_losses
    )
)
# save model if validation loss decrease
if valid_losses <= valid_loss_min:
    print(
        "Validation loss decreased {:.4f}--->{:.4f}  Saving model...".format(
            valid_loss_min, valid_losses
        )
    )
    # save current model
    torch.save(model.state_dict(), "model_cifer.pt")
    valid_loss_min = valid_losses
print("Learning rate: {:.5f}".format(optimizer.state_dict()["param_groups"][0]["lr"]))


title: Which python deep learning libraries compile at runtime?

body:
I am trying to wrap my head around C-optimized code in python. I have read a couple of times now that python achieves high-speed computing through C-extensions. In other words, whenever I work with libraries such as numpy, it basically calls a C-extension that calculates the result and returns it.

C-extensions using numpy

Say I want to add two numbers using np.add(x,y). If I understand it correctly, libraries such as numpy do not compile the python code but instead already come with executables that will simply take the values x and y and return the result. Is that correct?

Theano, Tensorflow, and PyTorch

In particular, I am wondering if this is also true for deep learning libraries. According to the official documentation of Theano, it requires g++ and gcc (at least they are highly recommended). Does this mean that Theano will compile C (or C++) code at runtime of the python script? If so, is it the same for PyTorch and Tensorflow?

I hope that someone can solve my confusion here! Thanks a lot!


title: Pytorch loss of convolution is 0.0 from start

body:
I am building a conv net that classifies dog and cat. Architecture is pretty simple. 2 Conv(with batch norm, leakyReLU, Maxpooling) to 1 fc. Input image size is resized to 64. The size is good. The problem is loss is 0.0 from the start. I have no clue what the cause is. I couldn't find any answer. I have wrote every detail that might be important. If you need anything else please tell me, I will edit.

  main.py

import torch
import torch.nn as nn
from torchvision import transforms, datasets
import PIL
import matplotlib.pyplot as plt
from Dataset import Dataset
from Network import Network
# Added to avoid     torch._C._cuda_init() \n RuntimeError: CUDA error: unknown error
torch.cuda.current_device()
# Hyper Parameters
batch_size = 1
img_size = 64
learning_rate = 0.001
num_epoch = 1

# Directories
trainDir = "D:/Programming/python/Deep learning/datasets/dogs-vs-cats/train"
testDir = "D:/Programming/python/Deep learning/datasets/dogs-vs-cats/test1"

print("Initializing...")

# Device
device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')

# Augmentation
transforms = transforms.Compose([
    transforms.Resize((img_size, img_size)),
    transforms.ColorJitter(hue=.05, saturation=.05),
    transforms.RandomHorizontalFlip(),
    transforms.RandomRotation(20, resample=PIL.Image.BILINEAR) ,
    transforms.ToTensor()
])

trainset = datasets.ImageFolder(root=trainDir, transform=transforms)
testset = datasets.ImageFolder(root=testDir, transform=transforms)
train_loader = torch.utils.data.DataLoader(
    trainset, batch_size=batch_size, shuffle=True)
test_loader = torch.utils.data.DataLoader(
    testset, batch_size=batch_size, shuffle=False)  # test set will not be shuffled

model = Network(img_size,2).to(device)

criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

total_step = len(train_loader)
print("Tranining started")
for epoch in range(num_epoch):
    for i, (images, labels) in enumerate(train_loader):
        images = images.to(device)
        labels = labels.to(device)

        # forward propagate
        outputs = model(images)
        loss = criterion(outputs, labels)

        # backpropagte and optimize
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if (i+1) % 100 == 0:
            print(
                "Epoch [{}/{}], Step[{}/{}], Loss: {}".format(
                    epoch+1, num_epoch, i+1, total_step, loss.item()
                )
            )
print("Tranining complete, validation started")

with torch.no_grad():
    correct = 0
    total = 0
    for images, labels in test_loader:
        images = images.to(device)
        labels = labels.to(device)
        outputs = model(images)
        _, predicted = torch.max(outputs.data, 1)
        total += labels.size(0)
        correct += (predicted == labels).sum().item()

        print('Test Accuracy: {} %'.format(100 * correct / total))

#
torch.save(model.state_dict(), "model.ckpy")

  Network.py

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

"""
    Input size for conv
    l = number of input feature maps
    k = number of output feature maps
    n, m = width and height of kernel
    total parameter = (n*m*l+1)*k
"""

class Network(nn.Module):
    def __init__(self, input_size, num_class):
        super(Network, self).__init__()
        self.conv1 = nn.Sequential(
            nn.Conv2d(3, 16, kernel_size=5, stride=1, padding=2),
            nn.BatchNorm2d(16),
            nn.LeakyReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2)
        ) # output size = (128, 128, 16)
        self.conv2 = nn.Sequential(
            nn.Conv2d(16, 32, kernel_size=5, stride=1, padding=2),
            nn.BatchNorm2d(32),
            nn.LeakyReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2)
        ) # output size = (64, 64, 32)

        self.fc1 = nn.Linear(
            int((input_size/4)**2*32), num_class
        )

    def forward(self, x):
        out = self.conv1(x)
        out = self.conv2(out)
        out = out.view(out.size(0), -1)
        out = self.fc1(out)
        return out

  Output

Epoch [1/1], Step[5800/25000], Loss: 0.0
Epoch [1/1], Step[5900/25000], Loss: 0.0
Epoch [1/1], Step[6000/25000], Loss: 0.0
Epoch [1/1], Step[6100/25000], Loss: 0.0
Epoch [1/1], Step[6200/25000], Loss: 0.0
Epoch [1/1], Step[6300/25000], Loss: 0.0
Epoch [1/1], Step[6400/25000], Loss: 0.0
Epoch [1/1], Step[6500/25000], Loss: 0.0

  Result after each layer

outputs of conv1,2
         [[ 3.0135e-01,  3.5849e-01,  4.7758e-01,  ...,  3.9759e-01,
            3.7988e-01,  9.7870e-01],
          [ 4.3010e-01,  6.0753e-03,  4.5642e-01,  ..., -8.5486e-04,
            4.4537e-02,  2.9074e-01],
          [ 3.8567e-01,  7.8431e-02,  2.3859e-01,  ..., -3.0013e-03,
           -5.5821e-03,  1.2284e-01],
          ...,
          [ 3.9181e-01,  3.9093e-01,  1.2053e-01,  ..., -4.7156e-03,
            5.6266e-01,  7.7017e-01],
outputs of fc1
          [[-0.0772,  0.2166]]


title: Implementing prox-linear optimizer in PyTorch

body:
It seems that the PyTorch optimizer class assumes it uses only gradient information. However, I would like to implement a different training algorithm, namely, the prox-linear method, which minimizes losses of the form g(l(prediction, label)), where g is some known convex function.

The optimizer needs:

access to the value of l(prediction, label), in addition to its gradient w.r.t the parameters
knowledge of the function g beyond its gradient

How should I proceed with designing such an optimizer? How can it gain access to the information it needs?

It is clear that g, in some form, should be an argument of the optimizer, and not some generic loss function. But what about l?


title: Grad is very small

body:
First I thought, there is no update after the loss calculation and update step but the follwing code gives me false which prove there is an update:

before = list(model.parameters())[0].clone()
loss.backward()
optimizer.step()
after = list(model.parameters())[0].clone()
logging.info(torch.equal(before.data, after.data))

Then I wanted to see the magnitude of chage by the following code:

list(model.parameters())[0].grad

Which give me very small amaount of change:

tensor([[ 3.2294e-07,  7.3983e-06,  6.5637e-06,  ..., -1.3529e-06,
2019-11-16T17:21:03.747068480Z          -4.9979e-06,  4.9799e-06],
2019-11-16T17:21:03.747074966Z         [ 5.5463e-08,  3.0771e-06, -9.5087e-07,  ...,  1.9265e-06,
2019-11-16T17:21:03.747080687Z          -4.5251e-06, -7.1564e-07].....]

Here is my model :

self.lstm_hidden_dim = lstm_hidden_dim
self.lstm = nn.LSTM(word_embedding_dim,lstm_hidden_dim, num_layers=1,batch_first=False,bidirectional=True)
self.firstHidden = nn.Linear(lstm_hidden_dim*2, 300)
self.relu=nn.ReLU()
self.secondlinear=nn.Linear(300, score_space_size)
self.softmax= nn.LogSoftmax(dim=1)

Loss function and optimizer:

loss_function = nn.NLLLoss()
optimizer = optim.SGD(model.parameters(), lr=0.1)

Any idea, why the grad is very small?


title: What does the interleave_keys() function in torchtext library do exactly?

body:
You can find this function at torchtext/data/utils.py file

I have given the official code with documentation below

def interleave_keys(a, b):
    """Interleave bits from two sort keys to form a joint sort key.

    Examples that are similar in both of the provided keys will have similar
    values for the key defined by this function. Useful for tasks with two
    text fields like machine translation or natural language inference.
    """
    def interleave(args):
        return ''.join([x for t in zip(*args) for x in t])
    return int(''.join(interleave(format(x, '016b') for x in (a, b))), base=2)

A more detailed explanation would be helpful to understand how it returns an integer based on how similar the given two strings are.

And the format function used inside it is the commonly used builtin function in python


title: NotImplementError while inspecting PyTorch nn model

body:
I'm trying to inspect the model(PyTorch) but it shows the following error!

Unable to display children:Error resolving variables Traceback (most recent call last):
  File "D:\Program Files\JetBrains\PyCharm 2019.2.3\plugins\python\helpers\pydev\_pydevd_bundle\pydevd_comm.py", line 1208, in do_it
    xml.write(pydevd_xml.var_to_xml(val, k, evaluate_full_value=evaluate_full_value))
  File "D:\Program Files\JetBrains\PyCharm 2019.2.3\plugins\python\helpers\pydev\_pydevd_bundle\pydevd_xml.py", line 369, in var_to_xml
    elif hasattr(v, "__len__") and not is_string(v):
  File "D:\Anaconda3\envs\pt37\lib\site-packages\torch\nn\backends\backend.py", line 10, in __getattr__
    raise NotImplementedError
NotImplementedError

What's the __getattr__ method and how do I implement it?


title: Is my classifier having memory leaks or other memory-related issues? If yes, how do I address them?

body:
I'm running a classifier for face recognition implemented by arsfutura that you can find at https://github.com/arsfutura/face-recognition, based on pytorch. I'm using ubuntu 18.04 in a 64bit artchitecture, python 3.6.8. Please tell me if there is some additional information that can be useful to address my issue, I will provide it.
I run the face recognition via webcam using the command

mprof run video_classifier.py

by running the script this way, a file with extention .dat is compiled in the current directory, and by plotting this file using

mprof plot file.dat

I get something like this:
used memory vs. time

Waiting more time, the used memory will slowly but steadily increase. Does this mean there is a memory leak or some other memory-related problem? How can I be sure? How can I figure out why and where this is happening?
Thanks in advance


title: Append feature maps to intermediate layers of network in PyTorch

body:
I have the following PyTorch network architecture shown below. It takes in the input of size 16x8x2048, and gives an output of size 256x128x3. I have a feature map that I want to append along the channel dimension after every upsampling layer in the below architecture. I would be scaling the feature maps to the appropriate resolution before appending. How can I append these feature maps in the intermediate layers of the network?

model = []
model += [Conv2dBlock(2048, 256, 3, 1, 1, norm='bn', activation=activ, pad_type=pad_type)]
model +=  [nn.Upsample(scale_factor=2, mode='bilinear')]
model +=  [Conv2dBlock(256, 128, 3, 1, 1, norm='bn', activation=activ, pad_type=pad_type)]
model +=  [nn.Upsample(scale_factor=2, mode='bilinear')]
model +=  [Conv2dBlock(128, 64, 3, 1, 1, norm='bn', activation=activ, pad_type=pad_type)]
model +=  [nn.Upsample(scale_factor=2, mode='bilinear')]
model +=  [Conv2dBlock(64, 32, 3, 1, 1, norm='bn', activation=activ, pad_type=pad_type)]
model +=  [nn.Upsample(scale_factor=2, mode='bilinear')]
model +=  [Conv2dBlock(32, 32, 3, 1, 1, norm='bn', activation=activ, pad_type=pad_type)]
model += [Conv2dBlock(32, 3, 3, 1, 1, norm='none', activation=activ, pad_type=pad_type)]
model = nn.Sequential(*model)


title: Pytorch Matrix element wise multiplication

body:
Let's say I have two tensors A of shape : [32 , 512] and B of shape : [32 , 512],
And I want to do element-wise multiplication between the vectors of the matrix to get a new matrix of shape : [32 , 1] (first row of A with first row of B, and second row of A with second row of B and so on.. ) , current methods I have tried simply mul the matrix values and not rows and gives a matrix of shape : [32 , 512].
Thanks!


title: PyTorch paste values into tensor by row index with increasing column index

body:
I have a tensor output into which I want to put some values. I know the row that each value should go in, but I don't have an index tensor describing the columns. Instead, if there are k values that belong to one row, they should go in columns 0, 1, ..., k-1. This is perhaps better explained with an example:

import torch

n = 4
max_cols = 5
output = torch.zeros(n, max_cols)
row_idx = torch.tensor([0, 0, 0, 0, 1, 1, 2, 2, 2, 3])
values = torch.arange(len(row_idx)).float() # the values could be anything, not just arange
# output[??] = values

Here the first 4 values should be in output at [0, 0], ... [0, 3], the next 2 values at [1, 0], [1, 1], and so on.

Here's how I'm doing this now

_, counts = torch.unique(row_idx, return_counts=True)
range_ = torch.arange(max_cols)
col_idx = torch.cat([range_[:c] for c in counts])
output[row_idx, col_idx] = values
output

tensor([[0., 1., 2., 3., 0.],
        [4., 5., 0., 0., 0.],
        [6., 7., 8., 0., 0.],
        [9., 0., 0., 0., 0.]])

Is there any more efficient way to paste these values into the appropriate positions?

(feel free to suggest a better title for this if you can think of one)


title: Is it possible to use Pytorch on AWS Lambda?

body:
I used zappa to upload my classifier project to AWS Lambda. After I imported the torch package in my program, the build failed. It fails only when I import torch or torchvision packages.
Is there a way to upload these packages?

Error:

Waiting for stack website-aws-api-dev to update..

6res [00:06,  1.05s/res]

Deploying API Gateway..

Error: Warning! Status check on the deployed lambda failed. A GET request to '/' yielded a 502 response code.


title: Is it possible to "mask out" unwanted inputs and feature maps in PyTorch so that PyTorch simply ignores them?

body:
Recently, I've been playing around with PyTorch and I'm curious if it's possible to modify a Fully Convolutional Network so that it operates on a nonrectangular region of an image only.
Let's take an arbitrary image. Within this image, there is a non-rectangular region of pixels that I already know that I am interested in (the blue region of this image, where each square is conveniently the size of the kernel). Is there a way for me to tell a network to only operate on the blue region and not bother with the rest of the image (yellow region)?
I know I can just turn the unwanted part of the image to zeroes, but that seems like a waste; the network will still slide a convolutional kernel over the zeroed-out part of the image uselessly. Is it possible to provide a mask or something to tell PyTorch "don't bother sliding the conv operator over this region of the image; everything downstream is automatically 0"?
I've read about geometric neural networks for graph inputs and such, but in my head, it seems a bit overkill for this idea.


title: pytorch image multi label recognition model does not converge

body:
I built a image multi label recognition model based on resnet-50, the network is as follows (it's exactly the same as in the torchvision.models.resnet50, with the only difference in the final softmax layer in the forward function)
class ResNetMultiTag(nn.Module):

    def __init__(self, block, layers, num_classes=1000, zero_init_residual=False,
                groups=1, width_per_group=64, replace_stride_with_dilation=None,
                norm_layer=None):
        super(ResNetMultiTag, self).__init__()
        if norm_layer is None:
            norm_layer = nn.BatchNorm2d
        self._norm_layer = norm_layer

        self.inplanes = 64
        self.dilation = 1
        if replace_stride_with_dilation is None:
            # each element in the tuple indicates if we should replace
            # the 2x2 stride with a dilated convolution instead
            replace_stride_with_dilation = [False, False, False]
        if len(replace_stride_with_dilation) != 3:
            raise ValueError("replace_stride_with_dilation should be None "
                            "or a 3-element tuple, got {}".format(replace_stride_with_dilation))
        self.groups = groups
        self.base_width = width_per_group
        self.conv1 = nn.Conv2d(3, self.inplanes, kernel_size=7, stride=2, padding=3,
                            bias=False)
        self.bn1 = norm_layer(self.inplanes)
        self.relu = nn.ReLU(inplace=True)
        self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1)
        self.layer1 = self._make_layer(block, 64, layers[0])
        self.layer2 = self._make_layer(block, 128, layers[1], stride=2,
                                    dilate=replace_stride_with_dilation[0])
        self.layer3 = self._make_layer(block, 256, layers[2], stride=2,
                                    dilate=replace_stride_with_dilation[1])
        self.layer4 = self._make_layer(block, 512, layers[3], stride=2,
                                    dilate=replace_stride_with_dilation[2])
        self.avgpool = nn.AdaptiveAvgPool2d((1, 1))
        self.fc = nn.Linear(512 * block.expansion, num_classes * 2) # special
        self.softmax = nn.Softmax(dim=-1) # special

        for m in self.modules():
            if isinstance(m, nn.Conv2d):
                nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')
            elif isinstance(m, (nn.BatchNorm2d, nn.GroupNorm)):
                nn.init.constant_(m.weight, 1)
                nn.init.constant_(m.bias, 0)

        # Zero-initialize the last BN in each residual branch,
        # so that the residual branch starts with zeros, and each residual block behaves like an identity.
        # This improves the model by 0.2~0.3% according to https://arxiv.org/abs/1706.02677
        if zero_init_residual:
            for m in self.modules():
                if isinstance(m, Bottleneck):
                    nn.init.constant_(m.bn3.weight, 0)
                elif isinstance(m, BasicBlock):
                    nn.init.constant_(m.bn2.weight, 0)

    def _make_layer(self, block, planes, blocks, stride=1, dilate=False):
        norm_layer = self._norm_layer
        downsample = None
        previous_dilation = self.dilation
        if dilate:
            self.dilation *= stride
            stride = 1
        if stride != 1 or self.inplanes != planes * block.expansion:
            downsample = nn.Sequential(
                conv1x1(self.inplanes, planes * block.expansion, stride),
                norm_layer(planes * block.expansion),
            )

        layers = []
        layers.append(block(self.inplanes, planes, stride, downsample, self.groups,
                            self.base_width, previous_dilation, norm_layer))
        self.inplanes = planes * block.expansion
        for _ in range(1, blocks):
            layers.append(block(self.inplanes, planes, groups=self.groups,
                                base_width=self.base_width, dilation=self.dilation,
                                norm_layer=norm_layer))

        return nn.Sequential(*layers)

    def forward(self, x):
        x = self.conv1(x)
        x = self.bn1(x)
        x = self.relu(x)
        x = self.maxpool(x)

        x = self.layer1(x)
        x = self.layer2(x)
        x = self.layer3(x)
        x = self.layer4(x)

        x = self.avgpool(x)
        x = x.reshape(x.size(0), -1)
        x = self.fc(x)
        x = x.reshape(x.size(0), -1, 2) # special
        x = self.softmax(x) # special

        return x

This is the loss function I use
def get_tag_loss(self, pred, label):
    '''
    pred: [batchsize, num_classes, 2]
    label: [batchsize, num_classes, 2]
    '''
    pred = pred.contiguous().view(-1,2)
    label = label.contiguous().view(-1,2)

    label_scalar = label[:, 1]

    logp = torch.nn.functional.log_softmax(pred, dim=-1)
    ce = -logp[range(label_scalar.shape[0]), label_scalar.long()]
    loss = torch.sum( ce )
    return loss

I use SGD with lr ranges from 1e-2 to 1e-4, I have 6000 images, 15 classes. Each image may have 1 to 5 labels. The number of labels is un-balanced but I have done the resampling to keep the balance.
However, the model does not converge.
I tried to train the model on only one image, this case the model converges to the correct labels. However, even if I use only two images, just add one more image, the model failed to converge.
What's wrong with my implementation?
Did I use the wrong loss function, wrong model, or I make other mistakes?
Thank you all for helping me!!


title: How to set environment variables in torch.nn.parallel.DistributedDataParallel?

body: I am trying to train my code with distributed data parallelism, I already trained using torch.nn.DataParallel and now I am trying to see how much gain I can get in training speed if I train using torch.nn.parallel.DistributedDataParallel since I read on numerous pages that its better to use DistributedDataParallel. So I followed one of the examples but I am not sure how to set the following environment variables (os.environ['MASTER_ADDR'] and os.environ['MASTER_PORT']) since I am using a cloud service so I am not sure which specific node my code gets allocated for training my model. Can anyone help me to set these variables?


title: Convert Output Size of Fourier Transformed Signal Using Pytorch (torch.rfft)

body:
I’m trying to run a image fourier transformation with various output sizes of signal using pytorch. For example,suppose that the input size of image is [3x32x32] and the output size of fourier transformation is [3x64x64], then ideal code can be represented as torch.rfft(input, signal_ndim=3, n=(3,64,64)) (if given n is the output size of signal).
However, pytorch doesn’t seem to offer the argument ‘n’. According to the pytorch fft documentation, N represents the output size of signal, but cannot be modified in the code.
SO, does anyone know how to change the output size of fourier transformed signal using pytorch? I would appreciate if anyone can tell me a solution to change the output size of fourier trasformed signal.
You can see that numpy-fft library uses the argument ‘s’ as the output size of the signal in the code.

Numpyhttps://numpy.org/doc/stable/reference/generated/numpy.fft.fftn.htmlnumpy.fft.fftn(a, s=None, axes=None, norm=None)
-> ‘s’ denotes shape of the output of signal.Simple code
import torch
import numpy as np

np_a = torch.rand(3, 32, 32).numpy()
np_a_ft = np.fft.fftn(a, (3,28, 28))
print(np_a_ft.shape)

Input size of image                    : 3, 32, 32
Output size of after fourier transform : 3, 28, 28

Pytorchhttps://pytorch.org/docs/stable/generated/torch.rfft.htmltorch.rfft(input, signal_ndim, normalized=False, onesided=True)
torch_x   = torch.rand(3, 32, 32, requires_grad=True)
torch_xf  = torch.rfft(torch_x, 3)
print(torch_xf.size())

torch.Size([3, 32, 17, 2])


title: Import error when using torch.utils.bottleneck

body:
I use torch.utils.bottleneck to profile the code.
But I got some import errors as follow:
Traceback (most recent call last):
...
  File "./benchmark/csm/gnn.py", line 17, in <module>
    from ...data import sc as snc
ImportError: attempted relative import with no known parent package

How can I use 'torch.utils.bottleneck' to profile my code without modifying my project code?
Because there are a lot of from ...xx import xx as xx in my project.


title: Sparse Cross Entropy Loss for calculating Loss in NLP problems. PyTorch

body:
My Input tensor Looks like :
torch.Size([8, 23])

// where,
// 8 -> batch size
// 23 -> words in each of them

My output tensor Looks like :
torch.Size([8, 23, 103])

// where,
// 8 -> batch size
// 23 -> words predictions
// 103 -> vocab size.

I want to calculate sparse cross Entropy Loss for this task, but I can’t since PyTorch only calculates the loss single element. How can I code it to work? Thanks for your help.


title: Using Transformer's decoder to extract sentences

body:
My question relates directly to the text summarization task. I know that there are a bunch of implementations with RNN networks (especially LSTMs) that use sentence-level attention to extract salient sentences of the source, using an attentive LSTM decoder. I have been digging into it to see if this is possible with the Transformer's networks, specifically the Transformer's decoder part, but do not really have an idea how to get this incorporated.
Look, for example, in terms of LSTM decoder, the LSTM encoder can produce contextualized encodings for the sentences that are in the source, then the last hidden state is passed to the LSTM decoder, and at each decoding timestep, the decoder attends to the source sentences (encoded by the encoder) to get an attention score over those. Finally, these scores and the sentence hidden states are combined to form the context vector, which is further processed with the decoder hidden state to predict the right sentence to pick up.
Assume that I have obtained the sentence encodings using Transformer's encoder, I'm just wondering how I can relate the scenario happening in LSTM network's decoder part to the Transformer's network decoder side.
Also, a question, how are these networks (both LSTM and Transformer's) that use sentence-level attention instead of word-level attention trained?

Update: the behaviour that I intended to achieve is as follows: I want the Transformer's decoder gets in sentences (instead of tokens which is then regarded as abstractive summarization), compute attention on the source sentences considering the partial summary that has been selected in prior timesteps, and then give a probability distribution over the source sentences denoting how much it is probable that a sentence is being copied into the target. So to make it explicit, I'm looking for an extractive summarizer with decoder.


title: Why does Pytorch need `torch.cuda.is_available()` to be called for it to run?

body:
This is bugging me as some weird behaviour and I'm hoping someone has met with a similar situation.
Basically my application starts in Nvidia Docker2, and shows the no CUDA-capable device is detected error until I add a line torch.cuda.is_available(), then it magically starts working again.
I've only managed to gather that its not some race condition on what I can control, and calling other cuda runtime commands don't have the same effect.
So now I need to add the line somewhere before I run my pytorch application and I'm baffled as to why?
edit 1:
torch.cuda.init() also works, so the question is then why does torch.cuda.is_available() work


title: PyTorch Optimizer Doesn’t update Weights

body:
i'm newbie in PyTorch. Can someone help? I am trying teach Neural Network to play in tetris, but can't understand why weights doesn't cange.
Neural Network:
class CNN(Module):
# define model elements

def __init__(self, n_channels):
    super(CNN, self).__init__()
    self.number_of_actions = 5
    self.gamma = 0.01
    self.final_epsilon = 0.0001
    self.initial_epsilon = 0.1
    self.number_of_iterations = 2000000
    self.replay_memory_size = 10000
    self.minibatch_size = 32
    # input to first hidden layer
    #self.hidden1 = Linear(n_channels, 50)
    self.hidden1 = Conv2d(1, 1, (22, 10))
    kaiming_uniform_(self.hidden1.weight, nonlinearity='relu')
    #self.act1 = ReLU()
    self.pool1 = MaxPool2d(2, 2)
    # second hidden layer
    self.hidden2 = Linear(1, 30)
    kaiming_uniform_(self.hidden2.weight, nonlinearity='relu')
    self.act2 = ReLU()
    # fully connected layer
    self.hidden3 = Linear(30, 10)
    kaiming_uniform_(self.hidden3.weight, nonlinearity='relu')
    self.act3 = ReLU()
    # output layer
    self.hidden4 = Linear(10, 1)
    xavier_uniform_(self.hidden4.weight)
    self.act4 = Softmax(dim=1)

# forward propagate input
def forward(self, X):
    # input to first hidden layer
    X = self.hidden1(X)
    X = self.pool1(X)
    # second hidden layer
    X = self.hidden2(X)
    X = self.act2(X)
    # third hidden layer
    X = self.hidden3(X)
    X = self.act3(X)
    # output layer
    X = self.hidden4(X)
    X = self.act4(X)
    return X

Learning loop:
        reward = torch.tensor([[0.]], requires_grad=True)

        inputs = torch.tensor([[self.BoardMatrix]])
        outputs = model(inputs)
        # compute the model output
        state_action_values = model(inputs)
        replay_memory.append(self.BoardMatrix)
        if len(replay_memory) > model.replay_memory_size:
            replay_memory.pop(0)

        if self.isWaitingAfterLine:
            self.isWaitingAfterLine = False
            self.newPiece()
        else:
            # learning

            if (state_action_values[0][0][0][0] > 0.6):
                self.action("left")
                action_batch = 0.8
            elif (state_action_values[0][0][0][0] > 0.2):
                self.action("right")
                action_batch = 0.6
            elif (state_action_values[0][0][0][0] > -0.2):
                self.action("down")
                action_batch = 0.4
            elif (state_action_values[0][0][0][0] > -0.6):
                self.action("up")
                action_batch = 0.2
            elif (state_action_values[0][0][0][0] > -1):
                self.action("space")
                action_batch = 0
            self.oneLineDown()

        minibatch = random.sample(replay_memory, min(len(replay_memory), model.minibatch_size))
        criterion: MSELoss = nn.MSELoss()
        reward_batch = reward
        y_batch = torch.cat(tuple(reward_batch if minibatch[i][4]
                                  else reward_batch + model.gamma * torch.max(state_action_values[i])
                                  for i in range(len(minibatch))))
        y_batch = y_batch.detach()
        q_value = torch.sum(outputs * action_batch, dim=1)
        optimizer.zero_grad()
        loss = criterion(q_value, y_batch[0])
        # credit assignment
        loss.backward()
        optimizer.step()
        print(model.hidden1.weight)
    else:
        super(Board, self).timerEvent(event)

NN, optimizer initialization:
model = CNN(boardSize)
criterion = CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=1e-6)

I working by this tutorial "https://www.toptal.com/deep-learning/pytorch-reinforcement-learning-tutorial", but try change it for tetris.
P.S. I also find out y_batch never change but i don't know why


title: Issue when Re-implement Matrix Factorization in Pytorch

body:
I try to implement matrix factorization in Pytorch as the data extractor and model.
The original model is written in mxnet. Here I try to use the same idea in Pytorch.
Here is my code, it can be runned directly in codelab
import torch
import torch.nn as nn
import pandas as pd
import numpy as np
from torch.utils.data import Dataset, DataLoader

import collections
from collections import defaultdict
from IPython import display
import math
from matplotlib import pyplot as plt
import os
import pandas as pd
import random
import re
import shutil
import sys
import tarfile
import time
import requests
import zipfile
import hashlib

# ============data obtained, not change the original code
DATA_HUB= {}

# Defined in file: ./chapter_multilayer-perceptrons/kaggle-house-price.md
def download(name, cache_dir=os.path.join('..', 'data')):
    """Download a file inserted into DATA_HUB, return the local filename."""
    assert name in DATA_HUB, f"{name} does not exist in {DATA_HUB}."
    url, sha1_hash = DATA_HUB[name]
    os.makedirs(cache_dir, exist_ok=True)
    fname = os.path.join(cache_dir, url.split('/')[-1])
    if os.path.exists(fname):
        sha1 = hashlib.sha1()
        with open(fname, 'rb') as f:
            while True:
                data = f.read(1048576)
                if not data:
                    break
                sha1.update(data)
        if sha1.hexdigest() == sha1_hash:
            return fname  # Hit cache
    print(f'Downloading {fname} from {url}...')
    r = requests.get(url, stream=True, verify=True)
    with open(fname, 'wb') as f:
        f.write(r.content)
    return fname

# Defined in file: ./chapter_multilayer-perceptrons/kaggle-house-price.md
def download_extract(name, folder=None):
    """Download and extract a zip/tar file."""
    fname = download(name)
    base_dir = os.path.dirname(fname)
    data_dir, ext = os.path.splitext(fname)
    if ext == '.zip':
        fp = zipfile.ZipFile(fname, 'r')
    elif ext in ('.tar', '.gz'):
        fp = tarfile.open(fname, 'r')
    else:
        assert False, 'Only zip/tar files can be extracted.'
    fp.extractall(base_dir)
    return os.path.join(base_dir, folder) if folder else data_dir

#1. obtain dataset
DATA_HUB['ml-100k'] = ('http://files.grouplens.org/datasets/movielens/ml-100k.zip',
    'cd4dcac4241c8a4ad7badc7ca635da8a69dddb83')

def read_data_ml100k():
    data_dir = download_extract('ml-100k')
    names = ['user_id', 'item_id', 'rating', 'timestamp']
    data = pd.read_csv(os.path.join(data_dir, 'u.data'), '\t', names=names,
                       engine='python')
    num_users = data.user_id.unique().shape[0]
    num_items = data.item_id.unique().shape[0]
    return data, num_users, num_items

# 2. Split data
#@save
def split_data_ml100k(data, num_users, num_items,
                      split_mode='random', test_ratio=0.1):
    """Split the dataset in random mode or seq-aware mode."""
    if split_mode == 'seq-aware':
        train_items, test_items, train_list = {}, {}, []
        for line in data.itertuples():
            u, i, rating, time = line[1], line[2], line[3], line[4]
            train_items.setdefault(u, []).append((u, i, rating, time))
            if u not in test_items or test_items[u][-1] < time:
                test_items[u] = (i, rating, time)
        for u in range(1, num_users + 1):
            train_list.extend(sorted(train_items[u], key=lambda k: k[3]))
        test_data = [(key, *value) for key, value in test_items.items()]
        train_data = [item for item in train_list if item not in test_data]
        train_data = pd.DataFrame(train_data)
        test_data = pd.DataFrame(test_data)
    else:
        mask = [True if x == 1 else False for x in np.random.uniform(
            0, 1, (len(data))) < 1 - test_ratio]
        neg_mask = [not x for x in mask]
        train_data, test_data = data[mask], data[neg_mask]
    return train_data, test_data

#@save
def load_data_ml100k(data, num_users, num_items, feedback='explicit'):
    users, items, scores = [], [], []
    inter = np.zeros((num_items, num_users)) if feedback == 'explicit' else {}
    for line in data.itertuples():
        user_index, item_index = int(line[1] - 1), int(line[2] - 1)
        score = int(line[3]) if feedback == 'explicit' else 1
        users.append(user_index)
        items.append(item_index)
        scores.append(score)
        if feedback == 'implicit':
            inter.setdefault(user_index, []).append(item_index)
        else:
            inter[item_index, user_index] = score
    return users, items, scores, inter

#@save
def split_and_load_ml100k(split_mode='seq-aware', feedback='explicit',
                          test_ratio=0.1, batch_size=256):
    data, num_users, num_items = read_data_ml100k()
    train_data, test_data = split_data_ml100k(data, num_users, num_items, split_mode, test_ratio)
    train_u, train_i, train_r, _ = load_data_ml100k(train_data, num_users, num_items, feedback)
    test_u, test_i, test_r, _ = load_data_ml100k(test_data, num_users, num_items, feedback)

    # Create Dataset
    train_set = MyData(np.array(train_u), np.array(train_i), np.array(train_r))
    test_set = MyData(np.array(test_u), np.array(test_i), np.array(test_r))

    # Create Dataloader
    train_iter = DataLoader(train_set, shuffle=True, batch_size=batch_size)
    test_iter = DataLoader(test_set, batch_size=batch_size)

    return num_users, num_items, train_iter, test_iter

class MyData(Dataset):
  def __init__(self, user, item, score):
    self.user = torch.tensor(user)
    self.item = torch.tensor(item)
    self.score = torch.tensor(score)

  def __len__(self):
    return len(self.user)

  def __getitem__(self, idx):
    return self.user[idx], self.item[idx], self.score[idx]

# create a nn class (just-for-fun choice :-)
class RMSELoss(nn.Module):
    def __init__(self, eps=1e-6):
        '''You should be careful with NaN which will appear if the mse=0, adding self.eps'''
        super().__init__()
        self.mse = nn.MSELoss()
        self.eps = eps

    def forward(self,yhat,y):
        loss = torch.sqrt(self.mse(yhat,y) + self.eps)
        return loss

class MF(nn.Module):
    def __init__(self, num_factors, num_users, num_items, **kwargs):
        super(MF, self).__init__(**kwargs)
        self.P = nn.Embedding(num_embeddings=num_users, embedding_dim=num_factors)
        self.Q = nn.Embedding(num_embeddings=num_items, embedding_dim=num_factors)
        self.user_bias = nn.Embedding(num_users, 1)
        self.item_bias = nn.Embedding(num_items, 1)

    def forward(self, user_id, item_id):
        P_u = self.P(user_id)
        Q_i = self.Q(item_id)

        b_u = self.user_bias(user_id)
        b_i = self.item_bias(item_id)

        outputs = (P_u * Q_i).sum() + b_u.squeeze() + b_i.squeeze()
        return outputs

# train
# Device configuration
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

# Hyper parameters
num_epochs = 50
batch_size = 512
lr = 0.001

num_users, num_items, train_iter, test_iter = split_and_load_ml100k(test_ratio=0.1, batch_size=batch_size)

model = MF(30, num_users, num_items).to(device)

# Loss and Optimizer
optimizer = torch.optim.SGD(model.parameters(), lr=lr)
criterion = RMSELoss()

# Train the Model
train_rmse = []
test_rmse = []
for epoch in range(num_epochs):
    train_loss = 0
    num_train = 0
    model.train()
    for users, items, scores in train_iter:
        users = users.to(device)
        items = items.to(device)
        scores = scores.float().to(device)

        # Forward pass
        outputs = model(users, items)
        loss = criterion(outputs, scores)

        # Backward and optimize
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        train_loss += loss.item()
        num_train += scores.shape[0]

    train_rmse.append(train_loss / num_train)

    model.eval()
    test_loss = 0
    num_test = 0
    with torch.no_grad():
        for users, items, scores in test_iter:
            users = users.to(device)
            items = items.to(device)
            scores = scores.float().to(device)

            outputs = model(users, items)
            loss = criterion(outputs, scores)

            test_loss += loss.item()
            num_test += scores.shape[0]

    test_rmse.append(test_loss / num_test)

# plot
%matplotlib inline
import matplotlib.pyplot as plt
plt.style.use('seaborn-whitegrid')

x = list(range(num_epochs))
fig = plt.figure()
ax = plt.axes()

plt.plot(x, train_rmse, label='train_rmse');
plt.plot(x, test_rmse, label='test_rmse');

leg = ax.legend();

I got the result

The MXNET result is here

Why I cannot get a beautiful shape. And my train_rmse is larger than test_rmse.


title: Conversion pytorch to coreml for element-wise maximum operation

body:
I tried to convert a PyTorch model to coreml with the element-wise maximum operation based on coremltools.
With torch.max operation, I got
ValueError: node input.2 (max) got 2 input(s), expected [3]

With torch.maximum operation
RuntimeError: PyTorch convert function for op 'maximum' not implemented.

Any idea to solve this issue?


title: PyTorch - using `param.grad` affects learning process? [Solved]

body:
I'm training ResNet34 on CIFAR-10. I have a very weird behavior when I try to manipulate param.grad for model.parameters().
The following function is where all the mess happens. It currently doesn't do anything useful as a result of trying to understand what happens.
def add_error(error):
    params = (param for param in model.parameters() if param.requires_grad)
    # [param.grad + err for param, err in zip(params, error)]                        # Line 2
    # new_error = [param.grad + err for param, err in zip(params, error)]            # Line 3
    for param in params:
      param.grad.zero_()
    # new_error = [torch.zeros(param.grad.shape, device=device) for param in params] # Line 6
    return new_error

It's used in the gradient descent step:
def step(model, optimizer, batch, labels, error):
  optimizer.zero_grad()
  loss = compute_loss(model, batch, labels)
  loss.backward()

  new_error = add_error(error=error)      <- add_error is called here
  optimizer.step()
  return new_error

where optimizer is optim.SGD(model.parameters(), lr=0.1) and compute_loss essentially calls nn.CrossEntropyLoss() on model(batch) and labels.
What I expect: Since I set the gradient to 0, no matter what I do, nothing should change: the loss should be around the original value (2.4) all the time
What actually happens:

When I uncomment only line 6, everything works as expected: the loss is close to the constant 2.4.
When I uncomment only line 3, The learning happens as if I didn't call add_error at all. I.e. loss decreases at the same rate as usual SGD: 2.4 -> 1.7 -> 1.3 -> ... (per epoch). In other words, somehow the gradients are propagated.
When I uncomment lines 2 and 6: the weirdest thing of all. The loss increases to 4.3, and then slowly decreases 4.3 -> 4.2 -> 4.14 -> 4.1 -> ... (I suspect this decrease to be a result of batch normalization).

Note that in neither case I actually use error, I actually never use error to update the gradient.
Also, adding more lines like Line 2 doesn't affect the outcome.
Question: What's happening?

Why loss decreases? The gradient should be $0$ all the time.
How is it possible to change the loss value itself?

If it helps, I may try to produce MCVE and post it on pastebin (it'll be a wall of code, too large to fit here).


title: How to stack matrices with different size

body:
I have a list of matrices with size of (63,32,1,600,600), when I want to stack it with torch.stack(matrices).cpu().detach().numpy() it's raising with error:
"stack expects each tensor to be equal size, but got [32, 1, 600, 600] at entry 0 and [16, 1, 600, 600] at entry 62". Is tried for resizing but it did not work. I appreciate any recommendations.


title: High loss in neural network sequence classification

body:
I am using neural network to classify sequence of length 340 to 8 classes, I am using cross entropy as loss. I am getting very high number for the loss . I am wondering if I did mistake in calculating the loss for each epoch. Or should i use other loss function .
criterion = nn.CrossEntropyLoss()
if CUDA:
    criterion = criterion.cuda()
optimizer = optim.SGD(model.parameters(), lr=LEARNING_RATE, momentum=0.9)
loss_list = []
for epoch in range(N_EPOCHES):
    tot_loss=0
    running_loss =0
    model.train()
    loss_values = []
    acc_list = []
    acc_list = torch.FloatTensor(acc_list)
    sum_acc = 0
    # Training
    for i, (seq_batch, stat_batch) in enumerate(training_generator):
        # Transfer to GPU
        seq_batch, stat_batch = seq_batch.to(device), stat_batch.to(device)
        optimizer.zero_grad()
        # Model computation
        seq_batch = seq_batch.unsqueeze(-1)
        outputs = model(seq_batch)
        loss = criterion(outputs.argmax(1), stat_batch.argmax(1))
        loss.backward()
        optimizer.step()
        # print statistics
        running_loss += loss.item()*seq_batch.size(0)
        loss_values.append(running_loss/len(training_set))
        if i % 2000 == 1999:  # print every 2000 mini-batches
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 50000),"acc",(outputs.argmax(1) == stat_batch.argmax(1)).float().mean())
            running_loss = 0.0
        sum_acc += (outputs.argmax(1) == stat_batch.argmax(1)).float().sum()
    print("epoch" , epoch, "acc", sum_acc/len(training_generator))
print('Finished Training')

[1,  2000] loss: 14.205 acc tensor(0.5312, device='cuda:0')
[1,  4000] loss: 13.377 acc tensor(0.4922, device='cuda:0')
[1,  6000] loss: 13.159 acc tensor(0.5508, device='cuda:0')
[1,  8000] loss: 13.050 acc tensor(0.5547, device='cuda:0')
[1, 10000] loss: 12.974 acc tensor(0.4883, device='cuda:0')
epoch 1 acc tensor(133.6352, device='cuda:0')
[2,  2000] loss: 12.833 acc tensor(0.5781, device='cuda:0')
[2,  4000] loss: 12.834 acc tensor(0.5391, device='cuda:0')
[2,  6000] loss: 12.782 acc tensor(0.5195, device='cuda:0')
[2,  8000] loss: 12.774 acc tensor(0.5508, device='cuda:0')
[2, 10000] loss: 12.762 acc tensor(0.5156, device='cuda:0')
epoch 2 acc tensor(139.2496, device='cuda:0')
[3,  2000] loss: 12.636 acc tensor(0.5469, device='cuda:0')
[3,  4000] loss: 12.640 acc tensor(0.5469, device='cuda:0')
[3,  6000] loss: 12.648 acc tensor(0.5508, device='cuda:0')
[3,  8000] loss: 12.637 acc tensor(0.5586, device='cuda:0')
[3, 10000] loss: 12.620 acc tensor(0.6016, device='cuda:0')
epoch 3 acc tensor(140.6962, device='cuda:0')
[4,  2000] loss: 12.520 acc tensor(0.5547, device='cuda:0')
[4,  4000] loss: 12.541 acc tensor(0.5664, device='cuda:0')
[4,  6000] loss: 12.538 acc tensor(0.5430, device='cuda:0')
[4,  8000] loss: 12.535 acc tensor(0.5547, device='cuda:0')
[4, 10000] loss: 12.548 acc tensor(0.5820, device='cuda:0')
epoch 4 acc tensor(141.6522, device='cuda:0')


title: Error while installing Pytorch from source

body:
I need to use Pytorch with open MPI, so I'm having to install it from source. I'm using CentOS 8.
while runnning -
python setup.py install
i'm running into the following error.
CMake Warning at caffe2/CMakeLists.txt:755 (add_library):
  Cannot generate a safe runtime search path for target torch_cpu because
  files in some directories may conflict with libraries in implicit
  directories:

    runtime library [libgomp.so.1] in /usr/lib/gcc/x86_64-redhat-linux/8 may be hidden by files in:
      /home/aditya/anaconda3/lib

  Some of these libraries may not be found correctly.

-- Generating done
-- Build files have been written to: /home/aditya/pytorch/build
cmake3 --build . --target install --config Release -- -j 1
[2/2038] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/Version.cpp.o
FAILED: caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/Version.cpp.o
/usr/bin/g++  -DCPUINFO_SUPPORTED_PLATFORM=1 -DFMT_HEADER_ONLY=1 -DFXDIV_USE_INLINE_ASSEMBLY=0 -DHAVE_MALLOC_USABLE_SIZE=1 -DHAVE_MMAP=1 -DHAVE_SHM_OPEN=1 -DHAVE_SHM_UNLINK=1 -DMINIZ_DISABLE_ZIP_READER_CRC32_CHECKS -DNNP_CONVOLUTION_ONLY=0 -DNNP_INFERENCE_ONLY=0 -DONNXIFI_ENABLE_EXT=1 -DONNX_ML=1 -DONNX_NAMESPACE=onnx_torch -DUSE_DISTRIBUTED -DUSE_EXTERNAL_MZCRC -DUSE_RPC -DUSE_TENSORPIPE -D_FILE_OFFSET_BITS=64 -Dtorch_cpu_EXPORTS -Iaten/src -I../aten/src -I. -I../ -isystem third_party/gloo -isystem ../cmake/../third_party/gloo -isystem ../cmake/../third_party/googletest/googlemock/include -isystem ../cmake/../third_party/googletest/googletest/include -isystem ../third_party/protobuf/src -isystem ../third_party/gemmlowp -isystem ../third_party/neon2sse -isystem ../third_party/XNNPACK/include -I../cmake/../third_party/benchmark/include -isystem ../third_party -isystem ../cmake/../third_party/eigen -isystem /home/aditya/anaconda3/include/python3.8 -isystem /home/aditya/anaconda3/lib/python3.8/site-packages/numpy/core/include -isystem ../cmake/../third_party/pybind11/include -isystem /home/aditya/anaconda3/include -Icaffe2/contrib/aten -I../third_party/onnx -Ithird_party/onnx -I../third_party/foxi -Ithird_party/foxi -isystem ../third_party/ideep/mkl-dnn/include -isystem ../third_party/ideep/include -I../torch/csrc/api -I../torch/csrc/api/include -I../caffe2/aten/src/TH -Icaffe2/aten/src/TH -Icaffe2/aten/src -Icaffe2/../aten/src -Icaffe2/../aten/src/ATen -I../torch/csrc -I../third_party/miniz-2.0.8 -I../third_party/kineto/libkineto/include -I../third_party/kineto/libkineto/src -I../aten/src/TH -I../aten/../third_party/catch/single_include -I../aten/src/ATen/.. -Icaffe2/aten/src/ATen -I../caffe2/core/nomnigraph/include -isystem include -I../third_party/FXdiv/include -I../c10/.. -Ithird_party/ideep/mkl-dnn/include -I../third_party/ideep/mkl-dnn/src/../include -I../third_party/pthreadpool/include -I../third_party/cpuinfo/include -I../third_party/QNNPACK/include -I../aten/src/ATen/native/quantized/cpu/qnnpack/include -I../aten/src/ATen/native/quantized/cpu/qnnpack/src -I../third_party/cpuinfo/deps/clog/include -I../third_party/NNPACK/include -I../third_party/fbgemm/include -I../third_party/fbgemm -I../third_party/fbgemm/third_party/asmjit/src -I../third_party/FP16/include -I../third_party/tensorpipe -Ithird_party/tensorpipe -I../third_party/tensorpipe/third_party/libnop/include -I../third_party/fmt/include -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow -DHAVE_AVX_CPU_DEFINITION -DHAVE_AVX2_CPU_DEFINITION -O2 -DNDEBUG -DNDEBUG -fPIC   -DCAFFE2_USE_GLOO -DHAVE_GCC_GET_CPUID -DUSE_AVX -DUSE_AVX2 -DTH_HAVE_THREAD -Wall -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Wno-write-strings -Wno-unknown-pragmas -Wno-missing-braces -Wno-maybe-uninitialized -fvisibility=hidden -O2 -fopenmp -DCAFFE2_BUILD_MAIN_LIB -pthread -DASMJIT_STATIC -std=gnu++14 -MD -MT caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/Version.cpp.o -MF caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/Version.cpp.o.d -o caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/Version.cpp.o -c ../aten/src/ATen/Version.cpp
../aten/src/ATen/Version.cpp: In function ‘std::__cxx11::string at::get_mkldnn_version()’:
../aten/src/ATen/Version.cpp:45:13: error: ‘mkldnn_version_t’ does not name a type; did you mean ‘dnnl_version_t’?
       const mkldnn_version_t* ver = mkldnn_version();
             ^~~~~~~~~~~~~~~~
             dnnl_version_t
../aten/src/ATen/Version.cpp:46:37: error: ‘ver’ was not declared in this scope
       ss << "Intel(R) MKL-DNN v" << ver->major << "." << ver->minor << "." << ver->patch
                                     ^~~
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
  File "setup.py", line 864, in <module>
    build_deps()
  File "setup.py", line 354, in build_deps
    build_caffe2(version=version,
  File "/home/aditya/pytorch/tools/build_pytorch_libs.py", line 58, in build_caffe2
    cmake.build(my_env)
  File "/home/aditya/pytorch/tools/setup_helpers/cmake.py", line 345, in build
    self.run(build_args, my_env)
  File "/home/aditya/pytorch/tools/setup_helpers/cmake.py", line 140, in run
    check_call(command, cwd=self.build_dir, env=env)
  File "/home/aditya/anaconda3/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake3', '--build', '.', '--target', 'install', '--config', 'Release', '--', '-j', '1']' returned non-zero exit status 1.


title: PyTorch: saving both weights and model definition

body:
During prototyping, I often perform numerous changes to a PyTorch model. For instance, suppose the first model I am experimenting with is:
class Model(nn.Module):
    def __init__(self, **args):
        super().__init__()
        self.l1 = nn.Linear(128, 1)

then I will add another layer:
class Model(nn.Module):
    def __init__(self, **args):
        super().__init__()
        self.l1 = nn.Linear(128, 32)
        self.l2 = nn.Linear(32, 1)

or maybe add some convolutions, and so on.
The problem is that I often get disorganized the more experiments I perform, as I haven't found a straightforward way of saving both model definition and its weights so that I can load a previous state.
I know I can do:
torch.save({'model': Model(), 'state': model.state_dict()}, path)
# or directly
torch.save(model, path)

but then loading the model also requires that the model class (here, Model) to exist in the current file.
In Keras you can simply do:
model = ...  # Get model (Sequential, Functional Model, or Model subclass)
model.save('path/to/location')

which saves model's architecture/config and weights, among other things. This means that you can load the model without having defined the architecture:
model = keras.models.load_model('path/to/location')

Refering to Keras model saving:

The SavedModel and HDF5 file contains:

the model's configuration (topology)
the model's weights
the model's optimizer's state (if any)

Thus models can be reinstantiated in the exact same state, without any
of the code used for model definition or training.

This is what I want to achieve in PyTorch.
Is there a similar approach for PyTorch? What is the best practice for these situations?


title: pytorch collections.OrderedDict' object has no attribute 'to'

body:
this is my main code,but I don't know how to fix the problem?
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = torch.load('./checkpoints/fcn_model_5.pth')  # 加载模型
model = model.to(device)


title: Ensure that all samples in a batch have the same shape in pytorch dataloader

body:
My task is to do a multi label classification on a custom dataset with pyTorch and BERT. My data contains about 1500 samples. The amount of words can vary between 1000 and 50k words. Because BERT can only handle a max sequence of 512 I am using a sliding window approach on my data. Please note that a data sample can  has several sentences.
For reference I´m working with the example notebooks here and from huggingface.
Here is a minimal version of my script:
import transformers
from torch.utils.data import Dataset, DataLoader, RandomSampler, SequentialSampler
from transformers import BertTokenizer, BertModel, BertConfig
import pandas as pd
import torch
from torch import cuda
import math
import transformers
from transformers import BertTokenizer, BertModel, BertConfig, AutoTokenizer
from torch.utils.data import Dataset, DataLoader, RandomSampler, SequentialSampler
device = 'cuda' if cuda.is_available() else 'cpu'

MAX_LEN = 400
STRIDE = 20
TRAIN_BATCH_SIZE = 8
VALID_BATCH_SIZE = 4
EPOCHS = 1
LEARNING_RATE = 1e-05

model_checkpoint = "bert-base-german-cased"
tokenizer = AutoTokenizer.from_pretrained(model_checkpoint, local_files_only=True)
assert isinstance(tokenizer, transformers.PreTrainedTokenizerFast)

class CustomDataset(Dataset):

    def __init__(self, dataframe, tokenizer, max_len, stride):
        self.tokenizer = tokenizer
        self.data = dataframe
        self.text = dataframe.text
        self.targets = self.data.labels
        self.max_len = max_len
        self.stride = stride

    def __len__(self):
        return len(self.text)

    def __getitem__(self, index):
        text = str(self.text[index])
        text = " ".join(text.split())

        inputs = self.tokenizer(
            text,
            None,
            max_length=MAX_LEN,
            stride=STRIDE,
            padding='max_length',
            truncation='only_first',
            return_overflowing_tokens=True,
        )
        ids = inputs['input_ids']
        mask = inputs['attention_mask']
        token_type_ids = inputs["token_type_ids"]

        return {
            'ids': torch.tensor(ids, dtype=torch.long),
            'mask': torch.tensor(mask, dtype=torch.long),
            'token_type_ids': torch.tensor(token_type_ids, dtype=torch.long),
            'targets': torch.tensor(self.targets[index], dtype=torch.float)
        }

I think the sliding window is working because if I run [len(x) for x in inputs["input_ids"]] I get a list of input_ids for my paragraph/text.
# Creating the dataset and dataloader for the neural network
train_size = 0.8
train_dataset=training_frame.sample(frac=train_size,random_state=200)
test_dataset=training_frame.drop(train_dataset.index).reset_index(drop=True)
train_dataset = train_dataset.reset_index(drop=True)

print("FULL Dataset: {}".format(training_frame.shape))
print("TRAIN Dataset: {}".format(train_dataset.shape))
print("TEST Dataset: {}".format(test_dataset.shape))

training_set = CustomDataset(train_dataset, tokenizer, MAX_LEN, STRIDE)
testing_set = CustomDataset(test_dataset, tokenizer, MAX_LEN, STRIDE)

train_params = {'batch_size': TRAIN_BATCH_SIZE,
                'shuffle': False,
                'num_workers': 0
                }

test_params = {'batch_size': VALID_BATCH_SIZE,
                'shuffle': True,
                'num_workers': 0
                }

training_loader = DataLoader(training_set, **train_params)
testing_loader = DataLoader(testing_set, **test_params)

Until here the script is running without any error, but if I try to iterate over training_set like here:
train_iter = iter(training_loader)
print(type(train_iter))
text, labels = train_iter.next()
print(text.size())
print(labels.size())

I get the following error:
RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 0. Got 133 and 75 in dimension 1 at /opt/conda/conda-bld/pytorch_1556653215914/work/aten/src/TH/generic/THTensor.cpp:711

Process finished with exit code 1

In this question André mentions that the loaded batches have different shapes and that´s how this error occures. He suggests to set the batch_size = 1. However I want to use the defined the batch_size defined in my script.
I think the sliding window causes the error because the list of input_ids can vary per sample in my batchsamples because the total length of the text can be different.
How can I ensure that my data fed to the network has always the same shape?


title: Pytorch: memory management

body:
Following one code example by pytorch, I am interested in the following lines

del model.rewards[:]
del model.saved_actions[:]

I wonder when the user has to handle memory clean up and when he does not.

I am mostly interested in critical problems or memory leaks that could be caused by not managing memory correctly.

I am less interested in memory optimizations.


title: Multiple cpu producers with few gpus not utilize 100% of the gpus (pytorch)

body:
I tried to implement board game self-play data generation in parallel using multiple cpus to do self-paly concurrently. For parent process, i created 4 NN model for 30cpus (1 model for 10 cpus and 1 model to train) each model is in different gpus.(the model is implemented as 20 blocks resnet-like architecture with batchnorm) Pseudo code as follows

nnet = NN(gpu_num=0)
nnet1 = NN(gpu_num=1)
nnet2 = NN(gpu_num=2)
nnet3 = NN(gpu_num=3)

for i in range(num_iteration):
    nnet1.load_state_dict(nnet.state_dict())
    nnet2.load_state_dict(nnet.state_dict())
    nnet3.load_state_dict(nnet.state_dict())
    samples = parallel_self_play()
    nnet.train(samples)

parallel_self_play() is implemented as follows

pool = mp.Pool(processes=num_cpu) #30
for i in range(self.args.numEps):
    results = []
    if i % 3 == 0:
        net = self.nnet1
    elif i % 3 == 1:
        net = self.nnet2
    else:
        net = self.nnet3

     results.append(pool.apply_async(AsyncSelfPlay, args=(net))
     # get results from results array then return it
     return results

My code work perfectly fine with almost 100% gpu utilization throughout the first self-play (less than 10 minutes for an iteration) but after the first iteration (training) when i loaded new weights into nnet1-3 gpu utilization never reach 80% again (~30min - 1hour per iteration). I notice a few things while mess around with me code

This model includes batchnorm layers, when switch model to train() mode -> train -> switch back to eval() causes the self-play (use forward pass from model) to not use gpu at all.
If it doesn't switch from eval() -> train() (train using eval mode) this causes gpu utilization to be lower  (30-50%) but not entirely gone.
If the models that are not the main one doesn't load the weights from the main one, self-play still utilize 100% gpu so my guess is that something happened during training process and change some states in the model.

This also happen when use only 8 cpus - 1gpu architecture and train model on the fly (no intermediate one).

Can someone guide me where to fix my code or how i should train my model?


title: pytorch: inception v3 parameter empty error

body:
I am using inception_v3 from torchvision.models as my base model and adding an FC layer at the end to get features. However, I am getting an empty parameter error.

import torch
import torch.nn as nn
import torchvision.models as models

class Baseline(nn.Module):
    def __init__(self, out_size):
        super().__init__()
        model = models.inception_v3(pretrained=True)
        model.fc = nn.Linear(2048, out_size)
        model.aux_logits = False

        # Freeze model weights
        for param in model.parameters():
            param.requires_grad = False

        self.parameters = nn.ParameterList()

    def forward(self, image):
        x = model(image)
        x = x.view(x.size(0), -1)
        x = model.fc = (x)
        return x


title: How can I find a solution for the "FileNotFoundError"

body:
I'm currently working on an image classifier project. During the testing of the predict function, I receive the error: FileNotFoundError: [Errno 2] No such file or directory: 'flowers/test/1/image_06760'

the path of the file is correct.
You can find the whole notebook here:
https://github.com/MartinTschendel/image-classifier-1/blob/master/190829-0510-Image%20Classifier%20Project.ipynb

Here is the respective predict function and the test of this function:

def predict(image_path, model, topk=5):
    ''' Predict the class (or classes) of an image using a trained deep learning model.
    '''

    #loading model
    loaded_model = load_checkpoint(model)

    # implement the code to predict the class from an image file
    img = Image.open(image_name)
    img = process_image(img)

    # convert 2D image to 1D vector
    img = np.expand_dims(img, 0)

    img = torch.from_numpy(img)

    #set model to evaluation mode and turn off gradients
    loaded_model.eval()
    with torch.no_grad():
        #run image through network
        output = loaded_model.forward(img)

    #calculate probabilities
    probs = torch.exp(output)
    probs_top = probs.topk(topk)[0]
    index_top = probs.topk(topk)[1]

    # Convert probabilities and outputs to lists
    probs_top_list = np.array(probs_top)[0]
    index_top_list = np.array(index_top[0])

    #load index and class mapping
    loaded_model.class_to_idx = train_data.class_to_idx
    #invert index-class dictionary
    idx_to_class = {x: y for y, x in class_to_idx.items()}

    #convert index list to class list
    classes_top_list = []
    for index in index_top_list:
        classes_top_list += [idx_to_class[index]]

    return probs_top_list, classes_top_list

# test predict function

# inputs equal paths to saved model and test image
model_path = '190824_checkpoint.pth'
image_name = data_dir + '/test' + '/1/' + 'image_06760'

probs, classes = predict(image_name, model_path, topk=5)

print(probs)
print(classes)

This is the error I receive:

FileNotFoundError: [Errno 2] No such file or directory: 'flowers/test/1/image_06760'


title: Applying a pytorch CNN to video?

body:
I am looking for some advice on how to apply a pytorch CNN to a video as opposed to an image.

Picture a drone flying over an area and using video to capture some objects below. I have a CNN trained on images of objects, and want to count the objects in the video.

Currently my strategy has been to convert the video to frames as PNGs and running the CNN on those PNGs. this seems inefficient, and I am struggling with how to count the objects without duplicating (frame 1 and frame 1+n will overlap).

It would be appreciated if someone had some advice, or a suggested tutorial/code set that did this.  Thanks in advance.


title: NN regression loss value not decreasing

body:
I'm training a NN with Pytorch to predict the expected price for the Boston dataset.
The network looks like this:

from sklearn.datasets import load_boston
from torch.utils.data.dataset import Dataset
from torch.utils.data import DataLoader
import torch.nn.functional as F
import torch
import torch.nn as nn
import torch.optim as optim

class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(13, 128)
        self.fc2 = nn.Linear(128, 64)
        self.fc3 = nn.Linear(64, 32)
        self.fc4 = nn.Linear(32, 16)
        self.fc5 = nn.Linear(16,1)

    def forward(self, x):
       x = self.fc1(x)
       x = self.fc2(x)
       x = F.relu(x)
       x = self.fc3(x)
       x = F.relu(x)
       x = self.fc4(x)
       x = F.relu(x)
       return self.fc5(x)

And the data loader:

class BostonData(Dataset):
    __xs = []
    __ys = []

    def __init__(self, train = True):
        df = load_boston()
        index = int(len(df["data"]) * 0.7)
        if train:
            self.__xs = df["data"][0:index]
            self.__ys = df["target"][0:index]
        else:
            self.__xs = df["data"][index:]
            self.__ys = df["target"][index:]

    def __getitem__(self, index):
        return self.__xs[index], self.__ys[index]

    def __len__(self):
        return len(self.__xs)

In my first attempt I didn't add the ReLU units, but after a little bit of research I saw that adding them is a common practice, but It didn't work out for me.

Here is the training code:

dset_train = BostonData(train = True)
dset_test = BostonData(train = False)
train_loader = DataLoader(dset_train, batch_size=30, shuffle=True)
test_loader = DataLoader(dset_train, batch_size=30, shuffle=True)

optimizer = optim.Adam(net.parameters(), lr = 0.001)
criterion = torch.nn.MSELoss()
EPOCHS = 10000

lloss = []

for epoch in range(EPOCHS):
    for trainbatch in train_loader:
        X,y = trainbatch
        net.zero_grad()
        output = net(X.float())
        loss = criterion(output, y)
        loss.backward()
        optimizer.step()
    lloss.append(loss)
    print(loss)

After 10k epochs, the loss graph looks like the following

where I don't see any clear decrease.
I don't know if I'm messing up with the torch.nn.MSELoss(), the optimizer or maybe with the net topology, so any help will be appreciated.

Edit:
Changing the learning rate and normalizing the data didn't work for me. I added the line self.__xs = (self.__xs - self.__xs.mean()) / self.__xs.std()
 and a change to lr = 0.01. The loss plot is very similar to the first one.

Same plot for lr = 0.01 and normalizing after 1000 epochs:


title: What is an efficient way to read and store input data from multiple files in python?

body:
I want to train a classifier for 8000 data samples in python (using sklearn or pytorch libraries). But, each data is a times-series stored in a .wav file.

Do I have to first read all 8000 files in a loop and store them in a big file, and read that big file each time I want to train a classifier? or there is a more efficient way for such cases?


title: Running in Google Colab

body:
Good evening.
I want to run my python program on Google collab, but in what place I should download files and when open in python file?
How do I open this file?


title: Pytorch JIT Class

body:
Is every method of JIT Class complied? If so why JIT function is little faster than the class method?
Please see the code below

import torch
import time

@torch.jit.script
class Test:
    def __init__(self):
        pass

    def calc(self, input: torch.Tensor) -> torch.Tensor:
        return 1.0 / torch.exp(input)

@torch.jit.script
def calc(a):
  return 1.0 / torch.exp(a)

for i in range(5):
  test = Test()
  start = time.time()
  a = torch.zeros(6000, 200)
  for j in range(1000):
    a = test.calc(a)
  print(time.time() - start)

for i in range(5):
  start = time.time()
  a = torch.zeros(6000, 200)
  for j in range(1000):
    a = calc(a)
  print(time.time() - start)


title: WGAN loss diverges

body:
I have some trouble to understand the WGAN Loss values.

I understand that we do not have a discriminator anymore, but a critic. Difference is, that the Discriminator tries to classify the input ergo map it to either 0 or 1 and that the critic tries to score them.
The scoring map to any real number and the Loss function is:

  L_critic = Critic(fake_input) - Critic(real_input) + weight *  gradient_penalty

And the Loss function for the generator which produces the fake_input is:

  L_generator = - Critic(fake_input)

My question is:
Over the Epochs my Loss values for the generator and Discriminator diverges and takes really high numbers. Is this common? I have seen elsewhere that the Loss of the Critic drop to 0, but what ensures that?
As far as I understood the Critic tries to score the real from fake as far apart as possible.

[]


title: Why does the output from VQ-Wav2Vec from FairSeq missing frames?

body:
I am using the fairseq library to run an example code for feature extraction with the VQ-Wav2Vec code as written below:

In [6]: import torch
   ...: from fairseq.models.wav2vec import Wav2VecModel

In [7]: cp = torch.load('wav2vec_models/checkpoint_best.pt')
   ...: model = Wav2VecModel.build_model(cp['args'], task=None)
   ...: model.load_state_dict(cp['model'])
   ...: model.eval()

In [9]: wav_input_16khz = torch.randn(1,10000)
   ...: z = model.feature_extractor(wav_input_16khz)
   ...: f, idxs = model.vector_quantizer.forward_idx(z)
   ...: print(idxs.shape, f.shape)

>>>> torch.Size([1, 60, 4]) torch.Size([1, 512, 60])

My understanding is that the vq-wav2vec processes every 10ms of input speech (assumed to be sampled at 16K samples / sec) samples and outputs a feature vector of size [512] samples for each of these 10ms of speech. So given that the input speech is 10000 samples, we are supposed to get 62 frames ( 62 * 160 = 9920 samples).

Why do I see only 60 frames?


title: NaN when I use Group Normalization in Pytorch

body:
i was trying groupnormalization for efficientnet ,
my model code is :
out_dim = 5
enet_type = 'efficientnet-b0'

pretrained_model = {
    'efficientnet-b0': '../input/efficientnet-pytorch/efficientnet-b0-08094119.pth'
}

class enetv2(nn.Module):
    def __init__(self, backbone, out_dim):
        super(enetv2, self).__init__()
        self.enet = enet.EfficientNet.from_name(backbone)
        self.enet.load_state_dict(torch.load(pretrained_model[backbone]))

        self.myfc = nn.Linear(self.enet._fc.in_features, out_dim)
        self.enet._fc = nn.Identity()

    def extract(self, x):
        return self.enet(x)

    def forward(self, x):
        x = self.extract(x)
        x = self.myfc(x)
        return x
model = enetv2(enet_type, out_dim=out_dim)
model = model.to(device)

with this model i get no error,everything works fine,however i wanted to change all the batch normalization layers to group normalization layer like this :
for  i in range(0,len(model.enet._blocks)):
    try:
        print("...........found BN layer  so Replacing Bn0 with GN.........")
        gn0 = model.enet._blocks[i]._bn0.num_features
        model.enet._blocks[i]._bn0 = nn.GroupNorm(1,num_channels = gn0)
        print(model.enet._blocks[i]._bn0)

    except:
        print("BN layer Not  found!!")

    gn1 = model.enet._blocks[i]._bn1.num_features
    gn2 = model.enet._blocks[i]._bn2.num_features

    model.enet._blocks[i]._bn1 = nn.GroupNorm(1,num_channels = gn1)
    model.enet._blocks[i]._bn2 = nn.GroupNorm(1,num_channels = gn2)

    print(model.enet._blocks[i]._bn1)
    print(model.enet._blocks[i]._bn2)

and now i am getting nan with that, after 1st and 2nd epoch i can see loss getting nan and after second epoch giving me this error : ValueError: Input contains NaN, infinity or a value too large for dtype(‘float32’).


title: Contrastive loss dose not change after some epochs

body:
I am trying to implement a Contrastive loss for Cifar10 in PyTorch and then in 3D images. I wrote the following pipeline and I checked the loss. Logically it is correct, I checked it. But I have three problems, the first problem is that the convergence is so slow. The second problem is that after some epochs the loss dose does not decrease anymore and it is fixed. My third problem is that for some settings, it does not proceed anymore, I mean it sticks for some reason and I have to restart the kernel. I will be very thankful if someone can help me to say where I am doing a mistake. Thank you very much in advance.
transform = transforms.Compose([transforms.ToTensor(),
                                transforms.Normalize([0.491, 0.482, 0.447], [0.247, 0.243,0.261])])
class PairwiseAg(object):
    def __init__(self, transform):
        self.transform = transform
    def __call__(self, vol):
        voli = self.transform(vol)
        volj = self.transform(vol)
        return(voli,volj)
dataset =datasets.CIFAR10('./data', train=True, transform=PairwiseAg(transform), download=True)
ss = ShuffleSplit(n_splits=1, test_size= 0.2,random_state=0)
for train_idx, val_idx in ss.split(dataset):
    train_idx = train_idx
    val_idx = val_idx
train_set = Subset(dataset,train_idx)
val_set = Subset(dataset,val_idx)
train_loader = DataLoader(train_set,batch_size=512,shuffle=True,num_workers=2,drop_last=False)
val_loader = DataLoader(val_set,batch_size=512,shuffle=True,num_workers=2,drop_last=False)
device = torch.device('cuda:4' if torch.cuda.is_available() else 'cpu')
model = torchvision.models.resnet18(pretrained=False, progress=True)
model.fc = nn.Linear(in_features=512,out_features=128)
model = model.to(device)
class NT_Xent(nn.Module):
    def __init__(self, temperature, device):
        super(NT_Xent, self).__init__()
        self.temperature = temperature
        self.device = device
        self.criterion = nn.CrossEntropyLoss(reduction="sum")
        self.similarity_f = nn.CosineSimilarity(dim=2)
    def forward(self, z_i, z_j):
        self.batch_size= z_i.size()[0]
        self.mask = torch.ones((self.batch_size * 2, self.batch_size * 2), dtype=bool)
        self.mask = self.mask.fill_diagonal_(0)
        for i in range(self.batch_size):
            self.mask[i, self.batch_size + i] = 0
            self.mask[self.batch_size + i, i] = 0
        z_i= F.normalize(z_i, dim=1)
        z_j= F.normalize(z_j, dim=1)
        p1 = torch.cat((z_i, z_j), dim=0)
        sim = self.similarity_f(p1.unsqueeze(1), p1.unsqueeze(0)) / self.temperature)
        sim_i_j = torch.diag(sim, self.batch_size)
        sim_j_i = torch.diag(sim, -self.batch_size)
        positive_samples = torch.cat((sim_i_j, sim_j_i), dim=0).reshape(self.batch_size * 2,1)
        negative_samples = sim[self.mask].reshape(self.batch_size * 2, -1)
        labels = torch.zeros(self.batch_size * 2).to(positive_samples.device).long()
        logits = torch.cat((positive_samples, negative_samples), dim=1)
        loss = self.criterion(logits, labels)
        loss /= 2 * self.batch_size
        return(loss)
optimizer = optim.Adam(model.parameters(),lr=5e-1)
criterion = NT_Xent(0.7,device)
def train(epoch):
    model.train()
    total_loss = 0
    for i,(X,_) in enumerate(train_loader):
        X1 = X[0].to(device)
        X2 = X[1].to(device)
        h1 = model(X1)
        h2 = model(X2)
        loss = criterion(h1.float(),h2.float())
        total_loss+=loss.item()
        optimizer.zero_grad()
        loss.backwar()
        optimizer.step()
    return(total_loss/len(train_loader.dataset))

def val():
    model.eval()
    total_loss = 0
    for i, (X,_) in enumerate(val_loader):
        X1 = X[0].to(device)
        X2 = X[1].to(device)
        h1 = model(X1)
        h2 = model(X2)
        loss = criterion(h1.float(),h2.float())
        total_loss+=loss.item()
    return(total_loss/len(val_loader.dataset))

def main(num_epochs):
    for epoch in range(num_epochs):
        tr_loss = train(epoch)
        val_loss = val()

output:
curren_lr:0.5
Epoch: 0 train_loss:0.0131  val_loss:0.0131
Epoch: 1 train_loss:0.0127  val_loss:0.0129
Epoch: 2 train_loss:0.0126  val_loss:0.0129
Epoch: 3 train_loss:0.0126  val_loss:0.0129
Epoch: 4 train_loss:0.0125  val_loss:0.0129
Epoch: 5 train_loss:0.0124  val_loss:0.0128
Epoch: 6 train_loss:0.0124  val_loss:0.0126
Epoch: 7 train_loss:0.0123  val_loss:0.0127
Epoch: 8 train_loss:0.0123  val_loss:0.0125
Epoch: 9 train_loss:0.0123  val_loss:0.0127
Epoch: 10 train_loss:0.0123  val_loss:0.0126
Epoch: 11 train_loss:0.0122  val_loss:0.0125```


title: Python 3.8 RAM owerflow and loading issues

body:
First, I want to mention, that this is our first project in a bigger scale and therefore we don't know everything but we learn fast.
We developed a code for image recognition. We tried it with a raspberry pi 4b but quickly faced that this is way to slow overall. Currently we are using a NVIDIA Jetson Nano. The first recognition was ok (around 30 sec.) and the second try was even better (around 6-7 sec.). The first took so long because the model will be loaded for the first time. Via an API the image recognition can be triggered and the meta data from the AI model will be the response. We use fast-API for this.
But there is a problem right now, where if I load my CNN as a global variable in the beginning of my classification file (loaded on import) and use it within a thread I need to use mp.set_start_method('spawn') because otherwise I will get the following error:
"RuntimeError: Cannot re-initialize CUDA in forked subprocess.
To use CUDA with multiprocessing, you must use the 'spawn' start method"
Now that is of course an easy fix. Just add the method above before starting my thread. Indeed this works but another challenge occurs at the same time. After setting the start method to 'spawn' the ERROR disappears but the Jetson starts to allocate way to much memory.
Because of the overhead and preloaded CNN model, the RAM is around 2.5Gig before the thread starts. After the start it doesn’t stop allocating RAM, it consumes all 4Gig of the RAM and also the whole 6Gig Swap. Right after this, the whole API process kill with this error: "cannot allocate memory" which is obvious.
I managed to fix that as well just by loading the CNN Model in the classification function. (Not preloading it on the GPU as in the two cases before). However, here I got problem as well. The process of loading the model to the GPU takes around 15s - 20s and this every time the recognition starts. This is not suitable for us and we are wondering why we cannot pre-load the model without killing the whole thing after two image-recognitions.  Our goal is to be under 5 sec with this.
#clasify
import torchvision.transforms as transforms
from skimage import io
import time
from torch.utils.data import Dataset
from .loader import *
from .ResNet import *

#if this part is in the classify() function than no allocation problem occurs
net = ResNet152(num_classes=25)
net = net.to('cuda')
save_file = torch.load("./model.pt", map_location=torch.device('cuda'))
net.load_state_dict(save_file)

def classify(imgp=""):
    #do some classification with the net
    pass

if __name__ == '__main__':
    mp.set_start_method('spawn') #if commented out the first error ocours
    manager = mp.Manager()
    return_dict = manager.dict()
    p = mp.Process(target=classify, args=('./bild.jpg', return_dict))
    p.start()
    p.join()
    print(return_dict.values())

Any help here will be much appreciated. Thank you.


How can I implement a loss function in Pytorch if some of its operations are not included in torch functions?
e.g. I have a network named Net and an input named Input, then we can calculate out = Net(Input).
Normally we can calculate the loss by loss = loss_fn(out, ground_truth).
However, what if I need to do something on out? Like, I want to firstly apply Fourier transform on it. Since FFT is not supported very well in pytorch functions, I may need to change to numpy to do this. I tried this but the loss backward seems strange. Can anyone tell me the right way of doing this? Thx a lot :)


title: Indexing list of tensors

body:
I have two identical lists of tensors (with different sizes) except that for the first one all of the tensors are assigned to the cuda device. For example:
list1=[torch.tensor([0,1,2]).cuda(),torch.tensor([3,4,5,6]).cuda(),torch.tensor([7,8]).cuda()]
>>> list1
[tensor([0, 1, 2], device='cuda:0'), tensor([3, 4, 5, 6], device='cuda:0'), tensor([7, 8], device='cuda:0')]

list2=[torch.tensor([0,1,2]),torch.tensor([3,4,5,6]),torch.tensor([7,8])]
>>> list2
[tensor([0, 1, 2]), tensor([3, 4, 5, 6]), tensor([7, 8])]

I want to extract some tensors from the lists according to an array of indices such as:
ind=torch.tensor([0,2])
>>> ind
tensor([0, 2])

So my solution was to do something like that:
np.array(list1)[ind]

np.array(list2)[ind]

My question is why it works with the first list with the tensors defined on the cuda device and gives an error with the second list as shown below:
>>> np.array(list1)[ind]
array([tensor([0, 1, 2], device='cuda:0'),
       tensor([7, 8], device='cuda:0')], dtype=object)

>>> np.array(list2)[ind]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: only one element tensors can be converted to Python scalars

EDIT:
Just to clarify, the error isn't raised because the tensors have different shapes. The following examples illustrate this point:
list3=[torch.tensor([1,2,3]).cuda()]
list4=[torch.tensor([1,2,3]).cuda(),torch.tensor([4,5,6]).cuda()]
list5=[torch.tensor([1,2,3])]
list6=[torch.tensor([1,2,3]),torch.tensor([4,5,6])]

And the results are:
>>> np.array(list3)
array([tensor([1, 2, 3], device='cuda:0')], dtype=object)
>>> np.array(list4)
array([tensor([1, 2, 3], device='cuda:0'),
       tensor([4, 5, 6], device='cuda:0')], dtype=object)
>>> np.array(list5)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: only one element tensors can be converted to Python scalars
>>> np.array(list6)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: only one element tensors can be converted to Python scalars