title: Learning rate scheduler - PyTorch

body:
I am trying to recreate results of “The Lottery Ticket  Hypothesis” by Frankle et al. in PyTorch. They are using CIFAR-10 which has 50K  training images. Using a batch size = 64 gives 781 iterations/steps in  one epoch. I am trying to implement this in PyTorch.
For VGG-18 & ResNet-18, the authors propose the following learning rate schedule

Linear learning rate warmup for first k = 7813 steps from 0.0 to 0.1

After 10 epochs or 7813 training steps, the learning rate schedule is as follows-

For the next 21094 training steps (or, 27 epochs), use a learning rate of 0.1
For the next 13282 training steps (or, 17 epochs), use a learning rate of 0.01
For any remaining training steps, use a learning rate of 0.001

I have implemented this in TensorFlow2 as follows:
from typing import Callable, List, Optional, Union

class WarmUp(tf.keras.optimizers.schedules.LearningRateSchedule):
    """
    Applies a warmup schedule on a given learning rate decay schedule.
    Args:
        initial_learning_rate (:obj:`float`):
            The initial learning rate for the schedule after the warmup (so this will be the learning rate at the end
            of the warmup).
        decay_schedule_fn (:obj:`Callable`):
            The schedule function to apply after the warmup for the rest of training.
        warmup_steps (:obj:`int`):
            The number of steps for the warmup part of training.
        power (:obj:`float`, `optional`, defaults to 1):
            The power to use for the polynomial warmup (defaults is a linear warmup).
        name (:obj:`str`, `optional`):
            Optional name prefix for the returned tensors during the schedule.
    """
    def __init__(
        self,
        initial_learning_rate: float,
        decay_schedule_fn: Callable,
        warmup_steps: int,
        power: float = 1.0,
        name: str = None,
    ):
        super().__init__()
        self.initial_learning_rate = initial_learning_rate
        self.warmup_steps = warmup_steps
        self.power = power
        self.decay_schedule_fn = decay_schedule_fn
        self.name = name

    def __call__(self, step):
        with tf.name_scope(self.name or "WarmUp") as name:
            # Implements polynomial warmup. i.e., if global_step < warmup_steps, the
            # learning rate will be `global_step/num_warmup_steps * init_lr`.
            global_step_float = tf.cast(step, tf.float32)
            warmup_steps_float = tf.cast(self.warmup_steps, tf.float32)
            warmup_percent_done = global_step_float / warmup_steps_float
            warmup_learning_rate = self.initial_learning_rate * tf.math.pow(warmup_percent_done, self.power)
            return tf.cond(
                global_step_float < warmup_steps_float,
                lambda: warmup_learning_rate,
                lambda: self.decay_schedule_fn(step - self.warmup_steps),
                name=name,
            )

    def get_config(self):
        return {
            "initial_learning_rate": self.initial_learning_rate,
            "decay_schedule_fn": self.decay_schedule_fn,
            "warmup_steps": self.warmup_steps,
            "power": self.power,
            "name": self.name,
        }

boundaries = [21093, 34376]
values = [0.1, 0.01, 0.001]

learning_rate_fn = tf.keras.optimizers.schedules.PiecewiseConstantDecay(boundaries, values)

warmup_shcedule = WarmUp(initial_learning_rate = 0.1, decay_schedule_fn = learning_rate_fn, warmup_steps = 7813)

optimizer = tf.keras.optimizers.SGD(learning_rate = warmup_shcedule, momentum = 0.9, decay = 0.0, nesterov = False)

I then train model using “tf.GradientTape” and view the learning rate as follows:
optimizer._decayed_lr('float32').numpy()

The resulting learning rate during the 60 epochs of training can be viewed visually as:

You can access the complete TF2 code here. Since I am new to PyTorch, can you show me how I can achieve the same learning rate scheduler in torch?
Thanks


title: TypeError: 'module' object is not subscriptable (Pytorch)

body:
I'm currently self studying and implementing PyTorch.
While doing transfer learning with this tutorial:
https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html
I encountered an error in this particular code block:
image_datasets = {x: datasets.ImageFolder(os.path.join(main_dataset, x),
                                          transforms[x])
                 for x in ['Train','Test']}

The error code is as below:
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-41-c83fc3ba4a1c> in <module>()
      1 image_datasets = {x: datasets.ImageFolder(os.path.join(main_dataset, x),
      2                                           transforms[x])
----> 3                 for x in ['Train','Test']}

<ipython-input-41-c83fc3ba4a1c> in <dictcomp>(.0)
      1 image_datasets = {x: datasets.ImageFolder(os.path.join(main_dataset, x),
      2                                           transforms[x])
----> 3                 for x in ['Train','Test']}

TypeError: 'module' object is not subscriptable

Which I am certain I have not used any module wrongly or any typos in this particular code block.
While searching the error code, I did not find anything related to this as well.
Please help, thanks.


title: TypeError: new(): argument 'size' must be tuple of ints, but found element of type NoneType at pos 2 when using pytorch, using nn.linear

body:
File "C:\Users\J2\Desktop\Pytorchseries\thenn.py", line 50, in
net = Net()
TypeError: new(): argument 'size' must be tuple of ints, but found element of type NoneType at pos 2
If it helps I was following the sentdex pytorch tutorial. Any help would be appreciated. I am new to machine learning, and I was hoping that this would work. Please help me out!

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
import matplotlib.pyplot as plt
import tqdm

training_data = np.load('training_data.npy', allow_pickle=True)
print(len(training_data))

X = torch.Tensor([i[0] for i in training_data]).view(-1,50,50)
X = X/255.0
y = torch.Tensor([i[1] for i in training_data])

plt.imshow(X[0], cmap='gray')
print(y[0])

class Net(nn.Module):
    def __init__(self):
        super().__init__() # just run the init of parent class (nn.Module)
        self.conv1 = nn.Conv2d(1, 32, 5) # input is 1 image, 32 output channels, 5x5 kernel / window
        self.conv2 = nn.Conv2d(32, 64, 5) # input is 32, bc the first layer output 32. Then we say the output will be 64 channels, 5x5 kernel / window
        self.conv3 = nn.Conv2d(64, 128, 5)

        x = torch.randn(50,50).view(-1,1,50,50)
        self._to_linear = None
        self.convs(x)

        self.fc1 = nn.Linear(self._to_linear, 512) #flattening.
        self.fc2 = nn.Linear(512, 2) # 512 in, 2 out bc we're doing 2 classes (dog vs cat).

    def convs(self, x):
        # max pooling over 2x2
        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
        x = F.max_pool2d(F.relu(self.conv2(x)), (2, 2))
        x = F.max_pool2d(F.relu(self.conv3(x)), (2, 2))

    def forward(self, x):
        x = self.convs(x)
        x = x.view(-1, self._to_linear)  # .view is reshape ... this flattens X before
        x = F.relu(self.fc1(x))
        x = self.fc2(x) # bc this is our output layer. No activation here.
        return F.softmax(x, dim=1)

        if self._to_linear is None:
            self._to_linear = x[0].shape[0]*x[0].shape[1]*x[0].shape[2]
        return x

net = Net()
print(net)

import torch.optim as optim

optimizer = optim.Adam(net.parameters(), lr=0.001)
loss_function = nn.MSELoss()

X = torch.Tensor([i[0] for i in training_data]).view(-1,50,50)
X = X/255.0
y = torch.Tensor([i[1] for i in training_data])

VAL_PCT = 0.1  # lets reserve 10% of our data for validation
val_size = int(len(X)*VAL_PCT)
print(val_size)

train_X = X[:-val_size]
train_y = y[:-val_size]

test_X = X[-val_size:]
test_y = y[-val_size:]

print(len(train_X), len(test_X))

BATCH_SIZE = 100
EPOCHS = 1

for epoch in range(EPOCHS):
    for i in tqdm(range(0, len(train_X), BATCH_SIZE)): # from 0, to the len of x, stepping BATCH_SIZE at a time. [:50] ..for now just to dev
        #print(f"{i}:{i+BATCH_SIZE}")
        batch_X = train_X[i:i+BATCH_SIZE].view(-1, 1, 50, 50)
        batch_y = train_y[i:i+BATCH_SIZE]

        net.zero_grad()

        outputs = net(batch_X)
        loss = loss_function(outputs, batch_y)
        loss.backward()
        optimizer.step()    # Does the update

    print(f"Epoch: {epoch}. Loss: {loss}")

correct = 0
total = 0
with torch.no_grad():
    for i in tqdm(range(len(test_X))):
        real_class = torch.argmax(test_y[i])
        net_out = net(test_X[i].view(-1, 1, 50, 50))[0]  # returns a list,
        predicted_class = torch.argmax(net_out)

        if predicted_class == real_class:
            correct += 1
        total += 1
print("Accuracy: ", round(correct/total, 3))


title: RuntimeError: Assertion `THIndexTensor_(size)(target, 0) == batch_size' failed. at ...ClassNLLCriterion.c:79

body:
I am working on a machine learning project. My task is to classify satellite images. I have 5 classes as my labels. The whole data is in Matlab format.
Posting the whole code would be too much, so I have decided to post the important part of it.

Here is the code-snippet:

...

from torch.autograd import Variable
import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 5)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 5 * 5)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

net = Net()

import torch.optim as optim

criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)

for epoch in range(2):  # loop over the dataset multiple times

    running_loss = 0.0
    for i in range(1):
        print(i)
        # get the inputs
        #inputs, labels = data
        inputs=in_data
        labels=in_labels
        print("A:",labels.shape)
        print("B:",inputs.shape)

        # wrap them in Variable
        inputs, labels = Variable(inputs), Variable(labels)
        print("A_2:",inputs.size())
        print("A_3:",labels.size())

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = net(inputs)
        labels = labels.squeeze_()
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

...

As you can easily see, this a CNN (Convolutional Neural Network). If I run my code, following error(s) will be shown:

0
A: torch.Size([100])
B: torch.Size([100, 3, 64, 64])
A_2: torch.Size([100, 3, 64, 64])
A_3: torch.Size([100])
Traceback (most recent call last):
  File "datasetAnalyser.py", line 283, in <module>
    main(args)
  File "datasetAnalyser.py", line 139, in main
    run(torchImage,torchLabel)
  File "datasetAnalyser.py", line 114, in run
    loss = criterion(outputs, labels)
  File "/home/qais/anaconda2/envs/Python3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 325, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/qais/anaconda2/envs/Python3/lib/python3.6/site-packages/torch/nn/modules/loss.py", line 601, in forward
    self.ignore_index, self.reduce)
  File "/home/qais/anaconda2/envs/Python3/lib/python3.6/site-packages/torch/nn/functional.py", line 1140, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, size_average, ignore_index, reduce)
  File "/home/qais/anaconda2/envs/Python3/lib/python3.6/site-packages/torch/nn/functional.py", line 1049, in nll_loss
    return torch._C._nn.nll_loss(input, target, weight, size_average, ignore_index, reduce)
RuntimeError: Assertion `THIndexTensor_(size)(target, 0) == batch_size' failed.  at /opt/conda/conda-bld/pytorch_1513368888240/work/torch/lib/THNN/generic/ClassNLLCriterion.c:79

Well,I have googled and found many threads about this problem. Somehow, none of them solved my issue. That's why I ask you again.

What should I do? I guess I have to change the integer values in the

class Net(nn.Module)

but I don't exactly know which values should be changed.

Thanks for your help!


title: AlexNet paper does not match dimensions they reported?

body:
I am following the paper where AlexNet was introduced, and the dimensions they report just don't match with the figure they attached.

The output of the first conv layer (which is 96 11x11x3 convolution) is 55x55x96 (for simple 1GPU case). Now the paper states that the second conv layer is applied to the output of a maxpooling layer. Assuming the MaxPool is a kernel of 3x3 with stride of 2 (as they report with s and z), this means that the input to the second convolution layer should be (55 - 3)/2 + 1 = 27 but in the picture provided for the AlexNet they write there is a Max Pooling operation there but they don't perform the dimension reduction of the pooling!

So the second conv layer should have been applied to a volumen with widht and height = 27 and not 55 right?

Furthermore, I took a look at how PyTorch was implementing it to see if I was missing something and they just changed the configuration, starting with 64 kernels... :

AlexNet(
  (features): Sequential(
    (0): Conv2d(3, 64, kernel_size=(11, 11), stride=(4, 4), padding=(2, 2))
    (1): ReLU(inplace)
    (2): MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=False)
    (3): Conv2d(64, 192, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2))
    (4): ReLU(inplace)
    (5): MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=False)
    (6): Conv2d(192, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (7): ReLU(inplace)
    (8): Conv2d(384, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (9): ReLU(inplace)
    (10): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (11): ReLU(inplace)
    (12): MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=False)
  )
  (classifier): Sequential(
    (0): Dropout(p=0.5)
    (1): Linear(in_features=9216, out_features=4096, bias=True)
    (2): ReLU(inplace)
    (3): Dropout(p=0.5)
    (4): Linear(in_features=4096, out_features=4096, bias=True)
    (5): ReLU(inplace)
    (6): Linear(in_features=4096, out_features=1000, bias=True)
  )
)


title: How to calculate gradient of output of a neural network w.r.t an input?

body:
I would like to calculate the partial derivative of (num1 * output1, num1 * output2) w.r.t. num1. The input of the neural network is (num1, 2, 3, 4). The expected dim of the output is 2. Is there a way to do that?

number_of_inputs = 4
hidden_layer = 64
number_of_outputs = 2

class NeuralNetwork(nn.Module):

    def __init__(self):
        super(NeuralNetwork, self).__init__()
        self.linear1 = nn.Linear(number_of_inputs,hidden_layer)
        self.linear2 = nn.Linear(hidden_layer,number_of_outputs)

        self.activation = nn.Tanh()

    def forward(self, x):
        output = self.linear1(x)
        output = self.activation(output)
        output = self.linear2(output)

        return output

a = NeuralNetwork()

# a(input1, input2, input3, input4) -> (output1, output2)

num1 = torch.tensor(1, dtype=torch.float, requires_grad=True)
b = a(torch.tensor([num1, 2, 3, 4], dtype=torch.float, requires_grad=True))


title: How to fix 'Expected object of backend CUDA' when trying to apply_tfms in custom LearnerCallback

body:
Very new to machine learning, fastai, pytorch, and python, and I was trying to adapt a LearnerCallback to do transformations after manually modifying the images.  When I start my learn.fit_one_cycle, it's immediately interrupted as shown below:

I've tried sticking .to(torch.device('cuda')) everywhere I could think of

    #...
    def on_batch_begin(self, last_input, last_target, train, **kwargs):
        if not train: return

        #Get new input
        new_input = last_input.clone()
        new_target = last_target.clone()
        tfms = get_transforms(max_zoom=1.5)

        # modify the images here in some other way

        # apply_tfms
        for i in range(len(new_input)):
            new_input[i] = Image(new_input[i]).apply_tfms(tfms[0]).data
            new_target[i] = Image(new_target[i]).apply_tfms(tfms[0], do_resolve=False).data
    #...

The 'apply_tfms' in the second to last line is the culprit in the traceback ending with:

    553     m[1,0] *= w/h
    554     c.flow = c.flow.view(-1,2)
--> 555     c.flow = torch.addmm(m[:2,2], c.flow,  m[:2,:2].t()).view(size)
    556     return c
    557

RuntimeError: Expected object of backend CUDA but got backend CPU for argument #4 'mat1'

Is there a way I can apply the transforms within a LearnerCallback without getting that error, or an alternate method where I can get my LearnerCallback added with learn.callback_fns.append to run before apply_tfms runs the same transforms on both the modified input and target images?  I need pixel information from the target image to modify the input image.  I also need this process applied during training and validation.

If it makes any difference, I get the same error even if I don't modify the cloned images before apply_tfms.


title: numpy argmax in array with multiple brackets

body:
I have an issue in apply argmax to an array which has multiple brackets.
In real life I am getting this as a result of a pytorch tensor.
Here I can put an example:

a = np.array([[1.0, 1.1],[2.1,2.0]])
np.argmax(a,axis=1)

array([1, 0])

It is correct. But:

a = np.array([[[1.0, 1.1]],[[2.1,2.0]]])
np.argmax(a,axis=1)

array([[0, 0],
       [0, 0]])

It does not give me what I expect.
Consider that in reality I have this level of inner brackets:

a = np.array([[[[1.0, 1.1]]],[[[2.1,2.0]]]])


title: PyTorch loss function referencing model parameters

body:
For an assignment, I need to create a movie recommender system that uses a supplied loss function:

sum(i=1,M) sum(j=1,M) indicator[i̸=j](viT vj − Xi,j )**2

What this means is, the dot product between two movie embeddings, Vi and Vj should be pretty darn close to Xi,j. Where Xi,j is the sum of users who liked both movie i and movie j. Entries where i == j are omitted by the indicator function, (set to 0.)

The assignment deliverable is a weight matrix from the hidden layer. Its dimensions should be 9724x300, where there are 9724 unique movie IDs, and 300 neurons. 300 is an arbitrary choice, influenced by use of 300 neurons in Google's word2vec.

What I have:

source_data: rows are users, columns are movies. A 1 in a given cell means that the corresponding user liked the corresponding movie (dislike = 0.)
preprocessed_data: matrix multiplication of source_data transposed on itself. (i.e. every cell is the sum of users who liked BOTH movie i and j. The diagonal entries are not of use to me, as they're just the sum of users who liked a single movie.)

Where I'm stuck:

Not sure how to define my own loss function in terms of i and j that can compare the sum of users who like movies i and j in preprocessed_data to rows, i and j, in hidden layer weight tensor.
Not sure how to organize my data into X and y tensors, such that the data is appropriate to the loss function.


title: How do you efficiently sum the occurences of a value in one array at positions in another array

body:
Im looking for an efficient 'for loop' avoiding solution that solves an array related problem I'm having. I want to use a huge 1Darray (A -> size = 250.000) of values between 0 and 40 for indexing in one dimension, and a array (B) with the same size with values between 0 and 9995 for indexing in a second dimension.

The result should be an array with size (41, 9996) with for each index the amount of times that any value from array 1 occurs at a value from array 2.

Example:

A = [0, 3, 2, 4, 3]
B = [1, 2, 2, 0, 2]
which should result in:
[[0, 1, 0,
 [0, 0, 0,
 [0, 0, 1,
 [0, 0, 2,
 [1, 0, 0]]

The dirty way is too slow as the amount of data is huge, what you would be able to do is:

out = np.zeros(41,9995)
for i in A:
  for j in B:
     out[i,j] += 1

which will take 238.000 * 238.000 loops...
I've tried this, which works partially:

out = np.zeros(41,9995)
out[A,B] += 1

Which generates a result with 1 everywhere, regardless of the amount of times the values occur.

Does anyone have a clue how to fix this? Thanks in advance!


title: pytorch question regarding backward argument used in blitz tutorial

body:
A pytorch question, regarding backward(). In the pytorch blitz tutorial copied and pasted below, they pass in a vector [0.1, 1.0, 0.0001] to backward() . I can intuitively guess why vector [0.1, 1.0, 0.0001] shape passed in is [3] , but I do not understand where the values 0.1, 1.0, 0.0001 come from. Another tutorial I looked at passes in one such that backwards on a vector is done like this : L.backward(torch.ones(L.shape))

# copied from blitz tutorial
Now in this case y is no longer a scalar. torch.autograd could not compute the full Jacobian directly, but if we just want the vector-Jacobian product, simply pass the vector to backward as argument:

v = torch.tensor([0.1, 1.0, 0.0001], dtype=torch.float)
y.backward(v)

print(x.grad)

If anyone can explain the reasoning for [0.1, 1.0, 0.0001], I would appreciate it.


title: Increasing batch_size of dataset for Pytorch neural network

body:
I currently have my neural network training with a batch_size =1 , To run it across multiple gpus i need to increase the batch size to be larger than the amount of gpus so i want batch_size=16, although the way i have my data set up i am not sure how to change that

The data is read from a csv file

raw_data = pd.read_csv("final.csv")
train_data = raw_data[:750]
test_data = raw_data[750:]

Then the data is normalized and turned to Tensors

# normalize features
scaler = MinMaxScaler(feature_range=(-1, 1))
scaled_train = scaler.fit_transform(train_data)
scaled_test = scaler.transform(test_data)
# Turn into Tensorflow Tensors
train_data_normalized = torch.FloatTensor(scaled_train).view(-1)
test_data_normalized = torch.FloatTensor(scaled_test).view(-1)

Then the data is turned into a Tensor Tuple of [input list, output] format
e.g (tensor([1,3,56,63,3]),tensor([34]))

# Convert to tensor tuples
def input_series_sequence(input_data, tw):
 inout_seq = []
 L = len(input_data)
 i = 0
 for index in range(L - tw):
    train_seq = input_data[i:i + tw]
    train_label = input_data[i + tw:i + tw + 1]
    inout_seq.append((train_seq, train_label))
    i = i + tw
 return inout_seq

train_inout_seq = input_series_sequence(train_data_normalized, train_window)
test_input_seq = input_series_sequence(test_data_normalized, train_window)

And then the model is trained like so

for i in range(epochs):

for seq, labels in train_inout_seq:
    optimizer.zero_grad()
    model.module.hidden_cell = model.module.init_hidden()
    seq = seq.to(device)
    labels = labels.to(device)
    y_pred = model(seq)

    single_loss = loss_function(y_pred, labels)
    single_loss.backward()
    optimizer.step()

So i want to know how exactly to change the batch_size from 1 -> 16 , Do i need to use Dataset and Dataloader? and if so how exactly would it fit in with my current code, thanks!

Edit: Model is defined like this, might have to change the forward function?

class LSTM(nn.Module):
def __init__(self, input_size=1, hidden_layer_size=100, output_size=1):
    super().__init__()
    self.hidden_layer_size = hidden_layer_size

    self.lstm = nn.LSTM(input_size, hidden_layer_size)

    self.linear = nn.Linear(hidden_layer_size, output_size)

    self.hidden_cell = (torch.zeros(1, 1, self.hidden_layer_size),
                        torch.zeros(1, 1, self.hidden_layer_size))

def init_hidden(self):
    return (torch.zeros(1, 1, self.hidden_layer_size),
            torch.zeros(1, 1, self.hidden_layer_size))

def forward(self, input_seq):
    lstm_out, self.hidden_cell = self.lstm(input_seq.view(len(input_seq), 1, -1), self.hidden_cell)
    predictions = self.linear(lstm_out.view(len(input_seq), -1))
    return predictions[-1]


title: When I converted a vgg model written by pytorch to caffe2, I encountered this problem

body:
model = VGGish()  # vgg model class
model.train(False)
x = torch.randn(10, 1, 64, 96, requires_grad=True)
# Export the model
torch_out = torch.onnx._export(model,             # model being run`enter code here`
                               x,                       # model input (or a tuple for multiple inputs)
                               "super_resolution.onnx", # where to save the model (can be a file or file-like object)
                               export_params=True)      # store the trained parameter weights inside the model file

model = onnx.load("modelsVGGish_conv.onnx")
onnx.checker.check_model(model)
onnx.helper.printable_graph(model.graph)
rep = backend.prepare(model, device="CPU") # or "CPU"

When the program runs to this code

rep = backend.prepare(model, device="CPU") # or "CPU"

the erro occurred:

Traceback (most recent call last):
  File "C:/Users/xxx/Desktop/python/新建文件夹/onnx_to_caffe2.py", line 159, in <module>
    rep = backend.prepare(model, device="CPU") # or "CPU"
  File "D:\environment\Anaconda\lib\site-packages\caffe2\python\onnx\backend.py", line 713, in prepare
    init_net, predict_net = cls._onnx_model_to_caffe2_net(model, device, opset_version, False)
  File "D:\environment\Anaconda\lib\site-packages\caffe2\python\onnx\backend.py", line 876, in _onnx_model_to_caffe2_net
    onnx_model = onnx.utils.polish_model(onnx_model)
  File "D:\environment\Anaconda\lib\site-packages\onnx\utils.py", line 21, in polish_model
    model = onnx.optimizer.optimize(model)
  File "D:\environment\Anaconda\lib\site-packages\onnx\optimizer.py", line 55, in optimize
    optimized_model_str = C.optimize(model_str, passes)
IndexError: invalid unordered_map<K, T> key

it just make me crazy！

Anyone can help me?


title: Building a deep reinforcement learning with a cnn q - approximation

body: I'm new in DRL. Starting from this code https://github.com/jaromiru/cwcf, I would like to substitute the MLP used for the q function approximation with a CNN, but I don't know how to do. Can anybody help me? Thanks


title: What is the best way of deploying a neural network trained in tensorflow to torch C++?

body: I'm working on a project where I need to forward pass the network in C++ with PyTorch. The weights of the network need to be imported from a model trained in Tensorflow. I will write the C++ version of the network using Torch but I can not train the network in Torch because of some issues. Which methods can I use to do this?


title: pytorch object too deep for array when saving image

body:
I am trying to run the code from the following github rep:

https://github.com/iamkrut/image_inpainting_resnet_unet

I havent changed anything in the code and it is causing a ValueError, that the object is too deep, when the code tries to save the image. The error seems to come from these two lines.

images = img_tensor.cpu().detach().permute(0,2,3,1)
plt.imsave(join(data_dir, 'samples', image), images[index,:,:,:3])

Here is the error statement

  File "train.py", line 205, in <module>
    data_dir=args.data_dir)
  File "train.py", line 94, in train_net
    plt.imsave(join(data_dir, 'samples', image), images[index,:,:,:]);
  File "C:\ProgramData\Anaconda3\envs\torch2\lib\site-packages\matplotlib\pyplot.py", line 2140, in imsave
    return matplotlib.image.imsave(fname, arr, **kwargs)
  File "C:\ProgramData\Anaconda3\envs\torch2\lib\site-packages\matplotlib\image.py", line 1498, in imsave
    _png.write_png(rgba, fname, dpi=dpi)
ValueError: object too deep for desired array

Anyone know what could be causing this or how to fix it?
Thank you


title: Leveraging Pretrained model in GAN learning

body:
I was recently studying GAN. However, I was curious about the reason why GAN didn't make the most of the power of transfer learning which is a common practice in object detection, semantic segmentation etc.,

As far as I know, at least in the image-translation case, the input of a generator is usually a random noise, which is entirely different from the pretrained classification CNN.

However, I found that classifier was used in ACGAN which applied multitasking in the discriminator. i.e., the discriminator is expected to perform the prediction of real/fake and the class of the given image. ACGAN is therefore proven effective to boost the quality of the generated images although the classification part didn't use a pretrained CNN because it is a multitasking branch inside the discriminator.

Therefore, I was thinking to use a pretrained CNN like Alexnet to replace the classification branch in the discriminator of ACGAN. However, another issue is the range of the input.

For example, in PyTorch, the input of FCN8s which uses pretrained VGG16 is between 0 & 1 and each channel is subtracted by the corresponding mean value ([103.939/255., 116.779/255., 123.68/255.] in BGR order).
However, the input of the generator is usually normalized to [-1,1] by subtracting the intensity of each pixel 0.5 and divided by 0.5. The last layer of the generator is usually "tanh" which is also [-1,1].
How can we deal with the output of the generator to later feed it into the classifier?
We could of course re-scale it to [0,1] and perform mean subtraction for it but I was wondering if gradient flow would be interfered.

Is there anyone to share the advantage of using pretrained CNN in GAN learning?
Thanks!

Hower, since ACGAN tried to
I guess somebody must have tried it before and didn't receive good results.


title: Multiple-Input Linear Regression works for one dataset but not another

body:
I am trying to run a logistic regression algorithm using Pytorch (and employing a neural network with one hidden layer), and I stumbled upon a problem. I am running the same algorithm for two different input data. My inputs are two-dimensional. The first set of data I created myself, while the second set of data come from real-world data which I got from a csv file, converted to lists and then to pytorch tensors.
For the first input data, the tensor I insert to the logistic regression code is:
 First tensor has rank
 torch.Size([1000, 2])

and it's given by:
T1= tensor([[ 0.6258,  0.9683],
        [-0.0833,  0.5691],
        [-0.4657, -0.8722],
        ...,
        [ 0.5868, -1.0565],
        [ 0.1611, -0.1716],
        [-0.1515, -0.8408]])

While the tensor for the second set of data is:
Second tensor has rank
 torch.Size([1064, 2])

and it's given by:
T2= tensor([[918.0600,  74.8220],
        [917.3477,  71.4038],
        [923.0400,  60.6380],
        ...,
        [916.6000,  71.0960],
        [912.6000,  58.4060],
        [921.5300,  77.7020]])

Now, for the first set of data, I get the following result:

So as you can see, the algorithm does a fairly good job with the reb/blue decision region, as most of the red points end up in the red region (and the same with the blue ones).
Now, for the second set of data, I get the following:

As you can see, it paints the whole region in red. I tried playing around with the number of neurons in my hidden layer and the learning rate, number of epochs and some other things, but nothing seems to work. I then thought that it might have to do with x-axis data having much larger values than the y-axis ones so I normalized them by dividing each with their mean, but this did not solve the problem.
The algorithm is the same, but it just doesn't work for this set of data. I was wondering if somebody who's more of an expert than me could have a hunch as to what might be going wrong here.


title: PyTorch weights not updating after softmax

body:
I am using PyTorch to perform an optimisation problem, which is finding a set of weights w such that the weighted average of x (sum(w * x) / sum(w)) can be used to estimate some variables say y.
Below is my pytorch's 'model',
dtype = torch.float
device = torch.device('cpu')

class WAvg(nn.Module):
    def __init__(self, p):
        super(WAvg, self).__init__()
        self.p = p
        self.q = nn.Parameter(torch.randn(self.p, 1, device=device, dtype=dtype))
        self.w = nn.functional.softmax(self.q, dim=0)
    def forward(self, x):
        w_avg = nn.functional.linear(x, self.w.T)
        return w_avg

Code for training,
x_tr = np.array([
    [1, 1, 1],
    [1, 4, 1],
    [2, 4, 6],
    [1, 2, 3],
    [4, 2, -3],
    [2, 2, 2]
])
y_tr = np.array([1, 2.1, 3.9, 2, 1.2, 1.8])

x_tr = torch.from_numpy(x_tr).float()
y_tr = torch.from_numpy(y_tr).float()

wa = WAvg(3)

criterion = nn.MSELoss()
optimizer = optim.Adam(wa.parameters(), lr=0.01)

for epoch in range(10):
    # Set running loss
    running_loss_tr = 0.0
    # zero the parameter gradients
    optimizer.zero_grad()
    # forward + backward + optimize
    y_pred_tr = wa(x_tr)
    loss_tr = criterion(y_pred_tr, y_tr)
    loss_tr.backward()
    optimizer.step()
    # print statistics
    print(epoch, loss_tr.item())

This will get an error

RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.

The argument retain_graph=True was added in loss_tr.backward() (as suggested in this post), but it seems that the parameters q or w are not updated. I think the problem should be caused by the softmax that makes a constraints on the weights so that their sum is one, is there any clue for fixing?
Output:
0 1.305460810661316
1 1.305460810661316
2 1.305460810661316
3 1.305460810661316
4 1.305460810661316
5 1.305460810661316
6 1.305460810661316
7 1.305460810661316
8 1.305460810661316
9 1.305460810661316


title: What version of CUDA do I require on a Nvidia GeForce GT 120 GPU under Mac OS El Capitan to run PyTorch

body: I have a Mac Pro 2009 4,1 with twin NVIDIA GeForce GT 120 512 MB GPUs. I'm looking to install PyTorch on to the box using CUDA. My question is what software do I need to load to drive the GPUs? The PyTorch installation info says Yosemite or higher is required, so El Capitan should suffice, but I will need to build from source for CUDA support.


title: Initializing conv2d Layer in Pytorch

body:
I have a following neural network architecture:
A(1x16x5x5)---\
                 -> concat(A,B) (1x32x5x5) --> Conv2D -> classification
B(1x16x5x5)---/

Conv2D=nn.Conv2d(32,16, kernel_size = 1, stride = 1 , padding = 0)

If I want to choose branch A only for testing, then I initialized Conv2d layer like this:
 convWeights = np.ones((16,32,1,1))
 convWeights[:,16:,:,:] = 0
 myNetwork.Conv2.weight = nn.Parameter(torch.from_numpy(convWeights).float().cuda())
 myNetwork.Conv2.bias.data.fill_(0)

However, it didn't give the expected classification results. Therefore, I think, I have a problem with the above codes and probably I'm missing something. Shortly, how do I initialize conv2d layer so that it chooses only branch A?


title: How to write a contextmanager to throw and catch errors

body:
I want to catch the runtime error CUDA out of memory on multiple occasions in my code. I do this to then rerun the whole training workflow with lower batch size. What is the best way to do that?
I am currently doing this:
try:
    result = model(input)
# if the GPU runs out of memory, start the experiment again with a smaller batch size
except RuntimeError as e:
    if str(e).startswith('CUDA out of memory.') and batch_size > 10:
        raise CudaOutOfMemory(e)
    else:
        raise e

I then catch the error CudaOutOfMemory outside my main function.
However, this is a pretty long piece of code that I need to repeat many times. Is there any way to make a context manager for this?
such that instead I can run:
with catch_cuda_out_of_mem_error:
  result = model(input)

Edit:
I want to create a context manager instead of a function because the functions I want to wrap the "try, except" around are not always the same. In my workflow, I have many functions that use a lot of GPU memory and I would like to catch this error in any of them.


title: Confused with the use of super() in Pytorch for neural net construction

body:
I understand the use of super() to inherit attributes and methods from a class when initializing your class. However, when using this in the context of neural networks such as:
class Net(nn.Module):
      def __init__(self, num_inputs, num_outputs, num_hiddens1, num_hiddens2,
                 is_training = True):
          super(Net, self).__init__()

I am confused on why super() is calling the same class. What does this mean and why is this done?
Thanks very much!


title: Amazon Sagemaker ModelError when serving model

body:
I have uploaded a transformer roberta model in S3 bucket. Am now trying to run inference against the model using Pytorch with SageMaker Python SDK. I specified the model directory s3://snet101/sent.tar.gz which is a compressed file of the model (pytorch_model.bin) and all its dependencies.  Here is the code
model = PyTorchModel(model_data=model_artifact,
                   name=name_from_base('roberta-model'),
                   role=role,
                   entry_point='torchserve-predictor2.py',
                   source_dir='source_dir',
                   framework_version='1.4.0',
                   py_version = 'py3',
                   predictor_cls=SentimentAnalysis)
predictor = model.deploy(initial_instance_count = 1, instance_type = 'ml.m4.xlarge')
test_data = {"text": "How many cows are in the farm ?"}
prediction = predictor.predict(test_data)

I get the following error on the predict method of the predictor object:
ModelError                                Traceback (most recent call last)
<ipython-input-6-bc621eb2e056> in <module>
----> 1 prediction = predictor.predict(test_data)

~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/predictor.py in predict(self, data, initial_args, target_model, target_variant)
    123
    124         request_args = self._create_request_args(data, initial_args, target_model, target_variant)
--> 125         response = self.sagemaker_session.sagemaker_runtime_client.invoke_endpoint(**request_args)
    126         return self._handle_response(response)
    127

~/anaconda3/envs/python3/lib/python3.6/site-packages/botocore/client.py in _api_call(self, *args, **kwargs)
    355                     "%s() only accepts keyword arguments." % py_operation_name)
    356             # The "self" in this scope is referring to the BaseClient.
--> 357             return self._make_api_call(operation_name, kwargs)
    358
    359         _api_call.__name__ = str(py_operation_name)

~/anaconda3/envs/python3/lib/python3.6/site-packages/botocore/client.py in _make_api_call(self, operation_name, api_params)
    674             error_code = parsed_response.get("Error", {}).get("Code")
    675             error_class = self.exceptions.from_code(error_code)
--> 676             raise error_class(parsed_response, operation_name)
    677         else:
    678             return parsed_response

ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received server error (0) from model with message "Your invocation timed out while waiting for a response from container model. Review the latency metrics for each container in Amazon CloudWatch, resolve the issue, and try again.". See https://us-east-1.console.aws.amazon.com/cloudwatch/home?region=us-east-1#logEventViewer:group=/aws/sagemaker/Endpoints/roberta-model-2020-12-16-09-42-37-479 in account 165258297056 for more information.

I checked the server log error
java.lang.IllegalArgumentException: reasonPhrase contains one of the following prohibited characters: \r\n: Can't load config for '/.sagemaker/mms/models/model'. Make sure that:
'/.sagemaker/mms/models/model' is a correct model identifier listed on 'https://huggingface.co/models'
or '/.sagemaker/mms/models/model' is the correct path to a directory containing a config.json file

How can I fix this?


title: WGAN-GP Very Negative Critic Loss

body:
I am training a WGAN-GP on a 123 x 123 B&W Image. My model is training but I get very negative crit_loss values as far as Gen_loss = 6, and Crit_Loss = -106 at 400 epochs Table: Generator and Critic Loss Values.
So far, I have taken the following suggestions into consideration: made my critic symmetric to my generator, drawn from a larger z-dim, and added more layers to the generator. I would appreciate any other recommendations or adjustments that may alleviate this issue.
Generator Model:
class Generator(nn.Module):
    def __init__(self, z_dim=10, im_chan=1, hidden_dim=64):
        super(Generator, self).__init__()
        self.z_dim = z_dim
        # Build the neural network
        self.gen = nn.Sequential(
            self.make_gen_block(z_dim, hidden_dim * 8),
            self.make_gen_block(hidden_dim * 8, hidden_dim * 4, kernel_size=5, stride=2, padding = 1),
            self.make_gen_block(hidden_dim * 4, hidden_dim * 2, kernel_size=5, stride=2, padding = 1),
            self.make_gen_block(hidden_dim * 2, hidden_dim),
            self.make_gen_block(hidden_dim, im_chan, kernel_size=3, final_layer=True),
        )

    def make_gen_block(self, input_channels, output_channels, kernel_size=5, stride=2, padding = 1, final_layer=False):
        if not final_layer:
            return nn.Sequential(
                nn.ConvTranspose2d(input_channels, output_channels, kernel_size, stride),
                nn.BatchNorm2d(output_channels),
                nn.ReLU(inplace=True),
            )
        else:
            return nn.Sequential(
                nn.ConvTranspose2d(input_channels, output_channels, kernel_size, stride),
                nn.Tanh(),
            )

    def forward(self, noise):
        x = noise.view(len(noise), self.z_dim, 1, 1)
        return self.gen(x)

Critic Model:
class Critic(nn.Module):
    def __init__(self, im_chan=1, hidden_dim=64):
        super(Critic, self).__init__()
        self.crit = nn.Sequential(
            self.make_crit_block(im_chan, hidden_dim),
            # self.make_crit_block(hidden_dim, hidden_dim * 8, kernel_size=5, stride=2, padding = 1),
            self.make_crit_block(hidden_dim, hidden_dim * 4, kernel_size=5, stride=2, padding = 1),
            self.make_crit_block(hidden_dim * 4, hidden_dim * 2, kernel_size=5, stride=2, padding = 1),
            self.make_crit_block(hidden_dim * 2, 1, kernel_size=3, final_layer=True),
        )
    def make_crit_block(self, input_channels, output_channels, kernel_size=4, stride=2, padding = 1, final_layer=False):
        if not final_layer:
            return nn.Sequential(
                nn.Conv2d(input_channels, output_channels, kernel_size, stride),
                nn.BatchNorm2d(output_channels),
                nn.LeakyReLU(0.2, inplace=True),
            )
        else:
            return nn.Sequential(
                nn.Conv2d(input_channels, output_channels, kernel_size, stride),
            )

    def forward(self, image):
        crit_pred = self.crit(image)
        return crit_pred.view(len(crit_pred), -1)


title: Input batch size doesn't match target batch size in CrossEntropyLoss function

body:
I've been trying to build a model from scratch to recognize handwritten digits from the MNIST dataset with the help of PyTorch and the DataLoader class from FastAI. So far, I've been using a linear model that has 784 inputs (a flattened grayscale 28 by 28 handwritten digit image tensor) and 10 outputs.
simple_linear = torch.nn.Linear(784, 10)

My training data is organized as such:
train_x = torch.cat([stacked_zeros, stacked_ones, stacked_twos, stacked_threes,
                     stacked_fours, stacked_fives, stacked_sixes, stacked_sevens,
                     stacked_eights, stacked_nines]).view(-1, 28*28)

train_y = torch.nn.functional.one_hot(tensor([0] * len(zeros) + [1] * len(ones) + [2] * len(twos) +
                 [3] * len(threes) + [4] * len(fours) + [5] * len(fives) +
                 [6] * len(sixes) + [7] * len(sevens) + [8] * len(eights) +
                 [9] * len(nines)).unsqueeze(1))

My x variables have shape [784] while y variables are labeled using one-hot encoded vectors with [1, 10] shape.
The loss function I chose based on research is torch.nn.CrossEntropyLoss and the following code gives me an error:
mnist_loss = torch.nn.CrossEntropyLoss()
mnist_loss(simple_linear(train_x[0]), train_y[0])

ValueError                                Traceback (most recent call last)
<ipython-input-245-03f54a6a43fb> in <module>()
----> 1 tst(simple_linear(x), y)

8 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)
   2260     if input.size(0) != target.size(0):
   2261         raise ValueError('Expected input batch_size ({}) to match target batch_size ({}).'
-> 2262                          .format(input.size(0), target.size(0)))
   2263     if dim == 2:
   2264         ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)

ValueError: Expected input batch_size (1) to match target batch_size (10).

I've tried reshaping my x variables and y variables but I always get a similar error. How must my data be structured in order for the loss function to work?


title: How to save your trained model's weight on pytorch?

body:
I wrote a code for yolo implementation with PyTorch in Colab. I can do the training, but I cannot save the model's weights. I keep getting a 'ran out of input' error. Is there any solution?
Code:
"""
Main file for training Yolo model on Pascal VOC dataset
"""

import torch
import torchvision.transforms as transforms
import torch.optim as optim
import torchvision.transforms.functional as FT #resim transformları için
from tqdm import tqdm  #progressbar için

from torch.utils.data import DataLoader
from model import Yolov1
from dataset import VOCDataset
from utils import (
    non_max_suppression,
    mean_average_precision,
    intersection_over_union,
    cellboxes_to_boxes,
    get_bboxes,
    plot_image,
    save_checkpoint,
    load_checkpoint,
)
from loss import YoloLoss

seed = 123
torch.manual_seed(seed) #same datasets loading

# Hyperparameters etc.
LEARNING_RATE = 2e-5
DEVICE = "cuda" if torch.cuda.is_available else "cpu"
BATCH_SIZE = 16
WEIGHT_DECAY = 0 #sadece bir batchte train yapmak için 0 dedik yani regularizasyon yapmıyoruz
EPOCHS = 1000
NUM_WORKERS = 2
PIN_MEMORY = True
LOAD_MODEL = True
LOAD_MODEL_FILE = "/content/drive/MyDrive/YOLO 5 adım/archive/best.pt" #over fit olan modeli aldık
IMG_DIR = "/content/drive/MyDrive/YOLO 5 adım/archive/images"
LABEL_DIR = "/content/drive/MyDrive/YOLO 5 adım/archive/labels"

class Compose(object): #buradaki neden, göndereceğimiz dönüşümümüzün yalnızca resim üzerinde işlemesi çünkü sadece resize yapıyoruz
    def __init__(self, transforms):
        self.transforms = transforms

    def __call__(self, img, bboxes):
        for t in self.transforms:
            img, bboxes = t(img), bboxes

        return img, bboxes

transform = Compose([transforms.Resize((448, 448)), transforms.ToTensor(),]) #resize yapıp tensöre çevirdik

def train_fn(train_loader, model, optimizer, loss_fn): #tüm datasette bir training döngüsü olacak
    loop = tqdm(train_loader, leave=True) #progressbar videosunda varmış
    mean_loss = []

    for batch_idx, (x, y) in enumerate(loop):
        x, y = x.to(DEVICE), y.to(DEVICE)
        out = model(x)
        loss = loss_fn(out, y)
        mean_loss.append(loss.item())
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        # update progress bar progress barda her batchteki lossu göreceğiz
        loop.set_postfix(loss=loss.item())

    print(f"Mean loss was {sum(mean_loss)/len(mean_loss)}")

def main():
    model = Yolov1(split_size=7, num_boxes=2, num_classes=20).to(DEVICE)
    optimizer = optim.Adam(
        model.parameters(), lr=LEARNING_RATE, weight_decay=WEIGHT_DECAY
    )
    loss_fn = YoloLoss()

    **if LOAD_MODEL:
        load_checkpoint(torch.load(LOAD_MODEL_FILE), model, optimizer)  ** I got the error from this line

    train_dataset = VOCDataset(
        "/content/drive/MyDrive/YOLO 5 adım/archive/8examples.csv", #eğer tüm dataseti istiyorsak train.csv yaparız biz burada 100 örnek aldık sadece
        transform=transform,
        img_dir=IMG_DIR,
        label_dir=LABEL_DIR,
    )

    test_dataset = VOCDataset( #test kısmı
        "/content/drive/MyDrive/YOLO 5 adım/archive/test.csv", transform=transform, img_dir=IMG_DIR, label_dir=LABEL_DIR,
    )

    train_loader = DataLoader(
        dataset=train_dataset,
        batch_size=BATCH_SIZE,
        num_workers=NUM_WORKERS,
        pin_memory=PIN_MEMORY,
        shuffle=True,
        drop_last=False,#8 tanede denerken burayı false yaptık eğer çok az örnekte deniyorsak false fazla örnekte deniyorsak true yapıyoruz. çok az örnek yani tek bir batchte gerçeklesen örnekse false
    )

    test_loader = DataLoader(
        dataset=test_dataset,
        batch_size=BATCH_SIZE,
        num_workers=NUM_WORKERS,
        pin_memory=PIN_MEMORY,
        shuffle=True,
        drop_last=True,
    )

    for epoch in range(EPOCHS):
        for x, y in train_loader:  #BU KISIM GÖRSELLEŞTİRME İÇİN SONUCU BİZE GÖSTERİYOR TEK TEK RESİMLERDE
            x = x.to(DEVICE)              #Bu kısmı uygularken yukarıda hiperparametreler kısmındaki LOAD_MODAL'ı True yapmamız gerekiyor.
            for idx in range(8):
                bboxes = cellboxes_to_boxes(model(x))
                bboxes = non_max_suppression(bboxes[idx], iou_threshold=0.5, threshold=0.4, box_format="midpoint")
                plot_image(x[idx].permute(1,2,0).to("cpu"), bboxes)

            import sys
            sys.exit()

        pred_boxes, target_boxes = get_bboxes( train_loader, model, iou_threshold=0.5, threshold=0.4 )

        mean_avg_prec = mean_average_precision(
            pred_boxes, target_boxes, iou_threshold=0.5, box_format="midpoint"
        )
        print(f"Train mAP: {mean_avg_prec}") #her epoch için yazdırdık

        if mean_avg_prec > 0.9:
            checkpoint = {
                "state_dict": model.state_dict(),
                "optimizer": optimizer.state_dict(),
            }
            save_checkpoint(checkpoint, filename=LOAD_MODEL_FILE)   #kaydetmek için buu
            import time
            time.sleep(10)

        train_fn(train_loader, model, optimizer, loss_fn)

if __name__ == "__main__":
    main()

https://colab.research.google.com/drive/1l6Z86Qk8qu7Oo7fV6YNaWmJRFB9NPjCv?usp=sharing

I got error from this line (in the picture)
#save_checkpoint function from utils
def save_checkpoint(state, filename="my_checkpoint.pth.tar"):
    print("=> Saving checkpoint")
    torch.save(state, filename)

def load_checkpoint(checkpoint, model, optimizer):
    print("=> Loading checkpoint")
    model.load_state_dict(checkpoint["state_dict"])
    optimizer.load_state_dict(checkpoint["optimizer"])


title: How to pickle with torch.load/save a custom object without a direct reference of it's class definition and without dill?

body:
I have a very simple scenario. I have 1 custom class (not a NN model) I am pickling with torch.save. When I try to load it it fails with error:
Traceback (most recent call last):
  File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydevd_bundle/pydevd_exec2.py", line 3, in Exec
    exec(exp, global_vars, local_vars)
  File "<input>", line 1, in <module>
  File "/Users/brando/anaconda3/envs/metalearning/lib/python3.8/site-packages/torch/serialization.py", line 594, in load
    return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
  File "/Users/brando/anaconda3/envs/metalearning/lib/python3.8/site-packages/torch/serialization.py", line 853, in _load
    result = unpickler.load()
AttributeError: Can't get attribute 'DagDataPreparation' on <module '__main__' from '/Users/brando/ML4Coq/ml4coq-proj/data_lib/dag/dag_dataloader.py'>

but the class is basically empty (though it's in a different file DagDataPreparation:
class DagDataPreparation:

    def __init__(self, root):
        print('1')

    def create_everything(self):
        return 1

if I use dill as an argument for torch.load and torch.save it works. It also works if I simply import the class at the top of the file
from data_lib.dag.dataset_preparation import DagDataPreparation

I understand why the error might be happening, it can't find the definition of the class...but this is really weird because I am 200% I've done this type of thing (saving/loading arbitrary classes with torch) and I've never had this issue before.
What might be the issue? Does everyone else do experience the same bug in my scenario if you have two files one that loads the pickle file and it throws that error at you?


title: Neural network learning to sum two numbers

body:
I am learning Pytorch, and I am trying to implement a really simple network which takes an input which is of length 2, i.e. a point in the plane, and aims to learn the sum of its components.
In principle the network should just learn a linear layer with weight matrix W = [1.,1.] and zero bias, so I expect to have very low training error. However, I don't see why I do not get this as expected.
The code I am writing is this:
import torch
from torch import nn, optim
import numpy as np

device = torch.device("cuda:0" if
torch.cuda.is_available() else "cpu")

N = 1000  # number of samples
D = 2  # input dimension
C = 1  # output dimension

def model(z):
  q = z[:,0]
  p = z[:,1]
  return q+p

X = torch.rand(N, D,requires_grad=True).to(device)
y = model(X)

lr = 1e-2 #Learning rate

Rete = nn.Sequential(nn.Linear(D, C))
Rete.to(device) #Convert to CUDA

criterion = torch.nn.MSELoss()
optimizer = torch.optim.Adam(Rete.parameters(), lr=lr)

for t in range(5000):

    y_pred = Rete(X)
    loss = criterion(y_pred, y)
    print("[EPOCH]: %i, [LOSS]: %.6f" % (t, loss.item()))
    optimizer.zero_grad()
    optimizer.step()


title: Iterative targeted-FGSM not working for minibatches

body:
I am trying to perform iterative fgsm attack to my neural network. I am refering to this blog https://savan77.github.io/blog/imagenet_adv_examples.html. However, the attack can only affect the prediction of the first data in a minibatch. And the resting is not affected.

I am using MNIST dataset and a very small neural network, which I have pretrained by myself.

def fgsm_attack(inputs, model, epsilon, attacked_target):
    criterion = nn.CrossEntropyLoss()
    inputs_tensor = inputs.data
    alpha = 0.025

    # Add iterations to make attack stronger
    for i in range(5):
        zero_gradients(inputs)
        outputs = model(inputs)
        loss_gen = criterion(outputs, attacked_targets)
        loss_gen.backward(retain_graph=True)

        # Collect the element-wise sign of the data gradient
        sign_inputs_grad = alpha * torch.sign(inputs.grad.data)

        # I am performing targeted attack, so this is minus
        adv_temp = inputs.data - sign_inputs_grad

        total_grad = adv_temp - inputs_tensor

        # Just to ensure that the amplitude will not exceed the threshold
        total_grad = torch.clamp(total_grad, -epsilon, epsilon)

        x_adv = inputs_tensor + total_grad
        inputs.data = x_adv

    return inputs, total_grad

If the batch size is 5:

If there is no attack, the predictions should be [7, 4, 6, 5, 0]

If we set the attacked target to be 1, the expected results should be [1, 1, 1, 1, 1]

However, I got [1, 4, 6, 5, 0].

This is how I generate the adversaries:

for i, (inputs, targets) in enumerate(val_loader):
    inputs.requires_grad = True
    perturbed_inputs, sign_inputs_grad = fgsm_attack(inputs, model, epsilon, attacked_target)
    outputs = model(perturbed_inputs)
    ...

I just call the function to generate adversaries.


title: Interpolate one range of numbers to another in a tensor in PyTorch

body:
I am trying to transform a set of numbers say {0, 4, 6} to a consecutive one {0, 1, 2}.
Currently I am using Scipy's interp1d function. My problem is that I want to change each element of a tensor. Is there a better way than broadcasting the interp1d function to each element in a tensor? If not how can I achieve this broadcasting of a custom function

Currently I am doing the following:

>>> a = torch.tensor([[0, 6, 4],
                      [4, 0, 4]])
>>> unique_values = np.unique(a)
>>> interpolator = interp1d(unique_values, range(len(unique_values)))
>>> interpolator(a[0, 1])
>>> 2


title: Pairwise cross product by pytorch & its precision problem

body:
I'm a student who just started study pytorch.

I hope to generate pairwise cross product matrix.

Example is below.

x=torch.Tensor([[1,0,0],[0,1,0]]) # 2*3
y=torch.Tensor([[2,0,0],[0,2,0],[0,0,2]]) # 3*3

#what I want to get is below using torch.cross

z=torch.Tensor([[[0,0,0],[0,0,2],[0,-2,0]],\
                 [0,0,-2],[0,0,0],[2,0,0]]]) # 2*3*3
#So I write code

tmp_x=0*y+x.unsqueeze(1)# 2*3*3 , horrible coding.. please modify this.
tmp_y=y+0*x.unsqueeze(1)
z=torch.cross(tmp_x,tmp_y,dim=-1)

I think my code is not good. May I have advice for more good code?

And there is precision problem, when I use it for float type. Cross product between a vector and itself is not zero-vector.
 When I print each matrix in above code like below, in this case x=y,

print x
print y
print '====='
print tmp_x
print tmp_y
print '====='
print z

then I got result.

tensor([[ 0.2957,  1.6024, -0.1010],
        [-1.4755,  0.1937, -0.6091],
        [ 1.3540,  0.7203,  0.4037]])
tensor([[ 0.2957,  1.6024, -0.1010],
        [-1.4755,  0.1937, -0.6091],
        [ 1.3540,  0.7203,  0.4037]])
=======
tensor([[[ 0.2957,  1.6024, -0.1010],
         [ 0.2957,  1.6024, -0.1010],
         [ 0.2957,  1.6024, -0.1010]],

        [[-1.4755,  0.1937, -0.6091],
         [-1.4755,  0.1937, -0.6091],
         [-1.4755,  0.1937, -0.6091]],

        [[ 1.3540,  0.7203,  0.4037],
         [ 1.3540,  0.7203,  0.4037],
         [ 1.3540,  0.7203,  0.4037]]])
tensor([[[ 0.2957,  1.6024, -0.1010],
         [-1.4755,  0.1937, -0.6091],
         [ 1.3540,  0.7203,  0.4037]],

        [[ 0.2957,  1.6024, -0.1010],
         [-1.4755,  0.1937, -0.6091],
         [ 1.3540,  0.7203,  0.4037]],

        [[ 0.2957,  1.6024, -0.1010],
         [-1.4755,  0.1937, -0.6091],
         [ 1.3540,  0.7203,  0.4037]]])
=======
tensor([[[ 5.0209e-09, -2.8436e-10, -1.0191e-08],
         [-9.5639e-01,  3.2915e-01,  2.4217e+00],
         [ 7.1970e-01, -2.5616e-01, -1.9567e+00]],

        [[ 9.5639e-01, -3.2915e-01, -2.4217e+00],
         [ 6.0023e-10, -2.4257e-08,  1.3432e-08],
         [ 5.1692e-01, -2.2897e-01, -1.3251e+00]],

        [[-7.1970e-01,  2.5616e-01,  1.9567e+00],
         [-5.1692e-01,  2.2897e-01,  1.3251e+00],
         [ 9.2649e-09,  5.7705e-09, -4.0787e-09]]])

z[0,0,:] must be zero vector, but it is not. What did I wrong?


title: PyTorch: Simple feedforward neural network not running without retain_graph=True

body:
Below is my code for training a Feedforward neural network (FFNN).

The labels are numbers between 0 and 50. The FFNN comprises of a single hidden layer with 50 neurons and an output layer with 51 neurons. Furthermore, I have used negative log likelihood loss.

I am very new to PyTorch so I used a couple of websites for guidance. The strange thing is that none of them required retain_graph to be set to True (they dont pass any arguments when calling backward()). Furthermore, it runs very slowly and the accuracy seems to be fluctuating around a fixed value instead of reducing.

Assuming that the input's format is correct, can someone please explain to me why the network is performing so badly and why the network requires retain_graph to be set to True?

Thank you very much!

n_epochs = 2
batch_size = 100
for epoch in range(n_epochs):
    permutation = torch.randperm(training_set.size()[0])
    for i in range(0, training_set.size()[0], batch_size):
        opt.zero_grad()
        indices = permutation[i:i + batch_size]
        batch_features = training_set[indices]
        batch_labels = torch.LongTensor([label for label, sent in train[indices]])
        batch_outputs = model(batch_features)
        loss = loss_function(batch_outputs, batch_labels)
        loss.backward(retain_graph=True)
        opt.step()


title: PyTorch Dataset and Conv1d using a ton of memory

body:
I am trying to write a convolutional neural network in pytorch. I'm very new to machine learning and PyTorch, so I'm not very familiar with the package.

I have written a custom dataset and it has loaded my data from a csv file properly. However, when I load it into a data loader, my system monitor shows python using a huge amount of memory. I'm currently using only a fraction of my data set, and one instance uses about 5 gigs as a data loader.

My dataset is 1 dimensional tensors. Each one is very long - about 33 million values. I used
sys.getsizeof(train_set.sample_list[0][0].storage())
to check the size of the underlying tensor, and it was only 271 megabytes.

Additionally, if I continue on and create an instance of my CNN, the initializer eats up memory until my kernel crashes. The reason for this is unclear to me.

Here is the code for my custom Dataset:

    def __init__(self, csv_file, train):

        self.train = train
        self.df_tmp = pd.read_csv(csv_file, header=None, sep='\t')
        self.df_tmp.drop(self.df_tmp.shape[1]-1, axis=1, inplace=True)
        self.df = self.df_tmp.transpose()
        self.sample_list = []

        for i in range(self.df.shape[0]): #num rows,  33 million ish
            sample = torch.tensor([self.df.iloc[i][1:].values])
            label = torch.tensor(self.df.iloc[i][0])
            self.sample_list.append((sample, label))

    def __len__(self):
        return len(self.sample_list)

    def __getitem__(self, idx):
        return self.sample_list[idx]

And the code for the NN:

    #input batch shape is (9 x 33889258 x 1)
    def __init__(self):
        super(CNN, self).__init__()

        #input channels 1, output 3
        self.conv1 = torch.nn.Conv1d(1, out_channels=3, kernel_size=(100), stride=10, padding=1)

        #size in is 3,1,33889258
        self.pool = torch.nn.MaxPool1d(kernel_size=2, stride=2, padding=0)

        self.fc1 = torch.nn.Linear(45750366, 1000) #3 * 1 * 3388917

        self.fc2 = torch.nn.Linear(1000, 2)

    def forward(self, x):
        #size: (1x1x33889258) to (3x1x33889258)
        tmp = self.conv1(x.float())
        x = F.relu(tmp) #

        x = self.pool(x)
        #whatever shape comes out of here needs to go into x.view

        x = x.view(45750366) #-1, 1*1*3388927
        x = self.fc1(x)
        x = F.relu(x)

        x = self.fc2(x)
        return(x)

Some of my input sizes might be off, I'm still working that out but the memory issue is preventing me from making progress


title: Zombie process does not allow deallocating GPU memory

body:
I am loading NLP models in GPU to do inferencing.
But once the inference is over the GPU does not deallocate its memory:

But then the command ps -a | grep python gave me

How do I solve this issue?


title: How to put image uploaded in tkinter into a function?

body:
I am trying to create a Python tkinter application where the user can upload an image from file and the image is put through a image segmentation function which outputs an matplotlib plot.

I have the image segmentation function, it takes two parameters: neural network, image file pathway.

from torchvision import models
from PIL import Image
import matplotlib.pyplot as plt
import torch
import torchvision.transforms as T
import numpy as np

fcn = models.segmentation.fcn_resnet101(pretrained=True).eval()

# Define the helper function
def decode_segmap(image, nc=21):

  label_colors = np.array([(0, 0, 0),  # 0=background
               # 1=aeroplane, 2=bicycle, 3=bird, 4=boat, 5=bottle
               (128, 0, 0), (0, 128, 0), (128, 128, 0), (0, 0, 128), (128, 0, 128),
               # 6=bus, 7=car, 8=cat, 9=chair, 10=cow
               (0, 128, 128), (128, 128, 128), (64, 0, 0), (192, 0, 0), (64, 128, 0),
               # 11=dining table, 12=dog, 13=horse, 14=motorbike, 15=person
               (192, 128, 0), (64, 0, 128), (192, 0, 128), (64, 128, 128), (192, 128, 128),
               # 16=potted plant, 17=sheep, 18=sofa, 19=train, 20=tv/monitor
               (0, 64, 0), (128, 64, 0), (0, 192, 0), (128, 192, 0), (0, 64, 128)])

  r = np.zeros_like(image).astype(np.uint8)
  g = np.zeros_like(image).astype(np.uint8)
  b = np.zeros_like(image).astype(np.uint8)

  for l in range(0, nc):
    idx = image == l
    r[idx] = label_colors[l, 0]
    g[idx] = label_colors[l, 1]
    b[idx] = label_colors[l, 2]

  rgb = np.stack([r, g, b], axis=2)
  return rgb

def segment(net, path):
  img = Image.open(path)
  plt.imshow(img); plt.axis('off'); plt.show()
  # Comment the Resize and CenterCrop for better inference results
  trf = T.Compose([T.Resize(256),
                   T.CenterCrop(224),
                   T.ToTensor(),
                   T.Normalize(mean = [0.485, 0.456, 0.406],
                               std = [0.229, 0.224, 0.225])])
  inp = trf(img).unsqueeze(0)
  out = net(inp)['out']
  om = torch.argmax(out.squeeze(), dim=0).detach().cpu().numpy()
  rgb = decode_segmap(om)
  plt.imshow(rgb); plt.axis('off'); plt.show()

I have not made the Tkinter GUI as I am unsure how I can take the image uploaded from file and convert it into a file path (string) and put it through the function. There is only one neural network for now, being fcn.


title: (ε, δ)-differential privacy accuracy/privacy trade-off

body:
I'm using Renyi (ε, δ)-differential privacy as implemented in PyTorch DNNs. I do this to guarantee the privacy of the training set to the degree ε. In order to implement this, I use PyTorch-DP. As the network fits more and more on the trainingset and the accuracy goes up, the amount of privacy decreases. However, at some turning point, the privacy decreases and the accuracy decreases too; I assume that the accuracy eventually goes down because too much (privacy) noise is added to the mean of the clipped gradients over a batch of training data and/or because it overfits on the training set.

Now what I want to achieve is the following: I want to train a model to reach (ε, δ)-differential privacy with a differing value of ε (i.e. 0.5, 1, 1.5, 2). Mind that the bigger the ε-value, the less privacy the model guarantees. In an ideal situation, the accuracy would improve as ε increases. Yet this does not happen because after that turning point, the accuracy decreases. My question is, is there any way to guarantee/influence this so that the accuracy is still at a near-optimum at the point that the algorithm reaches the specified ε. Or is this not really as controllable as I'd like and is the best way to achieve this through empirically changing the learning rate, or changing the amount of additive clipping noise as well as the number of epochs so that more/less privacy is added each epoch but for a shorter/longer time.

I'm asking for ideas really.


title: multiclass-segemenratation using pytorch and unet

body: I am doing landuse classification with 4 classes and as an output from softmax function of unet model i gained output of [8,4,128,128] and my mask image is [8,1,128,128] so to calculate loss I used nn.cross entropy function should i have to make any modification for good results as I find the output of testing image was 4 channels images which are not like mask image.``


title: Advice to go from MATLAB .mat file with structures to a numpy array?

body:
I am trying to read in a large structure from a .mat file into a jupyter notebook. I am a little new to python so I'm not sure why my solution isn't working.
The structure from MATLAB(2020) I am reading in is structured like this
pose.frames.ind

Where there are 44 frames and in each frame 63 ind. I am reading it into a jupyter notebook like this with mat4py. I am trying to adapt someone else's code so after I read it in I need to convert it to a tensor so it can go into another function.
from mat4py import loadmat
val = loadmat('pose.mat')
pose_body = val['pose']['frames']['ind'][0]
pose_body = np.asarray(pose_body)
pose_body = torch.FloatTensor(pose_body).to(comp_device)

When I feed pose_body = np.zeros([1,63]) into the line that changes it to a torch tensor, the code works fine. However, when I try to feed it this array I imported something goes wrong and I get the error of
RuntimeError: invalid argument 0: Tensors must have same number of dimensions: got 2 and 1 at C:/w/1/s/tmp_conda_3.7_055306/conda/conda-bld/pytorch_1556690124416/work/aten/src\THC/generic/THCTensorMath.cu:62
Is there an easier/better way to convert the data from matlab in the format I need? I am a little unfamiliar with python, and it seems like they're the same type of array. From doing type(pose_body), and tf.size(pose_body) I know that both numpy arrays have a shape of 63, a dtype of int32, and are of class "numpy.ndarray".


title: Lstm with different input sizes for each batch (pyTorch)

body:
I am switching from TensorFlow to PyTorch and I having some troubles with my net.
I have made a Collator(for the DataLoader) that pads each tensor(originally sentence) in each batch into the Maslen of each batch.
so I have different input sizes per each batch.
my network consists of LSTM -> LSTM -> DENSE
my question is, how can I specify this variable input size to the LSTM?
I assume that in TensorFlow I would do Input((None,x)) bedsore the LSTM.
Thank you in advance


title: How to multiply two tensors' axes in PyTorch?

body:
I have two 3D tensors of shape:
a = torch.full([1495, 110247, 1], 0.5)
b = torch.full([1495, 110247, 2], 1)

I want to multiply them so that the first two dimensions remain the same and the third dimension is the scalar product of the third dimension of a and b.


title: What is the exact equivalent of PyTorch's dist.rsample() in Tensorflow to sample from a distribution using the reparameterization trick?

body:
I coded the Soft-Actor-Critic algorithm in pyTorch and it works very fine. Now I want to translate that code to Tensorflow 2 with eager execution.
How can I make a Tensorflow/ tensorflow_probability distribution to flow the gradients through with reparameterization?
have a good day all!


title: How to implement Softmax regression with pytorch?

body:
I am working on a uni assignment where I need to implement Softmax Regression with Pytorch. The assignment says:
Implement Softmax Regression as an nn.Module and pipe its output with its output with torch.nn.Softmax.

As I am new to pytorch, I am not sure how to do it exactly. So far I have tried:
class SoftmaxRegression(nn.Module):  # inheriting from nn.Module!
def __init__(self, num_labels, num_features):

    super(SoftmaxRegression, self).__init__()
    self.linear = torch.nn.Linear(num_labels, num_features)

def forward(self, x):
    # should return the probabilities for the classes, e.g.
    # tensor([[ 0.1757,  0.3948,  0.4295],
    #         [ 0.0777,  0.3502,  0.5721],
    #         ...

    # not sure what to do here

does anybody have any idea how I could go about it? I am not sure what should be written in the forward method. I appreciate any help !


title: Best trained model not correctly saving with pytorch

body:
i want to save the best model when training but it keeps saving the first trained model
#Train the model for 4 epochs
from collections import defaultdict
#Define history and best accuracy
history = defaultdict(list)
best_accuracy = 0

for epoch in range(EPOCHS):

    print(f'Epoch: {epoch + 1} / {EPOCHS}')
    print('-' * 15)

    #Call the train epoch method
    train_acc, train_loss = train_epoch(model, train_data_loader, loss_fn, optimizer, device, scheduler, len(df_train))

    #Print the loss and accuracy
    print(f'Train loss: {train_loss}, Train accuracy: {train_acc}')

    #Call the val epoch method
    val_acc, val_loss = eval_model(model, val_data_loader, loss_fn, device, len(df_val))

    #Print the validation loss and accuracy
    print(f'Validation loss: {val_loss}, Validation accuracy: {val_acc}')

    #Update the history dictionary
    history['train_acc'].append(train_acc)
    history['train_loss'].append(train_loss)
    history['val_acc'].append(val_acc)
    history['val_loss'].append(val_loss)

    if val_acc > best_accuracy:
        torch.save(model.state_dict(), 'best_model_state.bin')
        best_accuracy = val_acc

this is the code for loading the best model file and i get the accuracy of the first model
 model.load_state_dict(torch.load("/content/best_model_state.bin"))
test_accuracy = eval_model(model, test_data_loader, loss_fn, device, len(df_test))
print("Test accuracy: ", test_accuracy[0].item())


The task being performed is simple classification on the MNIST dataset. The problem I'm facing is that for the architecture shown below, I'm able to perform the training step which was giving a similar error earlier but that was due to some architectural issue, but now it's giving the error for the test step. I've tried to change the batch size and checked the size of the inputs at each step, but couldn't understand what's going wrong.
Thank you for your help!
LeNet5 architecture:
class LeNet5(nn.Module):

def __init__(self, n_classes):
    super(LeNet5, self).__init__()

    self.feature_extractor = nn.Sequential(
        nn.Conv2d(in_channels=1, out_channels=6, kernel_size=4, stride=1),
        nn.Tanh(),
        nn.AvgPool2d(kernel_size=2),
        nn.Conv2d(in_channels=6, out_channels=16, kernel_size=4, stride=1),
        nn.Tanh(),
        nn.AvgPool2d(kernel_size=2),
        nn.Conv2d(in_channels=16, out_channels=120, kernel_size=4, stride=1),
        nn.Tanh()
    )

    self.classifier = nn.Sequential(
        nn.Linear(in_features=120*2*2, out_features=84),
        nn.Tanh(),
        nn.Linear(in_features=84, out_features=n_classes),
    )

def forward(self, x):
  # print(x.shape)
  x = self.feature_extractor(x)
  # print(x.shape)
  x = x.view(-1, 120*2*2)
  # print(x.shape)
  logits = self.classifier(x)
  # print(logits.shape)
  return logits

Training function:
def train(model, epoch, delta=1e-5):
  global best_train_acc, best_test_acc
  device = 'cuda'
  model.to(device)
  print('\nEpoch: %d' % epoch)
  model.train()
  train_loss = 0
  correct = 0
  total = 0
  total_acc = 0
  for batch_idx, (inputs, targets) in enumerate(trainloader):
      inputs, targets = inputs.to(device), targets.to(device)

      # for params in model.feature_extractor.parameters():
      #   params.requires_grad = False

      # model.feature_extractor[6].parameters().requires_grad = True
      optimizer.zero_grad()
      outputs = model(inputs)
      print(outputs.shape)
      loss = criterion(outputs, targets)
      loss.backward()
      optimizer.step()

      train_loss += loss.item()
      _, predicted = outputs.max(1)
      total += targets.size(0)
      correct += predicted.eq(targets).sum().item()
      epsilon, best_alpha = optimizer.privacy_engine.get_privacy_spent(delta)

      print(f'Epoch {epoch} Step {batch_idx}/{len(trainloader)}', 'Loss: %.3f | Acc: %.3f%% (%d/%d)'
                    % (train_loss/(batch_idx+1), 100.*correct/total, correct, total),  f"(ε = {epsilon:.2f}, δ = {delta}) for α = {best_alpha}")

      acc = 100.*correct/total
      total_acc += acc
      if acc>best_train_acc:
          best_train_acc = acc

  return total_acc/len(trainloader)

Test function:
def test(model, epoch, model_name, lr):
  global best_train_acc, best_test_acc
  device = 'cuda'
  model.eval()
  test_loss = 0
  correct = 0
  total = 0
  total_acc = 0
  with torch.no_grad():
      for batch_idx, (inputs, targets) in enumerate(testloader):
          # print(inputs.shape, inputs[0].shape)
          inputs, targets = inputs.to(device), targets.to(device)
          outputs = model(inputs)
          loss = criterion(outputs, targets)

          test_loss += loss.item()
          _, predicted = outputs.max(1)
          total += targets.size(0)
          correct += predicted.eq(targets).sum().item()

          print(f'Epoch {epoch} Step {batch_idx}/{len(testloader)}', 'Test Loss: %.3f | Test Acc: %.3f%% (%d/%d)'
                        % (test_loss/(batch_idx+1), 100.*correct/total, correct, total))

  # Save checkpoint.
  acc = 100.*correct/total
  total_acc += acc
  if acc > best_test_acc:
      print('Saving..')
      state = {
          'model': model.state_dict(),
          'acc': acc,
          'epoch': epoch,
      }
      if not os.path.isdir('/content/drive/My Drive/checkit'):
          os.mkdir('/content/drive/My Drive/checkit')
      torch.save(state, f'/content/drive/My Drive/checkit/ckpt_{model_name}_{lr}_MNIST.pth')
      best_test_acc = acc

  return total_acc

Error
    ValueError                                Traceback (most recent call last)
<ipython-input-25-5c2bb9b3b3d1> in <module>()
     22 for epoch in range(start_epoch, start_epoch+50):
     23     LeNet5_train_acc.append([train(model, epoch), epoch+1])
---> 24     LeNet5_test_acc.append([test(model, epoch, 'LeNet5', lr), epoch+1])
     25
     26 best_acc.append([best_train_acc, best_test_acc])

4 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)
   2260     if input.size(0) != target.size(0):
   2261         raise ValueError('Expected input batch_size ({}) to match target batch_size ({}).'
-> 2262                          .format(input.size(0), target.size(0)))
   2263     if dim == 2:
   2264         ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)

ValueError: Expected input batch_size (8) to match target batch_size (32).


title: What is switchable atrous convolution and why we use it?

body:
I was trying to understand a research paper called "DetectoRS: Detecting Objects with Recursive Feature Pyramid and Switchable Atrous Convolution". What I didn't understand is how the "Switchable Atrous Convolution" works and why we have used it? I know what Atrous/ Dilated convolution is, but what is Switchable here? How it is determined?
I had spent days trying to grasp the concept before asking this question here.
Here are the links that I have collected and read from (might help you as well):
Official research paper at arxiv
Medium blog to get high-level overview
Python Implementation of Switchable Atrous Convolution (official GitHub repo)
I really value your time.
Thank you.


title: RuntimeError 'CUDA error' when trying the Google Cloud AI Platform Tutorial for PyTorch

body:
I've been trying to get started with Google Cloud's AI Platform. I have been following this tutorial: https://cloud.google.com/ai-platform/training/docs/getting-started-pytorch#gpu_1
My own models are written in PyTorch, hence the choice to get started with PyTorch. Why I want to use the GPU I guess goes without saying.
I've tried to follow the instructions to the letter, and I've used the provided sample code. Yet I still run into errors. I can create a job without problems, but the job ends up failing with the following error:
RuntimeError: CUDA error: no kernel image is available for execution on the device
I'm relatively new to PyTorch and completely new to GCP, so I have no idea how I would go about fixing this and any help would be much appreciated.
Full trace:
The replica master 0 exited with a non-zero status of 1.
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/opt/conda/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/root/.local/lib/python3.7/site-packages/trainer/task.py", line 123, in <module>
    main()
  File "/root/.local/lib/python3.7/site-packages/trainer/task.py", line 119, in main
    experiment.run(args)
  File "/root/.local/lib/python3.7/site-packages/trainer/experiment.py", line 132, in run
    train(sequential_model, train_loader, criterion, optimizer, epoch)
  File "/root/.local/lib/python3.7/site-packages/trainer/experiment.py", line 37, in train
    for batch_index, data in enumerate(train_loader):
  File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 347, in __next__
    data = self._next_data()
  File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 387, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
    return self.collate_fn(data)
  File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py", line 74, in default_collate
    return {key: default_collate([d[key] for d in batch]) for key in elem}
  File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py", line 74, in <dictcomp>
    return {key: default_collate([d[key] for d in batch]) for key in elem}
  File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py", line 55, in default_collate
    return torch.stack(batch, 0, out=out)
RuntimeError: CUDA error: no kernel image is available for execution on the device


title: Longformer get last_hidden_state

body:
I am trying to follow this example in the huggingface documentation here https://huggingface.co/transformers/model_doc/longformer.html:
import torch
from transformers import LongformerModel, LongformerTokenizer
model = LongformerModel.from_pretrained('allenai/longformer-base-4096')
tokenizer = LongformerTokenizer.from_pretrained('allenai/longformer-base-4096')
SAMPLE_TEXT = ' '.join(['Hello world! '] * 1000)  # long input document
input_ids = torch.tensor(tokenizer.encode(SAMPLE_TEXT)).unsqueeze(0)  # batch of size 1
# Attention mask values -- 0: no attention, 1: local attention, 2: global attention
attention_mask = torch.ones(input_ids.shape, dtype=torch.long, device=input_ids.device) # initialize to local attention
global_attention_mask = torch.zeros(input_ids.shape, dtype=torch.long, device=input_ids.device) # initialize to global attention to be deactivated for all tokens
global_attention_mask[:, [1, 4, 21,]] = 1  # Set global attention to random tokens for the sake of this example
                                    # Usually, set global attention based on the task. For example,
                                    # classification: the <s> token
                                    # QA: question tokens
                                    # LM: potentially on the beginning of sentences and paragraphs
outputs = model(input_ids, attention_mask=attention_mask, global_attention_mask=global_attention_mask, output_hidden_states= True)
sequence_output = outputs[0].last_hidden_state
pooled_output = outputs.pooler_output

I suppose that this would return a document embedding for the sample text.
However, I run into the following error:
AttributeError: 'Tensor' object has no attribute 'last_hidden_state'

Why isnt it possible to call last_hidden_state?


How to concatenate BERT-like sentence representation and word embeddings - Keras & huggingface
I am following this Keras tutorial to combine Hugging Face transformers with other layers:
https://keras.io/examples/nlp/text_extraction_with_bert/
I want to concatenate the transformer embedding layer (included in the tutorial) with some regular word embeddings:
encoder = TFBertModel.from_pretrained("bert-base-uncased")

input_ids = layers.Input(shape=(max_transformer_len,), dtype=tf.int32)
token_type_ids = layers.Input(shape=(max_transformer_len,), dtype=tf.int32)
attention_mask = layers.Input(shape=(max_transformer_len,), dtype=tf.int32)
embedding1 = encoder(
    input_ids, token_type_ids=token_type_ids, attention_mask=attention_mask)[0]`

input_wordembedding = Input(shape=(max_sentence_len,), dtype='int32',
                 name='we_input')
embedding2 = Embedding(output_dim=wordembedding_VECTOR_SIZE,
                 input_dim=wordembedding_VOCAB_SIZE,
                 input_length=max_sentence_len,
                 weights = [emb_matrix],
                 name='emb1')(we_input)

z = Concatenate(name='merged')([embedding1, embedding2])

My problem is that the layer embedding1 has sub-words representations and embedding2 has words representations. Then, I want to do a max-pooling of the sub-words in the embedding1 layer. In this way, I will obtain word representations with a transformer model.
Does anyone know how to approach this problem in Keras? If it is impossible to solve it with Keras, is it doable in PyTorch? How?


title: After installig Pytorch cuda , torch.cuda.is_available() show false . What to do?

body:
I have install pytorch cuda by this ..
conda install pytorch torchvision torchaudio cudatoolkit=11.1 -c pytorch -c nvidia

My cuda version is 11.2 . I am using windows 10 .
Pytorch cuda 11.2 is not available right now . (pytorch.org)
So i have install 11.1 version .
(using nvidia-smi)
Cuda version
But it show false.
torch.cuda.is_available() >>> False

I have tried both 10.2  and 11.1  version .
As far i know,  i do not need to install cuda toolkit for pytorch


title: Torch's loss.backward() hanged on ParlAI

body:
I am interested in Memory Networks and Movie Dialog QA. Recently facebook announced AI training framework called ParlAI, which supports many models and datasets. Although I tried below command on ParlAI, the training stopped at first loss.backward() at memnn.py. I waited almost one day, but loss.backward() didn't finish. I have checked this by printing debug and [Using Cuda] printing. Actually my GPU was working because it used some memory. I checked this by nvidia-smi -l 1.

python examples/train_model.py -m memnn -t "#moviedd-qa" -bs 32 --gpu 0 -e 10

Then, I switched to simple task, and it finished a few minutes.

python examples/train_model.py -m memnn -t "babi:task1k:1" -bs 32 --gpu 0 -e 10

I realize #moviedd-qa is more complicated compared to babi task. But how long does it usually take to train this model in my setting? Does anyone try to train this model via ParlAI? I am afraid this is not bug of ParlAI. Could you advise me to proceed my work?

My Environment

Ubunt 16.04.03 LTS, 64 bit
python 3.6.1 (Anaconda 4.4.0 (64-bit))
GPU: GTX 1080 ti
CPU: Intel(R) Core(TM) i7-7700K CPU @ 4.20GHz
torch.version: '0.2.0_3'

I am also asking developers at ParlAI at their github, but no responses.


title: Unable to duplicate PyTorch's Neural Transfer example

body:
I'm following the tutorial from PyTorch's Neural Transfer. However, after downloading the Jupyter notebook and running it, I don't get the output as expected. Any ideas on what I'm doing wrong?

Observed output.

Expected output.

It should be noted that I'm in the following environment with no CUDA support (on a Mac laptop).

conda 4.5.0
python 2.7.17
jupyter 4.4.0
pytorch 0.3.1
torchvision 0.2.0

I did notice that the observed output is 128x128 and the expected is 512x512 (although, this observed difference is likely coming from the setup of using CPU or GPU).

The output of the observed style transfer looks like the following.

Building the style transfer model..
Optimizing..
run [50]:
Style Loss : 351316.375000 Content Loss: 53.188274

run [100]:
Style Loss : 318270.687500 Content Loss: 51.054543

run [150]:
Style Loss : 627803.375000 Content Loss: 59.788795

run [200]:
Style Loss : 351316.375000 Content Loss: 53.188274

run [250]:
Style Loss : 318270.687500 Content Loss: 51.054543

run [300]:
Style Loss : 627803.375000 Content Loss: 59.788795

And the expected output looks like the following.

Building the style transfer model..
Optimizing..
run [50]:
Style Loss : 0.147848 Content Loss: 0.470452

run [100]:
Style Loss : 0.043477 Content Loss: 0.343297

run [150]:
Style Loss : 0.035841 Content Loss: 0.315429

run [200]:
Style Loss : 0.031865 Content Loss: 0.306764

run [250]:
Style Loss : 0.030837 Content Loss: 0.302101

run [300]:
Style Loss : 0.030777 Content Loss: 0.299595


title: CNN weights getting stuck

body:
This is a slightly theoretical question. Below is a graph the plots the loss as the CNN is being trained. Y axis is MSE and X axis is number of Epochs.

Description of CNN:

class Net(nn.Module):

def __init__ (self):
    super(Net, self).__init__()
    self.conv1 = nn.Conv1d(in_channels = 1, out_channels = 5, kernel_size = 9) #.double
    self.pool1 = nn.MaxPool1d(3)
    self.fc1 = nn.Linear(5*30, 200)
    #self.dropout = nn.Dropout(p = 0.5)
    self.fc2 = nn.Linear(200, 99)

def forward(self, x):
    x = self.pool1(F.relu(self.conv1(x)))
    x = x.view(-1, 5 * 30)
    #x = self.dropout(F.relu(self.fc1(x)))
    x = F.relu(self.fc1(x))
    x = self.fc2(x)
    return x

def init_weights(m):
    if type(m) == nn.Linear:
       nn.init.xavier_uniform_(m.weight)
       m.bias.data.fill_(0.01)

 net = Net()
 net.apply(init_weights)

 criterion = nn.MSELoss()
 optimizer = optim.Adam(net.parameters(), lr=0.01-0.0001) # depends

Both input and output are an array of numbers. It is a multiregression ouput-problem.

This issue where the loss/weights gets stuck in an incorrect place doesn’t happen as much if I use a lower learning rate. However, it still happens. In some sense that means that the hyper dimensional space that is created by the parameters of the CNN is jagged with a lot of local minimums.  This could be true because the CNN ‘s inputs are very similar. Would increasing the layers of the CNN both conv layers and fully connected linears help solve this problem as they hyper dimensional space might be smoother? Or is this intuition completely incorrect?
A more broad question when should you be inclined to add more convolutional layers? I know that in practice you should almost never start from scratch and instead use another model’s first few layers. However, the inputs I am using are very different to anything I have found online and therefore cannot do this.


title: Pytorch XOR implementation rounding issue?

body:
I've implemented the XOR problem with a super simple NN in Pytorch as a learning exercise.

The network is trained, and now that I'm giving it new inputs, looking for output, I get the following:

print("(0, 0) = {}".format(model(Variable(torch.Tensor([0, 0]).unsqueeze(0)))))

print("(1, 0) = {}".format(model(Variable(torch.Tensor([1, 0]).unsqueeze(0)))))

7.7716e-07, 1

Why does it output:

1 (exactly ) for the input [1, 0]
7.7716e-07 (not exactly 0) for [0, 0]

I understand it's close to it, but why isn't it, unlike the other?

Does it simply require more training?


title: matplotlib numpy -- TypeError: Cannot read property 'props' of undefined -- Graph not showing up?

body:
This code works on one machine, but not the other. I can't seem to isolate the issue with the dependencies.

Sample code from: https://matplotlib.org/3.1.1/gallery/lines_bars_and_markers/simple_plot.html

--

import matplotlib
import matplotlib.pyplot as plt
import numpy as np

# Data for plotting
t = np.arange(0.0, 2.0, 0.01)
s = 1 + np.sin(2 * np.pi * t)

fig, ax = plt.subplots()
ax.plot(t, s)

ax.set(xlabel='time (s)', ylabel='voltage (mV)',
       title='About as simple as it gets, folks')
ax.grid()

fig.savefig("test.png")
plt.show()

--> TypeError: Cannot read property 'props' of undefined

This code does not produce a stack trace of explicitly fail. The TypeError: is returned by the call to plt.show(). I tried searching for this error, but I couldn't find anything reported. (I added a screenshot of what visual-studio on the server I am working on shows. I will likely just create a docker of my other environment as I should have done that in the first place to avoid having my configuration break.) The graph saves correctly to file. It's just an issue with plt.show().

np.version
'1.17.2'

matplotlib.version
'3.1.1'

screen-shot

graph-saves-correctly


title: HOW (how) does Keras and Pytorch handle the last batch when batch_size is not a multiple of size of training data?

body:
Before you happen to suggest or report as duplicate, I am asking HOW does the two libraries handle the problem NOT WHAT HAPPENS because I know it can make a batch from remaining data. And I do know that it handles but all I am asking is HOW.

If I have 100 images as training data and batch_size=15, the last batch will have 10 images to train. My Question is that when the Input() layer already knows that the data is coming as a shape of (Batch_size,channel,width,height) for PyTorch and (batch_size,width,height,channels) for Keras using Tensorflow as backend.

If the last batch has size=10, isn't the model supposed to throw an error because it will get (10,1,28,28) in place of (15,1,28,28) given we have (28,28) pixels Grayscale images?

What is happening behind the scenes?


title: Loss not Converging for CNN Model

body:
Image Transformation and Batch

transform = transforms.Compose([

                                transforms.Resize((100,100)),

                                transforms.ToTensor(),

                                transforms.Normalize([0.485,0.456,0.406],[0.229,0.224,0.225])

                                ])

data_set = datasets.ImageFolder(root="/content/drive/My Drive/models/pokemon/dataset",transform=transform)

train_loader = DataLoader(data_set,batch_size=10,shuffle=True,num_workers=6)

Below is my Model

class pokimonClassifier(nn.Module):

  def __init__(self):

    super().__init__()

    self.conv1 = nn.Conv2d(3,6,3,1)

    self.conv2 = nn.Conv2d(6,18,3,1)

    self.fc1 = nn.Linear(23*23*18,520)

    self.fc2 = nn.Linear(520,400)

    self.fc3 = nn.Linear(400,320)

    self.fc4 = nn.Linear(320,149)

  def forward(self,x):

    x = F.relu(self.conv1(x))

    x = F.max_pool2d(x,2,2)

    x = F.relu(self.conv2(x))

    x = F.max_pool2d(x,2,2)

    x = x.view(-1,23*23*18)

    x = F.relu(self.fc1(x))

    x = F.relu(self.fc2(x))

    x = F.relu(self.fc3(x))

    x = F.log_softmax(self.fc4(x), dim=1)

    return x

Creating Instance of model, Use GPU, Set Criterion and optimizer
Here is firsr set lr = 0.001 then later changed to 0.0001

model = pokimonClassifier()
model.to('cuda')
criterion = nn.CrossEntropyLoss()

optimizer = torch.optim.Adam(model.parameters(),lr = 0.0001)

Training Dataset

for e in range(epochs):

  train_crt = 0

  for b,(train_x,train_y) in enumerate(train_loader):

    b+=1

    train_x, train_y = train_x.to('cuda'), train_y.to('cuda')

    # train model

    y_preds = model(train_x)

    loss = criterion(y_preds,train_y)

    # analysis model

    predicted = torch.max(y_preds,1)[1]

    correct = (predicted == train_y).sum()

    train_crt += correct

    # print loss and accuracy

    if b%50 == 0:

        print(f'Epoch {e} batch{b} loss:{loss.item()} ')

    # updating weights and bais

    optimizer.zero_grad()

    loss.backward()

    optimizer.step()

  train_loss.append(loss)

  train_correct.append(train_crt)

My loss value remains between 4 - 3  and its not converging to 0.
I am super new to deep learning and  I don't know much about it.

The dataset I am using is here: https://www.kaggle.com/thedagger/pokemon-generation-one

A help will be much appreciated.
Thank You


title: Understanding LSTM with a simple dataset

body:
I wanted to make sure I understand LSTM so I implemented a dummy example using Pytorch framework.
As an input, I use sequences of consecutive numbers of length 10 and the value to predict is always the last number of sequence + 1. For instance:
x = [6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
y = 16

Since it's a very simple forecasting task, I expected the model to work well but I observe very poor performances. The model predicts a constant value by batch that keeps increasing during the training process.

I am wondering what I am missing. Below is the code I've made - any help would be highly appreciated.

from torch.utils.data import Dataset, TensorDataset, DataLoader, RandomSampler, SequentialSampler
import torch.nn as nn
import torch

class MyDataset(Dataset):

    def __init__(self):
        pass

    def __getitem__(self, index):
        x = torch.tensor([index-9,index-8,index-7,index-6,index-5,index-4,index-3,index-2,index-1,index])
        y = torch.tensor(index + 1)
        return x,y

    def __len__(self):
        return 1000

class LSTM(nn.Module):
    def __init__(self, hidden_layer_size=1, batch_size = 1):

        super().__init__()
        self.hidden_layer_size = hidden_layer_size
        self.batch_size = batch_size
        self.lstm = nn.LSTM(1, hidden_layer_size)
        self.linear = nn.Linear(10, 1)
        self.hidden_cell = (torch.zeros(1,self.batch_size,self.hidden_layer_size),
                            torch.zeros(1,self.batch_size,self.hidden_layer_size))

    def forward(self, input_seq):

        lstm_out, self.hidden_cell = self.lstm(input_seq.view(10 ,self.batch_size, -1), self.hidden_cell)
        predictions = self.linear(lstm_out.squeeze().T)
        return predictions

batch_size = 32
epochs = 1000

train = MyDataset()
sampler = RandomSampler(train)
train_dataloader = DataLoader(train, sampler=sampler, batch_size= batch_size , drop_last = True)

model = LSTM(batch_size = batch_size)
loss_function = nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)

for e in range(epochs):
    for step, batch in enumerate(train_dataloader) :

        seq, labels = batch
        optimizer.zero_grad()

        model.hidden_cell = (torch.zeros(1, batch_size, model.hidden_layer_size),
                             torch.zeros(1, batch_size, model.hidden_layer_size))

        y_pred = model(seq.float())

        print(y_pred)

        single_loss = loss_function(y_pred, labels.float())
        single_loss.backward()
        optimizer.step()


title: why time spent on cpu is equal to time spent on gpu in Pytorch with Profile?

body:
everyone!
I'm a new coder using Pytorch.I have a question, why the time spent on cpu is equal to time spent on gpu?
The time costs on cpu and time costs on cuda is very approximate,and I suspect the code run on cpu, but with tool of top and 'watch -n 3 nvidia-smi' in another terminal, I saw the change with the gpu when code run on gpu.
Is there any method to show, the code really run on cuda?
Thank you very much!
My Pytorch version is 1.3.0, CUDA 9.0.
Code is:
import torch
input = torch.randn(1000, 3, 224, 224)
conv1 = torch.Conv2d(3, 5, (3, 3))
conv2 = torch.Conv2d(5, 7, (5, 5))

with torch.autograd.Profiler.profile(use_cuda=True) as prof:
    out = conv1(input)
    out = torch.relu(out)
    out = conv2(out)
    out = torch.relu(out)
    out = torch.relu(out)

print("prof is:\n{}".format(prof))

//the code above is I memorize, but the thought is this, could be different with image
opkernel name and cpu time
cpu and cuda time