title: Is there a way to impose memory limit on google or tools pywrapknapsack_solver?

body:
I have created 5,000 instances of a knapsack dataset with each dataset including 100 weights, 100 values, the total weight and a capacity percentage. The weights and values have a normal distribution with mean 100 and std 5. The input file is a pickle formatted as [tensor(weights), tensor(values), tensor(total weight), capacity percentage].
The pickle input file
I have created a python program that takes in the input file, uses the google or tools pywrapknapsack_solver to solve the problem, and outputs the solution as a one-hot vector and the solution time.
import torch
from ortools.algorithms import pywrapknapsack_solver
import time
import pickle
import argparse

def translate_soln(packed_items, num_items):
    out = [0]*num_items
    for i in packed_items:
        out[int(i)] = 1
    return out

def knapsack_solution_generator(input_path, output_path):
    input_data = pickle.load( open(input_path, 'rb'))

    weights = input_data[0].tolist()
    values = input_data[1].tolist()

    total_weight = input_data[2].item()
    capacity_percentage = input_data[3]
    capacity = [total_weight * (capacity_percentage / 100)]

    num_items = len(weights)

    solver = pywrapknapsack_solver.KnapsackSolver(
            pywrapknapsack_solver.KnapsackSolver.
            KNAPSACK_MULTIDIMENSION_BRANCH_AND_BOUND_SOLVER, 'KnapsackExample')
    solver.set_time_limit(7200)

    start_time = time.perf_counter()
    solver.Init(values, [weights, capacity)

    computed_value = solver.Solve()
    end_time = time.perf_counter()

    solution_time = round(end_time - start_time, 5)

    packed_items = []
    for j in range(num_items):
        if solver.BestSolutionContains(j):
            packed_items.append(j)

    solution = translate_soln(packed_items, num_items)
    pickle.dump((solution, solution_time), open(output_path, 'wb'))

if __name__ == "__main__":
    parser = argparse.ArgumentParser()

    parser.add_argument('--input', type=str, help="Input file path")
    parser.add_argument('--output', type=str, help="Output file path")

    args = parser.parse_args()
    input_path = args.input
    output_path = args.output

    knapsack_solution_generator(input_path, output_path)

I have been using a Linux virtual machine to solve all of these problems independently. However, when the machine attempts to solve problems with capacity percentages of the range 10-90%, there are certain problem instances that will not solve. When looking at the output log, there is a "Killed error". To dig deeper I had the machine attempt to solve the problem instances again as I monitored with the linux top command and saw that the problems that would not solve were killed at 80% memory after about 20 minutes. However, certain problems would be solved in 2 hours. What is causing the solver to use up too much memory? Is there a way to implement a memory constraint?


import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

import torch
import torch.nn as nn
import torch.nn.functional as F
import numpy as np
from torch.autograd import Variable
from sklearn import preprocessing

batch_size = 32
num_classes = 8
epochs = 10

img_rows, img_cols = 256, 256
x_train = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)
x_test = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1)
input_shape = (img_rows, img_cols, 1)

print('x_train shape:', x_train.shape)
print("Training samples: {}".format(x_train.shape[0]))
print("Test samples: {}".format
      (x_test.shape[0]))
x_train = torch.Tensor(x_train).float()
x_test = torch.Tensor(x_test).float()
y_train = torch.LongTensor(y_train)
y_test = torch.LongTensor(y_test)
#Define model
model = nn.Sequential(
        nn.Conv2d(256,80,1, stride=1),
        nn.ReLU(),
        nn.Conv2d(80,40,1, stride=1),
        nn.ReLU(),
        nn.MaxPool2d(4,stride=1),
        nn.Conv2d(40,30,1, stride=1),
        nn.ReLU(),
        nn.Conv2d(30,15,1, stride=1),
        nn.ReLU(),
        nn.Dropout(0.2),
        nn.Flatten(),
        nn.Linear(32, 32),
        nn.Dropout(0.1),
        nn.Linear(num_classes, 256),
        nn.ReLU()
        )
criterion = nn.CrossEntropyLoss()# cross entropy loss

optimizer = torch.optim.Adam(model.parameters(), lr=0.01)

for epoch in range(100):
    optimizer.zero_grad()
    out = model(x_train)
    loss = criterion(out, y_train)
    loss.backward()
    optimizer.step()
    print(f"Epoch {epoch+1}, loss: {loss.item()}")

However when I run this code it produces the error at this specifc line
RuntimeError                              Traceback (most recent call last)
<ipython-input-10-96fa8b09f1ec> in <module>
     63 for epoch in range(100):
     64     optimizer.zero_grad()
---> 65     out = model(x_train)
     66     loss = criterion(out, y_train)
     67     loss.backward()

with the following error message
RuntimeError: Given input size: (40x256x1). Calculated output size: (40x253x-2). Output size is too small
I am unsure on how to fix this as I am new to pytorch and the original model did work in tensorflow.
Any help would be much appreciated


title: how to see the data in DataLoader in pytorch

body:
I see something like the following in the examples on Github.
How can I see the type of this data (shape and the other properties)?
train_data = MyDataset(int(1e3), length=50)
train_iterator = DataLoader(train_data, batch_size=1000, shuffle=True)


title: Testing my CNN on a small set of image but training has no effect

body:
I constructed a CNN to recognize 9 classes of gestures in images of 224x224x3. I try to test its functionality by training it on 16 images and see if it overfits to 100 accuracy. Here is my network
    import torch.nn as nn
    class learn_gesture(nn.Module):
        def __init__(self):
            super(learn_gesture, self).__init__()
            self.name = "gesture_learner"
            self.conv1 = nn.Conv2d(in_channels=3, out_channels=20, kernel_size=5, stride=1, padding=2)
            self.conv2 = nn.Conv2d(in_channels=20, out_channels=50, kernel_size=5, stride=1, padding=2)
            self.conv3 = nn.Conv2d(in_channels=50, out_channels=100, kernel_size=5, stride=1, padding=2)
            self.conv4 = nn.Conv2d(in_channels=100, out_channels=200, kernel_size=5, stride=1, padding=2)
            self.conv5 = nn.Conv2d(in_channels=200, out_channels=400, kernel_size=5, stride=1, padding=2)
            self.pool1 = nn.MaxPool2d(2,2)
            self.pool2 = nn.MaxPool2d(2,2)
            self.pool3 = nn.MaxPool2d(2,2)
            self.pool4 = nn.MaxPool2d(2,2)
            self.pool5 = nn.MaxPool2d(2,2)
            self.fc1 = nn.Linear(7*7*400, 10000)
            self.fc2 = nn.Linear(10000, 3000)
            self.fc3 = nn.Linear(3000, 9)

        def forward(self, x):
            x = self.pool1(F.relu(self.conv1(x))) # gives 112*20
            x = self.pool2(F.relu(self.conv2(x))) # gives 56*50
            x = self.pool3(F.relu(self.conv3(x))) # gives 28*100
            x = self.pool4(F.relu(self.conv4(x))) # gives 14*200
            x = self.pool5(F.relu(self.conv5(x))) # gives 7*400
            x = x.view(-1, 7*7*400)
            x = F.relu(self.fc1(x))
            x = F.relu(self.fc2(x))
            return F.softmax(self.fc3(x), dim=1)

And here is the training code:
    overfit_model = learn_gesture()
    num_epochs = 200   #set it high so that it will converge
    ## loss function and optimizer
    criterion = nn.CrossEntropyLoss()
    optimizer = optim.SGD(over_model.parameters(), lr=0.001, momentum=0.9)       #optimizer is SGD with momentum

    ## set up some empty np arrays to store our result for plotting later
    train_err = np.zeros(num_epochs)
    train_loss = np.zeros(num_epochs)
    ################################################ train the network
    for epoch in range(num_epochs):
        total_train_loss = 0
        total_train_err = 0
        total_epoch = 0
        for i, data in enumerate(smallLoader, 0):
            inputs, labels = data
            outputs = over_model(inputs)
            loss = criterion(outputs, labels)
            loss.backward()
            optimizer.step()
            optimizer.zero_grad()
            corr = (determine_corr(outputs, labels))    # get a list of bool representing right or wrong predictions in the batch
            total_train_err += corr.count(False)
            total_train_loss += loss.item()
            total_epoch += len(labels)
        train_err[epoch] = float(total_train_err) / total_epoch
        train_loss[epoch] = float(total_train_loss) / (i+1)
        print(("Epoch {}: Train err: {}, Train loss: {}").format(
                   enter code hereepoch + 1,
                   train_err[epoch],
                   train_loss[epoch]))

The training has no effect, and both the accuracy and loss has no improvement either. I just absolutely can't figure out where the error is. Any help is greatly appreciated!
###############  Update ##############
I got rid of the softmax in the forward function. Surprisingly, the performance of the model hasn't changed much. And I notice that some elements in the output now are negative and the elements across all classes do not add to 1. Is this supposed to happen?
output:
tensor([[ 0.0165, -0.0041,  0.0043,  0.0017,  0.0238,  0.0329, -0.0265, -0.0224,
     -0.0187],
    [ 0.0163, -0.0044,  0.0036,  0.0028,  0.0248,  0.0334, -0.0268, -0.0218,
     -0.0194],
    [ 0.0161, -0.0046,  0.0041,  0.0019,  0.0240,  0.0333, -0.0266, -0.0223,
     -0.0192],
    [ 0.0190, -0.0044,  0.0035,  0.0015,  0.0244,  0.0322, -0.0267, -0.0223,
     -0.0187],
    [ 0.0174, -0.0048,  0.0033,  0.0021,  0.0251,  0.0328, -0.0257, -0.0225,
     -0.0190],
    [ 0.0175, -0.0041,  0.0033,  0.0031,  0.0241,  0.0329, -0.0264, -0.0222,
     -0.0192],
    [ 0.0168, -0.0042,  0.0033,  0.0022,  0.0251,  0.0335, -0.0269, -0.0225,
     -0.0195],
    [ 0.0163, -0.0047,  0.0037,  0.0030,  0.0243,  0.0336, -0.0265, -0.0227,
     -0.0192],
    [ 0.0165, -0.0043,  0.0038,  0.0026,  0.0242,  0.0337, -0.0264, -0.0222,
     -0.0191],
    [ 0.0163, -0.0051,  0.0038,  0.0016,  0.0236,  0.0338, -0.0258, -0.0223,
     -0.0195],
    [ 0.0173, -0.0037,  0.0038,  0.0018,  0.0236,  0.0322, -0.0269, -0.0225,
     -0.0191],
    [ 0.0174, -0.0044,  0.0031,  0.0019,  0.0241,  0.0334, -0.0266, -0.0224,
     -0.0200],
    [ 0.0164, -0.0038,  0.0034,  0.0029,  0.0245,  0.0342, -0.0269, -0.0225,
     -0.0200],
    [ 0.0173, -0.0046,  0.0036,  0.0021,  0.0245,  0.0328, -0.0264, -0.0221,
     -0.0192],
    [ 0.0168, -0.0046,  0.0034,  0.0025,  0.0248,  0.0336, -0.0262, -0.0222,
     -0.0194],
    [ 0.0166, -0.0051,  0.0033,  0.0015,  0.0234,  0.0331, -0.0270, -0.0218,
     -0.0186]], grad_fn=<AddmmBackward>)
Epoch 199: Train err: 0.8125, Train loss: 2.1874701976776123


title: pic should be PIL Image or ndarray. Got

body:
Im trying to download cifar10 images and put it in dataloader, however the problem
'pic should be PIL Image or ndarray. Got <class 'tuple'>' keeps occur. Please help me.
transform_list={'train':transforms.Compose([transforms.Resize(256),
                                           transforms.RandomHorizontalFlip(),
                                           transforms.FiveCrop(size=224),
                                           transforms.ToTensor(),
                                           transforms.Normalize(mean=(0.5,0.5,0.5), std=(0.5,0.5,0.5))]),
               'test':transforms.Compose([transforms.Resize(224),
                                          transforms.ToTensor(),
                                          transforms.Normalize(mean=(0.5,0.5,0.5), std=(0.5,0.5,0.5))])}
    cifar10_train=torchvision.datasets.CIFAR10(root='./data',train=True,download=True,transform=transform_list['train'])
    cifar10_test=torchvision.datasets.CIFAR10(root='./data',train=False,download=True,transform=transform_list['test'])
    trainloader = torch.utils.data.DataLoader(cifar10_train, batch_size=128,
                                             shuffle=True)
    testloader = torch.utils.data.DataLoader(cifar10_test, batch_size=128,
                                             shuffle=True)

    print(cifar10_train)
    print(cifar10_train[0])

Error
TypeError                                 Traceback (most recent call last)
<ipython-input-25-03a0afd731d1> in <module>()
     15
     16 print(cifar10_train)
---> 17 print(cifar10_train[0])

3 frames
/usr/local/lib/python3.7/dist-packages/torchvision/transforms/functional.py in to_tensor(pic)
    100     """
    101     if not(F_pil._is_pil_image(pic) or _is_numpy(pic)):
--> 102         raise TypeError('pic should be PIL Image or ndarray. Got {}'.format(type(pic)))
    103
    104     if _is_numpy(pic) and not _is_numpy_image(pic):

TypeError: pic should be PIL Image or ndarray. Got <class 'tuple'>


title: How to increase batch size in GPT2 training for translation task?

body:
I am developing a code to use the pre-trained GPT2 model for a machine translation task. The length of my data's word-to-id is 91, and I developed the following code for my model:
import torch
from torch.utils.data import DataLoader
from transformers.models.gpt2.modeling_gpt2 import GPT2Model

# data preparation code

def batch_sequences(x, y, env):
    """
    Take as input a list of n sequences (torch.LongTensor vectors) and return
    a tensor of size (slen, n) where slen is the length of the longest
    sentence, and a vector lengths containing the length of each sentence.
    """
    lengths_x = torch.LongTensor([len(s) + 2 for s in x])
    lengths_y = torch.LongTensor([len(s) + 2 for s in y])
    max_length = max(lengths_x.max().item(), lengths_y.max().item())
    sent_x = torch.LongTensor(
        max_length, lengths_x.size(0)).fill_(env.pad_index)
    sent_y = torch.LongTensor(
        max_length, lengths_y.size(0)).fill_(env.pad_index)
    assert lengths_x.min().item() > 2
    assert lengths_y.min().item() > 2

    sent_x[0] = env.eos_index
    for i, s in enumerate(x):
        sent_x[1:lengths_x[i] - 1, i].copy_(s)
        sent_x[lengths_x[i] - 1, i] = env.eos_index

    sent_y[0] = env.eos_index
    for i, s in enumerate(y):
        sent_y[1:lengths_y[i] - 1, i].copy_(s)
        sent_y[lengths_y[i] - 1, i] = env.eos_index

    return sent_x, sent_y, max_length

def collate_fn(elements):
    """
    Collate samples into a batch.
    """
    x, y = zip(*elements)
    x = [torch.LongTensor([env.word2id[w]
                          for w in seq if w in env.word2id]) for seq in x]
    y = [torch.LongTensor([env.word2id[w]
                          for w in seq if w in env.word2id]) for seq in y]
    x, y, length = batch_sequences(x, y, env)
    return (x, length), (y, length), torch.LongTensor(nb_ops)

loader = DataLoader(data, batch_size=1, shuffle=False, collate_fn=collate_fn)
gpt2 = GPT2Model.from_pretrained('gpt2')
in_layer = nn.Embedding(len(env.word2id), 768)
out_layer = nn.Linear(768, len(env.word2id))

parameters = list(gpt2.parameters()) + list(in_layer.parameters()) + list(out_layer.parameters())
optimizer = torch.optim.Adam(parameters)
loss_fn = nn.CrossEntropyLoss()
for layer in (gpt2, in_layer, out_layer):
    layer.train()

accuracies = list()
n_epochs = 5
for i in range(n_epochs):
    for (x, x_len), (y, y_len) in loader:

        x = x.to(device=device)
        y = y.to(device=device)

        embeddings = in_layer(x.reshape(1, -1))
        hidden_state = gpt2(inputs_embeds=embeddings).last_hidden_state[:, :]
        logits = out_layer(hidden_state)[0]
        loss = loss_fn(logits, y.reshape(-1))
        accuracies.append(
            (logits.argmax(dim=-1) == y.reshape(-1)).float().mean().item())

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if len(accuracies) % 500 == 0:
            accuracy = sum(accuracies[-50:]) / len(accuracies[-50:])
            print(f'Samples: {len(accuracies)}, Accuracy: {accuracy}')

This code works pretty well when the batch size is 1. But it is so slow. I wanted to increase the batch size from 1 to 32, but I get some dimension compatibility problems. How can I increase the batch size without errors?
My data consists of pair of sentences, the first one is a sentence in the first language and the second one is its translation in the second language.
For example, assume that x.shape is (batch_size, 12) (meaning we have 'batch_size' sentences of length 12 as input and y.shape is also (batch_size, 12) (the translations). And also we have a word-to-id dictionary of length 90 that matches each word in a sentence with its index)


title: Pytorch function name demystification: gels for least squares estimation

body:
What does "gels" stand for in Pytorch?
It solves least squares, but what does the name stand for?

It is hard to get comfortable with a function without getting its name and it is surprising that these are not explained in the documentation.


title: Memory leak in python code

body:
I am trying to make my own dataset with torch in Python. However, the memory usage leaks in every file open.

The memory leak is occurred during the below for loop

for subj, _file in enumerate(filelist):

I deleted unnecessary in the code like import

parser.add_argument('--stddev', dest='stddev', default='subject', type=str, help="How to calculate ")
args = parser.parse_args()

# Options
if args.sess_num == 0:
  label_set = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

base_dir = './kriss_proto'
ori_base_dir = '{}/original'.format(base_dir)
filelist = os.listdir(ori_base_dir) # ['BD001.mat', ...]
filenum = len(filelist)

# Saving data
eeg_data = {}
eeg_data['dataset'] = {}
split_data = {}
split_data['splits'] = []
split_data['splits'].append({})

_meanstd = []
for i in range(32):
  _meanstd.append([])
  if args.stddev == 'subject':
    for j in range(filenum):
      _meanstd[i].append([])

idx = 0

for subj, _file in enumerate(filelist):
  filepath = os.path.join(ori_base_dir, _file)
  with h5py.File(filepath, 'r') as mat:

    ## Step by step (with object searching)
    bd_eeg = mat.get('BD_EEG')
    vst = bd_eeg.get('VST')

    for run in vst.keys():
      print("Processing {} - {}".format(_file, run))
      run_data = vst.get(run)
      eeg = run_data['data3D']
      eeg = th.Tensor(eeg)
      eeg = eeg.permute(2, 1, 0)

      label = run_data['class_labels']
      label = th.Tensor(label)
      label = th.squeeze(label)

      # DATA COPY
      for i in range(int(label.shape[0])):
        if (label[i] in label_set) and (int(eeg[i].shape[0]) > 400):
          eeg_data['dataset'][idx] = {}
          eeg_data['dataset'][idx]['eeg'] = eeg[i, :, :32]
          eeg_data['dataset'][idx]['label'] = label[i] - label_set[0] # label should start from 0
          if args.stddev == 'whole':
            for j in range(32):
              for k in range(int(eeg[i].shape[0])):
                _meanstd[j].append(eeg[i][k, j])
          elif args.stddev == 'subject':
            for j in range(32):
              for k in range(int(eeg[i].shape[0])):
                _meanstd[j][subj].append(eeg[i][k, j])

          idx += 1


title: Error in DeepMask's Pytorch Code

body:
I’m currently trying to implement Deepmask (Link to FAIR's Paper) using Pytorch, so far I have defined the Joint Loss Function, and the model’s learn-able parameters and the forward pass.

I was working on the training phase, and as the paper says that training must be done in an alternative back-propagation fashion across the two branches, I have written the code for the same.

But there is some problem with training, I tried to train the model with a fake data-set (a randomly generated data-set), for minibatches other than the first mini-batch the loss of the model is turning out to be nan.

What could be the reason for this nan loss?

Link to current version of my code


title: About autograd in pyorch, Adding new user-defined layers, how should I make its parameters update?

body:
everyone !

My demand is a optical-flow-generating problem. I have two raw images and a optical flow data as ground truth, now my algorithm is to generate optical flow using raw images, and the euclidean distance between generating optical flow and ground truth could be defined as a loss value, so it can implement a backpropagation to update parameters.

I take it as a regression problem, and I have to ideas now:

I can set every parameters as (required_grad = true), and compute a loss, then I can loss.backward() to acquire the gradient, but I don’t know how to add these parameters in optimizer to update those.

I write my algorithm as a model. If I design a “custom” model, I can initilize several layers such as nn.Con2d(), nn.Linear() in def init() and I can update parameters in methods like (torch.optim.Adam(model.parameters())), but if I define new layers by myself, how should I add this layer’s parameters in updating parameter collection???

This problem has confused me several days. Are there any good methods to update user-defined parameters? I would be very grateful if you could give me some advice!


title: When I code ` torch.addmm` in pytorch and then Segmentation fault, why?

body:
The version of python is 3.6.0
The version of pytorch is 0.4.1

When I do some coding below, the error happens and the programme exit completely. Why? How to fix it?


title: A Basic Python Question about Defining Function

body:
I have a basic question regarding Python code.

For example,

import torch
import torch.nn as nn

loss = nn.MSELoss()
input = torch.randn(3, 5, requires_grad=True)
target = torch.randn(3, 5)
output = loss(input, target)
output.backward()

Why do I need to define the loss function at the first line? I can't replace loss() at the 4th line with nn.MSELoss().

It is a question that confuses me for a while... Hope someone can help out. Thanks!


title: why does linear regression converge slower if we use two layers of linear module?

body:
I have tested linear regression of 1d input and 1d output by pytorch using three different ways.

One is using the formula from linear algebra,
the other one is using nn.Linear(1,1). These two always give identical solution.
However, when I use two layers: nn.Linear(1,2) and nn.Linear(2,1) sequentially for the third approach, the result does not converge at first. After I set learning rate much smaller and epoch number much bigger, it finally shows its convergence.

In theory, because composition of linear transform is again linear transform, they shall give the same answer no matter one layer and two layers.
Intuitively, I thought more neurons and layers make things more efficient. But this shows its opposite and I do not understand.

The code is in github. Please directly jump in the last shell for the third approach.
The expected result is given in both first and second approaches in the notebook.


title: What is the difference between these two neural network structures?

body:
first using nn.Parameter

class ModelOne(nn.Module):
  def __init__(self):
    super().__init__()
    self.weights = nn.Parameter(torch.randn(300, 10))
    self.bias = nn.Parameter(torch.zeros(10))
  def forward(self, x):
    return x @ self.weights + self.bias

when I do

mo = ModelOne()
[len(param) for param in mo.parameters()]

it gives
[300, 10]

second using nn.Linear

class ModelTwo(nn.Module):
  def __init__(self):
    super().__init__()
    self.linear = nn.Linear(300, 10)

  def forward(self, x):
    return self.linear(x)

same thing here gives
[10, 10]


title: How to persist a process in python

body:
I have the following Pytorch code, what the code does is:

Loads Pytorch object detection model into RAM using net=CRAFT()
using net.eval() it runs inference
Using the for loop I am supplying images to run inference on one by one

code:

def copyStateDict(state_dict):
    if list(state_dict.keys())[0].startswith("module"):
        start_idx = 1
    else:
        start_idx = 0
    new_state_dict = OrderedDict()
    for k, v in state_dict.items():
        name = ".".join(k.split(".")[start_idx:])
        new_state_dict[name] = v
    return new_state_dict

if __name__ == '__main__':
    # load net
    net = CRAFT()     # initialize

    print('Loading weights from checkpoint (' + args.trained_model + ')')
    if args.cuda:
        net.load_state_dict(copyStateDict(torch.load(args.trained_model)))
    else:
        net.load_state_dict(copyStateDict(torch.load(args.trained_model, map_location='cpu')))

    if args.cuda:
        net = net.cuda()
        net = torch.nn.DataParallel(net)
        cudnn.benchmark = False

    net.eval()

    t = time.time()

    # load data
    for k, image_path in enumerate(image_list):
        print("Test image {:d}/{:d}: {:s}".format(k+1, len(image_list), image_path), end='\r')
        image = imgproc.loadImage(image_path)

        bboxes, polys, score_text = test_net(net, image, args.text_threshold, args.link_threshold, args.low_text, args.cuda, args.poly)

        # save score text
        #filename, file_ext = os.path.splitext(os.path.basename(image_path))
        #mask_file = result_folder + "/res_" + filename + '_mask.jpg'
        #cv2.imwrite(mask_file, score_text)

        file_utils.saveResult(image_path, image[:,:,::-1], polys, dirname=result_folder)

    print("elapsed time : {}s".format(time.time() - t))

There is a caveat in this code, especially the fact that it occupies 1.8 GB of RAM each time I do step 1 and 2. The actual inference on images takes much lesser RAM (40mb per image), but the majority of my memory consumption comes from loading the ML model into RAM. Is there a way in which I can keep loaded this model into RAM on a server and then just perform inferences on the images?


title: How to exponentially decay a value along an axis?

body:
I have a tensor of shape (sequence_length,batch_size,1) (which is (5,5,1) in the following example). Every sequence has a single entry that I want to exponentially decay along the sequence (for example with a decay factor of 0.99) for n steps (here: n = 2).

Example:

Is there an efficient way to realize something like that (either in PyTorch or numpy).

Edit (visualizing the additive nature):


title: Is this how nn.Transformer works?

body:
If I want to transform an image to another image,
then

transformer_model = nn.Transformer(img_size, n_heads)
transformer_model(source_image, target_image)

is this the correct way to use nn.Transformer?


title: Debugging neural network dropout problem for the probability not lying inside [0,1]

body:
I tried to put a droprate to my neural network (NN) using torch and I got a strange error at the end. How can I fix it?

So the idea is that I wrote a NN inside a function to make it easier to call. The function is the following:
(I personally think the problem lies inside the class of the NN, but for the sake of having a working example I'm putting everything).

def train_neural_network(data_train_X, data_train_Y, batch_size, learning_rate, graph = True, dropout = 0.0 ):
  input_size = len(data_test_X.columns)
  hidden_size = 200
  num_classes = 4
  num_epochs = 120
  batch_size = batch_size
  learning_rate = learning_rate

  # The class of NN
  class NeuralNet(nn.Module):
    def __init__(self, input_size, hidden_size, num_classes, p = dropout):
        super(NeuralNet, self).__init__()
        self.fc1 = nn.Linear(input_size, hidden_size)
        self.fc2 = nn.Linear(hidden_size, hidden_size)
        self.fc3 = nn.Linear(hidden_size, num_classes)

    def forward(self, x, p = dropout):
          out = F.relu(self.fc1(x))
          out = F.relu(self.fc2(out))
          out = nn.Dropout(out, p) #drop
          out = self.fc3(out)
          return out

  # Prepare data
  X_train = torch.from_numpy(data_train_X.values).float()
  Y_train = torch.from_numpy(data_train_Y.values).float()

  # Loading data
  train = torch.utils.data.TensorDataset(X_train, Y_train)
  train_loader = torch.utils.data.DataLoader(train, batch_size=batch_size)

  net = NeuralNet(input_size, hidden_size, num_classes)

  # Loss
  criterion = nn.CrossEntropyLoss()

  # Optimiser
  optimiser = torch.optim.SGD(net.parameters(), lr=learning_rate)

  # Proper training
  total_step = len(train_loader)
  loss_values = []

  for epoch in range(num_epochs+1):
    net.train()

    train_loss = 0.0

    for i, (predictors, results) in enumerate(train_loader, 0):
      # Forward pass
      outputs = net(predictors)
      results = results.long()
      results = results.squeeze_()
      loss = criterion(outputs, results)

      # Backward and optimise
      optimiser.zero_grad()
      loss.backward()
      optimiser.step()

      # Update loss
      train_loss += loss.item()

    loss_values.append(train_loss / batch_size )
  print('Finished Training')

  return net

And when I call the function:

net = train_neural_network(data_train_X = data_train_X, data_train_Y = data_train_Y, batch_size = batch_size, learning_rate = learning_rate, dropout = 0.1)

The error is the following:

net = train_neural_network(data_train_X = data_train_X, data_train_Y = data_train_Y, batch_size = batch_size, learning_rate = learning_rate, dropout = 0.1)

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/dropout.py in __init__(self, p, inplace)
      8     def __init__(self, p=0.5, inplace=False):
      9         super(_DropoutNd, self).__init__()
---> 10         if p < 0 or p > 1:
     11             raise ValueError("dropout probability has to be between 0 and 1, "
     12                              "but got {}".format(p))

RuntimeError: bool value of Tensor with more than one value is ambiguous

Why do you think there is an error?

Before putting the droprate, everything was working. Additional points for you if you know how to
implement a bias inside my network! For example, on the hidden layer. I can't find any example online.


title: Having trouble writing training loop in pytorch

body:
I load data from a CSV FILE OF 20+6 Columns(Features and Labels). Im trying to run my data through Convolutional Neural Network in pytorch. I get error saying it expects a 3D input and Im giving it 1D input. I am using Conv1d .

import torch
import torch.nn as nn
import torch.nn.functional as F
import numpy as np
import pandas as pd
from torch.utils.data import Dataset,DataLoader
from sklearn.model_selection import train_test_split

#Read Data
data=pd.read_csv('Data.csv')
Features=data[data.columns[0:20]]
Labels=data[data.columns[20:]]
#Split Data
X_train, X_test, y_train, y_test = train_test_split( Features, Labels, test_size=0.33, shuffle=True)
#Create Tensors
train_in=torch.tensor(X_train.values)
train_out=torch.tensor(y_train.values)
test_in=torch.tensor(X_test.values)
test_out=torch.tensor(y_test.values)
#Model CNN
class CNN(nn.Module):
        def __init__(self):
                super(CNN,self).__init__()
                self.layer1 = nn.Sequential(
                                nn.Conv1d(20,40,kernel_size=5,stride=1,padding=2),
                                nn.ReLU(),
                                nn.MaxPool1d(kernel_size=2,stride=2)
                                )
                self.layer2 = nn.Sequential(
                                nn.Conv1d(40,60,kernel_size=5,stride=1,padding=2),
                                nn.ReLU(),
                                nn.MaxPool1d(kernel_size=2,stride=2)
                                )
                self.drop_out = nn.Dropout()
                self.fc1 = nn.Linear(60,30)
                self.fc2 = nn.Linear(30,15)
                self.fc3 = nn.Linear(15,6)

        def forward(self,x):
                out=self.layer1(x)
                out=self.layer2(out)
                out=self.drop_out(out)
                out=self.fc1(out)
                out=self.fc2(out)
                out=self.fc3(out)
                return out

Epochs=10
N_labels=len(Labels.columns)
N_features=len(Features.columns)
batch_size=100
learning_rate=0.001
#TRAIN MODEL
model = CNN()
#LOSS AND OPTIMIZER
criterion = torch.nn.SmoothL1Loss()
optimizer = torch.optim.Adam(model.parameters(),lr=learning_rate)
#TRAIN MODEL
model.train()
idx=0
for i in train_in:
        y=model(i)
        loss=criterion(y,train_out[idx])
        idx+=1
        loss.backward()
        optimizer.step()

How do I write the Training and Eval loop? All the examples I see on the internet all use images and they also use DataLoader.


title: segfault when using xarray, only when 'import torch'

body:
I get a segfault when I open a file using xarray, but only if I import torch (imagine how long it took me to figure this out):

import xarray as xr
import torch

xr.open_dataset('my/file.nc')

----

[1651963ee602:10863] *** Process received signal ***
[1651963ee602:10863] Signal: Segmentation fault (11)
[1651963ee602:10863] Signal code: Address not mapped (1)
[1651963ee602:10863] Failing at address: 0x440000e9
[1651963ee602:10863] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7f26f1e11890]
[1651963ee602:10863] [ 1] /usr/local/mpi/lib/libmpi.so.40(PMPI_Comm_set_errhandler+0x41)[0x7f26dc349691]
[1651963ee602:10863] [ 2] /opt/conda/lib/python3.6/site-packages/mpi4py/MPI.cpython-36m-x86_64-linux-gnu.so(+0x82370)[0x7f26cadc2370]
[1651963ee602:10863] [ 3] /opt/conda/lib/python3.6/site-packages/mpi4py/MPI.cpython-36m-x86_64-linux-gnu.so(+0x2dd79)[0x7f26cad6dd79]
[1651963ee602:10863] [ 4] /opt/conda/bin/python /opt/conda/bin/ipython(PyModule_ExecDef+0x7a)[0x55cad2262cca]
[1651963ee602:10863] [ 5] /opt/conda/bin/python /opt/conda/bin/ipython(+0x1ecd38)[0x55cad2262d38]
[1651963ee602:10863] [ 6] /opt/conda/bin/python /opt/conda/bin/ipython(PyCFunction_Call+0xf4)[0x55cad218cb54]
[1651963ee602:10863] [ 7] /opt/conda/bin/python /opt/conda/bin/ipython(_PyEval_EvalFrameDefault+0x539a)[0x55cad224183a]
[1651963ee602:10863] [ 8] /opt/conda/bin/python /opt/conda/bin/ipython(+0x170cf6)[0x55cad21e6cf6]
[1651963ee602:10863] [ 9] /opt/conda/bin/python /opt/conda/bin/ipython(+0x171c91)[0x55cad21e7c91]
[1651963ee602:10863] [10] /opt/conda/bin/python /opt/conda/bin/ipython(+0x1a1635)[0x55cad2217635]
[1651963ee602:10863] [11] /opt/conda/bin/python /opt/conda/bin/ipython(_PyEval_EvalFrameDefault+0x30a)[0x55cad223c7aa]
[1651963ee602:10863] [12] /opt/conda/bin/python /opt/conda/bin/ipython(+0x171a5b)[0x55cad21e7a5b]
[1651963ee602:10863] [13] /opt/conda/bin/python /opt/conda/bin/ipython(+0x1a1635)[0x55cad2217635]
[1651963ee602:10863] [14] /opt/conda/bin/python /opt/conda/bin/ipython(_PyEval_EvalFrameDefault+0x30a)[0x55cad223c7aa]
[1651963ee602:10863] [15] /opt/conda/bin/python /opt/conda/bin/ipython(+0x171a5b)[0x55cad21e7a5b]
[1651963ee602:10863] [16] /opt/conda/bin/python /opt/conda/bin/ipython(+0x1a1635)[0x55cad2217635]
[1651963ee602:10863] [17] /opt/conda/bin/python /opt/conda/bin/ipython(_PyEval_EvalFrameDefault+0x30a)[0x55cad223c7aa]
[1651963ee602:10863] [18] /opt/conda/bin/python /opt/conda/bin/ipython(+0x171a5b)[0x55cad21e7a5b]
[1651963ee602:10863] [19] /opt/conda/bin/python /opt/conda/bin/ipython(+0x1a1635)[0x55cad2217635]
[1651963ee602:10863] [20] /opt/conda/bin/python /opt/conda/bin/ipython(_PyEval_EvalFrameDefault+0x30a)[0x55cad223c7aa]
[1651963ee602:10863] [21] /opt/conda/bin/python /opt/conda/bin/ipython(_PyFunction_FastCallDict+0x11b)[0x55cad21e80cb]
[1651963ee602:10863] [22] /opt/conda/bin/python /opt/conda/bin/ipython(_PyObject_FastCallDict+0x26f)[0x55cad2189f0f]
[1651963ee602:10863] [23] /opt/conda/bin/python /opt/conda/bin/ipython(_PyObject_CallMethodIdObjArgs+0x100)[0x55cad21b3be0]
[1651963ee602:10863] [24] /opt/conda/bin/python /opt/conda/bin/ipython(PyImport_ImportModuleLevelObject+0x280)[0x55cad21804b0]
[1651963ee602:10863] [25] /opt/conda/bin/python /opt/conda/bin/ipython(+0x1abbda)[0x55cad2221bda]
[1651963ee602:10863] [26] /opt/conda/bin/python /opt/conda/bin/ipython(PyCFunction_Call+0xc6)[0x55cad218cb26]
[1651963ee602:10863] [27] /opt/conda/bin/python /opt/conda/bin/ipython(PyObject_Call+0x3e)[0x55cad218994e]
[1651963ee602:10863] [28] /opt/conda/bin/python /opt/conda/bin/ipython(PyObject_CallFunction+0xf4)[0x55cad21e6854]
[1651963ee602:10863] [29] /opt/conda/bin/python /opt/conda/bin/ipython(PyImport_Import+0x9e)[0x55cad2180a9e]
[1651963ee602:10863] *** End of error message ***
[1]    10863 segmentation fault (core dumped)  ipython

This works:

import xarray as xr

xr.open_dataset('my/file.nc')

This works as well:

import xarray as xr
import torch

print(torch.ones(1))

OS       ubuntu 18.04.3
python   3.6.7
xarray   0.15.0
torch    1.3.0a0+24ae9b5

I appreciate any help.


title: How to convert a trained model to a function?

body:
I have finished training a model, what I want to do next is hand over my model to my colleages who know nothing about deep learning, I just want to give them a function that they can run without installing tensorflow or Python on their machines, may be just python (ideally I would love it to run on Matlab). Is this doable? How can I abstract away all or codify everything from them?

I read about deployment of models but it's all about servers and stuff, this is not what I want.
PS: assume a TF model for now.


title: Resize torch tensor channels

body:
I have a torch tensor with 3 channels, and I want it to be 1 channel (all other dimensions should stay the same).
So if my current dimensions are torch.Size([6, 3, 512, 512]) I want it to be torch.Size([6, 1, 512, 512])

How can I do that?


title: How to evaluate a single image in PyTorch model?

body:
I used this code to train a model:

def train(model, epochs):
    for epoch in range(epochs):
        for idx, batch in enumerate(train_loader):
            x, bndbox = batch    # unpack batch
            pred_bndbox = model(x)# forward pass
            #print('label:', bndbox, 'prediction:', pred_bndbox)
            loss = criterion(pred_bndbox, bndbox)     # compute loss for this batch
            optimiser.zero_grad()# zero gradients of optimiser
            loss.backward()     # backward pass (find rate of change of loss with respect to model parameters)
            optimiser.step()# take optimisation step
            print('Epoch:', epoch, 'Batch:', idx, 'Loss:', loss.item())
            writer.add_scalar('DETECTION Loss/Train', loss, epoch*len(train_loader) + idx)    # write loss to a graph

train(cnn, epochs)

torch.save(cnn.state_dict(), str(time.time()))# save model

def visualise(model, n):
    model.eval()
    for idx, batch in enumerate(test_loader):
        x, y = batch
        pred_bndbox = model(x)
        S40dataset.show(batch, pred_bndbox=pred_bndbox)
        if idx == n:
            break

How do I evaluate the model prediction on a single image to check the operation of the neural network?


title: In pytorch, is there a built-in method to extract rows with given indexes?

body:
Suppose I have a torch tensor

import torch
a = torch.tensor([[1,2,3],
                  [4,5,6],
                  [7,8,9]])

and a list

b = [0,2]

Is there a built-in method to extract the rows 0 and 2 and put them in a new tensor:

tensor([[1,2,3],
        [7,8,9]])

In particular, is there a function that look likes this:

extract_rows(a,b) -> c

where c contains desired rows. Sure, this can done by a for loop, but a built-in method is in general faster.

Note that the example is only an example, there could be dozens of indexes in the list, and hundreds of rows in the tensor.


title: how to write a crossover function for genetic algorithm

body:
def crossover(parents):
    result = copy.deepcopy(parents[0].brain)
    for param in result.parameters():
        print((param.shape[0] / 2))
        if len(param.shape) == 2:  # weights of linear layer
            for i0 in range(param.shape[0]):
                for i1 in range(param.shape[1]):
                    if i0 < (param.shape[0] / 2) and i1 < (param.shape[1]):
                        param[i0][i1] = [param[i0][i1] for param in parents[0].brain.parameters() if
                                         len(param.shape) == 2]
                        # print("x is", x)
                        # print(f"param{i0}{i1}", param[i0][i1])
                        # param[i0][i1] = 2
                    else:
                        param[i0][i1] = [param[i0][i1] for param in parents[1].brain.parameters() if
                                         len(param.shape) == 2]

I am using pytorch library and trying to write a crossover function for my project. initially for the crossover result i just copied the brain of the one of the parents for the ease. then I'm cutting it from the middle and the merging the brains of both of the parents in it. The problem is the output of this "param[i0][i1] for param in parents[1].brain.parameters()" comes out to be a list of 3 tensors which shouldn't.
output

x is [tensor(0.0775), tensor(0.0372), tensor(0.2628)]
param07 tensor(0.0775)

how do i fix this?


title: Research Question (Help): RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same

body:
No idea what's happening in my case:

Error message:

Traceback (most recent call last):
  File "plot_parametric_pytorch.py", line 127, in <module>
    ops = opfun(X_train[smpl])
  File "plot_parametric_pytorch.py", line 81, in <lambda>
    opfun = lambda X: model.forward(Variable(torch.from_numpy(X)))
  File "/mnt_home/klee/LBSBGenGapSharpnessResearch/vgg.py", line 43, in forward
    x = self.features(x).to(device)
  File "/home/klee/anaconda3/envs/sharpenv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/klee/anaconda3/envs/sharpenv/lib/python3.7/site-packages/torch/nn/modules/container.py", line 100, in forward
    input = module(input)
  File "/home/klee/anaconda3/envs/sharpenv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/klee/anaconda3/envs/sharpenv/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 349, in forward
    return self._conv_forward(input, self.weight)
  File "/home/klee/anaconda3/envs/sharpenv/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 346, in _conv_forward
    self.padding, self.dilation, self.groups)
RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same

Source code:

[import statements]
cudnn.benchmark = True
(X_train, y_train), (X_test, y_test) = cifar10.load_data()
X_train = X_train.astype('float32')
X_train = np.transpose(X_train, axes=(0, 3, 1, 2))
X_test = X_test.astype('float32')
X_test = np.transpose(X_test, axes=(0, 3, 1, 2))
X_train /= 255
X_test /= 255
device = torch.device('cuda:0')

# This is where you can load any model of your choice.
# I stole PyTorch Vision's VGG network and modified it to work on CIFAR-10.
# You can take this line out and add any other network and the code
# should run just fine.
model = vgg.vgg11_bn()
model.to(device)

# Forward pass
opfun = lambda X: model.forward(Variable(torch.from_numpy(X))) <------

# Forward pass through the network given the input
predsfun = lambda op: np.argmax(op.data.numpy(), 1)

# Do the forward pass, then compute the accuracy
accfun = lambda op, y: np.mean(np.equal(predsfun(op), y.squeeze()))*100

# Initial point
x0 = deepcopy(model.state_dict())

# Number of epochs to train for
# Choose a large value since LB training needs higher values
# Changed from 150 to 30
nb_epochs = 30
batch_range = [25, 40, 50, 64, 80, 128, 256, 512, 625, 1024, 1250, 1750, 2048, 2500, 3125, 4096, 5000]

# parametric plot (i.e., don't train the network)
hotstart = False

if not hotstart:
    for batch_size in batch_range:
        optimizer = torch.optim.Adam(model.parameters())
        model.load_state_dict(x0)
        model.to(device)
        average_loss_over_epoch = '-'
        print('Optimizing the network with batch size %d' % batch_size)
        np.random.seed(1337) #So that both networks see same sequence of batches
        for e in range(nb_epochs):
            model.eval()
            print('Epoch:', e, ' of ', nb_epochs, 'Average loss:', average_loss_over_epoch)
            average_loss_over_epoch = 0
            # Checkpoint the model every epoch
            torch.save(model.state_dict(), "./models/30EpochC3ExperimentBatchSize" + str(batch_size) + ".pth")
            array = np.random.permutation(range(X_train.shape[0]))
            slices = X_train.shape[0] // batch_size
            beginning = 0
            end = 1

            # Training loop!
            for _ in range(slices):
                start_index = batch_size * beginning
                end_index = batch_size * end
                smpl = array[start_index:end_index]
                model.train()
                optimizer.zero_grad()
                ops = opfun(X_train[smpl]) <-----
                tgts = Variable(torch.from_numpy(y_train[smpl]).long().squeeze())
                loss_fn = F.nll_loss(ops, tgts)
                average_loss_over_epoch += loss_fn.data.numpy() / fractions_of_dataset
                loss_fn.backward()
                optimizer.step()
                beginning += 1
                end += 1

And here is where the model is constructed (and the forward method) [vgg.py]

import torch
import torch.nn as nn
F = nn.functional
import torch.utils.model_zoo as model_zoo
import math

__all__ = [
    'VGG', 'vgg11', 'vgg11_bn', 'vgg13', 'vgg13_bn', 'vgg16', 'vgg16_bn',
    'vgg19_bn', 'vgg19',
]

model_urls = {
    'vgg11': 'https://s3.amazonaws.com/pytorch/models/vgg11-fb7e83b2.pth',
    'vgg13': 'https://s3.amazonaws.com/pytorch/models/vgg13-58758d87.pth',
    'vgg16': 'https://s3.amazonaws.com/pytorch/models/vgg16-82412952.pth',
    'vgg19': 'https://s3.amazonaws.com/pytorch/models/vgg19-341d7465.pth',
}

class VGG(nn.Module):
    def __init__(self, features):
        super(VGG, self).__init__()
        self.features = features.cuda()
        self.classifier = nn.Sequential(
            nn.Dropout(),
            nn.Linear(512, 4096),
            nn.ReLU(True),
            nn.Dropout(),
            nn.Linear(4096, 4096),
            nn.ReLU(True),
            nn.Linear(4096, 10),
        )
        self._initialize_weights()

    def forward(self, x):
        device = torch.device('cuda:0')
        x.cuda()
        x.to(device)
        x = self.features(x)
        x = x.view(x.size(0), -1)
        x = self.classifier(x)
        return F.log_softmax(x)

My full source code is here https://github.com/kristyelee/LBSBGenGapSharpnessResearch/blob/master/plot_parametric_pytorch.py
https://github.com/kristyelee/LBSBGenGapSharpnessResearch/blob/master/vgg.py

I read this for some information, RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same but I'm not sure how to put the input tensors on the GPU.. I kind of tried to do this with x.cuda() not sure if this is right or not though. I really want to put my model on the GPU to train faster (because I'm currently just using the CPU otherwise, which is slow)


title: Bert forward reports error on GPU; but runs fine on CPU

body:
I would like to encode articles using Bert. My code runs fine on CPU, but failed on GPU. The GPU is a on a remote server.
My strategy is to tokenize the articles using CPU first. Then move both model and tokens to GPU. I am not sure if this is a good approach. Any suggestion of improvements are welcomed. Thank you in advance.
def assign_gpu(token):
    token_tensor = token['input_ids'].to('cuda')
    token_typeid = token['token_type_ids'].to('cuda')
    attention_mask = token['attention_mask'].to('cuda')
    output = {'input_ids': token_tensor,
              'token_type_ids': token_typeid,
              'attention_mask': attention_mask}
    return output

bs = 16
data_dl = DataLoader(PatentDataset(df), batch_size=bs, shuffle=False)
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased/')
model = BertModel.from_pretrained('bert-base-uncased/')
inputs_list = []
for a in data_dl:
     inputs_list.append(tokenizer.batch_encode_plus(a, pad_to_max_length=True, return_tensors='pt'))

# GPU part
model.cuda()
model.eval()
out_list = []
with torch.no_grad():
    for i, inputs in enumerate(inputs_list):
        inputs = assign_gpu(inputs)
        output = model(**inputs)[0][:, 0, :]
        out_list.append(output)

The error message is
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-3-f6979d077f60> in <module>
      6     for i, inputs in enumerate(inputs_list):
      7         inputs = assign_gpu(inputs)
----> 8         output = model(**inputs)[0][:, 0, :]
      9         out_list.append(output)

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--> 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

~/volta_pypkg/lib/python3.6/site-packages/transformers/modeling_bert.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, encoder_hidden_states, encoder_attention_mask, output_attentions, output_hidden_states)
    751
    752         embedding_output = self.embeddings(
--> 753             input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds
    754         )
    755         encoder_outputs = self.encoder(

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--> 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

~/volta_pypkg/lib/python3.6/site-packages/transformers/modeling_bert.py in forward(self, input_ids, token_type_ids, position_ids, inputs_embeds)
    180         token_type_embeddings = self.token_type_embeddings(token_type_ids)
    181
--> 182         embeddings = inputs_embeds + position_embeddings + token_type_embeddings
    183         embeddings = self.LayerNorm(embeddings)
    184         embeddings = self.dropout(embeddings)

RuntimeError: CUDA error: device-side assert triggered

Another issue is that if I rerun the GPU for loop on a new cell in the interactive Jupyter notebook, it reports error on the assign_gpu part. I am not sure if it's caused by my remote server or some error in my code.
inputs = assign_gpu(inputs_list[0])
output = model(**inputs)
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-5-ec07b46d1a63> in <module>
----> 1 inputs = assign_gpu(inputs_list[0])
      2 output = model(**inputs)

<ipython-input-1-cb4bdd793d7f> in assign_gpu(token)
     29
     30 def assign_gpu(token):
---> 31     token_tensor = token['input_ids'].to('cuda')
     32     token_typeid = token['token_type_ids'].to('cuda')
     33     attention_mask = token['attention_mask'].to('cuda')

RuntimeError: CUDA error: device-side assert triggered


title: intergation of flask for ML into a meteor.js app

body:
I have build a simple pure react and meteor web app. I am trying to connect a flask API to my meteor.js app for the machine learning component of my application. I have seen examples for pure react front end but cant get the same logic to work for meteor.
what I did is:

make a flask app and return the prediction results to localhost:5000 as a python dictionary e.g.

{'class': 'bird', 'confidence':'0.8932'}

Set up a proxy in my meteor app in package.json, I have meteor app running at localhost:3000:

"proxy":"http://127.0.0.1:5000/"

finally, this is where I am confused, I have a bunch of components in my home page, I am not sure if I have to render the flask results in a component or page, nor how to do that. What I tried Is to render the results in one of the components using the useEffect, useState functions.

I get an error that says something like I can't use this funtionality.

function App() {
  const [predictedClass, setPredictedClass] = useState(0);

  useEffect(() => {
    fetch('/prediction').then(res => res.json()).then(data => {
      setPredictedClass(data.class);
    });
  }, []);


title: What can I use in Pytorch to remplace caffe's weight filler

body:
I'm writing a pytorch model based on the caffe model below. Do you know how I can write weight filler and bias filler in pytorch ?
layer {
  name: "conv3"
  type: "Convolution"
  bottom: "pool2"
  top: "conv3"
  param {
    lr_mult: 1
    decay_mult: 1
  }
  convolution_param {
    num_output: 128
    pad_h: 0
    pad_w: 1
    kernel_h: 1
    kernel_w: 3
    stride: 1
    weight_filler {
      type: "gaussian"
      std: 0.1
    }
    bias_filler {
      type: "constant"
      value: 0
    }
  }
}

Thank you


title: Cause of Exploding NLLLoss

body:
I have been trying to make Transformer based language model, for the loss function Negative Log-likelihood is implemented. For some reason, after a few iterations, there is a steep increase in reconstruction loss.
Here is the log generated by the training program
root - WARNING - Loss: 203.81146240234375
root - WARNING - Loss: 124.32596588134766
root - WARNING - Loss: 62.59440612792969
root - WARNING - Loss: 59.84109115600586
root - WARNING - Loss: 59.247005462646484
root - WARNING - Loss: 48.832725524902344
root - WARNING - Loss: 57.592288970947266
root - WARNING - Loss: 50.18443298339844
root - WARNING - Loss: 46.474849700927734
root - WARNING - Loss: 52.12908172607422
root - WARNING - Loss: 50.090736389160156
root - WARNING - Loss: 66.04253387451172
root - WARNING - Loss: 49.094024658203125
root - WARNING - Loss: 36.69044494628906
root - WARNING - Loss: 48.54591369628906
root - WARNING - Loss: 60.71137237548828
root - WARNING - Loss: 40.35478591918945
root - WARNING - Loss: 49.070556640625
root - WARNING - Loss: 54.33742141723633
root - WARNING - Loss: 47.14014434814453
root - WARNING - Loss: 55.043060302734375
root - WARNING - Loss: 47.63726043701172
root - WARNING - Loss: 46.314571380615234
root - WARNING - Loss: 41.330291748046875
root - WARNING - Loss: 48.85242462158203
root - WARNING - Loss: 50.59345245361328
root - WARNING - Loss: 48.508975982666016
root - WARNING - Loss: 43.35681915283203
root - WARNING - Loss: 45.875431060791016
root - WARNING - Loss: 51.701438903808594
root - WARNING - Loss: 39.1783561706543
root - WARNING - Loss: 30.14274024963379
root - WARNING - Loss: 44.33928680419922
root - WARNING - Loss: 40.88005447387695
root - WARNING - Loss: 62.682804107666016
root - WARNING - Loss: 45.18329620361328
root - WARNING - Loss: 39.7137451171875
root - WARNING - Loss: 47.31813049316406
root - WARNING - Loss: 50.755348205566406
root - WARNING - Loss: 40.52918243408203
root - WARNING - Loss: 49.48160934448242
root - WARNING - Loss: 58.29778289794922
root - WARNING - Loss: 45.660675048828125
root - WARNING - Loss: 55.13115692138672
root - WARNING - Loss: 50.72150421142578
root - WARNING - Loss: 33.377098083496094
root - WARNING - Loss: 48.404151916503906
root - WARNING - Loss: 60.24494934082031
root - WARNING - Loss: 46.290470123291016
root - WARNING - Loss: 9.493173539216099e+24

After the weight updation of the last iteration, the loss becomes nan from that point forth. What are the possible reasons for this to occur ?


title: multiplication of all the combination of elements in certain dimension

body:
I'm trying to do multiplication of all the combination of elements in a certain dimension.
For example,
using for loops it would be
for a,i in enumerate(A):
    for b,j in enumerate(B):
        c[i][j] = a[1]*b[0]

not using for loop would be
A[:,None,1]*B[:,0]

this worked fine for 2-dimensions(A.shape=(4,2),B.shape=(4,2)).
However, When I am expanding above procedures to 3-dimensions(A.shape=(2,4,2),B.shape=(2,4,2)).
I tried with
A[:,:,1]*B[:,:,None,0]

but gives me an error.
How can I do this without using for loops???
For 2-dimensional tensors this is what I get
a = torch.FloatTensor([[0,1],[0,2],[0,3],[0,4]])
b = torch.FloatTensor([[1,0],[2,0],[3,0],[4,0]])
c = a[:,None,0] * b[:,1]

print(c)

tensor([[ 1.,  2.,  3.,  4.],
        [ 2.,  4.,  6.,  8.],
        [ 3.,  6.,  9., 12.],
        [ 4.,  8., 12., 16.]])

but in the case of
A = torch.cat([a.unsqueeze(0),a.unsqueeze(0)],dim=0)
B = torch.cat([b.unsqueeze(0),b.unsqueeze(0)],dim=0)

C = A[:,:,None,0] * B[:,:,1] << RuntimeError

RuntimeError: The size of tensor a (4) must match the size of tensor b (2) at non-singleton dimension 1

what I want is
tensor([[[ 1.,  2.,  3.,  4.],
         [ 2.,  4.,  6.,  8.],
         [ 3.,  6.,  9., 12.],
         [ 4.,  8., 12., 16.]],

        [[ 1.,  2.,  3.,  4.],
         [ 2.,  4.,  6.,  8.],
         [ 3.,  6.,  9., 12.],
         [ 4.,  8., 12., 16.]]])

I cannot think of a way to do this without using for loop in dimension 0.


title: Linear layer input size Pytorch

body:
I'm working on an assignement with 1D signals and I have trouble finding the right input size for the linear layer (XXX). My signals have different lengths and are padded in a batch. I read that the linear layear should always have the same input size (XXX) but I'm not sure how to find it when each batch has a different length. Does anybody have an advice?
Thanks
class NeuralNet(nn.Module):
    def __init__(self):
        super(NeuralNet, self).__init__()

        self.features = nn.Sequential(nn.Conv1d(in_channels = 1, out_channels = 128, kernel_size = 7, stride = 3),
                                      nn.BatchNorm1d(128),
                                      nn.ReLU(),
                                      nn.MaxPool1d(2, 3),
                                      nn.Conv1d(128, 32, 5, 1),
                                      nn.BatchNorm1d(32),
                                      nn.ReLU(),
                                      nn.MaxPool1d(2, 2),
                                      nn.Conv1d(32, 32, 5, 1),
                                      nn.ReLU(),
                                      nn.Conv1d(32, 128, 3, 2),
                                      nn.ReLU(),
                                      nn.MaxPool1d(2, 2),
                                      nn.Conv1d(128, 256, 7, 1),
                                      nn.ReLU(),
                                      nn.MaxPool1d(2, 2),
                                      nn.Conv1d(256, 512, 3, 1),
                                      nn.ReLU(),
                                      nn.Conv1d(512, 128, 3, 1),
                                      nn.ReLU()
                                      )

        self.classifier = nn.Sequential(nn.Linear(XXX, 512),
                                        nn.ReLU(),
                                        nn.Dropout(p = 0.1),
                                        nn.Linear(512,2)
                                        )

    def forward(self, x):

        x = self.features(x)
        x = torch.flatten(x)
        x = self.classifier(x)

        return x


title: Encounter the error one of the variables needed for gradient computation has been modified by an inplace operation

body:
I have encountered the following error. This is a different rnn structure. I have implemented to use in graph convolution. The problem is that the hidden is updated in-place operation. However, I have to update its value in each forward call. How can I do that? Thanks in advance.
    RuntimeError                              Traceback (most recent call last)
<ipython-input-110-b4425651d544> in <module>()
      8     out = model(x[i])
      9     loss = mael(out, x[i+1])
---> 10     loss.backward(retain_graph=True)
     11     optimizer.step()
     12     print(loss.item())

1 frames
/usr/local/lib/python3.6/dist-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
    130     Variable._execution_engine.run_backward(
    131         tensors, grad_tensors_, retain_graph, create_graph,
--> 132         allow_unreachable=True)  # allow_unreachable flag
    133
    134

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [1, 100]], which is output 0 of SelectBackward, is at version 1; expected version 0 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

This is a different rnn structure. I have implemented to use in graph convolution. The problem is that the hidden is updated in-place operation. However, I have to update its value in each forward call. How can I do that? Thanks in advance.
class RNN(nn.Module):
    def __init__(self, input_dim, hidden_dim):
        super(RNN,self).__init__()
        self.input_dim = input_dim
        self.hidden_dim = hidden_dim
        self.weight = Parameter(torch.rand(10,input_dim,hidden_dim, requires_grad=True))
        self.weight_h = Parameter(torch.rand(10,input_dim,hidden_dim, requires_grad=True))
        self.bias = Parameter(torch.rand(10,input_dim,hidden_dim, requires_grad=True))
        self.hidden = torch.rand(10,input_dim, hidden_dim)
        self.weight_2 = Parameter(torch.rand(10,input_dim,hidden_dim,requires_grad=True))
        self.weight_h_2 = Parameter(torch.rand(10,hidden_dim,hidden_dim, requires_grad=True))
        self.bias_2 = Parameter(torch.rand(10,input_dim,hidden_dim, requires_grad=True))
        self.tanh = Tanh()
        self.iteration = 0
        self.next_layer = False
        self.hidden_init = torch.rand(1,1)

    def set_hidden(self,x):
      y = self.tanh(mm(x, self.weight[self.iteration]) + mm(self.hidden_init, self.weight_h[self.iteration]) + self.bias[self.iteration])
      return y
    def set_hidden_state_layer_2(self, x, hidden):
      y = self.tanh(mm(x, self.weight_2[self.iteration]) + mm(hidden, self.weight_h_2[self.iteration]) + self.bias_2[self.iteration])
      return y

    def forward(self, x):
      try:
        dim_1, dim_2, dim_3 = x.shape
      except:
        x = torch.unsqueeze(x,0)

      if self.iteration == 10:
        self.next_layer = True
        self.iteration = 0
      if self.next_layer:
        self.hidden[self.iteration] = self.set_hidden_state_layer_2(x, self.hidden[self.iteration].clone())
        self.iteration = self.iteration + 1
        return self.hidden[self.iteration - 1]
      else:
        hidden_init = torch.rand(1,1)
        self.hidden[self.iteration] = self.tanh(mm(x, self.weight[self.iteration]) + mm(self.hidden_init, self.weight_h[self.iteration]) + self.bias[self.iteration])
        self.iteration = self.iteration + 1
        return self.hidden[self.iteration - 1]

model = RNN(1,100)
mael = nn.L1Loss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
x = torch.rand(11,1)
x_2 = torch.rand(11,1)
for i in range(10):
    optimizer.zero_grad()
    out = model(x[i])
    loss = mael(out, x[i+1])
    loss.backward(retain_graph=True)
    optimizer.step()
    print(loss.item())


title: PyTorch GPU memory management

body:
In my code, I want to replace values in the tensor given values of some indices are zero, for example
target_mac_out[avail_actions[:, 1:] == 0] = -9999999

But, it returns OOM
RuntimeError: CUDA out of memory. Tried to allocate 166.00 MiB (GPU 0; 10.76 GiB total capacity; 9.45 GiB already allocated; 4.75 MiB free; 9.71 GiB reserved in total by PyTorch)

I think there is no memory allocation because it just visits the tensor of target_mac_out and check the value and replace a new value for some indices.
Am I understanding right?


title: Convert and save PytorchTensor to .mat image on GPU in python

body:
I am a beginner in Pytorch and I am stuck on a question for days. I  want to save a image which is in Pytorch tensor form  as .mat file. I looked but there doesn't seem to be a direct method on converting Pytoch tensors to .mat file. One possible solution which I found was to convert it to numpy array, but since I am using Nvidia GPU, when I try converting Pytorch tensor to numpy array it gives me this error :
fake_images[0] = fake_images[0].numpy()
TypeError: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first
How do I save Pytorch tensor to .mat file while using GPU?


title: Scipy into Pytorch

body:
I am attempting to integrate this scipy function into PyTorch:
https://docs.scipy.org/doc/scipy/reference/generated/scipy.interpolate.UnivariateSpline.html
I’ve been looking at this guide:
https://pytorch.org/tutorials/advanced/numpy_extensions_tutorial.html
This is the code I have written so far:
from scipy.interpolate import UnivariateSpline

import torch
from torch.autograd import Function

class SPLFunction(Function):

    @staticmethod
    def forward(ctx, input, s):
        np_input = input.detach().numpy()
        s = s.detach()

        x, y = np_input[0], np_input[1]

        spl = UnivariateSpline(x, y,s=s)
        result = spl(y)
        ctx.save_for_backward(input, s)

        return torch.as_tensor(result, dtype=input[0].dtype)

    @staticmethod
    def backward(ctx, grad_output):
        input, s = ctx.saved_tensors
        np_input = input.detach().numpy()
        s = s.detach()

        x, y = np_input[0], np_input[1]
        spl = UnivariateSpline(x, y,s=s)
        df = spl.derivative(x)
        df = torch.from_numpy(df*grad_output)

        return(df)

However, this does not seem to be working. I am getting a variety of errors…if anyone has experience integrating scipy functions in to PyTorch, any tips in the right direction would be hugely appreciated!


title: How to load a Pytorch model when the parameters are saved as numpy arrays?

body:
On this GitHub repo, I've downloaded the pretrained model senet50_ft.
I load it like so:
import pickle
f = open('pretrained_models/senet50_ft_weight.pkl', 'rb')
state_dict = pickle.load(f, encoding='latin1')
f.close()

The state is loaded, the Github repos also provides the SENet model Class here.
So I managed to instanciate that model:
model = senet.senet50()

Then I Tried to load the state, but I got an error:
model.load_state_dict(state_dict)

Traceback (most recent call last):
  File "...\module.py", line 982, in _load_from_state_dict
    param.copy_(input_param)
TypeError: copy_(): argument 'other' (position 1) must be Tensor, not numpy.ndarray

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "...\module.py", line 1037, in load_state_dict
    load(self)
  File "...\module.py", line 1035, in load
    load(child, prefix + name + '.')
  File "...\module.py", line 1032, in load
    state_dict, prefix, local_metadata, True, missing_keys, unexpected_keys, error_msgs)
  File "...\module.py", line 988, in _load_from_state_dict
    .format(key, param.size(), input_param.size(), ex.args))
TypeError: 'int' object is not callable

I tried to convert ndarray to Tensor by doing the following:
for key in state_dict.keys():
    state_dict[key] = torch.from_numpy(state_dict[key])

But I got an another error and I think I'm not going anywhere.
I'm new to PyTorch but I suspect that this model was serialized with an old version of PyTorch. Do you know if a solution exists?


title: PyTorch ROCm is out - How to select Radeon GPU as device

body:
since Pytorch released the ROCm version, which enables me to use other gpus than nvidias, how can I select my radeon gpu as device in python?
Obviously, code like device = torch.cuda.is_available or device = torch.device("cuda") is not working. Thanks for any help. :)


title: Fine-tuning model's classifier layer with new label

body:
I would like to fine-tune already fine-tuned BertForSequenceClassification model with new dataset containing just 1 additional label which hasn't been seen by model before.
By that, I would like to add 1 new label to the set of labels that model is currently able of classifying properly.
Moreover, I don't want classifier weights to be randomly initialized, I'd like to keep them intact and just update them accordingly to the dataset examples while increasing the size of classifier layer by 1.
The dataset used for further fine-tuning could look like this:
sentece,label
intent example 1,new_label
intent example 2,new_label
...
intent example 10,new_label

My model's current classifier layer looks like this:
Linear(in_features=768, out_features=135, bias=True)

How could I achieve it?
Is it even a good approach?


title: Pytorch tensor to numpy gives "()" as shape

body:
I have a pytorch tensor

span_end = tensor([[[13]]])

I do the following

span_end = span_end.view(1).squeeze().data.numpy()
            print(type(span_end))
            print(span_end.shape)

This gives me the following output

<class 'numpy.ndarray'>
()

Then later when I try to access the 0th element of span_end I get IndexError because the shape is null somehow. What am I doing wrong here?


title: How to compute product between two sets of features in pytorch using a single loop?

body:
I wan to compute the product between two sets of feature matrices X and Y
of dimensions (H,W,12) each:

Inefficiently I would do:

H = []
for i in range(12):
    for j in range(12):
        h = X[:,:,i]*Y[:,:,j]
        H.append(h)

which will output H of dimension (H,W,144)

How can this be done in pytorch without iterating in two loops?

I have tried used tensordot solutions but cant replicate the behavior.


title: How do I concatenate tensors along a given axis?

body:
import torch

a = torch.Tensor(2,2,2)

b = myfunction(a)

print(a)
>>> [[[1,2],
      [5,6]],
     [[7,8],
      [9,10]]]

print(b)
>>> [[1,2,7,8],
     [5,6,9,10]]

How do I code myfunction to get b from a?

Is there some pytorch functions that transforms a in such way?


title: Is there any build in function for making 2D tensor from 1D tensor using specific calculation?

body:
Hi I'm student who just started for deep learning.

For example, I have 1-D tensor x = [ 1 , 2]. From this one, I hope to make 2D tensor y whose (i,j)th element has value (x[i] - x[j]), i.e y[0,:] = [0  ,  1] , y[1,:]=[ -1 , 0].

Is there built-in function like this in pytorch library?

Thanks.


title: Pytorch copy a neuron in a layer

body:
I am using pytorch 0.3.0. I'm trying to selectively copy a neuron and it's weights within the same layer, then replace the original neuron with an another set of weights. Here's my attempt at that:

reshaped_data2 = data2.unsqueeze(0)
new_layer_data = torch.cat([new_layer.data, reshaped_data2], dim=0)
new_layer_data[i] = data1
new_layer.data.copy_(new_layer_data)

First I unsqueezed data2 to make it a 1*X tensor instead of 0*X.
Then I concatenate my layer's tensor with the reshaped data2 along dimension 0.
I then replace the original data2 located at index i with data1.
Finally, I copy all of that into my layer.

The error I get is:

RuntimeError: inconsistent tensor size, expected tensor [10 x 128] and src [11 x 128] to have the same number of elements, but got 1280 and 1408 elements respectively at /Users/soumith/code/builder/wheel/pytorch-src/torch/lib/TH/generic/THTensorCopy.c:86

If I do a simple assignment instead of copy I get

RuntimeError: The expanded size of the tensor (11) must match the existing size (10) at non-singleton dimension 1. at /Users/soumith/code/builder/wheel/pytorch-src/torch/lib/TH/generic/THTensor.c:309

I understand the error, but what is the right way to go about this?


title: PyTorch Inequality Gradient

body:
I wrote a PyTorch model roughly as follows:

import torch.nn as nn
class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.layer1 = nn.Sequential(nn.Linear(64 * 64, 16), nn.LeakyReLU(0.2))
        self.layer2 = nn.Sequential(nn.Linear(16, 32), nn.LeakyReLU(0.2))
        self.layer3 = nn.Sequential(nn.Linear(32, 64), nn.LeakyReLU(0.2))
        self.layer4 = nn.Sequential(nn.Linear(64, 15), nn.Tanh())
    def forward(self, x):
        return (self.layer4(self.layer3(self.layer2(self.layer1(x)))) < 0).float()

Notice what I want to do: I want forward to return a tensor of 0s and 1s. However, this does not train, probably because the derivative of the inequality is zero.

How can I make a model like this one train, for example, if I want to do image segmentation?


title: TypeError: slice indices must be integers or None or have an __index__ method in zero padding CAGAN

body:
i have an error about TypeError: slice indices must be integers or None or have an index method when running zero padding method in CA GAN (https://github.com/fei-hdu/ca-gan/tree/new_parsing)

This is the code that is error (in data.py line 149)

def zero_padding(img, size0, pad1, pad2):
zero_padding = np.zeros((img.shape[0], size0, size0), dtype=np.float32)
print('img.shape[0] = ', img.shape[0])
pad1 = pad1 / 2
pad2 = pad2 / 2
zero_padding[:, pad1:size0 - pad1, pad2:size0 - pad2] = img
return zero_padding

This is the error

zero_padding[:, pad1:size0 - pad1, pad2:size0 - pad2] = img
TypeError: slice indices must be integers or None or have an index method


title: Why my accuracy values don't change on my train model

body:
When I start to train my model, Loss values decreasing but Accuracy values never change.I don't know why?

# -*- coding: utf-8 -*-

#Libraries
import torch
import torch.nn.functional as F
from torch import autograd, nn
from torch.autograd import Variable
import numpy as np
import matplotlib.pyplot as plt
from torchvision import transforms, datasets
from torch.utils import data

"""
Olivetti face dataset
"""
from sklearn.datasets import fetch_olivetti_faces

# Olivetti dataset download
olivetti = fetch_olivetti_faces()
train = olivetti.images
label = olivetti.target

X = train
Y = label
print("Format for X:", X.shape)
print("Format for Y: ", Y.shape)

print("\nDownload Ok")

"""
Set for train
"""
train_rate = 0.8

X_train = np.zeros([int(train_rate * X.shape[0]),64,64], dtype=float)
Y_train = np.zeros([int(train_rate * X.shape[0])], dtype=int)

X_val = np.zeros([int((1-train_rate) * X.shape[0]+1),64,64], dtype=float)
Y_val = np.zeros([int((1-train_rate) * X.shape[0]+1)], dtype=int)

#Split data for train and validation
for i in range(X.shape[0]):
    ie=0
    iv=0
    if (i%10)/10 >= train_rate:
        X_train[ie] = X[i]
        Y_train[ie] = Y[i]
        ie += 1
    else:
        X_val[iv] = X[i]
        Y_val[iv] = Y[i]
        iv += 1

X_train = X_train.reshape(320,-1,64,64)
X_val = X_val.reshape(80,-1,64,64)

print(Y_train.shape)
X_train = torch.Tensor(X_train)
Y_train = torch.Tensor(Y_train)

X_val = torch.Tensor(X_val)
Y_val = torch.Tensor(Y_val)

batch_size = 20

train_loader = torch.utils.data.DataLoader(X_train,
                                            batch_size=batch_size,
                                            )
val_loader = torch.utils.data.DataLoader(X_val,
                                            batch_size=batch_size,
                                            )

class CNNModule(nn.Module):
    def __init__(self):
        super(CNNModule, self).__init__()
        self.conv1 = nn.Conv2d(1, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 13 * 13, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 40)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 13 * 13)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

def make_train(model,dataset,n_iters,gpu):

    # Organize data
    X_train,Y_train,X_val,Y_val = dataset

    kriter = nn.CrossEntropyLoss()
    optimizer = torch.optim.SGD(model.parameters(),lr=0.01)

    #Arrays to save loss and accuracy
    tl=np.zeros(n_iters)    #For train loss
    ta=np.zeros(n_iters)    #For train accuracy
    vl=np.zeros(n_iters)    #For validation loss
    va=np.zeros(n_iters)    #For validation accuracy

    # Convert labels to long
    Y_train = Y_train.long()
    Y_val = Y_val.long()

    # GPU control
    if gpu:
        X_train,Y_train = X_train.cuda(),Y_train.cuda()
        X_val,Y_val = X_val.cuda(),Y_val.cuda()
        model = model.cuda() # Parameters to GPU!
        print("Using GPU")
    else:
        print("Using CPU")
        # print(X_train.shape)
        # print(Y_train.shape)

    for i in range(n_iters):

        # train forward
        train_out = model.forward(X_train)
        train_loss = kriter(train_out,Y_train)

        # Backward and optimization
        train_loss.backward()
        optimizer.step()
        optimizer.zero_grad()

        # Compute train accuracy
        train_predict = train_out.cpu().detach().argmax(dim=1)
        train_accuracy = (train_predict.cpu().numpy()==Y_train.cpu().numpy()).mean()

        # For validation
        val_out = model.forward(X_val)
        val_loss = kriter(val_out,Y_val)

        # Compute validation accuracy
        val_predict = val_out.cpu().detach().argmax(dim=1)
        val_accuracy = (val_predict.cpu().numpy()==Y_val.cpu().numpy()).mean()

        tl[i] = train_loss.cpu().detach().numpy()
        ta[i] = train_accuracy
        vl[i] = val_loss.cpu().detach().numpy()
        va[i] = val_accuracy

        # Show result each 5 loop
        if i%5==0:
            print("Loop --> ",i)
            print("Train Loss :",train_loss.cpu().detach().numpy())
            print("Train Accuracy :",train_accuracy)
            print("Validation Loss :",val_loss.cpu().detach().numpy())
            print("Validation Accuracy :",val_accuracy)

    model = model.cpu()

    #Print result
    plt.subplot(2,2,1)
    plt.plot(np.arange(n_iters), tl, 'r-')
    plt.subplot(2,2,2)
    plt.plot(np.arange(n_iters), ta, 'b--')

    plt.subplot(2,2,3)
    plt.plot(np.arange(n_iters), vl, 'r-')
    plt.subplot(2,2,4)
    plt.plot(np.arange(n_iters), va, 'b--')

dataset = X_train,Y_train,X_val,Y_val

gpu =  True
gpu = gpu and torch.cuda.is_available()

model = CNNModule()
make_train(model,dataset,100,gpu)

OUTPUT:

Using CPU
Loop -->  0
Train Loss : 3.6302185
Train Accuracy : 0.0
Validation Loss : 3.6171098
Validation Accuracy : 0.0
Loop -->  5
Train Loss : 3.557933
Train Accuracy : 0.996875
Validation Loss : 3.545982
Validation Accuracy : 0.9875
.
.
.
Loop -->  95
Train Loss : 0.04211783
Train Accuracy : 0.996875
Validation Loss : 0.13397054
Validation Accuracy : 0.9875


title: Python and pip versions are different and potentially causing problem

body:
I have both 3.6 and 3.8 python versions installed in my drive and I am using Anaconda for virtual environment with Python 3.6. When I say in cmd prompt:

  >python -V
Python 3.6.10 :: Anaconda, Inc.

However when I look for pip versions it shows

  >pip3 -V
pip 20.1.1 from c:\python38\lib\site-packages\pip (python 3.8)

I'm thinking this the cause of the problem when I use pip install -e . while my virtual environment is active:

ERROR: Could not find a version that satisfies the requirement torch==1.3.1 (from spinup==0.2.0) (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)
ERROR: No matching distribution found for torch==1.3.1 (from spinup==0.2.0)

How can I solve this ?


title: Multiplying matrices of different sizes

body:
I have a matrix full of zeros except for one zone, where I have values other than 0:

i=20
j=30
z = np.zeros((50,50))
while i < 30:
    while j < 40:
        z[j, i] = np.random.rand(1,1)
        j+=1

    j=30
    i+=1

matshow(z)
np.where(z == z.max())

Then I have another matrix, smaller than the first one. The distribution of values in this matrix follows a Gaussian curve:

import numpy as np
from scipy import signal

def gkern(kernlen=21, std=3):
    """Returns a 2D Gaussian kernel array."""
    gkern1d = signal.gaussian(kernlen, std=std).reshape(kernlen, 1)
    gkern2d = np.outer(gkern1d, gkern1d)
    return gkern2d
gkern(21, 3)

What I want is the following:

Identify where the maximum value of the 50x50 matrix is located. On this value, I want to center the maximum value of the smallest matrix, and then do the multiplication between the value of my small matrix and the values of the big matrices that are convered by the smaller matrix.


title: How to attach hooks to ReLUs in Inception V3 from torchvision

body:
I am using Inception v3 from torchvision. I tried to find the ReLUs within the model:

def recursively_find_submodules(model, submodule_type):
    module_list = []
    q = [model]
    while q:
        child = q.pop()
        if isinstance(child, submodule_type):
            module_list.append(child)
        q.extend(list(child.children()))
    return module_list

inception = torch.hub.load('pytorch/vision:v0.6.0', 'inception_v3', pretrained=True)
l = recursively_find_submodules(inception, torch.nn.ReLU) # l is empty!

So the ReLUs are not children of any module within the torch model. Upon closer inspection I found the ReLUs in the source code of torchvision but not as modules. In inception.py I found the following:

class BasicConv2d(nn.Module):

    def __init__(self, in_channels, out_channels, **kwargs):
        super(BasicConv2d, self).__init__()
        self.conv = nn.Conv2d(in_channels, out_channels, bias=False, **kwargs)
        self.bn = nn.BatchNorm2d(out_channels, eps=0.001)

    def forward(self, x):
        x = self.conv(x)
        x = self.bn(x)
        return F.relu(x, inplace=True)

So the BasicConv2d module uses the ReLU function to clamp it's output instead of the module (torch.nn.ReLU). I guess there is no way to hook up to ReLU functions and modify their input / output without modifying the whole model to use ReLU modules or is there a way to do this?


title: Multidimensional tensor product in PyTorch

body: In pytorch I have to tensors of dimensions [K,L,M] and [M,L,N]. I want to perform a standard tensor convolution product of those tensors along the middle two dimensions to obtain a [K,N] tensor. I couldn't find official documentation on how to perform those operations, perhaps it should be better done in some other library and then reconverted to pytorch tensor?


title: Problem of generalization of a model determining image similarity (Siamese Neural Network)

body:
I've trained a Siamese Neural Network to predict image similarity with 5 classes (1=not equal, ..., 5=exactly the same).
The similarity is measured by Cosine Similarity which results in values from -1 to 1. To train the model I've therefore transformed my original scale to these values, e.g. the "not equal" class 1 to -1, class 2 to -0.5, ... and "exactly the same" class 5 to 1.
Regarding the results of the prediction one can see that the predictions of the model has an ascending median with ascending similarity (from 1 to 5). But the scale (-1 to 1) is not fully used as the results are too close to each other. On the other hand the separation of the training data is very good, so I think that the problem can be solved, but right now the generalization on new image pairs which the model hasn't seen before does not yet work quite well.
I think this is a result of overfitting, but I don't know what else I can do to solve this problem.
That's why I ask you if someone has an idea what I could do to improve the separation of the classes.
Here are the configurations of my training model:

The training set is balanced (~500 Image Pairs / Class)
The CNN for the Siamese Neural Network is a trained ResNet34 to the specific images
Distance Function: Cosine Similarity
Optimizer: RMSPROP(lr=0.00001)
Epochs: 500
Batch Size: 32
Loss-Function: SmoothL1Loss
Avoid Overfitting: resize(105,105), randomRotation([-30,30]), ColorJitter(brightness=[0.7,1])
Architecture: cutted last layer from ResNet34 and adding a Linear Layer (see below)

def cutting_model(model):
    head = nn.Sequential(*list(model.children())[0])

    #Select last Layer
    body = nn.Sequential(*list(model.children())[1][:-1])
    model2 = nn.Sequential(head, body)

    return model2

class SiameseNetwork(nn.Module):
    def __init__(self, model):
        super(SiameseNetwork, self).__init__()

        self.cnn1 = cutting_model(model)#, False, 1)
        self.fc1 = nn.Sequential(nn.Linear(512,100))

    def forward_once(self, x):

        # Forward pass
        output = self.cnn1(x)
        output = output.view(output.size()[0], -1)
        output = self.fc1(output)
        return output

    def forward(self, input1, input2):

        output1 = self.forward_once(input1)
        output2 = self.forward_once(input2)
        return output1, output2

Train Method:

def train():
    cos = nn.CosineSimilarity(dim=1, eps=1e-6)

    for epoch in range(0,Config.train_number_epochs):

        for i, data in enumerate(train_dataloader,0):
            img0, img1 , label = data
            img0, img1 , label = img0.cuda(), img1.cuda() , label.cuda()
            optimizer.zero_grad()
            output1, output2 = net(img0,img1)
            cosine_distance = cos(output1, output2)
            label=label.squeeze()
            loss_function = criterion(cosine_distance, label)
            loss_function.backward()
            optimizer.step()

    return net

My current Siamese Neural Network:
SiameseNetwork(
  (cnn1): Sequential(
    (0): Sequential(
      (0): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
      (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (2): ReLU(inplace=True)
      (3): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)
      (4): Sequential(
        (0): BasicBlock(
          (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (relu): ReLU(inplace=True)
          (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
        (1): BasicBlock(
          (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (relu): ReLU(inplace=True)
          (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
        (2): BasicBlock(
          (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (relu): ReLU(inplace=True)
          (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
      )
      (5): Sequential(
        (0): BasicBlock(
          (conv1): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
          (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (relu): ReLU(inplace=True)
          (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (downsample): Sequential(
            (0): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False)
            (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          )
        )
        (1): BasicBlock(
          (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (relu): ReLU(inplace=True)
          (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
        (2): BasicBlock(
          (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (relu): ReLU(inplace=True)
          (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
        (3): BasicBlock(
          (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (relu): ReLU(inplace=True)
          (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
      )
      (6): Sequential(
        (0): BasicBlock(
          (conv1): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
          (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (relu): ReLU(inplace=True)
          (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (downsample): Sequential(
            (0): Conv2d(128, 256, kernel_size=(1, 1), stride=(2, 2), bias=False)
            (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          )
        )
        (1): BasicBlock(
          (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (relu): ReLU(inplace=True)
          (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
        (2): BasicBlock(
          (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (relu): ReLU(inplace=True)
          (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
        (3): BasicBlock(
          (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (relu): ReLU(inplace=True)
          (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
        (4): BasicBlock(
          (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (relu): ReLU(inplace=True)
          (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
        (5): BasicBlock(
          (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (relu): ReLU(inplace=True)
          (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
      )
      (7): Sequential(
        (0): BasicBlock(
          (conv1): Conv2d(256, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
          (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (relu): ReLU(inplace=True)
          (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (downsample): Sequential(
            (0): Conv2d(256, 512, kernel_size=(1, 1), stride=(2, 2), bias=False)
            (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          )
        )
        (1): BasicBlock(
          (conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (relu): ReLU(inplace=True)
          (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
        (2): BasicBlock(
          (conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (relu): ReLU(inplace=True)
          (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
      )
    )
    (1): Sequential(
      (0): AdaptiveConcatPool2d(
        (ap): AdaptiveAvgPool2d(output_size=1)
        (mp): AdaptiveMaxPool2d(output_size=1)
      )
      (1): Flatten()
      (2): BatchNorm1d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (3): Dropout(p=0.25, inplace=False)
      (4): Linear(in_features=1024, out_features=512, bias=True)
      (5): ReLU(inplace=True)
      (6): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (7): Dropout(p=0.5, inplace=False)
    )
  )
  (fc1): Sequential(
    (0): Linear(in_features=512, out_features=100, bias=True)
  )
)

I've tried:

different Optimizers (Adam, Eve)
different LearningRates (0.00025,0.0001,0.000075,0.000005, 0.0000025, 0.000001)
instead of CosineSimilarity (range -1:1) I've tried Cosine Distance (range 0:2)
different Epochs (200-3000)
different Batch-sizes (32,64)
different Loss-Function (L1Loss, L2Loss, SmoothL1Loss)
additional Data Augmentation (RandomPerspective, RandomHorizontalFlip(p=0.2))
different CNNs (AlexNet, ResNet101)
different Outputdimensions of the cutted CNN (1,5,10,25,75,100,250,500)

Thank you very much in advance for your help!