title: I tried to divide resnet into two parts using pytorch children(), but it doesn't work

body:
Here is a simple example. I tried to divide a network (Resnet50) into two parts: head and tail using children. Conceptually, this should work but it doesn't. Why is it?
import torch
import torch.nn as nn
from torchvision.models import resnet50

head = nn.Sequential(*list(resnet.children())[:-2])
tail = nn.Sequential(*list(resnet.children())[-2:])
x = torch.zeros(1, 3, 160, 160)

resnet(x).shape      # torch.Size([1, 1000])
head(x).shape        # torch.Size([1, 2048, 5, 5])
tail(head(x)).shape  # Error: RuntimeError: size mismatch, m1: [2048 x 1], m2: [2048 x 1000] at /pytorch/aten/src/TH/generic/THTensorMath.cpp:136

For information, the tail is nothing but
Sequential(
  (0): AdaptiveAvgPool2d(output_size=(1, 1))
  (1): Linear(in_features=2048, out_features=1000, bias=True)
)

So I actually know that if I can do like this. But then, why the reshaping function (view) is not in the children?
pool =resnet._modules['avgpool']
fc = resnet._modules['fc']
fc(pool(head(x)).view(1, -1))


title: how i can get reward in bank heist atari game?

body:
I'm struggling a bit with why my agent is not getting any rewards in the Atari game "bank heist". After each bank robbery when I render the environment I monitor the received award but when I run code below I'm not getting any reward.
What's more interesting is that if I run it on another environments - like taxi I get rewards but when I try it on bank heist I get nothing.
My code is below:
import gym
import torch

import matplotlib.pyplot as plt

#env = gym.make('Taxi-v3')
env = gym.make('BankHeist-ram-v0')

plt.style.use('ggplot')

number_of_states = env.observation_space.shape[0]
print(number_of_states)
number_of_actions = env.action_space.n
print(number_of_actions)
#number_of_states = env.observation_space.n
#number_of_actions = env.action_space.n

gamma = 0.9

egreedy = 0.1

Q = torch.zeros([number_of_states, number_of_actions])
print(Q)
num_episodes = 1000

steps_total = []
rewards_total = []

for i_episode in range(num_episodes):

    state = env.reset()
    step = 0

    while True:

    step += 1

   # env.render()

    random_for_egreedy = torch.rand(1)[0]

    if random_for_egreedy > egreedy:
        random_values = Q[state] + torch.rand(1,number_of_actions) / 1000
        action = torch.max(random_values,1)[1][0]
        action = action.item()
    else:
        action = env.action_space.sample()

    new_state, reward, done, info = env.step(action)

    Q[state, action] = reward + gamma * torch.max(Q[new_state])

    state = new_state

    if done:
        steps_total.append(step)
        rewards_total.append(reward)
        print("Episode finished after %i steps" % step )
        print("Episode achived %i rewards" % reward )
        break


title: How can I limit the range of parameters in pytorch?

body: So normally in pytorch, there is no strict limit to the parameters in models, but what if I wanted them to stay in the range [0,1]? Is there a way to block the update of parameters to outside that range?


title: How to use pre-trained FastText embeddings with existing Seq2Seq model?

body:
I'm new in NLP and I am trying to understand how to use pre-trained word embeddings like fastText with the existing Seq2Seq model. The Seq2Seq model I'm working with is the following. The encoder is simple and the decoder is Pointer Generator Network with CRF on the top. Both of them use an embedding layer.
The question: If I have my own dataset & vocab, how do I use both my own vocab and the one from the fastText? Do I have to use fastText weights in both the encoder and decoder?


title: Can I disable CUDA temporarily in PyTorch?

body:
I am trying to determine if the expense of a GPU is worth it for a particular task. I currently have PyTorch code running on a GPU enabled machine. I would love to be able to disable the GPU for a single run to do a direct comparison (everything else equal) of the benefits of the GPU to the task.
Is there anyway to tell PyTorch to ignore the GPU or to tell the OS to pretend that it doesn't have one? I'm running this on Ubuntu 20.04.1 LTS, using python 3.8.5 and torch 1.7.1.
UPDATE: This is code I inherited so I cannot speak to its efficiency but there are many places in the code where we have things like ...
tensors = get_tensors()
if torch.cuda.device_count() >= 1:
    tensers = tensors.cuda()

... and for the model its ...
num_cuda = torch.cuda.device_count()
model = get_model()
if num_cuda == 1:
    model = model.cuda()
elif num_cuda > 1:
    model = torch.nn.DataParallel(model)


title: Is there a way to only display frames from a video that only have detected object bounding boxes

body:
I'm using a video for testing my custom yolov5s object detection model on test data. However, not every frame in the video has a valid object to be detected.
[ When the inference testing is ran on the video, the image below shows when the model finds a frame that has an object that can be detected and classified
As you can see, frame 4401 has detected a yield traffic sign but frame 4399 and 4400 haven't detected anything. My question is how do I process through 4500 frames of a video and only display frames that have detected objects. I have extracted the frames as JPEGS but only want the valid JPEGS to evaluate test performance? Better still, is it possible to extract the data from relevant frames to a text file?
If anyone could help me out this would be a great help and time saver, many thanks in advance.
the new error


title: Predicting a Chess Board Position using Pytorch

body:
I want to predict a the current chess board using pytorch/keras. (Let's not worry about the input for now.)
How would I got about that?
A chess board has 8x8 positions (64) on each position could be a black or white piece (12) or no piece at all (1). I am planning on using this representation for the chess board (other suggestions are welcome!):
https://en.wikipedia.org/wiki/Board_representation_(computer_chess)#Square_list
For example:
 2 3 4 5 6 4 3 2
 1 1 1 1 1 1 1 1
 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0
-1-1-1-1-1-1-1-1
-2-3-4-5-6-4-3-2`

As far as I know it is not possible to predict something like this. Because the number of classes my final layer would have to predict are 448 (64x7) and I don't feel like a NN could do that. Additionally there is the problem that softmax wouldn't work (imo). Also the Loss function might become a problem as well.
Does someone have an idea on how to do that? Or could point me in the right direction, because multi-class classification isn't really the right term for this task. I was thinking about creating 6 networks that create a classification for each piece. So a 8x8 array that looks like this (for rooks):
 10000001
 00000000
 00000000
 00000000
 00000000
-1000000-1

But the problem is still quite similar.
I think creating 64 NNs that take care of one position each would simplify the problem a bit. But that would be a pain to train.
Looking forward to hearing your suggestions!


title: Split a torch tensor using a same-sized tensor of indices

body:
Let's say that I have tensor
t = torch.tensor([1,2,3,4,5])

I want to split it using a same-sized tensor of indices that tells me for each element, in which split it should go.
indices = torch.tensor([0,1,1,0,2])

So that the final result is
splits
[tensor([1,4]), tensor([2,3]), tensor([5])]

Is there a neat way to do this in Pytorch?
EDIT : In general there will be more than 2 or 3 splits.


title: Can I apply softmax only on specific output neurons?

body:
I am building an Actor-Critic neural network model in pytorch in order to train an agent to play the game of Quoridor (hopefully). For this reason, I have a neural network with two heads, one for the actor output which does a softmax on all the possible moves and one for the critic output which is just one neuron (for regressing the value of the input state).
Now, in quoridor, most of the times not all moves will be legal and as such I am wondering if I can exclude output neurons on the actor's head that correspond to illegal moves for the input state e.g. by passing a list of indices of all the neurons that correspond to legal moves. Thus, I want to not sum these outputs on the denominator of softmax.
Is there a functionality like this on pytorch (because I cannot find one)? Should I attempt to implement such a Softmax myself (kinda scared to, pytorch probably knows best, I ve been adviced to use LogSoftmax as well)?
Furthermore, do you think this approach of dealing with illegal moves is good? Or should I just let him guess illegal moves and penalize him (negative reward) for it in the hopes that eventually it will not pick illegal moves?
Or should I let the softmax be over all the outputs and then just set illegal ones to zero? The rest won't sum to 1 but maybe I can solve that by plain normalization (i.e. dividing by the L2 norm)?


title: How can I split my dataset using torchtext?

body:
I've been following this tutorial on github on sentiment analysis. The author has been using built-in datasets in torchtext.
My problem is now I want to try to what he's been doing in his tutorials using my own datasets that I've downloaded recently but I don't know how.
He's been using:
TEXT = data.Field(tokenize = 'spacy',
                  tokenizer_language = 'en_core_web_sm')

LABEL = data.LabelField()

train_data, test_data = datasets.TREC.splits(TEXT, LABEL, fine_grained=False)

train_data, valid_data = train_data.split(random_state = random.seed(SEED))

And I've imported the csv file into a dataframe and cleaned it up removing redundancies and empty rows and stuff like that and then I created the TEXT and LABEL variables but what should I do next? How can I get split my dataset the same way he did?


title: Can I find the number of specific numeric data in this MNIST training, test data?

body:
import torchvision.datasets as dsets
import torchvision.transforms as transforms
import torch.nn.init
import torch.nn.functional as F

device = "cuda" if torch.cuda.is_available() else "cpu"
print(device)

learning_rate = 0.001
training_epochs = 15
batch_size = 100

mnist_train = dsets.MNIST(root='MNIST_data/', # Specify download path
                          train=True, # Specify True to download as training data
                          transform=transforms.ToTensor(), # Convert to tensor
                          download=True)

mnist_test = dsets.MNIST(root='MNIST_data/', # Specify download path
                         train=False, # If false is specified, download as test data
                         transform=transforms.ToTensor(), # Convert to tensor
                         download=True)

This is the part that loads the data of MNIST data classification code using CNN
In the book I refer to, it is said that you can see how many specific numeric data are in the training set and the test set by mentioning only that part.
For example, can you tell how many '5' data are in that training or test set?
Just knowing that you can access data tensors with mnist_train.train_data or mnist_train.train_labels, etc., I don't know how much I can think of how to know the number of specific numeric data. help


title: Tensorflow and Torch on the same environment

body:
I am working on a goat detection problem. The main model identifies "goat" (single class) on an image and each goat then cropped from the original image. The cropped image then passes through a tensorflow model (trained tensorflow.keras.applications.InceptionV3) to find the current posture of the goat (sitting and standing - two classes).
The version of the torch should be 1.7+ and I am trying to use any version of the tensorflow (1.15.1/1.13.0 preferred). If I install tensorflow while torch is already installed, both tensorflow and torch disappear from the environment (pip freeze). Are there any compatible versions out there? Any specific version of TensorFlow and torch can be on the same environment?
Here are the commands -
conda create -n tst2 python=3.7
conda activate tst2

# install torch 1.7 gpu with cuda 10.1
conda install pytorch==1.7.1 torchvision==0.8.2 torchaudio==0.7.2 cudatoolkit=10.1 -c pytorch

if I import torch, it works fine.
Then I Installed tensorflow 2.1.0 gpu
conda install -c anaconda tensorflow-gpu

Now, I can import tensorflow but no torch
>>> import tensorflow
2021-05-02 17:47:23.158416: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
>>> import torch
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'torch'


title: Convert one-hot encoded dimension into the index of position of 1

body:
I have a tensor of three dimensions [batch_size, sequence_length, number_of_tokens].
The last dimension is one-hot encoded. I want to receive a tensor of two dimensions, where sequence_length consists of the index position of '1' of the number_of_tokens dimension.
For example, to turn a tensor of shape (2, 3, 4):
[[[0, 1, 0, 0]
[1, 0, 0, 0]
[0, 0, 0, 1]]
[[1, 0, 0, 0]
[1, 0, 0, 0]
[0, 0, 1, 0]]]

into a tensor of shape (2, 3) where number_of_tokens dimension is converted into the 1's position:
[[1, 0, 3]
[0, 0, 2]]

I'm doing it to prepare the model result to compare to reference answer when computing loss, I hope it is correct way.


title: pytorch language table for use in deep learning

body:
can anyone tell me the error here?

import pytorch_translator as pt
import pickle
import torch
# Set up the dictionary (vocab table)
pairs, src, tgt = pt.genLangs("de", "en", "train-short.txt", vocab_size=6000)
with open("langs.pkl", 'wb') as outfile:
     pickle.dump((src, tgt), outfile)

teacher_forcing_ratio = 0.5
hidden_size = 128

  AttributeError                            Traceback (most recent call last)
   in ()
        1 # Set up the dictionary (vocab table)
  ----> 2 pairs, src, tgt = pt.genLangs("de", "en", "train-short.txt", vocab_size=6000)
        3 with open("genLangs.pkl", 'wb') as outfile:
        4     pickle.dump((src, tgt), outfile)
        5

  AttributeError: 'module' object has no attribute 'genLangs'

There is something wrong with the genLangs but i can't find anything about this in pytorch, which suggests that it is wrong, but i don't know what is right to go there. completely removing it gives a syntax error.


title: Region of interest or mask as input to CNN, for faster feedforward computation

body: My input image is huge and only a small irregularly shaped region shall be used as input. I cannot crop it because the shape of the ROI causes the cropped region to be almost the same dimension as the input. I understand the receptive field and background context stuff, which impacts accuracy. Still, i am curious if any framework allows me to specify a binary mask as ROI?


title: Does pack-pad sacrifices accuracy?

body:
I tried pack-pad technique. It does reduce many computing time a lot, but my accuracy is significantly worse than former one.

It sounds make sense with approximation theory that keep longer computing time you will get more fine value. But this time it is neural network. I don't think training by embedded zero vector will get me more accuracy. Please correct me if I am wrong.

Here is my files. If you would like to see my code.
This is plain one with 50% accuracy

pack-pad with 25% accuracy

It is not a homework assignment. It is my self-study.

Questions:
1. Am I get a correct result?
2. Does pack-pad sacrifices accuracy?

PS:
I feel my question would fit to stackoverflow most than Datascience or CodeReview. Please let me move it if need.


title: PyTorch Experience Replay with multiple inputs

body:
I am running a modified version of the pyTorch deep Q tutorial which I have modified to pass in my own data rather than gym, and one additional input (two inputs in total)

Currently I am generating an individual state for each 'column' of inputs (not sure if this is the correct way though), when trying to pass the second input into my experience replay function it is returning:

__new__() takes 5 positional arguments but 7 were given

Code for expReplay():

class ReplayMemory(object):

    def __init__(self, capacity):
        self.capacity = capacity
        self.memory = []
        self.position = 0

    def push(self, *args):
        """Saves a transition."""
        if len(self.memory) < self.capacity:
            self.memory.append(None)
        self.memory[self.position] = Transition(*args)
        self.position = (self.position + 1) % self.capacity

    def sample(self, batch_size):
        return random.sample(self.memory, batch_size)

    def __len__(self):
        return len(self.memory)

And my Triggering of the function:

memory.push(state,rsistate, action, next_state, next_rsi_state, reward)

If anyone has any examples of experience replay using multiple inputs please fire away! <3


title: Where is this dimension error getting generated in my ADAM algorithm?

body:
I am working on main.py in this BRATS Unet

https://github.com/pykao/Modified-3D-UNet-Pytorch/blob/master/main.py

# create your optimizer
print ("Creating Optimizer")
##optimizer = optim.adam(net.parameteres(), lr=)
optimizer = torch.optim.Adam(model.parameters(), lr=args.lr)
print ("Created! \n")

trainloader = torch.utils.data.DataLoader(train_idx, batch_size=2, shuffle=True)
testloader  = torch.utils.data.DataLoader(test_idx, batch_size=2, shuffle=False)

for epoch in range(2):  # loop over the dataset multiple times
    running_loss = 0.0
    for i, data in enumerate(trainloader, 0):
        print("inside for")
        # get the inputs THIS ERRORS OUT
        inputs, labels = data

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = model(inputs)
        loss = criterion(outputs, target)
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()
        if i % 2000 == 1999:    # print every 2000 mini-batches
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 2000))
            running_loss = 0.0

print('Finished Training')

I get this output:

Creating Optimizer
Created!

inside for
Traceback (most recent call last):
  File "main.py", line 109, in <module>
    outputs = model(inputs)
  File "/home/MAHEUNIX/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/mnt/c/Users/MAHE/Modified Unet3D Master -TestRun/model.py", line 99, in forward
    out = self.conv3d_c1_1(x)
  File "/home/MAHEUNIX/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/MAHEUNIX/anaconda3/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 448, in forward
    self.padding, self.dilation, self.groups)
RuntimeError: Expected 5-dimensional input for 5-dimensional weight [16, 4, 3, 3, 3], but got 0-dimensional input of size [] instead

I am unfamiliar with PyTorch, and so trainloader, testloader are probably incorrectly used. Please assume I don't know much while you help me. Thanks.

New error:

Traceback (most recent call last):
  File "/mnt/c/Users/MAHE/Modified Unet3D Master -TestRun/main.py", line 91, in <module>
    for id, info in enumerate(trainloader,0):
  File "/home/MAHEUNIX/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 615, in __next__
    batch = self.collate_fn([self.dataset[i] for i in indices])
  File "/home/MAHEUNIX/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 615, in <listcomp>
    batch = self.collate_fn([self.dataset[i] for i in indices])
KeyError: 0


title: Simple Data recall RNN in Pytorch

body:
I am learning Pytorch and am trying to make a network that can remember previous inputs.
I have tried 2 different input/output structures(see below) but haven't gotten anything to work the way I would like.

input 1:

in:[4,2,7,8]

output [[0,0,4],[0,4,2],[4,2,7],[2,7,8]]

code:

def histroy(num_samples=4,look_back=3):
    data=np.random.randint(10,size=(num_samples)).tolist()
    lab=[[0]*look_back]
    for i in data:
        lab.append(lab[-1][1:]+[i])
    return data,lab[1:]

input2:

in:[4,2,7,8]

out:[0,4,2,7]

def histroy(num_samples=4):
    data=np.random.randint(10,size=(num_samples)).tolist()
    lab=[0]
    for i in data:
        lab.append(i)
    return data,lab

I have tried a number of different network structures and training methods but nothing seems to stick.

The only things I think I have right are net.hidden = net.init_hidden()
 should go outside of each epoch and loss.backward(retain_graph=True) but that doesnt seem to do anything

Currently, it can learn the last number in the sequence but never seems to learn any of the others

My last attempt:

import torch
import numpy as np
import torch.nn as nn
import torch.optim as optim

def histroy(num_samples=4,look_back=3):
    data=np.random.randint(10,size=(num_samples)).tolist()
    lab=[[0]*look_back]
    for i in data:
        lab.append(lab[-1][1:]+[i])
    return data,lab[1:]

class Net(nn.Module):
    def __init__(self, input_dim, hidden_dim, batch_size, output_dim=10, num_layers=1):
        super(Net, self).__init__()
        self.input_dim = input_dim
        self.hidden_dim = hidden_dim
        self.batch_size = batch_size
        self.num_layers = num_layers

        self.memory = nn.RNN(self.input_dim,self.hidden_dim,self.num_layers)
        self.linear = nn.Linear(self.hidden_dim, output_dim)
        self.first=True

    def init_hidden(self):
        # This is what we'll initialise our hidden state as
        return (torch.zeros(self.num_layers, self.batch_size, self.hidden_dim),
                torch.zeros(self.num_layers, self.batch_size, self.hidden_dim))

    def forward(self, input):
        self.memory_out, self.hidden = self.memory(input.view(len(input), self.batch_size, -1))
        y_pred = self.linear(self.memory_out[-1].view(self.batch_size, -1))
        return y_pred.view(-1)

if __name__ == '__main__':
    data_amount = 10000
    batch_size = 1  # default is 32
    data_amount-=data_amount%batch_size
    number_of_times_on_the_same_data = 250
    look_back=5

    net=Net(input_dim=1,hidden_dim=25,batch_size=batch_size,output_dim=look_back)
    data,labs=histroy(data_amount,look_back)
    data = torch.Tensor(data).float()
    labs = torch.Tensor(labs).float()

    optimizer = optim.Adam(net.parameters())
    criterion = torch.nn.MSELoss(size_average=False)

    for epoch in range(number_of_times_on_the_same_data):  # loop over the dataset multiple times
        running_loss = 0.0
        data, labs = histroy(data_amount, look_back)
        data = torch.Tensor(data).float()
        labs = torch.Tensor(labs).float()
        net.hidden = net.init_hidden()
        print("epoch",epoch)
        for i in range(0, data_amount, batch_size):
            inputs = data[i:i + batch_size]
            labels = labs[i:i + batch_size]
            optimizer.zero_grad()

            # forward + backward + optimize
            outputs = net(inputs)

            loss = criterion(outputs, labels)
            loss.backward(retain_graph=True)
            optimizer.step()
            running_loss += loss.item()

            if i >= data_amount-batch_size:
                print("loss",loss)
                net.hidden = net.init_hidden()
                print("Outputs",outputs)
                print("Input", data[-1*look_back:])
                print("labels",labels)


title: How to implement a variant of convolution operation using low-level pytorch APIs?

body:
My target is to implement a variant of convolution operation in pytorch which requires high inference speed.

I have implemented one in C++ and CUDA through pytorch CPP extension. However, the speed is not satisfactory, I guess I’ll have to implement it with lower level APIs. It seems aten/src/ATen/native/Convolution.cpp is the right place to do so, and it further calls aten/src/THNN/generic/SpatialConvolutionMM.c, so I modify SpatialConvolutionMM.c. The problem is after compiling, I cannot see any changes on the outputs. Maybe the files in THNN are wrapped in ATen/native in some way, I just cannot figure it out. Could anyone give me some suggestions? Thank you.


title: Why doesn´t raising the amount of layers make my network more accurate?

body:
I hope some of you can help me out with this one:

So, I have a network, made of 36 Inputs.
Normally, this network has more than half a million data-fields, but currently I´m working with a smaller amount of data. (About 40.000 datas)

The networks aim is to predict the amount of people in a room, depending on their environment. The environment is represented through the 36 input-neurons.

In first place, I tried using 4 layers, with 36 In- and Outputs.
The absolute difference between predicted people and the amount of people that should be there, is about 160. This is quite a lot, when the amount only moves between 0 and 2100.
When I do this with all the data it is about 200.

So, now I tried to increase the amount of layers and neurons in my network. As you can see in my code, I have 10 layers, all of them have at least 10.000 output-neurons. How can it be, that my network is still that in-accurate?
Does the change effect my network in any way? Can someone please tell me? I know the scheme behind neuronal networks, and increasing all these parameters should help...normally...

So this is my Network-Class:

    class Network(nn.Module):
    def __init__(self):
        super(Network, self).__init__()
        self.linear1 = nn.Linear(36, 10000)  #36 Input-Neurons, 36 Output-Neurons, Linearer Layer
        self.linear2 = nn.Linear(36, 10000)
        self.linear3 = nn.Linear(36, 10000)
        self.linear4 = nn.Linear(36, 10000)
        self.linear5 = nn.Linear(36, 20000)
        self.linear6 = nn.Linear(36, 20000)
        self.linear7 = nn.Linear(36, 10000)
        self.linear8 = nn.Linear(36, 10000)
        self.linear9 = nn.Linear(36, 10000)
        self.linear10 = nn.Linear(36, 1)

    def forward(self, x):
        pax_predict = F.torch.relu(self.linear1(x))
        pax_predict = F.torch.relu(self.linear2(x))
        pax_predict = F.torch.relu(self.linear3(x))
        pax_predict = F.torch.relu(self.linear4(x))
        pax_predict = F.torch.relu(self.linear5(x))
        pax_predict = F.torch.relu(self.linear6(x))
        pax_predict = F.torch.relu(self.linear7(x))
        pax_predict = F.torch.relu(self.linear8(x))
        pax_predict = F.torch.relu(self.linear9(x))
        pax_predict = self.linear10(x)
        return pax_predict

    def num_flat_features(self, pax_predict):
        size = pax_predict.size()[1:]
        num = 1
        for i in size:
            num *= i
        return num

network = Network()
print('Netzwerk:')
print(network)
print()

Followed by the training:

optimizer = torch.optim.Adam(network.parameters(), lr=50)
#optimizer = torch.optim.SGD(network.parameters(), lr=0.0022, momentum=0.8)

switch = 1

def training():

    target = y_train_tensor

    for epoch in range(200):

        input = x_train_tensor
        y_prediction = network(input)
        loss = criterion(y_prediction, target)
        wurzel = math.sqrt(loss)
        loss_avg = loss /len(y_train_tensor)
        wurzel_avg = math.sqrt(loss_avg)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        print('Epoch:', epoch, ' Total Loss:', loss.data,'Sqroot:', wurzel)
        print('Average Loss:', loss_avg, 'Avg. Sqrt:', wurzel_avg)
        print()

        plt.scatter(epoch, loss_avg.data, color='r', s=10, marker='.')

    plt.xlabel('Epochen')
    plt.ylabel('Loss')
    plt.savefig('./plot/loss/figure.png', dpi = 300)

    ## Gewichtsausgabe #########################################################################################
    ############################################################################################################

    weights_list = list(network.parameters())
    print('Gewichtsliste:')
    print (weights_list)

    weights_numpy = np.asarray(weights_list)

    print('Array:', weights_numpy)

I hope that someone of you can help me.

Thank you so much in advance!
Great community

Kind Regards
Christian Richter


title: Inference time varies over different GPUs using Torch

body:
I get a bug when running the below inference code. In the function recognize(), it takes 0.4s to finish prediction. It takes another 3s to return the result preds_str to the caller function. I found that if I set gpu_id=0 in file config, it returns instantly. How can I fix this bug? Thanks in advance.

def recognize(imgs, model, demo_loader):

          t = time()
          model.eval()
          with torch.no_grad():
              for image_tensors, image_path_list in demo_loader:
                    batch_size = image_tensors.size(0)
                    image = image_tensors.to(config.device)
                    # For max length prediction
                    length_for_pred = torch.IntTensor([config.batch_max_length] * batch_size).to(config.device)
                    text_for_pred = torch.LongTensor(batch_size, config.batch_max_length + 1).fill_(0).to(config.device)

                    preds = model(image, text_for_pred, is_train=False)
                    _, preds_index = preds.max(2)
                    preds_str = converter.decode(preds_index, length_for_pred)

           print('time elapsed before return:'time()-t) #0.4s
           return preds_str
def main():
     model = Model()
     self.model.cuda(config.device)
     model = torch.nn.DataParallel(model, device_ids=[config.device], output_device=[config.device]).to(config.device)
     model.load_state_dict(torch.load(config.saved_model, map_location=config.device))
     AlignCollate_demo = AlignCollate(imgH=config.imgH, imgW=config.imgW, keep_ratio_with_pad=config.PAD)
     imgs_dataset = ImageDataset(imgs)
     demo_loader = torch.utils.data.DataLoader(imgs_dataset, batch_size=config.batch_size,shuffle=False,num_workers=int(config.workers),collate_fn=AlignCollate_demo, pin_memory=True)
     start_time = time()
     # imgs = [img1, img2, ....]
     preds_str = recognize(imgs, model, demo_loader)
     print('time elapsed after return', time()-start_time) #3.4s

Config file:

    class ConfigWordRecognizer:
        gpu_id = 1 #troublesome line here
        device = torch.device('cuda:{}'.format(gpu_id) if torch.cuda.is_available() else 'cpu')
        imgH = 32
        imgW = 100
        batch_size = 80
        workers = 8
        batch_max_length = 25


title: Pytorch Autoencoder - How to improve loss?

body:
I've a UNET style autoencoder below, with a filter I wrote in Pytorch at the end. The network seems to be converging faster than it should and I don't know why. I have a dataset of 4000 images and I'm taking a 128x128 crop every time. I'm employing a training rate schedule and weight decay. I've tried fiddling with my parameters with a tiny dataset to see improvements but nothing seems to work. Once the learning rate goes down, the loss just bounces around and doesn't hit a floor, and in some cases goes back up. My network is as follows:

import torch
import torch.nn as nn
from wiener_3d import wiener_3d
from PIL import Image
import numpy as np
import matplotlib.pyplot as plt
import random

def np_to_pil(np_imgs):

img_num = np_imgs.shape[0]
channel_num = np_imgs.shape[1]
ar = np.clip(np_imgs*255, 0, 255).astype(np.uint8)

pil_imgs = []
for i in range(img_num):
    if channel_num == 1:
        img = ar[i][0]
    else:
        img = ar[i].transpose(1, 2, 0)
    pil_imgs.append(Image.fromarray(img))

return pil_imgs

class WienerFilter(nn.Module):
def __init__(self, param_b=16):
    super(WienerFilter, self).__init__()
    # self.register_parameter("param_a", nn.Parameter(torch.tensor(param_a)))
    # self.param_a = nn.Parameter(torch.tensor(param_a))
    # self.param_a.requires_grad = True
    self.param_b = param_b

def forward(self, input, std):
    tensors = input.shape[0]
    for i in range(tensors):
        tensor = input[i]
        tensor = torch.squeeze(tensor)
        # tensor = wiener_3d(tensor, self.param_a, self.param_b
        tensor = wiener_3d(tensor, 2*std, self.param_b)
        tensor = torch.unsqueeze(tensor, 0)
        input[i] = tensor
    return input

class AutoEncoder(nn.Module):
"""Autoencoder simple implementation """
def __init__(self):
    super(AutoEncoder, self).__init__()
    # Encoder
    # conv layer
    self.block1 = nn.Sequential(
        nn.Conv2d(1, 96, 3, padding=1),
        nn.BatchNorm2d(96),
        nn.LeakyReLU(0.1),
        nn.Conv2d(96, 96, 3, padding=1),
        nn.MaxPool2d(2),
        nn.BatchNorm2d(96),
        nn.LeakyReLU(0.1)

    )
    self.block2 = nn.Sequential(
        nn.Conv2d(96, 96, 3, padding=1),
        nn.MaxPool2d(2),
        nn.BatchNorm2d(96),
        nn.LeakyReLU(0.1)
    )
    self.block3 = nn.Sequential(
        nn.Conv2d(96, 96, 3, padding=1),
        nn.BatchNorm2d(96),
        nn.LeakyReLU(0.1),
        nn.ConvTranspose2d(96, 96, 2, 2),
        nn.BatchNorm2d(96),
        nn.LeakyReLU(0.1)
    )
    self.block4 = nn.Sequential(
        nn.Conv2d(192, 192, 3, padding=1),
        nn.BatchNorm2d(192),
        nn.LeakyReLU(0.1),
        nn.Conv2d(192, 192, 3, padding=1),
        nn.BatchNorm2d(192),
        nn.LeakyReLU(0.1),
        nn.ConvTranspose2d(192, 192, 2, 2),
        nn.BatchNorm2d(192),
        nn.LeakyReLU(0.1)
    )
    self.block5 = nn.Sequential(
        nn.Conv2d(288, 192, 3, padding=1),
        nn.BatchNorm2d(192),
        nn.LeakyReLU(0.1),
        nn.Conv2d(192, 192, 3, padding=1),
        nn.BatchNorm2d(192),
        nn.LeakyReLU(0.1),
        nn.ConvTranspose2d(192, 192, 2, 2),
        nn.BatchNorm2d(192),
        nn.LeakyReLU(0.1)
    )
    self.block6 = nn.Sequential(
        nn.Conv2d(193, 96, 3, padding=1),
        nn.BatchNorm2d(96),
        nn.LeakyReLU(0.1),
        nn.Conv2d(96, 64, 3, padding=1),
        nn.BatchNorm2d(64),
        nn.LeakyReLU(0.1),
        nn.Conv2d(64, 32, 3, padding=1),
        nn.LeakyReLU(0.1),
        nn.Conv2d(32, 1, 3, padding=1),
        nn.LeakyReLU(0.1)
    )

    self.wiener_filter = WienerFilter()

def forward(self, x, std):
    # torch.autograd.set_detect_anomaly(True)
    # print("input: ", x.shape)
    pool1 = self.block1(x)
    # print("pool1: ", pool1.shape)
    pool2 = self.block2(pool1)
    # print("pool2: ", pool2.shape)
    pool3 = self.block2(pool2)
    # print("pool3: ", pool3.shape)
    pool4 = self.block2(pool3)
    # print("pool4: ", pool4.shape)
    pool5 = self.block2(pool4)
    # print("pool5: ", pool5.shape)
    upsample5 = self.block3(pool5)
    # print("upsample5: ", upsample5.shape)
    concat5 = torch.cat((upsample5, pool4), 1)
    # print("concat5: ", concat5.shape)
    upsample4 = self.block4(concat5)
    # print("upsample4: ", upsample4.shape)
    concat4 = torch.cat((upsample4, pool3), 1)
    # print("concat4: ", concat4.shape)
    upsample3 = self.block5(concat4)
    # print("upsample3: ", upsample3.shape)
    concat3 = torch.cat((upsample3, pool2), 1)
    # print("concat3: ", concat3.shape)
    upsample2 = self.block5(concat3)
    # print("upsample2: ", upsample2.shape)
    concat2 = torch.cat((upsample2, pool1), 1)
    # print("concat2: ", concat2.shape)
    upsample1 = self.block5(concat2)
    # print("upsample1: ", upsample1.shape)
    concat1 = torch.cat((upsample1, x), 1)
    # print("concat1: ", concat1.shape)
    output = self.block6(concat1)
    path = "test"
    t_map = x - output

    filtering = self.wiener_filter(t_map, std)

    filtered_output = output + filtering

    return filtered_output

My current parameters are:
Adam optimizer,
learning rate decay by 0.1 if no improvement for 7 epochs,
intial learning rate 0.001,
0.0001 weight decay,
No batches.

I feel like I've tried everything at this stage. Could someone give me some advice on how to improve my network? Thank you.


title: Does the generator's optimizer also train the discriminator?

body:
While learning about GANs, I noticed that the code examples exhibit this pattern:

The discriminator is trained like this:

d_optim.zero_grad()

real_pred = d(real_batch)
d_loss = d_loss_fn(real_pred, torch.ones(real_batch_size, 1))
d_loss.backward()

fake_pred = d(g(noise_batch).detach())
d_loss = d_loss_fn(fake_pred, torch.zeros(noise_batch_size, 1))
d_loss.backward()

d_optim.step()

The generator is trained like this:

g_optim.zero_grad()

fake_pred = d(g(noise_batch))
g_loss = g_loss_fn(fake_pred, torch.ones(noise_batch_size, 1))
g_loss.backward()

g_optim.step()

It is mentioned that d(g(noise_batch).detach()) is written for the discriminator instead of d(g(noise_batch)) to prevent d_optim.step() from training g, but nothing is said about d(g(noise_batch)) for the generator; would g_optim.step() also train d?

In fact, why do we d(g(noise_batch).detach()) if, for example, d_optim = torch.optim.SGD(d.parameters(), lr=0.001)? Does this not specify that d.parameters() and not also g.parameters() are to be updated?


title: pytorch: ModuleNotFoundError exception on windows 10

body:
I have installed pytorch using

conda install pytorch torchvision cpuonly -c pytorch

In the jupyter notebook, the 'import torch' fails:

  ModuleNotFoundError: No module named 'torch._C'


title: How to keep model fixed during training?

body:
I am trying to implement a model that uses encoding from multiple pre-trained BERT models on different datasets and gets a combined representation using a fully-connected layer. In this, I want that BERT models should remain fixed and only fully-connected layers should get trained. Is it possible to achieve this in huggingface-transformers? I don't see any flag which allows me to do that.

PS: I don't want to go by the way of dumping the encoding of inputs for each BERT model and use them as inputs.


title: Google Cloud Platform taking a long time to load the app and stalling

body:
I'm using a google cloud instance to deploy a PyTorch model and when I gcloud app browse and open the link the app seems to be stalling and taking a very long time to load. I will include my app.yaml file because I think it has to do with gunivorn. In addition, when I un-comment the -w 24 -k uvicorn.workers.UvicornWorker I get a worker failed to boot message, i.e. gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>. Thanks for the help!

runtime: python
env: flex
runtime_config:
    python_version: 3
resources:
  cpu: 12
  memory_gb: 72
  disk_size_gb: 60
instance_class: F4_1G
entrypoint: gunicorn main:app #-w 24 -k uvicorn.workers.UvicornWorker


title: .eq() method is not giving same result as [ == ]

body:
I am having a hard time understanding why the results are not the same for following code.

I am trying to find the accuracy of a model but the first item gives a result of tensor(66.), and second item gives a result of tensor(105).

(y_test[y_test==y_predicted_cls].sum(), y_predicted_cls.eq(y_test).sum())

Both of the tensors (y_test and y_predicted_cls) have the same data type, torch.float32.
Output:

(tensor(66.), tensor(105))

I thought that first one would be equal to the second one.


title: What's the difference between memory_allocated and memory cached?

body: I have been struggling with these two. I tried to fix a cuda out of memory problem, So I was using them to track my GPU memory status. However between two checkpoints, only 450 M of new tensor was allocated. However memory_cached gave a result which is far more than allocated. And it was just growing like doubled after each iteration.


title: using 2d array as indices of a 4d array

body:
I have a Numpy 2D array (4000,8000) from a tensor.max() operation, that stores the indices of the first dimension of a 4D array (30,4000,8000,3). I need to obtain a (4000,8000,3) array that uses the indices over this set of images and extract the pixels of each position in the 2D max array.
A = np.random.randint( 0, 29, (4000,8000), dtype=int)
B = np.random.randint(0,255,(30,4000,8000,3),dtype=np.uint8)

final = np.zeros((B.shape[1],B.shape[2],3))

r = 0
c = 0
for row in A:
  c = 0
  for col in row:
    x = A[r,c]
    final[r,c] = B[x,r,c]
    c=c+1
  r=r+1

print(final.shape)

Is there any vectorised way to do that? I am fighting with the RAM usage using loops. Thanks


title: Why do I get a segmentation fault (SIGSEGV) when using PyTorch and calling a forward pass?

body:
I have been getting a segmentation violation error, also called SIGSEGV, when calling a forward pass in my PyTorch backend graph convolutional neural network:
Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)

I am using torch version 1.5.0.
This error comes after I have created an instance of my graph convolutional neural network and am calling the forward method on it for the first time, thereby passing the data.
This is part of my script:
# create model
model: LinkPredictHetero = LinkPredictHetero(
    g=graph,
    h_dim=h_dim,
    out_dim=h_out,
    num_hidden_layers=n_hidden_layers,
)

# call forward pass
embed = model(graph)

The graph is an instance of the dgl.heterograph.
The forward()method of the model links to the forward method of the base layer class, defined as follows:
def forward(self, graph):
    graph = graph.local_var()
    hs = {}
    hs = self.embed_layer(graph)
    for layer in self.layers:
        hs = layer(g, inputs = hs)
    return hs

Related to this forward method,
self.embed_layer is a torch.nn.ModuleDict():
self.embed_layer = torch.nn.ModuleDict()
for node_type in self.graph.node_types:
    in_size = list(self.graph.nodes[node_type].data['feature'].size())[1]
    self.embed_layer[node_type] = torch.nn.Linear(in_size, self.embed_size)

and self.layers is a torch.nn.ModuleList() to which different layer implementations are appended.
So when in the script I call model(deephet.graph), which should call the forward() method of the layer, the segmentation violation ends my program, and actually also terminates python.
Could you please help me to find out why this happens? I would really appreciate it.
Thank you, dear community.


title: How can I use torch.nn.DataParallel while I'm doing transfer learning?

body:
I'm trying to do my training on multiple GPUs by using the following code (the latest pytorch version):
from torchvision import models
model = model.vgg16(pretrained=True)
model.classifier._modules['6'] = torch.nn.Linear(4096, 10)
self.model = torch.nn.DataParallel(model, device_ids=[0,1,2]).cuda()
self.model = model.to(f'cuda:0')
...
def forward(self, input_data):
    output = self.model.forward(input_data)

I get this error when I call self.model.forward(input_data):
  File "/home/poahmadvand/py3env/lib/python3.7/site-packages/torchvision/models/vgg.py", line 43, in forward
    x = self.features(x)
  File "/home/poahmadvand/py3env/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/poahmadvand/py3env/lib/python3.7/site-packages/torch/nn/modules/container.py", line 100, in forward
    input = module(input)
  File "/home/poahmadvand/py3env/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/poahmadvand/py3env/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 345, in forward
    return self.conv2d_forward(input, self.weight)
  File "/home/poahmadvand/py3env/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 342, in conv2d_forward
    self.padding, self.dilation, self.groups)
RuntimeError: Expected tensor for argument #1 'input' to have the same device as tensor for argument #2 'weight'; but device 0 does not equal 1 (while checking arguments for cudnn_convolution)

How can I fix this error? thanks.


title: Gradient is equal to 'None'

body:
I have two networks. The output of the first network is the input to the other. In order to calculate the loss for the second network, I use vanilla policy gradient. I want to backpropagate this loss into the first network. After checking if the gradeints has changed, I see that they are all none.
I first load the first network (a pre-trained autoencoer in my network this way):
def load_checkpoint(filepath, model):
   checkpoint = torch.load(filepath)
   model.load_state_dict(checkpoint['state_dict'])
   for parameter in model.parameters():
       parameter.requires_grad = True

  model.train()
  return model

Then I define the optimizers for both networks this way:
class MultipleOptimizer(object):
    def __init__(self, *op):
        self.optimizers = op

    def zero_grad(self):
        for op in self.optimizers:
            op.zero_grad()

    def step(self):
        for op in self.optimizers:
            op.step()

opt = MultipleOptimizer(SGD(model.parameters(), lr=1, momentum=0.9), Adam(logits_net.parameters(), lr=lr))

the reward function is:
#Reward function
def reward(x, act):
    #print('action', act)
    #print('x  type', type(x))
    km = KMeans(act, n_init=20, n_jobs=4)
    y_pred = km.fit_predict(x.detach().cpu().numpy())# seems we can only get a centre from batch
    #print('k-means output type', type(y_pred))
    sil_score = sil(x.detach().cpu().numpy(), y_pred)
    #print('sil score', sil_score)

    return sil_score

The architecture of the second neural net and an alternative to avoid (logits=logits.mean(0)):
def mlp(sizes, activation=nn.Tanh, output_activation=nn.Identity):
# Build a feedforward neural network. outputs are the logits
    layers = []
    for j in range(len(sizes)-1):
        act = activation if j < len(sizes)-2 else output_activation
        layers += [nn.Linear(sizes[j], sizes[j+1]), act()]

    return nn.Sequential(*layers)

class mlp2(torch.nn.Module):
def __init__(self):

    super(mlp2, self).__init__()
    self.linear1 = nn.Linear(10,100)
    self.relu1 = nn.ReLU(inplace=True)
    self.linear2 = torch.nn.Linear(100,100)
    self.linear3 = torch.nn.Linear(100,20)
    self.linear4 = torch.nn.Linear(2000,100)
    self.ident = nn.Identity()

def forward(self, x):
    a = self.linear1(x)
    a = self.relu1(a)
    a = self.linear2(a)
    a = self.relu1(a)
    a = self.linear3(a)
    a = torch.flatten(a)
    a = self.linear4(a)
    a = self.relu1(a)
    a = self.linear3(a)
    out = self.ident(a)
    return out

Loss is calculated as in the following order:
def get_policy(obs):
    logits = logits_net(obs)
    return Categorical(logits=logits.mean(0))
def get_action(obs):
    return get_policy(obs).sample().item()
def Logp(obs, act):
    logp = get_policy(obs).log_prob(act.cuda())
    return logp
def compute_loss(logp, weights):
    return -(logp * weights).mean()

 def train_one_epoch():
    # make some empty lists for logging.
    batch_obs = []          # for observations
    batch_acts = []         # for actions
    batch_weights = []      # for R(tau) weighting in policy gradient
    batch_logp = []

    # reset episode-specific variables
    j = 1            # signal from environment that episode is over
    ep_rews = []            # list for rewards accrued throughout ep

    for i, data in enumerate(train_loader):

        #Create the mean image out of those 100 images
        x, label = data
        x = model(x.cuda())#torch.Size([100, 10])
        obs = x.data.cpu().numpy()#[100, 10] - a trajectory with only one state
        # Save obs
        batch_obs.append(obs.copy())

        #act in the environment
        #act = get_action(torch.as_tensor(obs, dtype=torch.float32))
        act = get_action(x)
        print('action type', type(act))
        #log probability
        #logp = Logp(torch.as_tensor(obs, dtype=torch.float32),act = torch.as_tensor(act, dtype=torch.int32))
        logp = Logp(x, act = torch.as_tensor(act, dtype=torch.int32))
        #rew = reward(obs, act+2)
        rew = reward(x, act+2)

        # save action, reward
        batch_acts.append(act)
        batch_weights.append(rew)#episode rewards
        batch_logp.append(logp)

    opt.zero_grad()
    batch_logp = torch.stack(batch_logp, dim=0)
    batch_loss = compute_loss(logp = torch.as_tensor(batch_logp, dtype=torch.float32),
                              weights = torch.as_tensor(batch_weights, dtype=torch.float32))

    batch_loss.backward() #does it return anything? gradients? print them!
    opt.step()

    for name, param in logits_net.named_parameters():
       print(name, param.grad)

I applied some changes with the assumption that maybe recreating some of the tensors maybe the issue:
I have the output of the first network, obs, converted like obs = x.data.cpu().numpy() this and then sent to get_action function: act = get_action(torch.as_tensor(obs, dtype=torch.float32)). I changes this to act = get_action(x) so, x is sent directly to this function. Also, change arguments of logp to  logp = Logp(x, act = torch.as_tensor(act, dtype=torch.int32)).
After these changes, I still get the none value for the gradient. Is there anyway possible to backpropagate the gradient when loss is calculated this way? any changes that I can apply?
any help is appreciated.


title: Schedule jobs between GPUs for PyTorch Models

body:
I'm trying to build up a system that trains deep models on requests. A user comes to my web site, clicks a button and a training process starts.
However, I have two GPUs and I'm not sure which is the best way to queue/handle jobs between the two GPUs: start a job when at least one GPU is available, queue the job if there are currently no GPUs available. I'd like to use one GPU per job request.
Is this something I can do in combination with Celery? I've used this in the past but I'm not sure how to handle this GPU related problem.
Thanks a lot!


title: Cannot get the value of hidden weights of RNN

body:
I declare my RNN as
self.rnn = torch.nn.RNN(input_size=encoding_dim, hidden_size=1, num_layers=1, nonlinearity='relu')

Later
self.rnn.all_weights
# [[Parameter containing:
tensor([[-0.8099, -0.9543,  0.1117,  0.6221,  0.5034, -0.6766, -0.3360, -0.1700,
         -0.9361, -0.3428]], requires_grad=True), Parameter containing:
tensor([[-0.1929]], requires_grad=True), Parameter containing:
tensor([0.7881], requires_grad=True), Parameter containing:
tensor([0.4320], requires_grad=True)]]

self.rnn.all_weights[0][0][0].values
# {RuntimeError}Could not run 'aten::values' with arguments from the 'CPU' backend. 'aten::values' is only available for these backends: [SparseCPU, Autograd, Profiler, Tracer].

Clearly I see the value of the weights, but cannot access to it. Documentation says I need to specify requires_grad=True, but that does not work.
Is there a more elegant and usable way than self.rnn.all_weights[0][0][0]?


title: Importing an externally defined function to pytorch Dataset

body:
I have the following problem. I am defining a dataset class for my pytorch dataloader. In the class, I would like to use an externally defined function in the __getitem__ method.
Here is the example of what I mean
import torch
from torch.utils.data import Dataset, DataLoader
import pandas as pd

def readOneElement(df, idx):
   # some fancy code which returns one element with index i based on full data set df (pandas)
   ...

# And here the Dataset class
class ModelDataset(Dataset):

def __init__(self, dataPath, transform = None):

     def listUniqueExamples(self, df):
         listExamples = df[['criterionId', 'adGroupId', 'campaignId', 'accountId']].drop_duplicates().reset_index();
         return listExamples

     self.transform = transform

     # Load data
     df = pd.read_csv(dataPath + '/trainData.csv')

     # Transform constants to self
     self.df = df
     self.listExamples = listUniqueExamples(self, self.df)
     self.length = len(self.listExamples)

def __len__(self):
     return self.length

def __getitem__(self, idx):
     # Create one example data
     sample = readOneElement(df = self.df, idx = idx) # !!!

     return sample

The line marked with #!!! will not work, because the function was defined outside this Dataset object. I get the following error
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)

in
----> 1 data[20]

in __getitem__(self, idx)
     31     def __getitem__(self, idx):
     32          # Create one example data
---> 33          sample = readOneElement(df = self.df, idx = idx)

NameError: name 'readOneElement' is not defined

If I would have defined this function inside the Dataset object (as I have done for the listUniqueExamples function) then it would have worked. However, in this exact case I want this function to be external.
Is there any way to import external functions to the Dataset class in pytorch?
Thank you in advance!


title: How to find the mean and the covariance of a 2d activation map (pytorch)

body: I have a tensor of shape [h, w], which consists of a normalized, 2-dimensional activation map. Considering this to be some distribution, I want to find the mean and the covariance within this activation map in pytorch. Is there an efficient way to do that?


title: Calculating the loss for each class in Multiclass segmentation with weights

body:
Calculating the loss with pytorch is easy using the reduction feature:
criterion = nn.CrossEntropyLoss(weight=cw, reduction="mean")

where cw is a 1D tensor of class weights.
masks_pred = net(imgs)
loss = criterion(masks_pred, masks)

In this case loss is a scalar (after loss.item()) that represents the weighted loss, which can is the same as using:
criterion = nn.CrossEntropyLoss(weight=cw, reduction="none")
masks_pred = net(imgs)
loss = criterion(masks_pred, masks)
loss = loss.sum() / cw[masks].sum()

My question is how to get the weighted loss for each class? In theory I know that I need to calculate the following:
sum loss for class x / sum of class x's weight
Things I tried:
First thing I tried is to get the sum of a specific class weight over masks (say class 0):
cw[masks==0]

but I get this error:
IndexError: too many indices for tensor of dimension 1
I also don't know how to sum the loss per class.


title: PyTorch: Perform add/sub/mul operations using 1-D tensor and multi-channel (3-D) image tensor

body:
Note: I am looking for the fastest/optimal way of doing or improving on this, as my constraint is time
Using PyTorch, I have an image as a 3D tensor, lets say of dimension 64 x 400 x 400, where 64 refers to channels and 400 is the image dimensions. With this, I have a 64 length 1D tensor, all with different values, where the intention is to use one value per channel. I want to use a value per 1D tensor to apply to the entire 400x400 block of the channel. So for example, when I want to add 3d_tensor + 1d_tensor, I want 1d_tensor[i] to be added to all 400x400 = 160000 values in 3d_tensor[i], with [i] ranging from 0 to 63.
What I previously did:
I tried doing it directly, by using the operators only:
output_add = 1d_tensor + 3d_tensor

This returned an error that the dimension of 3d_tensor (400) and 1d_tensor (64) are incompatible.
So my current form is using a for loop
for a, b in zip(3d_tensor, 1d_tensor):
   a += b

However at one stage I have four different 1D tensors to use at once in either addition, subtraction or multiplication, so is this for loop method the most efficient? I'm also planning on doing it 20+ times per image, so speed is key. I tried maybe extending the 1d tensor to a dimension of 64 x 400 x 400 also, so it could be used directly, but I could not get this right using tensor.repeat()


title: Arbitrary shaped Feedforward Neural Network in Pytorch

body:
I am making a script that has some generative aspect to it, and I need to generate arbitrary shaped  feedforward NNs. The idea is to pass a list With the number of number of neurons in each layer, and the number of layers is determined by the length of the list:
shape = [784,64,64,64,10]

I tried something like this:
shapenn = [784,64,64,64,10]
class Net(nn.Module):
    def __init__(self, shapenn):
        super().__init__()
        self.shapenn = shapenn
        self.fcl = [] # list with fully conected leyers
        for i in range(len(act_funs)):
            self.fcl.append(nn.Linear(self.nnarch[i],self.nnarch[i+1]))
net = Net(shapenn)

While the fully connected layers are created correctly in the list fcl, net is not initialized properly for example it has not net.parameters().
I am sure there is a correct way to do this, thank you very much in advance.


title: Pytorch model not working on Linux but works in Windows (problem with embedding weights)

body:
I have a simple flask application with a some pytorch models in it. If I start it on Windows, it works flawlessly but on linux, I get the following error:
    File "/home/emre/matura/matura-aws/bin/flask", line 8, in <module>
    sys.exit(main())
  File "/home/emre/matura/matura-aws/lib/python3.7/site-packages/flask/cli.py", line 967, in main
    cli.main(args=sys.argv[1:], prog_name="python -m flask" if as_module else None)
  File "/home/emre/matura/matura-aws/lib/python3.7/site-packages/flask/cli.py", line 586, in main
    return super(FlaskGroup, self).main(*args, **kwargs)
  File "/home/emre/matura/matura-aws/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/emre/matura/matura-aws/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/emre/matura/matura-aws/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/emre/matura/matura-aws/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/emre/matura/matura-aws/lib/python3.7/site-packages/click/decorators.py", line 73, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/home/emre/matura/matura-aws/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/emre/matura/matura-aws/lib/python3.7/site-packages/flask/cli.py", line 848, in run_command
    app = DispatchingApp(info.load_app, use_eager_loading=eager_loading)
  File "/home/emre/matura/matura-aws/lib/python3.7/site-packages/flask/cli.py", line 305, in __init__
    self._load_unlocked()
  File "/home/emre/matura/matura-aws/lib/python3.7/site-packages/flask/cli.py", line 330, in _load_unlocked
    self._app = rv = self.loader()
  File "/home/emre/matura/matura-aws/lib/python3.7/site-packages/flask/cli.py", line 392, in load_app
    app = locate_app(self, import_name, None, raise_if_not_found=False)
  File "/home/emre/matura/matura-aws/lib/python3.7/site-packages/flask/cli.py", line 240, in locate_app
    __import__(module_name)
  File "/home/emre/matura/Sentiment Analyzer/app.py", line 11, in <module>
    from app_lstm import predict_sentiment
  File "/home/emre/matura/Sentiment Analyzer/app_lstm.py", line 117, in <module>
    torch.load("Finished Models/Pytorch/LSTM.pt", map_location=torch.device("cpu"))
  File "/home/emre/matura/matura-aws/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1045, in load_state_dict
    self.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for RNN:
        size mismatch for embedding.weight: copying a param with shape torch.Size([101616, 100]) from checkpoint, the shape in current model is torch.Size([100944, 100]).

I use torchtext version 0.7 and the transformers library in the version 3.1.0 on WSL2. I want to deploy this model to AWS so I need to get it working under linux.


title: How to multiply torch[N,1] and torch[1,N]?

body:
I'd like to calculate a matrix (shape[N,N]) by multiplying 2 torch vectors A(shape[N,1]) and B=A'(shape[1,N]).
When I use torch.matmul or torch.mm, I got errors or A'A(shape[1,1]).
If A is denoted as
A = [a_1, a_2, ..., a_N]', I want to calculate a matrix C whose (i,j)element is
for i in range(N):
     for j in range(N):
       C(i,j) = a_i * a_j

I'd like to calculate this quickly. Do you have any ideas?
thank you for your help!


title: PyTorch: Calculate Hessian only for a subset of parameters?

body:
I am writing ElasticWeightConsolidation method and for that I need to compute Fisher matrix. As I understood Fisher Matrix is just Hessian of likelihood by weights of neural network. There is good function as torch.autograd.functional.hessian(func, inputs, create_graph=False, strict=False)
So I want to compute hessian(loss,weights) where loss = torch.nn.CrossEntropyLoss(). I also prepared weights of the network so that it its long 1D tensor, to have a possibility simply take diagonal elements of hessian like that:
def flat_param(model_param = yann_lecun.parameters()):
  ans_data = []
  ans_data = torch.tensor(ans_data, requires_grad=True)
  ans_data = ans_data.to(device)
  for p in model_param:
    temp_data = p.data.flatten()
    ans_data = torch.cat((ans_data,temp_data))
  return ans_data

ans = flat_param(yann_lecun.parameters())

then I tried so: hessian(loss, inputs = ans) but problem is that loss takes also targets, but I don't want to compute hessian of them. The task is mnist classification so that targets are integers 0,...,9
and if I add y_train to the parameters like that hessian(loss,inputs = (ans,y_train_01)
It is crashing with words "can't take gradient from integer". I tried also to make y_train_01.requires_grad = False but it didn't help. I understood that loss also depends on y_train_01, but is there any way to determine that targets are constants in my case?


title: pytorch logsumexp for tensors with different shapes

body:
I'm looking for ways to use torch.logsumexp with two tensors with different shapes. I currently implemented logsumexp manually like this below:
import torch
# t1.shape = (4,1,3)
t1 = torch.tensor([[[ 2,  1,  4]],[[-8, 23, -7]],[[ 8, -3,  3]],[[-4,  4,  6]]]).float()

# t2.shape = (4,6,3)
t2 = torch.tensor([[[ 3, 2, 2],[-1,-5, 1],[ 1, 1, 2],[-6, 7,-7],[ 3,-7, 1],[ 1,-1, 2]],
                   [[ 2, 1, 1],[ 3, 3,-1],[-5, 2, 4],[ 4, 3,-9],[ 1, 5, 1],[ 8,-5,-6]],
                   [[ 4, 6, 4],[ 1, 7,-7],[ 8, 8, 6],[-2, 1,-1],[ 9, 5, 9],[ 9, 2,-7]],
                   [[ 2,-6, 1],[-9, 9, 8],[ 3, 3, 2],[-3, 7, 4],[-6, 8,-5],[ 2, 4,-2]]]).float()

# t3.shape = (1,6,3)
t3 = torch.tensor([[[-1,-1, 0],[ 2, 2,-2],[ 3,-1, 0],[ 1, 0, 1],[ 2, 0,-1],[ 1,-2, 3]]]).float()

exp1 = torch.exp(t1)+torch.exp(t2)
log1 = torch.log(exp1)

exp2 = torch.exp(t1)+torch.exp(t3)
log2 = torch.log(exp2)

I would like to perform logsumexp for t1 and t2 and for t1 and t3. Because of exp and log operation, my code is very numerically unstable, so I'm hoping to use torch.logsumexp().
Is there any way to use torch.logsumexp() for those t1,t2, and t3 tensors? Or is there any way to make it numerical stable?


title: LSTM to Predict Pattern 010101... Understanding Hidden State

body:
I did a quick experiment to see if I could understand what the hidden state in an LSTM does...
I tried to make an LSTM predict a sequence of [1,0,1,0,1...] based off an input sequence of X with X[0] = 1 and the remainder as random noise.
X = [1, randFloat, randFloat, randFloat...]
label = [1, 0, 1, 0...]

In my head, the model would understand:

The inputs X mean nothing, or at least very little (as it's noise) - so it'd discard these values for the most part
Solely the hidden state from the previous sequence/timestep n would be used to predict the next timestep n+1... [1, 0, 1, 0...]
I also set X[0] = 1 so the first initial in an attempt to guide the net to predicting 1 on the first item (which it does)

So, this didn't work. In theory, should it not? Can you someone explain?
It essentially never converges, and is on the cusp of guessing between 0 or 1

## Code
import os
import numpy as np
import torch

from torchvision import transforms
from torch import nn
from sklearn import preprocessing
from util import create_sequences
import torch.optim as optim

Create some fake data
sequence_1 = torch.tensor(np.random.uniform(size=50)).float().detach()
sequence_1[0] = 1
sequence_2 = torch.tensor(np.random.uniform(size=50)).float().detach()
sequence_2[0] = 1

labels_1 = np.zeros(50)
labels_1[::2] = 1
labels_1 = torch.tensor(labels_1, dtype=torch.long)
labels_2 = labels_1.clone()

training_data = [sequence_1, sequence_2]
label_data = [labels_1, labels_2]

Create simple LSTM Model
class LSTM(nn.Module):
    def __init__(self, input_dim, hidden_dim, output_dim):
        super(LSTM, self).__init__()
        self.lstm = nn.LSTM(input_dim, hidden_dim)
        self.fc = nn.Linear(hidden_dim, output_dim)

    def forward(self, seq):
        lstm_out, _ = self.lstm(seq.view(len(seq), 1, -1))
        out = self.fc(lstm_out.view(len(seq), -1))
        out = F.log_softmax(out, dim=1)
        return out

We try to overfit on the dataset
INPUT_DIM = 1
HIDDEN_DIM = 6
model = LSTM(INPUT_DIM, HIDDEN_DIM, 2)

loss_function = nn.NLLLoss()
optimizer = optim.SGD(model.parameters(), lr=0.1)

for epoch in range(500):
    for i, seq in enumerate(training_data):
        labels = label_data[i]
        model.zero_grad()
        scores = model(seq)
        loss = loss_function(scores, labels)
        loss.backward()
        print(loss)

        optimizer.step()

with torch.no_grad():
    seq_d = training_data[0]
    tag_scores = model(seq_d)
    for score in tag_scores:
        print(np.argmax(score))


title: PyTorch Detecto Model: tensor incompatibiliy in predicition for a pretrained model

body:
Trying to train a very simple model and do a
image-prediction with the following code
for pytorch detecto:
from detecto import core, utils, visualize

dataset = core.Dataset('images/')

model = core.Model(['rect'])

model.fit(dataset)

modelName = 'model_weights_simpleRect.pth'
model.save(modelName)

image = utils.read_image('simple_image_to_test.jpg')
predictions = model.predict(image)

This leads to the following output:
Epoch 1 of 10
Begin iterating over training dataset
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:12<00:00,  1.56it/s]
Epoch 2 of 10
Begin iterating over training dataset
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:11<00:00,  1.80it/s]
Epoch 3 of 10
Begin iterating over training dataset
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:11<00:00,  1.80it/s]
Epoch 4 of 10
Begin iterating over training dataset
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:11<00:00,  1.79it/s]
Epoch 5 of 10
Begin iterating over training dataset
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:11<00:00,  1.79it/s]
Epoch 6 of 10
Begin iterating over training dataset
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:11<00:00,  1.80it/s]
Epoch 7 of 10
Begin iterating over training dataset
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:11<00:00,  1.78it/s]
Epoch 8 of 10
Begin iterating over training dataset
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:11<00:00,  1.80it/s]
Epoch 9 of 10
Begin iterating over training dataset
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:11<00:00,  1.78it/s]
Epoch 10 of 10
Begin iterating over training dataset
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:11<00:00,  1.80it/s]
Traceback (most recent call last):
  File "train_simpleRect_and_predict.py", line 15, in <module>
    predictions = model.predict(image)
  File "/home/std/anaconda3/envs/dri/lib/python3.7/site-packages/detecto/core.py", line 338, in predict
    preds = self._get_raw_predictions(images)
  File "/home/std/anaconda3/envs/dri/lib/python3.7/site-packages/detecto/core.py", line 294, in _get_raw_predictions
    preds = self._model(images)
  File "/home/std/anaconda3/envs/dri/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/std/anaconda3/envs/dri/lib/python3.7/site-packages/torchvision/models/detection/generalized_rcnn.py", line 52, in forward
    detections, detector_losses = self.roi_heads(features, proposals, images.image_sizes, targets)
  File "/home/std/anaconda3/envs/dri/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/std/anaconda3/envs/dri/lib/python3.7/site-packages/torchvision/models/detection/roi_heads.py", line 550, in forward
    boxes, scores, labels = self.postprocess_detections(class_logits, box_regression, proposals, image_shapes)
  File "/home/std/anaconda3/envs/dri/lib/python3.7/site-packages/torchvision/models/detection/roi_heads.py", line 474, in postprocess_detections
    pred_boxes = self.box_coder.decode(box_regression, proposals)
  File "/home/std/anaconda3/envs/dri/lib/python3.7/site-packages/torchvision/models/detection/_utils.py", line 168, in decode
    rel_codes.reshape(sum(boxes_per_image), -1), concat_boxes
RuntimeError: cannot reshape tensor of 0 elements into shape [0, -1] because the unspecified dimension size -1 can be any value and is ambiguous

How can I get more detailled information about the model dimension, where exactly
in the model the tensor incompatibilies occur and how to fix it?
Add. info: I used the same code with other data and it worked.
Thank you!


title: Can't access Tensorflow & Pytorch from Jupyter notebook or Lab?

body:
I can't access tensorflow or pytorch from Jupyter notebook or lab. My OS is macOS big sur . I installed python@3.8 using homebrew. If I try to run this:
import tensorflow as tf

import numpy as np

import matplotlib.pyplot as plt

print(tf.__version__)

simple code on Jupyter notebook , it is showing this output:
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-1-3c075791d0d7> in <module>
      1 # TensorFlow and tf.keras
----> 2 import tensorflow as tf
      3
      4 # Helper libraries
      5 import numpy as np

ModuleNotFoundError: No module named 'tensorflow'

I created separate virtual environment and install tensorflow on it. But I installed jupyter notebook & lab outside virtual environment. Is there anyway to access tensorflow & pytorch from jupyter notebook or lab without using Anaconda?


title: Pytorch weights sampling on TrainLoader

body:
# sklearn's dataloader, useful for large dataset
# it also does data shuffling
trainDT = CustomDataset(X_train, y_train)
testDT = CustomDataset(X_test, y_test)
global trainloader

#Calculate weights for samples (more emphasis for more recent samples)
#Weightage of 2 for the last 1972 data instances
sample_weights = []
sample_weights = DataFrame(sample_weights)
#sample_weights = torch.tensor(sample_weights)

sample_weights[5000:6971] = 2
sample_weights[0:5000] = 1

#Assign weights to all samples
sample_weights_all = sample_weights[trainDT]

#Add in sample weights for index samples
weighted_sampler = WeightedRandomSampler(
    weights=sample_weights,
    num_samples=len(sample_weights_all),
    replacement=False
)

trainloader = DataLoader(trainDT, batch_size=500, shuffle=True,sampler=weighted_sampler)
testloader = DataLoader(testDT, batch_size=500,shuffle=False)

Hi , What I am trying to do is to assign 2 X weightage for the last 1972 samples as seen in the code below. The following error occurred. Is anyone able to advise how do I get this through. The rationale is to place more emphasis on more recent samples (last 1972 samples).
Traceback (most recent call last):
  File "OptunaTest", line 615, in <module>
    n_instances, n_features, scores = run_analysis()
  File "OptunaTest", line 377, in run_analysis
    sample_weights_all = sample_weights[trainDT]
  File "/home/shar/.local/lib/python3.7/site-packages/pandas/core/frame.py", line 2995, in __getitem__
    indexer = self.columns.get_loc(key)
  File "/home/shar/.local/lib/python3.7/site-packages/pandas/core/indexes/base.py", line 2899, in get_loc
    return self._engine.get_loc(self._maybe_cast_indexer(key))
  File "pandas/_libs/index.pyx", line 107, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 131, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 1607, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 1614, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: <__main__.CustomDataset object at 0x7f0399e82c88>


title: what is the equivalent sytax for `Tensor.grad` in Tensorflow 2.0

body:
In the Pytorch, we can access the gradient of a variable  x by
z.grad

What is the same sytax in Tensorflow 2. My goad is to cut the gradient. Here is Pytorch code
if z.grad > 1000:
    z.grad = 10

Can tensorflow 2 apply the same functions?
Thanks


title: Overriding stem method in video classification model to change filter channel

body:
I am trying to use torchvision’s video classification models (R3D, R(2+1)D, MC18) but my data is single channel (grey scale video), and these model uses 3 channel input, in that case I am trying to override the stem class , can someone please confirm if what I am doing is correct?
For R3D18 and MC18  stem=BasicStem
class BasicStemModified(nn.Sequential):

    def __init__(self):
        super(BasicStemModified, self).__init__(
            nn.Conv3d(1, 45, kernel_size=(7, 7, 1),  #changing filter to 1 channel input
                      stride=(2, 2, 1), padding=(3, 3, 0),
                      bias=False),
            nn.BatchNorm3d(45),
            nn.ReLU(inplace=True),

            nn.Conv3d(45, 64, kernel_size=(1, 1, 3),
                      stride=(1, 1, 1), padding=(0, 0, 1),
                      bias=False),
            nn.BatchNorm3d(64),
            nn.ReLU(inplace=True))

model = torchvision.models.video.mc3_18(pretrained=False)

model.stem = BasicStemModified() #here assigning the modified stem

model.fc = nn.Sequential(
    nn.Dropout(0.3),
    nn.Linear(model.fc.in_features, num_classes)
)

model.to('cuda:0')

For R(2+1)D:
#For R(2+1)D model `stem=R2Plus1dStem`

class R2Plus1dStemModified(nn.Sequential):
    """R(2+1)D stem is different than the default one as it uses separated 3D convolution
    """
    def __init__(self):
        super(R2Plus1dStemModified, self).__init__(
            nn.Conv3d(3, 45, kernel_size=(1, 7, 7),   #changing filter to 1 channel input
                      stride=(1, 2, 2), padding=(0, 3, 3),
                      bias=False),
            nn.BatchNorm3d(45),
            nn.ReLU(inplace=True),
            nn.Conv3d(45, 64, kernel_size=(3, 1, 1),
                      stride=(1, 1, 1), padding=(1, 0, 0),
                      bias=False),
            nn.BatchNorm3d(64),
            nn.ReLU(inplace=True))

model = torchvision.models.video.mc3_18(pretrained=False)

model.stem = R2Plus1dStemModified() #here assigning the modified stem

model.fc = nn.Sequential(
    nn.Dropout(0.3),
    nn.Linear(model.fc.in_features, num_classes)
)

model.to('cuda:0')


title: Balanced KMeans: number of sizes provided must be greater or equal to the number of dimensions in the tensor

body:
I'm been trying to make a balanced clusters for my masther thesis using Google Colab (because I don't have a GPU) and balanced k-means Library (https://pypi.org/project/balanced-kmeans/), this is the process that I made:

Install the balanced k-means Library in Google Colab using this command:
!pip install balanced_kmeans

Import a CSV file to my Google Colab instance:
from google.colab import files

subido=files.upload()

Create a Pandas dataframe
import pandas as pd
import io
df = pd.read_csv(io.BytesIO(subido['prueba.csv']))

Then, changed the string data type to integer in the columns than I going to use:
df["bytes"] = pd.to_numeric(df["bytes"],errors='coerce')
df["bytes"] = df["bytes"].fillna(0)

df["paquete"]= pd.to_numeric(df["paquete"],errors='coerce')
df["paquete"]= df["paquete"].fillna(0)

Once I changed the data type, proceed to create a tensors (in this case 2 colums from the pandas dataframe) then stacked those tensors:
import torch
device = 'cuda'

A= torch.cuda.IntTensor(df['bytes'].values,device=device)

B=torch.cuda.IntTensor(df['paquete'].values,device=device)

prueba=[A,B]

X = (torch.stack(prueba,dim=1))

Finally I tried to create the clusters:
from balanced_kmeans import kmeans_equal
N=X.shape[0]
num_clusters=100
device='cuda'
cluster_size=N//num_clusters
choices,centers=kmeans_equal(X,num_clusters=num_clusters,cluster_size=cluster_size)

But unfourtanelly I got this error:

RuntimeError Traceback (most recent call last)
in ()
16 #print (W)
17
---> 18 choices, centers = kmeans_equal(X, num_clusters=num_clusters,cluster_size=cluster_size)

1 frames

/usr/local/lib/python3.6/dist-packages/balanced_kmeans/init.py in initialize(X, num_clusters)
16 indices = torch.empty(X.shape[:-1], device=X.device, dtype=torch.long)
17 for i in range(bs):
---> 18 indices = torch.randperm(num_samples, device=X.device)
19 initial_state = torch.gather(X, 1, indices.unsqueeze(-1).repeat(1, 1,X.shape[-1])).reshape(bs,num_clusters, -1, X.shape[-1]).mean(dim=-2)
20 return initial_state

RuntimeError: expand(torch.cuda.LongTensor{[20839]}, size=[]): the number of sizes provided (0) must be greater or equal to the number of dimensions in the tensor (1)

I don't know if this error is caused by the tensor X having a 1 dimention, and in the library the initialization function create a empty tensor with zero dimension (I took this code from initialize method in the init.py):
indices=torch.empty(X.shape[:-1], device=X.device, dtype=torch.long)
for i in range(bs):
  indices[i] = torch.randperm(num_samples, device=X.device)

So Am I doing right? (I'm new using tensors) or there is a bug in the initialization function?
I already checked the solution from PyTorch: The number of sizes provided (0) must be greater or equal to the number of dimensions in the tensor (1) but doesn't work for me becauseI don't need a scalar value.


title: Manually setting parameters' data in PyTorch model

body:
I train ResNet34 model on CIFAR-10 dataset. I'm simulating federated learning: I have one server model and multiple local models; all training happens on learning models and sometimes I synchronize them with the server model. Since I store models as lists of trainable parameters, synchronization is the following:
def communicate_to_server(*, local_data, prev_local_data, server_data, n_machines):
  for i in range(len(local_data)):
    server_data[i] += (local_data[i] - prev_local_data[i]) / n_machines
    local_data[i][:] = server_data[i]
    prev_local_data[i][:] = server_data[i]

So, the server model collects updates and shares its model with a local model.
To perform training for the local model, I have an instance of ResNet34 network. When I want to make an SGD step, I set model parameters to local model:
def set_params(model, data):
  params = [param for param in model.parameters() if param.requires_grad]
  for param, d in zip(params, data):
    param.data = d

And then do usual loss-backpropagation function stuff (I'm using optim.SGD with 0 momentum).
My question: Am I allowed to do these operations? I.e. am I allowed to simply substitute param.data with new values? Am I allowed to do assignments like local_data[i][:] = server_data[i] (I tried to insert .detach() in all possible places, it didn't seem to affect anything)? Can I be sure that I don't create any weird computational graph connections or don't break existing ones?
While I don't have direct evidence that something is wrong, I'm concerned that my testing accuracy is much lower compared to what the usual SGD achieves (77% vs 90%), while training accuracy is higher (100% vs 99%).


title: Why do I always get the same value as the result in a CNN in pytorch?

body:
Here is my code
dataset = pd.read_csv('augmented_data.csv')
dataset = dataset.sample(frac=1)

class ConvNet(nn.Module):
def __init__(self):
    super(ConvNet, self).__init__()
    self.conv1 = nn.Conv2d(3,6,5)
    self.pool = nn.MaxPool2d(2,2)
    self.conv2 = nn.Conv2d(6,16,5)
    self.fc1 = nn.Linear(1024144, 120)
    self.fc2 = nn.Linear(120, 84)
    self.fc3 = nn.Linear(84,1)

def forward(self, x):
    x = self.pool(F.relu(self.conv1(x)))
    x = self.pool(F.relu(self.conv2(x)))
    x = x.view(-1, 1024144)
    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(x))
    print(x)
    x = self.fc3(x)
    return x

files_read = 0
preprocess = transforms.Compose([
transforms.Resize(1024),
transforms.ToTensor(),
transforms.Normalize((0.5, 0.5, 0.5),(0.5, 0.5, 0.5))])

# device = torch.device('cuda' if torch.cuda.is_available else 'cpu')
device = torch.device('cpu')

# model = ConvNet().to(device)

criterion = torch.nn.MSELoss(reduction='sum')
optimizer = optim.Adam(model.parameters(), lr=0.001)

results = []
for index, row in dataset.iterrows():
    try:
      image = load_img('padded_images/' + row['image_name'] +'.jpg')
    except:
      image = load_img('augmented_images/' + row['image_name'] +'.jpeg')

    files_read += 1

    input_tensor = preprocess(image)
    input_batch = input_tensor.unsqueeze(0).to(device)

    if files_read <= 80 * len(dataset) // 100:
        output = model(input_batch)
        optimizer.zero_grad()

        y = torch.tensor([[float(row['target'])]]).to(device)
        loss = criterion(output, y)

        loss.backward()
        optimizer.step()
    else:
        model.eval()
        output = model(input_batch)
        results.append([1.0 if output[0][0].double() > 0.5 else 0, float(row['target'])])

So i am using pytorch CNN to classify 60k images in 2 classes. When i print the output after the model has trained, whatever the image as input, the ouput is always "tensor([[0.6384]], grad_fn=)". Always the same value. So it predicts only 1 (because it's greater than 0.5). The thing is, when i print the ouput while training, the results vary (16, 1 , 0, 4 ,0.6 etc) but when i print the output (with the same model but not trained) the results don't vary that much (0.5, 0.51, 0.49 0.52, 0.55). So I think it's safe to say that it is converging to a single value. I just don't know why. what could i do differently?


title: How to implement fractionally strided convolution layers in pytorch?

body:
Before everything, I searched google and StackOverflow but I do not find any similar questions so here I propose a new one.
I'm interested in  this paper and want to implement this SGAN for my project. The paper mentioned that its generator network is composed of "a stack of fractionally strided convolution layers", I found two different ways of implementing this in pytorch, one is:
torch.nn.Sequential(
    # other layers...
    torch.nn.ConvTranspose2d(),
    # other layers...
)

the other way is:
torch.nn.Sequential(
    # other layers...
    torch.nn.Upsample(scale_factor=2),
    torch.nn.Conv2D(),
    # other layers...
)

So, my question is, which is the better implementation of fractionally strided conv layer, or am I understanding something completely wrong?
Thanks in advance.
P.S, I found the second implementation here, in line 87 - 88.


title: How to put a list of tensor into a new tensor in pytorch?

body:
How can I merge a list of tensor into a single tensor?
I have a list of 64 images (128x128, RGB), and I want to create a batch of images with this size: torch.Size([64, 3, 128, 128])
I tried with torch.stack() but I can't get all elements of the list without iterating it.
How can I do it?
lfw_dataset = ImageFolder(os.path.join(root_dir), transform=test_transform)
lfw_dataset_train, lfw_dataset_val = torch.utils.data.random_split(lfw_dataset_train1, [train_values, validation_values])
train_loader = DataLoader(lfw_dataset_train, batch_size, num_workers=4, shuffle=True)
val_loader   = DataLoader(lfw_dataset_val,   batch_size, num_workers=4, shuffle=False)
# Define dictionary of loaders
loaders = {"train": train_loader,
           "val": val_loader}

positive_list= []
negative_list= []
positive_img = []
negative_img = []
for i, (input, labels) in enumerate(loaders["train"]):
  for num, x in enumerate(labels):
    target = x.item()
    k = [i for i, (imgs, label_pos) in enumerate(lfw_dataset.imgs) if label_pos==target]
    group_pos = (target, k)
    positive_list.append(group_pos)
    for i, (imgs, label_neg) in enumerate(lfw_dataset.imgs):
       if label_neg!=target:
         j = [i]
       break
    group_neg = (target, j)
    negative_list.append(group_neg)
    anchor_img=input[num]
    positive = random.choice(positive_list[num][1])
    negative = random.choice(negative_list[num][1])
    positive_img.append(lfw_dataset[positive][0])
    negative_img.append(lfw_dataset[negative][0])


title: DialoGPT model generating special symbols as chatbot reply

body:
I am using the DialoGPT model from Microsoft to build a chatbot. To fine-tune the model on my dataset, I am using this blog https://towardsdatascience.com/make-your-own-rick-sanchez-bot-with-transformers-and-dialogpt-fine-tuning-f85e6d1f4e30 whose reference is given on the DialoGPT GitHub page.
The fine-tuning done in this blog is for a fixed number of steps. Instead of keeping a fixed number of steps, I want my conversation to end only when my user says "bye" or a similar word. If the user says "bye" in say 6 or 10 steps, the fine-tuned model is working fine. However, if the conversation goes long, the model starts emitting special characters. Attached is the screenshot of the same.
Does anybody have any idea what causes a language model to emit special characters instead of words? I saw similar comments on this blog by others too but no solution.
Thank you in advance.
Screenshot of the conversation


title: How to import module only when I need to use it?

body:
I have a project which has optional dependencies, depending on the use case it may need torch library.  PyTorch takes a lot of memory, it's not optimal to install it when you don't need to use it. In my project, there are parts of the code which use torch, but most of the logic does not need it. I'm trying to implement optional import so that when I need to use torch I use it, here is my idea:
def _lazy_import_torch():
    import torch

def main():
    _lazy_import_torch()
    print(f'You have torch: {torch.__version__}')

if __name__ == '__main__':
    main()

When I run this code it appears that torch was not imported:
$ python main.py
Traceback (most recent call last):
  File "main.py", line 9, in <module>
    main()
  File "main.py", line 6, in main
    print(f'You have torch: {torch.__version__}')
NameError: name 'torch' is not defined

torch is installed in my environment.
I was wondering why it did not work and how can I make it works, please help me.


title: How to get value in order with a small average value

body:
I've got a data frame like
x  y    cluster
0  112  4
0  113  4
1  111  4

and I'll get locations from this code:
for n in range(0,9):
...
    location = np.array(cluseter_location )

I want to sort it out in the order of column 'cluster' with small columns 'y' means, so I tried:
for n in range(0,9):
    cluster_ = data2[data2['cluster_id']== n]

    ...


title: Convolutional AE always overfitting time series - what’s wrong?

body:
I've build a CAE for anomaly detection in time series, but it is always overfitting. I've tried data augmentation, short/long inputvector, dropout rates... I don't know what I'm doing wrong, may be you can give me a hint?
This is my architecture
class CAE(nn.Module):
    def __init__(self):
        super(CAE, self).__init__()

        self.encoder = Encoder()
        self.decoder = Decoder()

        self.encoder.to(get_device())
        self.decoder.to(get_device())

    def forward(self, x):
        x = self.encoder(x)
        x = self.decoder(x)
        return x

class Encoder(nn.Module):
    def __init__(self):
        super(Encoder, self).__init__()

        self.conv_enc = nn.Sequential(
            nn.AlphaDropout(p=0.5),
            nn.Conv1d(
                in_channels=1,
                out_channels=16,
                kernel_size=5,
                stride=2,
                padding=0
            ),
            nn.SELU(),
            nn.AlphaDropout(p=0.5),
            nn.Conv1d(
                in_channels=16,
                out_channels=8,
                kernel_size=5,
                stride=2,
                padding=0
            ),
            nn.SELU()
        )

        self.conv_enc.apply(self.init_weights)

    def init_weights(self, m):
        if type(m) == nn.Conv1d or type(m) == nn.ConvTranspose1d:
            torch.nn.init.xavier_normal_(m.weight)
            m.bias.data.fill_(0.01)

    def forward(self, x):
        x = self.conv_enc(x)
        return x

class Decoder(nn.Module):
    def __init__(self):
        super(Decoder, self).__init__()

        self.conv_dec = nn.Sequential(
             nn.ConvTranspose1d(
                in_channels=8,
                out_channels=16,
                kernel_size=5,
                stride=2,
                padding=0
            ),
            nn.SELU(),
            #nn.AlphaDropout(p=0.5),
            nn.ConvTranspose1d(
                in_channels=16,
                out_channels=1,
                kernel_size=6,
                stride=2,
                padding=0
            )
        )

        self.conv_dec.apply(self.init_weights)

    def init_weights(self, m):
        if type(m) == nn.Conv1d or type(m) == nn.ConvTranspose1d:
            torch.nn.init.xavier_normal_(m.weight)
            m.bias.data.fill_(0.01)

    def forward(self, x):
        x = self.conv_dec(x)
        return x

I'm using following setup

optimizer - AdamW with lr=0.0001
scheduler - OneCyleLR
Normalized Data with MinMaxScaler()
loss - L1Loss
batchsize - 250
trainingsize - 12000 sample
inputsize - 800

Loss
Acc
Thank you guys
Append
Inputsize = 140
Smaller Area of Time Series
Accuracy
Loss


title: Dot product along a dimension

body:
I have two tensors of shape [B, 3 , 240, 320] where B represents the batch size 3 represents the channels, 240 the height(H), 320 the width(W).
I need to find the dot product along the channels dimension(3 channels) thus the resulting tensor would be of shape [B, 1 , 240, 320].
My tensors have float32 elements in gpu(cuda to backprop).
Can you all please suggest how I can do that?
Thanks!
More clarification:
Let's say we have B=10, H=100, W=200.
So from the above would be common for both the first and seconds tensors. If we keep B, H, W constant we get a 1D vector as the resultant tensor(with 3 elements). I need to take the dot product of these two vectors. Thus the resultant tensor is of dimension [B, 1, 240, 320]


title: Pytorch loss is not zero but gradients are zero

body:
I want to get loss of each sub-module in the network, so every sub-module output its loss values.
Below is my code.
import torch
import torch.nn as nn
from torch.optim import SGD

class m1(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1,1,3)
        self.conv2 = nn.Conv2d(1,1,3)
        self.loss = nn.SmoothL1Loss()

    def forward(self, x):
        x = self.conv1(x)
        x = self.conv2(x)

        return dict(m1loss=self.loss(x, x + 1.1)), x

class m2(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1,1,3)
        self.relu = nn.ReLU()
        self.conv2 = nn.Conv2d(1,1,3)
        self.loss = nn.SmoothL1Loss()

    def forward(self, x):
        x = self.conv1(x)
        x = self.relu(x)
        x = self.conv2(x)
        return dict(m2loss=self.loss(x, x + 2.31)), x

class m(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1,1,3)
        self.conv2 = nn.Conv2d(1,1,1)
        self.relu = nn.ReLU()
        self.m1 = m1()
        self.m2 = m2()
        self.losses = dict()
        self.loss = nn.SmoothL1Loss()

    def forward(self, x):
        x = self.conv1(x)
        x = self.relu(x)
        x = self.conv2(x)
        loss, x = self.m1(x)
        self.losses.update(loss)
        loss, x = self.m2(x)
        self.losses.update(loss)
        self.losses.update(mloss=self.loss(x, x + 1.21))
        return self.losses

net = m()
sgd = SGD(net.parameters(), lr=10)

x = torch.rand((1,1,24,24))
out = net(x)
sgd.zero_grad()
l = sum(v for k,v in out.items())
print('Loss: ', l)
l.backward()
sgd.step()

for p in net.parameters():
    print('grad: ', p.grad)

I checked if the weights of the net is updated, but it was not updated at all.
When I printed the loss value and all gradients,
the loss value=3.12 but all gradients is zero.
What is wrong?
Loss:  tensor(3.1200, grad_fn=<AddBackward0>)
grad:  tensor([[[[0., 0., 0.],
          [0., 0., 0.],
          [0., 0., 0.]]]])
grad:  tensor([0.])
grad:  tensor([[[[0.]]]])
grad:  tensor([0.])
grad:  tensor([[[[0., 0., 0.],
          [0., 0., 0.],
          [0., 0., 0.]]]])
grad:  tensor([0.])
grad:  tensor([[[[0., 0., 0.],
          [0., 0., 0.],
          [0., 0., 0.]]]])
grad:  tensor([0.])
grad:  tensor([[[[0., 0., 0.],
          [0., 0., 0.],
          [0., 0., 0.]]]])
grad:  tensor([0.])
grad:  tensor([[[[0., 0., 0.],
          [0., 0., 0.],
          [0., 0., 0.]]]])
grad:  tensor([0.])


title: How to use view in 4-dimensional neural network

body:
I'm new to Neural Networks, so I hope u will forgive me if this is really basic stuff.
So far i managed to use view once in my first very simple project and now i tried to understand something more complex.
I found this code on github and I'm trying to understand it.
This is how network is built.
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 32, 3, 1)
        self.conv2 = nn.Conv2d(32, 64, 3, 1)
        self.dropout1 = nn.Dropout(0.25)
        self.dropout2 = nn.Dropout(0.5)
        self.fc1 = nn.Linear(9216, 128)
        self.fc2 = nn.Linear(128, 10)

    def forward(self, x):
        x = self.conv1(x)
        x = F.relu(x)
        x = self.conv2(x)
        x = F.relu(x)
        x = F.max_pool2d(x, 2)
        x = self.dropout1(x)
        x = torch.flatten(x, 1)
        x = x.view(-1, 64 * 12 * 12)
        x = self.fc1(x)
        x = F.relu(x)
        x = self.dropout2(x)
        x = self.fc2(x)
        output = F.log_softmax(x, dim=1)
        return output

I'm trying to get the image of number network failed to recognize, yet I don't really know how to use view here.
def test(model, device, test_loader):
    model.eval()
    test_loss = 0
    correct = 0
    mistakes = 0
    images, labels = next(iter(test_loader))
    img = images[0].view(1, 784)

    with torch.no_grad():
        logps = model(img)
        for data, target in test_loader:
            data, target = data.to(device), target.to(device)
            output = model(data)
            test_loss += F.nll_loss(output, target, reduction='sum').item()
            pred = output.argmax(dim=1, keepdim=True)
            correct += pred.eq(target.view_as(pred)).sum().item()

    test_loss /= len(test_loader.dataset)
    print('\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\n'.format(
        test_loss, correct, len(test_loader.dataset),
        100. * correct / len(test_loader.dataset)))

I couldn't find usefull examples so I would be gratefull for any kind of hint.
Every time I modify it I get one of these two errors.
builtins.RuntimeError: Expected 4-dimensional input for 4-dimensional weight [32, 1, 3, 3], but got 2-dimensional input of size [1, 784] instead
builtins.RuntimeError: shape '[1, 784, 3, 3]' is invalid for input of size 784
Thanks in advance for any kind of help.