title: Load GPU-trained Model to CPU in Pytorch

body:
Here are the piece of code when I try to load a gpu-trained model to cpu:
model_conv.load_state_dict(torch.load(model_file, map_location='cpu'))
model_conv = model_conv.cpu()

and the error message is:
Traceback (most recent call last):
  File "prediction.py", line 269, in <module>
    model_conv.load_state_dict(torch.load(resume_file, map_location='cpu'))
  File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/torch/serialization.py", line 229, in load
    return _load(f, map_location, pickle_module)
  File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/torch/serialization.py", line 377, in _load
    result = unpickler.load()
  File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/torch/serialization.py", line 348, in persistent_load
    data_type(size), location)
  File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/torch/serialization.py", line 246, in restore_location
    result = map_location(storage, location)
TypeError: 'str' object is not callable

My pytorch version is 0.1.12_1. Any idea how to fix this? I have checked how to load the gpu trained model into the cpu? but the solution seems not work in my case.
Any suggestion is appreciated!
---update---
Without map_location argument, the error msg is
Traceback (most recent call last):
  File "prediction.py", line 268, in <module>
    model_conv.load_state_dict(torch.load(resume_file))
  File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/torch/serialization.py", line 229, in load
    return _load(f, map_location, pickle_module)
  File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/torch/serialization.py", line 377, in _load
    result = unpickler.load()
  File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/torch/serialization.py", line 348, in persistent_load
    data_type(size), location)
  File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/torch/serialization.py", line 85, in default_restore_location
    result = fn(storage, location)
  File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/torch/serialization.py", line 67, in _cuda_deserialize
    return obj.cuda(device_id)
  File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/torch/_utils.py", line 57, in _cuda
    with torch.cuda.device(device):
  File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/torch/cuda/__init__.py", line 124, in __enter__
    _lazy_init()
  File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/torch/cuda/__init__.py", line 84, in _lazy_init
    _check_driver()
  File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/torch/cuda/__init__.py", line 58, in _check_driver
    http://www.nvidia.com/Download/index.aspx""")
AssertionError:
Found no NVIDIA driver on your system. Please check that you
have an NVIDIA GPU and installed a driver from
http://www.nvidia.com/Download/index.aspx


title: Transformers get named entity prediction for words instead of tokens

body:
This is very basic question, but I spend hours struggling to find the answer. I built NER using Hugginface transformers.
Say I have input sentence
input = "Damien Hirst oil in canvas"

I tokenize it to get
tokenizer = transformers.BertTokenizer.from_pretrained('bert-base-uncased')
tokenized = tokenizer.encode(input) #[101, 12587, 7632, 12096, 3514, 1999, 10683, 102]

Feed tokenized sentence to the model to get predicted tags for the tokens
['B-ARTIST' 'B-ARTIST' 'I-ARTIST' 'I-ARTIST' 'B-MEDIUM' 'I-MEDIUM'
 'I-MEDIUM' 'B-ARTIST']

prediction comes as output from the model. It assigns tags to different tokens.
How can I recombine this data to obtain tags for words instead of tokens? So I would know that
"Damien Hirst" = ARTIST
"Oil in canvas" = MEDIUM


title: PyTorch import broken, unable to pip install

body: So all of a sudden my code broke. Error: Unable to import torch, No module named torch. So I attempt to install torch; error, ModuleNotFoundError: No module named 'tools.nnwrap'. I delete my venv and re-create it, same thing. I try it outside the venv, same issue. I look up the issue, apparently I should go here. I do, try several of the install commands, with cuda, without, all broken. What do I do? Am on Arch Linux if that's relevant, maybe I broke something.


title: Importing JPG files from a Folder for PyTorch transformation (Another error)?

body:
I just recently posted a question about a file path error in which I thought I resolved... the folder that I have contains .jpg files. For some reason, i'm getting a new error:

RuntimeError: Found 0 files in subfolders of: C:\Users\Lyn\Desktop\UTKFaceDataSet\Train_Set
Supported extensions are: .jpg,.jpeg,.png,.ppm,.bmp,.pgm,.tif,.tiff,.webp

Here is my modified code:
data_dir = "C:\\Users\\Lyn\\Desktop\\UTKFaceDataSet"
train_dir = data_dir + '\\Train_Set'
test_dir = data_dir + '\\Test_Set'

training_transforms = transforms.Compose([transforms.Resize(100),
                                          transforms.ToTensor()])
testing_transforms = transforms.Compose([transforms.Resize(256),
                                         transforms.ToTensor()])

#Load the datasets with ImageFolder
training_dataset = datasets.ImageFolder(train_dir, transform=training_transforms)
testing_dataset = datasets.ImageFolder(test_dir, transform=testing_transforms)


title: Installing a CPU-based library version on a GPU enabled machine

body: I want to install a CPU version of PyTorch on a server which is equipped with a nVIDIA Tesla GPU. Will it work or can I only install a GPU version (with CUDA) on this server for PyTorch to function properly?


title: How can only giving number of channels and no height and width to my convolutional neural network work?

body:
Hello I am a bit new to the deep learning community and I have been really fed up with how to feed in data throught a neural network. So I was doing the sentdex pytorch series and I was learning convnets. He was using the cats and dogs dataset of microsoft on kaggle. He had resized the image to 50 by 50 and turned them into grayscale. If you want to see the video to answer my question here it is -
https://pythonprogramming.net/convnet-model-deep-learning-neural-network-pytorch/

So a few thoughts came in my mind while watching the video. The input he passed is only the colour channel of the image -

At once at seeing the input he entered it came in my mind why is he only passing the number of channels which is a grayscale image. When a conv2d takes 3 inputs.

And I mean it litterally works. I tried researching a bit but no where I found a  good explaination for the input shape that is being fed in here

So I have 2 thoughts and questions about this -

So does that line mean that the convolutional neural network will
only take in an image that is grayscale and is of any height and
width and if so please tell how to limit the dimensions like that to
make our cnn only accept a input shape of (50, 50, 1).
And if not then please explain what does it mean, and how we can make
it accept any input.


title: Index multidimensional torch tensor by another multidimensional tensor

body:
I have a tensor x in pytorch let's say of shape (5,3,2,6) and another tensor idx of shape (5,3,2,1) which contain indices for every element in first tensor. I want a slicing of the first tensor with the indices of the second tensor. I tried x= x[idx] but I get a weird dimensionality when I really want it to be of shape (5,3,2) or (5,3,2,1).
I'll try to give an easier example:
Let's say
x=torch.Tensor([[10,20,30],
                 [8,4,43]])
idx = torch.Tensor([[0],
                    [2]])

I want something like
y = x[idx]

such that 'y' outputs [[10],[43]] or something like.
The indices represent the position of the wanted elements the last dimension. for the example above where x.shape = (2,3) the last dimension are the columns, then the indices in 'idx' is the column. I want this but for more than 2 dimensions


title: Usage of quantized parameters from a pytorch quantized model

body:
Sorry if the question has been answered somewhere, I couldn’t find similar question across the forum so I would want to put my question here, and hope for your answer.
So we have a simple trained model, and applied the static quantization to get quantized model using ‘fbgemm’ as qconfig:
`myModel.qconfig = torch.quantization.get_default_qconfig('fbgemm')`

After this, we have quantized model with weights (int_repr()) exported.
I expect if I create a similar architecture, and import the int represented weight in, I can generate same result per layer as quantized model, but turn out the results are different.
Below is detailed flows:
#Notes: x_batch and x_quant were exported previously with quant model eval to pickle file and reload here for comparison
```

#Flow 1
#using x as input, calculate results through loaded quantized model
#forward: x--> x_quant = self.quant(x) --> f = self.featExt(x_quant)
# featExt definition: self.featExt = nn.Sequential(nn.Conv2d(1, 8,
#                   kernel_size=5, stride=5, bias=False), nn.ReLU())
x_quant_new, f, x_conv, y_hat = quant_net.forward(x_batch[0])
print('using saved quantized model: ')
print('x_quant to compare(int): ', x_quant_new.int_repr())
print('filter to compare(int): ', quant_net.featExt[0].weight().int_repr())
print('output to compare(int): ', f.int_repr())

#Flow 2
#using x_quant as input, calculate conv 2d using pytorch function
conv2d = nn.Conv2d(1, 8, kernel_size=5, stride=5, bias=False)
conv2d.weight.data = my_debug_net.featConv.weight.data
with torch.no_grad():
    conv2d.eval()
    res1 = conv2d(x_quant[0].type(torch.CharTensor))
print('*********using F.conv2d***********')
print('x_quant: ', x_quant[0])
print('filter: ', conv2d.weight.data)
print('F.conv2d Output ', res1)
print('F.relu Output ', F.relu(res1))

```

As a result, I have same print-out for x_quant and filter (input & weights) for 2 flows, but not the output.


title: From where can i get a detailed description of all the methods for the model in pytorch torchvison？

body:
I am a beginner of pytorch ，when i read the source code of a project about mask rcnn .I don't konw from where can i  get  some information about some methods that i  don't understand .The official documentation doesn't seem very detailed?
# load an instance segmentation model pre-trained pre-trained on COCO
model = torchvision.models.detection.maskrcnn_resnet50_fpn(pretrained=True)
# get number of input features for the classifier
in_features = model.roi_heads.box_predictor.cls_score.in_features
# replace the pre-trained head with a new one
model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)

Just like code above ,I could not get detailed information about the " roi_head" attribute from doc of model .From where i can learn about it？


title: Error invalid syntax in pytorch nn module

body:
I have an error in my pytorch code, and i really don't understand why.
  File "<ipython-input-11-89006c750b74>", line 3
    def GaussianBlur(torch.nn.Module):
                          ^
SyntaxError: invalid syntax

The rest of the code is here
import torch.nn as nn
def GaussianBlur(torch.nn.Module):
    def __init__(self, kernel_size, std_dev):
        self.kernel_size = kernel_size
        self.std_dev = std_dev

    def forward(self, img):
        image = np.array(img)
        image_blur = cv2.GaussianBlur(image, self.kernel_size, self.std_dev)
        return Image.fromarray(image_blur)

Anyone knows what's going on? Thanks


title: pytorch LSTM does not overfit single sample

body:
I try to overfit a single time series. Meaning, I try to perform the training on a single (X,Y) pair over and over again. I do this to get an impression of the capabilities of the hyperparameters. But it does not converge. This is the loss plot showing the MSE per iteration after about 800 iterations:

I hoped that the error completely vanishes, but it gets stuck on the plateau, where it still is, as I write this.
The time series has the length 29600 and the RNN maps a single value onto another single value. It consists of an LSTM-Cell with 1 input,200 hidden units and a fully connected Layer mapping onto a single value

My feeling tells me that the model may be not complex enough to fit the sample. But before I try to increase the complexity of the RNN, I have to be sure that my implementation of the training is correct. Maybe I did did not get the usage of autograd correctly. Since this is the first time, I ever tried to train a Neural network, I dont know, how long it should take.

import torch
import torch.nn as nn
import torch.optim as optim
from torch.autograd import Variable
import numpy as np

# the class containing the LSTM
class Sequence(nn.Module):
    def __init__(self):
        self.hidden_state_len = 200
        super(Sequence, self).__init__()
        self.lstm1 = nn.LSTMCell(1, self.hidden_state_len)
        self.linear = nn.Linear(self.hidden_state_len, 1)
        h_t = torch

    def forward(self, input):
        outputs = []
        h_t = torch.zeros(input.size(0), self.hidden_state_len, dtype=torch.double).cuda()
        c_t = torch.zeros(input.size(0), self.hidden_state_len, dtype=torch.double).cuda()

        for i, input_t in enumerate(input.chunk(input.size(1), dim=1)):
            h_t, c_t = self.lstm1(input_t, (h_t, c_t))
            output = self.linear(h_t)
            outputs += [output]

        return torch.cat(outputs, dim=1)

x1 = torch.load("/floyd/input/wav/x1.pt").double().cuda()[0][7400:37000].reshape(1,-1)
y1 = torch.load("/floyd/input/wav/y1.pt").double().cuda()[0][7400:37000].reshape(1,-1)

seq = Sequence()
seq.double()

criterion = nn.MSELoss()
seq = seq.cuda()
device = torch.device("cuda:0")
seq = seq.to(device)

optimizer = optim.Adam(seq.parameters())
starttime = datetime.datetime.now()
i = -1

# training for 4 hours on cloud GPU
while((datetime.datetime.now() - starttime).total_seconds() < 60*60*4) :
    i+=1
    optimizer.zero_grad()
    input = Variable(x1)
    target = Variable(y1)
    out = seq(input)
    loss = criterion(out, target)
    loss.backward()
    optimizer.step()


title: How to save intermediate weights when training a neural network

body:
I am training a neural network using pytorch, and I want to save the weights at every iteration. In other words, I want to create a list that contains all the weights the neural network has had during training.

I did the following:

for i, (images, labels) in enumerate(train_loader):

     (.....code that is used to train the model here.....)

     weight = model.fc2.weight.detach().numpy()
     weights_list.append(weight)

When I then print the entries of the list 'weights_list', I notice that they are all the same, which cannot be true, because I have printed the weights during the training and they do change (and the network actually does learn, so they have to).
My guess is that every entry of the list is actually a pointer to the weights of the network at the moment the list is checked. So:

1) Is my guess correct?
2) How can I solve this problem?

Thank you!


title: Best way to create a tic-tac-toe model with pytorch

body:
I want to create a model that can predict a 9x9 tic-tac-toe winner based on player moves.
Here is an example from my dataset:

..................................x.............................................. 14 L
..............o...................x.................x............................ 67 L
..............o...............x...x.................x..............o............. 2 L

There are 81 fields that can be X or o. Number on the left is the next step of the opponent(opponent is always 0). And letter represents the outcome of the game. I decided to replace all "."  with 0, x with 1 and o with 2. L or W with one hot encoding. I compress my positions with future step and feed it into the model. That's where i have trouble. My train_x dimension is (249561, 80, 1). My sample train is

tensor([0, 0, 0, 0, 1, 2, 1, 2, 0, 0, 2, 0, 0, 0, 0, 2, 1, 0, 0, 0, 1, 2, 2, 2,
        0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 2, 0, 2, 0, 1, 0, 0, 0, 2, 0, 1, 0, 1, 1,
        1, 0, 2, 0, 0, 1, 2, 1, 0, 0, 1, 2, 2, 1, 1, 0, 0, 0, 0, 0, 2, 1, 0, 2,
        0, 0, 1, 0, 0, 0, 2, 0], [67])

I tried this

self.fc = nn.Sequential(
        nn.Linear(80, 4096),
        nn.ReLU(),
        nn.Dropout(p=0.5),
        nn.Linear(4096, 2048),
        nn.ReLU(),
        nn.Dropout(p=0.5),
        nn.Linear(2048, 1),
    )

def forward(self, x):
    logit = self.fc(x)
    return logit

When i am going through training loop I get an error RuntimeError: Expected object of scalar type Float but got scalar type Long for argument #2 'mat1' in call to _th_addmm
I have two questions. Am i doing the data processing right and what model should I use?


title: L1 regularization neural network in Pytorch does not yield sparse solution

body:
I'm implementing a neural network with l1 regularization in pytorch. I directly add the l1 norm penalty to the loss function. The framework is basically the same as Lack of Sparse Solution with L1 Regularization in Pytorch, however, the solution is not sparse no matter how I adjust the tuning parameter. How do I make the solution sparse?

My code is pasted below.

class NeuralNet(nn.Module):
    """
    neural network class, with nn api
    """
    def __init__(self, input_size: int, hidden_size: List[int], output_size: int):
        """
        initialization function
        @param input_size: input data dimension
        @param hidden_size: list of hidden layer sizes, arbitrary length
        @param output_size: output data dimension
        """
        super().__init__()
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.output_size = output_size
        self.relu = nn.ReLU()
        self.softmax = nn.Softmax(dim=1)
        """layers"""
        self.input = nn.Linear(self.input_size, self.hidden_size[0], bias=False)
        self.hiddens = nn.ModuleList([
            nn.Linear(self.hidden_size[h], self.hidden_size[h + 1]) for h in range(len(self.hidden_size) - 1)])
        self.output = nn.Linear(hidden_size[-1], output_size)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        """
        forward propagation process, required by the nn.Module class
        @param x: the input data
        @return: the output from neural network
        """
        x = self.input(x)
        x = self.relu(x)
        for hidden in self.hiddens:
            x = hidden(x)
            x = self.relu(x)
        x = self.output(x)
        x = self.softmax(x)
        return x

def estimate(self, x, y, l1: bool = False, lam: float = None, learning_rate: float = 0.1, batch_size: int =
    32, epochs: int = 50):
    """
    estimates the neural network model
    @param x: training data
    @param y: training label
    @param l1: whether to use l1 norm regularization
    @param lam: tuning parameter
    @param learning_rate: learning rate
    @param batch_size: batch size
    @param epochs: number of epochs
    @return: null
    """
    input_size = x.shape[1]
    hidden_size = [50, 30, 10]
    output_size = 2
    model = NeuralNet(input_size, hidden_size, output_size)
    optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
    trainset = []
    for i in range(x.shape[0]):
        trainset.append([x[i, :], y[i]])
    trainloader = torch.utils.data.DataLoader(trainset, batch_size=batch_size, shuffle=True)
    for e in range(epochs):
        running_loss = 0
        for data, label in trainloader:
            input_0 = data.view(data.shape[0], -1)
            optimizer.zero_grad()
            output = model(input_0.float())
            loss = torch.nn.CrossEntropyLoss()(output, label.squeeze(1))
            if l1:
                if lam is None:
                    raise ValueError("lam needs to be specified when l1 is True.")
                else:
                    for w in model.parameters():
                        if w.dim() > 1:
                            loss = loss + lam * w.norm(1)
            loss.backward()
            optimizer.step()
            running_loss += loss.item()

Doesn't pytorch support sparse solution?


title: Filter out np.nan values from pytorch 1d tensor

body:
I have a 1d tensor looking kinda like this:

import numpy as np
import torch

my_list = [0, 1, 2, np.nan, np.nan, 4]
tensor = torch.Tensor(my_list)

How do i filter out the nan-values, so it becomes a tensor of size 4?


title: How to handle transforms.FiveCrop change in tensor size

body:
I am trying to add transforms.FiveCrop to my model. I understand that this method of data augmentation adds dimensions to my tensor but I am not sure how to handle it. The documentation notes:

This transform returns a tuple of images and there may be a mismatch in the number of inputs and targets your Dataset returns. See below for an example of how to deal with this.

Example
>>> transform = Compose([
>>>    FiveCrop(size), # this is a list of PIL Images
>>>    Lambda(lambda crops: torch.stack([ToTensor()(crop) for crop in crops])) # returns a 4D tensor
>>> ])
>>> #In your test loop you can do the following:
>>> input, target = batch # input is a 5d tensor, target is 2d
>>> bs, ncrops, c, h, w = input.size()
>>> result = model(input.view(-1, c, h, w)) # fuse batch size and ncrops
>>> result_avg = result.view(bs, ncrops, -1).mean(1) # avg over crops

...But I am not sure how to implement this.
Train loop:
    for batch_idx, (data, target) in enumerate(train_loader):

        print(f'Data: {data.shape}, Target: {target.shape}')
        # Before Fivecrop: Data: torch.Size([32, 3, 224, 224]), Target: torch.Size([32])
        # After Fivecrop: Data: torch.Size([32, 5, 3, 224, 224]), Target: torch.Size([32])

        indx_target = target.clone()
        data = data.to(train_config.device)
        target = target.to(train_config.device)

        optimizer.zero_grad()
        output = model(data)
        loss = F.cross_entropy(output, target)
        loss.backward()
        optimizer.step()

Can someone help explain how this is implemented in my train loop and what I am not understanding?
Thanks


title: Tensor assertion error when trying to split data into test and validation set?

body:
I have metabolite data and I'm trying to build a neural network model. I'm getting assert all(tensors[0].size(0) == tensor.size(0) for tensor in tensors) AssertionError error and I can't figure out what I should do to get this to work.
The issue happens here
#Split data into training and validation 70 / 30
data = torch.utils.data.TensorDataset(torch.Tensor(np.array(log_metabol)),
                                                torch.Tensor(labels))
train, val = torch.utils.data.random_split(data, [int(0.7 * len(data)),
                                                              len(data) - int(0.7 * len(data))])

The problem seems to originate from the first row of this block. How do I make this work?


title: migrating from keras to pytorch

body:
i'm newly in pytorch
it is the model with bidirectional lstm, is there any body to tel me what is the equivalent of this two different lstm & bi-lstm model?
i try some torch codes but it do not worked.because this code has suitable acc in keras,i want the exact model in torch and i unfortunately can't find it:(
fist_one:
def lstm_model(embedding_size, vocab_size):

    title = layers.Input(shape=(None,), dtype='int32', name='title')
    body = layers.Input(shape=(None,), dtype='int32', name='body')

    embedding = layers.Embedding(
        mask_zero=True,
        input_dim=vocab_size,
        output_dim=embedding_size,
        weights=[w2v_weights],
        trainable=True
    )

    lstm_1 = layers.LSTM(units=80, return_sequences=True)
    lstm_2 = layers.LSTM(units=80, return_sequences=False)
    emb_title = embedding(title)
    print("question embedding shape", emb_title.shape)
    sum_a = lstm_2(lstm_1(emb_title))
    print("q_output shape", sum_a.shape)

    emb_body = embedding(body)
    print("answer embedding shape", emb_body.shape)

    sum_b = lstm_2(lstm_1(emb_body))
    print("a_output shape", sum_a.shape)

    sim = layers.dot([sum_a, sum_b], axes=1, normalize=True)
    print("qa_similarity shape", sim.shape)

    #     sim = layers.Activation(activation='sigmoid')(sim)
    sim_model = models.Model(
        inputs=[title, body],
        outputs=[sim],
    )
    sim_model.compile(loss='mean_squared_error', optimizer='nadam', metrics=['accuracy'])

    embedding_model = models.Model(
        inputs=[title],
        outputs=[sum_a]
    )
    return sim_model, embedding_model

second one:

def bilstm_model(embedding_size, vocab_size):
    title = layers.Input(shape=(None,), dtype='int32', name='title')
    body = layers.Input(shape=(None,), dtype='int32', name='body')

    embedding = layers.Embedding(
        mask_zero=True,
        input_dim=vocab_size,
        output_dim=embedding_size,
        weights=[w2v_weights],
        trainable=True
    )

    lstm_1 = layers.Bidirectional(LSTM(activation='tanh', dropout=0.2, units=100, return_sequences=True))
    lstm_2 = layers.Bidirectional(LSTM(activation='tanh', dropout=0.2, units=100, return_sequences=False))
    sum_a = lstm_2(lstm_1(embedding(title)))
    sum_b = lstm_2(lstm_1(embedding(body)))

    sim = layers.dot([sum_a, sum_b], axes=1, normalize=True)
    #     sim = layers.Activation(activation='sigmoid')(sim)
    sim_model = models.Model(
        inputs=[title, body],
        outputs=[sim],
    )
    sim_model.compile(loss='mean_squared_error', optimizer='adam', metrics=['accuracy'])

    embedding_model = models.Model(
        inputs=[title],
        outputs=[sum_a]
    )
    return sim_model, embedding_model

i;m llokingo for true answer in weeks:(


title: How to automatically disable register_hook when model is in eval() phase in PyTorch?

body:
I require to update grads of an intermediate tensor variable using the register_hook method. Since the variable isn't a leaf-variable, I require to add the retain_grad() method to it after which, I can use the register_hook method to alter the grads.
score.retain_grad()
h = score.register_hook(lambda grad: grad * torch.FloatTensor(...))

This works perfectly fine during the training (model.train()) phase. However, it gives an error during the evaluation phase (model.eval()).
The error:
File "/home/envs/darthvader/lib/python3.6/site-packages/torch/tensor.py", line 198, in register_hook
    raise RuntimeError("cannot register a hook on a tensor that "
RuntimeError: cannot register a hook on a tensor that doesn't require gradient

How could the model automatically disable the register_hook method when it in eval() phase?


title: Estimating mixture of Gaussian models in Pytorch

body:
I actually want to estimate a normalizing flow with a mixture of gaussians as the base distribution, so I'm sort of stuck with torch.  However you can reproduce my error in my code by just estimating a mixture of Gaussian model in torch.  My code is below:
import numpy as np
import matplotlib.pyplot as plt
import sklearn.datasets as datasets

import torch
from torch import nn
from torch import optim
import torch.distributions as D

num_layers = 8
weights = torch.ones(8,requires_grad=True).to(device)
means = torch.tensor(np.random.randn(8,2),requires_grad=True).to(device)#torch.randn(8,2,requires_grad=True).to(device)
stdevs = torch.tensor(np.abs(np.random.randn(8,2)),requires_grad=True).to(device)
mix = D.Categorical(weights)
comp = D.Independent(D.Normal(means,stdevs), 1)
gmm = D.MixtureSameFamily(mix, comp)

num_iter = 10001#30001
num_iter2 = 200001
loss_max1 = 100
for i in range(num_iter):
    x = torch.randn(5000,2)#this can be an arbitrary x samples
    loss2 = -gmm.log_prob(x).mean()#-densityflow.log_prob(inputs=x).mean()
    optimizer1.zero_grad()
    loss2.backward()
    optimizer1.step()

The error I get is:
0
8.089411823514835
Traceback (most recent call last):

  File "/home/cameron/AnacondaProjects/gmm.py", line 183, in <module>
    loss2.backward()

  File "/home/cameron/anaconda3/envs/torch/lib/python3.7/site-packages/torch/tensor.py", line 221, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)

  File "/home/cameron/anaconda3/envs/torch/lib/python3.7/site-packages/torch/autograd/__init__.py", line 132, in backward
    allow_unreachable=True)  # allow_unreachable flag

RuntimeError: Trying to backward through the graph a second time, but the saved intermediate results have already been freed. Specify retain_graph=True when calling backward the first time.

After as you see the model runs for 1 iteration.


title: Pytorch showing the error: 'NoneType' object has no attribute 'zero_'

body:
I am using Python 3.8 and VSCode.
I tried to create a basic Neural Network without activations and biases but because of the error, I'm not able to update the gradients of the weights.
Matrix Details:
Layer Shape: (1, No. of Neurons)
Weight Layer Shape: (No. of Neurons in the previous layer, No. of Neurons in the next layer)
Here's my code:
# sample neural network

# importing libraries
import torch
import numpy as np
torch.manual_seed(0)

# hyperparameters
epochs = 100
lr = 0.01 # learning rate

# data
X_train = torch.tensor([[1]], dtype = torch.float)
y_train = torch.tensor([[2]], dtype = torch.float)

'''
Network Architecture:
    1 neuron in the first layer
    4 neurons in the second layer
    4 neurons in the third layer
    1 neuron in the last layer

    * I haven't added bias and activation.
'''

# initializing the weights
weights = []
weights.append(torch.rand((1, 4), requires_grad = True))
weights.append(torch.rand((4, 4), requires_grad = True))
weights.append(torch.rand((4, 1), requires_grad = True))

# calculating y_pred
y_pred = torch.matmul(torch.matmul(torch.matmul(X_train, weights[0]), weights[1]), weights[2])

# calculating loss
loss = (y_pred - y_train)**2

# calculating the partial derivatives
loss.backward()

# updating the weights and zeroing the gradients
with torch.no_grad():
    for i in range(len(weights)):
        weights[i] = weights[i] - weights[i].grad
        weights[i].grad.zero_()

It it showing the error:
File "test3.py", line 43, in <module>
weights[i].grad.zero_()
AttributeError: 'NoneType' object has no attribute 'zero_'

I don't understand why it is showing this error. Can someone please explain?


title: I am stuck on how to COnvert Pytorch code in to Keras of Attention Model

body:
How to Convert Pytorch Model to keras Layer
the Pytorch Code:

class SeqAttnMatch(nn.Module):
    """Given sequences X and Y, match sequence Y to each element in X.

    * o_i = sum(alpha_j * y_j) for i in X
    * alpha_j = softmax(y_j * x_i)
    """

    def __init__(self, input_size, identity=False):
        super(SeqAttnMatch, self).__init__()
        if not identity:
            self.linear = nn.Linear(input_size, input_size)
        else:
            self.linear = None

    def forward(self, x, y, y_mask):
        """
        Args:
            x: batch * len1 * hdim
            y: batch * len2 * hdim
            y_mask: batch * len2 (1 for padding, 0 for true)
        Output:
            matched_seq: batch * len1 * hdim
        """
        # Project vectors
        if self.linear:
            x_proj = self.linear(x.view(-1, x.size(2))).view(x.size())
            x_proj = F.relu(x_proj)
            y_proj = self.linear(y.view(-1, y.size(2))).view(y.size())
            y_proj = F.relu(y_proj)
        else:
            x_proj = x
            y_proj = y

        # Compute scores
        scores = x_proj.bmm(y_proj.transpose(2, 1))

        # Mask padding
        y_mask = y_mask.unsqueeze(1).expand(scores.size())
        scores.data.masked_fill_(y_mask.data, -float('inf'))

        # Normalize with softmax
        alpha_flat = F.softmax(dim=scores.view(-1, y.size(1)))
        alpha = alpha_flat.view(-1, x.size(1), y.size(1))

        # Take weighted average
        matched_seq = alpha.bmm(y)
        return matched_seq

class BilinearSeqAttn(nn.Module):
    """A bilinear attention layer over a sequence X w.r.t y:

    * o_i = softmax(x_i'Wy) for x_i in X.

    Optionally don't normalize output weights.
    """

    def __init__(self, x_size, y_size, identity=False, normalize=True):
        super(BilinearSeqAttn, self).__init__()
        self.normalize = normalize

        # If identity is true, we just use a dot product without transformation.
        if not identity:
            self.linear = nn.Linear(y_size, x_size)
        else:
            self.linear = None

    def forward(self, x, y, x_mask):
        """
        Args:
            x: batch * len * hdim1
            y: batch * hdim2
            x_mask: batch * len (1 for padding, 0 for true)
        Output:
            alpha = batch * len
        """
        Wy = self.linear(y) if self.linear is not None else y
        xWy = x.bmm(Wy.unsqueeze(2)).squeeze(2)
        xWy.data.masked_fill_(x_mask.data, -float('inf'))
        if self.normalize:
            alpha = F.softmax(dim=xWy)
        else:
            alpha = xWy.exp()
        return alpha

class LinearSeqAttn(nn.Module):
    """Self attention over a sequence:

    * o_i = softmax(Wx_i) for x_i in X.
    """

    def __init__(self, input_size):
        super(LinearSeqAttn, self).__init__()
        self.linear = nn.Linear(input_size, 1)

    def forward(self, x, x_mask):
        """
        Args:
            x: batch * len * hdim
            x_mask: batch * len (1 for padding, 0 for true)
        Output:
            alpha: batch * len
        """
        x_flat = x.view(-1, x.size(-1))
        scores = self.linear(x_flat).view(x.size(0), x.size(1))
        scores.data.masked_fill_(x_mask.data, -float('inf'))
        alpha = F.softmax(dim=scores)
        return alpha

I


title: Wasserstein GAN critic training ambiguity

body:
I'm running a DCGAN-based GAN, and am experimenting with WGANs, but am a bit confused about how to train the WGAN.

In the official Wasserstein GAN PyTorch implementation, the discriminator/critic is said to be trained Diters (usually 5) times per each generator training.

Does this mean that the critic/discriminator trains on Diters batches or the whole dataset Diters times? If I'm not mistaken, the official implementation suggests the discriminator/critic is trained on the whole dataset Diters times, but other implementations of WGAN (in PyTorch and TensorFlow etc.) do the opposite.

Which is correct? The WGAN paper (to me, at least), indicates that it is Diters batches. Training on the whole dataset is obviously orders of magnitude slower.

Thanks in advance!


title: With pytorch DataLoader how to take in two ndarray (data & label)?

body:
I have a training data features in ndarray of shape (100, 400, 3) as it's 100 images of 20x20 with RGB channel and label in shape (100, ). Do I need to combine them into one dataset or how can I pass it to Pytorch dataLoader in order to iterate over image and label later?

What I've tried so far

#turn ndarray of features and labels into tensors
transform = transforms.Compose([transforms.ToPILImage(),
                                transforms.ToTensor()])


title: Creating a Feed Forward NN Model in Pytorch with a dynamic number of hidden layers

body:
Why are these two code segments not equivalent:
Segment 1: Creating a model with 2 layers.

class FNNModule(nn.Module):
    def __init__(self, input_dim, output_dim, hidden_dim1, hidden_dim2, non_linear_function):
        super().__init__()
        self.hidden1 = nn.Linear(input_dim, hidden_dim1)
        self.hidden2 = nn.Linear(hidden_dim1, hidden_dim2)
        self.non_linear_function = non_linear_function()
        self.final_linear = nn.Linear(hidden_dim2, output_dim)

    def forward(self, x):
        out = self.hidden1(x)
        out = self.non_linear_function(out)
        out = self.hidden2(x)
        out = self.non_linear_function(out)
        out = self.final_linear(out)
        return out

Segment Two: Creating the same model but changing code where hidden_layers is a variable:

class FNNModuleVar(nn.Module):
    def __init__(self, input_dim, output_dim, hidden_dim_array = [], non_linear_function_array=[]):
        super().__init__()
        self.linear_functions = []
        self.non_linear_functions = [x() for x in non_linear_function_array]
        self.hidden_layers = len(hidden_dim_array)
        for l in range(self.hidden_layers):
            self.linear_functions.append(nn.Linear(input_dim, hidden_dim_array[l]))
            input_dim = hidden_dim_array[l]
        self.final_linear = nn.Linear(input_dim, output_dim)

    def forward(self, x):
        out = x
        for i in range(self.hidden_layers):
            out = self.linear_functions[i](out)
            out = self.non_linear_functions[i](out)
        out = self.final_linear(x)
        return out
modelVar = FNNModuleVar(input_dim, output_dim, [100, 50], [nn.Tanh, nn.Tanh])
model = FNNModule(input_dim, output_dim, 100, 50, nn.Tanh)

When I try to iterate through modelVar.parameters() and model.parameters() I see that I have very different models.

What am I doing wrong in modelVar?


title: loop through the batch image loader pytorch

body:
Let say I have batch with

  imgs = torch.Size([128, 1, 28, 28])

if I want to loop through the each image

for img in imgs:
   print(img.shpae) -> torch.Size([1, 28, 28])

if I want to get a  torch.Size([1,1, 28, 28]) for each image what should I do?


title: AssertionError: Torch not compiled with CUDA enabled for Key Information Extraction as Character-wise Classification with LSTM

body:
Im trying to implement the train.py file from this repo but the error I'm getting is:

Traceback (most recent call last):
  File "./src/train.py", line 74, in <module>
    main()
  File "./src/train.py", line 21, in main
    model = MyModel0(len(VOCAB), 20, args.hidden_size).to(args.device)
  File "/home/guest/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 426, in to
    return self._apply(convert)
  File "/home/guest/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 202, in _apply
    module._apply(fn)
  File "/home/guest/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 224, in _apply
    param_applied = fn(param)
  File "/home/guest/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 424, in convert
    return t.to(device, dtype if t.is_floating_point() else None, non_blocking)
  File "/home/guest/anaconda3/lib/python3.7/site-packages/torch/cuda/__init__.py", line 192, in _lazy_init
    _check_driver()
  File "/home/guest/anaconda3/lib/python3.7/site-packages/torch/cuda/__init__.py", line 95, in _check_driver
    raise AssertionError("Torch not compiled with CUDA enabled")
AssertionError: Torch not compiled with CUDA enabled

Even though the # torch.backends.cudnn.enabled = False, I'm still getting this error. Maybe there's more GPU dependencies which I'm not familiar with. I'm not so familiar with PyTorch, any suggestions, please?


title: LSTM cell implementation in Pytorch design choices

body:
I was looking for an implementation of an LSTM cell in Pytorch that I could extend, and I found an implementation of it in the accepted answer here. I will post it here because I'd like to refer to it. There are quite a few implementation details that I do not understand, and I was wondering if someone could clarify.

import math
import torch as th
import torch.nn as nn

class LSTM(nn.Module):

    def __init__(self, input_size, hidden_size, bias=True):
        super(LSTM, self).__init__()
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.bias = bias
        self.i2h = nn.Linear(input_size, 4 * hidden_size, bias=bias)
        self.h2h = nn.Linear(hidden_size, 4 * hidden_size, bias=bias)
        self.reset_parameters()

    def reset_parameters(self):
        std = 1.0 / math.sqrt(self.hidden_size)
        for w in self.parameters():
            w.data.uniform_(-std, std)

    def forward(self, x, hidden):
        h, c = hidden
        h = h.view(h.size(1), -1)
        c = c.view(c.size(1), -1)
        x = x.view(x.size(1), -1)

        # Linear mappings
        preact = self.i2h(x) + self.h2h(h)

        # activations
        gates = preact[:, :3 * self.hidden_size].sigmoid()
        g_t = preact[:, 3 * self.hidden_size:].tanh()
        i_t = gates[:, :self.hidden_size]
        f_t = gates[:, self.hidden_size:2 * self.hidden_size]
        o_t = gates[:, -self.hidden_size:]

        c_t = th.mul(c, f_t) + th.mul(i_t, g_t)

        h_t = th.mul(o_t, c_t.tanh())

        h_t = h_t.view(1, h_t.size(0), -1)
        c_t = c_t.view(1, c_t.size(0), -1)
        return h_t, (h_t, c_t)

1- Why multiply the hidden size by 4 for both self.i2h and self.h2h (in the init method)

2- I don't understand the reset method for the parameters. In particular, why do we reset parameters in this way?

3- Why do we use view for h, c, and x in the forward method?

4- I'm also confused about the column bounds in the activations part of the forward method. As an example, why do we upper bound with 3 * self.hidden_size for gates?

5- Where are all the parameters of the LSTM? I'm talking about the Us and Ws here:


title: Hugging Face Transformer multiprocessing training

body: I'm trying to train NER using distilbert on CPU. However, training is slow. Is there any way to do some CPU optimization to reduce the training time?


title: PyTorch: Using a target size (torch.Size([1])) that is different to the input size (torch.Size([1, 1]))

body:
I am new to PyTorch and working on the implementation of recommender systems.
I obtained my models from here:
https://blog.fastforwardlabs.com/2018/04/10/pytorch-for-recommenders-101.html
Following the instructions from the website, i feed the DenseNet model exactly the same way as the MatrixFactorization model.
models.py:
class MatrixFactorization(nn.Module):

def __init__(self, n_users, n_items, n_factors=20):
    super(MatrixFactorization, self).__init__()
    # create user embeddings
    self.user_factors = nn.Embedding(n_users, n_factors,
                                     sparse=True)
    # create item embeddings
    self.item_factors = nn.Embedding(n_items, n_factors,
                                     sparse=True)

def forward(self, user, item):
    # matrix multiplication
    prediction = (self.user_factors(user) * self.item_factors(item)).sum(1)
    # test
    return F.hardsigmoid(prediction)

def predict(self, user, item):
    return self.forward(user, item)

class DenseNet(nn.Module):

def __init__(self, n_users, n_items, n_factors, h1=128, d_out=1):
    """
    Simple Feedforward with Embeddings
    """
    super(DenseNet, self).__init__()
    # user and item embedding layers
    self.user_factors = torch.nn.Embedding(n_users, n_factors,
                                           sparse=True)
    self.item_factors = torch.nn.Embedding(n_items, n_factors,
                                           sparse=True)
    # linear layers
    self.linear1 = torch.nn.Linear(n_factors*2, h1)
    self.linear2 = torch.nn.Linear(h1, d_out)

def forward(self, users, items):
    users_embedding = self.user_factors(users)
    items_embedding = self.item_factors(items)
    # concatenate user and item embeddings to form input
    x = torch.cat([users_embedding, items_embedding], 1)
    h1_relu = F.relu(self.linear1(x))
    output_scores = self.linear2(h1_relu)
    return output_scores

def predict(self, users, items):
    # return the score
    output_scores = self.forward(users, items)
    return output_scores

training of densenet:
    index = 0
    model.train()

for user, item in zip(users, items):
    # get user, item and rating data
    # rating = Variable(torch.FloatTensor([ratings[user, item]]))
    rating = normalize(rating_values[index])
    rating = Variable(torch.FloatTensor([rating]))
    user = Variable(torch.LongTensor([int(user)]))
    item = Variable(torch.LongTensor([int(item)]))

    index += 1
    # predict
    prediction = model.predict(user, item)
    loss = loss_fn(prediction, rating)

    optimizer.zero_grad()
    # backpropagate
    loss.backward()

    # update weights
    optimizer.step()

Although i receive an output i get the UserWarning:
    AppData\Local\Programs\Python\Python39\lib\site-packages\torch\nn\modules\loss.py:446: UserWarning: Using a target size (torch.Size([1])) that is different to the input size (torch.Size([1, 1])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
  return F.mse_loss(input, target, reduction=self.reduction)

I thought this model takes the same input (tensor of user and item id for each user) as the mf model which works for me. What is wrong with the inputs? Which line produces this error?


title: Implementing Hierarchical Attention for Classification

body:
I am trying to implement the Hierarchical Attention paper for text classification. One of the challenges that I am finding is how to manage batching and updates to the weights of the network by the optimizer.  The architecture of the network is made of two encoders stacked one after the other: a sentence encoder, and a document encoder.

When the dataset is made of large documents, the following problem arises: for each pass through the document encoder, you will have multiple passes through the sentence encoder.  When the loss is calculated and the optimizer uses the calculated gradients to update the weights of the parameters of the network, I am assuming that the weights of the sentence encoder should be updated differently to the weights of the document encoder.  What is a good strategy to do so? How could that strategy could be implemented in libraries such as Keras or Pytorch?


title: Detach intermediate module in Pytorch with regard to part of the loss

body:
Let's say I have the following forward pass that results in two separate losses:

forward(self, input)
    x = self.layer1(input)

    y = self.layer2(x)

    z = self.layer3(y)

    return y, z

We then calculate loss1(y) and loss2(z). Then we can optimize loss = loss1 + loss2 with a single optimizer.

But I have two caveats: (1) I want d_loss1 to be calculated with regards to layer2 only (without layer1), and (2) I want d_loss2 to be calculated with regards to layer3 and layer1 - without layer2.
Essentially, I want to train non-consecutive parts of the network separately with separate losses.

I believe I can solve (1) by introducing a stop-gradient in the input to layer2 like so:

forward(self, input)
    x = self.layer1(input)

    y = self.layer2(x)
    y_stop_gradient = self.layer2(Variable(x.data))

    z = self.layer3(y)

    return y_stop_gradient, z

But how can I solve (2)?
In other words, I want loss2's gradients to "skip" layer2 while keeping layer2 trainable with regards to loss1.


title: Import from Github : How to fix ImportError

body:
I want to use the open source person re-identification library in Python

on Ubuntu 19.04
with Anaconda
no CUDA
in the terminal PyCharm (or not)
Python version 3.7.3
PyTorch version 1.1.0

For that I have to follow instruction like on their deposite git :

git clone https://github.com/Cysu/open-reid.git
cd open-reid
python setup.py install

python examples/softmax_loss.py -d viper -b 64 -j 2 -a resnet50 --logs-dir logs/softmax-loss/viper-resnet50

I receive the following error:

from sklearn.utils.extmath
import pinvh

  ImportError: cannot import name 'pinvh'

I have tried to create virtual environments with previous versions of PyTorch (0.4.1, 0.4.0 and 1.0.1) but I always got:

  File "examples/softmax_loss.py", line 12, in <module>
    from reid import datasets
ModuleNotFoundError: No module named 'reid'

I do not know how to fix it.

EDIT :

Hi thanks for the answer, the problem is that the import are like :

from reid import datasets
from reid import models
from reid.dist_metric import DistanceMetric
from reid.trainers import Trainer
from reid.evaluators import Evaluator
from reid.utils.data import transforms as T
from reid.utils.data.preprocessor import Preprocessor
from reid.utils.logging import Logger
from reid.utils.serialization import load_checkpoint, save_checkpoint

I tried :

from ../reid import datasets

But I got a

File "examples/softmax_loss.py", line 12
from ../reid import datasets
   ^
SyntaxError: invalid syntax

EDIT 2 :
After re-installing Python 3.7.3 and pytorch 1.1.0 the problem persist with pinvh... I still got this message :

ImportError: cannot import name 'pinvh' from 'sklearn.utils.extmath'

If you can tell me how to fix it or try to tell me if it works please


title: Federated learning using custom model in Pytorch/Pysyft

body:
I am trying to build a federated learning model. In my scenario, I have 3 workers and an orchestrator. The workers start the training and at the end of each training round, the models are being sent to the orchestrator, the orchestrator calculates the federated average and sends back the new model, the workers train on that new model etc. The custom network is an AutoEncoder that I have built from scratch.

Unfortunately I am getting this error message from the workers: RuntimeError: forward() is missing value for argument 'inputs'. Declaration: forward(ClassType self, Tensor inputs, Tensor outputs) -> (Tensor) which is weird because my forward function is defined as follows, inside the AE class:

class AutoEncoder(nn.Module):

    def __init__(self, code_size):
        super().__init__()
        self.code_size = code_size

        # Encoder specification
        self.enc_cnn_1 = nn.Conv2d(3, 10, kernel_size=5)
        self.enc_cnn_2 = nn.Conv2d(10, 20, kernel_size=5)
        self.enc_linear_1 = nn.Linear(53 * 53 * 20, 50)
        self.enc_linear_2 = nn.Linear(50, self.code_size)

        # Decoder specification
        self.dec_linear_1 = nn.Linear(self.code_size, 160)
        self.dec_linear_2 = nn.Linear(160, IMAGE_SIZE)

    def forward(self, images):
        code = self.encode(images)
        out = self.decode(code)
        return out, code

    def encode(self, images):
        code = self.enc_cnn_1(images)
        code = F.selu(F.max_pool2d(code, 2))

        code = self.enc_cnn_2(code)
        code = F.selu(F.max_pool2d(code, 2))
        code = code.view([images.size(0), -1])
        code = F.selu(self.enc_linear_1(code))

        code = self.enc_linear_2(code)
        return code

    def decode(self, code):
        out = F.selu(self.dec_linear_1(code))
        out = F.sigmoid(self.dec_linear_2(out))
        out = out.view([code.size(0), 3, IMAGE_WIDTH, IMAGE_HEIGHT])
        return out

Loss function (cross entropy)

```
@torch.jit.script
def loss_fn(inputs, outputs):
    return torch.nn.functional.mse_loss(input=inputs, target=outputs)

def set_gradients(model, finetuning):
    """Helper function to exclude all gradients from training
    used for transfer learning in feature extract mode; See: https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html

    Args:
        model (torch.nn.Module): model object.
        finetuning (bool):  if true, nothing will be changed; transfer learning will be used in finetuning mode, i.e., all gradients are trained;
                            if false, all gradients get excluded from training, used in feature extract mode
    """

    if not finetuning:
        for param in model.parameters():
            param.requires_grad = False
```

```
def initialize_model():
    model = AutoEncoder(code_size)
    set_gradients(model, False)
    return model
async def train_model_on_worker(
    worker: websocket_client.WebsocketClientWorker,
    traced_model: torch.jit.ScriptModule,
    dataset_key: str,
    batch_size: int,
    curr_round: int,
    lr: float,
):

    traced_model.train()
    print("train mode on")
    train_config = sy.TrainConfig(
        model=traced_model,
        loss_fn=loss_fn,
        batch_size=batch_size,
        shuffle=True,
        epochs=1,
        optimizer="Adam",
        optimizer_args={"lr": lr}
    )
logger.info(worker.id + " send trainconfig")
    train_config.send(worker)
    print("Model sent to the worker")
    logger.info(worker.id + " start training")
    await worker.async_fit(dataset_key=DATASET_KEY, return_ids=[0])
    logger.info(worker.id + " training done")
    results = dict()
logger.info(worker.id + " get model")
    model = train_config.model_ptr.get().obj

    results["worker_id"] = worker.id
    results["model"] = model

    return results
def validate_model(identifier, model, dataloader, criterion):
model.eval() # changes the mode of the model, in evaluation mode we don't have dropout

    loss = []
    for i, (inputs,_) in enumerat(dataloader):
        print("validation mode on")
        #with torch.set_grad_enabled(False):
        outputs, code  = model(Variable(inputs)) #a tensor with 2 values: one for leak and one for no leak
        loss = criterion(outputs, inputs)

        loss = loss.sqrt()
        loss.append(loss.item())
    print("Loss = %.3f" % loss.data)
async def main():
    args = define_and_get_arguments()
    hook = sy.TorchHook(torch) #with this we can override some pytorch methods with pysyft

    # Create WebsocketClientWorkers using IDs, Ports and IP addresses from arguments
    worker_instances = []
    for i in range(len(args.workers) // 3):
        j = i * 3
        worker_instances.append(websocket_client.WebsocketClientWorker(
            id=args.workers[j], port=args.workers[j + 1], host=args.workers[j + 2], hook=hook, verbose=args.verbose))

    model = initialize_model()

    # optional loading of predefined model weights (= dictionary):
    if args.basic_model:
        model.load_state_dict(torch.load(args.basic_model))

    # model serialization (creating an object of type ScriptModule):
    model.eval()
    traced_model = torch.jit.trace(model, torch.rand([1, 3, 224, 224], dtype=torch.float)) #we need to change the form of the model in order to make it
    #serialisable and send it to the workers

    # Data / picture transformation:
    data_transforms = transforms.Compose([
        transforms.Resize(INPUT_SIZE),
        transforms.CenterCrop(INPUT_SIZE),
        transforms.ToTensor()
        #transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ])

    # Create validation dataset and dataloader
    validation_dataset = datasets.ImageFolder(os.path.join(args.dataset_path, 'val'), data_transforms)
    validation_dataloader = torch.utils.data.DataLoader(validation_dataset, batch_size=args.test_batch_size, shuffle=True, num_workers=4)

    # Create test dataset and dataloader
    test_dataset = datasets.ImageFolder(os.path.join(args.dataset_path, 'test'), data_transforms)
    test_dataloader = torch.utils.data.DataLoader(test_dataset, batch_size=args.test_batch_size, shuffle=True, num_workers=4)

    # Lists to plot loss and acc after training
    train_loss_values = []
    train_acc_values = []
    val_loss_values = []
    val_acc_values = []

    np.set_printoptions(formatter={"float": "{: .0f}".format})

    for curr_round in range(1, args.training_rounds + 1):
        logger.info("Training round %s/%s", curr_round, args.training_rounds)
        print("entered training ")
        # reduce learn rate every 5 training rounds (adaptive learnrate)
        lr = args.lr * (0.1 ** ((curr_round - 1) // 5))
        completed, pending = await asyncio.wait(
            [
                *[
                    train_model_on_worker(
                        worker=worker,
                        traced_model=traced_model,
                        dataset_key=DATASET_KEY,
                        batch_size=args.batch_size,
                        curr_round=curr_round,
                        lr=lr,
                    )
                    for worker in worker_instances
                ]
            ],
            timeout=40
        )

        results = []
        for entry in completed:
            print("entry")
            print(entry)
            results.append(entry.result())

        for entry in pending:
            entry.cancel()

        new_worker_instances = []
        for entry in results:
            for worker in worker_instances:
                if (entry["worker_id"] == worker.id):
                    new_worker_instances.append(worker)

        worker_instances = new_worker_instances

        # Setup the loss function
        criterion = torch.nn.functional.mse_loss()
        #optimizer = optimizer_cls(autoencoder.parameters(), lr=lr)

        # Federate models (note that this will also change the model in models[0]
        models = {}
        for worker in results:
            if worker["model"] is not None:
                models[worker["worker_id"]] = worker["model"]

        logger.info("aggregation")
        traced_model = utils.federated_avg(models)
        logger.info("aggregation done")

        # Evaluate federated model
        logger.info("Validate..")
        loss = validate_model("Federated", traced_model, validation_dataloader, criterion)
        logger.info("Validation done")
        val_loss_values.append(loss)
        #val_acc_values.append(acc)
if __name__ == "__main__":
    # Logging setup
    date_time = datetime.now().strftime("%m-%d-%Y_%H:%M:%S")
    FORMAT = "%(asctime)s | %(message)s"
    logging.basicConfig(filename='logs/orchestrator_' + date_time + '.log', format=FORMAT)
    logger = logging.getLogger("orchestrator")
    logger.setLevel(level=logging.INFO)

    asyncio.get_event_loop().run_until_complete(main())

The code of the workers:
def load_dataset(dataset_path):
    """Helper function for setting up the local datasets.

    Args:
        dataset_path (string):  path to dataset, images must be arranged in this way
                                dataset_path/train/class1/xxx.jpg
                                dataset_path/train/class2/yyy.jpg
    """

    data_transform = transforms.Compose([
        transforms.RandomResizedCrop(INPUT_SIZE),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor()
        #transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ])
    dataset = datasets.ImageFolder(os.path.join(dataset_path, 'train'), data_transform)

    return dataset

def start_websocket_server(id, port, dataset, verbose):
    """Helper function for spinning up a websocket server.

    Args:
        id (str or id): the unique id of the worker.
        port (int): the port on which the server should be run.
        dataset: dataset, which the worker should provide.
        verbose (bool): a verbose option - will print all messages sent/received to stdout.
    """

    hook = sy.TorchHook(torch)
    server = WebsocketServerWorker(id=id, host="0.0.0.0", port=port, hook=hook, verbose=verbose)
    server.add_dataset(dataset, key=DATASET_KEY)
    server.start()

    return server

    def _fit(self, model, dataset_key, loss_fn):
        logger = logging.getLogger("worker")
        logger.info(dataset_key)
        print("dataset key")
        model.train()
        data_loader = torch.utils.data.DataLoader(dataset, batch_size=args.test_batch_size, shuffle=True, num_workers=4)
        #data_loader = self._create_data_loader(
            #dataset_key=dataset_key, shuffle=self.train_config.shuffle
        #)
        print("worker")
        print(data_loader)

        loss = None
        iteration_count = 0

        for _ in range(self.train_config.epochs):
            for data in enumerate(data_loader):
                # Set gradients to zero
                self.optimizer.zero_grad()

                # Update model
                output,code = model(data)
                logger.info(data)
                logger.info(output)
                loss = loss_fn(data, output)
                loss.backward()
                self.optimizer.step()

                # Update and check interation count
                iteration_count += 1
                if iteration_count >= self.train_config.max_nr_batches >= 0:
                    break

        return model

if __name__ == "__main__":

    # Parse args
    args = define_and_get_arguments()

    # Logging setup
    date_time = datetime.now().strftime("%m-%d-%Y_%H:%M:%S")
    FORMAT = "%(asctime)s %(levelname)s %(filename)s(l:%(lineno)d, p:%(process)d) - %(message)s"
    logging.basicConfig(filename='logs/worker_' + args.id + '_' + date_time + '.log', format=FORMAT)
    logger = logging.getLogger("worker")
    logger.setLevel(level=logging.INFO)

    # Load train dataset
    dataset = load_dataset(args.dataset_path)

    # Start server
    server = start_websocket_server(
        id=args.id,
        port=args.port,
        dataset=dataset,
        verbose=args.verbose,
    )

Does anybody know what the problem is?


title: exceptions when loading the checkpoint of a PyTorch NN model

body:
when calling the function defined in the following cell, an exception is thrown 'TypeError: forward() takes 2 positional arguments but 9 were given '
This document provides further detail

def load_checkpoint(chkptJP):
checkpoint = torch.load(chkptJP)
model2 = model1(checkpoint['input_size'],
              checkpoint['output_size'],
              checkpoint['fc1'],
              checkpoint['fc2'],
              checkpoint['optimizer_state_dict'],
              checkpoint['epoch'],
              checkpoint['class_to_idx'],
              checkpoint['learning_rate'])
model2.load_state_dict(checkpoint['state_dict'])
return model2

The code that wrote out the checkpoint is as follows:

checkpoint ={'input_size':512,
         'output_size':102,
         'fc1':256,
         'fc2':102,
         'state_dict': model.state_dict(),
         'optimizer_state_dict': optimizer.state_dict(),
         'epoch': epoch+1,
         'class_to_idx': model.class_to_idx,
         'learning_rate': 0.003}
torch.save(checkpoint,chkptJP)


title: Speed of L2 Regularization on Pytorch

body:
I'm trying to manually implement L2 regularisation and a couple of its variations in a neural network. What I'm doing is the following:

for name, param in model.state_dict():
    if 'weight' in name:
        l2_reg += torch.sum(param**2)

loss = cross_entropy(outputs, labels) + 0.0001*l2_reg

Is this equivalent to adding 'weight_decay = 0.0001' inside my optimizer? i.e.:

torch.optim.SGD(model.parameters(), lr=learning_rate , momentum=0.9, weight_decay = 0.0001)

My problem is that I thought they were equivalent, but the manual procedure is about 100x slower than adding 'weight_decay = 0.0001'. Why is that? How can I fix it?

Note that I need to also implement my own variation of L2 regularization, so just adding 'weight_decay = 0.0001' won't help.


title: T5 model custom vocabulary

body:
Is there a way to choose my custom vocabulary in T5-model while fine-tuning for a text summarization task?
I tried using a sentencepiece model to create my custom tokenizer but the model predicted some tokens which was not present in my tokenizer and hence the tokenizer takes it as an unknown token.


I am trying to run a model on TPU as given in colab notebook. The model was working fine, but today I could not run the model.
I used the following code to install pytorch-xla.
VERSION = "nightly"  #@param ["1.5" , "20200325", "nightly"]
!curl https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py -o pytorch-xla-env-setup.py
!python pytorch-xla-env-setup.py --version $VERSION

I try to install required libraries as below:
!pip install -U nlp
!pip install sentencepiece
!pip install numpy --upgrade

However, when I try the following
import nlp

It gives the following error:
OSError: libmkl_intel_lp64.so.1: cannot open shared object file: No such file or directory

I searched the error and I tried the followings, but still does not work. Any ideas how to fix it? Note: It was working a few days ago, however, today it is not.
!pip install mkl
#!export PATH="$PATH:/opt/intel/bin"
#!export LD_LIBRARY_PATH="$PATH:opt/intel/mkl/lib/intel64_lin/"
!export LID_LIBRAEY_PATH="$LID_LIBRARY_PATH:/opt/intel/mkl/lib/intel64_lin/"


title: How to check the correct Training Class In PyTorch

body:
I'm new with Pytorch, I tried to write my training class, but I had this error

import numpy as np
import torch
import torch.nn as nn
from torch.autograd import Variable
import torch.nn.functional as F
import torch.optim as optim

import tqdm

class MLPNet(nn.Module):
    def __init__(self):
        super(MLPNet, self).__init__()
        self.first_fully_connected = nn.Linear(8*8, 100)
        self.last_fully_connected = nn.Linear(100, 10)

    def forward(self, x):
        x = x.view(-1, 8*8) # reshape input tensor to the size (batch_size, 8*8)
        x = F.sigmoid(self.first_fully_connected(x))
        x = F.sigmoid(self.last_fully_connected(x))
        return x

def training(mlp, X, y, epochs=1, lr=.2, batch_size=101):
    # solver
    # loss
    solver = torch.optim.SGD(mlp.parameters(), lr=lr, momentum=0.9)
    loss = nn.CrossEntropyLoss() #  nn.NLLLoss()

    n_batches = (len(X) + batch_size - 1) // batch_size
    for epoch in tqdm.tqdm(range(epochs)):
        for i in range(n_batches):
            slice_ = np.s_[i::n_batches]
            X_batch = Variable(torch.from_numpy(X[slice_])).float()
            y_batch = Variable(torch.from_numpy(y[slice_, np.newaxis])).float()

#             X_batch = Variable(torch.from_numpy(X[slice_])).long()
#             y_batch = Variable(torch.from_numpy(y[slice_, np.newaxis])).long()

            print(type(X_batch.data))
            print(type(y_batch.data))
            ### BEGIN: your optim step here. do not forget to reset gradients

            # Clear gradients w.r.t. parameters
            solver.zero_grad()

            prediction = mlp(X_batch)
            # Forward pass to get output/logits
            #outputs = mlp(X_batch)

            # Calculate Loss: softmax --> cross entropy loss
            #loss = criterion(outputs, y_batch)
            loss_f = loss(prediction, y_batch)

            # Getting gradients w.r.t. parameters
            loss_f.backward()

            # Updating parameters
            solver.step()
            ### END

    return mlp

mlp = nn.Sequential(
    #### Your net here
    nn.Linear(2, 64),
    nn.ReLU(),
    nn.Linear(64, 2)
)
model_mlp = training(mlp, X_std, y_std)

But I got this error,
I tried the change the type but still faced that error.
I tried the changed the loss function too but still that error.

RuntimeError Traceback (most recent call last)
 in ()
----> 1 model_mlp = fit(mlp, X_std, y_std)

RuntimeError: Expected object of type Variable[torch.FloatTensor] but found type Variable[torch.LongTensor] for argument #1 'mat1'

I really appreciate any help you can provide.
Thank you so much


title: pytorch: how can I use picture as label in dataloader?

body:
I want to do some image reconstruction using autoencoders in pytorch, however, I didn't find a way to use image as label for an input image.(the label image is different from original ones)

I've tried the image folder method, but I think that's for classfication and I am currently unable to come up with one solution. Should I create a custom dataset for this...

Thanks in advance!


title: PyTorch - unexpected shape of model parameters weights

body:
I created a fully connected network in Pytorch with an input layer of shape (1,784) and a first hidden layer of shape (1,256).
To be short: nn.Linear(in_features=784, out_features=256, bias=True)

Method 1 : model.fc1.weight.data.shape gives me torch.Size([128, 256]), while

Method 2 : list(model.parameters())[0].shape gives me torch.Size([256, 784])

In fact, between an input layer of size 784 and a hidden layer of size 256, I was expecting a matrix of shape (784,256).
So, in the first case, I see the shape of the next hidden layer (128), which does not make sense for the weights between the input and first hidden layer, and, in the second case, it looks like Pytorch took the transform of the weight matrix.

I don't really understand how Pytorch shapes the different weight matrices, and how can I access individual weights after the training. Should I use method 1 or 2? When I display the corresponding tensors, the displays look totally similar, while the shapes are different.


title: pytorch custom loss function on minimizing the angle between vectors

body:
The goal is to minimize the angle between the actual and predicted vectors in a neural network setting. Can someone please check if the following execution is correct?
criterion = nn.CosineSimilarity()
loss = torch.mean(torch.abs(criterion(actual_vectors,predicted_vectors)))

#back-propagation on the above *loss* will try cos(angle) = 0. But I want angle between the vectors to be 0 or cos(angle) = 1.

loss = 1 - loss

#To me, the above does not seem right. Isn't back-propagation on the above 'loss' similar to minimizing the negative of 'loss' from line 2?
#Does '1' have no role to play here when back-propagation is applied?

loss.backward()


title: ModuleNotFoundError: No module named 'numpy.core._multiarray_umath' on matplotlib import

body:
I'm trying to run a simple testfile on a remote Server. But it throws a numpy error for matplotlib.pyplot. Here is the code
import matplotlib.pyplot as plt
import numpy as np

# Fixing random state for reproducibility
np.random.seed(19680801)

x, y = np.random.randn(2, 100)
print('x')
print(x)
print('y')
print(y)
fig, [ax1, ax2] = plt.subplots(2, 1, sharex=True)
ax1.xcorr(x, y, usevlines=True, maxlags=50, normed=True, lw=2)
ax1.grid(True)

ax2.acorr(x, usevlines=True, normed=True, maxlags=50, lw=2)
ax2.grid(True)

plt.show()

Here is the error message.

PyTorch/1.7-py36-cuda11/numpy/core/overrides.py", line 7, in
from numpy.core._multiarray_umath import (
ModuleNotFoundError: No module named 'numpy.core._multiarray_umath'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "graph_test.py", line 1, in
import matplotlib.pyplot as plt

/PyTorch/1.7-py36-cuda11/numpy/core/init.py", line 48, in
raise ImportError(msg)
ImportError:
IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!
Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.
We have compiled some common reasons and troubleshooting tips at:
https://numpy.org/devdocs/user/troubleshooting-importerror.html
Please note and check the following:

The Python version is: Python3.7 from "/projects/smiles/Model/venv/bin/python"
The NumPy version is: "1.19.4"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.
Original error was: No module named 'numpy.core._multiarray_umath'

Python Version: 3.7.5
Numpy Version: 1.19.4
Matplotlib version: 3.3.3


title: Torchvision normalize - how it operates on tuple of means/sds?

body:
I don't understand how this transform works from torchvision. Ultimately I want to build a custom normalize class so I need to figure out how this works first.

Here in the docs it describes the init like this:

def __init__(self, mean, std, inplace=False):
        self.mean = mean
        self.std = std
        self.inplace = inplace

And when I pass these parameters usually (not custom class) I pass them as a list or tuple for each channel:

transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])

But if I look at the call:

return F.normalize(tensor, self.mean, self.std, self.inplace)

All this passes the tuple to is F.normalize() which only accepts a single value for the p parameter.

The class must iterate through the channels somehow to allow this to be implemented but how does it do this and how can I implement it in custom class?

Based on this tutorial, I would describe it like this:

class Normalize(object):
    """Convert ndarrays in sample to Tensors."""

    def __init__(self, mean, std, inplace=False):
        self.mean = mean
        self.std = std
        self.inplace = inplace

    def __call__(self, sample):
        image, landmarks = sample['image'], sample['landmarks']
        return {'image': F.normalize(image, self.mean, self.std, self.inplace),
                'landmarks': landmarks}

But this does not work because it does not go through each channel.


title: OSError: [Errno 24] Too many open files in Windows 10 when using pytorch dataloader

body:
I'm doing a model of a CNN network which takes images that are located in a directory "images", and each image is located inside that directory in a directory named as the ID of the image (ex. image 1 is in: images/1/1.png).

class SSMDataset(Dataset):
    songs_list = []
    def __init__(self, root_dir, transform=None):
        """lista de nombre de canciones"""
        self.root_dir = root_dir
        self.songs_list = []
        self.labels_list = []
        for (dirpath, dirnames, filenames) in os.walk(self.root_dir):
            for f in filenames:
                if f.endswith('.png'):
                    o = {}
                    o['img_path'] = dirpath + '/' + f
                    self.songs_list.append(o)
                elif f.endswith('.csv'):
                    o['label_path'] = dirpath + '/' + f
                    self.labels_list.append(o)
        self.transform = transform

    def __len__(self):
        return len(self.songs_list)

    def __getitem__(self, index):
        img_path = self.songs_list[index]['img_path']
        image = Image.open(img_path)
        fp = image.fp
        image.load()
        fp.closed
        image = np.array(image)
        image = image[np.newaxis, :, :]
        label_path = self.labels_list[index]['label_path']
        label = np.genfromtxt(label_path)
        if self.transform is not None:
            for t in self.transform:
                image, labels = t(image, label)
        return image, label

When I train the model, at the epoch 23 and always in taht epoch the following error raises:

Epoch: 23
39% 136/350 [00:32<00:37,  5.67it/s, accuracy=0.221, loss=1.97]

Traceback (most recent call last):
File dataloader.py line 560, in next
File dataloader.py line 560, in
File  line 55, in getitem
File Image.py line 2543, in open

OSError: [Errno 24] Too many open files: C:\Users\...\images\1296
/1296.png

Process finished with exit code 1

I have made proof with different libraries as cv2, matplotlib...to open the image, I have use the command "with Image.open(img_path) as im:" to make sure the image is closed after it is taken but it always raises the same error and in the same epoch. I also included:

import torch.multiprocessing
torch.multiprocessing.set_sharing_strategy('file_system')

in the code but the error continues to appear at the same epoch. Any idea of how could it be solved? Maybe is it a Windows problem...?


title: Installing Pytorch Transformers in AWS Sagemaker

body:
I'm trying to install the pytorch transformers package for my AWS Sagemaker notebook instance. However, it keeps giving me error of "No Module Found" for the package when i run my entry point script.

I saw in an example for TensorFlowModel which requires to set up env but for Pytorch it is not the case (How do I load python modules which are not available in Sagemaker?). Anyway, below is my code :

estimator = PyTorch(entry_point='model.py',
                role=role,
                framework_version='1.4.0',
                train_instance_count=2,
                train_instance_type='ml.c4.xlarge',
                source_dir = 'src',
                hyperparameters={
                    'train_path': 's3://bucket-train',
                    'validation_path': 's3://bucket-val',
                    'epochs': 3,
                    'backend': 'gloo'
                })


title: CNN + RNN architecture for video recognition

body:
I am trying to replicate the ConvNet + LSTM approach presented in this paper using pytorch. But I am struggling to find the correct way to combine the CNN and the LSTM in my model. Here is my attempt :
class VideoRNN(nn.Module):
  def __init__(self, hidden_size, n_classes):
    super(VideoRNN, self).__init__()

    self.hidden_size = hidden_size

    vgg = models.vgg16(pretrained=True)
    embed = nn.Sequential(*list(vgg.classifier.children())[:-1])
    vgg.classifier = embed

    for param in vgg.parameters():
      param.requires_grad = False

    self.embedding = vgg
    self.GRU = nn.GRU(4096, hidden_size)

  def forward(self, input, hidden=None):
    embedded = self.embedding(input)
    output, hidden = self.gru(output, hidden)
    output = self.classifier(output.view(-1, 4096))

    return output, hidden

As my videos have variable length, I provide a PackedSequence as an input. It is created from a Tensor with shape (M,B,C,H,W) where M is the maximum sequence length and B the batch size. The C,H,W are the channels, height and width of each frame.
I want the pre-trained CNN to be part of the model as I may later unfreeze some layer to finetune the CNN for my task. That's why I didn't compute the embedding of the images separately.
My questions are then the following :

Is the shape of my input data correct in order to handle batches of videos in my context or should I use something else than a PackedSequence?

In my forward function, how can I handle the batch of sequences of images with my VGG and my GRU unit ? I cannot feed directly the PackedSequence as an input to my VGG so how can I proceed?

Does this approach seem to respect the "pytorch way of doing things" or should is my approach flawed?


title: Low accuracy binary classification with Pytorch

body:
In practicing deep learning for binary classification with Pytorch on Breast-Cancer-Wisconsin-Diagnostic-DataSet.
I've tried different approaches, and the best I can get as below, the accuracy is still low at 61%.
What's the way to improve the accuracy?
Thank you.
import pandas as pd
import io

dataset = pd.read_excel(base_dir + "Breast-Cancer-Wisconsin-Diagnostic.xlsx")

number_of_columns = dataset.shape[1]

# training and testing split of 70:30
dataset['diagnosis'] = pd.Categorical(dataset['diagnosis']).codes
dataset = dataset.sample(frac=1, random_state=1234)
train_input = dataset.values[:398, :number_of_columns-1]
train_target = dataset.values[:398, number_of_columns-1]
test_input = dataset.values[398:, :number_of_columns-1]
test_target = dataset.values[398:, number_of_columns-1]

import torch
torch.manual_seed(1234)
hidden_units = 5
net = torch.nn.Sequential(
torch.nn.Linear(number_of_columns-1, hidden_units),
torch.nn.ReLU(),
torch.nn.Linear(hidden_units, 2))

# choose optimizer and loss function
criterion = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(net.parameters(), lr=0.1,momentum=0.9)

# train
epochs = 50
for epoch in range(epochs):
    inputs = torch.autograd.Variable(torch.Tensor(train_input).float())
    targets = torch.autograd.Variable(torch.Tensor(train_target).long())
    optimizer.zero_grad()
    out = net(inputs)
    loss = criterion(out, targets)
    loss.backward()
    optimizer.step()
    if epoch == 0 or (epoch + 1) % 10 == 0:
        print('Epoch %d Loss: %.4f' % (epoch + 1, loss.item()))

# Epoch 1 Loss: 412063.1250
# Epoch 10 Loss: 0.6628
# Epoch 20 Loss: 0.6639
# Epoch 30 Loss: 0.6592
# Epoch 40 Loss: 0.6587
# Epoch 50 Loss: 0.6588

import numpy as np
inputs = torch.autograd.Variable(torch.Tensor(test_input).float())
targets = torch.autograd.Variable(torch.Tensor(test_target).long())
optimizer.zero_grad()
out = net(inputs)
_, predicted = torch.max(out.data, 1)
error_count = test_target.size - np.count_nonzero((targets == predicted).numpy())

print('Errors: %d; Accuracy: %d%%' % (error_count, 100 * torch.sum(targets == predicted) // test_target.size))

# Errors: 65; Accuracy: 61%


title: PyTorch minibatch training very slow

body:
When training my model on the adult income data set and using minibatches training is very slow regardless if I use PyTorch's DataLoader or a basic implementation for minibatch training.
Is there a problem with my code or is there another way to speed up training for the adult income data set? I want to use one-hot encoding and cross-entropy loss + softmax. Do I have to use a different loss function or remove the softmax layer?
import pandas as pd
from pandas import read_csv
from sklearn.preprocessing import LabelEncoder
from sklearn.preprocessing import KBinsDiscretizer
from sklearn.model_selection import train_test_split
import torch
from torch.autograd import Variable
from torch.utils.data import DataLoader, Dataset, TensorDataset
import numpy as np
import matplotlib.pyplot as plt
import torch.nn as nn
import torch.nn.functional as F
import warnings

warnings.filterwarnings('ignore')
device = torch.device("cpu")

class Model(nn.Module):
    def __init__(self, input_dim):
        super(Model, self).__init__()
        self.layer1 = nn.Linear(input_dim, 12)
        self.layer2 = nn.Linear(12, 2)

    def forward(self, x):
        x = F.sigmoid(self.layer1(x))
        x = F.softmax(self.layer2(x))  # To check with the loss function
        return x

# load dataset
filename = './datasets/adult-all.csv'
dataframe = read_csv(filename, header=None, na_values='?')
# drop rows with missing
dataframe = dataframe.dropna()
# summarize the class distribution
target = dataframe.values[:, -1]
# split into inputs and outputs
last_ix = len(dataframe.columns) - 1
X_, y = dataframe.drop(last_ix, axis=1), dataframe[last_ix]
# select categorical and numerical features
cat_ix = X_.select_dtypes(include=['object', 'bool']).columns
num_ix = X_.select_dtypes(include=['int64', 'float64']).columns
# label encode the target variable to have the classes 0 and 1
y = LabelEncoder().fit_transform(y)
# one-hot encoding of categorical features
df_cat = pd.get_dummies(X_[cat_ix])
# binning of numerical features
x = X_.drop(columns=cat_ix, axis=1)
est = KBinsDiscretizer(n_bins=3, encode='onehot-dense', strategy='uniform')
df_num = est.fit_transform(x)
X = pd.concat([df_cat.reset_index(drop=True), pd.DataFrame(df_num).reset_index(drop=True)], axis=1)
# split training and test data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
X_tr = Variable(torch.tensor(X_train.values, dtype=torch.float))
X_te = Variable(torch.tensor(X_test.values, dtype=torch.float))
y_tr = Variable(torch.tensor(y_train, dtype=torch.long))
y_te = Variable(torch.tensor(y_test, dtype=torch.long))

def binary_cross_entropy_one_hot(input, target):
    return torch.nn.CrossEntropyLoss()(input, target)

def _accuracy(y_pred, y_true):
    classes = torch.argmax(y_pred, dim=1)
    labels = y_true
    accuracy = torch.mean((classes == labels).float())
    return accuracy

model = Model(X.shape[1])
learning_rate = 1e-3
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
epochs = 1000
accuracy = 0.0
minibatch = True

# training loop
train_loss = []
for epoch in range(epochs):
    if minibatch:
        batch_size = 128  # or whatever
        permutation = torch.randperm(X_tr.size()[0])
        for i in range(0, X_tr.size()[0], batch_size):
            optimizer.zero_grad()
            indices = permutation[i:i + batch_size]
            batch_x, batch_y = X_tr[indices], y_tr[indices]
            # in case you wanted a semi-full example
            outputs = model.forward(batch_x)
            loss = binary_cross_entropy_one_hot(outputs, batch_y)
            loss.backward()
            optimizer.step()
        if epoch % 100 == 0:
            print(f'epoch: {epoch:2}  loss: {loss:10.8f}')
        # train_ds = TensorDataset(X_tr, y_tr)
        # train_dl = DataLoader(train_ds, batch_size=256, shuffle=True)
        # batch_loss = 0.0
        # batch_accuracy = 0.0
        # for nb, (x_batch, y_batch) in enumerate(train_dl):  # manually set number of batches?
        #     optimizer.zero_grad()
        #     y_pred_train = model(x_batch)
        #     loss = binary_cross_entropy_one_hot(y_pred_train, y_batch)
        #     loss.backward()
        #     optimizer.step()
        #     batch_loss += loss.item()
        #     batch_accuracy += _accuracy(y_pred_train, y_batch)
        # train_loss.append(batch_loss / (nb + 1))
        # accuracy = batch_accuracy / (nb + 1)
        # if epoch % 100 == 0:
        #     print(f'epoch: {epoch:2}  loss: {train_loss[epoch]:10.8f}')
    else:
        optimizer.zero_grad()
        y_pred = model(X_tr)
        # computing the loss function
        loss = binary_cross_entropy_one_hot(y_pred, y_tr)
        if epoch % 100 == 0:
            print(f'epoch: {epoch:2}  loss: {loss.item():10.8f}')
        loss.backward()
        optimizer.step()
        accuracy = _accuracy(y_pred, y_tr)
# evaluation on test data
with torch.no_grad():
    model.eval()
    y_pred = model(X_te)
    test_loss = binary_cross_entropy_one_hot(y_pred, y_te)
    test_acc = _accuracy(y_pred, y_te)
print("Loss on test data: {:.4}".format(test_loss))
print("Accuracy on test data: {:.4}".format(test_acc))


title: Why RNN need two bias vectors?

body:
In pytorch RNN implementation, there are two biases, b_ih and b_hh.
Why is this? Is it different from using one bias? If yes, how? Will it affect performance or efficiency?


title: Matching PyTorch w/ CNTK (VGG on CIFAR)

body:
I am trying to understand how PyTorch works and want to replicate a simple CNN training on CIFAR. The CNTK script gets to 0.76 accuracy after 168 seconds of training (10 epochs), which is similar to my MXNet script (0.75 accuracy after 153 seconds).

However, my PyTorch script is lagging behind a lot at 0.71 accuracy and 354 seconds. I appreciate I will get differences in accuracy due to stochastic weight initialisation, etc. However the difference across frameworks is much greater than difference within a framework, initialising randomly between runs.

The reasons I can think of:

MXNet and CNTK are initialized to xavier/glorot uniform; not sure how to do this in PyTorch and so perhaps the weights are initialised to 0
CNTK does gradient-clipping by default; not sure if PyTorch has the equivalent
Perhaps the bias is dropped in PyTorch by default
I use SGD with momentum; perhaps the PyTorch implementation of momentum is a bit different

Edit:

I have tried specifying the weight initialisation, however it seems to have no big effect:

self.conv1 = nn.Conv2d(3, 50, kernel_size=3, padding=1)
init.xavier_uniform(self.conv1.weight, gain=np.sqrt(2.0))
init.constant(self.conv1.bias, 0)


title: PyTorch equivalent of a Tensorflow loss function

body:
I was trying to reimplement a TensorFlow code using PyTorch framework. Below I have included the TF sample code and my PyT interpretation, for a target of size (Batch, 9, 9, 4) and a network output of size (Batch, 9, 9, 4)

TensorFlow implementation:

loss = tf.nn.softmax_cross_entropy_with_logits(labels=target, logits=output)
loss = tf.matrix_band_part(loss, 0, -1) - tf.matrix_band_part(loss, 0, 0)

PyTorch implementation:

output = torch.tensor(output, requires_grad=True).view(-1, 4)
target = torch.tensor(target).view(-1, 4).argmax(1)

loss = torch.nn.CrossEntropyLoss(reduction='none')
my_loss = loss(output, target).view(-1,9,9)

For the PyTorch implementation, I'm not sure how to implement tf.matrix_band_part. I was thinking about defining a mask, but I was not sure if that would hurt the backpropagation or not. I am aware of torch.triu, but this function does not work for tensors with more than 2 dimensions.


title: With pytorch/python, is it better to overwrite variables or define new ones?

body:
Suppose I have an algorithm that does the following in python with pytorch. Please ignore whether the steps are efficient. This is just a silly toy example.

 def foo(input_list):
      # input_list is a list of N 2-D pytorch tensors of shape (h,w)
      tensor = torch.stack(input_list) # convert to tensor.shape(h,w,N)
      tensor1 = torch.transpose(tensor,0,2).unsqueeze(1) # convert to tensor.shape(N,1,h,w)
      tensor2 = torch.interpolate(tensor1,size=(500,500) # upsample to new shape (N,1,500,500)

 def bar(input_list):
      tensor = torch.stack(input_list) # convert to tensor.shape(h,w,N)
      tensor = torch.transpose(tensor,0,2).unsqueeze(1) # convert to tensor.shape(N,1,h,w)
      tensor = torch.interpolate(tensor,size=(500,500) # upsample to new shape (N,1,500,500)

My question is whether it makes more sense to use method foo() or bar() or if it doesn't matter. My thought was that I save memory by rewriting over the same variable name (bar), since I will never actually need those intermediate steps. But if the CUDA interface is creating new memory spaces for each function, then I'm spending the same amount of memory with both methods.


title: Fixing error img should be PIL Image. Got

body:
I tried to create Custom dataset but when show the some images it had the error. Here is my Dataset class and tranforms:
transforms = transforms.Compose([transforms.Resize(224,224)])

class MyDataset(Dataset):
    def __init__(self, path, label,  transform=None):
        self.path = glob.glob(os.path.join(path, '*.jpg'))
        self.transform = transform
        self.label = label

    def __getitem__(self, index):
        img = io.imread(self.path[index])
        img = torch.tensor(img)
        labels = torch.tensor(int(self.label))
        if self.transform:
          img = self.transform(img)
        return (img,labels)

    def __len__(self):
        return len(self.path)

And here error line:
images, labels = next(iter(train_loader))


title: how to detach list of pytorch tensors to array

body:
There is a list of PyTorch's Tensors and I want to convert it to array but it raised with error:

'list' object has no attribute 'cpu'

How can I convert it to array?
import torch
result = []
for i in range(3):
    x = torch.randn((3, 4, 5))
    result.append(x)
a = result.cpu().detach().numpy()


title: How to run a pytorch project with CPU?

body: A Pytorch project is supposed to run on GPU. I want to run it on my laptop only with CPU. There are a lot of places calling .cuda() on models, tensors, etc., which fail to execute when cuda is not available. Is it possible to do it without changing the code everywhere?


title: Problems with LSTM model

body:
I try to realise LSTM model in PyTorch and got such problem: loss don't reduce.
My task is so: I have sessions with different features. Session length is fixed and equals to 20. My goal is to predict will the last session been skipped or not.
I tried to scale input features, I tried to pass target into features(maybe provided features are absolutely uninformative, I thought this should lead to overfitting and loss should be near 0), but always my loss reduction looks like this:

print(X.shape)
#(82770, 20, 31) where 82770 is count of sessions, 20 is seq_len, 31 is count of features
print(y.shape)
#(82770, 20)

I defined also get_batches function. And yes, I know about problems with last batch in this generator

def get_batches(X, y, batch_size):
'''Create a generator that returns batches of size
   batch_size x seq_length from arr.
'''
assert X.shape[0] == y.shape[0]
assert X.shape[1] == y.shape[1]
assert len(X.shape) == 3
assert len(y.shape) == 2

seq_len = X.shape[1]
n_batches = X.shape[0]//seq_len

for batch_number in range(n_batches):
    #print(batch_number*batch_size, )
    batch_x = X[batch_number*batch_size:(batch_number+1)*batch_size, :, :]
    batch_y = y[batch_number*batch_size:(batch_number+1)*batch_size, :]
    if batch_x.shape[0] == batch_size:
        yield batch_x, batch_y
    else:
        print('batch_x shape: {}'.format(batch_x.shape))
        break

Here is my RNN

class BaseRNN(nn.Module):

def __init__(self, n_features, hidden_size, n_layers, drop_p=0.3, lr=0.001, last_items=10):
    super(BaseRNN, self).__init__()
    # constants
    self.n_features = n_features
    self.hidden_size = hidden_size
    self.n_layers = n_layers
    self.drop_p = drop_p
    self.lr = lr
    self.last_items = last_items

    # layers
    self.lstm = nn.LSTM(
        n_features, n_hidden, n_layers,
        dropout=drop_p, batch_first=True
    )
    self.dropout = nn.Dropout(self.drop_p)
    self.linear_layer = nn.Linear(self.hidden_size, 1)
    self.sigm = nn.Sigmoid()

def forward(self, x, hidden):
    out, hidden = self.lstm(x, hidden)
    batch_size = x.shape[0]
    out = self.dropout(out)
    out = out.contiguous().view(-1, self.hidden_size)
    out = self.linear_layer(out)
    out = self.sigm(out)
    # use only last elements
    out = out.view(batch_size, -1)
    out = out[:, -1]
    return out, hidden

def init_hidden(self, batch_size):
    #initialize with zeros
    weight = next(self.parameters()).data
    hidden = (weight.new(self.n_layers, batch_size, self.hidden_size).zero_(),
                  weight.new(self.n_layers, batch_size, self.hidden_size).zero_())

    return hidden

Here is my train function:

def train(net, X, y,
      n_epochs=10, batch_size=10, clip=5):
'''
pass
'''
n_features = X.shape[2]
seq_len = X.shape[1]
net.train()
opt = torch.optim.Adam(net.parameters(), lr=net.lr)
criterion = nn.BCELoss()
counter = 0
losses = []
for e in range(n_epochs):
    h = net.init_hidden(batch_size)
    for x, y in get_batches(X=X, y=y, batch_size=batch_size):
        counter += 1
        h = net.init_hidden(batch_size)
        inputs, targets = torch.from_numpy(x).float(), torch.from_numpy(y.astype(int))
        targets = targets[:,-net.last_items:].float().view(net.last_items*batch_size)
        h = tuple([each.data for each in h])
        net.zero_grad()
        output, h = net(inputs, h)
        loss = criterion(output.view(net.last_items*batch_size), targets)
        losses.append(loss.item())
        loss.backward()
        nn.utils.clip_grad_norm_(net.parameters(), clip)
        opt.step()
return losses

Run training:

n_hidden = 100
n_layers = 1
n_features = X.shape[2]
net = BaseRNN(n_features, n_hidden, n_layers,
              lr=0.01, drop_p=0.1, last_items=1)

losses = train(net, X, y, n_epochs=5, batch_size=1000, lr=0.001, clip=5)
plt.plot(losses)

After all these steps I get plot like in the top of my question. I think I get a huge error somewhere because I put target variable in features, but still no loss reduction.
Where I am wrong?

PS.How to generate sample data? I will use real y data and add some noise.

Y = np.array([[0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 1],
       [1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
       [1, 1, 1, 1, 1, 1, 1, 0, 1, 0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1],
       [0, 1, 1, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0],
       [0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
       [1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 1, 1, 1, 1],
       [1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 0, 1, 1, 1]])
print(Y.shape)
#(10, 20)

# add 5 features with random noise
random_noise = np.random.randn(10*20*5).reshape(10,20,5)
X = np.concatenate((Y.reshape(10,20,1), random_noise), axis=2)
print(X.shape)
#(10, 20, 6)


title: Pytorch NN and communication between classes

body:
I am new at python, and pytorch, and I have a problem understanding how it works.

    import torch.nn as nn
    import torch.nn.functional as F
    import torch.optim as optim

    class Net(nn.Module):
        def __init__(self):
            ..
        def forward(self, x):
            ..
            return x
    net = Net()

criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)

for epoch in range(2):  # loop over the dataset multiple times

    running_loss = 0.0
    for i, data in enumerate(trainloader, 0):
        inputs, labels = data
        optimizer.zero_grad()
        outputs = net(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

So, this is the code, and I draw what I understand from the code in the picture.
I have some questions:

A) Why could not I use nn.CrossEntropy instead of 'criterion' directly in the code? What difference does it make if I assign it to a variable? I get this error: bool value of Tensor with more than one value is ambiguous

B) Why when the class Net gets an object (nn) (I assumed when 'as' is used, an object is created), then class Net can simply use backward afterward? It is supposed to be part of nn, not Net. Could you please make sense of it for me?

C) While optim is a different object, how do the parameters, which were optimized by optim, can affect nn? I don't understand how they pass variable and update each other?


title: How can I save model weights to mlflow tracking sever using pytorch-lightning?

body:
I would like to save model weights to mlflow tracking using pytorch-lightning.
pytorch-lightning supports logging.
However, it seems that saving model weights as a artifact on mlflow is not supported.

At first, I planed to override ModelCheckpoint class to do it, but I found it is difficult for me because of complex Mixin operations.

Anybody knows simple way to accomplish it?


title: Pytorch isnt found by my code using import torch

body:
I have installed torch using pip. And when I try to run

import torch

I get an error saying

raceback (most recent call last):
  File "C:\Users\leoga\Documents\AI Image Classifier\Classifier.py", line 2, in <module>
    import torch
  File "C:\Python38\lib\site-packages\torch\__init__.py", line 81, in <module>
    ctypes.CDLL(dll)
  File "C:\Python38\lib\ctypes\__init__.py", line 373, in __init__
    self._handle = _dlopen(self._name, mode)
FileNotFoundError: Could not find module 'C:\Python38\lib\site-packages\torch\lib\caffe2_nvrtc.dll'. Try using the full path with constructor syntax.

I am using python 3.8


title: How does pytorch compute derivatives for simple functions?

body:
When we talk about the auto-differentiation in the pytorch, we are usually presented a graphical structures of tensors based on their formulas, and pytorch will compute the gradients by tracing down the graphical tree using chain rules. However, I want to know what will happen at the leaf nodes? Does pytorch hardcode a whole list of basic functions with their analytical derivatives, or does it compute the gradients using numerical methods? A quick example:
import torch

def f(x):
    return x ** 2
x = torch.tensor([1.0], requires_grad=True)
y = f(x)
y.backward()
print(x.grad) # 2.0

In this example, does pytorch compute the derivative by $$ (x^2)' = 2x = 2 * 1 = 2 $$, or does pytorch compute in a way similar to $$ (1.00001^2 - 1^2) / (1.000001 - 1) ~ 2 $$ ?
Thanks!