title: Preprocessing a video in android for pytorch

body:
What is the best way to preprocess video data in Android Kotlin, in preparation to feed into a PyTorch Android model? Specifically, I have a ready-made model in PyTorch, and I've converted it to be ready for PyTorch Mobile.
During training, the model takes in raw footage from phones and is preprocessed to (1) be greyscale, and (2) compressed to a specific smaller resolution that I have specified, (3) converted to a Tensor to be fed into a neural net (or potentially send the compressed video to a remote server). I use OpenCV for this, but I'm wondering what the easiest way to do this would be in Android Kotlin?
Python code for reference:

def save_video(filename):

    frames = []

    cap = cv2.VideoCapture(filename)
    frameCount = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
    frameWidth = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
    frameHeight = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))

    buf_c = np.empty((frameCount, frameHeight, frameWidth, 3), np.dtype('uint8'))
    buf = np.empty((frameCount, frameHeight, frameWidth), np.dtype('uint8'))

    fc = 0
    ret = True

    # 9:16 ratio
    width = 121
    height = 216
    dim = (width, height)

    # Loop until the end of the video
    while fc < frameCount and ret:
        ret, buf_c[fc] = cap.read()

        # convert to greyscale
        buf[fc] = cv2.cvtColor(buf_c[fc], cv2.COLOR_BGR2GRAY)

        # reduce resolution
        resized = cv2.resize(buf[fc], dim, interpolation = cv2.INTER_AREA)

        frames.append(resized)
        fc += 1

    # release the video capture object
    cap.release()

    # Closes all the windows currently opened.
    cv2.destroyAllWindows()

    return frames


title: OpenNMT confusion use of `Optim.updateLearningRate`: `valid_loss` used as `ppl` arg

body:
In train.py line 246, we have

# update the learning rate
optim.updateLearningRate(valid_loss, epoch)

However, in onmt/Optim.py, inside function updateLearningRate() at line 37, we have:

def updateLearningRate(self, ppl, epoch):
...
    self.last_ppl = ppl

Why use valid_loss in the place of ppl for updateLearningRate()?

Aren't valid_loss and ppl different measures?

Could anyone help me here, Thanks!


title: Mismatching Conda and Pycharm

body:
I'm new in python and I'm confused about  mismatching Conda packages list and Pycharm. In a project I need to install pytorch. Installing with pycharm lead to some error and when I install it through conda, It does not appear in pycharm. Both list is the same env.

Thanks in advance.

pycharm list

Anaconda list


title: PyTroch, Gradient calculations

body:
https://colab.research.google.com/github/pytorch/tutorials/blob/gh-pages/_downloads/neural_networks_tutorial.ipynb

Hi I am trying to understand the NN with pytorch.
I have doubts in gradient calculations..

import torch.optim as optim

create your optimizer
optimizer = optim.SGD(net.parameters(), lr=0.01)

```
# in your training loop:
optimizer.zero_grad()   # zero the gradient buffers
output = net(input)
loss = criterion(output, target)
loss.backward()
optimizer.step()    # Does the update
```

From the about code, I understood loss.backward() calculates the gradients.
I am not sure, how these info shared with optimizer to update the gradient.

Can anyone explain this..

Thanks in advance !


title: What is the logic behind this assignment: understanding in-place assignment operations in numpy

body:
I have two fairly simple codes that give different answer. I understand it is due to the reference shared but I am not very clear what exactly happens in the 2nd case

a = np.ones(5)
b = torch.from_numpy(a)
a=np.add(a, 1, out=a)
print(a)
print(b)

  [out]:

  [2. 2. 2. 2. 2.]

  tensor([2., 2., 2., 2., 2.], dtype=torch.float64)

a = np.ones(5)
b = torch.from_numpy(a)
a=a+1
print(a)
print(b)

  [out]:

  [2. 2. 2. 2. 2.]

  tensor([1., 1., 1., 1., 1.], dtype=torch.float64)

Why isn't b changed in the second case ?


title: Pytorch can not save torchvision models pretrained weights

body:
I try to fine tune the torchvision models for some classification tasks. When I trained the inception code and saved the best accuracy model, some strange things happened. This is simplified train and test code:

net = torchvision.models.inception_v3(pretrained=True)
for epoch in range(60):
    for data in trainLoader:
         ....
    for data in testLoader:
         ....
    if test_acc > best_acc:
       best_acc = test_acc
       best_model_state_dict = copy.deepcopy(net.state_dict())
       best_epoch = epoch
       state = {
            "epoch": best_epoch,
             "model_state": best_model_state_dict,
             "best_acc": best_acc
            }

       save_path = "inception_model/{}_inceptionv3_{}_classroomClassification.pkl".format(str(best_acc), exp_i)
       torch.save(state, save_path)

When I seperately test the same data using the code below, the accuracy is lower than the value recorded.

net = torchvision.models.inception_v3()
net.aux_logits = False
net.fc = nn.Linear(2048, 6)
model_path = r"D:\aha\0.8785714285714286_inceptionv3_2_classroomClassification.pkl"

state = torch.load(model_path)
net.load_state_dict(state["model_state"])
net.eval().cpu()
conf_mat = np.zeros([6, 6])
test_running_loss = 0.0
test_acc = 0.0
for data in tqdm(testLoader):
    img_feature, label = data
    img_feature = Variable(img_feature).cpu()
    net.eval()
    img_feature = Variable(img_feature).cpu()
    label = Variable(label).long().cpu()
    out = net(img_feature)
    pred = torch.max(out.data, 1)[1]
    test_acc += torch.sum(pred == label).item()
test_acc = test_acc / len(testData)
print(test_acc)

But when I change net = torchvision.models.inception_v3() into net = torchvision.models.inception_v3(pretrained=True), the accuracy is changed back. This seems like that pytorch can not save the pretrained weights.


title: Size Mismatch when passing a state batch to network

body:
Since I’m a beginner in ML, this question or the design overall may sound silly, sorry about that. I’m open to any suggestions.

I have a simple network with three linear layers one of which is output layer.

self.fc1 = nn.Linear(in_features=2, out_features=12)
self.fc2 = nn.Linear(in_features=12, out_features=16)
self.out = nn.Linear(in_features=16, out_features=4)

My states are consisting of two values, coordinate x and y. That’s why input layer has two features.

In main.py I’m sampling and extracting memories in ReplayMemory class and pass them to get_current function:

    experiences = memory.sample(batch_size)
    states, actions, rewards, next_states = qvalues.extract_tensors(experiences)

    current_q_values = qvalues.QValues.get_current(policy_net, states, actions)

Since a single state is consisting of two values, length of the states tensor is batchsize x 2 while length of the actions is batchsize. (Maybe that’s the problem?)

When I pass “states” to my network in get_current function to obtain predicted q-values for the state, I get this error:

size mismatch, m1: [1x16], m2: [2x12]

It looks like it is trying to grab the states tensor as if it is a single state tensor. I don’t want that. In the tutorials that I follow, they pass the states tensor which is a stack of multiple states, and there is no problem. What am I doing wrong? :)

This is how I store an experience:

memory.push(dqn.Experience(state, action, next_state, reward))

This is my extract tensors function:

def extract_tensors(experiences):
    # Convert batch of Experiences to Experience of batches
    batch = dqn.Experience(*zip(*experiences))

    state_batch = torch.cat(tuple(d[0] for d in experiences))
    action_batch = torch.cat(tuple(d[1] for d in experiences))
    reward_batch = torch.cat(tuple(d[2] for d in experiences))
    nextState_batch = torch.cat(tuple(d[3] for d in experiences))

    print(action_batch)

    return (state_batch,action_batch,reward_batch,nextState_batch)

Tutorial that I follow is this project's tutorial.

https://github.com/nevenp/dqn_flappy_bird/blob/master/dqn.py

Look between 148th and 169th lines. And especially 169th line where it passes the states batch to the network.


title: Pytorch DataLoader pipelining

body:
I would expect DataLoader to load batches concurrently to the main process, filling up the buffer as soon as a batch is consumed from the buffer. However, when I track the utilization of GPU and order of loading vs execution I see some different behavior:

loading of the whole buffer (expected)
consuming the whole buffer by execution of all the batches until the buffer is empty (not expected)
loading the whole buffer again without parallel execution (not expected)
goto 2.

This obviously results in dips in GPU utilization when in step 3.

I set:

num_workers >= 1

pin_memory = True/False (doesn't influence the described behavior)

Did anyone experience the same? What could be the issue?


title: How do I crop a Landsat image into smaller chunks for training and then predict on the original image

body:
I am looking at using Landsat imagery to train a CNN for unsupervised pixel-wise semantic segmentation classification.  That said, I have been unable to find a method that allows me to crop images from the larger Landsat image for training and then predict on the original image. Essentially here is what I am trying to do:

Original Landsat image (5,000 x 5,000 - this is an arbitrary size, not exactly sure of the actual dimensions off-hand) -> crop the image into (100 x 100) chunks -> train the model on these cropped images -> output a prediction for each pixel in the original (uncropped) image.

That said, I am not sure if I should predict on the cropped images and stitch them together after they are predicted or if I can predict on the original image.

Any clarification/code examples would be greatly appreciated.  For reference, I use both pytorch and tensorflow.

Thank you!
Lance D


title: Importing OpenCV after importing PyTorch messes with CPU affinity

body:
This is on the extreme end of weird software issues I ever had to deal with. Looking for help!

Consider this Python script:

import psutil
import torch

def run():
    curr_process = psutil.Process()
    curr_process.cpu_affinity([3])
    print('Before', curr_process.cpu_affinity())
    import cv2
    print('After', curr_process.cpu_affinity())

import multiprocessing
multiprocessing.Process(target=run).start()

You would think that importing OpenCV has nothing to do with CPU affinity and the output before and after would be the same, but no. The output of this script on my machine is:

Before [3]
After [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]

The issue, of course, arose in a big application and it took considerable effort to boil it down to this minimal example. I found that reproducing it requires a key condition: PyTorch and OpenCV should be imported from different processes. One before, and one after the fork. It does not reproduce if I import both in the child process, or in the main process, but it does reproduce if I swap cv2 and torch in this script.

First of all, what does importing the library has to do with the CPU affinity? Perhaps this is some kind of optimization attempt by the authors of the library that went rogue?

Then, how do I fix something like this? I could, of course, import everything before fork as a quick hacky solution, but I'd like to get to the bottom of things.

My versions of libraries (both installed from Conda repo):

opencv=4.1.0=py37h3aa1047_6
pytorch=1.4.0=py3.7_cuda10.1.243_cudnn7.6.3_0


title: Sum over indices with value (1)

body:
I have a tensor with a shape (2, 2, 3) like:

a= tensor ([[[2, 0, 2],[1, 0, 0]],[[1, 0, 1],[0, 1, 0]]])

I want to find the indices of the values (1), then make 2 to the power of those indices and finally add the results for the last dimension, so the final result should be like:

tensor ([[[],[2^0]],[[2^0+2^2],[2^1]]])

My actual tensor is much bigger than this example, so I don’t want to use "for" loop, and I have to use broadcasting…

I was thinking of something like torch.pow(2,(a == 1).nonzero()).sum(), but it doesn’t work. I have to find a way to apply (a== 1).nonzero() only for the last dimension, any suggestion? Thanks.


title: Array indexing in pytorch

body:
I am experiencing weird behavior while dealing with the following scenario. tne is the batch size, while N is the total number of possible locations for each example. Each example has exactly 24 scalar outputs, where their locations are stored in dof tensor (size of dof is (tne X 24)). Fint_e has a size of tneX24 (i.e., the 24 outputs for each example). I am trying to construct a large tensor, which has a size of tne X N. When I do the following, it fills in the wrong manner. Any advice?

Fint_MAT        = torch.zeros((tne,N))
Fint_MAT[:,dof]  = Fint_e

The dof tensor, which has the size of batch size X 24, has different indices for each example, but each example has in total 24 indices.

For instance,

dof[0,:] = 0, 1, 6, 9, … (24 in total)

dof[1,:] = 1,100, 151, 300,… (24 in total)

Any hint would be appreciated.

I include below a simple scenario for better elaboration:

tne = 3
N   = 48
Fint_MAT = torch.zeros((tne,N))
Fint_e   = torch.randn((tne, 24))
v1 = torch.arange(24).unsqueeze(0)
v2 = torch.arange(12, 36).unsqueeze(0)
v3 = torch.arange(24, 48).unsqueeze(0)
dof      = torch.cat((v1,v2,v3), axis=0).long()
Fint_MAT[:,dof]  = Fint_e


title: Access lower dimensional encoded data of autoencoder

body:
Here is an autoencoder I’m working on from tutorial:https://debuggercafe.com/implementing-deep-autoencoder-in-pytorch/

I’m just learning about autoencoders and I’ve modified the source encode a custom small dataset which consists of:

[0,1,0,1,0,1,0,1,0],[0,1,1,0,0,1,0,1,0],[0,1,1,0,0,1,0,1,0],[0,1,1,0,0,1,0,1,0]

It seems to work ok, but I’m unsure how to access the lower dimensional embedding values of dimension 2 (set by parameter out_features).

I've added a methods to the Autoencoder class to return the embedding , is this the recommended method of accessing the embedding's ?

Code:

import torch
import torchvision
from torch import nn
from torch.autograd import Variable
from torch.utils.data import DataLoader
from torchvision import transforms
from torchvision.datasets import MNIST
from torchvision.utils import save_image
import warnings
import os

# import packages
import os
import torch
import torchvision
import torch.nn as nn
import torchvision.transforms as transforms
import torch.optim as optim
import matplotlib.pyplot as plt
import torch.nn.functional as F

from torchvision import datasets
from torch.utils.data import DataLoader
from torchvision.utils import save_image
import numpy as np

# utility functions
def get_device():
    if torch.cuda.is_available():
        device = 'cuda:0'
    else:
        device = 'cpu'
    return device

device = get_device()

features = torch.tensor(np.array([ [0,1,0,1,0,1,0,1,0],[0,1,1,0,0,1,0,1,0],[0,1,1,0,0,1,0,1,0],[0,1,1,0,0,1,0,1,0] ])).float()
tic_tac_toe_data_loader = torch.utils.data.DataLoader(features, batch_size=1, shuffle=True)

class Encoder(nn.Module):
    def __init__(self):
        super(Encoder, self).__init__()
        self.fc1 = nn.Linear(in_features=9, out_features=2)

    def forward(self, x):
        return F.sigmoid(self.fc1(x))

class Decoder(nn.Module):
    def __init__(self):
        super(Decoder, self).__init__()
        self.fc1 = nn.Linear(in_features=2, out_features=9)

    def forward(self, x):
        return F.sigmoid(self.fc1(x))

class Autoencoder(nn.Module):

    def __init__(self):
        super(Autoencoder, self).__init__()
        self.fc1 = Encoder()
        self.fc2 = Decoder()

    def forward(self, x):
        return self.fc2(self.fc1(x))

net = Autoencoder()
net.to(device)

NUM_EPOCHS = 50
LEARNING_RATE = 1e-3

criterion = nn.MSELoss()
optimizer = optim.Adam(net.parameters(), lr=LEARNING_RATE)

# image transformations
transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((0.5,), (0.5,))
])

outputs = None

def train(net, trainloader, NUM_EPOCHS):
    train_loss = []
    for epoch in range(NUM_EPOCHS):
        running_loss = 0.0
        for data in trainloader:
            img = data
            img = img.to(device)
            img = img.view(img.size(0), -1)
#             print('img.shape' , img.shape)
            optimizer.zero_grad()

            outputs = net(img)
            loss = criterion(outputs, img)
            loss.backward()
            optimizer.step()
            running_loss += loss.item()

        loss = running_loss / len(trainloader)
        train_loss.append(loss)

    return train_loss

# train the network
train_loss = train(net, tic_tac_toe_data_loader, NUM_EPOCHS)

I can access the lower dimensional embedding using

print(Encoder().forward( torch.tensor(np.array([0,1,0,1,0,1,0,1,0])).float()))

But is this using the trained weight values for the embedding ? If I call Encoder multiple times with same values:

print(Encoder().forward( torch.tensor(np.array([0,1,0,1,0,1,0,1,0])).float()))
print(Encoder().forward( torch.tensor(np.array([0,1,0,1,0,1,0,1,0])).float()))

different results are returned:

tensor([0.5083, 0.5020], grad_fn=<SigmoidBackward>)
tensor([0.4929, 0.6940], grad_fn=<SigmoidBackward>)

Why is this the case ? Is an extra training step being invoked as a result of calling Encoder ?


title: Estimate the std and mean of a large image dataset using Welford method using pytorch

body:
I’m trying to estimate the std and mean of a large image dataset using a Welford algorithm. As far as I read this algorithm is more robust than the method that accumulates the sum and the sum square of the values. However, I’m facing some issues. The first one is that I tested the algorithm in a small random dataset(6 images) and the mean and std estimated by Welford method are not exactly the same as the mean and std calculated using the whole dataset at the same time. The difference is very small, but I’m not sure if I’m doing something wrong.

import torch
torch.set_printoptions(precision=10)
class ImgRunningStats:

        def __init__(self):
            self.n = torch.zeros(1, dtype=torch.long)
            self.old_m = torch.zeros(3, dtype=torch.float64)
            self.new_m = torch.zeros(3, dtype=torch.float64)
            self.old_s = torch.zeros(3, dtype=torch.float64)
            self.new_s = torch.zeros(3, dtype=torch.float64)

        def push(self, x):
            self.n += 1

            if self.n == 1:
                self.old_m = self.new_m = x
                self.old_s = 0
            else:
                self.new_m = self.old_m + (x - self.old_m) / self.n
                self.new_s = self.old_s + (x - self.old_m) * (x - self.new_m)

                self.old_m = self.new_m
                self.old_s = self.new_s

        def mean(self):
            return self.new_m if self.n else 0.0

        def var(self):
            return self.new_s / (self.n - 1) if self.n > 1 else 0.0

        def std(self):
            return torch.sqrt(self.var())

    rs= ImgRunningStats()

    test_img_0 = torch.rand([2,3,250,250]) test_img_1 = torch.rand([2,3,250,250]) test_img_2 = torch.rand([2,3,250,250])

    images = [test_img_0, test_img_1, test_img_2]

    test_img = torch.cat([test_img_0, test_img_1, test_img_2], dim=0)

    print('mean full: ', test_img.mean(dim=[0,2,3])) print('std full: ', test_img.std(dim=[0,2,3], unbiased=True))

    for img in images:
        pixels = img.view(-1, img.shape[1])
        for px in pixels:
            rs.push(px)

    print(f'Mean Welford: {rs.mean()}, Std Welford: {rs.std()}')

However, the major issue that I have is that the algorithm is very slow… I don’t know how I can improve the performance of the algorithm. As far as I saw, it is not feasible to run this algorithm in a large image dataset with the current implementation.

Is it possible to parallelize this algorithm?


title: How to implement a custom dataset to pytorch project

body:
I’d like to train a NN with a given dataset (all including some kind of object, for example: a dog), after the training the NN should help me classifying my images (downloaded from instagram) as “image includes a dog (with probability:0.XX)” or “image doesn’t include a dog (with probability: 0.XX)”.

Obviously images from instagram-images do not always have the same size (but they all have the same format (.jpg) due to filtering), and the images from my dataset do not have the same size as well.

While testing, I'm getting this Error:

    Traceback (most recent call last):
  File "/venv/nn.py", line 129, in <module>
    train(model=globalModel, hardware=hw, train_loader=loader, optimizer=optimizer, epoch=1)
  File "/venv/nn.py", line 74, in train
    for batch_idx, (data, target) in enumerate(train_loader):
  File "\venv\lib\site-packages\torch\utils\data\dataloader.py", line 345, in __next__
    data = self._next_data()
  File "\venv\lib\site-packages\torch\utils\data\dataloader.py", line 384, in _next_data
    index = self._next_index()  # may raise StopIteration
  File "\venv\lib\site-packages\torch\utils\data\dataloader.py", line 339, in _next_index
    return next(self._sampler_iter)  # may raise StopIteration
  File "\venv\lib\site-packages\torch\utils\data\sampler.py", line 200, in __iter__
    for idx in self.sampler:
  File "\venv\lib\site-packages\torch\utils\data\sampler.py", line 62, in __iter__
    return iter(range(len(self.data_source)))
TypeError: object of type 'type' has no len()

with this code: https://pastebin.com/DcvbeMcq

Does anyone know how to implement a custom dataset right?


title: How to overlap NCCL communication with PyTorch training

body:
I am using PyTorch to distributed train a model. What I want to do is to overlap NCCL communication with the backward(computing) of the model. I am using python multi-threading. However, it seems that when the two threads are running together, the time of backward increases.
Here is part of my code related to this problem. I have simplified my code to make it clearer.
semaphore = threading.Semaphore(0)

def bk_hook(*args):
    semaphore.release()

for layer in model.__dict__['_modules'].items():
    layer[1].register_backward_hook(bk_hook)

def training():
    # trains the model

def communication():
    global model, semaphore
    layers = len(list(model.__dict__['_modules']))
    for layer in range(layers):
        semaphore.acquire()
        dist.broadcast(...) # NCCL communication

t1 = threading.Thread(target=training)
t2 = threading.Thread(target=communication)
t1.start()
t2.start()

I have tried running a thread that only trains the model and another thread that does not use NCCL. In this case, the time of backward does not increase. So it is NCCL communication that leads to the increase of backward time. I am using 3 nodes on one server, in which case NCCL uses cudaMemcpy. I wonder why the communication and the computing cannot overlap.


title: Propagation of hidden state in LSTM while training to identify trigger points in a sequence

body:
I have slightly modified the PyTorch LSTM example from here. My aim is to train the LSTM on continuous time-series data in order to identify trigger points.
Basically, the input to my network would be a float-vector and the output would be a binary classification (trigger point - yes or no). The trigger is a gradual event, and also dependent on the last few vectors in the sequence. So, I need to propagate the hidden state across the training iterations.
My question is - whether this is the right way to use an LSTM? Specifically this part - taking the hidden state produced by the LSTM cell in the current iteration, detaching it from the computational graph, and passing it on to the LSTM again in the next iteration? I'm new to LSTMs and would appreciate any insights in this regard.
class LSTMTagger(nn.Module):

    def __init__(self, embedding_dim, hidden_dim, tagset_size):
        super(LSTMTagger, self).__init__()
        self.hidden_dim = hidden_dim

        # The LSTM takes word embeddings as inputs, and outputs hidden states
        # with dimensionality hidden_dim.
        self.lstm = nn.LSTM(embedding_dim, hidden_dim)

        # The linear layer that maps from hidden state space to tag space
        self.hidden2tag = nn.Linear(hidden_dim, tagset_size)
        self.softmax = nn.LogSoftmax(dim=2)
        # initialize the hidden state.
        self.hidden = (torch.randn(1, 1, hidden_dim),
                  torch.randn(1, 1, hidden_dim))

    def forward(self, embeds):
        lstm_out, (ht, ct) = self.lstm(embeds, self.hidden)
        self.hidden = (ht.detach(), ct.detach())     # Is this correct ?
        tag_space = self.hidden2tag(lstm_out)
        tag_scores = self.softmax(tag_space)
        return tag_scores.view(-1, 2)


title: Different kernels within a same batch for a causal convolution

body:
Is there a way in PyTorch (1.6) to compute dynamic causal convolution: where the kernel size is different per sample? One could use the argument group of F.conv1d. However, the kernels must have the same shape, which makes the padding complicate.
For example, I took a similar case with 5 inputs and 5 differents kernels, I obtain a wrong output due to the largest kernel_size:
import torch
import torch.nn as nn
import torch.nn.functional as F

inputs1 = [0 for _ in range(20)]
inputs1[8] = 1
inputs2 = [0 for _ in range(20)]
inputs2[13] = 1
inputs3 = [0 for _ in range(20)]
inputs3[1] = 1
inputs4 = [0 for _ in range(20)]
inputs4[0] = 1
inputs5 = [0 for _ in range(20)]
inputs5[18] = 1
inputs = torch.FloatTensor([inputs1, inputs2, inputs3, inputs4, inputs5])

kernel_size1 = 6
kernel_size2 = 4
kernel_size3 = 5
kernel_size4 = 8
kernel_size5 = 4
largest_kernel = max(kernel_size1, kernel_size2, kernel_size3, kernel_size4, kernel_size5)
kernel1 = torch.cat([torch.ones(kernel_size1), torch.zeros(kernel_size1 - 1 + max(largest_kernel + (largest_kernel - 1) - (kernel_size1 + (kernel_size1 - 1)), 0))])
kernel2 = torch.cat([torch.ones(kernel_size2), torch.zeros(kernel_size2 - 1 + max(largest_kernel + (largest_kernel - 1) - (kernel_size2 + (kernel_size2 - 1)), 0))])
kernel3 = torch.cat([torch.ones(kernel_size3), torch.zeros(kernel_size3 - 1 + max(largest_kernel + (largest_kernel - 1) - (kernel_size3 + (kernel_size3 - 1)), 0))])
kernel4 = torch.cat([torch.ones(kernel_size4), torch.zeros(kernel_size4 - 1 + max(largest_kernel + (largest_kernel - 1) - (kernel_size4 + (kernel_size4 - 1)), 0))])
kernel5 = torch.cat([torch.ones(kernel_size5), torch.zeros(kernel_size5 - 1 + max(largest_kernel + (largest_kernel - 1) - (kernel_size5 + (kernel_size5 - 1)), 0))])
kernels = torch.cat([kernel1.unsqueeze(0), kernel2.unsqueeze(0), kernel3.unsqueeze(0), kernel4.unsqueeze(0), kernel5.unsqueeze(0)], axis=0)

inputs = inputs.unsqueeze(0) # Inverse channel and batch
kernels = kernels.unsqueeze(1)

print('Predicted')
conv_res = F.conv1d(inputs, kernels, padding=largest_kernel - 1, groups=inputs.size(1)).long()
for x in conv_res.squeeze().tolist():
    print('({})'.format(len(x)), x)
print('Expected')
print('(20) [0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0]')
print('(20) [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0]')
print('(20) [0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]')
print('(20) [1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]')
print('(20) [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1]')

Produces
Predicted
(20) [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0]
(20) [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1]
(20) [0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
(20) [1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
(20) [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Expected
(20) [0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0]
(20) [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0]
(20) [0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
(20) [1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
(20) [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1]

We can see the padding is wrong because of the largest kernel. Any idea how to do the padding properly?


title: cuda device side assert triggered pytorch

body:
i want to train 5 network at the same time
model = VAE_NET(device=device).to(device)
netG = UNet(in_channel = 3, out_channel = 3, n_kernel = 64).to(device)
p_d = Discriminator_P(in_channel = 3, n_ker = 32, device=device).to(device)
g_d = Discriminator_G(in_channel = 3, n_ker = 32, device=device).to(device)
triplet = TripletNet(device=device).to(device)

rec_m, (rec_f, mean_f, logvar_f, latent_f), (rec_l, mean_l, logvar_l, latent_l) = model(origin)

first, global discriminator gD
            optim_g_D.zero_grad()

            g_pred_real = g_d(origin)
            g_pred_recon = g_d(recon.detach())
            g_pred_rand_recon = g_d(rand_recon.detach())

            loss_G_d_real = CELoss(g_pred_real, torch.ones_like(g_pred_real))
            loss_G_d_recon = CELoss(g_pred_recon, torch.zeros_like(g_pred_recon))
            loss_G_d_rand = CELoss(g_pred_rand_recon, torch.zeros_like(g_pred_rand_recon))
            loss_G_d = (loss_G_d_rand + loss_G_d_recon + loss_G_d_real) * (1/3)

            loss_G_d.backward()
            optim_g_D.step()

second, patch discriminator pD
            optim_p_D.zero_grad()

            p_pred_real = p_d(origin)
            p_pred_recon = p_d(recon.detach())
            p_pred_rand_recon = p_d(rand_recon.detach())

            loss_P_d_real = CELoss(p_pred_real, torch.ones_like(p_pred_real))
            loss_P_d_recon = CELoss(p_pred_recon, torch.zeros_like(p_pred_recon))
            loss_P_d_rand = CELoss(p_pred_rand_recon, torch.zeros_like(p_pred_rand_recon))

            loss_P_d = loss_P_d_real + loss_P_d_recon + loss_P_d_rand
            loss_P_d = loss_P_d * (1/3)

            loss_P_d.backward()
            optim_p_D.step()

third, Generative model G and Triplet Network for triplet loss
            optim_G.zero_grad()

            l1 = triplet(origin[:10], origin[10:20], origin[20:])
            l2 = triplet(recon_s1t, origin[:10], origin[20:])
            l3 = triplet(recon_s2t, origin[10:20], origin[20:])

            loss_iden_12t = IdentLoss(l1[0], l1[1], l1[2])
            loss_iden_rcs1s1t = IdentLoss(l2[0], l2[1], l2[2])
            loss_iden_rcs2s2t = IdentLoss(l3[0], l3[1], l3[2])

            loss_identy = loss_iden_12t + loss_iden_rcs1s1t + loss_iden_rcs2s2t

            p_pred_recon = p_d(recon)

            g_pred_recon = g_d(recon)

            loss_G_g = CELoss(p_pred_recon, torch.ones_like(p_pred_recon))
            loss_G_p = CELoss(g_pred_recon, torch.ones_like(g_pred_recon))

            loss_G = 20 * loss_G_g + 30 * loss_G_p  + 100 * loss_identy

            loss_G.backward()
            optim_G.step()

last, VAE Network
            optim_vae.zero_grad()
            !nvidia-smi
            lm_loss = CELoss(rec_l, lm)
            f_loss = L1Loss(rec_f, f)
            m_loss = CELoss(rec_m, m)

            lm_kld_loss = classifier(latent_l)
            f_kld_loss = classifier(latent_f)

            loss_vae = 4000*(f_loss + m_loss) +  2000 * lm_loss + 30 * (lm_kld_loss + f_kld_loss)

            loss_vae.backward()
            optim_vae.step()

but when i training altogether, cuda device-side assert trigger occur randomly in training VAE Network
so i first train vae network and train others with eval vae network mode, there's no error
is any way to train network altogether or missing part in code??


title: Why is Loss of SGD for a dataset is not matching the pytorch code with the scratch python code for linear regression?

body:
I'm trying to implement Multiple Linear regression on the wine dataset. But when I compare the results of Pytorch with scratch code of Python the losses are not coming same.
My Scratch Code:
Functions:
def yinfer(X, beta):
  return beta[0] + np.dot(X,beta[1:])

def cost(X, Y, beta):
  sum = 0
  m = len(Y)
  for i in range(m):
    sum = sum + ( yinfer(X[i],beta) - Y[i])*(yinfer(X[i],beta) - Y[i])
  return  sum/(1.0*m)

Main Code:
alpha = 0.005
b=[0,0.04086357 ,-0.02831656  ,0.09622949 ,-0.15162516  ,0.60188454  ,0.47528714,
  -0.6066466  ,-0.22995654 ,-0.58388734  ,0.20954669 ,-0.67851365]
beta = np.array(b)
print(beta)
iterations = 1000
arr_cost = np.zeros((iterations,2))
m = len(Y)
temp_beta = np.zeros(12)
for i in range(iterations):
  for k in range(m):
        temp_beta[0] =  yinfer(X[k,:], beta) - Y[k]
        temp_beta[1:] = (yinfer(X[k,:], beta) - Y[k])*X[k,:]
        beta = beta - alpha*temp_beta/(1.0*m)    #(m*np.linalg.norm(temp_beta))
  arr_cost[i] = [i,cost(X,Y,beta)]
  #print(cost(X,Y,beta))
plt.scatter(arr_cost[0:iterations,0], arr_cost[0:iterations,1])

I have used same weights that were used in Pytorch code
My Pytorch code:
class LinearRegression(nn.Module):
  def __init__(self,n_input_features):
    super(LinearRegression,self).__init__()
    self.linear=nn.Linear(n_input_features,1)
    # self.linear.weight.data=b.view(1,-1)
    self.linear.bias.data.fill_(0.0)
    nn.init.xavier_uniform_(self.linear.weight)
    # nn.init.xavier_normal_(self.linear.bias)
  def forward(self,x):
    y_predicted=self.linear(x)
    return y_predicted
model=LinearRegression(11)

criterion = nn.MSELoss()
num_epochs=1000
for epoch in range(num_epochs):
  for x,y in train_data:
    y_pred=model(x)
    loss=criterion(y,y_pred)
    # print(loss)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()

My DataLoader:
class Data(Dataset):
    def __init__(self):
        self.x=x_train
        self.y=y_train
        self.len=self.x.shape[0]
    def __getitem__(self,index):
      return self.x[index],self.y[index]
    def __len__(self):
        return self.len
dataset=Data()
train_data=DataLoader(dataset=dataset,batch_size=1,shuffle=False)

Can someone please tell me why is that happening or is there any faults in my code?


title: Defining python class method using arguments from __init__

body:
What I want to achieve
I want to define a class with two modes A and B, so that the forward method of the class changes accordingly.
class MyClass():
    def __init__(self, constant):
        self.constant=constant

    def forward(self, x1,x2,function):
        if function=='A':
            return x1+self.constant

        elif function=='B':
            return x1*x2+self.constant

        else:
            print('please provide the correct function')
model1 = MyClass(2)
model1.forward(2, None, 'A')
output>>>4
model2 = MyClass(2)
model2.forward(2, 2, 'B')
output>>>6

It works, but it is not optimal, since every time when calling the forward method, it will check which function to use. However, the forward function is already set and will never be changed once the class is define, therefore, checking which function to use inside forward is super redundant in my case. (For those who notice this, I am writing my neural network model using PyTorch, two models share 90% of the network architecture, the only 10% differences is the way they do feedforward).
My desired version
I want to set the forward method when the class is defined, so that I can achieve this
model1 = MyClass(2, 'A')
model1.forward(2)
output>>>4

model2 = MyClass(2, 'B')
model2.forward(2, 2)
output>>>6

So I rewrote my class to be:
class MyClass():
    def __init__(self, constant, function):
        self.constant=constant # There would be a lot of shared parameters for the two methods
        self.function=function # This controls the feedforward method of this class

    if self.function=='A':
        def forward(self, x1):
            return x1+self.constant

    elif self.function=='B':
        def forward(self, x1, x2):
            return x1*x2+self.constant

    else:
        print('please provide the correct function')

However, it gives me the following error.

NameError: name 'self' is not defined

How do I write the class for that it defines different forward method based on the args from __init__?


title: Loading manually annotated data to train RNN POS tagger

body:
I've got a large manually annotated data. I would like to train a part of speech tagger using RNN. The data is something similar to the text below :
Lorem <NP> Ipsum <NP> dummy <N> text <ADV> printing <VREL> typesetting <NUMCR> Ipsum <VREL> Ipsum <NP> Ipsum <NP> Lorem <N> Ipsum <NP> Ipsum <N> Ipsum <NP> Lorem <ADJ> Lorem <NP> Ipsum <N> Lorem <VN> Lorem <ADJ> Lorem <N> Lorem <N> ፣ <PUNC> Lorem <ADJ> Lorem <ADJ> Ipsum <NC> Ipsum <NC> Ipsum <NP>

Please guide me on how to load this data to train an RNN based tagger.


title: Apache Airflow + GCP ML engine for Pytorch example

body:
Everyone.
I want to use airflow MLEngineTrainingOperator to train model on gcp.
Then, because default runtime version just for tensorflow or XGboost, habe not pytorch.
But i want to train with pytorch framework now. But i not sure how to implement and haven't seen other examples.
If you know how to implement, please let me know how to do.
I will thanks a lot.


title: How to correctly define a Pytorch dataset and dataloader for data which is scattered accross disk, AND in each file?

body:
I have data scattered across several files, say f1, f2, f3 whose paths are p1, p2, p3.
Each file has say 1000 samples which can be loaded a pandas dataframe with to_pd(path).
Files must be loaded in order due to an existing file-fetching system I already have in place.
Data within each file can be sampled randomly.

I've seen solutions for loading from within a single file
and for reading a list of files,
but not for doing both.

I can't think of a way to just define a custom pytorch Dataset because I don't have all paths in advance, but they come at runtime, only loading one after I'm finished with the last one.
I could make a dataset per file, but then I would have to create a dataloader that knows how to handle those multiple datasets.

What's the correct way of doing this in Pytorch?
I am also using Pytorch lightning if it helps.


title: Segmentation to one-hot encoding

body:
I have a batch of segmented images of size
seg --> [batch, channels, imsize, imgsize] --> [16, 6, 50, 50]
each scalar in this tensor specifies one of the segmentation classes.
We have 2000 total segmentation classes.
Now the goal is to convert
[16, 6, 50, 50] --> [16, 2000, 50, 50]
where each class is encoded in one hot fashion.
How do I do it with pytorch api?
I can only think of ridiculously inefficient looping construction.
Example
Here we will have only 2 initial channels (instead of 6), 4 labels (instead of 2000), size batch 1 (instead of 16) and 4x4 image instead of 50x50.
0, 0, 1, 1
0, 0, 0, 1
1, 1, 1, 1
1, 1, 1, 1

3, 3, 2, 2
3, 3, 2, 2
3, 3, 2, 2
3, 3, 2, 2

Now this turns into 4 channel output
1, 1, 0, 0
1, 1, 1, 0
0, 0, 0, 0
0, 0, 0, 0

0, 0, 1, 1
0, 0, 0, 1
1, 1, 1, 1
1, 1, 1, 1

1, 1, 0, 0
1, 1, 0, 0
1, 1, 0, 0
1, 1, 0, 0

0, 0, 1, 1
0, 0, 1, 1
0, 0, 1, 1
0, 0, 1, 1

The key observation is that a particular label appears only on a single input channel.


title: Why is the code not able to approximate the square function?

body:
WHy does the following code not work as a square approximator? I am getting weird dimensions. When I tried plotting loss, the graph somehow does not show anything. I am a beginner with pytorch, so I would be grateful for any help.
import torch
from torch import nn
import matplotlib.pyplot as plt
import numpy as np

data = [[i] for i in range(-10000, 10000)]
y = [[i[0] * i[0]] for i in data]
data=torch.FloatTensor(data)
y=torch.FloatTensor(y)

class MyModel(nn.Module):
    def __init__(self, numfeatures, outfeatures):
        super().__init__()
        self.modele = nn.Sequential(
                                    nn.Linear( numfeatures,  2*numfeatures),
                                    nn.ReLU(),
                                    nn.Linear(2 * numfeatures, 4 * numfeatures),
                                    nn.ReLU(),
                                    nn.Linear(4* numfeatures, 2 * numfeatures),
                                    nn.ReLU(),
                                    nn.Linear(2*numfeatures, numfeatures),
                                    )
    def forward(self, x):
        return self.modele(x)

model = MyModel(1, 1)
criterion = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)

n_epochs = 10000
epoch_loss= []

for i in range(n_epochs):
    y_pred = model(data)
    loss = criterion(y_pred, y)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    epoch_loss.append(loss.item())

plt.plot(epoch_loss)


title: running_mean should contain %% elements not ** when use BatchNorm1d

body:
code:
##% first ,I set a model:

class net(nn.Module):
    def __init__(self, max_len, feature_linear, rnn, input_size, hidden_size, output_dim, num__rnn_layers, bidirectional, batch_first=True, p=0.2):
        super(net, self).__init__()
        self.max_len = max_len
        self.feature_linear = feature_linear
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.bidirectional = bidirectional
        self.num_directions = 2 if bidirectional == True else 1
        self.p = p
        self.batch_first = batch_first
        self.linear1 = nn.Linear(max_len, feature_linear)
        init.kaiming_normal_(self.linear1.weight, mode='fan_in')
        self.BN1 = BN(feature_linear)

        self.linear2 = nn.Linear(feature_linear, feature_linear)
        init.kaiming_normal_(self.linear2.weight, mode='fan_in')
        self.BN2 = BN(feature_linear)

        self.linear5 = nn.Linear(feature_linear, feature_linear)
        init.kaiming_normal_(self.linear5.weight, mode='fan_in')
        self.BN5 = BN(feature_linear)

        self.rnn = rnn(input_size, hidden_size, num_rnn_layers, bidirectional=bidirectional, batch_first=batch_first)
        self.hidden_to_out1 = nn.Linear(hidden_size * self.num_directions, hidden_size, bias=False)
        init.kaiming_normal_(self.hidden_to_out1.weight, mode='fan_in')
        self.BN6 = BN(hidden_size)

    def forward(self, xb, seq_len_crt):
        rnn_input = torch.zeros(xb.shape[0], self.feature_linear, self.input_size)
        for i in range(self.input_size):
            out = self.linear1(xb[:, :, i]) # xb[:,:,i].shape:(1,34), out.shape(1,100)
            out = F.relu(out) # 输入：out.shape(1,100), 输出：out.shape(1,100)
            self.BN1.eval()
            out = self.BN1(out) # 输入：out.shape(1,100)，输出：out.shape(1,100)

            out = self.linear2(out)
            out = F.relu(out)
            self.BN2.eval()
            out = self.BN2(out)

            out = self.linear5(out)
#             out = self.dropout2(out)
            out = F.relu(out)
            self.BN5.eval()
            out = self.BN5(out)

            rnn_input[:, :, i] = out

        packed = nn.utils.rnn.pack_padded_sequence(rnn_input, seq_len_crt, batch_first=self.batch_first, enforce_sorted=False)
        out, states = self.rnn(packed)
        padded_out, input_sizes = nn.utils.rnn.pad_packed_sequence(out, batch_first=self.batch_first)
        # assert torch.all(torch.tensor(seq_len_crt).eq(input_sizes))

        y_hat = self.hidden_to_out1(padded_out)
        y_hat = F.relu(y_hat)
        self.BN6.eval()
        y_hat = self.BN6(y_hat)

  return y_hat.squeeze(-1)

##% second,I make a model as a funtion and optimize it

input_size = 5
hidden_size = 32
output_dim = 1
num_rnn_layers = 2
bidirectional = True
rnn = nn.LSTM
batch_size = batch_size
feature_linear = 60
BN = nn.BatchNorm1d

model = net(max_len, feature_linear, rnn, input_size, hidden_size, output_dim, num_rnn_layers, bidirectional, p=0.1)
model.train()
loss_func = nn.MSELoss(reduction='none')
# optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.9)
# optimizer = optim.Adam(model.parameters(), lr=0.01)
optimizer = optim.AdamW(model.parameters(), lr=0.001, weight_decay=0.05)

then I fit the model
fit(500, model, loss_func, optimizer, train_dl, valid_dl, valid_ds, seq_len_train_iter, seq_len_valid_iter)

but I met an error , the key error message is :
running_mean should contain 25 elements not 32

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-221-fb062ad3f20e> in <module>
----> 1 fit(500, model, loss_func, optimizer, train_dl, valid_dl, valid_ds, seq_len_train_iter, seq_len_valid_iter)

<ipython-input-216-330ddbc44040> in forward(self, xb, seq_len_crt)
     88         y_hat = F.relu(y_hat)
     89         self.BN6.eval()
---> 90         y_hat = self.BN6(y_hat)
     91
     92         y_hat = self.hidden_to_out2(y_hat)

RuntimeError: running_mean should contain 25 elements not 32

so I  checked :
self.hidden_to_out1 = nn.Linear(hidden_size * self.num_directions, hidden_size, bias=False)
init.kaiming_normal_(self.hidden_to_out1.weight, mode='fan_in')
self.BN6 = BN(hidden_size)

hidden_size = 32, So ,I don't know why and how to modify it


title: AllenNLP 2.0: Can't get FBetaMultiLabelMeasure to run

body:
I would like to compute the f1-score for a classifier trained with allen-nlp. I used the working code from a allen-nlp guide, which computed accuracy, not F1, so I tried to adjust the metric in the code.
According to the documentation, CategoricalAccuracy and FBetaMultiLabelMeasure take the same inputs. (predictions: torch.Tensor of shape [batch_size, ..., num_classes], gold_labels: torch.Tensor of shape [batch_size, ...])
But for some reason the input that worked perfectly well for the accuracy results in a RuntimeError when given to the f1-multi-label metric.
I condensed the problem to the following code snippet:
>>> from allennlp.training.metrics import CategoricalAccuracy, FBetaMultiLabelMeasure
>>> import torch
>>> labels = torch.LongTensor([0, 0, 2, 1, 0])
>>> logits = torch.FloatTensor([[ 0.0063, -0.0118,  0.1857], [ 0.0013, -0.0217,  0.0356], [-0.0028, -0.0512,  0.0253], [-0.0460, -0.0347,  0.0400], [-0.0418,  0.0254,  0.1001]])
>>> labels.shape
torch.Size([5])
>>> logits.shape
torch.Size([5, 3])
>>> ca = CategoricalAccuracy()
>>> f1 = FBetaMultiLabelMeasure()
>>> ca(logits, labels)
>>> f1(logits, labels)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".../lib/python3.8/site-packages/allennlp/training/metrics/fbeta_multi_label_measure.py", line 130, in __call__
true_positives = (gold_labels * threshold_predictions).bool() & mask & pred_mask
RuntimeError: The size of tensor a (5) must match the size of tensor b (3) at non-singleton dimension 1

Why is this error happening? What am I missing here?


title: Error training PyTorch model IndexError: index out of range in self

body:
I am new to PyTorch and trying to implement a recommender system.
I am working on my first model (matrix factorization) which i found here:
https://blog.fastforwardlabs.com/2018/04/10/pytorch-for-recommenders-101.html
I have a list of user ids and item ids (refer to movie ids) and the ratings which i pass to the
model as described on the website. I read the files with the pandas function.
training.py:

ratings = pd.read_csv('../data/ratings.csv')
movies = pd.read_csv('../data/movies.csv')

n_users = int(ratings.userId.nunique())
n_items = int(ratings.movieId.nunique())

users = pd.Series.tolist(ratings.userId)
items = pd.Series.tolist(ratings.movieId)
rating_values = pd.Series.tolist(ratings.rating)

model = MatrixFactorization(n_users, n_items, n_factors=20)
loss_fn = torch.nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(),
                            lr=1e-6)

index = 0
for user, item in zip(users, items):
    # get user, item and rating data
rating = Variable(torch.FloatTensor([rating_values[index]]))
user = Variable(torch.LongTensor([int(user)]))
item = Variable(torch.LongTensor([int(item)]))

index += 1
# predict rating
prediction = model(user, item)
loss = loss_fn(prediction, rating)
print(loss)

optimizer.zero_grad()

# backpropagate
loss.backward()

# update weights
optimizer.step()

models.py:
class MatrixFactorization(nn.Module):

    def __init__(self, n_users, n_items, n_factors=20):
        super().__init__()
        # create user embeddings
        self.user_factors = nn.Embedding(n_users, n_factors,
                                         sparse=True)
        # create item embeddings
        self.item_factors = nn.Embedding(n_items, n_factors,
                                         sparse=True)

    def forward(self, user, item):
        # matrix multiplication
        return (self.user_factors(user) * self.item_factors(item)).sum(1)

    def predict(self, user, item):
        return self.forward(user, item)

At the end i get the following error:
Traceback (most recent call last):
  File "src\training.py", line 31, in <module>
    prediction = model(user, item)
  File "AppData\Local\Programs\Python\Python39\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "src\models.py", line 20, in forward
    return (self.user_factors(user) * self.item_factors(item)).sum(1)
  File "Python\Python39\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "Python\Python39\lib\site-packages\torch\nn\modules\sparse.py", line 124, in forward
    return F.embedding(
  File "Python\Python39\lib\site-packages\torch\nn\functional.py", line 1852, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
IndexError: index out of range in self

I don't understand what went wrong. Is something wrong with the model or with my inputs?


title: Why do I get a ValueError when using CrossEntropyLoss

body:
I am trying to train a pretty simple 2-layer neural network for a multi-class classification class. I am using CrossEntropyLoss and I get the following error: ValueError: Expected target size (128, 4), got torch.Size([128]) in my training loop at the point where I am trying to calculate the loss.
My last layer is a softmax so it outputs the probabilities of each of the 4 classes. My target values are a vector of dimension 128 (just the class values). Am I initializing the CrossEntropyLoss object incorrectly?
I looked up existing posts, this one seemed the most relevant:
https://discuss.pytorch.org/t/valueerror-expected-target-size-128-10000-got-torch-size-128-1/29424 However, if I had to squeeze my target values, how would that work? Like right now they are just class values for e.g., [0 3 1 0]. Is that not how they are supposed to look? I would think that the loss function maps the highest probability from the last layer and associates that to the appropriate class index.
Details:

This is using PyTorch
Python version is 3.7
NN architecture is: embedding -> pool -> h1 -> relu -> h2 -> softmax
Model Def (EDITED):

self.embedding_layer = create_embedding_layer(embeddings)
self.pool = nn.MaxPool1d(1)
self.h1 = nn.Linear(embedding_dim, embedding_dim)
self.h2 = nn.Linear(embedding_dim, 4)
self.s = nn.Softmax(dim=2)

forward pass:
    x = self.embedding_layer(x)
    x = self.pool(x)
    x = self.h1(x)
    x = F.relu(x)
    x = self.h2(x)
    x = self.s(x)
    return x


title: How to deal with None in a PyTorch custom dataset class?

body:
I have an object detection dataset with RGB images and annotations in Json. I use a custom DataLoader class to read the images and the labels. One issue that I’m facing is that I would like to skip images when training my model if/when labels don’t contain certain objects.
For example, If one image doesn’t contain any target labels belonging to the class ‘Cars’, I would like to skip them. When parsing my Json annotation, I tried checking for labels that don’t contain the class ‘Cars’ and returned None. Subsequently, I used a collate function to filter the None but unfortunately, It is not working.
import torch
from torch.utils.data.dataset import Dataset
import json
import os
from PIL import Image
from torchvision import transforms
#import cv2
import numpy as np
general_classes = {
    # Cars
    "Toyota Corolla" : 0,
    "VW Golf" : 0,
    "VW Beetle" : 0,

    # Motor-cycles
    "Harley Davidson" : 1,
    "Yamaha YZF-R6" : 1,
}

car_classes={
"Toyota Corolla" : 0,
"VW Golf" : 0,
"VW Beetle" : 0
}

def get_transform(train):
    transforms = []
    # converts the image, a PIL image, into a PyTorch Tensor
    transforms.append(T.ToTensor())
    if train:
        # during training, randomly flip the training images
        # and ground-truth for data augmentation
        transforms.append(T.RandomHorizontalFlip(0.5))
    return T.Compose(transforms)

def my_collate(batch):
    batch = list(filter(lambda x: x is not None, batch))
    return torch.utils.data.dataloader.default_collate(batch)

class FilteredDataset(Dataset):
    # The dataloader will skip the image and corresponding labels based on the dictionary 'car_classes'
    def __init__(self, data_dir, transforms):
        self.data_dir = data_dir
        img_folder_list = os.listdir(self.data_dir)
        self.transforms = transforms

        imgs_list = []
        json_list = []
        self.filter_count=0
        self.filtered_label_list=[]

        for img_path in img_folder_list:
            #img_full_path = self.data_dir + img_path
            img_full_path=os.path.join(self.data_dir,img_path)
            json_file = os.path.join(img_full_path, 'annotations-of-my-images.json')
            img_file = os.path.join(img_full_path, 'Image-Name.png')

            json_list.append(json_file)
            imgs_list.append(img_file)
        self.imgs = imgs_list
        self.annotations = json_list
        total_count=0

        for one_annotation in self.annotations:
            filtered_obj_id=[]
            with open(one_annotation) as f:
                img_annotations = json.load(f)

            parts_list = img_annotations['regions']
            for part in parts_list:
                current_obj_id = part['tags'][0] # bbox label
                check_obj_id = general_classes[current_obj_id]
                if(check_obj_id==0):
                    subclass_id=car_classes[current_obj_id]
                    filtered_obj_id.append(subclass_id)
                    total_count=total_count+1

            if(len(filtered_obj_id)>0):
                self.filter_count=self.filter_count+1
                self.filtered_label_list.append(one_annotation)

        print("The total number of the objects in all images: ",total_count)

    # get one image and the bboxes,img_id, labels of parts, etc in the image as target.
    def __getitem__(self, idx):

        img_path = self.imgs[idx]
        image_id = torch.tensor([idx])

        with open(self.annotations[idx]) as f:
            img_annotations = json.load(f)
        parts_list = img_annotations['regions']
        obj_ids = []
        boxes = []
        for part in parts_list:
            obj_id = part['tags'][0]
            check_obj_id = general_classes[obj_id]
            if(check_obj_id==0):
               obj_id=car_classes[obj_id]
               obj_ids.append(obj_id)
                #print("---------------------------------------------------")

        if(len(obj_ids)>0):
            img = Image.open(img_path).convert("RGB")
            labels = torch.as_tensor(obj_ids, dtype = torch.int64)
            target = {}
            target['labels'] = labels

            if self.transforms is not None:
                img, target = self.transforms(img, target)
                return img, target
        else:
            return None

    def __len__(self):
        return len(self.filtered_label_list)

train_data_path = "path-to-my-annotation"
# Generators
train_dataset = FilteredDataset(train_data_path,get_transform(train=True))
print("Total files in the train_dataset: ",len(train_dataset))
#print("The first instance in the train dataset : ",train_dataset[0])
#training_generator = torch.utils.data.DataLoader(train_dataset)
training_generator = torch.utils.data.DataLoader(train_dataset,collate_fn=my_collate)
print("\n\n Iterator in action! ")
print("---------------------------------------------------------")
count=0
for img,target in training_generator:
    #print("The img name : ",img[0])
    count=count+1
    print("target name : ",target)
    print("count : ",count)
    print("**************************************************")

However, I get the following error,

Could anyone please suggest a way to skip the images that do not contain a particular categorical label?


title: How does one setup the set_sharing_strategy strategy for multiprocessing in Pytorch?

body:
where does one run the following:
import torch.multiprocessing
torch.multiprocessing.set_sharing_strategy('file_system')

suppose to be ran by every process or only the main one?
Traceback (most recent call last):
  File "/var/spool/pbs/mom_priv/jobs/486213.iam-pbs.SC", line 321, in <module>
    main_distributed()
  File "/var/spool/pbs/mom_priv/jobs/486213.iam-pbs.SC", line 239, in main_distributed
    mp.spawn(fn=parallel_train, args=(opts,), nprocs=opts.world_size)
  File "/homes/miranda9/.conda/envs/automl-meta-learning/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 199, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
  File "/homes/miranda9/.conda/envs/automl-meta-learning/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 148, in start_processes
    process.start()
  File "/homes/miranda9/.conda/envs/automl-meta-learning/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/homes/miranda9/.conda/envs/automl-meta-learning/lib/python3.8/multiprocessing/context.py", line 283, in _Popen
    return Popen(process_obj)
  File "/homes/miranda9/.conda/envs/automl-meta-learning/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/homes/miranda9/.conda/envs/automl-meta-learning/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/homes/miranda9/.conda/envs/automl-meta-learning/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 58, in _launch
    self.pid = util.spawnv_passfds(spawn.get_executable(),
  File "/homes/miranda9/.conda/envs/automl-meta-learning/lib/python3.8/multiprocessing/util.py", line 427, in spawnv_passfds
    errpipe_read, errpipe_write = os.pipe()
OSError: [Errno 24] Too many open files

ref:

https://github.com/pytorch/pytorch/issues/11201
https://pytorch.org/docs/stable/multiprocessing.html#file-descriptor-file-descriptor

related: https://discuss.pytorch.org/t/how-does-one-setp-up-the-set-sharing-strategy-strategy-for-multiprocessing/113302


title: Is there a better way to multiply & sum two Pytorch tensors along the first dimension?

body:
I have two Pytorch tensors, a & b, of shape (S, M) and (S, M, H) respectively. M is my batch dimension. I want to multiply & sum the two tensors such that the output is of shape (M, H). That is, I want to compute the sum over s of a[s] * b[s].
For example, for S=2, M=2, H=3:
>>> import torch
>>> S, M, H = 2, 2, 3
>>> a = torch.arange(S*M).view((S,M))
tensor([[0, 1],
        [2, 3]])
>>> b = torch.arange(S*M*H).view((S,M,H))
tensor([[[ 0,  1,  2],
         [ 3,  4,  5]],

        [[ 6,  7,  8],
         [ 9, 10, 11]]])

'''
DESIRED OUTPUT:
= [[0*[0, 1, 2] + 2*[6, 7, 8]],
   [1*[3, 4, 5] + 3*[9, 10, 11]]]

= [[12, 14, 16],
   [30, 34, 38]]

note: shape is (2, 3) = (M, H)
'''

I've found one way that sort of works, using torch.tensordot:
>>> output = torch.tensordot(a, b, ([0], [0]))
tensor([[[12, 14, 16],
         [18, 20, 22]],

        [[18, 22, 26],
         [30, 34, 38]]])
>>> output.shape
torch.Size([2, 2, 3]) # always (M, M, H)
>>> output = output[torch.arange(M), torch.arange(M), :]
tensor([[12, 14, 16],
        [30, 34, 38]])

But as you can see, it makes a lot of unnecessary computations and I have to slice the ones that are relevant for me.
Is there a better way to do this that doesn't involve the unnecessary computations?


how to import a torch 1.7.1 when torch 1.4.0 is also installed
When I run the command: ! pip list
It lists all libraries with : torch    1.7.1
Now when I run:
>>>import torch
>>>torch.__version__
'1.4.0'

How Do I import torch==1.7.1 in the python program?
I am using python 3.8.3 and windows 10


title: Manually assign weights using PyTorch

body:
I am using Python 3.8 and PyTorch 1.7 to manually assign and change the weights and biases for a neural network. As an example, I have defined a LeNet-300-100 fully-connected neural network to train on MNIST dataset. The code for class definition is:
class LeNet300(nn.Module):
    def __init__(self):
        super(LeNet300, self).__init__()

        # Define layers-
        self.fc1 = nn.Linear(in_features = input_size, out_features = 300)
        self.fc2 = nn.Linear(in_features = 300, out_features = 100)
        self.output = nn.Linear(in_features = 100, out_features = 10)

        self.weights_initialization()

    def forward(self, x):
        out = F.relu(self.fc1(x))
        out = F.relu(self.fc2(out))
        return self.output(out)

    def weights_initialization(self):
        '''
        When we define all the modules such as the layers in '__init__()'
        method above, these are all stored in 'self.modules()'.
        We go through each module one by one. This is the entire network,
        basically.
        '''
        for m in self.modules():
            if isinstance(m, nn.Linear):
                nn.init.xavier_normal_(m.weight)
                nn.init.constant_(m.bias, 0)

To experiment with trying to change the weights for this model-
# Instantiate model-
mask_model = LeNet300()

To assign all of the weights in each of the layers to one (1), I use the code-
with torch.no_grad():
    for layer in mask_model.state_dict():
        mask_model.state_dict()[layer] = nn.parameter.Parameter(torch.ones_like(mask_model.state_dict()[layer]))

# Sanity check-
mask_model.state_dict()['fc1.weight']

This output shows that the weights are not equal to 1.
I also tried the code-
for param in mask_model.parameters():
    # print(param.shape)
    param = nn.parameter.Parameter(torch.ones_like(param))

But this does not work as well.
Help?


title: Pytorch RNN data from three different csv. Python 3

body:
i need to train a RNN on pythorch on features vectors taken from 3 different frames of the same object. The features dimensions are alway the same.

I have a csv for first frames, csv for seconds frames, csv for third one.

Need to take the same raw from different csvs in order to get the features about the specific frame.
(raw 1 in csv1 is frame 1 of object x, raw 1 in csv2 is frame 2 of object x, raw 1 in csv3 is frame 3 of object x)

I need to create a RNN that at state t0, t1, t2 take frame 1 features, frame 2 features and frame 3.

I saw character examples and time prediction ones where for each state are fed a single character of a world or a single time value of the dataset.

How can I organize the training in order to take a whole features vector to train that state, then the second feature and so on?

thanks in advance


title: How can I fully utilise the GPU when training my model in FloyHub?

body:
I am training the below model on FloydHub using Jupyter Notebook. But, whenever I train the model it takes a lot of time(1 minute). The stats below the notebook displays that only 2% of the GPU is utilised.
I have tried running the command torch.cuda.is_available() and it returns True.

import torch
from torch import nn,optim
import torch.nn.functional as F
from torchvision import datasets,transforms

transform = transforms.Compose([transforms.ToTensor(),
                               transforms.Normalize((0.5,0.5,0.5),(0.5,0.5,0.5)),
                               ])
trainset = datasets.MNIST('~/.pytorch/MNIST_data/',download=True,train=True,transform=transform)
trainloader = torch.utils.data.DataLoader(trainset,batch_size=64,shuffle=True)

testset = datasets.MNIST('~/.pytorch/MNIST_data/',download=True,train=False,transform=transform)
testloader = torch.utils.data.DataLoader(testset,batch_size=64,shuffle=True)

class Classifier(nn.Module):

    def __init__(self):
        super().__init__()

        self.hidden = nn.Linear(784,256).cuda()
        self.output = nn.Linear(256,10).cuda()
        self.dropout = nn.Dropout(p=0.2).cuda()

    def forward(self,x):
        x = x.view(x.shape[0],-1).cuda()
        x = self.hidden(x).cuda()
        x = torch.sigmoid(x).cuda()
        x = self.dropout(x).cuda()
        x = self.output(x).cuda()
        x = F.log_softmax(x,dim=1).cuda()
        return x.cuda()

model = Classifier()
model.cuda()
criterion = nn.NLLLoss().cuda()
optimizer = optim.SGD(model.parameters(),lr=0.5)

epochs = 30
training_losses = []
test_losses = []

for e in range(epochs):
    train_loss = 0
    test_loss = 0
    accuracy = 0
    for images,labels in trainloader:
        optimizer.zero_grad()
        output = model(images)
        labels = labels.cuda()
        loss = criterion(output,labels)
        loss.backward()
        optimizer.step()
        train_loss+=loss.item()

    with torch.no_grad():
        # set the model to testing mode
        model.eval()
        for images,labels in testloader:
            output = model(images)
            labels = labels.cuda()
            test_loss+=criterion(output,labels)
            ps = torch.exp(output)
            # get the class with the highest probability
            _,top_class = ps.topk(1,dim=1)
            equals = top_class == labels.view(*top_class.shape)
            accuracy+=torch.mean(equals.type(torch.FloatTensor))

    model.train()

    training_losses.append(train_loss/len(trainloader))
    test_losses.append(test_loss/len(testloader))

    if((e+1)%5 == 0):
        print(f"Epoch:{e+1}\n",
                f"Training Loss:{train_loss/len(trainloader)}\n",
                f"Test Loss:{test_loss/len(testloader)}\n",
                f"Test Accuracy:{(accuracy/len(testloader)*100)}\n\n")


title: where do I add -DCUDA_HOST_COMPILER=/usr/bin/gcc-5 (Pytorch)

body:
I am installing pytorch from the source, and am running into issues where it is using my current gcc compiler and not gcc-5

I have tried to google this problem for hours, searched stack and everything.

I have tried:

exporting the C and CXX variables to no avail
linking the variable

I am using arch linux


title: How to parallelize a for loop inside some_module.forward(some_input) (on the GPU)?

body:
Let's say I have a model (in pseudo-code, mostly):

class SomeLayer(nn.Module):
    def __init__(self, s):
        #init some layers etc
        self.N = s*s
    def forward(self, input_tensor):
        #intialize some variables
        some_results=[]
        for iter_i in range(self.N):
            # do independent operations on different parts of input_tensor
            # each operation is basically a copy of a subtensor of input_tensor
            # such that its size depends on iter_i
            # append_results to some_results
        return some_results

What is the correct way of parallelizing that sort of for loop? Currently I'm planing to write a small CUDA kernel for that and load that from python, but it feels a bit overkill, and I asssume there should be a simple way to do that although I haven't been able to find it in the documentation.


title: Output hidden state in OpenNMT-py

body: I just have a short question regarding the pytorch version of OpenNMT. There does not seem to be an option to return the hidden state of encoder and decoder in the options. Am I missing a flag or is this not an option in OpenNMT-py?


title: Why does broadcasting give an empty tensor?

body:
x = torch.randn(1, 1, 0)

y = torch.randn(4, 1, 1)

(x+y)

tensor([], size=(4, 1, 0))

(x + y).shape

torch.Size([4, 1, 0])

shouldn’t it have been 4, 1, 1, just y?


title: Constraint on dimensions of activation/feature map in convolutional network

body:
Let's say input to intermediate CNN layer is of size 512×512×128 and that in the convolutional layer we apply 48 7×7 filters at stride 2 with no padding. I want to know what is the size of the resulting activation map?

I checked some previous posts (e.g., here or here) to point to this Stanford course page. And the formula given there is (W − F + 2P)/S + 1 = (512 - 7)/2 + 1, which would imply that this set up is not possible, as the value we get is not an integer.

However if I run the following snippet in Python 2.7, the code seems to suggest that the size of activation map was computed via (512 - 6)/2, which makes sense but does not match the formula above:

>>> import torch
>>> conv = torch.nn.Conv2d(in_channels=128, out_channels=48, kernel_size=7, stride=2, padding=0)
>>> conv
Conv2d(128, 48, kernel_size=(7, 7), stride=(2, 2))
>>> img = torch.rand((1, 128, 512, 512))
>>> out = conv(img)
>>> out.shape
(1, 48, 253, 253)

Any help in understanding this conundrum is appreciated.


title: Compute loss for for multi-class classification when using cross validation

body:
I am training my neural network on the MNIST data set with handwritten digits. I use PyTorch and wondered what's the right way to compute the loss when using cross validation (Kfold) with 5 folds? I want to plot a loss graph which is comparing train and test loss.

I am using cross-entropy loss as the loss function. Should I take the average of the loss for each epoch in each fold? Such that if I am using 5 epochs, the loss graph consists of 5 datapoints, where the first point corresponds to the average loss for the first epoch of all 5 folds?

I also use a batch-size of 128. At first the loss is around 1,61 and at the end it's around around 1,47. The model has a accuracy of 97.7%, is this little decrease in loss strange?


title: Error while downloading pre-trained GAN model in Pytorch : 'memory' file not found

body:
I was following the steps given in https://modelzoo.co/model/pytorch-cyclegan-and-pix2pix to download a pre-trained model.
These where the first 3 commands given there:
git clone https://github.com/pytorch/vision
cd vision
**python setup.py install**

However when I ran the third line, I got an error:
fatal error: 'memory' file not found

#include <memory>

error: command 'gcc' failed with exit status 1

If anyone has some idea on how to overcome this error, it would be really helpful.


title: Pytorch Gridsample - Can't get Toy example to work

body:
I am trying to understand how to use Pytorch's grid sample function. I understand that we pass in a [B * Channel * H * W] and a flowfield [B * H * W * 2] of UV pixels

But it doesn't seem to work.

import numpy as np
import torch
import torch.nn.functional as F
from torch import nn
from torch.autograd import Variable
import time

import cv2

rgbimg = np.ones((100,100,3)).astype(np.float32)*0.5
rgbimg[0:50,0:50,:] = 0

rgbimg_tensor = torch.tensor(rgbimg.transpose(2,0,1)).unsqueeze(0)

flowfield = torch.ones((1,rgbimg.shape[0], rgbimg.shape[1], 2))*0

rgbimg_tensor_trans = F.grid_sample(rgbimg_tensor, flowfield)

rgbimg_trans = rgbimg_tensor_trans[0,:,:,:].numpy().transpose(1,2,0)

print(rgbimg_trans.shape)

cv2.namedWindow("fimage")
cv2.moveWindow("fimage", 2500, 50)
cv2.imshow("fimage", rgbimg_trans)
cv2.waitKey(0)

Here I have set the flowfield to zero, but it just seems to set the final image to the same value everywhere. What am I doing wrong

becomes


title: PyTorch Model throwing error for list of layers

body:
I have designed the following torch model with 2 conv2d layers. It works without any error.

import torch.nn as nn
from torchsummary import summary

class mini_unet(nn.Module):
    def __init__(self):
        super(mini_unet, self).__init__()
        self.c1 = nn.Conv2d(1, 1, 3, padding = 1)
        self.r1 = nn.ReLU()
        self.c2 = nn.Conv2d(1, 1, 3, padding = 1)
        self.r2 = nn.ReLU()

    def forward(self, x):
        x = self.c1(x)
        x = self.r1(x)
        x = self.c2(x)
        x = self.r2(x)
        return x

a = mini_unet().cuda()

print(a)

But, let's say I have too many layers, I don't want to explicitly write each of them in the forward function. So, I used a list to automate it like below.

import torch.nn as nn
from torchsummary import summary

class mini_unet2(nn.Module):
    def __init__(self):
        super(mini_unet2, self).__init__()
        self.layers = nn.ModuleList([nn.Conv2d(1, 1, 3, padding = 1),
        nn.ReLU(),
        nn.Conv2d(1, 1, 3, padding = 1),
        nn.ReLU()])

    def forward(self, x):
        for l in self.layers:
            x = l(x)
        return x

a2 = mini_unet2().cuda()
print(a2)
summary(a2, (1,4,4))

This gives me the following error which is strange, I have used cuda() why it doesn't work?

RuntimeError                              Traceback (most recent call last)
<ipython-input-36-1d71e75b96e0> in <module>
     17 a2 = mini_unet2().cuda()
     18 print(a2)
---> 19 summary(a2, (1,4,4))

~/anaconda3/envs/torch/lib/python3.6/site-packages/torchsummary/torchsummary.py in summary(model, input_size, batch_size, device)
     70     # make a forward pass
     71     # print(x.shape)
---> 72     model(*x)
     73
     74     # remove these hooks

~/anaconda3/envs/torch/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    487             result = self._slow_forward(*input, **kwargs)
    488         else:
--> 489             result = self.forward(*input, **kwargs)
    490         for hook in self._forward_hooks.values():
    491             hook_result = hook(self, input, result)

<ipython-input-36-1d71e75b96e0> in forward(self, x)
     12     def forward(self, x):
     13         for l in self.layers:
---> 14             x = l(x)
     15         return x
     16

~/anaconda3/envs/torch/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    487             result = self._slow_forward(*input, **kwargs)
    488         else:
--> 489             result = self.forward(*input, **kwargs)
    490         for hook in self._forward_hooks.values():
    491             hook_result = hook(self, input, result)

~/anaconda3/envs/torch/lib/python3.6/site-packages/torch/nn/modules/conv.py in forward(self, input)
    318     def forward(self, input):
    319         return F.conv2d(input, self.weight, self.bias, self.stride,
--> 320                         self.padding, self.dilation, self.groups)
    321
    322

RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the same


title: The tensorflow equivalent implementation of pytorch gather()

body:
If I have a tensor S whose size is (2,1,64,512) and an index tensor D whose size is (2,5,64,512). In pytorch, I can extract values from S along axis=1 according to D by: o = torch.gather(S, D, axis=1), which results in a new tensor of size (2,5,64,512).

I am wondering is there a way that can produce the same results using tensorflow even the tensor's size is different.

Thanks,


title: Parallel LSTM Model with shared parameters

body:
I am trying to build parallel LSTM models for each input feature with shared parameters. For instance, suppose I have 10 input features and I want to build a separate LSTM for each feature with shared parameters. Each feature should have their own hidden states, only parameters should be shared. I am looking to build something like this :

Parallel LSTM Model for each input feature

I want to do this using keras/pytorch. Any help would be appreciated.


title: why a simple python MLP model can outperform complex pytorch RNN-LSTM model tremendously

body:
I have compared simple MLP (in fact even just one layer with 10 nodes!) using python (followed Andrew Trask instruction) versus RNN-LSTM model defined in Pytorch for the same movie sentiment binary classification problem. To my surprise the simple model outperforms pytorch LSTM overwhelmingly; it takes less than a minute for the first model to be trained on CPU (20,000 reviews) and generates 85% of test accuracy while for the latter it took me ~ 75 minutes to reach more or less the same accuracy! That makes me curious is there any guideline when to use just simple models over complex ones? I mean, should the demanding models like LSTM just be used for huge datasets? As a complementary point, I know data pre-processing is important but not sure if it makes such a mind-blowing difference!

Here is the github link: https://github.com/mghanava/SentimentAnalysisPractice.git
Please check Sentiment_Classification_Comparison.ipynb or its pdf version.

Thanks


title: accuracy is not increasing in classification images

body:
I try to implement the classification of images with bayesian CNN using dropout.
I have defined two classes:

with dropout for the training phase
without dropout for the test(Don’t drop out on testing? to be confirmed please)

When I started the program I remade that the train/test accuracy remain stable they don’t increase. I don’t see what the problem is.
I don’t know if it’s because of convolution and pooling layer parameters or what? Any idea please.
class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5, padding=2)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5, padding=2)
        self.fc1 = nn.Linear(16 * 8 * 8, 1024)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 192 * 8 * 8)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

            # Lenet with MCDO
    class Net_MCDO(nn.Module):
        def __init__(self):
            super(Net_MCDO, self).__init__()
            self.conv1 = nn.Conv2d(3, 6, 5, padding=2)
            self.pool = nn.MaxPool2d(2, 2)
            self.conv2 = nn.Conv2d(16, 192, 5, padding=2)
            self.fc1 = nn.Linear(16 * 8 * 8, 120)
            self.fc2 = nn.Linear(120, 84)
            self.fc3 = nn.Linear(84, 10)
            self.dropout = nn.Dropout(p=0.3)

    def forward(self, x):
            x = self.pool(self.dropout(self.conv1(x)))
            x = self.pool(self.dropout(self.conv2(x)))
            x = x.view(-1, 192 * 8 * 8)
            x = F.relu(self.fc1(x))
            x = F.relu(self.fc2(self.dropout(x)))
            x = F.softmax(self.fc3(self.dropout(x)),dim=1)
            return x

    net=Net()
    mcdo=Net_MCDO()

    CE = nn.CrossEntropyLoss()
    learning_rate=0.001
    optimizer=optim.SGD(net.parameters(), lr=learning_rate, momentum=0.9)
    epoch_num = 30
    train_accuracies=np.zeros(epoch_num)
    test_accuracies=np.zeros(epoch_num)
    for epoch in range(epoch_num):
        average_loss = 0.0
        total=0
        success=0

        for i, data in enumerate(trainloader, 0):
            inputs, labels = data
            inputs, labels = Variable(inputs), Variable(labels)

            optimizer.zero_grad()
            outputs = mcdo(inputs)
            loss=CE(outputs, labels)
            loss.backward()
            optimizer.step()

            average_loss += loss.item()
            _, predicted = torch.max(outputs.data, 1)
            total += labels.size(0)
            success += (predicted==labels.data).sum()

        train_accuracy = 100.0*success/total
        succes=0
        total=0

        for (inputs, labels) in testloader:
            inputs, labels = Variable(inputs), Variable(labels)
            outputs = net(inputs)
            _,predicted = torch.max(outputs.data, 1)
            total += labels.size(0)
            success += (predicted==labels.data).sum()

        test_accuracy = 100.0*success/total
        print(u"epoch{}, average_loss{}, train_accuracy{},
          test_accuracy{}".format(
          epoch,
          average_loss/n_batches,
          train_accuracy,
          100*success/total
          ))
            #save
            train_accuracies[epoch] = train_accuracy
            test_accuracies[epoch] = 100.0*success/total

    plt.plot(np.arange(1, epoch_num+1), train_accuracies)
    plt.plot(np.arange(1, epoch_num+1), test_accuracies)
    plt.show()


title: Signal denoising experiment - using cvxlayers and pytorch

body:
I am trying to use this code with different data to get a better understanding of how the algorithm works, however I am coming up with an error on the train and test -
the below is what i have tried -
import math

import cvxpy as cp
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import torch
from cvxpylayers.torch import CvxpyLayer

import latexify
latexify.latexify()

torch.set_default_tensor_type(torch.DoubleTensor)
%matplotlib inline

import yfinance as yf
in_test = yf.download("SPY", start="2012-01-01", end="2017-04-30")['Close']
lenght=len(in_test)
outputs=torch.from_numpy(in_test.to_numpy())
outputs=torch.reshape(outputs,[len(outputs),1])
ou_xx=np.linspace(1,100, len(yy))
inputs=torch.from_numpy(ou_xx)
inputs=torch.reshape(inputs,[len(inputs),1])
N_train=lenght-200
N_val=200
outputs

X_train = inputs[:N_train]
Y_train = outputs[:N_train]

X_val = inputs[N_train:]
Y_val = outputs[N_train:]

Y_val

def create_layer():
    y_cp = cp.Variable(n)
    x_minus_y = cp.Variable(n)

    x_param = cp.Parameter(n)
    theta_param = cp.Parameter((n, n))
    lambda_param = cp.Parameter(pos=True)
    objective = (
        cp.sum_squares(theta_param @ x_minus_y) +
        lambda_param*cp.sum_squares(cp.diff(y_cp))
    )
    constraints = [
        x_minus_y == x_param - y_cp
    ]
    problem = cp.Problem(cp.Minimize(objective), constraints)
    layer = CvxpyLayer(
        problem,
        parameters=[x_param, theta_param, lambda_param],
        variables=[y_cp])
    return layer

layer = create_layer()

import torch
from torch.utils.data import TensorDataset, DataLoader
import numpy as np
from cvxpylayers.torch import CvxpyLayer

torch.set_default_dtype(torch.double)

from tqdm.notebook import tqdm

#this function is where i am running into problems
def fit(loss, params, X, Y, Xval, Yval, batch_size=128, lr=1e-3, epochs=100, verbose=False, print_every=1, callback=None):
    """

    Arguments:
        loss: given x and y in batched form, evaluates loss.
        params: list of parameters to optimize.
        X: input data, torch tensor.
        Y: output data, torch tensor.
        Xval: input validation data, torch tensor.
        Yval: output validation data, torch tensor.
    """

    train_dset = TensorDataset(X, Y)
    train_loader = DataLoader(train_dset, batch_size=batch_size, shuffle=True)
    opt = torch.optim.Adam(params, lr=lr)

    train_losses = []
    val_losses = []
    for epoch in tqdm(range(epochs)):
        if callback is not None:
            callback()

        with torch.no_grad():
            val_losses.append(loss(Xval, Yval).item())
        if verbose and epoch % print_every == 0:
            print("val loss %03d | %3.5f" % (epoch + 1, val_losses[-1]))

        batch = 1
        train_losses.append([])
        for Xbatch, Ybatch in train_loader:
            opt.zero_grad()
            l = loss(Xbatch, Ybatch)
            l.backward()
            opt.step()
            train_losses[-1].append(l.item())
            if verbose and epoch % print_every == 0:
                print("batch %03d / %03d | %3.5f" %
                      (batch, len(train_loader), np.mean(train_losses[-1])))
            batch += 1
    return val_losses, train_losses

theta_tch = torch.eye(n, requires_grad=True)
lambda_tch = torch.tensor(0.5, requires_grad=True)
params = [theta_tch, lambda_tch]

def loss_fn(X, actual):
    preds = layer(X, theta_tch, lambda_tch)[0]
    mse_per_example = (preds - actual).pow(2).mean(axis=1)
    return mse_per_example.mean()

val_losses, train_losses =  fit(
    loss_fn, params, X_train, Y_train, X_val, Y_val, lr=1e-2, batch_size=8,
    epochs=15, verbose=True, print_every=1)

The above gives the error below
HBox(children=(FloatProgress(value=0.0, max=15.0), HTML(value='')))

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-68-cd369ca9dfdf> in <module>
      9
     10
---> 11 val_losses, train_losses =  fit(
     12     loss_fn, params, X_train, Y_train, X_val, Y_val, lr=1e-2, batch_size=8,
     13     epochs=15, verbose=True, print_every=1)

<ipython-input-67-f19c59cb9b44> in fit(loss, params, X, Y, Xval, Yval, batch_size, lr, epochs, verbose, print_every, callback)
     32
     33         with torch.no_grad():
---> 34             val_losses.append(loss(Xval, Yval).item())
     35         if verbose and epoch % print_every == 0:
     36             print("val loss %03d | %3.5f" % (epoch + 1, val_losses[-1]))

<ipython-input-68-cd369ca9dfdf> in loss_fn(X, actual)
      4
      5 def loss_fn(X, actual):
----> 6     preds = layer(X, theta_tch, lambda_tch)[0]
      7     mse_per_example = (preds - actual).pow(2).mean(axis=1)
      8     return mse_per_example.mean()

~/miniconda3/envs/myenv1/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    720             result = self._slow_forward(*input, **kwargs)
    721         else:
--> 722             result = self.forward(*input, **kwargs)
    723         for hook in itertools.chain(
    724                 _global_forward_hooks.values(),

~/cvxpylayers/cvxpylayers/torch/cvxpylayer.py in forward(self, solver_args, *params)
    150             info=info,
    151         )
--> 152         sol = f(*params)
    153         self.info = info
    154         return sol

~/cvxpylayers/cvxpylayers/torch/cvxpylayer.py in forward(ctx, *params)
    224                 p_shape = p.shape if batch_size == 0 else p.shape[1:]
    225                 if not np.all(p_shape == param_order[i].shape):
--> 226                     raise ValueError(
    227                         "Inconsistent parameter shapes passed in. "
    228                         "Expected parameter {} to have non-batched shape of "

ValueError: Inconsistent parameter shapes passed in. Expected parameter 0 to have non-batched shape of (100,) but got torch.Size([200, 1]).

I am not sure how to format the tensor in order to get it to train through the algorithm.
Please assist.
The reference for the above code is at the following github repo -
https://github.com/cvxgrp/cvxpylayers/blob/master/examples/torch/signal_denoising.ipynb


title: Vectorized way to apply a 3-dimension mask to RGB in pytorch

body:
I have a HxWx3 tensor representing an RGB image and a HxWx3 mask (boolean) tensor as input.
It is assumed that for each (i,j) in the mask tensor there's exactly one true value (that is exactly one of R\G\B is on).
I want to apply the mask to the image to result in a HxW (or HxWx1) tensor V where V[i,j]='the matching R\G\B value according to the mask'.
Using Problem applying binary mask to an RGB image with numpy I was able to achieve the following:
>>> X*mask
tensor([[[ 9., 10.],
         [ 0.,  0.]],

        [[ 0.,  0.],
         [ 0., 20.]],

        [[ 0.,  0.],
         [30.,  0.]]])

But as stated, I want a single dim HxW and not HxWx3 as result.
Illustration:


title: Getting not defined error while training clinical EHR pytorch model

body:
I am working on clinical EHR. I am currently referring to this blog and github link here. I have generated the dataset and processed it as per the instructions in the notebooks present in the repository. I am facing an issue trying to train the model.
build_EHRNN class.
torch.manual_seed(1)

class build_EHRNN(nn.Module):
    def __init__(self, inputDimSize=4894, hiddenDimSize=[200,200], batchSize=100, embSize=200, numClass=4894, dropout=0.5, logEps=1e-8):
        super(build_EHRNN, self).__init__()

        self.inputDimSize = inputDimSize
        self.hiddenDimSize = hiddenDimSize
        self.numClass = numClass
        self.embSize = embSize
        self.batchSize = batchSize
        self.dropout = nn.Dropout(p=0.5)
        self.logEps = logEps

        # Embedding inputs
        self.W_emb = nn.Parameter(torch.randn(self.inputDimSize, self.embSize).cuda())
        self.b_emb = nn.Parameter(torch.zeros(self.embSize).cuda())

        self.W_out = nn.Parameter(torch.randn(self.hiddenDimSize, self.numClass).cuda())
        self.b_out = nn.Parameter(torch.zeros(self.numClass).cuda())

        self.params = [self.W_emb, self.W_out,
                       self.b_emb, self.b_out]

    def forward(self,x, y, h, lengths, mask):
        self.emb = torch.tanh(torch.matmul(x, self.W_emb) + self.b_emb)
        input_values = self.emb
        self.outputs = [input_values]
        for i, hiddenSize in enumerate([self.hiddenDimSize, self.hiddenDimSize]):  # iterate over layers
            rnn = EHRNN(self.inputDimSize,hiddenSize,self.embSize,self.batchSize,self.numClass) # calculate hidden states
            hidden_state = []
            h = self.init_hidden().cuda()
            for i,seq in enumerate(input_values): # loop over sequences in each batch
                h = rnn(seq, h)
                hidden_state.append(h)
            hidden_state = self.dropout(torch.stack(hidden_state))    # apply dropout between layers
            input_values = hidden_state

        y_linear = torch.matmul(hidden_state, self.W_out)  + self.b_out # fully connected layer
        yhat = F.softmax(y_linear, dim=1)  # yhat
        yhat = yhat*mask[:,:,None]   # apply mask

        # Loss calculation
        cross_entropy = -(y * torch.log(yhat + self.logEps) + (1. - y) * torch.log(1. - yhat + self.logEps))
        last_step = -torch.mean(y[-1] * torch.log(yhat[-1] + self.logEps) + (1. - y[-1]) * torch.log(1. - yhat[-1] + self.logEps))
        prediction_loss = torch.sum(torch.sum(cross_entropy, dim=0),dim=1)/ torch.cuda.FloatTensor(lengths)
        cost = torch.mean(prediction_loss) + 0.000001 * (self.W_out ** 2).sum() # regularize
        return (yhat, hidden_state, cost)

    def init_hidden(self):
        return torch.zeros(self.batchSize, self.hiddenDimSize)  # initial state

Creating instance and training model
model = build_EHRNN(inputDimSize=4894, hiddenDimSize=200, batchSize=100, embSize=200, numClass=4894, dropout=0.5, logEps=1e-8)
model = model.to(device)
optimizer = torch.optim.Adadelta(model.parameters(), lr = 0.01, rho=0.90)
max_epochs = 10

loss_all = []
iteration = 0

for e in range(max_epochs):
    for index in random.sample(range(n_batches), n_batches):
        batchX = train[0][:n_batches*batchSize][index*batchSize:(index+1)*batchSize]
        batchY = train[1][:n_batches*batchSize][index*batchSize:(index+1)*batchSize]

        optimizer.zero_grad()

        x, y, lengths, mask = padding(batchX, batchY, 4894, 4894)

        if torch.cuda.is_available():
            x, y, lenghts, mask = x.cuda(), y.cuda(), lengths, mask.cuda()

        outputs, hidden, cost = model(x,y, h, lengths, mask)

        if torch.cuda.is_available():
            cost.cuda()
        cost.backward()
        nn.utils.clip_grad_norm_(model.parameters(), 5)
        optimizer.step()

Error:
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-14-cff1f002dced> in <module>()
     17             x, y, lenghts, mask = x.cuda(), y.cuda(), lengths, mask.cuda()
     18
---> 19         outputs, hidden, cost = model(x,y, h, lengths, mask)
     20
     21         if torch.cuda.is_available():

NameError: name 'h' is not defined

Update:
Removing 'h' param produces the following error
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-14-6495250d91c9> in <module>()
     18
     19         # outputs, hidden, cost = model(x,y, h, lengths, mask)
---> 20         outputs, hidden, cost = model(x, y, lengths, mask)
     21
     22         if torch.cuda.is_available():

1 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    720             result = self._slow_forward(*input, **kwargs)
    721         else:
--> 722             result = self.forward(*input, **kwargs)
    723         for hook in itertools.chain(
    724                 _global_forward_hooks.values(),

<ipython-input-7-3c831fe3ca8d> in forward(self, x, y, lengths, mask)
     36                 h = rnn(seq, h)
     37                 hidden_state.append(h)
---> 38             hidden_state = self.dropout(torch.stack(hidden_state))    # apply dropout between layers
     39             input_values = hidden_state
     40

RuntimeError: stack expects a non-empty TensorList


title: Inference with pytorch and tensorflow model on single GPU

body:
I have two models one for Task1 which is trained using tf and other for Task2 which is trained pytorch. I want to use both for inferring depending on task. I have managed to achieve this using below code.
The problem though is, sometimes the dataloader crash/hang because of multiprocessing. I have checked dataloader and found this to be issue. Things work fine without multiprocessing.
Is there a way around ? I also tried p.terminate after p.join(). It did not help.
           if Task1:
                p = multiprocessing.Process(target= pytorch_fn,
                                                    args=)
                p.start()
                p.join()

            else:
                p = multiprocessing.Process(target= tensorflow_fn,
                                                    args=)

                p.start()
                p.join()


title: Loading a model checkpoint in lesser amount of memory

body:
I had a question that I can't find any answers to online. I have trained a model whose checkpoint file is about 20 GB. Since I do not have enough RAM with my system (or Colaboratory/Kaggle either - the limit being 16 GB), I can't use my model for predictions.
I know that the model has to be loaded into memory for the inferencing to work. However, is there a workaround or a method that can:

Save some memory and be able to load it in 16 GB of RAM (for CPU), or the memory in the TPU/GPU
Can use any framework (since I would be working with both) TensorFlow + Keras, or PyTorch (which I am using right now)

Is such a method even possible to do in either of these libraries? One of my tentative solutions was not load it in chunks perhaps, essentially maintaining a buffer for the model weights and biases and performing calculations accordingly - though I haven't found any implementations for that.

I would also like to add that I wouldn't mind the performance slowdown since it is to be expected with low-specification hardware. As long as it doesn't take more than two weeks :) I can definitely wait that long...


title: Equivalence beteween Pytroch and TensorFlow padding (reflect mode)

body:
I'm currently working on a basic convolutionnal neuron network. Actually, the CNN is ready to use using Pytorch. Nevertheless I'm trying to replicate the CNN and the entire process using TensorFlow.
I have a problem with the preprocessing of the images. Actually they are classical 3D images (R,G,B channels) of size 32x32. The issue comes from the padding (reflect mode) which seems to be different between the two libraries. Here's the code using Pytorch :
# Image preprocessing
trans_train = transforms.Compose([transforms.ToPILImage(),
                                  transforms.Pad(32, padding_mode='reflect'),
                                  transforms.ToTensor(),
                                  transforms.Normalize(mean=[0.5, 0.5, 0.5],std=[0.5, 0.5, 0.5])])

# Data generators
dataset_train = MyDataset(df_data=train, data_dir=train_path, transform=trans_train)

loader_train = DataLoader(dataset = dataset_train, batch_size=batch_size, shuffle=True, num_workers=0)

At this time I'm supposed to have "images" resized like 96x96x3.
Here's my equivalent using TensorFlow :
def load_and_preprocess_image(imagefile):

  def prepocess_image(image):

    def normalize_tensor(tensor,mean,std):

      mean_tensor = tf.constant(mean, shape=[1,3])
      std_tensor = tf.constant (std, shape=[1,3])
      tensor = tf.math.subtract(tensor, mean_tensor, name= None)
      tensor = tf.math.divide(tensor,std_tensor, name=None)

      return tensor

    image = tf.image.decode_jpeg(image,channels=3) # R,G,B mode, .jpeg format
    image = tf.pad(image, paddings=[[31,31],[31,31],[0,0]] , mode='REFLECT', name=None)
    image = image /255 # when imported with Pytorch images are scaled between 0.0 and 1.0
    image = normalize_tensor(image,[0.5,0.5,0.5],[0.5,0.5,0.5])

    return image

  image = tf.io.read_file(imagefile)
  return prepocess_image(image)

But at this time the returned tensor got a shape like 94x94x3, which is due to the padding with the 'reflect' mode has to respect this :
If mode is "REFLECT" then both paddings[D, 0] and paddings[D, 1] must be no greater than tensor.dim_size(D) - 1 (TensorFlow documentation)
The problem is that the CNN was designed to have images with a shape of 96x96x3 as inputs.
Is there a way to bypass this padding problem without changing the design of the CNN ?
Thanks


title: converting einsum notation to for loops

body:
I am trying to convert the following einsum notation into for loops as I cannot fully understand what is happening to the matrices and am unable to use loops to replicate the results myself:
np.einsum('bijc,bijd->bcd', x, x)
Any help is appreciated.


title: creating a train and a test dataloader

body:
I have actually a directory RealPhotos containing 17000 jpg photos. I would be interested in creating a train dataloader and a test dataloader
ls RealPhotos/
2007_000027.jpg  2008_007119.jpg  2010_001501.jpg  2011_002987.jpg
2007_000032.jpg  2008_007120.jpg  2010_001502.jpg  2011_002988.jpg
2007_000033.jpg  2008_007123.jpg  2010_001503.jpg  2011_002992.jpg
2007_000039.jpg  2008_007124.jpg  2010_001505.jpg  2011_002993.jpg
2007_000042.jpg  2008_007129.jpg  2010_001511.jpg  2011_002994.jpg
2007_000061.jpg  2008_007130.jpg  2010_001514.jpg  2011_002996.jpg
2007_000063.jpg  2008_007131.jpg  2010_001515.jpg  2011_002997.jpg
2007_000068.jpg  2008_007133.jpg  2010_001516.jpg  2011_002999.jpg
2007_000121.jpg  2008_007134.jpg  2010_001518.jpg  2011_003002.jpg
2007_000123.jpg  2008_007138.jpg  2010_001520.jpg  2011_003003.jpg
...

I know I can subclassing TensorDataset to make it compatible with unlabeled data with
class UnlabeledTensorDataset(TensorDataset):
    """Dataset wrapping unlabeled data tensors.

    Each sample will be retrieved by indexing tensors along the first
    dimension.

    Arguments:
        data_tensor (Tensor): contains sample data.
    """
    def __init__(self, data_tensor):
        self.data_tensor = data_tensor

    def __getitem__(self, index):
        return self.data_tensor[index]

And something along these lines for training the autoencoder
X_train     = rnd.random((300,100))
train       = UnlabeledTensorDataset(torch.from_numpy(X_train).float())
train_loader= data_utils.DataLoader(train, batch_size=1)

for epoch in range(50):
    for batch in train_loader:
        data = Variable(batch)
        optimizer.zero_grad()
        output = model(data)
        loss = criterion(output, data)