title: Applying a 2d convolution on a 2d Grid in Pytorch

body:
I am trying to do a 2d convolution, one a 2d grid, represented by a tensor of the following shape

[batch_dim, width, height]

Where the first dimensions is the 'batch dimension' an the second and third dimension represent the 2d grid

When I try to apply a 2d convolution using the following code:

nn.Conv2d(1, 32, kernel_size=3, stride=1)

I get the following error:

RuntimeError: Expected 4-dimensional input for
4-dimensional weight 32 1 1 1, but got 3-dimensional
input of size [1, 132, 7] instead

Why does it a require 4d tensor for 2d grid with one channel? How should I implement a 2d convolution on a 2d grid?


title: ValuerError regarding dimensions when declaring PyTorch tensor

body:
I'm currently trying to convert a list of values into a PyTorch tensor and am facing some difficulties.

The exact code that's causing the error is:

input_tensor = torch.cuda.FloatTensor(data)

Here, data is a list with two elements: The first element is another list of NumPy arrays and the second element is a list of tuples. The sizes of both lists differ, and I believe this is causing the following error:

*** ValueError: expected sequence of length x at dim 2 (got y)

Usually y is larger than x. I've tried playing around with an IPython terminal to see what's wrong, and it appears that trying to convert data of this format directly into PyTorch tensors doesn't work. Taking each individual element of the data list and converting those into tensors works, though.

Does anybody know why this doesn't work and perhaps also be able to provide some feedback on how to achieve my original goal? Thanks in advance.


title: Customized MPII dataset python3 class for pytorch 1.3

body:
I'm new to this kind of image processing work. And I want to build an MPII dataset described hereclass for training a pytorch network on it. I tend to work with Python 3.6 and pytorch 1.4

The problem is mainly from that MPII dataset annotation (download here) is in .mat form, which I'm not very familiar with. There are also some implementations I found on Github, like this deep high resolution net, and also this Microsoft MPII dataset for python. The latter is actually adopted in many implementations on the MPII dataset, but it's way too old. It's built with pytorch==0.4.0 and python2, creating a lot of environmental conflict problems.

What I want to do now is just to build a simple pytorch dataset which can automatically extract the joint positions in the 'annotation/' directory, and combine it with the specific image in 'image/' directory. I hope the return value is just a dictionary with these two items, that's all I need.

I also noticed that many researchers tend to convert the original '.mat' annotation file to '.json' format. I think this will make it much easier to operate, but there is some troubles when I'm trying to do this.

Python 3.6.10 |Anaconda, Inc.| (default, Jan  7 2020, 21:14:29)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import json
>>> from scipy.io import loadmat
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/billy/miniconda3/envs/datasci/lib/python3.6/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/home/billy/miniconda3/envs/datasci/lib/python3.6/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/home/billy/miniconda3/envs/datasci/lib/python3.6/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/home/billy/miniconda3/envs/datasci/lib/python3.6/json/encoder.py", line 180, in default
  File "/home/billy/miniconda3/envs/datasci/lib/python3.6/json/encoder.py", line 180, in default
    o.__class__.__name__)
TypeError: Object of type 'bytes' is not JSON serializable

I also saw this question on this site, but I am not sure I understand what is wrong... I 'd appreciate any help very much!!


title: How can I determine neighborhood from pairwise distance matrix efficiently?

body:
I have a M * N pairwise distance matrix between M points from group A and N points from group B.

I want to get the list of neighbor points from group B for each points from group A.

Is there any efficient code for this problem using pytorch? instead of multiple 'for' loop.

Thanks


title: Define a network in pytorch with incomplete connections, like convolution

body:
I'd like to train a small neural network in Pytorch that takes as an input an 8-dimensional vector and predicts one of three possible categories. The first hidden layer should contain 6 neurons, where each neuron takes the activations of only 3 consecutive dimensions in the input layer. The second hidden layer should also contain 6 nodes and be fully connected, and the last layer should be the output layer with 3 neurons. Thus the topology is:

network topology

Let's say that a mini batch consists of 64 (8-dimensional) data points.

I tried to implement the first layer using 1D convolution. Since a 1D convolution filter assumes the input is a sequence of points, I thought a good approach is to define 6 filters operating on 8 1-dimensional points:

import torch.nn as nn
import torch.nn.functional as functional

class ExampleNet(nn.Module):

    def __init__(self, batch_size, input_channels, output_channels):
        super(ExampleNet, self).__init__()
        self._layer1 = nn.Conv1d(in_channels=1, out_channels=input_channels - 2, kernel_size=3, stride=1)
        self._layer2 = nn.Linear(in_features=input_channels - 2, out_features=input_channels - 2)
        self._layer3 = nn.Linear(in_features=input_channels - 2, out_features=output_channels)

    def forward(self, x):
        x = functional.relu(self._layer1(x))
        x = functional.relu(self._layer2(x))
        x = functional.softmax(self._layer3(x))
        return x

net = ExampleNet(64, 8, 3)

I know that Pytorch expects a sequence of arrays of size 64 x 8 x 1 each when training the network. However, since I apply 1D convolutional filters in an untraditional way, I think I should have input arrays of size 64 x 1 x 8, and I am expecting an output of size 64 x 3. I use the following mini-batch of random points to run through the network:

# Generate a mini-batch of 64 samples
input = torch.randn(64, 1, 8)
out = net(input)
print(out.size())

And the output I get tells me that I defined a wrong topology. How would you advise me to define the layers I need? Is using Conv1d a good approach in my case? I saw that another approach is to use a masked layer but I don't know how to define it.


title: An appropriate way of adding a feature to a time series forecasting model input

body:
I have been working on a demand forecasting model for a while. I am using an LSTM model to predict the  future demand of a product family of a company. To solidify and exemplify my raw data, an example is as below;

Unprocessed data
np.random.seed(1)

raw_data = pd.DataFrame({"product_type": ["A"]*3 + ["B"]*3 + ["C"]*3, "product_family": ["x", "y", "z", "t", "u", "y", "p", "k", "l"]})

for col in [str(x)+"-"+str(y) for x in range(2015, 2020) for y in range(1, 13)]:
    raw_data[col] = np.random.randint(10, 50, 9)

raw_data.head()

  product_type product_family  2015-1  ...  2019-10  2019-11  2019-12
0            A              x      47  ...       15       39       38
1            A              y      22  ...       37       28       29
2            A              z      18  ...       41       41       37
3            B              t      19  ...       32       44       29
4            B              u      21  ...       22       29       25
[5 rows x 62 columns]

As can be seen above, the data has two nominal feature, and the rest are the past demand data.

First, let me interpret what I do in my case:

I first select the product_family to be predicted and let that product_family be "x":
prod_family_data = raw_data.loc[raw_data.product_family == "x", raw_data.columns[2:]].to_numpy()

Then I create the x and y of the training set:
x_train, y_train = [], []

for i in range(0, len(prod_family_data) - 12):
    x_train.append(prod_family_data[i: i + 12])
    y_train.append(prod_family_data[i + 12])

x_train = np.array(x_train)

y_train = np.array(y_train)

array([[47, 11, 21, 32, 34, 14, 35, 49, 44, 42, 31, 18],
       .
       .
       .
       [14, 20, 45, 13, 48, 43, 45, 49, 49, 37, 15, 39]], dtype=object)

y_train

array([28, 38, 12, 12, 23, 29, 19, 23, 39, 38, 18, 40, 46, 48, 44, 27, 10,
       24, 25, 22, 15, 28, 44, 46, 22, 12, 45, 47, 38, 21, 46, 26, 12, 21,
       18, 14, 20, 45, 13, 48, 43, 45, 49, 49, 37, 15, 39, 38])

x_train = x_train.reshape(x_train.shape[0], x_train.shape[1], 1)

x_train.shape

(48, 12, 1)

y_train.shape

(48,)

Then I predict the product_family's demand with a LSTM model, then I go back to the start, select another product_family, rinse and repeat.
What I wonder is if there is a way to add the product_family feature to the input (and may be product_type and other nominal qualities of products in the future too) of the model, and feed it to the model all at once?
Also, is there a way to bound the demand data with the timestamps to the input so that the model will catch the trend or seasonality of the data/


title: Is there a way in Python to select only certain variables (from the data set) for input features used in Deep Learning?

body:
I am very new to implementing deep-learning for survival analysis using Pycox package.

I am using The Molecular Taxonomy of Breast Cancer International Consortium (METABRIC), which
contains variables x0 to x8 with time duration and event (0=no, 1=yes).

When I did the example from the Pycox Pycox Github, I realized that by default, all the variables
from x0 to x8 are being used as input features for training, testing, and validation

However, I want to train the LogisticHazard (Nnet-survival) including only certain features as follows:

Model 1=x0, x1, x2

Model 2=x0, x1, x2, x3, x4, x5

Model 3=x0, x1, x2, x3, x4, x5, x6, x7, x8

Pycox-LogisticHazard Example with METABRIC

df_train = metabric.read_df()

df_test = df_train.sample(frac=0.2)

df_train = df_train.drop(df_test.index)

df_val = df_train.sample(frac=0.2)

df_train = df_train.drop(df_val.index)

df.train.head()

Data Set Overview

How do I easily set Model 1, Model 2, Model 3 for training, testing, and validation sets?
so that I can customize which input variables (e.g Model 1: x0, x1, x2) to use for x_train, x_val, x_test ?

Any help will be appreciated.


title: Pytorch model forward speed is lowered while using torch.multiprocessing

body:
I am new to pytorch and recently I found the model forward speed is lowered using torch.multiprocessing when I running the code in Pytorch Documentation Multiprocessing Best Practice and pytorch example of mnist_hogwild.

When I set the num_processes as 1, the forward time is about 5ms. But the forward time increased to 10ms when num_processes = 2 and 16-30ms when num_processes = 4

forward time when num_process = 1

forward time when num_process = 2

forward time when num_process = 4

At first I thought it was because the model was shared only with parameters and there is conflicts in computing resources. So I changed the code and create model instances inside the subprocesses. There are the test results below.

forward time when num_process = 1 and models are independent in subprocesses

forward time when num_process = 2 and models are independent in subprocesses

forward time when num_process = 4 and models are independent in subprocesses

The forward speed of model is lowered though models are indepedent.

What's going on there?


title: Runtime Error when using custom linear layer in Pytorch

body:
I am using a custom Linear Layer in Pytorch, whose job is to add noise to the network.
class NoisyLinear(nn.Linear):

    def __init__(self, in_features, out_features, sigma_init=0.017, bias=True):
        super(NoisyLinear, self).__init__(in_features, out_features, bias=bias)
        self.sigma_weight = nn.Parameter(torch.full((out_features, in_features), sigma_init))
        self.register_buffer("epsilon_weight", torch.zeros(out_features, in_features))
        if bias:
            self.sigma_bias = nn.Parameter(torch.full((out_features,), sigma_init))
            self.register_buffer("epsilon_bias", torch.zeros(out_features))
        self.reset_parameters()

    def reset_parameters(self):
        std = math.sqrt(3 / self.in_features)
        self.weight.data.uniform_(-std, std)
        self.bias.data.uniform_(-std, std)

    def forward(self, input):
        self.epsilon_weight.normal_()
        bias = self.bias
        if bias is not None:
            self.epsilon_bias.normal_()
            bias = bias + self.sigma_bias * self.epsilon_bias
        return F.linear(input, self.weight + self.sigma_weight * self.epsilon_weight, bias)

the Input size would be 5 and the output would be 256.
but it gave me that error.
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [256, 5]] is at version 3; expected version 2 instead.

Although when I replace this custom layer with the normal linear layer it works fine.


title: How to make a neural network generalizes better?

body: I designed a neural network model with large number of output predicted by softmax function. However, I want categorize all the outputs into 5 outputs without modifying the architecture of other layers. The model performs well in the first case but when I decrease the number of output it loses accuracy and get a bad generalization. My question is : Is there a method to make my model performs well even if there is just 5 outputs ? for example : adding dropout layer before output layer, using other activation function, etc.


title: Perform batch normalization with PyTorch in Java

body:
I am wondering if I could perform batch normalization in Java. I know that PyTorch in Java could do forward if we load a model. I am wondering how could I?

Save bn.weights/bn.bias/running_average/running_std somewhere in local(I could get these in Python then either save as checkpoint file (but I am not sure how would Java know where to find these info when Java load a model, somehow related to my 2. method.) Or save these number separately in a .txt file) then let Java read these info.

or

Directly extract bn.weights/bn.bias/running_average/running_std with the help of PyTorch library in Java (is this supported in Java PyTorch?)

Some other method?

Then perform batch normalization in Java?


title: VGG16 faster than Mobilenet_v2 when batch_size equal to 1 on inference,use pytorch 1.0.0 GPU RTX2080Ti

body:
I use pytorch to evaluate the run time of  VGG16 and Mobilenet_v2,and I use batch size equal to 1.
When batch size is 1,my VGG16 result is Avg time: 0.0051 s (197.2 fps).
Mobilenet_v2 result is Avg time: 0.0058 s (173.0 fps).
I wonder why VGG16 is faster than Mobilenet_v2 when batch size 1.
if Batch size equal to 12,the result of VGG16 is Avg time: 0.0340 s (29.4 fps).
While Mobilenet_v2 is Avg time: 0.0060 s (165.4 fps).
is my code have something wrong? Why is VGG16 so faster? Or evaluation run time can't use batch size equal to 1.
Thank.
import torch
#import csv
import torch.nn as nn
from torch.utils.data import Dataset
from torch.utils.data import DataLoader
#from PIL import Image, ImageFile
import torchvision.transforms as transforms
from torchvision import models
import torch.utils.model_zoo as model_zoo
import torch.nn.functional as F
import torch.optim as optim
import os
import numpy as np
from backbone import *
import cv2
from tensorboardX import SummaryWriter
import argparse
from torchsummaryX import summary
import time
import warnings
warnings.filterwarnings("ignore")

if __name__ == "__main__":
    device = torch.device('cuda')
    torch.backends.cudnn.benchmark = False
    torch.backends.cudnn.deterministic =True
    parser = argparse.ArgumentParser()
    parser.add_argument('--backbone', help='the backbone model', type=str, default="Mobile_v2")
    args = parser.parse_args()
    #print(args)
    phase = 'train'
    shape_r,shape_c = 224,224
    model_name = f'{backbone_name}'
    print(model_name)
    if(backbone_name == 'VGG16'):
        backbone = models.vgg16(pretrained=True).to(device)
    if(backbone_name == 'Mobile_v2'):
        backbone = mobilenet_v2(pretrained=True).to(device)
    #Time it
    def time_cal2(m):
        times = []
        global device
        batchsize = 1
        with torch.no_grad():
            m.eval()
            x_i = torch.rand(batchsize,3,shape_r,shape_c).to(device)
            y = m.forward(x_i)
            print(y.size())
            for i in range(100):
                x_i = torch.rand(batchsize,3,shape_r,shape_c).to(device)
                torch.cuda.synchronize()
                t0 = time.time()
                y = m.forward(x_i)
                torch.cuda.synchronize()
                dt = time.time() - t0
                times.append(dt)
        dt = sum(times) / len(times)
        print(f"Avg time: {dt:.4f} s ({1 / dt:.1f} fps)")
        return dt
    summary(backbone,torch.rand(1,3,shape_r,shape_c).to(device))
    print(backbone_name)
    time_cal2(backbone)


title: Pytorch cifar dataset: RuntimeError: size mismatch, m1: [4 x 2048], m2: [1568 x 10]

body:
I am making my own network with 2 basic layers and perform training on CIFAR10 dataset. I am getting mismatch error.
import torch
import torchvision
import torchvision.transforms as transforms
from torch.utils.data import Dataset, DataLoader
import torch.nn as nn

device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')

num_epochs = 5
num_classes = 10
batch_size = 4
learning_rate = 0.001

print("---------Train Dataset----------")
train_dataset = torchvision.datasets.CIFAR10(root='../data/', train=True,
                                             transform=transforms.ToTensor(), download=True)

print("---------Train Loader----------")
train_loader = torch.utils.data.DataLoader(dataset=train_dataset, batch_size=batch_size, shuffle=True)

class ConvNet(nn.Module):
    def __init__(self, num_classes=10):
        super(ConvNet, self).__init__()

        # First Layer
        self.layer1 = nn.Sequential(
            nn.Conv2d(3, 16, kernel_size=3, stride=1, padding=1),
            nn.BatchNorm2d(16),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2))

        # Second Layer
        self.layer2 = nn.Sequential(
            nn.Conv2d(16, 32, kernel_size=3, stride=1, padding=1),
            nn.BatchNorm2d(32),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2))

        # Fully connected Layer
        self.fc = nn.Linear(7 * 7 * 32, num_classes)

    def forward(self, x):
        out = self.layer1(x)
        out = self.layer2(out)
        out = out.reshape(out.size(0), -1)
        out = self.fc(out)
        return out

model = ConvNet(num_classes).to(device)

criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

print("--------Train Model----------")

total_step = len(train_loader)
print("Total Step", total_step)

for epoch in range(num_epochs):
    for i, (images, labels) in enumerate(train_loader):

        images = images.to(device)
        labels = labels.to(device)

        print("-----Forward Pass-------")
        outputs = model(images)
        loss = criterion(outputs, labels)

        print ("----Backward Pass-------")
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if (i+1) % 100 == 0:
            print(epoch+1, num_epochs, i+1, total_step, loss.item())


title: Installation of GANet

body:
I'm trying to use the following NN:
GANet
This error happening when I doesn't use apex
Traceback (most recent call last):
  File "predict.py", line 187, in <module>
    test(leftname, rightname, savename)
  File "predict.py", line 158, in test
    prediction = model(input1, input2)
  File "/home/master/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/master/anaconda3/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 141, in forward
    return self.module(*inputs[0], **kwargs[0])
  File "/home/master/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/master/Work/Project01/GANet/models/GANet_deep.py", line 396, in forward
    g = self.conv_start(x)
  File "/home/master/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/master/anaconda3/lib/python3.7/site-packages/torch/nn/modules/container.py", line 92, in forward
    input = module(input)
  File "/home/master/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/master/Work/Project01/GANet/models/GANet_deep.py", line 42, in forward
    x = self.bn(x)
  File "/home/master/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/master/Work/Project01/GANet/libs/sync_bn/modules/sync_bn.py", line 121, in forward
    self.activation, self.slope).view(input_shape)
  File "/home/master/Work/Project01/GANet/libs/sync_bn/functions/sync_bn.py", line 95, in forward
    y = sync_bn_gpu.batchnorm_forward(x, _ex, _exs, gamma, beta, ctx.eps)
RuntimeError: cudaGetLastError() == cudaSuccess ASSERT FAILED at src/gpu/sync_bn_cuda.cu:289, please report a bug to PyTorch. (BatchNorm_Forward_CUDA at src/gpu/sync_bn_cuda.cu:289)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x45 (0x7fca7ac1bcc5 in /home/master/anaconda3/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #1: BatchNorm_Forward_CUDA(at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, float) + 0x2be (0x7fca76e56ff1 in /home/master/Work/Project01/GANet/libs/sync_bn/build/lib/sync_bn_gpu.cpython-37m-x86_64-linux-gnu.so)
frame #2: <unknown function> + 0x22ab3 (0x7fca76e54ab3 in /home/master/Work/Project01/GANet/libs/sync_bn/build/lib/sync_bn_gpu.cpython-37m-x86_64-linux-gnu.so)
frame #3: <unknown function> + 0x201a5 (0x7fca76e521a5 in /home/master/Work/Project01/GANet/libs/sync_bn/build/lib/sync_bn_gpu.cpython-37m-x86_64-linux-gnu.so)
<omitting python frames>
frame #11: THPFunction_apply(_object*, _object*) + 0x5a1 (0x7fcaab0b4bf1 in /home/master/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_python.so)

following the installation recommendations:

I installed apex, the only installation that worked for me is:
pip uninstall apex
rm -rf apex

git clone https://github.com/ptrblck/apex.git
cd apex
git checkout apex_no_distributed
pip install -v --no-cache-dir ./

source
with apex (and replacing all the BatchNorm2d and BatchNorm3d to apex.parallel.SyncBatchNorm) the NN running but return all zeros result matrix.
Any suggestions?
Working on ubuntu 18.04
with conda,
python 3.7.4,
torch 1.0.0,
cuda 10.2
gcc 7.5


title: pytorch : How to stack 2 tensors

body:
I’m trying to stack 2 tensors A.shape=(64,16,16) and B.shape=(64,16,16) in a tensor of shape C.shape=(1,128,16,16)
and non of functions I’ve tried work where
torch.stack => C.shape=(2,64,16,16) and
torch.cat => C.shape=(128,16,16)
can enyones help me


title: How to ensure the repeatability of the results of the Pytorch model across devices？

body:
I am a little bit confused about the randomness of the pytorch model.
I used the following code to fix the random seed so that the training results of the model can be repeated on the same device：
def seed_torch(seed=2020):
    random.seed(seed)
    os.environ["PYTHONHASHSEED"] = str(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed(seed)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False

But what is confusing is that when the code is run on another device with the same hardware configuration and software environment, it will produce different results.
I reinstalled the virtual environment to ensure that the versions of the libraries are consistent, but this problem still puzzles me.
Feel free to ask if more code is needed to explain the problem.


title: RNN Variational Autoencoder produces good reconstruction, but poor generation

body:
I am trying to reproduce the results of this paper by training the RNN based variational autoencoder. While reconstruction of the original text is working very well, the generation of new text is terrible. I have given my model architecture below. It is loosely based on this repository
class SentenceVAE(nn.Module):
    def __init__(self, embedding_size, vocab_size, hidden_size, latent_dim, dropout, device,
                 max_len = 50, pad_idx = 0, start_idx = 1, end_idx = 2, unk_idx = 3):
        super(SentenceVAE, self).__init__()

        self.tensor = torch.cuda.FloatTensor if torch.cuda.is_available() else torch.Tensor
        self.embed = nn.Embedding(vocab_size, embedding_size, pad_idx)
        self.hidden_to_mu = nn.Linear(hidden_size, latent_dim)
        self.hidden_to_logvar = nn.Linear(hidden_size, latent_dim)
        self.dropout = nn.Dropout(dropout)
        self.encoder_gru = nn.GRU(embedding_size, hidden_size, batch_first=True)
        self.decoder_gru = nn.GRU(embedding_size, hidden_size, batch_first=True)
        self.flow_fc = nn.Sequential(
            nn.Linear(latent_dim, 1024),
            nn.GELU(),
            nn.Linear(1024, hidden_size)
        )
        self.out = nn.Linear(hidden_size, vocab_size)
        self.device = device
        self.latent_dim = latent_dim
        self.unk_idx = unk_idx
        self.start_idx = start_idx
        self.end_idx = end_idx
        self.pad_idx = pad_idx

    def reparameterize(self, mu, logvar):
        eps = torch.randn_like(logvar)
        std = torch.exp(0.5 * logvar)
        return mu + eps * std

    def decode(self, hidden, dec_in):
        decoder_input = self.embed(dec_in)
        if len(hidden.size()) < 3:
            hidden = hidden.unsqueeze(0)
        outputs, hidden = self.decoder_gru(decoder_input, hidden)
        out = self.out(outputs)
        return out, hidden

    def sample_sentence(self, z = None):
        max_len = 20
        batch = 1
        if z == None:
            z = torch.randn((batch,self.latent_dim))
            z = z.to(self.device)
        hidden = self.flow_fc(z)
        pred = [[self.start_idx]]
        out_sent = []
        for i in range(max_len):
            pred_tensor = torch.tensor(pred)
            pred_tensor = pred_tensor.to(device)
            preds, hidden = self.decode(hidden, pred_tensor)
            preds = preds[:, -1, :]
            pred_index = torch.argmax(preds, dim = -1)
            pred[0] = [pred_index.item()]
            out_sent.append(pred_index.item())
            if pred_index.item() == self.end_idx:
                break
        return out_sent

    def forward(self, x):
        enc_in, dec_in = x, x
        encoder_input = self.embed(enc_in)
        _, rnn_hidden = self.encoder_gru(encoder_input)
        rnn_hidden = rnn_hidden.squeeze(0)
        mu = self.hidden_to_mu(rnn_hidden)
        logvar = self.hidden_to_logvar(rnn_hidden)
        z = self.reparameterize(mu, logvar)
        ## Randomly replace words with <unk>
        dec_in_copy = dec_in.clone()
        prob = torch.rand(dec_in.size())
        prob[(dec_in == self.start_idx) | (dec_in == self.end_idx) | (dec_in == self.pad_idx)] = 1
        dec_in_copy[prob < dropout] = self.unk_idx
        hidden = self.flow_fc(z)
        out, _= self.decode(hidden, dec_in_copy)

        return mu, logvar, out

Here are some sample reconstruction outputs, the first line is the input and second is the output
#1
<SOS> gondry 's direction is adequate ... but what gives human nature its unique feel is kaufman 's script . <EOS>
<SOS> it 's direction is adequate ... but what gives human nature its unique feel is kaufman 's approach . <EOS>

#2
<SOS> there seems to be no clear path as to where the story 's going , or how long it 's going to take to get there . <EOS>
<SOS> there seems to be no amount path , to where the most 's going , or even long it 's going to take to get there . <EOS>

Now if I use the sample_sentence method in the class SentenceVAE to generate a new sentence, the output is always:
<SOS> <SOS> <SOS> <SOS> <SOS> <SOS> <SOS> <SOS> <SOS> <SOS> <SOS> <SOS> <SOS> <SOS> <SOS> <SOS> <SOS> <SOS> <SOS> <SOS>

While debugging I noticed that the output is always the token which is inside pred = [[self.start_idx]]  which repeats max_len number of times. In the above case the <SOS> was given as the input token in sample_sentence.


title: How to formulate this particular learning rate scheduler in PyTorch?

body:
I'm working with transformer based model, but unable to formulate this learning rate scheduler with PyTorch
lrate = d_model ^ 0.5 * min( step_num ^ 0.5, step_num * warmup_steps ^ -1.5)

Thanks


title: How to train a reinforcement learning model by the total steps' rewards

body:
I am trying to use reinforcement learning to solve an optimal combination problem. The combination problem means I have like 50 actions, and I hope to find the optimal sequence of these actions to achieve the highest total rewards. Here is the code I am using:
class ConvNet(nn.Module):
   ### I build the neural network here
   return

class QNet_Agent(object):
    def __init__(self):
       self.nn = ConvNet(1, 256).to(device)
       self.loss_func = nn.MSELoss()
       self.optimizer = optim.Adam(params=self.nn.parameters(), lr=learning_rate)

    def select_action(self, state, epsilon, visited_list):
        random_for_egreedy = torch.rand(1)[0]
        if random_for_egreedy > epsilon:
           with torch.no_grad():
              state = state.to(device)
              action_from_nn = self.nn(state)
              # to avoid repeated chosing, I force already chosed action equals to -1
              action_from_nn[0][visited_list] = -1.0
              action = torch.max(action_from_nn, 1)[1]
              action = action.item()
        else:
            random.seed()
            action = np.random.choice(action_candidate, 1)[0]
    return action

    def optimize(self):
      if (len(memory) < batch_size):
          return
      ### the memory sample is used for experience reply
      state, action, new_state, reward, done = memory.sample(batch_size)
      state = dgl.batch(state).to(device)
      new_state = dgl.batch(new_state).to(device)
      reward = Tensor([reward]).to(device)
      action = LongTensor(action).to(device)
      done = Tensor(done).to(device)
      new_state_values = self.nn(new_state).detach()
      max_new_state_values = torch.max(new_state_values, 1)[0]
      target_value = reward + (1 - done) * gamma * max_new_state_values
      predicted_value = self.nn(state).gather(1, action.unsqueeze(1)).squeeze(1).unsqueeze(0)
      loss = self.loss_func(predicted_value, target_value)
      self.optimizer.zero_grad()
      loss.backward()
      self.optimizer.step()

qnet_agent = QNet_Agent()
for i_episode in range(num_episodes):
     env.reset()
     reward = initial_reward
     status = status
     step = 0
     while True:
        step += 1
        epsilon = calculate_epsilon(frames_total)
        action = qnet_agent.select_action(env.status, epsilon, visited_list)
        new_status, new_reward = env.update(action)
        rewards_difference = new_reward - reward
        reward = new_reward
        already_visit.append(action)
        memory.push(status, action, new_status, rewards_difference, done)
    qnet_agent.optimize()
    status = new_status
    frames_total += 1
    if done or (step > 200):
       print(episode information)

Sorry for the long code, I tried to remove unnecessary part of the codes and only kept the framework. Basically, as you may see, I use every step reward to update the Q-value, and the rewards function is how much reward improved compared to the last step.
My current questions are:

Since it is a combination problem, what I actually care about is the total rewards. The total rewards is the sum of the reward of each step. I am thinking how should I update the model by the final total rewards, rather than the reward of each time step? The most difficult part for me is memory.push(status, action, new_status, rewards, done). Since the model needs to use the status and new status to calculate the Q value, I don't know how to move this line to the end of each episode. I think a better way is to move this line after the if done:, but I don't know how to deal with the variable status and new_status.

When I change the rewards function by rewards = reward, rather than the difference, the model cannot converge. I used 700 training episodes, the final reward keeps fluctuating at the same level until the end. Intuitively, I am feeling directly use the reward is right. But what may cause the reason for this non-converge?

I hope this long question won't confuse you. Please let me know any confusion in the comment.


title: Why the predictions are very off (which causes very large losses)?

body:
I have this very simple resnet18 network that I am trying to train from scratch for task of landmark estimation (I have 4 landmarks):
num_classes = 4 * 2 #4 coordinates X and Y flattened --> 4 of 2D keypoints or landmarks

class Network(nn.Module):
    def __init__(self,num_classes=8):
        super().__init__()
        self.model_name = 'resnet18'
        self.model = models.resnet18()
        self.model.fc = nn.Linear(self.model.fc.in_features, num_classes)

    def forward(self, x):
        x = x.float()
        out = self.model(x)
        return out

For the following piece of code:
network = Network()
network.cuda()

criterion = nn.MSELoss()
optimizer = optim.Adam(network.parameters(), lr=0.0001)

loss_min = np.inf
num_epochs = 1

start_time = time.time()
for epoch in range(1,num_epochs+1):

    loss_train = 0
    loss_test = 0
    running_loss = 0

    network.train()
    print('size of train loader is: ', len(train_loader))

    for step in range(1,len(train_loader)+1):

        batch = next(iter(train_loader))
        images, landmarks = batch['image'], batch['landmarks']
        #RuntimeError: Given groups=1, weight of size [64, 3, 7, 7], expected input[64, 600, 800, 3] to have 3 channels, but got 600 channels instead
        #using permute to fix the above error
        images = images.permute(0,3,1,2)

        images = images.cuda()

        landmarks = landmarks.view(landmarks.size(0),-1).cuda()

        ##images = torchvision.transforms.Normalize(images) #find the args later
        ##landmarks = torchvision.transforms.Normalize(landmarks) #find the args later

        predictions = network(images)

        # clear all the gradients before calculating them
        optimizer.zero_grad()

        print('predictions are: ', predictions.float())
        print('landmarks are: ', landmarks.float())
        # find the loss for the current step
        loss_train_step = criterion(predictions.float(), landmarks.float())

        loss_train_step = loss_train_step.to(torch.float32)
        print("loss_train_step before backward: ", loss_train_step)

        # calculate the gradients
        loss_train_step.backward()

        # update the parameters
        optimizer.step()

        print("loss_train_step after backward: ", loss_train_step)

        loss_train += loss_train_step.item()

        print("loss_train: ", loss_train)
        running_loss = loss_train/step
        print('step: ', step)
        print('running loss: ', running_loss)

        print_overwrite(step, len(train_loader), running_loss, 'train')

    network.eval()
    with torch.no_grad():

        for step in range(1,len(test_loader)+1):

            batch = next(iter(train_loader))
            images, landmarks = batch['image'], batch['landmarks']
            images = images.permute(0,3,1,2)
            images = images.cuda()
            landmarks = landmarks.view(landmarks.size(0),-1).cuda()

            predictions = network(images)

            # find the loss for the current step
            loss_test_step = criterion(predictions, landmarks)

            loss_test += loss_test_step.item()
            running_loss = loss_test/step

            print_overwrite(step, len(test_loader), running_loss, 'Validation')

    loss_train /= len(train_loader)
    loss_test /= len(test_loader)

    print('\n--------------------------------------------------')
    print('Epoch: {}  Train Loss: {:.4f} Valid Loss: {:.4f}'.format(epoch, loss_train, loss_test))
    print('--------------------------------------------------')

    if loss_test < loss_min:
        loss_min = loss_test
        torch.save(network.state_dict(), '../moth_landmarks.pth')
        print("\nMinimum Valid Loss of {:.4f} at epoch {}/{}".format(loss_min, epoch, num_epochs))
        print('Model Saved\n')

print('Training Complete')
print("Total Elapsed Time : {} s".format(time.time()-start_time))

I get this NAN as well as very large MSE losses (showing only for one step):
size of train loader is:  90
predictions are:  tensor([[-0.0380, -0.1871,  0.0729, -0.3570, -0.2153,  0.3066,  1.1273, -0.0558],
        [-0.0316, -0.1876,  0.0317, -0.3613, -0.2333,  0.3023,  1.0940, -0.0665],
        [-0.0700, -0.1882,  0.0068, -0.3201, -0.1884,  0.2953,  1.0516, -0.0567],
        [-0.0844, -0.2009,  0.0573, -0.3166, -0.2597,  0.3127,  1.0343, -0.0573],
        [-0.0486, -0.2333,  0.0535, -0.3245, -0.2310,  0.2818,  1.0590, -0.0716],
        [-0.0240, -0.1989,  0.0572, -0.3135, -0.2435,  0.2912,  1.0612, -0.0560],
        [-0.0942, -0.2439,  0.0277, -0.3147, -0.2368,  0.2978,  1.0110, -0.0874],
        [-0.0356, -0.2285,  0.0064, -0.3179, -0.2432,  0.3083,  1.0300, -0.0756]],
       device='cuda:0', grad_fn=<AddmmBackward>)
landmarks are:  tensor([[501.9200, 240.1600, 691.0000, 358.0000, 295.0000, 294.0000, 488.6482,
         279.6466],
        [495.6300, 246.0600, 692.0000, 235.0000, 286.0000, 242.0000, 464.0000,
         339.0000],
        [488.7100, 240.8900, 613.4007, 218.3425, 281.0000, 220.0000, 415.9966,
         338.4796],
        [502.5721, 245.4983, 640.0000, 131.0000, 360.0000, 143.0000, 542.9840,
         321.8463],
        [505.1393, 246.4364, 700.0000, 306.0000, 303.0000, 294.0000, 569.6925,
         351.8367],
        [501.0900, 244.0100, 724.0000, 251.0000, 302.0000, 276.0000, 504.6415,
         291.7443],
        [495.9500, 244.2800, 608.0000, 127.0000, 323.0000, 166.0000, 491.0000,
         333.0000],
        [490.2500, 241.3400, 699.0000, 304.0000, 398.6197, 313.8339, 429.1374,
         303.8483]], device='cuda:0')
loss_train_step before backward:  tensor(166475.6875, device='cuda:0', grad_fn=<MseLossBackward>)
loss_train_step after backward:  tensor(166475.6875, device='cuda:0', grad_fn=<MseLossBackward>)
loss_train:  166475.6875
step:  1
running loss:  166475.6875

The other thing besides Network I am also suspicious of is the transforms:
Here's the transforms I am using:
transformed_dataset = MothLandmarksDataset(csv_file='moth_gt.csv',
                                           root_dir='.',
                                           transform=transforms.Compose(
                                               [
                                               Rescale(256),
                                               RandomCrop(224),

                                               ToTensor()#,
                                               ##transforms.Normalize(mean = [ 0.485, 0.456, 0.406 ],
                                               ##         std = [ 0.229, 0.224, 0.225 ])
                                               ]
                                                                        )
                                           )

How can I fix it? Also, with having Normalize I do still get same problem (NAN and very large loss values since the predictions are very off).
How can I basically debug why the predictions are very off?


title: predict the position of an image in another image

body:
If one image is a part of another image, then how to compute the accurate location in deep learning way?
Now I could compute this by extracting and matching key points using OpenCV, but I hope to solve it with neural networks.
Any ideas to design the networks and loss functions?
Thanks very much.


title: pytorch narrow is not applicable

body:
How to do slicing in pytorch? I tried narrow and numpy slicing? Both are not working for outputing train data and test data. Is there any suggestion for solving it?
x1 = (max-min)*torch.rand(1, 21, dtype=torch.float) + min
x2 = (max-min)*torch.rand(1, 21, dtype=torch.float) + min
zipped_list = zip(x1, x2)
y = torch.empty(1, 21)
y = [torch.sin(2*x1+2) * torch.cos(0.5*x2)+0.5 for (x1, x2) in zipped_list]

print(y)

train_data = y.narrow(0,1)
test_data = y[11:21]
print(train_data)

Output is
AttributeError: 'list' object has no attribute 'narrow'

However when I do normal slicing, the test data will not be slice properly
train_data = y[0:11]
test_data = y[11:21]

Train Data is: [tensor([-0.0515,  0.4574,  0.5141,  0.4865,  0.9266,  1.0984,  0.5364,  0.7042,
         0.1741, -0.4839,  0.4332,  0.2962,  0.2311,  0.6169,  0.4321,  0.4088,
         0.2443,  0.1982,  0.7978,  0.6651, -0.4453])]
Test Data is: []


title: what is the result from the pytorch source code of LSTM?

body:
https://pytorch.org/docs/stable/_modules/torch/nn/modules/rnn.html#LSTM
I am trying to make customized LSTM cell with pytorch but have some questions related with the source code.
The link of the source code of LSTM in Pytorch is above.
From the source code, at forward function, it returns
return output, self.permute_hidden(hidden, unsorted_indices)
The output is from

I am wondering what the real value of result is.
From the code, result[0] becomes the output value and rest of the result becomes the hidden and cell states.
I tried to print all the values from the cloned git page of Pytorch, got problem with version moudle...
Any help would be very appreciated.
Thank you!


title: PyTorch is giving me a different value for a scalar

body:
When I create a tensor from float using PyTorch, then cast it back to a float, it produces a different result. Why is this, and how can I fix it to return the same value?
num = 0.9
float(torch.tensor(num))

Output:
0.8999999761581421


title: Pytorch does not pad the layers

body:
While making a network I used deconvolution layer :
def deconv3d(cin,cout,k=4,s=2,pad=-1):
    pad = (k - 1) // 2 if pad < 0 else pad
    return nn.Sequential(
        nn.ConvTranspose3d(cin,cout,kernel_size=k,stride=s,padding = pad,bias=False),
        nn.ReLU(inplace=True)
    )

And I adjusted this to networks and it keeps reducing the size of layer.
self.conv_f5 = conv3d(128, 128, k=3, s=1, pad=1)
    self.conv_f6 = deconv3d(128,64,k=3,s=2,pad=1)
    self.conv_f7 = conv3d(64,64,k=3,s=1,pad=1)
    self.conv_f8 = deconv3d(64,32,k=3,s=2,pad=1)
    self.conv_f9 = conv3d(32,32,k=3,s=1,pad=1)

This is the layers I made and the result is :
Cost Volume: torch.Size([2, 128, 100, 120, 160])
fc5 torch.Size([2, 128, 50, 60, 80])
fc6 torch.Size([2, 64, 99, 119, 159])
fc8 torch.Size([2, 32, 197, 237, 317])

I cannot understand why f3 has smaller size than expected and finding how to fix it.
Please tell me how and thank you very much.


title: tf.nn.softmax_cross_entropy_with_logits and sequential applying softmax and cross_entropy give different gradients

body:
I can't fully understand, tf.nn.softmax_cross_entropy_with_logits performs under the hood. I have two versions of computing cross entropy loss after applying softmax.
First:
def tf_softmax_cross_entropy_with_logits(logits, targets):
    return tf.nn.softmax_cross_entropy_with_logits(labels=targets, logits=logits)

Second:
@tf.function
def cross_entropy(x, y, epsilon = 1e-9):
    return -1 * tf.reduce_sum(y * tf.math.log(x + epsilon), -1)

def tf_softmax_cross_entropy_with_logits_v1(logits, targets):
  logits_sf = tf.nn.softmax(
    logits, axis=2
  )
  return cross_entropy(logits_sf, tf.dtypes.cast(targets, dtype=tf.dtypes.float32))

On the same input data
logits:  <tf.Variable 'logits:0' shape=(2, 3, 4) dtype=float32, numpy=
array([[[0.81269646, 0.07857466, 0.8916855 , 0.16925514],
        [0.06311357, 0.54531074, 0.5037316 , 0.9248222 ],
        [0.66955376, 0.9281193 , 0.12239242, 0.8532245 ]],

       [[0.90477383, 0.7104306 , 0.40681756, 0.5755513 ],
        [0.8547678 , 0.59606934, 0.77619946, 0.97301054],
        [0.06244731, 0.33562684, 0.22166848, 0.32035887]]], dtype=float32)>
targets:  <tf.Variable 'targets:0' shape=(2, 3, 4) dtype=int64, numpy=
array([[[1, 0, 0, 1],
        [0, 0, 0, 0],
        [2, 0, 1, 1]],

       [[0, 0, 1, 2],
        [2, 2, 0, 0],
        [0, 2, 2, 2]]])>

They produce the same result, but gradients for logits are different:
Applying first version
with tf.GradientTape() as g:
  g.watch(logits)
  result = tf_softmax_cross_entropy_with_logits(logits, targets)
  print("Result: ", result)
print("Gradient: ", g.gradient(result, logits))

gives:
Result:  tf.Tensor(
[[2.8982296 0.        5.9832497]
 [4.5992084 5.8805046 8.007221 ]], shape=(2, 3), dtype=float32)
Gradient:  tf.Tensor(
[[[-0.67612386  0.15543681  0.3504963  -0.82980925]
  [ 0.15289354  0.24763085  0.23754568  0.3619299 ]
  [-1.7546182   0.31778693 -0.85802454 -0.7051442 ]]

 [[ 0.31739688  0.26133674 -0.8070952  -1.7716384 ]
  [-1.7383577  -1.7979975   0.24187233  0.29448295]
  [ 0.20916311 -1.7251312  -1.754736   -1.729296  ]]], shape=(2, 3, 4), dtype=float32)

Applying second version
with tf.GradientTape() as g:
  g.watch(logits)
  result = tf_softmax_cross_entropy_with_logits_v1(logits, targets)
  print(result)
print(g.gradient(result, logits))

gives:
tf.Tensor(
[[ 2.8982296 -0.         5.9832497]
 [ 4.5992084  5.8805046  8.007221 ]], shape=(2, 3), dtype=float32)
tf.Tensor(
[[[-0.3522477   0.31087363  0.7009926  -0.65961856]
  [-0.         -0.         -0.         -0.        ]
  [-1.0184727   1.2711477  -0.43209818  0.17942311]]

 [[ 0.95219064  0.78401023 -0.42128566 -1.3149152 ]
  [-0.95343083 -1.1919901   0.96748936  1.1779318 ]
  [ 1.2549787  -0.35078713 -0.52841544 -0.37577602]]], shape=(2, 3, 4), dtype=float32)

Can anybody help me and give an answer what is the reason of this difference? Thanks a lot


title: No model named torch

body:
Dear fellow humans and others,
i have a question about using and importing pytorch into my editor.
I installed pytorch through Conda and in my terminal editor I can use pytorch,see image but not in Visual studio code or Pycharm!
No model named torch error code in VSC
Why is that?
Very sorry for the basic question and thank you for sparing some time!
Should I use pip?


title: Error(s) in loading state_dict for ModuleList:

body:
I have problem to load a state to a new model using  load_state_dict() in pysyft.
I created a model with SyNet class and i want to pass the parameters after I trained it in a new model.
The procedure I followed is :
model = SyNet( categorical_embedding_sizes, numerical_data.shape[1], 2, [200,100,50], p=0.4 ,torch_ref= th)

#train

state_1=model.state_dict()

for layer in state_1:
  state_1[layer]=state_1[layer]/2

So I want to pass the parameters multiplied by 0.5
Βut when I pass them to a new model I got the following message .

This is the source code of pysyft module.py if someone knows the source of the error


title: CycleGAN training going slowly in PyTorch even on GPU

body:
I am training my implementation of CycleGAN on a colab notebook using the GPU but the training seems to be quite slow, I'm training with a batch size of 1 and this is my training code:
def train():

    generator_A2B = Generator(3).train().to(device)
    generator_B2A = Generator(3).train().to(device)
    discriminator_A = Discriminator(3).to(device)
    discriminator_B = Discriminator(3).to(device)

    L2Loss = nn.MSELoss().to(device)
    L1Loss = nn.L1Loss().to(device)
    train_step = 0

    optimizer_G_A2B = optim.Adam(generator_A2B.parameters(), lr=lr, betas=(0.5, 0.999))
    optimizer_G_B2A = optim.Adam(generator_B2A.parameters(), lr=lr, betas=(0.5, 0.999))
    optimizer_D_A = optim.Adam(discriminator_A.parameters(), lr=lr, betas=(0.5, 0.999))
    optimizer_D_B = optim.Adam(discriminator_B.parameters(), lr=lr, betas=(0.5, 0.999))

    transforms_list = transforms.Compose([transforms.ToTensor(),
                                          transforms.Normalize(mean=[0.5, 0.5, 0.5], std=[0.5, 0.5, 0.5])])

    dataset = CycleGANData(data_root_x, data_root_y, transforms_list=transforms_list)

    data_loader = DataLoader(dataset, batch_size=batch_size, shuffle=True)
    iter_nums = len(data_loader)
    print(f"Total Iterations per Epoch {iter_nums}")
    for epoch in range(1, num_epochs + 1):

            for i, (img_a, img_b) in enumerate(data_loader):

                for Disc in range(1):

                    train_step += 1

                    discriminator_A.zero_grad()
                    discriminator_B.zero_grad()

                    realA = img_a.to(device)
                    realB = img_b.to(device)
                    fakeB = generator_A2B(realA)
                    fakeA = generator_B2A(realB)

                    D_A_real = discriminator_A(realA)
                    D_A_fake = discriminator_A(fakeA)
                    D_B_real = discriminator_B(realB)
                    D_B_fake = discriminator_B(fakeB)

                    y_real = torch.ones_like(D_A_real)
                    y_fake = -torch.ones_like(D_A_real)

                    D_A_real_loss = L2Loss(D_A_real, y_real)
                    D_B_real_loss = L2Loss(D_B_real, y_real)
                    D_A_fake_loss = L2Loss(D_A_fake, y_fake)
                    D_B_fake_loss = L2Loss(D_B_fake, y_fake)

                    D_A_GANloss = (D_A_real_loss + D_A_fake_loss)/2
                    D_B_GANloss = (D_B_real_loss + D_B_fake_loss)/2

                    D_A_GANloss.backward(retain_graph=True)
                    D_B_GANloss.backward(retain_graph=True)
                    optimizer_D_A.step()
                    optimizer_D_B.step()

                    if train_step % 1 == 0:

                        generator_A2B.zero_grad()
                        generator_B2A.zero_grad()

                        fakeB = generator_A2B(realA)
                        fakeA = generator_B2A(realB)

                        D_A_fake = discriminator_A(fakeA)
                        D_B_fkae = discriminator_B(fakeB)

                        G_A2B_GANLoss = L2Loss(D_B_fake, y_real)
                        G_B2A_GANLoss = L2Loss(D_A_fake, y_real)

                        B_idt = generator_A2B(realB)
                        A_idt = generator_B2A(realA)

                        G_A2B_iden_loss = L1Loss(B_idt, realB)
                        G_B2A_iden_loss = L1Loss(A_idt, realA)

                        A_cycle = generator_B2A(generator_A2B(realA))
                        B_cycle = generator_A2B(generator_B2A(realB))
                        A_cycle_loss = L1Loss(A_cycle, realA)
                        B_cycle_loss = L1Loss(B_cycle, realB)

                        G_loss = (G_A2B_GANLoss + G_B2A_GANLoss) + lambda_identity * (G_A2B_iden_loss + G_B2A_iden_loss) + lambda_cyc * (A_cycle_loss + B_cycle_loss)
                        G_loss.backward()
                        optimizer_G_B2A.step()
                        optimizer_G_A2B.step()

                print(f"[Epoch - {epoch}] [Step - {i}/{iter_nums}] [Generator Loss - {G_loss}] [Discriminator A Loss - {D_A_GANloss}] [Discriminator B Loss - {D_B_GANloss}]")

train()

Is there something I could fix to speed up training or is there an immediate problem to fix the training time, Thanks in advance.


title: parallelize nested for loop in pytorch

body:
I am working with a pytorch based code. Part of my code has 4 nested for loops. Basically, it is finding patches of image and estimating similarity between two patches. Since the parameters are torch elements, none of the python libraries like joblib are working. I am a newbie to pycuda and I kindly seek help to parallelize this code. At present, this is slow computation that takes more than 1.5 seconds.  Here is part of my code.
import torch
import numpy as np
import cv2
import time

def mul_val(a,b,ax=None):
    return torch.mean(((b-a)/(b+0.01))**2)

def np_to_torch(img_np):
    return torch.from_numpy(img_np)[None,None, :]

def main_fn(a,alpha = 0.25):
    start = time.time()
    final_u = torch.zeros(w,h)
    frame1 = a
    for y1 in range(w):
        i = y1*grid_size
        for x1 in range(h):

            j = x1*grid_size  ## batch, channel, width, height
            block1 = frame1[:,:,i:i+grid_size, j:j+grid_size]
            corr_list = []
            for y2 in range(y1-radius,y1+radius+1):
                    i2 = y2*grid_size
                    if not (0 <= y2 < h):
                        continue
                    for x2 in range(x1-radius,x1+radius+1):
                        j2 = x2*grid_size
                        if not (0 <= x2 < w):
                            continue

                        block2 = frame1[:,:,i2:i2+grid_size, j2:j2+grid_size]
                        if not (block1.shape == block2.shape):
                            continue
                        corr = mul_val(block1, block2)
                        corr_list.append(corr)

            corr_list=sorted(corr_list, reverse = True)
            del corr_list[20:]
            uncorr = 1.0 - (sum(corr_list)/20.0)
            final_u[y1,x1] = torch.mul(a[:,:,i,j],uncorr)

    new_val = torch.norm(torch.sub(1.0,final_u))
    final_value = alpha*new_val
    print("Time taken is ..", time.time()-start)
    return final_value

grid_size = 9
radius = 3
im1 = cv2.imread("1.png", 0)
img = np.asarray(im1)
img = img.astype(np.float64)
img_torch = np_to_torch(img)   ### made this as a torch element intentionally.
frame_width = img.shape[0]
frame_height = img.shape[1]
h = int(frame_height//grid_size)
w = int(frame_width//grid_size)
main_fn(img_torch)

The above code opens an image and converts to torch element just to replicate the parameters as torch elements.
Any help in this regard is highly appreciated.
Thank you


title: Pytorch autoencoder print accuracy positional argument error

body:
I got this autoencoder
class AE(nn.Module):
    def __init__(self, **kwargs):
        super(AE,self).__init__()
        self.encoder_hidden_layer = nn.Linear(
            in_features=kwargs["input_shape"], out_features=128
        )
        self.encoder_output_layer = nn.Linear(
            in_features=128, out_features=128
        )
        self.decoder_hidden_layer = nn.Linear(
            in_features=128, out_features=128
        )
        self.decoder_output_layer = nn.Linear(
            in_features=128, out_features=kwargs["input_shape"]
        )

    def forward(self, features):
        activation = self.encoder_hidden_layer(features)
        activation = torch.relu(activation)
        code = self.encoder_output_layer(activation)
        code = torch.sigmoid(code)
        activation = self.decoder_hidden_layer(code)
        activation = torch.relu(activation)
        activation = self.decoder_output_layer(activation)
        reconstructed = torch.sigmoid(activation)
        return reconstructed

and I am trying to build a accuracy calculator
with torch.no_grad():
    for batch_features in test_loader:
        batch_features = batch_features[0]
        test_examples = batch_features.view(-1, 784)
        reconstruction = model(test_examples)

        images, labels = data[0].to(device), data[1].to(device)
        print(labels)

        outputs_auto = AE(images.to(device))
        _, predicted_auto = torch.max(outputs_auto.data, 1)
        total += labels.size(0)
        correct += (predicted_auto == labels).sum().item()
        break
    print('Accuracy of the network on the 10000 test images: %d %%' % (
    100 * correct / total))

but it shows this type of error. I have tried some modification of the code but it failed. I am new to autoencoder
---> 21         outputs_auto = AE(images.to(device))
     22         _, predicted_auto = torch.max(outputs_auto.data, 1)
     23         total += labels.size(0)
TypeError: __init__() takes 1 positional argument but 2 were given

I just want to print accuracy in percent. If you need more code please ask to edit this question


title: Function with **kwargs on a class returns "unexpected argument"

body:
I'm trying to follow this article https://towardsdatascience.com/residual-network-implementing-resnet-a7da63c7b278 but i get unexpected argument.
def conv_bn(in_channels, out_channels, conv, *args, **kwargs):
    return nn.Sequential(conv(in_channels, out_channels, *args, **kwargs), nn.BatchNorm1d(out_channels))

Running it with this function i get the right response
conv_bn(3, 3, nn.Conv1d, kernel_size=4, bias=False)
Sequential(
  (0): Conv1d(3, 3, kernel_size=(4,), stride=(1,), bias=False)
  (1): BatchNorm1d(3, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)

But in the class i get the error:
class ResNetBasicBlock(ResNetResidualBlock):
    """
    Modificado para conv1d
    """
    expansion = 1
    def __init__(self, in_channels, out_channels, *args, **kwargs):
        super().__init__(in_channels, out_channels, *args, **kwargs)
        self.blocks = nn.Sequential(
            conv_bn(self.in_channels, self.out_channels, conv=self.conv, bias=False,stride=self.downsampling),
            activation_func(self.activation),
            conv_bn(self.out_channels, self.expanded_channels, conv=self.conv, bias=False),
        )
dummy = torch.ones((1, 32, 224, 224))

block = ResNetBasicBlock(32, 64)
block(dummy).shape
print(block)

Error
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-83-51655749b6e2> in <module>
      1 dummy = torch.ones((1, 32, 224, 224))
      2
----> 3 block = ResNetBasicBlock(32, 64)
      4 block(dummy).shape
      5 print(block)

<ipython-input-82-67a486dfad55> in __init__(self, in_channels, out_channels, *args, **kwargs)
      7         super().__init__(in_channels, out_channels, *args, **kwargs)
      8         self.blocks = nn.Sequential(
----> 9             conv_bn(self.in_channels, self.out_channels, conv=self.conv, bias=False,stride=self.downsampling),
     10             activation_func(self.activation),
     11             conv_bn(self.out_channels, self.expanded_channels, conv=self.conv, bias=False),

<ipython-input-80-0ac97ae302a5> in conv_bn(in_channels, out_channels, conv, *args, **kwargs)
      1 def conv_bn(in_channels, out_channels, conv, *args, **kwargs):
----> 2     return nn.Sequential(conv(in_channels, out_channels, *args, **kwargs), nn.BatchNorm1d(out_channels))

c:\users\carlos\appdata\local\programs\python\python39\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

TypeError: forward() got an unexpected keyword argument 'bias'


title: Training loss is not changing at all while training model

body:
I’m trying to solve a VQA classification problem. my training loss is not changing at all while training the model.
I put in comment the CNN model and try to run it with the text only, but still, no change in loss value.
I pass through those models:
class question_lstm(nn.Module):

    def __init__(self, input_dim, emb_dim, hid_dim, n_layers, dropout, output_dim, que_size):
        super(question_lstm, self).__init__()
        self.hid_dim = hid_dim
        self.n_layers = n_layers
        self.embedding = nn.Embedding(input_dim, emb_dim)
        self.tanh = nn.Tanh()
        self.lstm = nn.LSTM(emb_dim, hid_dim, n_layers, dropout = dropout)
        self.dropout = nn.Dropout(dropout)
        #self.fc1=nn.Linear(n_layers*hid_dim,que_size)
        self.fc1=nn.Linear(n_layers*output_dim,que_size)

    def forward(self, question):
      emb_question=self.embedding(question) #(batchsize, input_dim, emb_dim=256)
      emb_question=self.dropout(emb_question)
      emb_question=self.tanh(emb_question)
      emb_question = emb_question.transpose(0, 1)  #(input_dim, batchsize, emb_dim)
      output, (hidden, cell) = self.lstm(emb_question)
      qu_feature = torch.cat((hidden, cell), dim=2)
      qu_feature = qu_feature.transpose(0, 1) #(batchsize=100, num_layer=2, hid_dim=2048)
      question_output =self.fc1(qu_feature)
      return question_output

class vqamodel(nn.Module):

  def __init__(self, output_dim,input_dim, emb_dim, hid_dim, n_layers, dropout, answer_len, que_size,):

    super(vqamodel,self).__init__()
    #self.image=img_CNN(img_size,image_feature)
    self.question=question_lstm(input_dim, emb_dim, hid_dim, n_layers, dropout,output_dim,que_size)
    self.tanh=nn.Tanh()
    self.relu=nn.ReLU()
    self.dropout=nn.Dropout(dropout)
    self.fc1=nn.Linear(que_size,output_dim)
    self.fc2=nn.Linear(output_dim,answer_len)

  def forward(self, image, question):
    question_emb=self.question(question)
    combine =question_emb   #*img_emb
    out_feature=self.fc1(combine) #(batchsize=100, output_dim=2048)
    out_feature=self.relu(out_feature)
    out_feature=self.dropout(out_feature)
    out_feature=self.fc2(out_feature) #(batchsize=100, answer_len=1000)
    return (out_feature)

I’m using cross entropy loss and Adam:
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(vqa_model.parameters(),lr=0.001)
scheduler = optim.lr_scheduler.StepLR(optimizer, step_size=7, gamma=0.1)

any idea what can cause this constant loss value?
the train loop:
def train(model,criterion,optimizer,scheduler):

  start_time = time.time() #the time we start the train

  for epoch in range(num_epochs):
      train_loss = 0
      #test_loss = 0
      train_correct = 0
      #test_correct = 0
      vqa_model.train()
      for  i,sample in enumerate(train_VQAdataset_loader):
              #image = sample['image'].to(device=device)
              question = sample['question'].to(torch.int64).to(device=device)
              label = sample['answer'].to(device=device)

              output = vqa_model(image, question) # forward
              loss = criterion(output, label)
              optimizer.zero_grad() # Zero the gradients
              loss.backward() # backprop
              optimizer.step() # Update weights
              scheduler.step()

          # Statitcs
              train_loss += loss.item() # save the loss for the entire epoch

_, predictions = torch.max(output, 1)
train_correct += (predictions == label).sum() #number of success - cumulative
train_losses.append(train_loss / len(train_VQAdataset_loader))


title: How to create a trainable customized convolutional layer?

body:
I am wondering how to customize a convolutional layer, where the convolutional filter is determined by some functions instead of independent parameters. I made a toy example below. a and b are parameters. A 2X2 filter is created as torch.tensor([[[[a+b, a-b], [a-b,a+b]]]]). As we can see, it is determined by a and b instead of four independent parameters. In my experiment, the network is simply composed of one such filter. I made a simple learning task to test my code, where there is only one training instance: input: torch.tensor([[[[1.0, 0],[0,0]]]]), true output: torch.tensor([[2.0]]). I attach my code and experiment below.
class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        # in my later experiment there might be multiple parameters,
        # so I made the generate_parameters() function
        a, b = self.generate_parameters()
        self.conv = nn.Conv2d(1, 1, 2, bias=False)

        kernel = self.generate_kernel(a, b)
        self.conv.weight = nn.Parameter(kernel)

    def forward(self, x):
        x = self.conv(x)
        return x

    def generate_parameters(self):
        a = nn.Parameter(torch.zeros(1, 1))
        b = nn.Parameter(torch.zeros(1, 1))
        return a, b

    def generate_kernel(self, a, b):
        return torch.tensor([[[[a+b, a-b], [a-b,a+b]]]])

model = MyModel()
criterion = nn.MSELoss()
optimizer = optim.SGD(model.parameters(), lr=0.1)

inputs, labels = torch.tensor([[[[1.0, 0],[0,0]]]]), torch.tensor([[2.0]])

for epoch in range(10):  # loop over the dataset multiple times
    for param in model.parameters():
        print(param.data, param.size())

    # zero the parameter gradients
    optimizer.zero_grad()

    # forward + backward + optimize
    outputs = model(inputs)
    loss = criterion(outputs, labels)
    loss.backward()
    optimizer.step()

experiment result:
epoch: 0 tensor([[[[0., 0.],
          [0., 0.]]]]) torch.Size([1, 1, 2, 2])
epoch: 1 tensor([[[[0.4000, 0.0000],
          [0.0000, 0.0000]]]]) torch.Size([1, 1, 2, 2])
epoch: 2 tensor([[[[0.7200, 0.0000],
          [0.0000, 0.0000]]]]) torch.Size([1, 1, 2, 2])
epoch: 3 tensor([[[[0.9760, 0.0000],
          [0.0000, 0.0000]]]]) torch.Size([1, 1, 2, 2])
epoch: 4 tensor([[[[1.1808, 0.0000],
          [0.0000, 0.0000]]]]) torch.Size([1, 1, 2, 2])
epoch: 5 tensor([[[[1.3446, 0.0000],
          [0.0000, 0.0000]]]]) torch.Size([1, 1, 2, 2])
epoch: 6 tensor([[[[1.4757, 0.0000],
          [0.0000, 0.0000]]]]) torch.Size([1, 1, 2, 2])
epoch: 7 tensor([[[[1.5806, 0.0000],
          [0.0000, 0.0000]]]]) torch.Size([1, 1, 2, 2])
epoch: 8 tensor([[[[1.6645, 0.0000],
          [0.0000, 0.0000]]]]) torch.Size([1, 1, 2, 2])
epoch: 9 tensor([[[[1.7316, 0.0000],
          [0.0000, 0.0000]]]]) torch.Size([1, 1, 2, 2])

As we can see, the trained filter isn’t as we expect (which should have been a symmetric 2X2 matrix). Ideally I thought the model should have only two parameters a and b. Any advice to fix my code or experiment? Thank you!


title: Input to an RNN produces - RuntimeError: Expected object of scalar type Double but got scalar type Float

body:
I'm trying to get a RNN to predict the next value in a sequence. I think my problem is getting the input tensor to be in the right datatype and/or correct dimensions. How can I resolve the error message below?
import torch
import torch.nn as nn

import seaborn as sns
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.metrics import mean_squared_error

The data I am using is imported from a CSV. Instead of using the actual data in this post, I've created dummy data that is the same data type (np.float64), as the data I'm using. I think the data's type might be the source of the problem. I've tried altering it but with no success.
# create random data that mimics actual data I am using

sampl = np.random.uniform(low=-3, high=3, size=(50,))
sampl.astype(np.float64)

# define seq length
seq_length = 10

data_x = sampl.copy()

# number of sequences w/i data
n_seq = len(data_x)//seq_length

# only keep enough data to make full sequences
data_x = data_x[:n_seq * seq_length]

# Number of input data we can have
nbr_input = len(data_x[:-seq_length])

# create feature data and target data
features = []
targets = []

# create data one seq at a time
for i in range(nbr_input):
    features.append(data_x[i:i+seq_length])
    targets.append(data_x[i+seq_length])

# convert data into pytorch tensors
feature_tensor = torch.from_numpy(np.array(features))
target_tensor  = torch.from_numpy(np.array(targets))

# add a last dimension so that data is (batch_size, seq_length, features)
feature_tensor = feature_tensor.unsqueeze(-1)
target_tensor  = target_tensor.unsqueeze(-1)

# UPDATE
# convert tensors to scalar type double
feature_tensor = feature_tensor.double()
target_tensor = target_tensor.double()

# DEFINE THE RNN

class RNN(nn.Module):
    def __init__(self, input_size, output_size, hidden_dim, n_layers):
        super(RNN, self).__init__()
        self.hidden_dim=hidden_dim
        self.rnn = nn.RNN(input_size, hidden_dim, n_layers, batch_first=True)
        self.fc = nn.Linear(hidden_dim, output_size)

    def forward(self, x, hidden):
        batch_size = x.size(0)
        r_out, hidden = self.rnn(x, hidden)
        r_out = r_out.view(-1, self.hidden_dim)
        output = self.fc(r_out)
        return output, hidden

# test that dimensions of the model are as expected
test_rnn = RNN(input_size=1, output_size=1, hidden_dim=32, n_layers=2)

print(test_rnn)

print('Input size: ', feature_tensor.size())

# test out rnn sizes

test_out, test_h = test_rnn(feature_tensor, None)

print('Output size: ', test_out.size())
print('Hidden state size: ', test_h.size())

Input size:  torch.Size([40, 10, 1])

RuntimeError: Expected object of scalar type Double but got scalar type Float for argument #3 'mat2' in call to _th_addmm_out


title: How to introduce a loss to get two matrices similar

body:
I am training a neural network and I want two matrices to be similar (the covariance matrices). My naive approach was to use a loss based on the difference, such as the L1 loss. But this also forced the matrices to become small, which is not what I want.
Does anyone have an idea here?
Thanks a lot!


title: Loss function Neuronal Network Binary classification averaged labels

body:
Background information:
I am trying to build a Neuronal Network to predict a probability.
Unfortunately I only have Class labels, (Specific event happens or not) which are very stochastic in nature. Meaning, a seemingly very big part is random or at least not correlated in the featuredata.
It is also a time series data, but for different reasons I treat every timestamp as a different datapoint with time encoded as input label.
My labels are 0 or 1. When I downsample my dataset by using the following line:
input_data = block_reduce(input_data, block_size=(4, 1, 1), func=np.mean)

This results in some labels being of value: 0.25, 0.5 or 0.75
Up to now I only downsampled by picking every nth line and used BCELoss as my loss function.
However I realized the following:
>>> torch_label = torch.tensor([[1., 0.5], [1., 0.5]])
>>> torch_pred = torch.tensor([[1., 0.5], [1., 0.5]])
>>> loss_func = torch.nn.BCELoss()
>>> print(loss_func(torch_pred,torch_label))
tensor(0.3466)

Meaning that it does not behave the way I want it to. I tried MSELoss, but it seams wrong since I still would say i have a classification problem.
Also I actually care about the probability of the event. I currently use SGD, since adam did not work for me, since its to stachastic and a 4 layer NN. Which I optimized using grid search. I would also appreciate any tips.


title: Why using GPT2Tokenizer convert Arabic characters to symbols?

body:
I am trying to use GPT2 for Arabic text classification task as follows:
    tokenizer = GPT2Tokenizer.from_pretrained(model_path)
    model = GPT2ForSequenceClassification.from_pretrained(model_path,
                                                          num_labels=len(lab2ind))

However, when I use the tokenizer it converts the Arabic characters to symbols like this
'ĠÙĥØªÙĬØ±'


title: how to convert the following model from RGB into 1 chanel (gray only)

body:
I need to change the following code into 1 color (gray) instead of RGB, so how can I do it ?
when i tried to train my network it worked, but when i started the prediction i got some errors like : RuntimeError: Given groups=1, weight of size [12, 3, 3, 3], expected input[1, 1, 40, 24] to have 3 channels, but got 1 channels instead, so i need to change something in this code (important note : the dataset that i used is gray , and i did the train as RGB which is wrong) , also i know if changed the pictures of prediction with RGB pic it will works , but i only have gray pictures , so i need to edit the model
#CNN network
class ConvNet(nn.Module):
  def __init__(self, num_classes = 12):
    super(ConvNet,self).__init__()

    #output size after convolution filter , w = 24 , f = 3 , P = s = 1 , when we calculate it = 24
    #((w-f+2P/s)) +1
    #input shape = (3,3,40,24) , 3 => batch_size , 3 => RGB color , 40*24 pixels

    self.conv1 = nn.Conv2d(in_channels = 3,out_channels = 12, kernel_size= 3, stride = 1, padding = 1)
    #Shape = (3,12,40,24)
    self.bn1 = nn.BatchNorm2d(num_features = 12)
    #Shape = (3,12,40,24)
    self.relu1 = nn.ReLU()
    #Shape = (3,12,40,24)

    self.pool = nn.MaxPool2d(kernel_size = 2)
    #reduce the image size by factor 2
    #Shape = (3,12,20,12)

    self.conv2 = nn.Conv2d(in_channels = 12,out_channels = 20, kernel_size= 3, stride = 1, padding = 1)
    #Shape = (3,20,20,12)
    self.relu2 = nn.ReLU()
    #Shape = (3,20,20,12)

    self.conv3 = nn.Conv2d(in_channels = 20,out_channels = 32, kernel_size= 3, stride = 1, padding = 1)
    #Shape = (3,32,20,12)
    self.bn3 = nn.BatchNorm2d(num_features = 32)
    #Shape = (3,32,20,12)
    self.relu3 = nn.ReLU()
    #Shape = (3,32,20,12)

    self.fc = nn.Linear(in_features = 32*20*12, out_features = num_classes)

    #Feed forward Function
def forward(self,input):
    output = self.conv1(input)
    output = self.bn1(output)
    output = self.relu1(output)

    output = self.pool(output)

    output = self.conv2(output)
    output = self.relu2(output)

    output = self.conv3(output)
    output = self.bn3(output)
    output = self.relu3(output)

        #Above output will be in matrix form, with shape of (3,32,20,12)

    output = output.view(-1,32*20*12)
    output = self.fc(output)
    return output


title: Get a 10x10 patch from a 100x100 pytorch tensor with torus style wrap around the boundries

body:
How can I get a 10x10 patch from a 100x100 pytorch tensor, with the added constraint that if a patch would go outside the boundaries of the array, then it wraps around the edges (as if the array was a torus, with the top joined to the bottom, and the left joined to the right)?
I wrote this code that will do the job, I'm looking for something more elegant, efficient and clear:
def shift_matrix(a, distances) -> Tensor:
  x, y = distances
  a = torch.cat((a[x:], a[0:x]), dim=0)
  a = torch.cat((a[:, y:], a[:, :y]), dim=1)
  return a

def randomly_shift_matrix(a) -> Tensor:
  return shift_matrix(a, np.random.randint(low = 0, high = a.size()))

def random_patch(a, size) -> Tensor:
  full_shifted_matrix = randomly_shift_matrix(a)
  return full_shifted_matrix[0:size[0], 0:size[1]]

I feel like something with negative index slices should work.  I haven't found it though.
You can see the code in google colab here.


title: How to keep torch models in memory while waiting for requests to process data

body: I have several models for image processing. GPU memory is large enough to hold all of them at once. Currently my flask server is waiting for a request to process photos. When the request arrives I load each model one by one and process the image. This is very slow because of the load/unload step. How to tell torch to keep models in memory and call model(image) on loaded models?


title: How to detect memory usage by a pytorch network

body:
I'm building a GAN model, and I'd like to know how much memory it uses on my GPU, so I run os.system("nvidia-smi") immediately after I created my networks, but the result I get was mysterious. Here is my code:
ngpu=1
nc = 1
nz = 20  # noOfChannel of Z
gen_layers = 4
dis_layers = 4
device = torch.device("cuda:0" if (torch.cuda.is_available()) else "cpu")
gen_filters = [256, 128, 64, nc]
dis_filters = [64, 128, 256, 1]

class Generator(nn.Module):
    def __init__(self, ngpu):
        super(Generator, self).__init__()
        self.ngpu = ngpu
        self.main = nn.Sequential(

            # Hard code "same" padding.
            nn.ConvTranspose2d(nz, gen_filters[0], gen_kernel_size, 2, 3, bias=False),
            nn.BatchNorm2d(gen_filters[0]),
            nn.ReLU(True),

            nn.ConvTranspose2d(gen_filters[0], gen_filters[1], gen_kernel_size, 2, 3, bias=False),
            nn.BatchNorm2d(gen_filters[1]),
            nn.ReLU(True),

            nn.ConvTranspose2d(gen_filters[1], gen_filters[2], gen_kernel_size, 2, 3, bias=False),
            nn.BatchNorm2d(gen_filters[2]),
            nn.ReLU(True),

            nn.ConvTranspose2d(gen_filters[2], gen_filters[3], gen_kernel_size, 2, 3, bias=False),
            nn.Tanh()
        )

    def forward(self, input):
        if input.is_cuda and self.ngpu > 1:
            output = nn.parallel.data_parallel(self.main, input, range(self.ngpu))
        else:
            output = self.main(input)
        return output

class Discriminator(nn.Module):
    def __init__(self, ngpu):
        super(Discriminator, self).__init__()
        self.ngpu = ngpu
        self.main = nn.Sequential(
            # Hard code "same" padding
            nn.Conv2d(nc, dis_filters[0], dis_kernel_size, 2, 1),
            nn.BatchNorm2d(dis_filters[0]),
            nn.LeakyReLU(inplace=True),

            nn.Conv2d(dis_filters[0], dis_filters[1], dis_kernel_size, 2, 1),
            nn.BatchNorm2d(dis_filters[1]),
            nn.LeakyReLU(inplace=True),

            nn.Conv2d(dis_filters[1], dis_filters[2], dis_kernel_size, 2, 1),
            nn.BatchNorm2d(dis_filters[2]),
            nn.LeakyReLU(inplace=True),

            nn.Conv2d(dis_filters[2], dis_filters[3], dis_kernel_size, 2, 1),
            nn.Sigmoid()
        )

    def forward(self, input):
        if input.is_cuda and self.ngpu > 1:
            output = nn.parallel.data_parallel(self.main, input, range(self.ngpu))
        else:
            output = self.main(input)

        return output.view(-1, 1).squeeze(1)

# The main method
print("Before doing anything:")
os.system("nvidia-smi")

netG = Generator(ngpu).to(device)
print("Gen created")
os.system("nvidia-smi")

netD = Discriminator(ngpu).to(device)
print("Dis created")
os.system("nvidia-smi")

The result is:

As you can see, my generator used 427MiB memory on my GPU. However, after I created my discriminator, memory used in GPU is still 427MiB, how is this possible?


title: Adding custom weights to training data in PyTorch

body: Is it possible to add custom weights to the training instances in PyTorch? More explicitly, I'd like to add a custom weight for every row in my dataset. By default, the weights are 1, which means every data is equally important for my model.


title: How to decipher these 2 lines of code? What does it mean?

body:
This is the code:
real_idx = torch.rand(len(real)) > 0.2
real_selected = real[real_idx][: int(len(real)*p_real)]

p_real is actually 0.2
We're supposed to mix 2 tensors, namely the real and the fake tensor.


title: How to transform tensor size from [a, b] to [a, b, k] in PyTorch

body:
I want to add one extra dimension to a tensor and set the value of this dimension as a specific value.
For examples,
print(a.size)
torch.Size([10, 5])

# after tranformation and set a to b
print(b.size)
torch.Size([10, 5, 1])

# after transformation and set a to c
print(c.size)
torch.Size([10, 2, 5])

Thansk in advances.


title: TorchServe MAR multiple Python files per model

body:
Is there a way to split a model between multiple files before calling MAR?
For example, if I have the file base_model.py:
import torch
from torch import nn

class BaseModel(nn.Module):
    ...

And a model that inherits from BaseModel:
from base_model import BaseModel

class FullModel(BaseModel):
    ...

The same goes for the handler. I don't know how to split the code between multiple files.


title: Purpose of '*' in front of Python3 function

body:
I saw the code for a ResNet CNN in Python3 and PyTorch here as follows:
def resnet_block(input_channels, num_channels, num_residuals,
                 first_block=False):
    blk = []
    for i in range(num_residuals):
        if i == 0 and not first_block:
            blk.append(Residual(input_channels, num_channels,
                                use_1x1conv=True, strides=2))
        else:
            blk.append(Residual(num_channels, num_channels))
    return blk

To add the modules, the following code is used-
b2 = nn.Sequential(*resnet_block(64, 64, 2, first_block=True))
b3 = nn.Sequential(*resnet_block(64, 128, 2))
b4 = nn.Sequential(*resnet_block(128, 256, 2))
b5 = nn.Sequential(*resnet_block(256, 512, 2))

What does "*resnet_block()" mean/do?


title: Pytorch only sees 15GB memory when the device should have more

body:
I just kicked off a g4dn.8xlarge instance on AWS. According to their website, this instance should have 128GB memory, as shown in the screenshot below:

However, I notice my model keeps running out of memory. When I investigated, I see this:
>>> print(torch.cuda.get_device_properties('cuda:0'))
_CudaDeviceProperties(name='Tesla T4', major=7, minor=5, total_memory=15109MB, multi_processor_count=40)

It turns out the device can only use about 15GB of memory! Could someone please explain why this might be? Do I need to use a different AWS instance to get more memory, or is PyTorch just not able to use the rest of the memory that's available?


title: Using Apex AMP with PyTorch optimizers causes Attribute Error

body:
I'm facing some issues when I'm trying to use PyTorch optimizers with Apex AMP. My environment is:

OS: Ubuntu 18.04.5
Python: 3.8.5
PyTorch: 1.7.1
CUDA: 11.0
Apex: 0.9.10.dev0 (Build: pypi_0 / Channel: pypi)
Transformers: 4.3.3

You can reproduce my error with the following code:
from apex import amp
from transformers import AutoModel
from transformers.optimization import AdamW, get_linear_schedule_with_warmup

model = AutoModel.from_pretrained('bert-base-cased')
model = model.cuda()

new_layer = ["extractor", "bilinear"]
optimizer_grouped_parameters = [{"params": [p for n, p in model.named_parameters() if not any(nd in n for nd in new_layer)], },{"params": [p for n, p in model.named_parameters() if any(nd in n for nd in new_layer)], "lr": 1e-4},]
optimizer = AdamW(optimizer_grouped_parameters, lr=1e-5, eps=1e-10)

model, optimizer = amp.initialize(model, optimizer, opt_level="O1", verbosity=0)

scheduler = get_linear_schedule_with_warmup(optimizer, num_warmup_steps=4000, num_training_steps=10000)

Commenting out the model, optimizer = amp.initialize line runs the code fine. However, running this script returns the following:
Traceback (most recent call last):

  File "../train.py", line 33, in finetune
    scheduler = get_linear_schedule_with_warmup(optimizer, num_warmup_steps=warmup_steps, num_training_steps=total_steps)
  File "/home/User/anaconda3/lib/python3.8/site-packages/transformers/optimization.py", line 98, in get_linear_schedule_with_warmup
    return LambdaLR(optimizer, lr_lambda, last_epoch)
  File "/home/User/anaconda3/lib/python3.8/site-packages/torch/optim/lr_scheduler.py", line 205, in __init__
    super(LambdaLR, self).__init__(optimizer, last_epoch, verbose)
  File "/home/User/anaconda3/lib/python3.8/site-packages/torch/optim/lr_scheduler.py", line 74, in __init__
    self.optimizer.step = with_counter(self.optimizer.step)
  File "/home/User/anaconda3/lib/python3.8/site-packages/torch/optim/lr_scheduler.py", line 56, in with_counter
    instance_ref = weakref.ref(method.__self__)
AttributeError: 'function' object has no attribute '__self__'

I'm wondering if this is an issue with CUDA 11.0, because for another server where I'm using 10.1 this works fine. But yeah, I'm a little lost as to what the issue is and how I should fix it. Any tips are appreciated.


title: pytorch lstm output not match hidden state

body:
Here is the code:
import torch
from torch.nn.utils.rnn import pack_sequence, pad_packed_sequence
rnn = torch.nn.LSTM(2, 3, 1, bidirectional=True)
x1= torch.randn(1,2)
x2=torch.randn(2,2)
x=pack_sequence([x1,x2], enforce_sorted=False)
y, (hn, cn) = rnn(x)
y,lens=pad_packed_sequence(y)

print the hidden state result:
print(torch.cat([hn[-2], hn[-1]], dim=1))

get result:
tensor([[-0.0265, -0.1149, -0.0466,  0.1080,  0.0901,  0.0865],
        [ 0.0736, -0.2503, -0.0634,  0.0801,  0.1401, -0.0032]],
       grad_fn=<CatBackward>)

print the output:
for i,j in enumerate(lens):
    print(y[i][j-1])

get result:
tensor([-0.0265, -0.1149, -0.0466,  0.1080,  0.0901,  0.0865],
       grad_fn=<SelectBackward>)
tensor([ 0.0736, -0.2503, -0.0634,  0.0932,  0.0962, -0.0915],
       grad_fn=<SelectBackward>)

the second tensor is not the same with the hidden state!
Why?


title: problem changing input type in pytorch reinforcement learning

body:
I'm trying to tun a code I found on github, but keep crashing in one part and getting TypeError: only size-1 arrays can be converted to Python scalars error. I've been trying to solve it for two days now. If I understand the problem correctly I have incompatible dtypes of tensors yet I've no idea how to fix this. Everytime I try to change my tensor type to say DoubleTensor, or float64 I get other errors. I simply haven't got a clue now which part needs to be changed and how.
This is my model defined:
class Policy(nn.Module):
    def __init__(self):
        super(Policy, self).__init__()
        self.input_layer = nn.Linear(11, 128)
        self.hidden_1 = nn.Linear(128, 128)
        self.hidden_2 = nn.Linear(32,31)
        self.hidden_state = torch.tensor(torch.zeros(2,1,32)).cuda()
        self.rnn = nn.GRU(128, 32, 2)
        self.action_head = nn.Linear(31, 5)
        self.value_head = nn.Linear(31, 1)
        self.saved_actions = []
        self.rewards = []

    def reset_hidden(self):
        self.hidden_state = torch.tensor(torch.zeros(2,1,32)).cuda()

    def forward(self, x):
        x = torch.tensor(x).cuda()
        x = torch.sigmoid(self.input_layer(x))
        x = torch.tanh(self.hidden_1(x))
        x, self.hidden_state = self.rnn(x.view(1,-1,128), self.hidden_state.data)
        x = F.relu(self.hidden_2(x.squeeze()))
        action_scores = self.action_head(x)
        state_values = self.value_head(x)
        return F.softmax(action_scores, dim=-1), state_values

        def forward(self, x):
          conv_out = self.conv(x).view(x.size()[0], -1)
          val = self.fc_val(conv_out)
          adv = self.fc_adv(conv_out)
          return val + (adv - adv.mean(dim=1, keepdim=True))

    def act(self, state):
        probs, state_value = self.forward(state)
        m = Categorical(probs)
        action = m.sample()
        if action == 1 and env.state[0] < 1: action = torch.LongTensor([2]).squeeze().cuda.DoubleTensor()
        if action == 4 and env.state[1] < 1: action = torch.LongTensor([2]).squeeze().cuda.DoubleTensor()
        if action == 6 and env.state[2] < 1: action = torch.LongTensor([2]).squeeze().cuda.DoubleTensor()
        self.saved_actions.append((m.log_prob(action), state_value))
        return action.item()

This is where it crashes
env.reset()
# In case you're running this a second time with the same model, delete the gradients
del model.rewards[:]
del model.saved_actions[:]

gamma = 0.9
log_interval = 60

def finish_episode():
    R = 0
    saved_actions = model.saved_actions
    policy_losses = []
    value_losses = []
    rewards = []
    for r in model.rewards[::-1]:
        R = r + (gamma * R)
        rewards.insert(0, R)
    rewards = torch.tensor(rewards)

    epsilon = (torch.rand(1) / 1e4) - 5e-5
    # With different architectures, I found the following standardization step sometimes
    # helpful, sometimes unhelpful.
    #
    rewards = (rewards - rewards.mean()) / (rewards.std(unbiased=False) + epsilon)
    # Alternatively, comment it out and use the following line instead:
    rewards += epsilon

    for (log_prob, value), r in zip(saved_actions, rewards):
        reward = torch.tensor(r - value.item()).cuda()
        policy_losses.append(-log_prob * reward)
        value_losses.append(F.smooth_l1_loss(value, torch.tensor([r]).cuda()))

    optimizer.zero_grad()
    loss = torch.stack(policy_losses).sum() + torch.stack(value_losses).sum()
    loss = torch.clamp(loss, -1e-5, 1e5)
    loss.backward()
    optimizer.step()
    del model.rewards[:]
    del model.saved_actions[:]

running_reward = 0
for episode in range(0, 4000):
    state = env.reset()
    state = state.type(torch.float32)
    reward = 0
    done = False
    msg = None
    while not done:
        action = model.act(state)
        state, reward, done, msg = env.step(action)
        model.rewards.append(reward)
        if done:
            break
    running_reward = running_reward * (1 - 1/log_interval) + reward * (1/log_interval)
    finish_episode()
    # Resetting the hidden state seems unnecessary - it's effectively random from the previous
    # episode anyway, more random than a bunch of zeros.
    # model.reset_hidden()
    if msg["msg"] == "done" and env.portfolio_value() > env.starting_portfolio_value * 1.1 and running_reward > 500:
        print("Early Stopping: " + str(int(reward)))
        break
    if episode % log_interval == 0:
        print("""Episode {}: started at {:.1f}, finished at {:.1f} because {} @ t={}, \
last reward {:.1f}, running reward {:.1f}""".format(episode, env.starting_portfolio_value, \
              env.portfolio_value(), msg["msg"], env.cur_timestep, reward, running_reward))

This is the error I get
/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:18: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-74-21b617d2e36f> in <module>()
     46     msg = None
     47     while not done:
---> 48         action = model.act(state)
     49         state, reward, done, msg = env.step(action)
     50         model.rewards.append(reward)

4 frames
/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py in linear(input, weight, bias)
   1751     if has_torch_function_variadic(input, weight):
   1752         return handle_torch_function(linear, (input, weight), input, weight, bias=bias)
-> 1753     return torch._C._nn.linear(input, weight, bias)
   1754
   1755

RuntimeError: expected scalar type Double but found Float

This is the github I'm using:
https://github.com/tomgrek/RL-stocktrading/blob/master/Finance%20final.ipynb
I was suggested that teh problem is because when I use "state" as a parameter torch throws an error cause it expects a numeric type, but I cannot change state to any float, because I get another error that list can't be changed to float32.
I'll be super grateful if you could show me idiot-proof what I'm doing wrong.


title: How to properly implement discriminative learning rate in PyTorch

body:
I want to implement discriminative learning rate into my PyTorch code but I'm not sure how to go about this. Right now I have this code that creates an SGD optimizer and a cyclic learning rate scheduler.
params, lr_arr, _ = discriminative_lr_params(model, slice(1e-5, 1e-3))
optimizer = torch.optim.SGD(params, lr=1e-3, momentum=0.9, weight_decay=1e-1)
scheduler = torch.optim.lr_scheduler.CyclicLR(optimizer, base_lr=list(lr_arr), max_lr=list(lr_arr*100))

where I'm using the code from this github: https://github.com/vdouet/Discriminative-learning-rates-PyTorch
for the discriminative_lr_params function. I'm not sure if this is the correct way to use discriminative learning rates in PyTorch but it's the only way I was able to find. However, when training with this, I'm getting really weird results:

Training using just SGD and using SGD+Cyclic LR works fine:
SGD:
optimizer = torch.optim.SGD(model.parameters(), lr=1e-3, momentum=0.9, weight_decay=1e-1)
SGD+Cyclic LR:
optimizer = torch.optim.SGD(model.parameters(), lr=1e-3, momentum=0.9, weight_decay=1e-1)
scheduler = torch.optim.lr_scheduler.CyclicLR(optimizer, base_lr=1e-3, max_lr=1e-2)


title: NumPy/PyTorch slicing using an array of indices from an argmax call

body:
I have the following tensors / ndarrays I am operating with.
a_intents of shape (b, n_i) - Scalar at position ij is the activation of intent j for the i-th element in the batch.
u_intents of shape (b, n_i, d_m) - Vector of dimension d_m at index ij is the pose vector of intent j for the i-th element in the batch.
I want to get the index of the intent with the biggest activation scalar, and I do:
 max_activations = argmax(a_intents, dim=-1, keepdim=False)

Now, using those indices, I want to get the corresponding vectors in u_intents.
max_activation_poses = u_intents[?, ?,:]

How do I use max_activations to indicate only those indices on dim 1? My intuition tells me I will end with an incorrect shape if I do
[:, max_activations, :]

The shape I am trying to get is (b, d_m) - The vector with the same index as the highest activation, for each element in the batch.
Thanks


title: module 'torch.jit' has no attribute 'unused'

body:
I am a student trying to get my hands into facial recognition using Ultra96 and I am having troubles running my program.
I have tried to install nightly version into my Ultra96 however, it does not solve my problem. The current OS is pynq 2.6 which is Linux and it is using jupyter notebook to run the codes.
Please offer me some guidance!