title: Cublas runtime error in PyTorch? Code worked in a Jupyter notebook but not in a terminal

body:
I'm new to python and trying to figure out where my error messages are coming from is difficult. I wrote a code to use some existing (more complicated) code in a Jupyter notebook and it seems to work fine. I get this RuntimeError message in the Jupyter notebook but everything seems to be working fine and the files I'm writing are created regardless. I've been trying to adapt it to run inside the anaconda3 terminal and now things are falling apart. I created the python file and put the same "working code" from the jupyter notebook in and now the runtime error prevents the code from running. I have no idea why it works in the jupyter notebook but not in the terminal.
Anyway, here's the runtime error"
RuntimeError: cublas runtime error : the GPU program failed to execute at C:/w/1/s/tmp_conda_3.7_055306/conda/conda-bld/pytorch_1556690124416/work/aten/src/THC/THCBlas.cu:450

I've been googling around and someone suggested to update CUDA. I tried that and I don't think it's working. I'm wondering if anyone could knows what this error means and how I can solve it?
Here's the information I got about my CUDA version when I did nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2020 NVIDIA Corporation
Built on Wed_May__6_19:10:02_Pacific_Daylight_Time_2020
Cuda compilation tools, release 11.0, V11.0.167
Build cuda_11.0_bu.relgpu_drvr445TC445_37.28358933_0

Here's the traceback I get before the RuntimeError:
  File "AMASSBody.py", line 42, in <module>
    body = bm(pose_body=pose_body, betas=betas, root_orient=root_orient)
  File "C:\Users\emmay\anaconda3\lib\site-packages\torch\nn\modules\module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "c:\users\emmay\src\human-body-prior\human_body_prior\body_model\body_model.py", line 251, in forward
    dtype=self.dtype)
  File "C:\Users\emmay\anaconda3\lib\site-packages\smplx\lbs.py", line 179, in lbs
    v_shaped = v_template + blend_shapes(betas, shapedirs)
  File "C:\Users\emmay\anaconda3\lib\site-packages\smplx\lbs.py", line 265, in blend_shapes
    blend_shape = torch.einsum('bl,mkl->bmk', [betas, shape_disps])
  File "C:\Users\emmay\anaconda3\lib\site-packages\torch\functional.py", line 211, in einsum
    return torch._C._VariableFunctions.einsum(equation, operands)

I'm running python3 on windows 10. Let me know if there's more information you guys need.


title: Applying a function along all indices on an axis in PyTorch

body:
I'm trying to implement the Wasserstein Loss function in PyTorch, and I'm referencing the Scipy implementation for this. Since using PyTorch functions within the forward() method implies not having to write the backward() function, I have done this in my code (with the Scipy version just having the equivalent Numpy functions involved). Here is what I have:
class WassersteinLoss(nn.Module):
  def __init__(self):
    super(WassersteinLoss, self).__init__()
  def forward(self,u,v):
    result = torch.empty((len(u)))
    for i in range(len(u)):
      u_values,v_values = u[i],v[i]
      u_sorter,v_sorter = torch.argsort(u_values),torch.argsort(v_values)
      all_values = torch.cat((u_values,v_values))
      all_values,idx = torch.sort(all_values)

      # Compute the differences between pairs of successive values of u and v.
      deltas = torch.sub(all_values[1:],all_values[:-1])

      # Get the respective positions of the values of u and v among the values of
      # both distributions.
      u_cdf_indices = torch.searchsorted(u_values[u_sorter],all_values[:-1],right=True)
      v_cdf_indices = torch.searchsorted(v_values[v_sorter],all_values[:-1],right=True)

      # Calculate the CDFs of u and v
      u_cdf = torch.div(u_cdf_indices,len(u_values))
      v_cdf = torch.div(v_cdf_indices,len(v_values))

      # Compute the value of the integral based on the CDFs.
      result[i] = torch.sum(torch.multiply(torch.abs(u_cdf-v_cdf),deltas))
    return result.mean()

In the above function, u and v are the vectors of shape (NxM), where N is the number of samples in the batch. Since my for loop essentially goes over all the samples, the calculations on each of them are independent, since the samples in a batch depend on each other. I believe I'd see a significant speedup if I can do away with this for loop. Until now, I have tried performing all computations along the dim=1 axis, but this does not work.
Here is the code for a test case:
from scipy.stats import wasserstein_distance
import torch
import torch.nn as nn

print(wasserstein_distance([0, 1, 3], [5, 6, 8]))
#Output is 5

criterion = WassersteinLoss()
print(criterion(torch.tensor([[0, 1, 3]]), torch.tensor([[5, 6, 8]])))
#Output is tensor(5.)

Any inputs on how to modify the forward() function to eliminate the for loop would be greatly appreciated.


title: Is Tensorflow always faster than Pytorch when running in graph mode

body:
I have a simple model implemented using Tensorflow and Pytorch. I noticed that the tensorflow implementation runs over 3x faster than the Pytorch implementation when I use graph mode in tf (as I know Pytorch only supports eager mode). Does that mean Tensorflow is faster than Pytorch when running in graph mode?
I run both implementations on CPU.
Tensorflow version: 2.4.1,
Pytorch version: 1.7.1,
OS: windows 7
Tensorflow implementation
import tensorflow as tf
import numpy as np
import time

number_of_neural_units = 10
number_of_points = 101
number_of_epochs = 50000
loss_eva_interval = 1000

x_train = np.linspace(0, 2, number_of_points, endpoint=True).reshape((number_of_points, 1))
y_actual = np.exp(-0.5 * x_train**2) / (1 + x_train + x_train**3) + x_train**2
x_train_ts = tf.constant(x_train, dtype=tf.float32)

loss_array = np.zeros([int(number_of_epochs / loss_eva_interval) + 1])

class Model:
    def __init__(self):
        self.W = tf.Variable(tf.random.normal([1, number_of_neural_units]), name='W')
        self.b = tf.Variable(tf.random.normal([number_of_neural_units]), name='b')
        self.W1 = tf.Variable(tf.random.normal([number_of_neural_units, 1]), name='W1')
        self.b1 = tf.Variable(tf.random.normal([1]), name='b1')

        self.y1 = None
        self.y = None

        self.trainable_weights = [self.W, self.b, self.W1, self.b1]

    def __call__(self, x):
        self.y1 = tf.nn.sigmoid(tf.add(tf.matmul(x, self.W), self.b))
        self.y = tf.add(tf.matmul(self.y1, self.W1), self.b1)
        return self.y

def total_loss():
    model(x_train_ts)
    lq = (1 + 3 * (x_train_ts ** 2)) / (1 + x_train_ts + x_train_ts ** 3)

    dif = tf.matmul(tf.multiply(model.y1 * (1 - model.y1), model.W), model.W1)

    t_loss = (dif + (x_train_ts + lq) * model.y - x_train_ts**3 - 2 * x_train_ts - lq * x_train_ts**2)**2
    loss = tf.reduce_mean(t_loss) + (model.y[0] - 1)**2
    return loss

@tf.function
def updateWeights():
    with tf.GradientTape() as tape:
        loss = total_loss()
    grads = tape.gradient(loss, model.trainable_weights)
    opt.apply_gradients(zip(grads, model.trainable_weights))

def trainModel():
    t_start = time.time()

    current_lost = total_loss().numpy()
    loss_array[0] = current_lost
    print('epoch {}: loss: {}'.format(0, current_lost))

    for epoch in range(1, number_of_epochs + 1):

        updateWeights()

        if epoch % loss_eva_interval == 0:
            current_lost = total_loss().numpy()
            loss_array[int(epoch / loss_eva_interval)] = current_lost
            print('epoch {}: loss: {}'.format(epoch, current_lost))

    t_end = time.time()
    print('total time: {}'.format(t_end - t_start))

if __name__ == '__main__':
    model = Model()
    opt = tf.optimizers.Adam(learning_rate=0.01)
    trainModel()

Pytorch implementation:
import torch
from torch.autograd import Variable
import numpy as np
import time

number_of_neural_units = 10
number_of_points = 101
number_of_epochs = 50000
loss_eva_interval = 1000

x_train = np.linspace(0, 2, number_of_points, endpoint=True, dtype=np.float32).reshape((number_of_points, 1))
y_actual = np.exp(-0.5 * x_train ** 2) / (1 + x_train + x_train ** 3) + x_train ** 2
x_train_ts = torch.from_numpy(x_train)

loss_array = np.zeros([int(number_of_epochs / loss_eva_interval) + 1])

class Model:
    def __init__(self):
        self.W = Variable(torch.randn(1, number_of_neural_units), requires_grad=True)
        self.b = Variable(torch.randn(number_of_neural_units), requires_grad=True)
        self.W1 = Variable(torch.randn(number_of_neural_units, 1), requires_grad=True)
        self.b1 = Variable(torch.randn(1), requires_grad=True)

        self.y1 = None
        self.y = None

        self.trainable_weights = [self.W, self.b, self.W1, self.b1]

    def __call__(self, x):
        self.y1 = torch.sigmoid(torch.add(torch.matmul(x, self.W), self.b))
        self.y = torch.add(torch.matmul(self.y1, self.W1), self.b1)
        return self.y

def total_loss():
    model(x_train_ts)
    lq = (1 + 3 * (x_train_ts ** 2)) / (1 + x_train_ts + x_train_ts ** 3)

    dif = torch.matmul(torch.multiply(model.y1 * (1 - model.y1), model.W), model.W1)  # dy/dx [100,1]

    t_loss = (dif + (x_train_ts + lq) * model.y - x_train_ts ** 3 - 2 * x_train_ts - lq * x_train_ts ** 2) ** 2
    loss = t_loss.mean() + (model.y[0] - 1) ** 2
    return loss

def updateWeights():
    opt.zero_grad()
    loss = total_loss()
    loss.backward()
    opt.step()

def trainModel():
    t_start = time.time()

    current_lost = total_loss().detach().numpy()
    loss_array[0] = current_lost
    print('epoch {}: loss: {}'.format(0, current_lost))

    for epoch in range(1, number_of_epochs + 1):

        updateWeights()

        if epoch % loss_eva_interval == 0:
            current_lost = total_loss().detach().numpy()
            loss_array[int(epoch / loss_eva_interval)] = current_lost
            print('epoch {}: loss: {}'.format(epoch, current_lost))

    t_end = time.time()
    print('total time: {}'.format(t_end - t_start))

if __name__ == '__main__':
    model = Model()

    opt = torch.optim.Adam(model.trainable_weights, lr=0.01)
    trainModel()


title: Installing PyTorch with CUDA in setup.py

body:
I'm trying to specify PyTorch with CUDA in install_requires. The command to install with pip is
pip install torch==1.8.0+cu111 torchvision==0.9.0+cu111 torchaudio===0.8.0 -f https://download.pytorch.org/whl/torch_stable.html
How do I do that in the setup.py install_requires?


title: How to concatenate tensor to another list of tensor in pytorch?

body:
I have a tensor of shape "torch.Size([2, 2, 3])" and another tensor of shape "torch.Size([2, 1, 3])". I want a concatenated tensor of shape "torch.Size([2, 2, 6])".
 For example :
a=torch.tensor([[[2,3,5],[12,13,15]],[[20,30,50],[120,130,150]]])
b=torch.tensor([[[99,99,99]],[[999,999,999]]])

I want the output as : [[[99,99,99,2,3,5],[99,99,99,12,13,15]],[[999,999,999,20,30,50],[999,999,999,120,130,150]]]
I have written a O(n2) solution using two for loops but,
This is taking a lot of time with millions of calculation, Does anyone help me in doing this efficiently ?? May be some matrix calculation trick for tensors ??


title: Memory leak with en_core_web_trf model, Spacy

body:
there is a Memory leak when using pipe of en_core_web_trf model, I run the model using GPU with 16GB RAM, here is a sample of the code.
!python -m spacy download en_core_web_trf

import en_core_web_trf
nlp = en_core_web_trf.load()

#it's just an array of 100K sentences.
data = dataload()

for index, review in enumerate( nlp.pipe(data, batch_size=100) ):
    #doing some processing here
    if index % 1000: print(index)

this code cracks when reaching 31K, and raises OOM error.
CUDA out of memory. Tried to allocate 46.00 MiB (GPU 0; 11.17 GiB total capacity; 10.44 GiB already allocated; 832.00 KiB free; 10.72 GiB reserved in total by PyTorch)

I just use the pipeline to predict, not train any data or other stuff and tried with different batch sizes, but nothing happened,
still, crash.
Your Environment

spaCy version: 3.0.5
Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
Python version: 3.7.10
Pipelines: en_core_web_trf (3.0.0)


title: pytorch produce Process finished with exit code 139 when running on CUDA?

body:
I have created a simple model and training function to train a simple linear regression training, when running on CPU the code runs but when running on CUDA the code crashes half way randomly could not figure it out what exactly is the problem.
Here is the model:
import torch.nn as nn
import torch

class LinearRegression1D:
    def __init__(self, in_feature=1, out_feature=1):
        self.in_feature = in_feature
        self.out_feature = out_feature
        self.device = torch.device("cuda")
        self.loss_func = nn.MSELoss()

    def set_device(self, device: str):
        devices = ["cuda", "cpu"]
        if device not in devices:
            raise ValueError("Only {} is valid as device name".format(" and ".join(devices)))
        self.device = torch.device(device)

    def get_device(self):
        return self.device

    def get_loss_func(self):
        return self.loss_func

    def __call__(self, *args, **kwargs):
        model = nn.Linear(self.in_feature, self.out_feature)
        return model

Here is the training function:
import torch
from utills import move_to
from tqdm import tqdm

def train_simple_network(
    model,
    loss_func,
    training_loader,
    epochs=20,
    device="cpu"
):
    optimizer = torch.optim.SGD(model.parameters(), lr=0.001)
    model.to(device)
    for _ in tqdm(range(epochs), desc="Epoch"):
        model = model.train()
        running_loss = 0.0
        for inputs, labels in tqdm(training_loader, desc="Batch", leave=False):
            inputs = move_to(inputs, device)
            labels = move_to(labels, device)

            optimizer.zero_grad()
            y_hat = model(inputs)
            loss = loss_func(y_hat, labels)
            loss.backward()
            optimizer.step()
            running_loss += loss.item()

The move to function is a general function to move any data structure to the selected device:
def move_to(obj, device):
    """
    Based on the type move python object
    :param obj: the python object to move to a device, or to move its contents to a device
    :param device: the compute device to move objects to
    :return: python obj
    """
    if isinstance(obj, list):
        return [move_to(x, device) for x in obj]
    elif isinstance(obj, tuple):
        return tuple(move_to(list(obj), device))
    elif isinstance(obj, set):
        return set(move_to(list(obj), device))
    elif isinstance(obj, dict):
        to_ret = dict()
        for key, value in obj.items():
            to_ret[move_to(key, device)] = move_to(value, device)
        return to_ret
    elif hasattr(obj, "to"):
        return obj.to(device)
    else:
        return obj

Here is the final execution:
import torch
from torch.utils.data import DataLoader
import seaborn as sns

from chapter2.dataloaders import Simple1DRegressionDataSet
from chapter2.datagenerators import generate_1d_data
from chapter2.models import LinearRegression1D
from chapter2.train import train_simple_network

# utility function to generate 1 feature and 1 target set
features, targets = generate_1d_data()
# sns.scatterplot(x=features, y=targets)

# Convert dataset object to iterator
training_loader = DataLoader(Simple1DRegressionDataSet(features=features, targets=targets), shuffle=True)
# initialize model
model_instance = LinearRegression1D()

# If cuda is used it throws error: Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)
model_instance.set_device("cpu")

model = model_instance()
loss_func = model_instance.get_loss_func()
device = model_instance.get_device()

train_simple_network(model=model, loss_func=loss_func, training_loader=training_loader, device=device)

For reuse purpose I have separated the codes into files. The seperated code can be found in the following github repo . The final main code can be found in the following file repo.
Can some one help me to figure out the problem.


title: Installing PyTorch on Jetson Nano Ubuntu 18

body:
I am trying to install PyTorch on Jetson Nano Ruining Ubuntu 1804. My reference is https://dev.to/evanilukhin/guide-to-install-pytorch-with-cuda-on-ubuntu-18-04-5217
When I try the following command this is what I get:
(my_env) crigano@crigano-desktop:~$ python3.8 -m pip install numpy ninja pyyaml mkl mkl-include setuptools cmake cffi typing
Collecting numpy
  Using cached numpy-1.20.2-cp38-cp38-manylinux2014_aarch64.whl (12.7 MB)
Collecting ninja
  Using cached ninja-1.10.0.post2.tar.gz (25 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting pyyaml
  Using cached PyYAML-5.4.1-cp38-cp38-manylinux2014_aarch64.whl (818 kB)
ERROR: Could not find a version that satisfies the requirement mkl
ERROR: No matching distribution found for mkl


title: Pytorch "Unfold" equivalent in Tensorflow

body:
Say I have grayscale images of size (50*50), with a batch size of 2 in this case, and i use the Pytorch Unfold function as follows:
import numpy as np
from torch import nn
from torch import tensor

image1 = np.random.rand(1,50,50)
image2 = np.random.rand(1,50,50)
image = np.stack((image1,image2))
image = tensor(image)

ds = nn.Unfold(kernel_size=(2,2),stride=2)
x = ds(image).numpy()
x.shape

## OUTPUT: (2, 4, 625)

What would be the equivalent tensorflow implementation so that the output of the tensorflow implementation would exactly match 'x'? I've tried using the tf.image.extract_patches function but it seems to not be giving me what I quite want.
The question is then: What is the tensorflow implementation of Unfold?


title: How to reshape multichannel image with a PyTorch encoder?

body:
I have a tensor with dimensions [18, 512, 512], representing grayscale heatmaps for up to 18 specific objects on a 512x512 image. In order to generate a suitable representation of this image for my conditional GAN, I need to reshape this tensor into a [512, 4, 4] shape using an encoder. However, I can't understand how this transformation can be achieved, since given dimensions apppear too mismatched for direct linear or convolutional transformations.
class HeatmapEncoder(torch.nn.Module):
  def __init__(self):
    # source = 18x512x512
    # target = 512x4x4
    self.encoder = torch.nn.Sequential(
        nn.Linear(),
        nn.ReLU(),
        nn.Linear()
    )

    def forward(self, x):
      pass

It is possible to use `nn.Flatten()' with start_dim=0 here, but the result will be a simplified tensor that can't be used as an input by the linear layer.
The decoder part is not especially important right now, since I only need a low-dimensional representation of the heatmaps to condition my GAN and not to recreate those images.


title: How to replace torch.norm with other pytorch function?

body:
I would like to replace the torch.norm function using the other Pytorch function.
I was able to replace torch.norm in the case where x is not a matrix, as shown in the following code.
import torch
x = torch.randn(9)
out1 = torch.norm(x)
out2 = sum(abs(x)**2)**(1./2)
out1 == out2
>> tensor(True)

But I don't know how to replace it when x is a matrix.
Especially, I want to replace it in my case of dim=1 and keepdim=True.
x = torch.randn([3, 136, 64, 64])
out1 = torch.norm(x, dim=1, keepdim=True)
out2 = ???
out1 == out2

Background:
I'm converting a Pytorch model to CoreML, but the _VF.frobenius_norm operator defined in the torch.norm function is not implemented with CoreMLTools.
(The implementation inside torch.norm can be found here.)
A few people have trouble with this problem, but CoreMLTools is still unsupported (You can check from this issue).
So I'd like to replace it without the operator used in torch.norm.
I have tried torch.linalg.norm() and numpy.linalg.norm but they were not supported.
I have created a simple colaboratory notebook that reproduces this.
Please test it using the following colab.
https://colab.research.google.com/drive/11o6rTxHzEgZ_Rc7nFZHd3TvPugybB88h?usp=sharing


title: How to make VScode launch.json for a Python module

body:
I'm reseaching self-supervised muchine learning code.
And I have wanted to debug the code with python debugger not pdb.set_trace().
This is python command for ubuntu terminal.
python -m torch.distributed.launch --nproc_per_node=1 main_swav.py \
--data_path /dataset/imagenet/train \
--epochs 400 \
--base_lr 0.6 \
--final_lr 0.0006 \
--warmup_epochs 0 \
--batch_size 8 \
--size_crops 224 96 \
--nmb_crops 2 6 \
--min_scale_crops 0.14 0.05 \
--max_scale_crops 1. 0.14 \
--use_fp16 true \
--freeze_prototypes_niters 5005 \
--queue_length 380 \
--epoch_queue_starts 15\
--workers 10

In order to debug the code with VScode, I tried to revise launch.json like below as referring stackoverflow -question
{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Python: Current File",
            "type": "python",
            "module": "torch.distributed.launch --nproc_per_node=1 main_swav.py",
            "request": "launch",
            "console": "integratedTerminal",
            "args": ["--data_path", "/dataset/imagenet/train"]
        }
    ]
}

I knew this would not work... TT
Could you give me some advice?
Thank you for your time.


I run this code in google colab there showing an unexpected type error
Code:
First Cell -
#CNN Network
class convNet(nn.Module):
def __init__(self,num_classes=4):
super(convNet,self).__init__()

#Input shape= (32,3,256,256)

self.conv1=nn.Conv2d(in_channels=3,out_channels=32,kernel_size=3,stride=4,padding=1)
#Shape= (32,32,65,65)
self.bn1=nn.BatchNorm2d(num_features=12)
#Shape= (32,32,65,65)
self.relu1=nn.ReLU()
#Shape= (32,32,65,65)

self.pool1=nn.MaxPool2d(kernel_size=1)
#Shape= (32,32,64,64)

self.conv2=nn.Conv2d(in_channels=32,out_channels=64,kernel_size=3,stride=1,padding=1)
#Shape= (32,64,64,64)
self.relu2=nn.ReLU()
#Shape= (32,64,64,64)

self.pool2=nn.MaxPool2d(kernel_size=1)
#Shape= (32,64,63,63)

self.conv3=nn.Conv2d(in_channels=64,out_channels=128,kernel_size=3,stride=1,padding=1)
#Shape= (32,128,63,63)
self.relu3=nn.ReLU()
#Shape= (32,128,63,63)

self.pool3=nn.MaxPool2d(kernel_size=1)
#Shape= (32,128,62,62)

self.conv4=nn.Conv2d(in_channels=128,out_channels=256,kernel_size=3,stride=2,padding=1)
#Shape= (32,256,32,32)
self.bn2=nn.BatchNorm2d(num_features=32)
#Shape= (32,256,32,32)
self.relu4=nn.ReLU()
#Shape= (32,256,32,32)

self.pool4=nn.MaxPool2d(kernel_size=2)
#Shape= (32,256,16,16)

self.fc=nn.Linear(in_features=256*16*16,out_features=num_classes)

#Feed forward function
def forward(self,input):
  output=self.conv1(input)
  output=self.bn1(output)
  output=self.relu1(output)

  output=self.pool1(output)

  output=self.conv2(output)
  output=self.relu2(output)

  output=self.pool2(output)

  output=self.conv3(output)
  output=self.relu3(output)

  output=self.pool3(output)

  output=self.conv4(output)
  output=self.bn2(output)
  output=self.relu4(output)

  output=self.pool4(output)

  output=output.view(-1,256*16*16)

  output=self.fc(output)

  return output

Second Cell -
model=convNet(num_classes=4).to(device)

Error:
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-60-26ab8bb36717> in <module>()
----> 1 model=convNet(num_classes=4).to(device)

<ipython-input-59-49271bc77f04> in __init__(self, num_classes)
  6     #Input shape= (32,3,256,256)
  7
----> 8     self.conv1=nn.Conv2d(in_channels=3,out_channels=32,kernal_size=3,stride=4,padding=1)
  9     #Shape= (32,32,65,65)
 10     self.bn1=nn.BatchNorm2d(num_features=12)

TypeError: __init__() got an unexpected keyword argument 'kernal_size'

Thanks in advance.


title: How to use return value from one class in another?

body:
My code looks like this :

class RegularConv(nn.module):
    def __init__(self):
        super().__init__.()
        self.pool1 = nn.MaxPool2d(2, stride=2, return_indices=True)

    def forward (self, inputs):
        inputs, indices = self.pool1(inputs)
        return inputs, indices

class UpsamplingConv(nn.module):
    def __init__(self, indices):
        super().__init__.()
        self.unpool = nn.MaxUnpool2d(2, stride=2, padding=0)

    def forward (self, output):
        output = self.unpool(output, indices, output_size=output_size)
        return output
class Net(nn.module):
    def __init__(self):
        super().__init__.()
        self.layer1 = RegularConv()
        self.layer2 = UpsamplingConv()

    def forward(self, input):
        input, indices = self.layer1(input)
        input = self.layer2(input, indices)

The forward method in RegularConv class returns inputs and indices and the forward method in UpsamplingConv class has to use those indices.

I use both of these classes in another class Net. I want to use the return value from one class and use in another class. I want to pass the returned indices from RegularConv as an argument in UpsamplingConv


title: Pytorch - Are my weights being shared in this network?

body:
I have the following network where I am trying to do triplet loss:

First, I have a custom Convolution class ConvBlock(nn.Module):

def __init__(self, ngpu, input_c, output_c, mode=0):
    super(ConvBlock, self).__init__()
    self.ngpu = ngpu
    self.input_c = input_c
    self.output_c = output_c
    self.mode = mode

    self.b1 = nn.Sequential(
            nn.Conv2d(input_c, output_c, 3, stride=1, padding=1),
            #nn.BatchNorm2d(output_c),
            nn.PReLU(),
        )
    self.b2 = nn.Sequential(
            nn.Conv2d(output_c, output_c, 3, stride=1, padding=1),
            #nn.BatchNorm2d(output_c),
            nn.PReLU(),
        )
    self.pool = nn.Sequential(
            nn.MaxPool2d(2, 2),
        )

def forward(self, input):

    batch_size = input.size(0)
    if self.mode == 0:
        b1 = self.b1(input)
        hidden = self.pool(b1)
        return hidden, b1
    elif self.mode == 1:
        b1 = self.b1(input)
        b2 = self.b2(b1)
        hidden = self.pool(b2)
        return hidden, b2
    elif self.mode == 2:
        b1 = self.b1(input)
        hidden = self.b2(b1)
        return hidden

I now have an encoder module:

class _Encoder(nn.Module):

def __init__(self, ngpu,nc,nef,out_size,nz):
    super(_Encoder, self).__init__()
    self.ngpu = ngpu
    self.nc = nc
    self.nef = nef
    self.out_size = out_size
    self.nz = nz

    self.c1 = ConvBlock(self.ngpu, nc, nef, 0)       # 3 - 64
    self.c2 = ConvBlock(self.ngpu, nef, nef*2, 0)    # 64-128
    self.c3 = ConvBlock(self.ngpu, nef*2, nef*4, 1)  # 128-256
    self.c4 = ConvBlock(self.ngpu, nef*4, nef*8, 1)  # 256 -512
    self.c5 = ConvBlock(self.ngpu, nef*8, nef*8, 2)  # 512-512

    # 8 because..the depth went from 32 to 32*8
    self.mean = nn.Linear(nef * 8 * out_size * (out_size/2), nz)
    self.logvar = nn.Linear(nef * 8 * out_size * (out_size/2), nz)

#for reparametrization trick
def sampler(self, mean, logvar):
    std = logvar.mul(0.5).exp_()
    if args.cuda:
        eps = torch.cuda.FloatTensor(std.size()).normal_()
    else:
        eps = torch.FloatTensor(std.size()).normal_()
    eps = Variable(eps)
    return eps.mul(std).add_(mean)

def forward(self, input):
    batch_size = input.size(0)
    if isinstance(input.data, torch.cuda.FloatTensor) and self.ngpu > 1:
        c1_out, c1_x = nn.parallel.data_parallel(self.c1, input, range(self.ngpu))
        c2_out, c2_x = nn.parallel.data_parallel(self.c2, c1_out, range(self.ngpu))
        c3_out, c3_x = nn.parallel.data_parallel(self.c3, c2_out, range(self.ngpu))
        c4_out, c4_x = nn.parallel.data_parallel(self.c4, c3_out, range(self.ngpu))
        hidden = nn.parallel.data_parallel(self.c5, c4_out, range(self.ngpu))

        # hidden = nn.parallel.data_parallel(self.encoder, input, range(self.ngpu))
        hidden = hidden.view(batch_size, -1)
        mean = nn.parallel.data_parallel(self.mean, hidden, range(self.ngpu))
        logvar = nn.parallel.data_parallel(self.logvar, hidden, range(self.ngpu))
    else:
        c1_out, c1_x = self.c1(input)
        c2_out, c2_x = self.c2(c1_out)
        c3_out, c3_x = self.c3(c2_out)
        c4_out, c4_x = self.c4(c3_out)
        hidden = self.c5(c4_out)

        # hidden = self.encoder(input)
        hidden = hidden.view(batch_size, -1)
        mean, logvar = self.mean(hidden), self.logvar(hidden)

    latent_z = self.sampler(mean, logvar)
    if ADD_SKIP_CONNECTION:
        return latent_z,mean,logvar,{"c1_x":c1_x, "c2_x":c2_x, "c3_x":c3_x, "c4_x":c4_x}
    else:
        return latent_z,mean,logvar,{"c1_x":None, "c2_x":None, "c3_x":None, "c4_x":None}

I initialize my encoder as a single object:

encoder = _Encoder(ngpu,nc,nef,out_size,nz)
encoder = encoder.cuda()

And then I am applying some functions:

    latent_x,mean_x,logvar_x,skip_x = self.encoder(x)
    latent_y,mean_y,logvar_y,skip_y = self.encoder(y)
    latent_z,mean_z,logvar_z,skip_z = self.encoder(z)
    dist_a = F.pairwise_distance(mean_x, mean_y, 2)
    dist_b = F.pairwise_distance(mean_x, mean_z, 2)
    loss_triplet = triplet_loss(dist_a, dist_b, target)

    optimizer.zero_grad()
    loss_triplet.backward()
    optimizer.step()

I am starting to doubt if the weights are actually being shared across the 3 encoder blocks. Please help me check an tell me if it does


title: In tensorflow, how to enumerate training data (compared to pytorch)

body:
In pytorch, this is how I enumerate training data.

for epoch in range(0, args.epoches):
    for i, batch in enumerate(train_data):
        model.update(batch)

train_data contains multiple batches and batches are getting enumerated and updating the model, which is very clear to me.

I think this is a basic example of how tensorflow treats the batches.

for step in range(num_steps):
    batch_data, batch_labels = generate_batch(batch_size, num_skips, skip_window)
    feed_dict = {train_dataset : batch_data, train_labels : batch_labels}
    _, l = session.run([optimizer, loss], feed_dict=feed_dict)

Maybe this is a very obvious question, but I'm not clear how enumerating training batches is handled by session.run in tensorflow. I can't find batches are getting looped through in the code. All I see is feed_dict and I assume it handles the looping.

Can someone shed some light on this?


title: VGG network in pytorch produces identical outputs during test for different inputs

body:
I am using (untrained) VGG-19 with batch normalization on CIFAR10 images. When I turn the model into eval mode, I noticed that the model produced identical output logits values for different inputs. Is this normal? The parameters of the VGG network is random. Below is my code.

import torch
import torch.nn as nn

cfg = {
    'vgg11': [64, 'M', 128, 'M', 256, 256, 'M', 512, 512, 'M', 512, 512, 'M'],
    'vgg13': [64, 64, 'M', 128, 128, 'M', 256, 256, 'M', 512, 512, 'M', 512, 512, 'M'],
    'vgg16': [64, 64, 'M', 128, 128, 'M', 256, 256, 256, 'M', 512, 512, 512, 'M', 512, 512, 512, 'M'],
    'vgg19': [64, 64, 'M', 128, 128, 'M', 256, 256, 256, 256, 'M', 512, 512, 512, 512, 'M', 512, 512, 512, 512, 'M'],
}

class VGG(nn.Module):
    def __init__(self, vgg_name):
        super(VGG, self).__init__()
        self.features = self._make_layers(cfg[vgg_name])
        self.classifier = nn.Linear(512, 10)

    def forward(self, x):
        out = self.features(x)
        out = out.view(out.size(0), -1)
        out = self.classifier(out)
        return out

    @staticmethod
    def _make_layers(cfg):
        layers = []
        in_channels = 3
        for x in cfg:
            if x == 'M':
                layers += [nn.MaxPool2d(kernel_size=2, stride=2)]
            else:
                layers += [nn.Conv2d(in_channels, x, kernel_size=3, padding=1),
                           nn.BatchNorm2d(x),
                           nn.ReLU(inplace=True)]
                in_channels = x
        layers += [nn.AvgPool2d(kernel_size=1, stride=1)]
        return nn.Sequential(*layers)


title: Detailed examples of what Keras cannot do but Pytorch can?

body: I read many articles saying Keras is too high level and hard to be used for research. I found Keras has Lambda layer, and custom layer, so what are some detailed examples of what Pytorch can achieve while Keras cannot or is very tricky to? Thanks.


title: Defining parameters by some transformation OR Retaining sub-graphs, but not the whole graph

body:
I'm coming across an issue I haven't seen come up before. I work in Bayesian Machine Learning and as such make a lot of use of the distributions in PyTorch. One common thing to do is to define some of the parameters of distributions in terms of the log of their parameter, so that in optimisation they cannot go negative (e.g. the standard deviation of a Normal distribution).

In order to be distribution independent however, I do not want to have to manually recompute the conversion of this parameter. To demonstrate via example:

The following code will NOT run. After the first backward pass, the part of the graph the calculates the exponential of the parameter is automatically removed, and not re-added.

import torch
import torch.nn as nn
import torch.distributions as dd

log_std = nn.Parameter(torch.Tensor([1])) # Define the log of the parameter as an nn.Parameter, this is what we want to optimise
std = torch.exp(log_std) # Define the transformation we want to apply to the parameter to using it in the distribution
mean = nn.Parameter(torch.Tensor([1])) # A normal parameter
dist = dd.Normal(loc=mean, scale=std) # Define the distribution. From here I want to ONLY refer to this, not the other variables

optim = torch.optim.SGD([log_std, mean], lr=0.01) # Standard optimiser
target = dd.Normal(5,5) # Target distribution to match

for i in range(50):
    optim.zero_grad()

    samples = dist.rsample((1000,)) # Sample our model, note no reference to log_std

    cost = -(target.log_prob(samples) - dist.log_prob(samples)).sum() # KLdivergence cost metric
    cost.backward()
    optim.step()
    print(i)
    print(log_std, mean, cost)
    print()

The next set of code WILL run, but I must explicitly reference the log_std parameter in the loop, and recreate the distribution. If I wanted to change the distribution type, it would not be possible without considering the specific case.

import torch
import torch.nn as nn
import torch.distributions as dd

log_std = nn.Parameter(torch.Tensor([1])) # Define the log of the parameter as an nn.Parameter, this is what we want to optimise
mean = nn.Parameter(torch.Tensor([1])) # A normal parameter

optim = torch.optim.SGD([log_std, mean], lr=0.001) # Standard optimiser
target = dd.Normal(5,5) # Target distribution to match

for i in range(50):
    optim.zero_grad()

    std = torch.exp(log_std)  # Define the transformation we want to apply to the parameter to using it in the distribution
    dist = dd.Normal(loc=mean, scale=std)  # Define the distribution.

    samples = dist.rsample((1000,)) # Sample our model, note no reference to log_std

    cost = -(target.log_prob(samples) - dist.log_prob(samples)).sum() # KL divergence cost metric
    cost.backward()
    optim.step()
    print(i)
    print(mean, std, cost)
    print()

The first example does however work in Tensorflow, as the graphs there are static. Does anyone have some ideas on how I might fix this? If it were possible to keep only the part of the graph that defines the relationship std = torch.exp(log_std) then this could work. I have also tried playing with backwards gradient hooks, but unfortunately to calculate the new gradient properly you need access to the parameter value and the learning rate.

Thanks in advance!
Michael

EDIT

I have been asked for an example of how I might want to change the distribution. Taking the code which currently will NOT work, and changing the distributions to Gamma distributions:

import torch
import torch.nn as nn
import torch.distributions as dd

log_rate = nn.Parameter(torch.Tensor([1])) # Define the log of the parameter as an nn.Parameter, this is what we want to optimise
rate = torch.exp(log_std) # Define the transformation we want to apply to the parameter to usi it in the distribution
concentration = nn.Parameter(torch.Tensor([1])) # A normal parameter
dist = dd.Gamma(concentration=concentration, rate=std) # Define the distribution. From here I want to ONLY refer to this, not the other variables

optim = torch.optim.SGD([log_rate, concentration], lr=0.01) # Standard optimiser
target = dd.Gamma(5,5) # Target distribution to match

for i in range(50):
    optim.zero_grad()

    samples = dist.rsample((1000,)) # Sample our model, note no reference to log_std

    cost = -(target.log_prob(samples) - dist.log_prob(samples)).sum() # KL divergence cost metric
    cost.backward()
    optim.step()
    print(i)
    print(log_std, mean, cost)
    print()

Looking at the code that currently works however:

import torch
import torch.nn as nn
import torch.distributions as dd

log_rate = nn.Parameter(torch.Tensor([1])) # Define the log of the parameter as an nn.Parameter, this is what we want to optimise
mean = nn.Parameter(torch.Tensor([1])) # A normal parameter

optim = torch.optim.SGD([log_rate, concentration], lr=0.001) # Standard optimiser
target = dd.Gamma(5,5) # Target distribution to match

for i in range(50):
    optim.zero_grad()

    rate = torch.exp(log_rate)  # Define the transformation we want to apply to the parameter to usi it in the distribution
    dist = dd.Gamma(concentration=concentration, rate=rate)  # Define the distribution.

    samples = dist.rsample((1000,)) # Sample our model, note no reference to log_std

    cost = -(target.log_prob(samples) - dist.log_prob(samples)).sum() # KL divergence cost metric
    cost.backward()
    optim.step()
    print(i)
    print(mean, std, cost)
    print()

And you see we have to change the code inside the loop to allow the algorithm to work. Its not a big issue in this small example, but this is just a demonstration for much larger algorithms where it would be incredibly beneficial to not have to worry about


title: torch.matmul calculation not exact in Python

body:
I encountered an astonishing problem. For a neural network, I multiply two matrices with matrixA.matmul(MatrixB). matrixA is of size 8x648 (8 Batches) and matrixB of 648x600. Although every batch is identical (which is to say that each row of matrixA is identical) the seventh and eighth row of the output matrix differ from the other six. Despite the fact that they only differ slightly, this has an immense impact on the results of my network. Now I wonder how this can be. Stunningly, the problem doesn't occur, if I calculate matrixA[0].matmul(matrixB) and matrixA[7].matmul(matrixB) seperately.
Does anybody know why?


title: Pytorch CNN with demonstrated architecture will learn, but accuracy won't increase

body:
My Pytorch CNN runs and learns but the accuracy doesn't improve. The train loss decreases, but when testing on the train set, it has random accuracy. I implemented the same architecture in Keras and I had 90%+ accuracy. I'm basically just wondering if there's something clearly wrong in my structure. Here it is:

    def __init__(self):
        super().__init__()

        self.conv1 = nn.Conv2d(1, 16, 3)
        self.conv2 = nn.Conv2d(16, 32, 3)
        self.conv3 = nn.Conv2d(32, 64, 3)

        self.fc1 = nn.Linear(5*5*64, 1024)
        self.fc2 = nn.Linear(1024, 2048)
        self.fc3 = nn.Linear(2048, 1)

    def forward(self, x):
        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
        x = F.max_pool2d(F.relu(self.conv2(x)), (2, 2))
        x = F.max_pool2d(F.relu(self.conv3(x)), (2, 2))

        x = x.view(x.size(0), -1)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.dropout(x, 0.5)
        x = F.sigmoid(self.fc3(x))
        return x

loss_function = nn.BCELoss()


title: Turn CNN model into class

body:
I am trying to build a CNN for multilabel classification in Pytorch (each image can have more than one label). So far I have built the model as follows:

model.fc = nn.Sequential(nn.Linear(2048, 512),
                                 nn.ReLU(),
                                 nn.Dropout(0.2),
                                 nn.Linear(512, 10),
                                 nn.LogSigmoid())
                                 # nn.LogSoftmax(dim=1))

criterion = nn.NLLLoss()
# criterion = nn.BCELoss()
optimizer = optim.Adam(model.fc.parameters(), lr=0.003)

But I would like to build it using a class like the following example:

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 10, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(10, 20, 5)
        self.fc1 = nn.Linear(20 * 22 * 39, 100)
        self.fc2 = nn.Linear(100, 50)
        self.fc3 = nn.Linear(50, 10)
        self.fc4 = nn.Linear(10, 3)

    def forward(self, x):
        x = x.view(-1, 3, 100, 170)
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 20 * 22 * 39)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.relu(self.fc3(x))
        return self.fc4(x)

What would be the best way of accomplishing this given that I am dealing with a multilabel classification problem? Any insights I would appreciate it.


title: How do I blend together tensors in patterns other than 2x2?

body:
I have created code that blends tensors together, by first blending them together into rows and then blending the rows together into the final output. It works well for 4 tensors in a 2x2 pattern, but it fails to do a 2x3 (6 tensors), 3x3 (9 tensors), 4x4 (16 tensors) pattern.

The tensors are in the form of (B x C x H x W), where B is batch size, C is channels, H is height, and W is width.

For both the tiles to rows (tile_overlay()), and rows to final image (row_overlay()), I create a base tensor that I add the tiles/rows to. I suspect the issue with my code lies either with how I get the base tensor's dimensions, how I track where to put the rows/tiles on the base tensor, or the issue with both of those things.

import torch
from PIL import Image
import torchvision.transforms as transforms

def preprocess(image_name, image_size):
    image = Image.open(image_name).convert('RGB')
    if type(image_size) is not tuple:
        image_size = tuple([int((float(image_size) / max(image.size))*x) for x in (image.height, image.width)])
    Loader = transforms.Compose([transforms.Resize(image_size), transforms.ToTensor()])
    tensor = (Loader(image) * 256).unsqueeze(0)
    return tensor

def deprocess(output_tensor):
    output_tensor = output_tensor.clone().squeeze(0).cpu() / 256
    output_tensor.clamp_(0, 1)
    Image2PIL = transforms.ToPILImage()
    image = Image2PIL(output_tensor.cpu())
    return image

def prepare_tile(tile, overlap, side='both'):
    lin_mask_left = torch.linspace(0,1,overlap).repeat(tile.size(3),1).repeat(3,1,1).unsqueeze(0)
    lin_mask_right = torch.linspace(1,0,overlap).repeat(tile.size(3),1).repeat(3,1,1).unsqueeze(0)
    if side == 'both' or side == 'right':
        tile[:,:,:,overlap:] = tile[:,:,:,overlap:] * lin_mask_right
    if side == 'both' or side == 'left':
        tile[:,:,:,:overlap] = tile[:,:,:,:overlap] * lin_mask_left
    return tile

def overlay_tiles(tile_list, rows, overlap):
    c = 1
    f_tiles = []
    base_length = 0
    for i, tile in enumerate(tile_list):
        if c == 1:
             f_tile = prepare_tile(tile.clone(), overlap, side='right')
             if i + 1<= rows[1]:
                 base_length += tile.clone().size(3) - overlap
        elif c == rows[1]:
             f_tile = prepare_tile(tile.clone(), overlap, side='left')
             if i + 1<= rows[1]:
                 base_length += tile.size(3) - overlap
        elif c > 0 and c < rows[1]:
             f_tile = prepare_tile(tile.clone(), overlap, side='both')
             if i + 1<= rows[1]:
                 base_length += tile.size(3) - (overlap*2)
        f_tiles.append(f_tile)
        if c == rows[1]:
             c = 0
        c+=1

    base_length += overlap
    base_tensor = torch.zeros(3, tile_list[0].size(2), base_length).unsqueeze(0)
    row_list = []
    for row in range(rows[1]):
        row_list.append(base_tensor.clone())

    row_val, num_tiles = 0, 0
    l_max = tile_list[0].size(3)
    for y in range(rows[0]):
        for x in range(rows[1]):
            if num_tiles % rows[1] != 0:
                l_max += (f_tiles[num_tiles].size(3)-overlap)*x
                l_min = l_max - f_tiles[num_tiles].size(3)
                row_list[row_val][:, :, :, l_min:l_max] = row_list[row_val][:, :, :, l_min:l_max] + f_tiles[num_tiles]
            else:
                row_list[row_val][:, :, :, :f_tiles[num_tiles].size(3)] = f_tiles[num_tiles]
                l_max = tile_list[0].size(3)
            num_tiles+=1
        row_val+=1
    return row_list

def prepare_row(row_tensor, overlap, side='both'):
    lin_mask_top = torch.linspace(0,1,overlap).repeat(row_tensor.size(3),1).rot90(3).repeat(3,1,1).unsqueeze(0)
    lin_mask_bottom = torch.linspace(1,0,overlap).repeat(row_tensor.size(3),1).rot90(3).repeat(3,1,1).unsqueeze(0)
    if side == 'both' or side == 'top':
        row_tensor[:,:,:overlap,:] = row_tensor[:,:,:overlap,:]  * lin_mask_top
    if side == 'both' or side == 'bottom':
        row_tensor[:,:,overlap:,:] = row_tensor[:,:,overlap:,:] * lin_mask_bottom
    return row_tensor

def overlay_rows(row_list, rows, overlap):
    c = 1
    f_rows = []
    base_height = 0
    for i, row_tensor in enumerate(row_list):
        if c == 1:
             f_row = prepare_row(row_tensor.clone(), overlap, side='bottom')
             if i + 1<= rows[0]:
                 base_height += row_tensor.size(2) - overlap
        elif c == rows[1]:
             f_row = prepare_row(row_tensor.clone(), overlap, side='top')
             if i + 1<= rows[0]:
                 base_height += row_tensor.size(2) - overlap
        elif c > 0 and c < rows[0]:
             f_row = prepare_row(row_tensor.clone(), overlap, side='both')
             if i + 1<= rows[0]:
                 base_height += tile.size(2) - (overlap*2)
        f_rows.append(f_row)
        if c == rows[0]:
             c = 0
        c+=1

    base_height += overlap
    base_tensor = torch.zeros(3, base_height, row_list[0].size(3)).unsqueeze(0)

    num_rows = 0
    l_max = row_list[0].size(3)
    for y in range(rows[0]):
            if num_rows > 0:
                l_max += (f_rows[num_rows].size(2)-overlap)*y
                l_min = l_max - f_rows[num_rows].size(2)
                base_tensor[:, :, l_min:l_max, :] = base_tensor[:, :, l_min:l_max, :] + f_rows[num_rows]
            else:
                base_tensor[:, :, :f_rows[num_rows].size(2), :] = f_rows[num_rows]
                l_max = row_list[0].size(2)
            num_rows+=1
    return base_tensor

def rebuild_image(tensor_list, rows, overlap_hw):
    row_tensors = overlay_tiles(tensor_list, rows, overlap_hw[1])
    full_tensor = overlay_rows(row_tensors, rows, overlap_hw[0])
    return full_tensor

test_tensor_1 = preprocess('brad_pitt.jpg', (1080,1080))
test_tensor_2 = preprocess('starry_night_google.jpg', (1080,1080))

tensor_list = [test_tensor_1.clone(), test_tensor_2.clone(), test_tensor_1.clone(), test_tensor_2.clone()]
rows = [2, 2]
overlap = [540, 540]
complete_tensor = rebuild_image(tensor_list, rows, overlap)

ft = deprocess(complete_tensor.clone())
ft.save('complete_tensor_2x2.png')

tensor_list = [test_tensor_1.clone(), test_tensor_2.clone(),test_tensor_1.clone(), \
test_tensor_1.clone(), test_tensor_2.clone(),test_tensor_1.clone(), \
test_tensor_1.clone(), test_tensor_2.clone(),test_tensor_1.clone(),]
rows = [3, 3]
overlap = [540, 540]
complete_tensor = rebuild_image(tensor_list, rows, overlap)

ft = deprocess(complete_tensor.clone())
ft.save('complete_tensor_3x3.png')

tensor_list = [test_tensor_1.clone(), test_tensor_2.clone(), test_tensor_1.clone(), test_tensor_2.clone(), \
test_tensor_1.clone(), test_tensor_2.clone(), test_tensor_1.clone(), test_tensor_2.clone(), \
test_tensor_1.clone(), test_tensor_2.clone(), test_tensor_1.clone(), test_tensor_2.clone(), \
test_tensor_1.clone(), test_tensor_2.clone(), test_tensor_1.clone(), test_tensor_2.clone()]
rows = [4, 4]
overlap = [540, 540]
complete_tensor = rebuild_image(tensor_list, rows, overlap)

ft = deprocess(complete_tensor.clone())
ft.save('complete_tensor_4x4.png')

Running the above code will result in this error message when trying to create the 3x3 output:

Traceback (most recent call last):
  File "t0.py", line 148, in <module>
    complete_tensor = rebuild_image(tensor_list, rows, overlap)
  File "t0.py", line 126, in rebuild_image
    row_tensors = overlay_tiles(tensor_list, rows, overlap_hw[1])
  File "t0.py", line 68, in overlay_tiles
    row_list[row_val][:, :, :, l_min:l_max] = row_list[row_val][:, :, :, l_min:l_max] + f_tiles[num_tiles]
RuntimeError: The size of tensor a (0) must match the size of tensor b (1080) at non-singleton dimension 3

This is an example of what the 2x2 output looks like:

And this is a visual diagram with two examples of what I am doing:


title: What will happen if I use batch normalzation but set batch_size=1?

body:
Because I am using 3D medical images as training dataset, the batch size can only be set to 1 because of GPU limitation. Some people told me that std dev will be 0 when batch_size = 1. And (x-mean)/std_dev will lead to error because of division by 0.

But why did errors not occur when I set batch_size = 1? Why my network was trained as good as I expected? Could anyone explain it?

Some people argued that:

  The ZeroDivisionError may not be encountered because of two cases. First, the exception is caught in a try catch block. Second, a small rational number is added ( 1e-19 ) to the variance term so that it is never zero.

But some people disagree. They said that:

  You should calculate mean and std across all pixels in the images of the batch. (So even batch_size = 1, there are still a lot of pixels in the batch. So the reason why batch_size=1 can still work is not because of 1e-19)

I have checked the pytorch source code, and from the code I think the latter one is right.

Does anyone have different opinion???


title: Can one hardcode convolutional filters to detect characters in a CNN?

body:
In Pytorch, you can hardcode your filters to be whatever you like.
At the moment, I'm doing text detection and I need to identify the location of a certain information. This information always starts with the letter 'X'. Could this radically improve detection performance if I hardcode the 'X' filter?
Here's what I have so far:
import torch
import torch.nn as nn
import matplotlib.pyplot as plt

kernel = (torch.zeros((9, 9)) + \
          torch.eye(9) + \
          torch.rot90(torch.eye(9))).type(torch.bool)*1

print(kernel)

tensor([[1, 0, 0, 0, 0, 0, 0, 0, 1],
        [0, 1, 0, 0, 0, 0, 0, 1, 0],
        [0, 0, 1, 0, 0, 0, 1, 0, 0],
        [0, 0, 0, 1, 0, 1, 0, 0, 0],
        [0, 0, 0, 0, 1, 0, 0, 0, 0],
        [0, 0, 0, 1, 0, 1, 0, 0, 0],
        [0, 0, 1, 0, 0, 0, 1, 0, 0],
        [0, 1, 0, 0, 0, 0, 0, 1, 0],
        [1, 0, 0, 0, 0, 0, 0, 0, 1]])

We can visualize it like this:
plt.imshow(kernel)
plt.show()

Then, we can set the filter weights as such:
conv = nn.Conv2d(in_channels=1,
                 out_channels=1,
                 kernel_size=3,
                 stride=3,
                 bias=None)

conv.weight.data = kernel


title: Is it possible to make the nodes and tress of MCTS work on GPU-only with PyTorch?

body:
I've viewed some discussion about MCTS and GPU. It's said there's no advantage using GPU, as it doesn't have many matrix-multiply. But it does have a drawback using CPU, as the data transfering between devices really takes time.
Here I mean the nodes and tree should be on GPU. Then they may process the data on GPU, without copying the the data from CPU. If I just create class node and tree, they will let their methods work on CPU.
So I wonder whether I can move the searching part to GPU. Is there any example?


title: Get value of variable index in particular dimension

body:
Say if i have a tensor that is

value = torch.tensor([

    [[0, 0, 0], [1, 1, 1]],
    [[2, 2, 2], [3, 3, 3]],
])

essentially with shape (2,2,3).

Now say if i have an index = [1, 0], which means I want to take:

# row 1 of [[0, 0, 0], [1, 1, 1]], giving me: [1, 1, 1]
# row 0 of [[2, 2, 2], [3, 3, 3]], giving me: [2, 2, 2]

So that the final output:

output = torch.tensor([[1, 1, 1], [2, 2, 2]])

is there a vectorized way to achieve this?


title: Pytorch crashes on input in eval mode

body:
My model trains perfectly fine, but when I switch it to evaluation mode it does not like the data types of the input samples:

Traceback (most recent call last):
  File "model.py", line 558, in <module>
    main_function(train_sequicity=args.train)
  File "model.py", line 542, in main_function
    out = model(user, bspan, response_, degree)
  File "/home/memduh/git/project/venv/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "model.py", line 336, in forward
    self.params['bspan_size'])
  File "model.py", line 283, in _greedy_decode_output
    out = decoder(input_, encoder_output)
  File "/home/memduh/git/project/venv/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "model.py", line 142, in forward
    tgt = torch.cat([go_tokens, tgt], dim=0)  # concat GO_2 token along sequence lenght axis
RuntimeError: Expected object of scalar type Long but got scalar type Float for sequence element 1 in sequence argument at position #1 'tensors'

This seems to occur in a part of the code where concatenation happens. This is in an architecture similar to the pytorch transformer, just modified to have two decoders:

      def forward(self, tgt, memory):
          """ Call decoder
          the decoder should be called repeatedly

          Args:
              tgt: input to transformer_decoder, shape: (seq, batch)
              memory: output from the encoder

          Returns:
              output from linear layer, (vocab size), pre softmax

          """
          go_tokens = torch.zeros((1, tgt.size(1)), dtype=torch.int64) + 3  # GO_2 token has index 3

          tgt = torch.cat([go_tokens, tgt], dim=0)  # concat GO_2 token along sequence lenght axis

+
          mask = tgt.eq(0).transpose(0,1)  # 0 corresponds to <pad>
          tgt = self.embedding(tgt) * self.ninp
          tgt = self.pos_encoder(tgt)
          tgt_mask = self._generate_square_subsequent_mask(tgt.size(0))
          output = self.transformer_decoder(tgt, memory, tgt_mask=tgt_mask, tgt_key_padding_mask=mask)
          output = self.linear(output)
          return output

The concatenation bit in the middle of the codeblock is where the problem happens. The odd thing is that it works perfectly fine and trains, with loss going down in train mode. This issue only comes up in eval mode. What could the problem be?


title: How to concatenate a tensor WITHIN axis=1?

body:
I have a tensor of shape(2,2,2,2):

tensor([[[[   5.,    5.],
          [   5.,    5.]],

         [[  10.,   10.],
          [  10.,   10.]]],

        [[[ 100.,  100.],
          [ 100.,  100.]],

         [[1000., 1000.],
          [1000., 1000.]]]], device='cuda:0')

I want to transform it such that the tensor along axis=1 are repeated 3 times. And after applying .view(-1) to that I get a 1D resultant tensor as:

tensor([   5.,    5.,    5.,    5.,   5.,    5.,    5.,    5.,   5.,    5.,    5.,    5.,   10.,   10.,   10.,   10.,   10.,   10.,   10.,   10.,   10.,   10.,   10.,   10.,  100.,  100.,  100.,  100.,  100.,  100., 100.,  100.,  100.,  100., 100.,  100.,  100.,  100.,  100.,  100., 1000., 1000., 1000., 1000.  1000., 1000., 1000., 1000. 1000., 1000., 1000., 1000. 1000., 1000., 1000., 1000.], device='cuda:0')

How to do this?


title: Pytorch merging list of tensors together

body:
Let's say I have a list of tensors ([A , B , C ] where each tensor of is of shape [batch_size X 1024].
I want to merge all the tensors into a single tensor in the following way :
The first row in A is the first row in the new tensor, and the first row of B is the seocnd row in the new tensor, and the first row of C is the third row of the new tensor and so on and so forth.
So far I did it with for loops and this is not effictive at all.
Would love to hear about more efficent ways.

Thanks


title: How can I solve the problem of pytorch program?

body:
I want to run python program using pytorch with my own dataset.I come across with the error:
Traceback (most recent call last):
  File "C:\Users\hp\Downloads\efficient_densenet_pytorch-master\demoEmotion.py", line 345, in
    fire.Fire(demo)
  File "C:\Users\hp\Anaconda3\envs\tf-gpu\lib\site-packages\fire\core.py", line 138, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "C:\Users\hp\Anaconda3\envs\tf-gpu\lib\site-packages\fire\core.py", line 468, in _Fire
    target=component.name)
  File "C:\Users\hp\Anaconda3\envs\tf-gpu\lib\site-packages\fire\core.py", line 672, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "C:\Users\hp\Downloads\efficient_densenet_pytorch-master\demoEmotion.py", line 323, in demo
    n_epochs=n_epochs, batch_size=batch_size, seed=seed)
  File "C:\Users\hp\Downloads\efficient_densenet_pytorch-master\demoEmotion.py", line 202, in train
    n_epochs=n_epochs,
  File "C:\Users\hp\Downloads\efficient_densenet_pytorch-master\demoEmotion.py", line 83, in train_epoch
    output = model(input)
  File "C:\Users\hp\Anaconda3\envs\tf-gpu\lib\site-packages\torch\nn\modules\module.py", line 555, in call
    result = self.forward(*input, **kwargs)
  File "C:\Users\hp\Downloads\efficient_densenet_pytorch-master\models\densenet.py", line 151, in forward
    features = self.features(x)
  File "C:\Users\hp\Anaconda3\envs\tf-gpu\lib\site-packages\torch\nn\modules\module.py", line 555, in call
    result = self.forward(*input, **kwargs)
  File "C:\Users\hp\Anaconda3\envs\tf-gpu\lib\site-packages\torch\nn\modules\container.py", line 100, in forward
    input = module(input)
  File "C:\Users\hp\Anaconda3\envs\tf-gpu\lib\site-packages\torch\nn\modules\module.py", line 555, in call
    result = self.forward(*input, **kwargs)
  File "C:\Users\hp\Anaconda3\envs\tf-gpu\lib\site-packages\torch\nn\modules\container.py", line 100, in forward
    input = module(input)
  File "C:\Users\hp\Anaconda3\envs\tf-gpu\lib\site-packages\torch\nn\modules\module.py", line 555, in call
    result = self.forward(*input, **kwargs)
  File "C:\Users\hp\Anaconda3\envs\tf-gpu\lib\site-packages\torch\nn\modules\pooling.py", line 557, in forward
    self.padding, self.ceil_mode, self.count_include_pad, self.divisor_override)
RuntimeError: Given input size: (150x1x1). Calculated output size: (150x0x0). Output size is too small
Please guide me how to sove this problem.Thanks in advance!


title: PyTorch LSTM dimension

body:
I am trying to teach my first LSTM with numeric sequence data to predict a singular value.

My training set is a numpy matrix shape=207x7 where each input vector has 7 features.
I am not sure how to setup and train my LSTM properly. I have CNN experience, but now first LSTM.

class LSTM_NN(nn.Module):
"""
Stores the network format
"""
def __init__(self, input_size=7, hidden_layer_size=100, output_size=1):
    super(self.__class__, self).__init__()
    self._hidden_layer_size = hidden_layer_size
    self._lstm = nn.LSTM(input_size, hidden_layer_size)
    self._hidden_cell = (torch.zeros(1, 1, self._hidden_layer_size),
                         torch.zeros(1, 1, self._hidden_layer_size))
    self._linear = nn.Linear(hidden_layer_size, output_size)

def forward(self, input_data):
    """
    Forward propagation
    """

    lstm_out, self._hidden_cell = self._lstm(
        torch.tensor(np.expand_dims(input_data, 0)),
        self._hidden_cell)

training_data.shape
# (200, 7)
model = LSTM_NN(input_size=training_data.shape[1])
model.forward(training_data)

But I get this error:

Expected hidden[0] size (1, 200, 100), got (1, 1, 100)
  File "train_lstm.py", line 44, in forward


title: i have an error while training BERT model

body:
I'm trying to train my model using GPUs.
When I execute it I get this error below:
File "main.py", line 95, in train loss.backward()
File "/opt/conda/lib/python3.7/site-packages/torch/tensor.py", line 198, in backward torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "/opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py", line 100, in backward allow_unreachable=True)  # allow_unreachable flag
RuntimeError: transform: failed to synchronize: cudaErrorIllegalAddress: an illegal memory access was encountered
# Added identdation for better readability

This error occurs when i use single or multiple GPUs.
How can i deal with this issue? Thank you
Here is the code of the BERT model (no segment input, no pretraining)
c.f. When I use another model, this issue didn't occure.
"""code"""
import math
import torch
import torch.nn as nn
import torch.nn.functional as F

class HateClassification(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.config = config

        self.bert = BERT(self.config)
        # classfier
        self.projection_cls = nn.Linear(self.config.d_hidn, self.config.n_output, bias=False)
        self.sigmoid = nn.Sigmoid()

    def forward(self, inputs):
        inputs = torch.transpose(inputs,0,1)
        # (bs, n_enc_seq, d_hidn), (bs, d_hidn), [(bs, n_head, n_enc_seq, n_enc_seq)]
        outputs, outputs_cls, attn_probs = self.bert(inputs)
        # (bs, n_output)
        logits_cls = self.sigmoid(self.projection_cls(outputs_cls))
        # (bs, n_output), [(bs, n_head, n_enc_seq, n_enc_seq)]
        return logits_cls.squeeze()
""" bert """
class BERT(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.config = config

        self.encoder = Encoder(self.config)

        self.linear = nn.Linear(config.d_hidn, config.d_hidn)
        self.activation = torch.tanh

    def forward(self, inputs):
        # (bs, n_seq, d_hidn), [(bs, n_head, n_enc_seq, n_enc_seq)]
        outputs, self_attn_probs = self.encoder(inputs)
        # (bs, d_hidn)
        outputs_cls = outputs[:, 0].contiguous()
        outputs_cls = self.linear(outputs_cls)
        outputs_cls = self.activation(outputs_cls)
        # (bs, n_enc_seq, n_enc_vocab), (bs, d_hidn), [(bs, n_head, n_enc_seq, n_enc_seq)]
        return outputs, outputs_cls, self_attn_probs

    def save(self, epoch, loss, path):
        torch.save({
            "epoch": epoch,
            "loss": loss,
            "state_dict": self.state_dict()
        }, path)

    def load(self, path):
        save = torch.load(path)
        self.load_state_dict(save["state_dict"])
        return save["epoch"], save["loss"]
""" encoder """
class Encoder(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.config = config

        self.enc_emb = nn.Embedding(self.config.n_enc_vocab, self.config.d_hidn)
        self.pos_emb = nn.Embedding(self.config.n_enc_seq + 1, self.config.d_hidn)

        self.layers = nn.ModuleList([EncoderLayer(self.config) for _ in range(self.config.n_layer)])

    def forward(self, inputs):
        positions = torch.arange(inputs.size(1), device=inputs.device, dtype=inputs.dtype).expand(inputs.size(0), inputs.size(1)).contiguous() + 1
        pos_mask = inputs.eq(self.config.i_pad)
        positions.masked_fill_(pos_mask, 0)

        # (bs, n_enc_seq, d_hidn)
        outputs = self.enc_emb(inputs) + self.pos_emb(positions)

        # (bs, n_enc_seq, n_enc_seq)
        attn_mask = get_attn_pad_mask(inputs, inputs, self.config.i_pad)

        attn_probs = []
        for layer in self.layers:
            # (bs, n_enc_seq, d_hidn), (bs, n_head, n_enc_seq, n_enc_seq)
            outputs, attn_prob = layer(outputs, attn_mask)
            attn_probs.append(attn_prob)
        # (bs, n_enc_seq, d_hidn), [(bs, n_head, n_enc_seq, n_enc_seq)]
        return outputs, attn_probs
""" encoder layer """
class EncoderLayer(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.config = config

        self.self_attn = MultiHeadAttention(self.config)
        self.layer_norm1 = nn.LayerNorm(self.config.d_hidn, eps=self.config.layer_norm_epsilon)
        self.pos_ffn = PoswiseFeedForwardNet(self.config)
        self.layer_norm2 = nn.LayerNorm(self.config.d_hidn, eps=self.config.layer_norm_epsilon)

    def forward(self, inputs, attn_mask):
        # (bs, n_enc_seq, d_hidn), (bs, n_head, n_enc_seq, n_enc_seq)
        att_outputs, attn_prob = self.self_attn(inputs, inputs, inputs, attn_mask)
        att_outputs = self.layer_norm1(inputs + att_outputs)
        # (bs, n_enc_seq, d_hidn)
        ffn_outputs = self.pos_ffn(att_outputs)
        ffn_outputs = self.layer_norm2(ffn_outputs + att_outputs)
        # (bs, n_enc_seq, d_hidn), (bs, n_head, n_enc_seq, n_enc_seq)
        return ffn_outputs, attn_prob

""" attention pad mask """
def get_attn_pad_mask(seq_q, seq_k, i_pad):
    batch_size, len_q = seq_q.size()
    batch_size, len_k = seq_k.size()
    pad_attn_mask = seq_k.data.eq(i_pad)
    pad_attn_mask= pad_attn_mask.unsqueeze(1).expand(batch_size, len_q, len_k)
    return pad_attn_mask
""" multi head attention """
class MultiHeadAttention(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.config = config

        self.W_Q = nn.Linear(self.config.d_hidn, self.config.n_head * self.config.d_head)
        self.W_K = nn.Linear(self.config.d_hidn, self.config.n_head * self.config.d_head)
        self.W_V = nn.Linear(self.config.d_hidn, self.config.n_head * self.config.d_head)
        self.scaled_dot_attn = ScaledDotProductAttention(self.config)
        self.linear = nn.Linear(self.config.n_head * self.config.d_head, self.config.d_hidn)
        self.dropout = nn.Dropout(config.dropout)

    def forward(self, Q, K, V, attn_mask):
        batch_size = Q.size(0)
        # (bs, n_head, n_q_seq, d_head)
        q_s = self.W_Q(Q).view(batch_size, -1, self.config.n_head, self.config.d_head).transpose(1,2)
        # (bs, n_head, n_k_seq, d_head)
        k_s = self.W_K(K).view(batch_size, -1, self.config.n_head, self.config.d_head).transpose(1,2)
        # (bs, n_head, n_v_seq, d_head)
        v_s = self.W_V(V).view(batch_size, -1, self.config.n_head, self.config.d_head).transpose(1,2)

        # (bs, n_head, n_q_seq, n_k_seq)
        attn_mask = attn_mask.unsqueeze(1).repeat(1, self.config.n_head, 1, 1)

        # (bs, n_head, n_q_seq, d_head), (bs, n_head, n_q_seq, n_k_seq)
        context, attn_prob = self.scaled_dot_attn(q_s, k_s, v_s, attn_mask)
        # (bs, n_head, n_q_seq, h_head * d_head)
        context = context.transpose(1, 2).contiguous().view(batch_size, -1, self.config.n_head * self.config.d_head)
        # (bs, n_head, n_q_seq, e_embd)
        output = self.linear(context)
        output = self.dropout(output)
        # (bs, n_q_seq, d_hidn), (bs, n_head, n_q_seq, n_k_seq)
        return output, attn_prob
""" feed forward """
class PoswiseFeedForwardNet(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.config = config

        self.conv1 = nn.Conv1d(in_channels=self.config.d_hidn, out_channels=self.config.d_ff, kernel_size=1)
        self.conv2 = nn.Conv1d(in_channels=self.config.d_ff, out_channels=self.config.d_hidn, kernel_size=1)
        self.active = F.gelu
        self.dropout = nn.Dropout(config.dropout)

    def forward(self, inputs):
        # (bs, d_ff, n_seq)
        output = self.conv1(inputs.transpose(1, 2))
        output = self.active(output)
        # (bs, n_seq, d_hidn)
        output = self.conv2(output).transpose(1, 2)
        output = self.dropout(output)
        # (bs, n_seq, d_hidn)
        return output
""" scale dot product attention """
class ScaledDotProductAttention(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.config = config
        self.dropout = nn.Dropout(config.dropout)
        self.scale = 1 / (self.config.d_head ** 0.5)

    def forward(self, Q, K, V, attn_mask):
        # (bs, n_head, n_q_seq, n_k_seq)
        scores = torch.matmul(Q, K.transpose(-1, -2))
        scores = scores.mul_(self.scale)
        scores.masked_fill_(attn_mask, -1e9)
        # (bs, n_head, n_q_seq, n_k_seq)
        attn_prob = nn.Softmax(dim=-1)(scores)
        attn_prob = self.dropout(attn_prob)
        # (bs, n_head, n_q_seq, d_v)
        context = torch.matmul(attn_prob, V)
        # (bs, n_head, n_q_seq, d_v), (bs, n_head, n_q_seq, n_v_seq)
        return context, attn_prob


title: Properly shaping LSTM input/output for text sequence prediction?

body:
I'm trying to learn how to use an LSTM to predict the next character based on the previous character. I've successfully created a function that batches a long string of text (after one-hot encoding) into shape (batch_size, seq_len, one_hot_features). one_hot_features is simply the number of unique characters in the text.
Since my batching function gives me both training sequences and 'ground truth' sequences, I need the output of the LSTM network to be the same shape as the 'ground truth' batches so that I can plug both output and labels into the loss function.
So my questions are:

How do I correctly define my network architecture to meet the above requirements.
How do I shape my data as it passes through the various layers in the forward function?

This is my attempt, and the code will run, but the output of the network is incorrectly shaped for comparison to the 'ground truth' sequence batches:
class charNN(nn.Module):
    def __init__(self, vocab, hidden_size, n_layers, dropout=0.5):
        super().__init__()

        self.vocab_length = len(vocab)
        self.hidden_size = hidden_size
        self.lstm = nn.LSTM(self.vocab_length, hidden_size, n_layers, batch_first=True)
        self.fc   = nn.Linear(hidden_size, self.vocab_length)

    def forward(self, x, hidden=None):
        x, hidden = self.lstm(x, hidden)     # -> (n_batches, seq_len, hidden_size)
        x = x.reshape(-1, self.hidden_size)  # -> (n_batches * seq_len, hidden_size)
        x = self.fc(x)                       # -> (n_batches * seq_len, vocab_length)

        return x, hidden

Thank you in advance. I've been struggling with this one for a couple days. I will also happily provide the code for training the network, if it is required. It does not throw any errors except to tell me that the output size and the true label sizes do not match.


title: Reinforcement learning actor predicting same actions during initial training

body:
I have a reinforcement learning Actor Critic model with lstm.
During initial training it is giving same action value for all the states.
Can someone expert in AI/RL please help to let me know if this is normal behavior during training?
Also can you please help to let me know what should be the ideal size of lstm and linear layers if I have a state_dimension = 50 and action_dimension = 3.
Thanks in advance


title: How to make a 3-d like visualization of a CNN model developed using pytorch?

body:
Currently, I have a CNN model that I developed in Pytorch. I have used hiddenlayer package to create an image like shown in image 1. But, I want to create an image of the model that should look like something in image 2. Is there any package that I can use to achieve this?
image 1:

image 2:


title: Why my CNN network can't learn high level features

body:
I am trying to predict thousands of images like the below one via CNN network. Following you can see CNN architecture and some of the details. I am wondering why the network is not learning properly? As you can see the predicted image values(contour) are far from the ground truth (predicted values are 10 times more). It seems the network is doing fine in edge detection (low-level features) but not in high-level features.
kernel = 3
num_filters = 12
batch_size = 128
lr = 1e-5

class Model(nn.Module):
    def __init__(self, kernel, num_filters, res = ResidualBlock):
        super(Model, self).__init__()

        self.conv0 = nn.Sequential(
            nn.Conv2d(4, num_filters, kernel_size = kernel*3,
                       padding = 4),
            nn.BatchNorm2d(num_filters),
            nn.ReLU(inplace=True))

        self.conv1 = nn.Sequential(
            nn.Conv2d(num_filters, num_filters*2, kernel_size = kernel,
                      stride=2, padding = 1),
            nn.BatchNorm2d(num_filters*2),
            nn.ReLU(inplace=True))

        self.conv2 = nn.Sequential(
            nn.Conv2d(num_filters*2, num_filters*4, kernel_size = kernel, stride=2, padding = 1),
            nn.BatchNorm2d(num_filters*4),
            nn.ReLU(inplace=True))

        self.conv3 = nn.Sequential(
            nn.Conv2d(num_filters*4, num_filters*8, kernel_size = kernel, stride=2, padding = 2),
            nn.BatchNorm2d(num_filters*8),
            nn.ReLU(inplace=True))

        self.conv4 = nn.Sequential(
            nn.Conv2d(num_filters*8, num_filters*16, kernel_size = kernel, stride=2, padding = 1),
            nn.BatchNorm2d(num_filters*16),
            nn.ReLU(inplace=True))

        self.tsconv0 = nn.Sequential(
            nn.ConvTranspose2d(num_filters*16, num_filters*8, kernel_size = kernel, padding =1),
            nn.Upsample(scale_factor=2, mode='bilinear', align_corners=True),
            nn.ReLU(inplace=True),
            nn.BatchNorm2d(num_filters*8))

        self.tsconv1 = nn.Sequential(
            nn.ConvTranspose2d(num_filters*8, num_filters*4, kernel_size = kernel, padding = 1),
            nn.Upsample(scale_factor=2, mode='bilinear', align_corners=True),
            nn.ReLU(inplace=True),
            nn.BatchNorm2d(num_filters*4))

        self.tsconv2 = nn.Sequential(
            nn.ConvTranspose2d(num_filters*4, num_filters*2, kernel_size = kernel, padding = 1),
            nn.Upsample(scale_factor=2, mode='bilinear', align_corners=True),
            nn.ReLU(inplace=True),
            nn.BatchNorm2d(num_filters*2))

        self.tsconv3 = nn.Sequential(
            nn.ConvTranspose2d(num_filters*2, num_filters, kernel_size = kernel, padding = 1),
            nn.Upsample(scale_factor=2, mode='bilinear', align_corners=True),
            nn.ReLU(inplace=True),
            nn.BatchNorm2d(num_filters))

        self.tsconv4 = nn.Sequential(
            nn.Conv2d(num_filters, 1, kernel_size = kernel*3, padding = 0, bias=False),
            nn.ReLU(inplace=True))

    def forward(self, x):
        x0 = self.conv0(x)           #([6, 600, 600])
        # print(x0.shape)
        x1 = self.conv1(x0)          #([12, 300, 300])
        # print(x1.shape)
        x2 = self.conv2(x1)          #([24, 150, 150])
        # print(x2.shape)
        x3 = self.conv3(x2)          #([48, 76, 76])
        # print(x3.shape)
        x4 = self.conv4(x3)          #([96, 38, 38])
        # print(x4.shape)
        x5 = self.tsconv0(x4)        #([48, 76, 76])
        # print(x5.shape)
        x6 = self.tsconv1(x5)        #([24, 152, 152])
        # print(x6.shape)
        x7 = self.tsconv2(x6)        #([12, 304, 304)
        # print(x7.shape)
        x8 = self.tsconv3(x7)        #([6, 608, 608])
        # print(x8.shape)
        x9 = self.tsconv4(x8)        #([1, 600, 600])
        # print(x9.shape)
        return x9


title: Pytorch and batches

body:
I'm having trouble understanding how batches play a role into the Pytorch framework.
In this model:
class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()

        # 28x28x1 => 26x26x32
        self.conv1 = nn.Conv2d(in_channels=1, out_channels=32, kernel_size=3)
        self.d1 = nn.Linear(26 * 26 * 32, 128)
        self.d2 = nn.Linear(128, 10)

    def forward(self, x):
        # 32x1x28x28 => 32x32x26x26
        x = self.conv1(x)
        x = F.relu(x)

        # flatten => 32 x (32*26*26)
        x = x.flatten(start_dim = 1)
        #x = x.view(32, -1)

        # 32 x (32*26*26) => 32x128
        x = self.d1(x)
        x = F.relu(x)

        # logits => 32x10
        logits = self.d2(x)
        out = F.softmax(logits, dim=1)
        return out

In the forward definition, we pass in some x, ie. aggregated images for a batch from a DataLoader. Here, the 32x1x28x28 dimension indicates that there are 32 images in a batch. Do we just ignore this fact and Pytorch handles applying Conv2d to each sample? The forward propagation seems to be just relative to a single image.


title: How to display Detectron2's bounding box on an image in Android

body:
This is probably a very generic question.
I want to know how to display a bounding box & its segmentation, on an image in Android, using Java. Is there any standardized way to approach this problem?
My output from Detectron2 is something like this:
Boxes(tensor([[ 20.3178, 0.0000, 219.9704, 166.3912],
              [ 35.7135, 0.0000, 214.5163, 163.8396]]))


title: How to include loading weights for a separate module in the torchscript?

body:
I have two weights files. One is the main network, but the second is a part of the main network - feature extraction layers.
So when I tried to debug and find the reason why the results of my Python and C++ script differed, I found out that the enc0 and other are fully zeros, but in the Python script not all values are zeros. My opinion is that torchscript does not save the load weights function, because of only forward scripting in the part of network.
How can I correctly load and script that?
class FPN(nn.Module):

    def __init__(self, norm_layer, num_filters=256):
        super().__init__()

        self.inception = inceptionresnetv2(num_classes=1000, pretrained='imagenet')

        self.enc0 = self.inception.conv2d_1a
        self.enc1 = nn.Sequential(
            self.inception.conv2d_2a,
            self.inception.conv2d_2b,
            self.inception.maxpool_3a,
        ) # 64
        ...
    def forward(self, x):
        enc0 = self.enc0(x)
        enc1 = self.enc1(enc0)
        ...


title: which value is used for padding in tensorflow convolution?

body:
In PyTorch, there are two choices for padding_mode when using convolution: zeros and circular.
However, I cannot find a similar parameter in the docs of TensorFlow.
In addition, I find that TensorFlow is not using 0 as the value to pad the inputs.
Does anyone know the answer?


title: PyTorch CNN unable to initialize due to excessive memory consumption

body:
I'm currently attempting to utilize a convolutional neural network to classify demented and non-demented MRI scans. All of the code shown below has been run on using a Core i7 CPU, 8.00 GB of RAM, and a GTX 1060 GPU. After sifting through journal articles and studies regarding the use of machine learning to conduct similar tasks, I've seen that it will require a highly complex model to see any success. I've attached the code for the CNN below.
class Hl_Model(nn.Module):

torch.cuda.set_device(0)

def __init__(self):
    super().__init__()

    self.conv1 = nn.Conv2d(1, 64, 3, stride=1)
    self.conv2 = nn.Conv2d(64, 64, 3, stride=1)
    self.conv3 = nn.Conv2d(64, 128, 3, stride=1)
    self.conv4 = nn.Conv2d(128, 128, 3, stride=1)
    self.conv5 = nn.Conv2d(128, 256, 3, stride=1)
    self.conv6 = nn.Conv2d(256, 256, 3, stride=1)
    self.conv7 = nn.Conv2d(256, 256, 3, stride=1)
    self.conv8 = nn.Conv2d(256, 256, 3, stride=1)
    self.conv9 = nn.Conv2d(256, 512, 3, stride=1)
    self.conv10 = nn.Conv2d(512, 512, 3, stride=1)

    x = torch.randn(216,144).view(-1,1,216,144)
    self._to_linear = None
    self.convs(x)

    self.fc1 = nn.Linear(self._to_linear, 512, bias=True)
    self.fc2 = nn.Linear(512, 256)
    self.fc3 = nn.Linear(256, 128, bias=True)
    self.fc4 = nn.Linear(128, 3)

def convs(self, x):

    x = F.relu(self.conv1(x))
    x = F.max_pool2d(F.relu(self.conv2(x)), (2, 2), stride=1)
    x = F.relu(self.conv3(x))
    x = F.max_pool2d(F.relu(self.conv4(x)), (2, 2), stride=1)
    x = F.relu(self.conv5(x))
    x = F.relu(self.conv6(x))
    x = F.relu(self.conv7(x))
    x = F.relu(self.conv8(x))
    x = F.relu(self.conv9(x))
    x = F.max_pool2d(F.relu(self.conv10(x)), (2,2), stride=1)

    if self._to_linear is None:
        self._to_linear = x[0].shape[0]*x[0].shape[1]*x[0].shape[2]
    return x

def forward(self, x):
    x = self.convs(x)
    x = x.view(-1, self._to_linear)
    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(x))
    x = F.relu(self.fc3(x))
    x = self.fc4(x)
    return F.softmax(x, dim=1)

When ran, the program quickly fails due to excessive memory consumption and returns the following error.
Traceback (most recent call last):
  File "c:\Users\Jack Diskin\Documents\OASIS_CNN.py", line 114, in <module>
    hl_model = Hl_Model()
  File "c:\Users\Jack Diskin\Documents\OASIS_CNN.py", line 77, in __init__
    self.fc1 = nn.Linear(self._to_linear, 512, bias=True)
  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\linear.py", line 72, in __init__
    self.weight = Parameter(torch.Tensor(out_features, in_features))
RuntimeError: [enforce fail at ..\c10\core\CPUAllocator.cpp:72] data. DefaultCPUAllocator: not enough
memory: you tried to allocate 24487395328 bytes. Buy new RAM!

However, in studies such as "Multi-Grade Brain Tumor Classification using Deep
CNN with Extensive Data Augmentation" (Sajjad et al, 2018), the authors use a significantly more complex architecture with the same 8 GB of RAM possessed by my computer. For this reason, I'm confused as to why my model will not run.


title: TensorFlow 2.4.0 - Parameters associated with BatchNorm and Activation

body:
I am printing a tensorflow.keras.Model instance summary. The type is tensorflow.python.keras.engine.functional.Functional object.
This model has layers with activations and batch normalization associated. When I print the list of parameters, I see

weights
bias
4 items co-dimensional with the bias

These four items are (I guess) the batch normalization and activations.
My question is: why do we have parameters associated with batch normalization and activations? And what could be the other two items?

My aim is to transpose this Keras model to a PyTorch counterpart, so I need to know the order of the parameters and what these parameters represent


title: Difference between CrossEntropyLoss and NNLLoss with log_softmax in PyTorch?

body:
When I am building a classifier in PyTorch, I have 2 options to do

Using the nn.CrossEntropyLoss without any modification in the model
Using the nn.NNLLoss with F.log_softmax added as the last layer in the model

So there are two approaches.
Now, what approach should anyone use, and why?


title: How to stretch and squeeze an image in PyTorch?

body:
I am working on MNIST Digit Data and looking for a way to either stretch or squeeze the image. The data is of the shape (28,28) and I want the final image to be of the same shape but either stretched or squeezed.
Is there any way to do it using torchvision.transforms or using any other method?


title: Updating weights manually in Pytorch

body:
import torch
import math

# Create Tensors to hold input and outputs.
x = torch.linspace(-math.pi, math.pi, 2000)
y = torch.sin(x)

# For this example, the output y is a linear function of (x, x^2, x^3), so
# we can consider it as a linear layer neural network. Let's prepare the
# tensor (x, x^2, x^3).
p = torch.tensor([1, 2, 3])
xx = x.unsqueeze(-1).pow(p)

model = torch.nn.Sequential(
    torch.nn.Linear(3, 1),
    torch.nn.Flatten(0, 1)
)

loss_fn = torch.nn.MSELoss(reduction='sum')

learning_rate = 1e-6

Then I print the weights
parameters = list(model.parameters())
print(parameters)

Results:
[Parameter containing:
tensor([[ 0.0407,  0.2680, -0.1148]], requires_grad=True), Parameter containing:
tensor([-0.0132], requires_grad=True)]

y_pred = model(xx)
loss = loss_fn(y_pred, y)
model.zero_grad()
loss.backward()

Updating the weights:
with torch.no_grad():
    for param in model.parameters():
        param -= 1e-6 * param.grad

and then
list(model.parameters())
[Parameter containing:
 tensor([[ 0.0532,  0.2472, -0.0393]], requires_grad=True),
 Parameter containing:
 tensor([-0.0167], requires_grad=True)]

The weights were updated. I got confused. How is that possible? I thought just param variable in the for loop changed, not model.parameters().
But when change the code a bit
with torch.no_grad():
    for param in model.parameters():
        param -= 1e-6

The weights didn't change. So I guess it related to param.grad. Can you guys explain to me about that?


title: LSTM Autoencoder Output Layer

body:
I am trying to get a LSTM autoencoder to recreate its inputs. So far I have:
class getSequence(nn.Module):
    def forward(self, x):
        out, _ = x
        return out

class getLast(nn.Module):
    def forward(self, x):
        out, states = x
        states = states[len(states) - 1]
        return states

class AEncoder(nn.Module):
    def __init__(self, input_size, first_layer, second_layer, n_layers):
        super(AEncoder, self).__init__()
        self.n_layers = n_layers
        self.encode = nn.Sequential(nn.LSTM(input_size, first_layer, batch_first=True),
                                    getSequence(),
                                    nn.ReLU(True),
                                    nn.LSTM(first_layer, second_layer),
                                    getLast())
        self.decode = nn.Sequential(nn.LSTM(second_layer, first_layer),
                                    getSequence(),
                                    nn.ReLU(True),
                                    nn.LSTM(first_layer, input_size),
                                    getSequence())

    def forward(self, x):
        x = x.float()
        x = self.encode(x)
        x = x.repeat(32, 1, 1) # repeating last hidden state of self.encode
        x = self.decode(x)
        return x

While researching I have been seeing some people adding a time-distributed dense layer at the end of the self.decode. I am confused if that final layer is specific to other tasks autoencoders are used for, if so, can I ignore that layer if I am only trying to recreate inputs?


title: The reason behind rgb image normalization parameters in pytorch

body: I've seen transforms.Normalize((0.485, 0.456, 0.406), (0.229, 0.224, 0.225)) both in lots of tutorials and pytorch docs, I know the first param is mean and the second one is std. I can't understand why the values for different channels differ.


title: Using PyTorch Transformer Encoder Block for Unsupervised Clustering

body:
I am trying to perform an unsupervised clustering task on a music dataset (notes represented as integers). I want to use a Transformer to encode relatedness between different songs before using k-means to cluster them, and I'm a little stuck on how to best approach this.
My initial idea was to take the output of the encoder layer and apply the clustering method directly, but the dimensions of each vector increases dramatically (from 100 to 10000) from input to output, and this would not be suitable for clustering. I'm not sure if this is my own implementation error, but I am wondering the following:

What should the dimensions of the encoder layer output be relative to the input
What do these outputs actually mean? (I have only found explanations addressing the decoder layer outputs)
Would it be appropriate to cluster these outputs directly or should I apply other functions first?


title: PyTorch - indices of corresponding values in another tensor

body:
I have a tensor, where I want to copy only some of the values (columnwise). The same values are in another tensor but in a random order. What I want, are the column indices from tensor2 of the values of tensor1. Here is an example:
copy_ind = torch.tensor([0, 1, 3], dtype=torch.long)
tensor1 = torch.tensor([[4, 6, 5, 1, 8],[10, 0, 8, 2, 1]])
temp = torch.index_select(tensor1, 1, copy_ind) # values to copy
tensor2 = torch.tensor([[1, 4, 5, 6, 8],[2, 10, 8, 0, 1]], dtype=torch.long)
_, t_ind = torch.sort(temp[0], dim=0)
t2_ind = copy_ind[t_ind] # indices of tensor2

The output should be:
t2_ind = [1, 3, 0]

Here is another example where I want to get the values of the tensor according to c1_new:
c1 = torch.tensor([[6, 7, 7, 8, 6, 8, 9, 4, 7, 6, 1, 3],[5, 11, 5, 7, 2, 9, 5, 5, 7, 11, 10, 7]], dtype=torch.long)
copy_ind = torch.tensor([1, 2, 3, 5, 7, 8], dtype=torch.long)
c1_new = torch.index_select(c1, 1, copy_ind)

indices = torch.as_tensor([[1, 3, 4, 6, 6, 6, 7, 7, 7, 8, 8, 9], [10, 7, 5, 2, 5, 11, 5, 7, 11, 7, 9, 5]])
values = torch.randn(12)
tensor = torch.sparse.FloatTensor(indices, values, (12, 12))

_, t_ind = torch.sort(c1[0], dim=0)
ind = t_ind[copy_ind] # should be [8, 6, 9, 10, 2, 7]

Unfortunately, the indices ind are not correct. Can someone please help me?


title: Gunicorn worker, threads for GPU tasks to increase concurrency/parallelism

body:
I'm using Flask with Gunicorn to implement an AI server. The server takes in HTTP requests and calls the algorithm (built with pytorch). The computation is run on the nvidia GPU.
I need some input as to how can I achieve concurrency/parallelism in this case. The machine has 8 vCPUs, 20 GB memory and 1 GPU, 12 GB memory.

1 worker occupies, 4 GB memory, 2.2GB GPU memory.
max workers I can give is 5. (Because of GPU memory 2.2 GB * 5 workers = 11 GB )
1 worker = 1 HTTP request (max simultaneous requests = 5)

The specific question is

How can I increase the concurrency/parallelism?
Do I have to specify number of threads for computation on GPU?

Now my gunicorn command is

gunicorn --bind 0.0.0.0:8002 main:app --timeout 360 --workers=5 --worker-class=gevent --worker-connections=1000


title: Launching the right number of CUDA blocks for a custom PyTorch activation function

body:
I'm currently writing a CUDA kernel for a custom operation (an activation) for PyTorch, but I'm quite unfamiliar with any form of GPU programming. For reference, I was following the Custom C++ & CUDA extension tutorial.
A simplified example of the sort of operation I want to do:
Let's say I have an input tensor, X_in, which can be of any shape and dims (e.g. something like (16, 3, 50, 100) ). Let's say I also have a 1D tensor, T (for example, T can be a tensor of shape (100,) ).
For each value in X_in, I want to calculate an "index" value that should be < len(T). Then the output would be basically the value of that index in T, multiplied or added with some constant. This is something like a "lookup table" operation.
An example kernel:
__global__ void inplace_lookup_kernel(
    const scalar_t* __restrict__ T,
    scalar_t* __restrict__ X_in,
    const int N) {

    const int i = blockIdx.x * blockDim.x + threadIdx.x;
    const int idx = int(X_in[i]) % N;

    X_in[i] = 5 * T[idx] - 3;
}

I also wish to do the operation in-place, which is why the output is being computed into X_in.
My question is, for an operation like this, which is to be applied pointwise on each value of X_in, how to determine the way to launch a good number of threads/blocks? In the Custom C++ & CUDA extension tutorial, they do so by:
const int threads = 1024;
const dim3 blocks((state_size + threads - 1) / threads, batch_size);

For their use-case, the operation (an lstm variant) has a specific format of input, and thus a fixed number of dimensions, from which blocks can be calculated.
But the operation I'm writing should accept inputs of any dimensions and shape. What is the right way to calculate the block number for this situation?
I'm currently doing the following:
const int threads = 1024;
const int nelems = int(X_in.numel());
const dim3 blocks((nelems + threads - 1) / threads);

However I'm doing this by intuition, and not with any certainty. Is there a better or correct way to do this? And is there any computational advantage if I define blocks in the format blocks(otherdim_size, batch_size) like in the tutorial?


title: I got runtime Error in making model with CNN

body:
I want to make A model with Cnn.but it always show me Error on Tensor Size.I canot find in any part of my code this kind of sizes! it's ok with two classes params and it accured when I change it to 4
here is my code
from __future__ import print_function, division

import torch
import torch.nn as nn
import torch.optim as optim
from torch.optim import lr_scheduler
import torchvision
from torchvision import datasets, models, transforms
import matplotlib.pyplot as plt
import numpy as np
import time
import os
import copy

import os
import shutil
import re

two_train =  "/Data/train/2"
two_val =  "/Data/val/2"
nine_train =  "/Data/train/9"
nine_val = "/Data/val/9"
seven_train =  "/Data/train/7"
seven_val = "/Data/val/7"
eight_train =  "/Data/train/8"
eight_val = "/Data/val/8"

two_files = os.listdir(two_train)
nine_files = os.listdir(nine_train)
seven_files = os.listdir(seven_train)
eight_files = os.listdir(eight_train)

# Make transforms and use data loaders

# We'll use these a lot, so make them variables
mean_nums = [0.485, 0.456, 0.406]
std_nums = [0.229, 0.224, 0.225]

chosen_transforms = {'train': transforms.Compose([
        transforms.ToTensor(),
        transforms.Normalize(mean_nums, std_nums)
]), 'val': transforms.Compose([
        transforms.ToTensor(),
        transforms.Normalize(mean_nums, std_nums)
]),
}

# Set the directory for the data
data_dir = '/content/drive/MyDrive/Data/'

# Use the image folder function to create datasets
chosen_datasets = {x: datasets.ImageFolder(os.path.join(data_dir, x),
  chosen_transforms[x])
                  for x in ['train', 'val']}

# Make iterables with the dataloaders
dataloaders = {x: torch.utils.data.DataLoader(chosen_datasets[x], batch_size=4,
  shuffle=True, num_workers=8)
              for x in ['train', 'val']}

dataset_sizes = {x: len(chosen_datasets[x]) for x in ['train', 'val']}
class_names = chosen_datasets['train'].classes

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

dataset_sizes
# class_names
# device
mean_nums

def imshow(inp, title=None):
    inp = inp.numpy().transpose((1, 2, 0))
    mean = np.array([mean_nums])
    std = np.array([std_nums])
    inp = std * inp + mean
    inp = np.clip(inp, 0, 1)
    plt.imshow(inp)
    if title is not None:
        plt.title(title)
    plt.pause(0.001)  # Pause a bit so that plots are updated

# Grab some of the training data to visualize
inputs, classes = next(iter(dataloaders['train']))

and I got This Error :
Caught RuntimeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/worker.py", line 198, in _worker_loop
    data = fetcher.fetch(index)
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
    return self.collate_fn(data)
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/collate.py", line 83, in default_collate
    return [default_collate(samples) for samples in transposed]
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/collate.py", line 83, in <listcomp>
    return [default_collate(samples) for samples in transposed]
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/collate.py", line 55, in default_collate
    return torch.stack(batch, 0, out=out)
RuntimeError: stack expects each tensor to be equal size, but got [3, 65, 36] at entry 0 and [3, 61, 39] at entry

Error occured in last line and show me this error on console!
I seearch on the Web but I got nothing


title: Transformer Autoencoder with low loss and good accuracy can't reconstruct song without teacher forcing

body:
I am working on an Adversarial Autoencoder with Compressive Transformer for music generation and interpolation.
The input for the decoder is a sequence of 8 bars, where each bars is made by 200 tokens.
Each bar has 4 tracks which are respectively: drums, bass, guitar and strings.
Each note is represented with a sequence <starting time> <note pitch> <note duration>.
The compressed representation is the first 16 latent of the encoder output on the last bar (the  8th one), which should capture information about all the 8 provided bars thanks to the memories of the Compressive Transformer.
The training is done with the WGAN-GP loss.
The encoder output distribution matches the prior distribution (a mixture of 4 gaussians), the accuracy reaches the value of 0.6 and the loss decrases well:

When I try to sample from the distribution, the song generated is kinda weird:
generated song
But the problem is when I try to reconstruct a song:
original
reconstructed
I think that the problem is releated to what is discussed here: at training time, when the decoder needs to reconstruct the token in position k, the previous <k token are the right ones because of the teacher forcing, while at testing time the previous token are feed in an autoregressive way and, because not all of them are the right ones, this confuses the Decoder.
The code for Greedy Decoding is the following:
    def greedy_decode(self, latent, n_bars, desc):
        _, _, d_mems, d_cmems = get_memories(n_batch=1)
        outs = []
        for _ in tqdm(range(n_bars), position=0, leave=True, desc=desc):
            trg = np.full((4, 1, 1), config["tokens"]["sos"])
            trg = torch.LongTensor(trg).to(config["train"]["device"])
            for _ in range(config["model"]["seq_len"] - 1):  # for each token of each bar
                trg_mask = create_trg_mask(trg.cpu().numpy())
                out, _, _, _, _, _ = self.decoder(trg, trg_mask, None, latent, d_mems, d_cmems)
                out = torch.max(out, dim=-2).indices
                out = out.permute(2, 0, 1)
                trg = torch.cat((trg, out[..., -1:]), dim=-1)
            trg_mask = create_trg_mask(trg.cpu().numpy())
            out, _, _, d_mems, d_cmems, _ = self.decoder(trg, trg_mask, None, latent, d_mems, d_cmems)
            out = torch.max(out, dim=-2).indices
            out = out.permute(2, 0, 1)
            outs.append(out)
        return outs

I also tried Beam Search, but it does not seem to solve the problem.
Now I am trying to use the previous generated token instead to the true one with a probability of 1/2, but the training is very slow.
Does anyone know a way to teach the transformer to be not too dependent on teacher forcing?


title: Run custom model's Intermediate Representation (IR), OpenVino

body: I converted my custom pytorch model into Intermediate Representation (IR), to run it in OpenVino. However, I can't find any (simple) tutorial/guide to do that. I know OpenVino has it's own build-in files to run some famous models (link). But, is there any simple guide to run IR in OpenVino?