title: Custom c++ extension error: pasting “pybind11_init_” and “‘sigmoid’” does not give a valid preprocessing token

body:
I’m trying to add custom c++ extension with pytorch.

I’m following the tutorial at https://pytorch.org/tutorials/advanced/cpp_extension.html

I have created two files.

project/
  main.py
  sigmoid.cpp

main.py

from torch.utils.cpp_extension import load
lltm_cpp = load(name=‘sigmoid’, sources=[‘sigmoid.cpp’], verbose=True)

sigmoid.cpp

#include <torch/extension.h>

#include <iostream>

torch::Tensor d_sigmoid(torch::Tensor z) {
  auto s = torch::sigmoid(z);
  return (1 - s) * s;
}

PYBIND11_MODULE('sigmoid', m) {
  m.def('d_sigmoid', &d_sigmoid, 'sigmoid');
}

when I run main.py I get the following error

  error: pasting “pybind11_init_” and “‘sigmoid’” does not give a valid
  preprocessing token

How can I fix the error?

Full error report

Using /tmp/torch_extensions as PyTorch extensions root…
Emitting ninja build file /tmp/torch_extensions/sigmoid/build.ninja…
Building extension module sigmoid…
[1/2] c++ -MMD -MF sigmoid.o.d -DTORCH_EXTENSION_NAME=sigmoid -DTORCH_API_INCLUDE_EXTENSION_H -isystem /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include -isystem /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/torch/csrc/api/include -isystem /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/TH -isystem /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/THC -isystem /home/wickrama/anaconda3/envs/pytorch/include/python3.6m -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++11 -c /home/wickrama/projects/torch_cpp_ext/sigmoid.cpp -o sigmoid.o
FAILED: sigmoid.o
c++ -MMD -MF sigmoid.o.d -DTORCH_EXTENSION_NAME=sigmoid -DTORCH_API_INCLUDE_EXTENSION_H -isystem /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include -isystem /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/torch/csrc/api/include -isystem /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/TH -isystem /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/THC -isystem /home/wickrama/anaconda3/envs/pytorch/include/python3.6m -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++11 -c /home/wickrama/projects/torch_cpp_ext/sigmoid.cpp -o sigmoid.o
In file included from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/pytypes.h:12:0,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/cast.h:13,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/attr.h:13,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/pybind11.h:44,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/torch/csrc/utils/pybind.h:6,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/python.h:12,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/torch/extension.h:6,
from /home/wickrama/projects/torch_cpp_ext/sigmoid.cpp:1:
/home/wickrama/projects/torch_cpp_ext/sigmoid.cpp:10:1: error: pasting “pybind11_init_” and “‘sigmoid’” does not give a valid preprocessing token
PYBIND11_MODULE(‘sigmoid’, m) {
^
/home/wickrama/projects/torch_cpp_ext/sigmoid.cpp:10:1: warning: character constant too long for its type
PYBIND11_MODULE(‘sigmoid’, m) {
^~~~~~~~~~~~~~
In file included from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/pytypes.h:12:0,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/cast.h:13,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/attr.h:13,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/pybind11.h:44,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/torch/csrc/utils/pybind.h:6,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/python.h:12,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/torch/extension.h:6,
from /home/wickrama/projects/torch_cpp_ext/sigmoid.cpp:1:
/home/wickrama/projects/torch_cpp_ext/sigmoid.cpp:10:1: error: pasting “PyInit_” and “‘sigmoid’” does not give a valid preprocessing token
PYBIND11_MODULE(‘sigmoid’, m) {
^
/home/wickrama/projects/torch_cpp_ext/sigmoid.cpp:10:1: warning: character constant too long for its type
PYBIND11_MODULE(‘sigmoid’, m) {
^~~~~~~
In file included from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/pytypes.h:12:0,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/cast.h:13,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/attr.h:13,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/pybind11.h:44,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/torch/csrc/utils/pybind.h:6,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/python.h:12,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/torch/extension.h:6,
from /home/wickrama/projects/torch_cpp_ext/sigmoid.cpp:1:
/home/wickrama/projects/torch_cpp_ext/sigmoid.cpp:10:1: error: pasting “pybind11_init_” and “‘sigmoid’” does not give a valid preprocessing token
PYBIND11_MODULE(‘sigmoid’, m) {
^
/home/wickrama/projects/torch_cpp_ext/sigmoid.cpp:10:1: warning: character constant too long for its type
PYBIND11_MODULE(‘sigmoid’, m) {
^~~~~~~~~~~~~~
In file included from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/pytypes.h:12:0,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/cast.h:13,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/attr.h:13,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/pybind11.h:44,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/torch/csrc/utils/pybind.h:6,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/python.h:12,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/torch/extension.h:6,
from /home/wickrama/projects/torch_cpp_ext/sigmoid.cpp:1:
/home/wickrama/projects/torch_cpp_ext/sigmoid.cpp:10:1: error: pasting “pybind11_init_” and “‘sigmoid’” does not give a valid preprocessing token
PYBIND11_MODULE(‘sigmoid’, m) {
^
/home/wickrama/projects/torch_cpp_ext/sigmoid.cpp:10:1: warning: character constant too long for its type
PYBIND11_MODULE(‘sigmoid’, m) {
^~~~~~~~~~~~~~
/home/wickrama/projects/torch_cpp_ext/sigmoid.cpp:11:9: warning: character constant too long for its type
m.def(‘d_sigmoid’, &d_sigmoid, ‘sigmoid’);
^~~~~~~~~~~
/home/wickrama/projects/torch_cpp_ext/sigmoid.cpp:11:34: warning: character constant too long for its type
m.def(‘d_sigmoid’, &d_sigmoid, ‘sigmoid’);
^~~~~~~~~
In file included from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/pytypes.h:12:0,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/cast.h:13,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/attr.h:13,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/pybind11.h:44,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/torch/csrc/utils/pybind.h:6,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/python.h:12,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/torch/extension.h:6,
from /home/wickrama/projects/torch_cpp_ext/sigmoid.cpp:1:
/home/wickrama/projects/torch_cpp_ext/sigmoid.cpp:10:17: error: expected initializer before ‘\x6d6f6964’
PYBIND11_MODULE(‘sigmoid’, m) {
^
/home/wickrama/projects/torch_cpp_ext/sigmoid.cpp:10:17: error: expected initializer before ‘\x6d6f6964’
PYBIND11_MODULE(‘sigmoid’, m) {
^
/home/wickrama/projects/torch_cpp_ext/sigmoid.cpp:10:17: error: expected initializer before ‘\x6d6f6964’
PYBIND11_MODULE(‘sigmoid’, m) {
^
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
File “/home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/utils/cpp_extension.py”, line 949, in _build_extension_module
check=True)
File “/home/wickrama/anaconda3/envs/pytorch/lib/python3.6/subprocess.py”, line 438, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command ‘[‘ninja’, ‘-v’]’ returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/home/wickrama/projects/torch_cpp_ext/main.py”, line 3, in
lltm_cpp = load(name=‘sigmoid’, sources=[‘sigmoid.cpp’], verbose=True)
File “/home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/utils/cpp_extension.py”, line 644, in load
is_python_module)
File “/home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/utils/cpp_extension.py”, line 813, in _jit_compile
with_cuda=with_cuda)
File “/home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/utils/cpp_extension.py”, line 866, in _write_ninja_file_and_build
_build_extension_module(name, build_directory, verbose)
File “/home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/utils/cpp_extension.py”, line 962, in _build_extension_module
raise RuntimeError(message)
RuntimeError: Error building extension ‘sigmoid’


title: PyTorch modules inside Numba jitclass

body:
I am new to using Numba. I am trying to create jitclass with pytorch modules in it but not able to find numba types to assign to them. Following is the sample code.

from numba import jitclass, int32
import torch

spec=[('val',int32),('network',),('loss',),('optimizer',)]

@jitclass(spec)
class Test():
    def __init__(self, val, network, loss, optimizer):
        self.val=val
        self.network=network
        self.loss=loss
        self.opimizer=optimizer

val=10
network= torch.nn.Sequential(torch.nn.Linear(10,50),torch.nn.ReLU(),
                     torch.nn.Linear(50,5),torch.nn.ReLU())
MseLoss= torch.nn.MSELoss()

optimizer= torch.optim.Adam(network.parameters(), lr=0.001)

obj=Test(val,network,MseLoss,optimizer)


title: Can I generate heat map using method such as Grad-CAM in concatenated CNN?

body:
I am trying to apply GradCAM to my pre-trained CNN model to generate heat maps of layers. My custom CNN design is shown as follows:
- It adopted all the convolution layers and the pre-trained weights from the VGG16 model.
- Extract lower level features (early convolution layers) from VGG16.
- Train the fully connected layers of both normal/high and lower level features from VGG16.
- Concatenate outputs of both normal/high- and lower-level f.c. layers and then train more f.c. layers before the final prediction.

model design

I want to use GradCAM to visualize the feature maps of the low-level route and the normal/high-level route and I have done such heatmaps on non-concatenate fine-tuned VGG using the last convolutional layers. My question is, on a concatenated CNN model, can the Grad-CAM method still work using the gradient of the prediction with respect to the low- and high-level feature map feature maps respectfully? If not, are there other methods that can do the heatmaps visualization for such a model? Is using the shared fully connected layer an option?

Any idea and suggestions are much appreciated!


title: Adding softmax layer to LSTM network "freezes" output

body:
I've been trying to teach myself the basics of RNN's with a personnal project on PyTorch. I want to produce a simple network that is able to predict the next character in a sequence (idea mainly from this article http://karpathy.github.io/2015/05/21/rnn-effectiveness/ but I wanted to do most of the stuff myself).
My idea is this : I take a batch of B input sequences of size n (np array of n integers), one hot encode them and pass them through my network composed of several LSTM layers, one fully connected layers and one softmax unit.
I then compare the output to the target sequences which are the input sequences shifted one step ahead.
My issue is that when I include the softmax layer, the output is the same every single epoch for every single batch. When I don't include it, the network seems to learn appropriately. I can't figure out what's wrong.
My implementation is the following :
class Model(nn.Module):
def __init__(self, one_hot_length, dropout_prob, num_units, num_layers):

    super().__init__()

    self.LSTM = nn.LSTM(one_hot_length, num_units, num_layers, batch_first = True, dropout = dropout_prob)

    self.dropout = nn.Dropout(dropout_prob)
    self.fully_connected = nn.Linear(num_units, one_hot_length)
    self.softmax = nn.Softmax(dim = 1)
    # dim = 1 as the tensor is of shape (batch_size*seq_length, one_hot_length) when entering the softmax unit

def forward_pass(self, input_seq, hc_states):

    output, hc_states = self.LSTM (input_seq, hc_states)
    output = output.view(-1, self.num_units)
    output = self.fully_connected(output)
    # I simply comment out the next line when I run the network without the softmax layer
    output = self.softmax(output)
    return output, hc_states

one_hot_length is the size of my character dictionnary (~200, also the size of a one hot encoded vector)
num_units is the number of hidden units in a LSTM cell, num_layers the number of LSTM layers in the network.
The inside of the training loop (simplified) goes as follows :
input, target = next_batches(data, batch_pointer)

input = nn.functional.one_hot(input_seq, num_classes = one_hot_length).float().

for state in hc_states:
    state.detach_()

optimizer.zero_grad()

output, states = net.forward_pass(input, hc_states)

loss = nn.CrossEntropyLoss(output, target)
loss.backward()

nn.utils.clip_grad_norm_(net.parameters(), MaxGradNorm)

optimizer.step()

With hc_states a tuple with the hidden states tensor and the cell states tensor, input, is a tensor of size (B,n,one_hot_length), target is (B,n).
I'm training on a really small dataset (sentences in a .txt of ~400Ko) just to tune my code, and did 4 different runs with different parameters and each time the outcome was the same : the network doesn't learn at all when it has the softmax layer, and trains somewhat appropriately without.
I don't think it is an issue with tensors shapes as I'm almost sure I checked everything.
My understanding of my problem is that I'm trying to do classification, and that the usual is to put a softmax unit at the end to get "probabilities" of each character to appear, but clearly this isn't right.
Any ideas to help me ?
I'm also fairly new to Pytorch and RNN so I apologize in advance if my architecture/implementation is some kind of monstrosity to a knowledgeable person. Feel free to correct me and thanks in advance.


title: Can we share memory between workers in a Pytorch DataLoader?

body:
My dataset depends on a 3GB tensor. This tensor could either be on the CPU or the GPU. The bottleneck of my code is the data loading preprocessing. But I can't add more than a few workers without killing my RAM.
This sounds silly for me: why could each worker receives a copy of the 3GB tensor, when this one is exactly the same across each worker?
Is there any solution for letting the workers access to a single version of this tensor?
Thanks,


title: PyTorch Semantic Segmentation with weighted classes

body:
I'm studying how to the semantic segmentation on images, but I would like to assign a weight to the classes. I'm using this Semantic Segmentation PyTorch and with the actual implementation there isn't the possibility to apply weight to the classes.
I tried to changes the util/functional.py like the code reported below but the training stage does not increase the performance. In particular the IOU_Score oscillate on 0.26 value without any good performance.
I'm using the notebook reported in the examples folder with a multiclass configuration.
Can anyone help me to understand how to changes metrics or how to use a different metrics with the weight assigned to the classes.
Thank you for the help.
def iou(pr, gt, eps=1e-7, threshold=None, ignore_channels=None, weights=None):
    """Calculate Intersection over Union between ground truth and prediction
    Args:
        pr (torch.Tensor): predicted tensor
        gt (torch.Tensor):  ground truth tensor
        eps (float): epsilon to avoid zero division
        threshold: threshold for outputs binarization
    Returns:
        float: IoU (Jaccard) score
    """

    pr = _threshold(pr, threshold=threshold)
    pr, gt = _take_channels(pr, gt, ignore_channels=ignore_channels)

    #print('PR', pr.shape, type(pr))
    #print('GT', gt.shape, type(gt))

    pr1 = torch.zeros(pr.shape[0], pr.shape[1], pr.shape[2], pr.shape[3])
    gt1 = torch.zeros(gt.shape[0], gt.shape[1], gt.shape[2], gt.shape[3])

    pr1[:, 0, :, :] = pr[:, 0, :, :] / 100
    pr1[:, 1, :, :] = pr[:, 1, :, :] / 2
    pr1[:, 2, :, :] = pr[:, 2, :, :] / 2
    pr1[:, 3, :, :] = pr[:, 3, :, :] / 100
    pr1[:, 4, :, :] = pr[:, 4, :, :] / 100

    gt1[:, 0, :, :] = gt[:, 0, :, :] / 100
    gt1[:, 1, :, :] = gt[:, 1, :, :] / 2
    gt1[:, 2, :, :] = gt[:, 2, :, :] / 2
    gt1[:, 3, :, :] = gt[:, 3, :, :] / 100
    gt1[:, 4, :, :] = gt[:, 4, :, :] / 100

    intersection = torch.sum(gt1 * pr1)
    union = torch.sum(gt1) + torch.sum(pr1) - intersection + eps
    return (intersection + eps) / union

jaccard = iou

def f_score(pr, gt, beta=1, eps=1e-7, threshold=None, ignore_channels=None, weights=None):
    """Calculate F-score between ground truth and prediction
    Args:
        pr (torch.Tensor): predicted tensor
        gt (torch.Tensor):  ground truth tensor
        beta (float): positive constant
        eps (float): epsilon to avoid zero division
        threshold: threshold for outputs binarization
    Returns:
        float: F score
    """

    pr = _threshold(pr, threshold=threshold)
    pr, gt = _take_channels(pr, gt, ignore_channels=ignore_channels)

    #print('PR', pr.shape, type(pr))
    #print('GT', gt.shape, type(gt))

    pr1 = torch.zeros(pr.shape[0], pr.shape[1], pr.shape[2], pr.shape[3])
    gt1 = torch.zeros(gt.shape[0], gt.shape[1], gt.shape[2], gt.shape[3])

    pr1[:, 0, :, :] = pr[:, 0, :, :] / 100
    pr1[:, 1, :, :] = pr[:, 1, :, :] / 2
    pr1[:, 2, :, :] = pr[:, 2, :, :] / 2
    pr1[:, 3, :, :] = pr[:, 3, :, :] / 100
    pr1[:, 4, :, :] = pr[:, 4, :, :] / 100

    gt1[:, 0, :, :] = gt[:, 0, :, :] / 100
    gt1[:, 1, :, :] = gt[:, 1, :, :] / 2
    gt1[:, 2, :, :] = gt[:, 2, :, :] / 2
    gt1[:, 3, :, :] = gt[:, 3, :, :] / 100
    gt1[:, 4, :, :] = gt[:, 4, :, :] / 100

    tp = torch.sum(gt1 * pr1)
    fp = torch.sum(pr1) - tp
    fn = torch.sum(gt1) - tp

    score = ((1 + beta ** 2) * tp + eps) \
            / ((1 + beta ** 2) * tp + beta ** 2 * fn + fp + eps)

    return score


title: UnsupportedNodeError: try blocks aren't supported fastai

body:
I am trying to save my model using torch.jit.trace. However, when i try to run the cell, it returns me this error below
UnsupportedNodeError: try blocks aren't supported:
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/fastai/layers.py", line 572
def _has_children(m:nn.Module):
    try: next(m.children())
    ~~~ <--- HERE
    except StopIteration: return False
    return True

To Reproduce
Steps to reproduce the behavior:
To reproduce the behaviour, I will use the data from the docs https://docs.fast.ai/vision.learner.html#unet_learner
path = untar_data(URLs.PETS)
fnames = get_image_files(path/"images")
pat = r'^(.*)_\d+.jpg$'
dls = ImageDataLoaders.from_name_re(path, fnames, pat, item_tfms=Resize(224))
learn = cnn_learner(dls, models.resnet34, loss_func=CrossEntropyLossFlat(), ps=0.25)
learn.save('restnet34')
img_size = 224
trace_input = torch.ones(1,3,img_size,img_size).cuda()
learn.model.eval()
jit_model = torch.jit.trace(learn.model.float(), trace_input)

Previously I was able to save the file, but now it keeps returning me that error. Any help will be appreciated. I have also tried torch.jit.script below but it still gives me the same error
This is the full stack trace error message
---------------------------------------------------------------------------
UnsupportedNodeError                      Traceback (most recent call last)
<ipython-input-12-e980bd2003d6> in <module>
      2 trace_input = torch.ones(1,3,img_size,img_size).cuda()
      3 learn.model.eval()
----> 4 jit_model = torch.jit.trace(learn.model.float(), trace_input)

~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/torch/jit/_trace.py in trace(func, example_inputs, optimize, check_trace, check_inputs, check_tolerance, strict, _force_outplace, _module_class, _compilation_unit)
    740             strict,
    741             _force_outplace,
--> 742             _module_class,
    743         )
    744

~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/torch/jit/_trace.py in trace_module(mod, inputs, optimize, check_trace, check_inputs, check_tolerance, strict, _force_outplace, _module_class, _compilation_unit)
    926         register_submods(mod, "__module")
    927
--> 928         module = make_module(mod, _module_class, _compilation_unit)
    929
    930         for method_name, example_inputs in inputs.items():

~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/torch/jit/_trace.py in make_module(mod, _module_class, _compilation_unit)
    558         if _module_class is None:
    559             _module_class = TopLevelTracedModule
--> 560         return _module_class(mod, _compilation_unit=_compilation_unit)
    561
    562

~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/torch/jit/_trace.py in __init__(self, orig, id_set, _compilation_unit)
   1038         for name, submodule in orig._modules.items():
   1039             tmp_module._modules[name] = make_module(
-> 1040                 submodule, TracedModule, _compilation_unit=None
   1041             )
   1042

~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/torch/jit/_trace.py in make_module(mod, _module_class, _compilation_unit)
    558         if _module_class is None:
    559             _module_class = TopLevelTracedModule
--> 560         return _module_class(mod, _compilation_unit=_compilation_unit)
    561
    562

~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/torch/jit/_trace.py in __init__(self, orig, id_set, _compilation_unit)
   1038         for name, submodule in orig._modules.items():
   1039             tmp_module._modules[name] = make_module(
-> 1040                 submodule, TracedModule, _compilation_unit=None
   1041             )
   1042

~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/torch/jit/_trace.py in make_module(mod, _module_class, _compilation_unit)
    558         if _module_class is None:
    559             _module_class = TopLevelTracedModule
--> 560         return _module_class(mod, _compilation_unit=_compilation_unit)
    561
    562

~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/torch/jit/_trace.py in __init__(self, orig, id_set, _compilation_unit)
   1038         for name, submodule in orig._modules.items():
   1039             tmp_module._modules[name] = make_module(
-> 1040                 submodule, TracedModule, _compilation_unit=None
   1041             )
   1042

~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/torch/jit/_trace.py in make_module(mod, _module_class, _compilation_unit)
    558         if _module_class is None:
    559             _module_class = TopLevelTracedModule
--> 560         return _module_class(mod, _compilation_unit=_compilation_unit)
    561
    562

~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/torch/jit/_trace.py in __init__(self, orig, id_set, _compilation_unit)
   1042
   1043         script_module = torch.jit._recursive.create_script_module(
-> 1044             tmp_module, lambda module: (), share_types=False
   1045         )
   1046

~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/torch/jit/_recursive.py in create_script_module(nn_module, stubs_fn, share_types)
    350     check_module_initialized(nn_module)
    351     concrete_type = get_module_concrete_type(nn_module, share_types)
--> 352     return create_script_module_impl(nn_module, concrete_type, stubs_fn)
    353
    354 def create_script_module_impl(nn_module, concrete_type, stubs_fn):

~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/torch/jit/_recursive.py in create_script_module_impl(nn_module, concrete_type, stubs_fn)
    363     cpp_module = torch._C._create_module_with_type(concrete_type.jit_type)
    364     method_stubs = stubs_fn(nn_module)
--> 365     property_stubs = get_property_stubs(nn_module)
    366
    367     def init_fn(script_module):

~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/torch/jit/_recursive.py in get_property_stubs(nn_module)
    599     """
    600     module_ty = type(nn_module)
--> 601     properties_asts = get_class_properties(module_ty, self_name="RecursiveScriptModule")
    602     rcbs = {}
    603

~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/torch/jit/frontend.py in get_class_properties(cls, self_name)
    149     for prop in props:
    150         if prop[0] not in unused_properties and not should_drop(prop[1].fget):
--> 151             getter = get_jit_def(prop[1].fget, f"__{prop[0]}_getter", self_name=self_name)
    152             setter = get_jit_def(prop[1].fset, f"__{prop[0]}_setter", self_name=self_name) if prop[1].fset else None
    153             properties.append(Property(getter.range(), Ident(getter.range(), prop[0]), getter, setter))

~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/torch/jit/frontend.py in get_jit_def(fn, def_name, self_name)
    219             arg.annotation = unused_def.args.args[0].annotation
    220
--> 221     return build_def(ctx, fn_def, type_line, def_name, self_name=self_name)
    222
    223

~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/torch/jit/frontend.py in build_def(ctx, py_def, type_line, def_name, self_name)
    253     return Def(Ident(r, def_name),
    254                decl,
--> 255                build_stmts(ctx, body))
    256
    257

~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/torch/jit/frontend.py in build_stmts(ctx, stmts)
    125
    126 def build_stmts(ctx, stmts):
--> 127     stmts = [build_stmt(ctx, s) for s in stmts]
    128     return list(filter(None, stmts))
    129

~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/torch/jit/frontend.py in <listcomp>(.0)
    125
    126 def build_stmts(ctx, stmts):
--> 127     stmts = [build_stmt(ctx, s) for s in stmts]
    128     return list(filter(None, stmts))
    129

~/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/torch/jit/frontend.py in __call__(self, ctx, node)
    226         method = getattr(self, 'build_' + node.__class__.__name__, None)
    227         if method is None:
--> 228             raise UnsupportedNodeError(ctx, node)
    229         return method(ctx, node)
    230


title: Pytorch LSTM in ONNX.js - Uncaught (in promise) Error: unrecognized input '' for node: LSTM_4

body:
I am trying to run a Pytorch LSTM network in browser. But I am getting this error:
graph.ts:313 Uncaught (in promise) Error: unrecognized input '' for node: LSTM_4
    at t.buildGraph (graph.ts:313)
    at new t (graph.ts:139)
    at Object.from (graph.ts:77)
    at t.load (model.ts:25)
    at session.ts:85
    at t.event (instrument.ts:294)
    at e.initialize (session.ts:81)
    at e.<anonymous> (session.ts:63)
    at onnx.min.js:14
    at Object.next (onnx.min.js:14)

How can I resolve this? Here is my code for saving the model to onnx:
net = torch.load('trained_model/trained_model.pt')
net.eval()

with torch.no_grad():
    input = torch.tensor([[1,2,3,4,5,6,7,8,9]])
    h0, c0 = net.init_hidden(1)
    output, (hn, cn) = net.forward(input, (h0,c0))

    torch.onnx.export(net, (input, (h0, c0)), 'trained_model/trained_model.onnx',
                    input_names=['input', 'h0', 'c0'],
                    output_names=['output', 'hn', 'cn'],
                    dynamic_axes={'input': {0: 'sequence'}})

I put input as the only dynamic axis since it is the only one that can vary in size. With this code, the model saves properly as trained_model.onnx. It does give me a warning:
UserWarning: Exporting a model to ONNX with a batch_size other than 1, with a variable length with LSTM can cause an error when running the ONNX model with a different batch size. Make sure to save the model with a batch size of 1, or define the initial states (h0/c0) as inputs of the model.
  warnings.warn("Exporting a model to ONNX with a batch_size other than 1, "

This warning a little confusing since I am exporting it with a batch_size of 1:

input has shape torch.Size([1, 9])
h0 has shape torch.Size([2, 1, 256]) - corresponding to  (num_lstm_layers, batch_size, hidden_dim)
c0 also has shape torch.Size([2, 1, 256])

But since I do define h0/c0 as inputs of the model I don't think this relates to the problem.
This is my javascript code for running in the browser:
<script src="https://cdn.jsdelivr.net/npm/onnxjs/dist/onnx.min.js"></script>
<!-- Code that consume ONNX.js -->
<script>
  // create a session
  const myOnnxSession = new onnx.InferenceSession();

  console.log('trying to load the model')

  // load the ONNX model file
  myOnnxSession.loadModel("./trained_model.onnx").then(() => {

    console.log('successfully loaded model!')

    // after this I generate input and run the model
    // since my code fails before this it isn't relevant
  });
</script>

Based on the console.log statements, it is failing to load to the model. How should I resolve this? If relevant I'm using Python 3.8.5, Pytorch 1.6.0, ONNX 1.8.0.


title: pytorch torch segmentation model syntax error in jupyter notebook

body:
i get the error:
nn.MaxPool2d(kernel_size=2, stride=2),
^
SyntaxError: invalid syntax
with the following code:
import torch
import torch.nn as nn
import torch.nn.functional as F

class CNNSEG(nn.Module): # Define your model
    def __init__(self, num_classes=1):
        super(CNNSEG, self).__init__()
        #Adds one extra class to stand for the zero-padded pixels
        self.num_classes = num_classes + 1

        self.conv1 = nn.Sequential(
            nn.Conv2d(in_channels=1, out_channels=32, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2),
            nn.LocalResponseNorm(size=5, alpha=1e-4, beta=0.75),
        )

        self.conv2 = nn.Sequential(
            nn.Conv2d(in_channels=32, out_channels=256, kernel_size=5, stride=1, padding=2, groups=2),
            nn.ReLU(),j
            nn.MaxPool2d(kernel_size=2, stride=2),
            nn.LocalResponseNorm(size=5, alpha=1e-4, beta=0.75),
        )

        self.conv3 = nn.Sequential(
            nn.Conv2d(in_channels=256, out_channels=512, kernel_size=5, stride=1, padding=2, groups=2),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=3, stride=2),
            nn.LocalResponseNorm(size=5, alpha=1e-4, beta=0.75),
        )

        self.score_conv = nn.Conv2d(in_channels=512, out_channels=num_classes, kernel_size=1, padding=0)
        self.deconv = nn.ConvTranspose2d(in_channels=num_classes, out_channels=num_classes,
            kernel_size=16, stride=8, bias=False)
        self.out_activation = nn.Softmax((num_classes, 4))

    def forward(self, x):
        out1 = self.conv1(x)
        out2 = self.conv2(out1)
        out3 = self.conv3(out2)
        out_score = self.score_conv(out3)
        out_deconv = self.deconv(out_score)
        return out_deconv

model = CNNSEG(num_classes=4) # We can now create a model using your defined segmentation model
print(model)

i haven't got any open brackets as far as I can see, I don't see what can be wrong here ? thanks


title: TensorFlow vs PyTorch convolution confusion

body:
I am confused on how to replicate Keras (TensorFlow) convolutions in PyTorch.
In Keras, I can do something like this. (the input size is (256, 237, 1, 21) and the output size is (256, 237, 1, 1024).
import tensorflow as tf
x = tf.random.normal((256,237,1,21))
y = tf.keras.layers.Conv1D(filters=1024, kernel_size=5,padding="same")(x)
print(y.shape)
(256, 237, 1, 1024)

However, in PyTorch, when I try to do the same thing I get a different output size:
import torch.nn as nn
x = torch.randn(256,237,1,21)
m = nn.Conv1d(in_channels=237, out_channels=1024, kernel_size=(1,5))
y = m(x)
print(y.shape)
torch.Size([256, 1024, 1, 17])

I want PyTorch to give me the same output size that Keras does:
This previous question seems to imply that Keras filters are PyTorch's out_channels but thats what I have. I tried to add the padding in PyTorch of padding=(0,503) but that gives me torch.Size([256, 1024, 1, 1023]) but that still not correct. This also takes so much longer than keras does so I feel that I have incorrectly assigned a parameter.
How can I replicate what Keras did with convolution in PyTorch?


title: Why does pow return nan during the backward pass?

body:
I am trying to evaluate the following expression as a loss function:

This function describes a family of closed surfaces. I am using a neural network to try to predict the surface parameters e1, e2, ax, ay, az that minimize the least squares distance from a given set of points. The quantity F(p) - 1, though not proportional, gives a metric of how far a point lies with respect to the surface. This is my code:
def eval_inout_loss(self, p, x):
    #x: BxNx3 tensor -> x, y, z
    #p: Bx5 tensor -> e1, e2, ax, ay, az

    ###########################################################################################
    # INSIDE - OUTSIDE FUNCTION LOSS
    ###########################################################################################
    # Evaluating F(x,y,z) and calculating Σ (F(x,y,z) - 1)^2

    #x^(2/e2), y^(2/e2), z^(2/e1)
    xs = torch.pow(torch.abs(x[:,:,0]), 2/p[:,1].unsqueeze(-1))
    ys = torch.pow(torch.abs(x[:,:,1]), 2/p[:,1].unsqueeze(-1))
    zs = torch.pow(torch.abs(x[:,:,2]), 2/p[:,0].unsqueeze(-1))

    s = torch.pow(xs + ys, p[:,1].unsqueeze(-1))
    s = torch.pow(s, 1/p[:,0].unsqueeze(-1)) + zs
    s = torch.pow(s, p[:,0].unsqueeze(-1))
    s = torch.abs(s - 1)

    #multiplying by axayaz to prioritize smaller volumes
    volparams = p[:,2:].prod(-1)
    s = volparams * s.sum(-1)
    l1 = s.mean()

I understand that low values of e1, e2 would cause this expression to be numerically unstable, so I have restrained the network output to the range [0.2, 1.9] by applying a sigmoid function at the output of the network:
x = 1.7 * torch.nn.functional.sigmoid(self.mlp6(x)) + 0.2

An obvious suspect would be trying to raise a negative number to a fractional exponent, thereby producing nan as the output. However this cannot be the case here, since all numbers are squared and therefor strictly non-negative. To further enhance the numerical stability I read in a relevant paper that it's useful to raise F(p)^e1, and I achieve this with the line s = torch.pow(s, p[:,0].unsqueeze(-1)) shown above.
Setting autograd anomaly detection tool to true it gives me the following error:
  File "/home/..../Desktop/code/file.py", line 391, in eval_inout_loss
  s = torch.pow(xs + ys, p[:,1].unsqueeze(-1))
  (function print_stack)

Traceback (most recent call last):
  File "/home/...../Desktop/code/samplenet.py", line 477, in <module>
  loss.backward()
  File "/home/..../.local/lib/python3.8/site-packages/torch/tensor.py", line 185, in backward
  torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/vlassis/.local/lib/python3.8/site-packages/torch/autograd/__init__.py", line 125, in backward
  Variable._execution_engine.run_backward(
  RuntimeError: Function 'PowBackward1' returned nan values in its 0th output.

This can happen during the first iteration or several hundred iterations later, but it always happens. The output of the function doesn't seem to be particularly abnormal when this happens. For example, a possible sequence goes something like this: l1 = 0.2560 -> l1 = 0.2458 -> l1 = nan. I have tried disabling the anomaly detection tool to print the values of the tensors before nan values appear:
e1 min max mean tensor(nan, device='cuda:0', grad_fn=<MinBackward1>) tensor(nan, device='cuda:0', grad_fn=<MaxBackward1>) tensor(nan, device='cuda:0', grad_fn=<MeanBackward0>)
e2 min max mean tensor(nan, device='cuda:0', grad_fn=<MinBackward1>) tensor(nan, device='cuda:0', grad_fn=<MaxBackward1>) tensor(nan, device='cuda:0', grad_fn=<MeanBackward0>)
x min max mean tensor(-1.8769, device='cuda:0') tensor(1.6930, device='cuda:0') tensor(0.0006, device='cuda:0')
y min max mean tensor(-1.4985, device='cuda:0') tensor(1.5802, device='cuda:0') tensor(0.0108, device='cuda:0')
z min max mean tensor(-1.6468, device='cuda:0') tensor(1.8991, device='cuda:0') tensor(-0.0091, device='cuda:0')
xs min max tensor(nan, device='cuda:0', grad_fn=<MinBackward1>) tensor(nan, device='cuda:0', grad_fn=<MaxBackward1>)
ys min max tensor(nan, device='cuda:0', grad_fn=<MinBackward1>) tensor(nan, device='cuda:0', grad_fn=<MaxBackward1>)
zs min max tensor(nan, device='cuda:0', grad_fn=<MinBackward1>) tensor(nan, device='cuda:0', grad_fn=<MaxBackward1>)

What are other possible causes for nan to appear?


title: How to load a fine tuned model from BertForSequenceClassification and use it to tokenize a sentence?

body:
I'm following this tutorial (https://mccormickml.com/2019/07/22/BERT-fine-tuning/#a1-saving--loading-fine-tuned-model) in order to fine-tune a BertForSequenceClassification. After I train the model, I want to load this model to code a function "classify_sentence(sentence)": it takes a sentence and return a logit vector of predictions.
def classify_sentence(self, sentence):

    self.model = BertForSequenceClassification.from_pretrained(output_dir)
    self.tokenizer = BertTokenizer.from_pretrained(output_dir)

    encoded_dict = self.tokenizer.encode_plus(
                sentence,                      # Sentence to encode.
                add_special_tokens = True, # Add '[CLS]' and '[SEP]'
                max_length = 64,           # Pad & truncate all sentences.
                pad_to_max_length = True,
                return_attention_mask = True,   # Construct attn. masks.
                return_tensors = 'pt',     # Return pytorch tensors.
    )

    # Add the encoded sentence to the list.
    input_id = encoded_dict['input_ids']
    # And its attention mask (simply differentiates padding from non-padding).
    attention_mask = encoded_dict['attention_mask']

    input_id = torch.cat(input_id, dim=0)
    attention_mask = torch.cat(attention_mask, dim=0)

    with torch.no_grad():

        output = self.model(input_id,
        token_type_ids=None,
        attention_mask=attention_mask
        )

    logits = outputs[0]

    return logits

output_dir is a directory that contains these files: config.json, pytorch_model.bin, special_tokens_map.json, tokenizer_config.json and vocab.txt.
When I run this function, I get an error:
AttributeError: 'BertTokenizer' object has no attribute 'encode_plus'
However I used this method to encode the sentence during the train. Is there any alternative way to tokenize a sentence after load a trained BERT model?


title: Append a tensor vector to tensor matrix

body:
I have a tensor matrix that i simply want to append a tensor vector as another column to it.
For example:
    X = torch.randint(100, (100,5))
    x1 = torch.from_numpy(np.array(range(0, 100)))

I've tried torch.cat([x1, X) with various numbers for both axis and dim but it always says that the dimensions don't match.


title: How iterate over BucketIterator instance when the data parameter of BucketIterator constructor is Dataset with one customizable Example instance?

body:
my source-code is structured as follows:
Define the fields
TEXT = data.Field(
    sequential=True,
    tokenize = 'spacy',
    #batch_first = True,
)
LABEL = data.LabelField()

UPDATED: Get Vocabulary
TEXT.build_vocab(

    train,

    max_size=25000,

    #vectors = "glove.6B.100d",

    vectors = vectors,

    #unk_init = torch.Tensor.normal_

    #min_freq=50,

)

LABEL.build_vocab(

    train

)

Creating Example and Dataset instances
fields=[
        ('name', TEXT),
        ('component', LABEL),
    ]

examples= []

example = Example.fromdict({"name": "exmaple", "component": "Undefined"}, {"name": ("name", TEXT), "component": ("component", LABEL)})

examples.append(example)

instance = Dataset(examples=[example], fields=fields)

Creating bucket iterator instance
instance_iterator = data.BucketIterator(instance,batch_size=64,sort_key=lambda x: x.name,sort= True, train=True,device=device)

UPDATED: I also try with data.BucketIterator.split
BATCH_SIZE = 64

train_iterator,  instance_iterator = data.BucketIterator.splits(
    (train, instance),
    sort_key = lambda x: x.name,
    sort = True,
    batch_size = BATCH_SIZE,
    device = device)

Iterate over bucket iterator
for batch in instance_iterator:
  print(batch.name)

When try iterate over bucket iterator with customizable Dataset, it occurs the following error:
/usr/local/lib/python3.6/dist-packages/torchtext/data/iterator.py in __iter__(self)
    160                     else:
    161                         minibatch.sort(key=self.sort_key, reverse=True)
--> 162                 yield Batch(minibatch, self.dataset, self.device)
    163             if not self.repeat:
    164                 return

/usr/local/lib/python3.6/dist-packages/torchtext/data/batch.py in __init__(self, data, dataset, device)
     26             self.dataset = dataset
     27             self.fields = dataset.fields.keys()  # copy field names
---> 28             self.input_fields = [k for k, v in dataset.fields.items() if
     29                                  v is not None and not v.is_target]
     30             self.target_fields = [k for k, v in dataset.fields.items() if

/usr/local/lib/python3.6/dist-packages/torchtext/data/batch.py in <listcomp>(.0)
     27             self.fields = dataset.fields.keys()  # copy field names
     28             self.input_fields = [k for k, v in dataset.fields.items() if
---> 29                                  v is not None and not v.is_target]
     30             self.target_fields = [k for k, v in dataset.fields.items() if
     31                                   v is not None and v.is_target]

AttributeError: 'tuple' object has no attribute 'is_target'


title: YoloV5 killed at first epoch

body:
I'm using a virtual machine on Windows 10 with this config:
Memory 7.8 GiB
Processor Intel® Core™ i5-6600K CPU @ 3.50GHz × 3
Graphics llvmpipe (LLVM 11.0.0, 256 bits)
Disk Capcity 80.5 GB
OS Ubuntu 20.10 64 Bit
Virtualization Oracle

I installed docker for Ubuntu as described in the official documentation.
I pulled the docker image as described on the yolo github section for docker.
Since I have no NVIDIA GPU I could not install a driver or CUDA.
I pulled the aquarium from roboflow and installed it on a folde aquarium.
I ran this command to start the image and have my aquarium folder mounted
sudo docker run --ipc=host -it -v "$(pwd)"/Desktop/yolo/aquarium:/usr/src/app/aquarium ultralytics/yolov5:latest

And was greeted with this banner

=============
== PyTorch ==
NVIDIA Release 21.03 (build 21060478) PyTorch Version 1.9.0a0+df837d0
Container image Copyright (c) 2021, NVIDIA CORPORATION.  All rights
reserved.
Copyright (c) 2014-2021 Facebook Inc. Copyright (c) 2011-2014 Idiap
Research Institute (Ronan Collobert) Copyright (c) 2012-2014 Deepmind
Technologies    (Koray Kavukcuoglu) Copyright (c) 2011-2012 NEC
Laboratories America (Koray Kavukcuoglu) Copyright (c) 2011-2013 NYU
(Clement Farabet) Copyright (c) 2006-2010 NEC Laboratories America
(Ronan Collobert, Leon Bottou, Iain Melvin, Jason Weston) Copyright
(c) 2006      Idiap Research Institute (Samy Bengio) Copyright (c)
2001-2004 Idiap Research Institute (Ronan Collobert, Samy Bengio,
Johnny Mariethoz) Copyright (c) 2015      Google Inc. Copyright (c)
2015      Yangqing Jia Copyright (c) 2013-2016 The Caffe contributors
All rights reserved.
NVIDIA Deep Learning Profiler (dlprof) Copyright (c) 2021, NVIDIA
CORPORATION.  All rights reserved.
Various files include modifications (c) NVIDIA CORPORATION.  All
rights reserved.
This container image and its contents are governed by the NVIDIA Deep
Learning Container License. By pulling and using the container, you
accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license
WARNING: The NVIDIA Driver was not detected.  GPU functionality will
not be available.    Use 'nvidia-docker run' to start this container;
see    https://github.com/NVIDIA/nvidia-docker/wiki/nvidia-docker .
NOTE: MOFED driver for multi-node communication was not detected.
Multi-node communication performance may be reduced.

So no error there.
I installed pip and with pip wandb I added wandb. I used wandb login and set my API key.
I ran following command:
# python train.py --img 640 --batch 16 --epochs 10 --data ./aquarium/data.yaml --weights yolov5s.pt --project ip5 --name aquarium5 --nosave --cache

And received this output:
github: skipping check (Docker image)
YOLOv5  v5.0-14-g238583b torch 1.9.0a0+df837d0 CPU

Namespace(adam=False, artifact_alias='latest', batch_size=16, bbox_interval=-1, bucket='', cache_images=True, cfg='', data='./aquarium/data.yaml', device='', entity=None, epochs=10, evolve=False, exist_ok=False, global_rank=-1, hyp='data/hyp.scratch.yaml', image_weights=False, img_size=[640, 640], label_smoothing=0.0, linear_lr=False, local_rank=-1, multi_scale=False, name='aquarium5', noautoanchor=False, nosave=True, notest=False, project='ip5', quad=False, rect=False, resume=False, save_dir='ip5/aquarium5', save_period=-1, single_cls=False, sync_bn=False, total_batch_size=16, upload_dataset=False, weights='yolov5s.pt', workers=8, world_size=1)
tensorboard: Start with 'tensorboard --logdir ip5', view at http://localhost:6006/
hyperparameters: lr0=0.01, lrf=0.2, momentum=0.937, weight_decay=0.0005, warmup_epochs=3.0, warmup_momentum=0.8, warmup_bias_lr=0.1, box=0.05, cls=0.5, cls_pw=1.0, obj=1.0, obj_pw=1.0, iou_t=0.2, anchor_t=4.0, fl_gamma=0.0, hsv_h=0.015, hsv_s=0.7, hsv_v=0.4, degrees=0.0, translate=0.1, scale=0.5, shear=0.0, perspective=0.0, flipud=0.0, fliplr=0.5, mosaic=1.0, mixup=0.0
wandb: Currently logged in as: pebs (use `wandb login --relogin` to force relogin)
wandb: Tracking run with wandb version 0.10.26
wandb: Syncing run aquarium5
wandb: ⭐ View project at https://wandb.ai/pebs/ip5
wandb:  View run at https://wandb.ai/pebs/ip5/runs/1c2j80ii
wandb: Run data is saved locally in /usr/src/app/wandb/run-20210419_102642-1c2j80ii
wandb: Run `wandb offline` to turn off syncing.

Overriding model.yaml nc=80 with nc=7

                 from  n    params  module                                  arguments
  0                -1  1      3520  models.common.Focus                     [3, 32, 3]
  1                -1  1     18560  models.common.Conv                      [32, 64, 3, 2]
  2                -1  1     18816  models.common.C3                        [64, 64, 1]
  3                -1  1     73984  models.common.Conv                      [64, 128, 3, 2]
  4                -1  1    156928  models.common.C3                        [128, 128, 3]
  5                -1  1    295424  models.common.Conv                      [128, 256, 3, 2]
  6                -1  1    625152  models.common.C3                        [256, 256, 3]
  7                -1  1   1180672  models.common.Conv                      [256, 512, 3, 2]
  8                -1  1    656896  models.common.SPP                       [512, 512, [5, 9, 13]]
  9                -1  1   1182720  models.common.C3                        [512, 512, 1, False]
 10                -1  1    131584  models.common.Conv                      [512, 256, 1, 1]
 11                -1  1         0  torch.nn.modules.upsampling.Upsample    [None, 2, 'nearest']
 12           [-1, 6]  1         0  models.common.Concat                    [1]
 13                -1  1    361984  models.common.C3                        [512, 256, 1, False]
 14                -1  1     33024  models.common.Conv                      [256, 128, 1, 1]
 15                -1  1         0  torch.nn.modules.upsampling.Upsample    [None, 2, 'nearest']
 16           [-1, 4]  1         0  models.common.Concat                    [1]
 17                -1  1     90880  models.common.C3                        [256, 128, 1, False]
 18                -1  1    147712  models.common.Conv                      [128, 128, 3, 2]
 19          [-1, 14]  1         0  models.common.Concat                    [1]
 20                -1  1    296448  models.common.C3                        [256, 256, 1, False]
 21                -1  1    590336  models.common.Conv                      [256, 256, 3, 2]
 22          [-1, 10]  1         0  models.common.Concat                    [1]
 23                -1  1   1182720  models.common.C3                        [512, 512, 1, False]
 24      [17, 20, 23]  1     32364  models.yolo.Detect                      [7, [[10, 13, 16, 30, 33, 23], [30, 61, 62, 45, 59, 119], [116, 90, 156, 198, 373, 326]], [128, 256, 512]]
[W NNPACK.cpp:80] Could not initialize NNPACK! Reason: Unsupported hardware.
Model Summary: 283 layers, 7079724 parameters, 7079724 gradients, 16.4 GFLOPS

Transferred 356/362 items from yolov5s.pt
Scaled weight_decay = 0.0005
Optimizer groups: 62 .bias, 62 conv.weight, 59 other
train: Scanning '/usr/src/app/aquarium/train/labels.cache' images and labels... 448 found, 0 missing, 1 empty, 0 corrupted: 100%|█| 448/448 [00:00<?, ?
train: Caching images (0.4GB): 100%|████████████████████████████████████████████████████████████████████████████████| 448/448 [00:01<00:00, 313.77it/s]
val: Scanning '/usr/src/app/aquarium/valid/labels.cache' images and labels... 127 found, 0 missing, 0 empty, 0 corrupted: 100%|█| 127/127 [00:00<?, ?it
val: Caching images (0.1GB): 100%|██████████████████████████████████████████████████████████████████████████████████| 127/127 [00:00<00:00, 141.31it/s]
Plotting labels...

autoanchor: Analyzing anchors... anchors/target = 5.17, Best Possible Recall (BPR) = 0.9997
Image sizes 640 train, 640 test
Using 3 dataloader workers
Logging results to ip5/aquarium5
Starting training for 10 epochs...

     Epoch   gpu_mem       box       obj       cls     total    labels  img_size
  0%|                                                                                | 0/28 [00:00<?, ?it/s]Killed
root@cf40a6498016:~# /opt/conda/lib/python3.8/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 6 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

From this output I would think that there were 0 epochs completed.
My data.yaml contains this code:
train: /usr/src/app/aquarium/train/images
val: /usr/src/app/aquarium/valid/images

nc: 7
names: ['fish', 'jellyfish', 'penguin', 'puffin', 'shark', 'starfish', 'stingray']

wandb.ai does not display any metrics, but I have the files config.yaml, requirements.txt, wandb-metadata.json and wandb-summary.json.
Why am I not getting any output?
Has there in fact be no training at all?
If there was a training, how can I use my model?


title: Pytorch deep convolutional network does not converge on CIFAR10

body:
I copied the CIFAR10 sample network from PyTorch tutorial and added more layers, including BN. Even after 45 epochs, the network keeps achieving 68% classification accuracy on the test set.

The network consists of:

2 convolutional layers with 3x3 kernels (input size reduces from 32px to 28px)
one max pooling layer (input size reduces from 28px to 14px)
3 convolutional layers with 3x3 kernels (input size reduces from 14px to 8px)
A fully connected network with 3 layers of 256->256->10 neurons
batch normaliation is applied on all layers, including the convolutional layers, except for the last FC layer
Relu is applied on all the convolutional layers and all the hidden FC layers

Did I build/use anything improperly?

import torch
import torchvision
import torchvision.transforms as transforms
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1_1 = nn.Conv2d(3, 16, 3)  # 32 -> 30
        self.bn1_1 = nn.BatchNorm2d(16)
        self.conv1_2 = nn.Conv2d(16, 16, 3) # 30 - > 28
        self.bn1_2 = nn.BatchNorm2d(16)
        self.pool = nn.MaxPool2d(2, 2)  # 28 -> 14
        self.conv2_1 = nn.Conv2d(16, 16, 3) # 14 -> 12
        self.bn2_1 = nn.BatchNorm2d(16)
        self.conv2_2 = nn.Conv2d(16, 16, 3) # 12 -> 10
        self.bn2_2 = nn.BatchNorm2d(16)
        self.conv2_3 = nn.Conv2d(16, 16, 3) # 10 -> 8
        self.bn2_3 = nn.BatchNorm2d(16)
        self.fc1 = nn.Linear(16 * 8 * 8, 256)
        self.bn4 = nn.BatchNorm1d(256)
        self.fc2 = nn.Linear(256, 256)
        self.bn5 = nn.BatchNorm1d(256)
        self.fc3 = nn.Linear(256, 10)

    def forward(self, x):
        x = F.relu(self.bn1_1(self.conv1_1(x)))
        x = self.pool(F.relu(self.bn1_2(self.conv1_2(x))))
        x = F.relu(self.bn2_1(self.conv2_1(x)))
        x = F.relu(self.bn2_2(self.conv2_2(x)))
        x = F.relu(self.bn2_3(self.conv2_3(x)))
        x = x.view(-1, 16 * 8 * 8)
        x = F.relu(self.bn4(self.fc1(x)))
        x = F.relu(self.bn5(self.fc2(x)))
        x = self.fc3(x)
        return x

net = Net()
device = 'cuda:0'
net.to(device)
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)

transform = transforms.Compose(
        [transforms.ToTensor(),
         transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

trainset = torchvision.datasets.CIFAR10(root='./data', train=True,
                                            download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=8,
                                              shuffle=True, num_workers=2)

for epoch in range(128):  # loop over the dataset multiple times
    for i, data in enumerate(trainloader, 0):
        # get the inputs
        inputs, labels = data
        inputs, labels = inputs.to(device), labels.to(device)

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = net(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

Note: added the "Python" tag so the code gets highlighted

Note: updated the forward method to apply F.relu on the hidden FC layers


title: I have a question about TRANSLATION WITH A SEQUENCE TO SEQUENCE in the pytorch tutorials

body:
I am currently learning about the Seq2seq translation. I am trying to understand and following PyTorch tutorial from this website "https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#attention-decoder".

In the website, They talk about the Attention technique. I would like to know which technique are they used between Luong and Bahdanau?
Another question, Why do they apply Relu layer before GRU cell?
Finally, the red box in the figure is called a context vector, right?


I want to have the standard LSTM/GRU/RNN set up but swap the linear function with a convolution. Is that possible to do in Pytorch in an clean and efficient manner?

Ideally it still works with packing, varying sequence length etc.

Small sample code of a trivial way to pass data through it would be super useful like:

# Based on Robert Guthrie tutorial

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

from pdb import set_trace as st

torch.manual_seed(1)

def step_by_step(net,sequence,hidden):
    '''
    an example of an LSTM processing all the sequence one token at a time (one time step at a time)
    '''
    ## process sequence one element at a time
    print()
    print('start processing sequence')
    for i, token in enumerate(sequence):
        print(f'-- i = {i}')
        #print(f'token.size() = {token.size()}')
        ## to add fake batch_size and fake seq_len
        h_n, c_n = hidden # hidden states, cell state
        processed_token = token.view(1, 1, -1) # torch.Size([1, 1, 3])
        print(f'processed_token.size() = {processed_token.size()}')
        print(f'h_n.size() = {h_n.size()}')
        #print(f'processed_token = {processed_token}')
        #print(f'h_n = {h_n}')
        # after each step, hidden contains the hidden state.
        out, hidden = lstm(processed_token, hidden)
    ## print results
    print()
    print(out)
    print(hidden)

def whole_seq_all_at_once(lstm,sequence,hidden):
    '''
        alternatively, we can do the entire sequence all at once.
        the first value returned by LSTM is all of the hidden states throughout
        #the sequence.
        The second is just the most recent hidden state
        (compare the last slice of "out" with "hidden" below, they are the same)
        The reason for this is that:
        "out" will give you access to all hidden states in the sequence
        "hidden" will allow you to continue the sequence and backpropagate,
        by passing it as an argument  to the lstm at a later time
        Add the extra 2nd dimension
    '''
    h, c = hidden
    Tx = len(sequence)
    ## concatenates list of tensors in the dim 0, i.e. stacks them downwards creating new rows
    sequence = torch.cat(sequence) # (5, 3)
    ## add a singleton dimension of size 1
    sequence = sequence.view(len(sequence), 1, -1) # (5, 1, 3)
    print(f'sequence.size() = {sequence.size()}')
    print(f'h.size() = {h.size()}')
    print(f'c.size() = {c.size()}')
    out, hidden = lstm(sequence, hidden)
    ## "out" will give you access to all hidden states in the sequence
    print()
    print(f'out = {out}')
    print(f'out.size() = {out.size()}') # (5, 1, 25)
    ##
    h_n, c_n = hidden
    print(f'h_n = {h_n}')
    print(f'h_n.size() = {h_n.size()}')
    print(f'c_n = {c_n}')
    print(f'c_n.size() = {c_n.size()}')

if __name__ == '__main__':
    ## model params
    hidden_size = 6
    input_size = 3
    lstm = nn.LSTM(input_size=input_size,hidden_size=hidden_size)
    ## make a sequence of length Tx (list of Tx tensors)
    Tx = 5
    sequence = [torch.randn(1, input_size) for _ in range(Tx)]  # make a sequence of length 5
    ## initialize the hidden state.
    hidden = (torch.randn(1, 1, hidden_size), torch.randn(1, 1, hidden_size))
    #step_by_step(lstm,sequence,hidden)
    whole_seq_all_at_once(lstm,sequence,hidden)
    print('DONE \a')

Cross posted:

How does one implement convolutional GRUs/LSTMs/RNNs in Pytorch?
https://discuss.pytorch.org/t/how-to-implement-convolutional-grus-lstms-rnns/53471


title: How to increase numerical accuracy of Pytorch model?

body:
With the following line:

print(self.netG(self.real_A)-self.netG(self.real_A))

I think I can get a all zero tensor, but no.

tensor([[ [[-0.0032,  0.0089, -0.0085,  ..., -0.0027,  0.0004, -0.0022],
          [-0.0019, -0.0022,  0.0775,  ...,  0.0236, -0.0277, -0.0125],
          [ 0.0049,  0.0159,  0.0203,  ..., -0.0212,  0.0010, -0.0069],
          ...,
          [ 0.0042,  0.0081, -0.0127,  ..., -0.0097,  0.0136, -0.0002],
          [-0.0010,  0.0020, -0.0066,  ...,  0.0260,  0.0433,  0.0088],
          [-0.0023,  0.0095,  0.0125,  ...,  0.0005,  0.0090,  0.0029]]]],
       device='cuda:0', grad_fn=<SubBackward0>)


title: Best way to vectorize generating a batch of randomly rotated matrices in Numpy/PyTorch?

body:
I’d like to generate batches of randomly rotated matrices based on an initial starting matrix (which has a shape of, for example, (4096, 3)), where the rotation applied to each matrix in the batch is randomly chosen from a group of rotation matrices (in my code in the original post, I only want to randomly select from 8 possible rotation angles). Therefore, what I end up with is a tensor of shape (batch_size, 4096, 3).

My current approach is that I pre-make the possible rotated matrices (since I’m only dealing with 8 possible random rotations), and then use a for loop to generate the batch by randomly picking one of the eight pre-made rotated matrices for each item in the batch. This isn’t super efficient, so I was hoping to vectorize the whole process somehow.

Right now, this is how I loop over a batch to generate a batch of rotated matrices one by one:

for view_i in range(batch_size):
        # Get rotated view grid points randomly
        idx = torch.randint(0, 8, (1,))
        pointsf = rotated_points[idx]

In the code below, I generate a pre-made set of random rotation matrices that get randomly selected from in a for-loop over the batch.

The make_3d_grid function generates a (grid_dim * grid_dim * grid_dim, 3) shaped matrix (basically a 2D array of x, y, z coordinate points). The get_rotation_matrix function returns a (3, 3) rotation matrix, where theta is used for rotation around the x-axis.

rotated_points = []
grid_dim = 16

pointsf = make_3d_grid((-1,)*3, (1,)*3, (grid_dim,)*3)

view_angles = torch.tensor([0, np.pi / 4.0, np.pi / 2.0, 3 * np.pi / 4.0, np.pi, 5 * np.pi / 4.0, 3 * np.pi / 2.0, 7 * np.pi / 4.0])

for i in range(len(view_angles)):
    theta = view_angles[i]

    rot = get_rotation_matrix(theta, torch.tensor(0.0), torch.tensor(0.0))

    pointsf_rot = torch.mm(pointsf, rot)

    rotated_points.append(pointsf_rot)

Any help in vectorizing this would be greatly appreciated! If code for this can be done in Numpy that works fine too, since I can convert it to PyTorch myself.


title: Indexing in two dimensional PyTorch Tensor using another Tensor

body:
Suppose that tensor A is defined as:

 1  2  3  4
 5  6  7  8
 9 10 11 12
13 14 15 16

I'm trying to extract a flat array out of this matrix by using another tensor as indices. For example, if the second tensor is defined as:

0
1
2
3

I want the result of the indexing to be 1-D tensor with the contents:

1
6
11
16

It doesn't seem to behave like NumPy; I've tried A[:, B] but it just throws an error for not being able to allocate an insane amount of memory and I've no idea why!


title: Imdb Error shows up, when i try to download LSUN dataset with pytorch

body:
Recently I try to read LSUN dataset and train a DL network. But error shows up when I run the following code.

import torch
import torchvision
import torchvision.transforms as transforms
import time
from torch import nn, optim
import numpy as np
import matplotlib.pyplot as plt
root='d:/datasets/'
lsun_dataset = torchvision.datasets.LSUN(root=root, classes=['restaurant_train'], transform=transforms.Compose([
                                   transforms.Resize(96),
                                   transforms.RandomCrop(64),
                                   transforms.ToTensor(),
                                   transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),
                               ]))

print(lsun_dataset);

the error message is as below.

    Traceback (most recent call last):
  File "D:\coding\paper\lsgan\dataloadingtest.py", line 13, in <module>
    transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),
  File "d:\anaconda\lib\site-packages\torchvision\datasets\lsun.py", line 83, in __init__
    transform=transform))
  File "d:\anaconda\lib\site-packages\torchvision\datasets\lsun.py", line 25, in __init__
    readahead=False, meminit=False)
lmdb.Error: d:/datasets//restaurant_train_lmdb: \u03f5\u0373\ufffd\u04b2\ufffd\ufffd\ufffd\u05b8\ufffd\ufffd\ufffd\ufffd��\ufffd\ufffd\ufffd\ufffd

can anyone tell me how to fix it? Thx.

Platform: Windows 10, Python version: 3.7.3


title: Gpt2 generation of text larger than 1024

body:
I know the context supported by GPT2 is 1024, but I assume there's some technique they utilized to train and generate text longer than that in their results. Also, I saw many gpt2-based repos training text with length longer than 1024. But when I tried generating text using run_generation.py to generate text longer than 1024 it throws a runtime error :The size of tensor a (1025) must match the size of tensor b (1024) at non-singleton dimension 3. I have the following questions:

Shouldn't it be possible to generate longer text since a sliding window is used?
Can you please explain what's necessary to generate longer text? What changes will I have to make to the run_generation.py code?


title: pytorch: How to implement one link per neuron?

body:
For example, I would like to have a standard feed-forward neural network with the following structure:

n input neurons
n neurons on the second layer
2 neurons on the third layer
n neurons on the fourth layer

where

the i-th neuron in the first layer is connected precisely to the i-th neuron in the second layer (don't know how to do that)
the second and the third layer are fully connected, the same goes for the third and the fourth layer (I know how to do that - using nn.Linear)
loss function is MSE + L1 norm of the (vector of) weights between the first two layers (depends on the solution of the question whether I can do that)

Motivation: I want to implement an autoencoder and try to achieve some sparsity (this is why the inputs are multiplied by a single weight (going from the first to the second layer)).


AssertionError                            Traceback (most recent call last)
AssertionError: Caught AssertionError in DataLoader worker process 0.
Original Traceback (most recent call last):
AssertionError: Expected boolean as argument for 'return_batch', got type <class 'PIL.Image.Image'>. Call augment() only with named arguments, e.g. augment(images=<array>).


title: What is the goal of Variable using pytorch?

body:
I have this code :
from torch.autograd import Variable
d_real_data = Variable(d_sampler(d_input_size))

But I wonder what is the difference between Variable(d_sampler(d_input_size)) and d_sampler(d_input_size)
I think it is two tensors but the values are different. So I was wondering what is the goal of this function Variable ?


title: How to get stable output for torch.nn.Transformer

body:
Looks like Transformer layers of pytorch give not reproducible outputs. It happens both for cpu and gpu. I know that it sometimes happens because of parallel computations on gpu.
emb = nn.Embedding(10, 12).to(device)
inp1 = torch.LongTensor([1, 2, 3, 4]).to(device)
inp1 = emb(inp1).reshape(inp1.shape[0], 1, 12) #S N E

encoder_layer = nn.TransformerEncoderLayer(d_model=12, nhead=4)
transformer_encoder = nn.TransformerEncoder(encoder_layer, num_layers=4)

out1 = transformer_encoder(inp1)
out2 = transformer_encoder(inp1)

out1 and out2 are different. It can be multiprocessing on cpu, but results looks too shaky. How to fix this?


title: How can I have submodules of a PyTorch Module that are not attributes of the module

body:
I would like to have a PyTorch sub-class of Module that keeps sub-modules in a list (because there may be a variable number of sub-modules depending on the constructor's arguments). I set this list in the following way:
self.hidden_layers = [torch.nn.Linear(i, o) for i, o in pairwise(self.layer_sizes)]

According to this and this question, a submodule is only registered by __setattr__, when a Module object is assigned to an attribute of self. Because hidden_layers is not assigned an object of type Module, the submodules in the list are not registered as submodules, and as a result self.parameters() does not iterate over the submodules' parameters.
I suppose I could explicitly call __subattr__ for each element of the list but that would be quite ugly. Is there a more correct way to register a submodule that is not a direct attribute of Module?


I am trying to run the code on github(https://github.com/AayushKrChaudhary/RITnet)
I did not get the semantic segmentation dataset of OpenEDS, so I tried to download the png image from the Internet and put it in Semantic_Segmentation_Dataset\test\ to run the test program.
That code gives the following error:
Traceback (most recent call last):
  File "test.py", line 59, in <module>
    for i, batchdata in tqdm(enumerate(testloader),total=len(testloader)):
  File "C:\Users\b0743\AppData\Local\Continuum\anaconda3\envs\Machine_Learning\lib\site-packages\torch\utils\data\dataloader.py", line 291, in __iter__
    return _MultiProcessingDataLoaderIter(self)
  File "C:\Users\b0743\AppData\Local\Continuum\anaconda3\envs\Machine_Learning\lib\site-packages\torch\utils\data\dataloader.py", line 737, in __init__
    w.start()
  File "C:\Users\b0743\AppData\Local\Continuum\anaconda3\envs\Machine_Learning\lib\multiprocessing\process.py", line 105, in start
    self._popen = self._Popen(self)
  File "C:\Users\b0743\AppData\Local\Continuum\anaconda3\envs\Machine_Learning\lib\multiprocessing\context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Users\b0743\AppData\Local\Continuum\anaconda3\envs\Machine_Learning\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
  File "C:\Users\b0743\AppData\Local\Continuum\anaconda3\envs\Machine_Learning\lib\multiprocessing\popen_spawn_win32.py", line 65, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Users\b0743\AppData\Local\Continuum\anaconda3\envs\Machine_Learning\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: can't pickle cv2.CLAHE objects

(Machine_Learning) C:\Users\b0743\Downloads\RITnet-master\RITnet-master>Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\b0743\AppData\Local\Continuum\anaconda3\envs\Machine_Learning\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\Users\b0743\AppData\Local\Continuum\anaconda3\envs\Machine_Learning\lib\multiprocessing\spawn.py", line 115, in _main
    self = reduction.pickle.load(from_parent)
EOFError: Ran out of input

and my environment is:
# Name                    Version
cudatoolkit               10.1.243
cudnn                     7.6.5
keras-applications        1.0.8
keras-base                2.3.1
keras-gpu                 2.3.1
keras-preprocessing       1.1.0
matplotlib                3.3.1
matplotlib-base           3.3.1
numpy                     1.19.1
numpy-base                1.19.1
opencv                    3.3.1
pillow                    7.2.0
python                    3.6.10
pytorch                   1.6.0
scikit-learn              0.23.2
scipy                     1.5.2
torchsummary              1.5.1
torchvision               0.7.0
tqdm                      4.48.2

I don’t know if this is a stupid question, but I hope someone can try to answer it for me.


title: AttributeError in PyTorch model quantization for Flair's SequenceTagger

body:
I am working on a custom class inheriting FlairEmbedding from the Flair NLP library. In this class, I would like to implement model quantization using PyTorch's torch.quantization module. In order to do that, I need to train the model over a few batches to gather statistics and select the proper quantization parameters. The model will be used in a sequence tagger downstream, so I'm using Flair's SequenceTagger class with the same parameters as the ones I use in the downstream task. This is what the class looks like:
class CustomEmbeddings(FlairEmbeddings):
    def __init__(
                 self, tag_dictionary, tag_type, corpus, mini_batch_size, train_with_dev, # Used for training
                 model, fine_tune, chars_per_chunk,with_whitespace, tokenized_lm # Base FlairEmbeddings arguments
                ):

        super().__init__(model, fine_tune, chars_per_chunk, with_whitespace, tokenized_lm)

        self.lm.qconfig = torch.quantization.default_config
        torch.quantization.prepare(self.lm.qconfig, inplace=True)

        # Small training to gather statistics
        tagger = SequenceTagger(hidden_size=256, embeddings=self, tag_dictionary=tag_dictionary, tag_type=tag_type)
        trainer = ModelTrainer(tagger, corpus)
------> trainer.train('model', mini_batch_size=mini_batch_size, max_epochs=10, train_with_dev=train_with_dev)

        torch.quantization.convert(self.lm, inplace=True)

This code is failing with the following error:
  File "/home/pie3636/project/main.py", line 28, in __init__
    embeddings = CustomEmbeddings(name, **params)
  File "/home/pie3636/project/custom_embeddings.py", line 35, in __init__
    trainer.train('model', mini_batch_size=mini_batch_size, max_epochs=10, train_with_dev=train_with_dev)
  File "/usr/local/lib/python3.6/dist-packages/flair/trainers/trainer.py", line 371, in train
    loss = self.model.forward_loss(batch_step)
  File "/usr/local/lib/python3.6/dist-packages/flair/models/sequence_tagger_model.py", line 603, in forward_loss
    features = self.forward(data_points)
  File "/usr/local/lib/python3.6/dist-packages/flair/models/sequence_tagger_model.py", line 608, in forward
    self.embeddings.embed(sentences)
  File "/usr/local/lib/python3.6/dist-packages/flair/embeddings/base.py", line 60, in embed
    self._add_embeddings_internal(sentences)
  File "/usr/local/lib/python3.6/dist-packages/flair/embeddings/token.py", line 610, in _add_embeddings_internal
    text_sentences, start_marker, end_marker, self.chars_per_chunk
  File "/usr/local/lib/python3.6/dist-packages/flair/models/language_model.py", line 157, in get_representation
    _, rnn_output, hidden = self.forward(batch, hidden)
  File "/usr/local/lib/python3.6/dist-packages/flair/models/language_model.py", line 80, in forward
    output, hidden = self.rnn(emb, hidden)
  File "/home/m.meloux/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 726, in _call_impl
    hook_result = hook(self, input, result)
  File "/home/m.meloux/.local/lib/python3.6/site-packages/torch/quantization/quantize.py", line 74, in _observer_forward_hook
    return self.activation_post_process(output)
  File "/home/m.meloux/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/m.meloux/.local/lib/python3.6/site-packages/torch/quantization/observer.py", line 276, in forward
    x = x_orig.detach()  # avoid keeping autograd tape
AttributeError: 'tuple' object has no attribute 'detach'

I am not sure whether the problem comes from my code, or if it is something to be reported to either PyTorch or Flair's issue tracker. The stacktrace is making me think that it's the interaction between those two libraries that is failing rather that my code, especially since PyTorch's quantization module is still in beta, but I could be mistaken. Any input as to what the error might be would be appreciated.


title: PyTorch: Inference on a single very large image using multiple GPUs?

body: I want to perform inference (i.e. semantic segmentation) on a very large satellite image without splitting it into pieces. I have access to 4 GPUs (each having 15 GBs of memory) and was wondering if it is possible to somehow use all the memory of these GPUs combined (i.e. 60 GB) for inference on the image in PyTorch?


title: Implementation Differences between SGD Tensorflow and PyTorch SGD

body:
Some issues I am getting while training with identical code ported from Pytorch to TF2.

Model.fit converges in a completely different manner than Gradient Tape. (And more similar to Pytorch)
tf.keras.optimizers.SGD converges very different than pytorch SGD. (e.g. a LR of 0.1 for tf is unstable at training while the same starting LR for Pytorch is used in many SOTA implementations)

I couldn't come up with a toy example to demonstrate the issue without needing many epochs to train and can be demonstrated in a few lines of code. Any suggestions?


title: Numpy sum elements in a multi-dimensional array according to indices

body:
I am dealing with a very large multi-dimensional data , but let me take a 2D array for example. Given a value array that is changing every iteration,
arr = np.array([[ 1, 2, 3, 4, 5], [5, 6, 7, 8, 9]]) # a*b

and an index array that is fixed all the time.
idx = np.array([[[0, 1, 1], [-1, -1, -1]],
                [[5, 1, 3], [1, -1, -1]]]) # n*h*w, where n = a*b,

Here -1 means no index will be applied. And I wish to get a result
res = np.array([[1+2+2, 0],
                [5+2+4, 2]]) # h*w

In real practice, I am doing with a very large 3D tensor (n ~ trillions), with a very sparse idx (i.e. lots of -1). As idx is fixed, my current solution is to pre-compute a n*(h*w) array index_tensor by filling 0 and 1, and then do
tmp = arr.reshape(1, n)
res = (tmp @ index_tensor).reshape([h,w])

It works fine but takes a huge memory to store the index_tensor. Is there any approach that I can take the advantage of the sparsity and unchangeableness of idx to reduce the memory cost and keep a fair running speed in python (using numpy or pytorch would be the best)? Thanks in advance!


title: Multi Layer Perceptron Deep Learning in Python using Pytorch

body:
I am having errors in executing the train function of my code in MLP.
This is the error:
mat1 and mat2 shapes cannot be multiplied (128x10 and 48x10)
My code for the train function is this:
class net(nn.Module):
def __init__(self, input_dim2, hidden_dim2, output_dim2):
    super(net, self).__init__()
    self.input_dim2 = input_dim2
    self.fc1 = nn.Linear(input_dim2, hidden_dim2)
    self.relu = nn.ReLU()
    self.fc2 = nn.Linear(hidden_dim2, hidden_dim2)
    self.fc3 = nn.Linear(hidden_dim2, output_dim2)
def forward(self, x):
  x = self.fc1(x)
  x = self.relu(x)
  x = self.fc2(x)
  x = self.relu(x)
  x = self.fc3(x)
  x = F.softmax(self.fc3(x))

  return x

model = net(input_dim2, hidden_dim2, output_dim2) #create the network
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.RMSprop(model.parameters(), lr = learning_rate2)

def train(num_epochs2):
for i in range(num_epochs2):
  tmp_loss = []
  for (x,y) in train_loader:
    print(y.shape)
    print(x.shape)
    outputs = model(x) #forward pass
    print(outputs.shape)
    loss = criterion(outputs, y) #loss computation
    tmp_loss.append(loss.item()) #recording the loss
    optimizer.zero_grad() #all the accumulated gradient
    loss.backward()  #auto-differentiaton - accumulation of gradient
    optimizer.step() # a gradient step

  print("Loss at {}th epoch: {}".format(i, np.mean(tmp_loss)))

I don't know where I'm wrong. My code seems to work okay.


title: how to solve this (Pytorch RuntimeError: 1D target tensor expected, multi-target not supported)

body:
i am newbie in pytorch and deep learning
my data set 53502 x 58,
i have problem this my code
model = nn.Sequential(
    nn.Linear(58,64),
    nn.ReLU(),
    nn.Linear(64,32),
    nn.ReLU(),
    nn.Linear(32,16),
    nn.ReLU(),
    nn.Linear(16,2),
    nn.LogSoftmax(1)
)

criterion = nn.NLLLoss()
optimizer = optim.AdamW(model.parameters(), lr = 0.0001)

epoch = 500
train_cost, test_cost = [], []
for i in range(epoch):
    model.train()
    cost = 0
    for feature, target in trainloader:
        output = model(feature)          #feedforward
        loss = criterion(output, target) #loss
        loss.backward()                  #backprop

        optimizer.step()                 #update weight
        optimizer.zero_grad()            #zero grad

        cost += loss.item() * feature.shape[0]
    train_cost.append(cost / len(train_set))

    with torch.no_grad():
        model.eval()
        cost = 0
        for feature, target in testloader:
            output = model(feature)          #feedforward
            loss = criterion(output, target) #loss

            cost += loss.item() * feature.shape
        test_cost.append(cost / len(test_set))

    print(f'\repoch {i+1}/{epoch} | train_cost: {train_cost[-1]} | test_cost : {test_cost[-1]}', end = "")

and then i get problem like this
   2262                          .format(input.size(0), target.size(0)))
   2263     if dim == 2:
-> 2264         ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
   2265     elif dim == 4:
   2266         ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)

RuntimeError: 1D target tensor expected, multi-target not supported

whats wrong?
how to solve this problem?
why this happend?
Thank you very much in advance!


title: Python: BERT Error - Some weights of the model checkpoint at were not used when initializing BertModel

body:
I am creating an entity extraction model in PyTorch using bert-base-uncased but when I try to run the model I get this error:
Error:
Some weights of the model checkpoint at D:\Transformers\bert-entity-extraction\input\bert-base-uncased_L-12_H-768_A-12 were not used when initializing BertModel:
['cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight',   'cls.predictions.transform.LayerNorm.bias', 'cls.seq_relationship.bias',
 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.weight',
 'cls.predictions.bias']
    - This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
    - This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).

I have downloaded the bert model from here and the additional files from here
Code
Following is the code for my model:
import config
import torch
import transformers
import torch.nn as nn

def loss_fn(output, target, mask, num_labels):

    lfn = nn.CrossEntropyLoss()
    active_loss = mask.view(-1) == 1
    active_logits = output.view(-1, num_labels)
    active_labels = torch.where(
        active_loss,
        target.view(-1),
        torch.tensor(lfn.ignore_index).type_as(target)
    )
    loss = lfn(active_logits, active_labels)
    return loss

class EntityModel(nn.Module):
    def __init__(self, num_tag, num_pos):
        super(EntityModel, self).__init__()

        self.num_tag = num_tag
        self.num_pos = num_pos
        self.bert = transformers.BertModel.from_pretrained(config.BASE_MODEL_PATH)
        self.bert_drop_1 = nn.Dropout(p = 0.3)
        self.bert_drop_2 = nn.Dropout(p = 0.3)
        self.out_tag = nn.Linear(768, self.num_tag)
        self.out_pos = nn.Linear(768, self.num_pos)

    def forward(self, ids, mask, token_type_ids, target_pos, target_tag):
        o1, _ = self.bert(ids,
                          attention_mask = mask,
                          token_type_ids = token_type_ids)

        bo_tag = self.bert_drop_1(o1)
        bo_pos = self.bert_drop_2(o1)

        tag = self.out_tag(bo_tag)
        pos = self.out_pos(bo_pos)

        loss_tag = loss_fn(tag, target_tag, mask, self.num_tag)
        loss_pos = loss_fn(pos, target_pos, mask, self.num_pos)

        loss = (loss_tag + loss_pos) / 2

        return tag, pos, loss

print("model.py run success!")


title: How to use pytorch 0.4.1 model to init pytorch 0.2.0?

body:
I have met a problem that I train my model in pytorch 0.4.1, but I can't find a tool to convert it into caffe model.

How to use a pytorch 0.4.1 model to init pytorch 0.2.0?
Or how to convert a pytorch 0.4.1 model to a caffe model?


title: RuntimeError: size mismatch, m1: [4 x 3136], m2: [64 x 5] at c:\a\w\1\s\tmp_conda_3.7_1

body:
I used  python 3 and when i insert  transform random crop size 224 it gives miss match error.

here my code

what did i wrong ?


title: Why can't I learn XOR function with this network and constraints?

body:
Let's say I have the following constraints and the network:

The architecture is fixed (see this image) (note that there are no biases)
Activation function for the hidden layer is ReLU
There's no activation function for the output layer (should just return the sum of the inputs it receive).

I tried to implement this in pytorch with various initialization schemes and different data sets but I failed (the code is at the bottom).

My questions are:

Is there anything wrong with my NN training process?
Is this a feasible problem? If yes, how?
If this is doable, can we still achieve that by constraining the weights to be in the set {-1, 0, 1}

Code:

import torch
import torch.nn as nn
import torch.optim as optim
import torch.utils.data as data_utils
import numpy as np

class Network(nn.Module):

     def __init__(self):

        super(Network, self).__init__()
        self.fc1 = nn.Linear(2,2,bias=False)
        self.fc2 = nn.Linear(2,1, bias=False)
        self.rl = nn.ReLU()

      def forward(self, x):
        x = self.fc1(x)
        x = self.rl(x)
        x = self.fc2(x)
        return x

#create an XOR data set to train
rng = np.random.RandomState(0)
X = rng.randn(200, 2)
y = np.logical_xor(X[:, 0] > 0, X[:, 1] > 0).astype('int32')

# test data set
X_test = np.array([[0,0],[0,1], [1,0], [1,1]])

train = data_utils.TensorDataset(torch.from_numpy(X).float(), \
                         torch.from_numpy(y).float())
train_loader = data_utils.DataLoader(train, batch_size=50, shuffle=True)

test = torch.from_numpy(X_test).float()

# training the network
num_epoch = 10000

net = Network()
net.fc1.weight.data.clamp_(min=-1, max=1)
net.fc2.weight.data.clamp_(min=-1, max=1)

# define loss and optimizer
criterion = nn.MSELoss()
optimizer = optim.Adam(net.parameters())

for epoch in range(num_epoch):
  running_loss = 0 # loss per epoch
  for (X, y)in train_loader:
    # make the grads zero
    optimizer.zero_grad()
    # forward propagate
    out = net(X)
    # calculate loss and update
    loss = criterion(out, y)
    loss.backward()
    optimizer.step()
    running_loss += loss.data
  if epoch%500== 0:
      print("Epoch: {0} Loss: {1}".format(epoch, running_loss))

The loss doesn't improve. It gets stuck in some value after a few epochs ( i'm not sure how to make this reproducible as I'm getting different values every time)

net(test) returns a set of predictions that are no way close to XOR output.


title: CycleGAN with 1-channel tiffs both as input and as output

body:
I am running CycleGAN with different types of tiffs in trainA and trainB. The tiffs are 256x256 pixels in size and have 1 channel per pixel. I am using tiffs to have a wide range of values.

I changed the code as suggested in the pytorch-CycleGAN-and-pix2pix
repo (https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/issues/320 and similar), but what I got out during the training in ./checkpoints are three-channels PNGs. Do you think it would be possible to change the code so that it goes from 1 channel tiff to 1 channel tiff with no information loss? As far as I understand, at present the code is converting the imported files to PNGs along the way. In other words: I would like my tensors to be [256*256*int_range,1]. Thanks for the help!


title: Discriminator loss and Generator loss not converging in regularisation phase of Adversarial autoencoder

body:
I am trying to implement a simple Adversarial autoencoder for MNIST dataset with pytorch. Only encoder-decoder(without regularisation phase) is converging with decrease in error as expected. But when I am trying to train discriminator and encoder for regularisation phase, generator loss is increasing and discriminator loss is decreasing, opposite of what is expected.

class Encoder(nn.Module):
    def __init__(self):
        super(Encoder, self).__init__()
        self.lin1 = nn.Linear(784, 400)
        self.lin2 = nn.Linear(400,100)
        self.lin3 = nn.Linear(100,2)
    def forward(self, x):
        #Without dropout
        x = self.lin1(x)
        x = F.relu(x)
        x = self.lin2(x)
        x = F.relu(x)
        x = self.lin3(x)
        return x
class Discriminator(nn.Module):
    def __init__(self):
        super(Discriminator, self).__init__()
        self.lin1 = nn.Linear(2,10)
        self.lin2 = nn.Linear(10,10)
        self.lin3 = nn.Linear(10,2)
    def forward(self, x):
        x = self.lin1(x)
        x = F.relu(x)
        x = self.lin2(x)
        x = F.relu(x)
        x = self.lin3(x)
        return torch.sigmoid(x)
encoder = Encoder().cuda()
discriminator = Discriminator().cuda()
encoder_gen_optimizer = optim.Adam(encoder.parameters(), lr = 3e-4)
discriminator_optimizer = optim.Adam(discriminator.parameters(), lr=3e-4)
for step in range(500):
    scheduler_disc.step()
    scheduler_gen.step()
    for batch_idx, (data, target) in enumerate(data_loader):
        x = Variable(data.view(data.size(0), -1)).cuda()

        #Regularisation loss - train discriminator to detect fake distribution
        discriminator_optimizer.zero_grad()
        encoder_gen_optimizer.zero_grad()
        z_fake = encoder(x)
        D_fake = discriminator(z_fake)
        target_fake = torch.zeros(batchsize, dtype=torch.int64).cuda()
        target_real = torch.ones(batchsize, dtype=torch.int64).cuda()
        z_real = Variable(torch.randn(batchsize,2)).cuda()
        D_real = discriminator(z_real)
        disc_loss = F.cross_entropy(D_real, target_real) + F.cross_entropy(D_fake, target_fake)
        disc_loss.backward()
        discriminator_optimizer.step()

        #Train generator(encoder) to generate normal distribution
        encoder_gen_optimizer.zero_grad()
        discriminator_optimizer.zero_grad()
        z = encoder(x)
        d = discriminator(z)
        t = torch.ones(batchsize, dtype=torch.int64).cuda()
        gen_loss = F.cross_entropy(d, t)
        gen_loss.backward()
        encoder_gen_optimizer.step()
        generator_loss.append(gen_loss)
        discriminator_loss.append(disc_loss)

Generator_Loss and Discriminator_Loss :


title: What does "Gathers values along an axis specified by dim." mean?

body:
I want to understand what does "Gathers values along an axis specified by dim." mean in the below code. How to structure the operation of function on data in my head. What is this function doing to data and how ?

Please refer this link https://pytorch.org/docs/stable/torch.html#torch.gather

torch.gather(input, dim, index, out=None, sparse_grad=False)

Gathers values along an axis specified by dim.

For a 3-D tensor the output is specified by:

out[i][j][k] = input[index[i][j][k]][j][k]  # if dim == 0
out[i][j][k] = input[i][index[i][j][k]][k]  # if dim == 1
out[i][j][k] = input[i][j][index[i][j][k]]  # if dim == 2


title: Vectorize loss function with complex indexing

body:
I have written a loss function in pytorch, however, it is too slow.
Thus, I'm trying to vectorize it.

Description

Given two meshes, it computes the distortion between patches (i.e. set of vertices) defined on them. The distortion is how much the distances between each pair of points changes.

Patches may have different sizes
This assumes a correspondence between M and N.
The loss accept batches.

Non-vectorized code

def loss_func(y_pred, batch):
    y_true = batch['vertices']
    patches: Tuple(Tensor,...) = batch['patches']

    loss = 0
    for i in range(y_pred.shape[0]):
        patch_loss = 0
        for patch in patches:
            dist_a = torch.pdist(y_pred[i][patch])
            dist_b = torch.pdist(y_true[i][patch])

            patch_loss += torch.sum((dist_a - dist_b) ** 2)

        loss += patch_loss
    return loss / y_pred.shape[0]

Vectorization attempt

def loss_func(y_pred, batch):
    y_true = batch['vertices']
    patches: Tuple(Tensor,...) = batch['patches']

    pred_dists = torch.cdist(y_pred, y_pred)
    true_dists = torch.cdist(y_true, y_true)

    diff_dists = (pred_dists - true_dists) ** 2

    loss = 0
    for patch in patches:
        d = diff_dists[:, patch, :][:, :, patch]
        loss += torch.sum(d, dim=(1, 2)) / 2

    return loss.mean()

Problem

It is still too slow.
I'm encountering difficulties in the vectorization since the patches may have different dimensions, thus they can't be put in a tensor.

I have the feeling that I could use directly the adjacency matrix to vectorize the code, without gathering the vertex indices in patches.

The patches are computed in this way:

def get_patch(vertex_index, vertex2vertex):
    return vertex2vertex[vertex_index, :].nonzero().squeeze(-1)

Working example

I prepared a self contained example to test and experiment with the two functions, with some benchmarks.
In the benchmark there is the same loss function without patches (i.e. just the distortion between all the vertices) as reference time

import torch
import time

device = 'cuda'
precision = torch.double
batch_size = 8
num_vertices = 400
dim = 2

y_pred = torch.rand(batch_size, num_vertices, dim, device=device, dtype=precision)
y_true = torch.rand(batch_size, num_vertices, dim, device=device, dtype=precision)

adjacency = (torch.rand(num_vertices, num_vertices, device=device, dtype=precision) > 0.95).to(precision)

def get_patch(vertex_index, vertex2vertex):
    return vertex2vertex[vertex_index, :].nonzero().squeeze(-1)

patches = tuple(get_patch(i, adjacency) for i in range(num_vertices))
batch = {
    'vertices': y_true,
    'patches': patches
}

def loss_func(y_pred, batch):
    y_true = batch['vertices']
    patches: Tuple(Tensor,...) = batch['patches']

    loss = 0
    for i in range(y_pred.shape[0]):
        patch_loss = 0
        for patch in patches:
            dist_a = torch.pdist(y_pred[i][patch])
            dist_b = torch.pdist(y_true[i][patch])

            patch_loss += torch.sum((dist_a - dist_b) ** 2)

        loss += patch_loss
    return loss / y_pred.shape[0]

def loss_func_vec(y_pred, batch):
    y_true = batch['vertices']
    patches: Tuple(Tensor,...) = batch['patches']

    pred_dists = torch.cdist(y_pred, y_pred)
    true_dists = torch.cdist(y_true, y_true)

    diff_dists = (pred_dists - true_dists) ** 2

    loss = 0
    for patch in patches:
        d = diff_dists[:, patch, :][:, :, patch]
        loss += torch.sum(d, dim=(1, 2)) / 2

    return loss.mean()

def reference_no_patches(y_pred, batch):
    y_true = batch['vertices']
    da = torch.cdist(y_pred, y_pred)
    db = torch.cdist(y_true, y_true)
    return torch.sum((da - db) ** 2) / y_true.shape[0] / 2

print(f'LOSS FUNCTION (ITERATION): TIME (LOSS)')
print()

for i in range(5):
    start_time = time.time()
    res = loss_func(y_pred, batch)
    torch.cuda.synchronize()
    print(f'Standard {i}:\t{time.time() - start_time:.4f}s ({res:.5f})')
print()

for i in range(5):
    start_time = time.time()
    res = loss_func_vec(y_pred, batch)
    torch.cuda.synchronize()
    print(f'Vectorized {i}:\t{time.time() - start_time:.4f}s ({res:.5f})')
print()

for i in range(5):
    start_time = time.time()
    res = reference_no_patches(y_pred, batch)
    torch.cuda.synchronize()
    print(f'Reference without patches {i}:\t{time.time() - start_time:.4f}s ({res:.5f})')

LOSS FUNCTION (ITERATION): TIME (LOSS)

Standard 0: 0.3311s (9744.87875)
Standard 1: 0.2853s (9744.87875)
Standard 2: 0.2929s (9744.87875)
Standard 3: 0.2972s (9744.87875)
Standard 4: 0.2714s (9744.87875)

Vectorized 0:   0.0254s (9744.87875)
Vectorized 1:   0.0251s (9744.87875)
Vectorized 2:   0.0262s (9744.87875)
Vectorized 3:   0.0242s (9744.87875)
Vectorized 4:   0.0241s (9744.87875)

Reference without patches 0:    0.0063s (9695.41083)
Reference without patches 1:    0.0062s (9695.41083)
Reference without patches 2:    0.0062s (9695.41083)
Reference without patches 3:    0.0066s (9695.41083)
Reference without patches 4:    0.0067s (9695.41083)

Context

The matrices involved are small. The number of vertices in each mesh is ~500, thus the adjacency matrix is [500, 500].
The patches may vary in dimension, however each one is << 500. There is one patch for each vertex (i.e. each vertex is the center of a patch), thus the number of patches is O(n!) (thus, I can't stack the patches into a single vector to perform the indexing).

I used the double precision in the example only to avoid numerical errors in the results, I plan to use float precision.

Finally, I will be happy to receive any suggestions on how to improve speed!

Update

Due to this bug I have to be careful with the memory allocation.
I rewrote the vectorized code as:

def loss_func_vec_mem(y_pred, batch):
    y_true = batch['vertices']
    patches: Tuple(Tensor,...) = batch['patches']

    loss = 0
    for patch in patches:
        pred = y_pred[:, patch, :]
        true = y_true[:, patch, :]

        pred_dists = torch.cdist(pred, pred)
        true_dists = torch.cdist(true, true)

        diff_dists = (pred_dists - true_dists) ** 2

        loss += torch.sum(diff_dists, dim=(1, 2))
    return loss.mean() / 2

Although it's even slower:

LOSS FUNCTION (ITERATION): TIME (LOSS)

Vec mem 0:  0.0484s (14685.45521)
Vec mem 1:  0.0489s (14685.45521)
Vec mem 2:  0.0471s (14685.45521)
Vec mem 3:  0.0471s (14685.45521)
Vec mem 4:  0.0459s (14685.45521)

Vectorized 0:   0.0271s (29370.91043)
Vectorized 1:   0.0262s (29370.91043)
Vectorized 2:   0.0320s (29370.91043)
Vectorized 3:   0.0266s (29370.91043)
Vectorized 4:   0.0263s (29370.91043)

Standard 0: 0.1680s (14685.45521)
Standard 1: 0.1614s (14685.45521)
Standard 2: 0.1639s (14685.45521)
Standard 3: 0.1606s (14685.45521)
Standard 4: 0.1694s (14685.45521)

Reference without patches 0:    0.0049s (12874.01275)
Reference without patches 1:    0.0105s (12874.01275)
Reference without patches 2:    0.0103s (12874.01275)
Reference without patches 3:    0.0092s (12874.01275)
Reference without patches 4:    0.0100s (12874.01275)


title: How to take depth of neural network as argument while constructing Network in Pytorch

body:
I have written following code to take depth of network as parameter in Pytorch.
Later I realized even if I am using many hidden layers, the learnable parameters remain the same.

class Net3(torch.nn.Module):
    def __init__(self, n_feature, n_hidden, n_output, depth, init):
        super(Net3, self).__init__()
        self.input = torch.nn.Linear(n_feature, n_hidden).float().to(device)
        self.hidden = torch.nn.Linear(n_hidden, n_hidden).float().to(device)
        self.predict = torch.nn.Linear(n_hidden, n_output).float().to(device)   =
        self.depth = depth

    def forward(self, x):
        x = F.relu(self.input(x))      # activation function for hidden layer
        for i in range(self.depth):
           x = F.relu(self.hidden(x))      # activation function for hidden layer
        x = self.predict(x)
        return x

Is there any other way to achieve this ?