title: Speeding up the trainning - RNN with LSTM in PyTorch

body:
I am trying to train a LSTM for energy demand forecast but it takes too long. I do not understand why because the model looks “simple” and there is no much data. Might it be because I am not using the DataLoader? How could I use it with RNN since I have a sequence?
Complete code is in Colab: https://colab.research.google.com/drive/130rG8_j1Lf8RQoVRrfXCeo5h_CcC5NU6?usp=sharing
The interesting part to be improved may be this:
for seq, y_train in train_data:
    optimizer.zero_grad()
    model.hidden = (torch.zeros(1,1,model.hidden_size),
                    torch.zeros(1,1,model.hidden_size))
    y_pred = model(seq)
    loss = criterion(y_pred, y_train)
    loss.backward()
    optimizer.step()

Thanks in advance to anyone helping me.


title: Linear classifier with PyTorch from scratch questions

body:
I'm trying to implement linear classifier in PyTorch, using 1 layer with tensors W and b, softmax and cross entropy loss. For each batch I have to:

Calculate logits
Transform logits to probabilities with softmax
Compute most probable classes
Calculate cross entropy between true and predicted classes
Use an optimizer to change W and b

So far I have (I have flat MNIST loaded with Scikit-learn):
# convert Numpy arrays to PyTorch tensor Variables
input_X_train = torch.from_numpy(X_train_flat).float().to(device)
input_X_val = torch.from_numpy(X_val_flat).float().to(device)
input_X_test = torch.from_numpy(X_test_flat).float().to(device)

input_y_train = torch.from_numpy(y_train).long().to(device)
input_y_val = torch.from_numpy(y_val).long().to(device)
input_y_test = torch.from_numpy(y_test).long().to(device)

# model parameters: W and b
W = torch.randn(input_dim, output_dim, device=device, dtype=dtype, requires_grad=True)
b = torch.randn(1, device=device, dtype=dtype, requires_grad=True)

BATCH_SIZE = 512
EPOCHS = 40
LEARNING_RATE = 1e-6

# create torch.optim.Adam optimizer for loss function minimization
optimizer = torch.optim.Adam([W, b], lr=LEARNING_RATE)

# create negative log loss function object for loss function evaluation
# use mean loss value from all batch samples
loss_fn = torch.nn.NLLLoss(reduction="mean")

for t in range(EPOCHS):
    # logits for input_X, resulting shape should be [input_X.shape[0], 10]
    logits = torch.matmul(input_X_train, W) + b

    # apply torch.nn.functional.softmax (torch_F.softmax) to logits
    probas = torch_f.softmax(logits, dim=1)

    # apply torch.argmax to find a class index with highest probability
    classes = torch.argmax(probas, dim=1)

    # loss should be a scalar number: average loss over all the objects with torch.mean()
    # PyTorch implements negative log loss (NLL) *without* log - you have to first compute log of
    # softmax, then negative log loss, which will swap sign

    # Use torch.nn.functional.log_softmax (torch_f.log_softmax) on top of input_y and logits
    # It is identical to calculating cross-entropy (log and then NLL) on top of probas,
    # but is more numerically friendly (read the docs).
    log_probas = torch_f.log_softmax(logits, dim=1)
    loss = loss_fn(log_probas, input_y_train)

    # Before the backward pass, use the optimizer object to zero all of the
    # gradients for the variables it will update (which are the learnable
    # weights of the model). This is because by default, gradients are
    # accumulated in buffers( i.e, not overwritten) whenever .backward()
    # is called. Checkout docs of torch.autograd.backward for more details.
    optimizer.zero_grad()

    # calculate backward gradients for backpropagation
    loss.backward()

    # Calling the step function on an Optimizer makes an update to its parameters
    optimizer.step()

For some reason, the W and b don't change. What am I doing wrong?
EDIT:
I've seen and tried in the code above e. g. this minimal working example https://discuss.pytorch.org/t/minimal-working-example-of-optim-sgd/11623/2.
EDIT 2:
Gradients W.grad are often, I think it should not be like that. Probabilities of classes are definitely right (so it's not e. g. like this example), since I've checked sum of every row and probabilities of all classes for each sample sum to 1.


title: increasing batch size during inference

body:
I believed that the inference time per batch was independent of the batch size when using a GPU, but this minimal example tells me that this doesn't seem true:
import torch
from torch import nn
from tqdm import tqdm

BATCH_SIZE = 32
N_ITER = 10000

class NN(nn.Module):
    def __init__(self):
        super(NN, self).__init__()
        self.layer = nn.Conv2d(3, 32, kernel_size=5, stride=1, padding=3, bias=False)

    def forward(self, input):
        out = self.layer(input)
        return out

cnn = NN().cuda()
cnn.eval()
tensor = torch.rand(BATCH_SIZE, 3, 999, 999).cuda()
with torch.no_grad():
    for _ in tqdm(range(N_ITER), mininterval=0.1):
        out = cnn(tensor)

When increasing BATCH_SIZE, the "it/s" shown by tqdm increases proportionally:
Plot of inference time vs batch size
It was my believe that the GPU can process the entire tensor simultaneously, as long as it doesn't use all the memory. Maybe I don't understand something about how GPUs process data in parallel, so I would appreciate some insights here.
I am using a NVIDIA GeForce 2080 Ti, pytorch 1.6.0 and CUDA 10.2.


title: Get the polygon coordinates of predicted output mask in YOLACT/YOLACT++

body:
I am using Yolact https://github.com/dbolya/yolact ,an instance segmentation algorithm which outputs the test image with a mask on the detected object. As the input images are given with the coordinates of polygons around the input classes in the annotations.json, I want to get an output like this. But I can't figure out how to extract the coordinates of those contours/polygons.
As far as I understood from this script https://github.com/dbolya/yolact/blob/master/eval.py the output is list of tensors for detected objects. It contains classes, scores, boxes and mask for evaluated image. The eval.py script returns recognized image with all this information. Recognition is saved in 'preds' in evalimg function (line 595), and post-processing of predict result is in the "def prep_display" (line 135)
Now how do I extract those polygon coordinates and save it in .JSON file or whatever else?
I also tried to look at these but couldn't figure out sadly!
https://github.com/dbolya/yolact/issues/286
and
https://github.com/dbolya/yolact/issues/256


title: Pytorch: copy.deepcopy vs torch.tensor.contiguous()?

body:
In python torch, it seems copy.deepcopy method is generally used to create deep-copies of torch tensors instead of creating views of existing tensors.
Meanwhile,
as far as I understood, the torch.tensor.contiguous() method turns a non-contiguous tensor into a contiguous tensor, or a view into a deeply copied tensor.
Then, do the two code lines below work equivalently if I want to deepcopy src_tensor into dst_tensor?
org_tensor = torch.rand(4)
src_tensor = org_tensor

dst_tensor = copy.deepcopy(src_tensor) # 1
dst_tensor = src_tensor.contiguous() # 2

If the two work equivalent, which method is better in deepcopying tensors?


title: Failed to install neural_render_pytorch

body:
I try to install neural_render_pytorch by command "pip install neural_renderer_pytorch".
But have a big error and i do not understand it.
Error message:
    Running setup.py install for neural-renderer-pytorch ... error
ERROR: Command errored out with exit status 1:
 command: /home/danyilzozulia/PycharmProjects/test/venv/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-bhf3anci/neural-renderer-pytorch_8cf9e80a49224bd798a82d088a984e58/setup.py'"'"'; __file__='"'"'/tmp/pip-install-bhf3anci/neural-renderer-pytorch_8cf9e80a49224bd798a82d088a984e58/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-ilhc49xm/install-record.txt --single-version-externally-managed --compile --install-headers /home/danyilzozulia/PycharmProjects/test/venv/include/site/python3.8/neural-renderer-pytorch
     cwd: /tmp/pip-install-bhf3anci/neural-renderer-pytorch_8cf9e80a49224bd798a82d088a984e58/
Complete output (209 lines):
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.8
creating build/lib.linux-x86_64-3.8/neural_renderer
copying neural_renderer/load_obj.py -> build/lib.linux-x86_64-3.8/neural_renderer
copying neural_renderer/projection.py -> build/lib.linux-x86_64-3.8/neural_renderer
copying neural_renderer/look.py -> build/lib.linux-x86_64-3.8/neural_renderer
copying neural_renderer/mesh.py -> build/lib.linux-x86_64-3.8/neural_renderer
copying neural_renderer/vertices_to_faces.py -> build/lib.linux-x86_64-3.8/neural_renderer
copying neural_renderer/perspective.py -> build/lib.linux-x86_64-3.8/neural_renderer
copying neural_renderer/look_at.py -> build/lib.linux-x86_64-3.8/neural_renderer
copying neural_renderer/renderer.py -> build/lib.linux-x86_64-3.8/neural_renderer
copying neural_renderer/get_points_from_angles.py -> build/lib.linux-x86_64-3.8/neural_renderer
copying neural_renderer/rasterize.py -> build/lib.linux-x86_64-3.8/neural_renderer
copying neural_renderer/__init__.py -> build/lib.linux-x86_64-3.8/neural_renderer
copying neural_renderer/lighting.py -> build/lib.linux-x86_64-3.8/neural_renderer
copying neural_renderer/save_obj.py -> build/lib.linux-x86_64-3.8/neural_renderer
creating build/lib.linux-x86_64-3.8/neural_renderer/cuda
copying neural_renderer/cuda/__init__.py -> build/lib.linux-x86_64-3.8/neural_renderer/cuda
running build_ext
building 'neural_renderer.cuda.load_textures' extension
creating /tmp/pip-install-bhf3anci/neural-renderer-pytorch_8cf9e80a49224bd798a82d088a984e58/build/temp.linux-x86_64-3.8
creating /tmp/pip-install-bhf3anci/neural-renderer-pytorch_8cf9e80a49224bd798a82d088a984e58/build/temp.linux-x86_64-3.8/neural_renderer
creating /tmp/pip-install-bhf3anci/neural-renderer-pytorch_8cf9e80a49224bd798a82d088a984e58/build/temp.linux-x86_64-3.8/neural_renderer/cuda
Emitting ninja build file /tmp/pip-install-bhf3anci/neural-renderer-pytorch_8cf9e80a49224bd798a82d088a984e58/build/temp.linux-x86_64-3.8/build.ninja...
Compiling objects...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/2] /usr/bin/nvcc --generate-dependencies-with-compile --dependency-output /tmp/pip-install-bhf3anci/neural-renderer-pytorch_8cf9e80a49224bd798a82d088a984e58/build/temp.linux-x86_64-3.8/neural_renderer/cuda/load_textures_cuda_kernel.o.d -I/home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include -I/home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/TH -I/home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/THC -I/home/danyilzozulia/PycharmProjects/test/venv/include -I/usr/include/python3.8 -c -c /tmp/pip-install-bhf3anci/neural-renderer-pytorch_8cf9e80a49224bd798a82d088a984e58/neural_renderer/cuda/load_textures_cuda_kernel.cu -o /tmp/pip-install-bhf3anci/neural-renderer-pytorch_8cf9e80a49224bd798a82d088a984e58/build/temp.linux-x86_64-3.8/neural_renderer/cuda/load_textures_cuda_kernel.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=load_textures -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_61,code=compute_61 -gencode=arch=compute_61,code=sm_61 -std=c++14
FAILED: /tmp/pip-install-bhf3anci/neural-renderer-pytorch_8cf9e80a49224bd798a82d088a984e58/build/temp.linux-x86_64-3.8/neural_renderer/cuda/load_textures_cuda_kernel.o
/usr/bin/nvcc --generate-dependencies-with-compile --dependency-output /tmp/pip-install-bhf3anci/neural-renderer-pytorch_8cf9e80a49224bd798a82d088a984e58/build/temp.linux-x86_64-3.8/neural_renderer/cuda/load_textures_cuda_kernel.o.d -I/home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include -I/home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/TH -I/home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/THC -I/home/danyilzozulia/PycharmProjects/test/venv/include -I/usr/include/python3.8 -c -c /tmp/pip-install-bhf3anci/neural-renderer-pytorch_8cf9e80a49224bd798a82d088a984e58/neural_renderer/cuda/load_textures_cuda_kernel.cu -o /tmp/pip-install-bhf3anci/neural-renderer-pytorch_8cf9e80a49224bd798a82d088a984e58/build/temp.linux-x86_64-3.8/neural_renderer/cuda/load_textures_cuda_kernel.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=load_textures -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_61,code=compute_61 -gencode=arch=compute_61,code=sm_61 -std=c++14
nvcc fatal   : Unknown option '-generate-dependencies-with-compile'
[2/2] c++ -MMD -MF /tmp/pip-install-bhf3anci/neural-renderer-pytorch_8cf9e80a49224bd798a82d088a984e58/build/temp.linux-x86_64-3.8/neural_renderer/cuda/load_textures_cuda.o.d -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include -I/home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/TH -I/home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/THC -I/home/danyilzozulia/PycharmProjects/test/venv/include -I/usr/include/python3.8 -c -c /tmp/pip-install-bhf3anci/neural-renderer-pytorch_8cf9e80a49224bd798a82d088a984e58/neural_renderer/cuda/load_textures_cuda.cpp -o /tmp/pip-install-bhf3anci/neural-renderer-pytorch_8cf9e80a49224bd798a82d088a984e58/build/temp.linux-x86_64-3.8/neural_renderer/cuda/load_textures_cuda.o -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=load_textures -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
FAILED: /tmp/pip-install-bhf3anci/neural-renderer-pytorch_8cf9e80a49224bd798a82d088a984e58/build/temp.linux-x86_64-3.8/neural_renderer/cuda/load_textures_cuda.o
c++ -MMD -MF /tmp/pip-install-bhf3anci/neural-renderer-pytorch_8cf9e80a49224bd798a82d088a984e58/build/temp.linux-x86_64-3.8/neural_renderer/cuda/load_textures_cuda.o.d -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include -I/home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/TH -I/home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/THC -I/home/danyilzozulia/PycharmProjects/test/venv/include -I/usr/include/python3.8 -c -c /tmp/pip-install-bhf3anci/neural-renderer-pytorch_8cf9e80a49224bd798a82d088a984e58/neural_renderer/cuda/load_textures_cuda.cpp -o /tmp/pip-install-bhf3anci/neural-renderer-pytorch_8cf9e80a49224bd798a82d088a984e58/build/temp.linux-x86_64-3.8/neural_renderer/cuda/load_textures_cuda.o -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=load_textures -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
In file included from /home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/ATen/Parallel.h:140,
                 from /home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/utils.h:3,
                 from /home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/nn/cloneable.h:5,
                 from /home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/nn.h:3,
                 from /home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/all.h:13,
                 from /home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/torch.h:3,
                 from /tmp/pip-install-bhf3anci/neural-renderer-pytorch_8cf9e80a49224bd798a82d088a984e58/neural_renderer/cuda/load_textures_cuda.cpp:1:
/home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/ATen/ParallelOpenMP.h:83: warning: ignoring #pragma omp parallel [-Wunknown-pragmas]
   83 | #pragma omp parallel for if ((end - begin) >= grain_size)
      |
/tmp/pip-install-bhf3anci/neural-renderer-pytorch_8cf9e80a49224bd798a82d088a984e58/neural_renderer/cuda/load_textures_cuda.cpp: In function ‘at::Tensor load_textures(at::Tensor, at::Tensor, at::Tensor, at::Tensor, int, int)’:
/tmp/pip-install-bhf3anci/neural-renderer-pytorch_8cf9e80a49224bd798a82d088a984e58/neural_renderer/cuda/load_textures_cuda.cpp:15:39: warning: ‘at::DeprecatedTypeProperties& at::Tensor::type() const’ is deprecated: Tensor.type() is deprecated. Instead use Tensor.options(), which in many cases (e.g. in a constructor) is a drop-in replacement. If you were using data from type(), that is now available from Tensor itself, so instead of tensor.type().scalar_type(), use tensor.scalar_type() instead and instead of tensor.type().backend() use tensor.device(). [-Wdeprecated-declarations]
   15 | #define CHECK_CUDA(x) AT_CHECK(x.type().is_cuda(), #x " must be a CUDA tensor")
      |                                       ^
/tmp/pip-install-bhf3anci/neural-renderer-pytorch_8cf9e80a49224bd798a82d088a984e58/neural_renderer/cuda/load_textures_cuda.cpp:17:24: note: in expansion of macro ‘CHECK_CUDA’
   17 | #define CHECK_INPUT(x) CHECK_CUDA(x); CHECK_CONTIGUOUS(x)
      |                        ^~~~~~~~~~
/tmp/pip-install-bhf3anci/neural-renderer-pytorch_8cf9e80a49224bd798a82d088a984e58/neural_renderer/cuda/load_textures_cuda.cpp:28:5: note: in expansion of macro ‘CHECK_INPUT’
   28 |     CHECK_INPUT(image);
      |     ^~~~~~~~~~~
In file included from /home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/ATen/Tensor.h:3,
                 from /home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/ATen/Context.h:4,
                 from /home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/ATen/ATen.h:9,
                 from /home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/types.h:3,
                 from /home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader_options.h:4,
                 from /home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/base.h:3,
                 from /home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/stateful.h:3,
                 from /home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader.h:3,
                 from /home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data.h:3,
                 from /home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/all.h:8,
                 from /home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/torch.h:3,
                 from /tmp/pip-install-bhf3anci/neural-renderer-pytorch_8cf9e80a49224bd798a82d088a984e58/neural_renderer/cuda/load_textures_cuda.cpp:1:
/home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/ATen/core/TensorBody.h:303:30: note: declared here
  303 |   DeprecatedTypeProperties & type() const {
      |                              ^~~~
/tmp/pip-install-bhf3anci/neural-renderer-pytorch_8cf9e80a49224bd798a82d088a984e58/neural_renderer/cuda/load_textures_cuda.cpp:15:23: error: ‘AT_CHECK’ was not declared in this scope; did you mean ‘DCHECK’?
   15 | #define CHECK_CUDA(x) AT_CHECK(x.type().is_cuda(), #x " must be a CUDA tensor")
      |                       ^~~~~~~~
/tmp/pip-install-bhf3anci/neural-renderer-pytorch_8cf9e80a49224bd798a82d088a984e58/neural_renderer/cuda/load_textures_cuda.cpp:15:23: note: in definition of macro ‘CHECK_CUDA’
   15 | #define CHECK_CUDA(x) AT_CHECK(x.type().is_cuda(), #x " must be a CUDA tensor")
      |                       ^~~~~~~~
/tmp/pip-install-bhf3anci/neural-renderer-pytorch_8cf9e80a49224bd798a82d088a984e58/neural_renderer/cuda/load_textures_cuda.cpp:28:5: note: in expansion of macro ‘CHECK_INPUT’
   28 |     CHECK_INPUT(image);
      |     ^~~~~~~~~~~
/tmp/pip-install-bhf3anci/neural-renderer-pytorch_8cf9e80a49224bd798a82d088a984e58/neural_renderer/cuda/load_textures_cuda.cpp:15:39: warning: ‘at::DeprecatedTypeProperties& at::Tensor::type() const’ is deprecated: Tensor.type() is deprecated. Instead use Tensor.options(), which in many cases (e.g. in a constructor) is a drop-in replacement. If you were using data from type(), that is now available from Tensor itself, so instead of tensor.type().scalar_type(), use tensor.scalar_type() instead and instead of tensor.type().backend() use tensor.device(). [-Wdeprecated-declarations]
   15 | #define CHECK_CUDA(x) AT_CHECK(x.type().is_cuda(), #x " must be a CUDA tensor")
      |                                       ^
/tmp/pip-install-bhf3anci/neural-renderer-pytorch_8cf9e80a49224bd798a82d088a984e58/neural_renderer/cuda/load_textures_cuda.cpp:17:24: note: in expansion of macro ‘CHECK_CUDA’
   17 | #define CHECK_INPUT(x) CHECK_CUDA(x); CHECK_CONTIGUOUS(x)
      |                        ^~~~~~~~~~
/tmp/pip-install-bhf3anci/neural-renderer-pytorch_8cf9e80a49224bd798a82d088a984e58/neural_renderer/cuda/load_textures_cuda.cpp:29:5: note: in expansion of macro ‘CHECK_INPUT’
   29 |     CHECK_INPUT(faces);
      |     ^~~~~~~~~~~
In file included from /home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/ATen/Tensor.h:3,
                 from /home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/ATen/Context.h:4,
                 from /home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/ATen/ATen.h:9,
                 from /home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/types.h:3,
                 from /home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader_options.h:4,
                 from /home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/base.h:3,
                 from /home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/stateful.h:3,
                 from /home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader.h:3,
                 from /home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data.h:3,
                 from /home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/all.h:8,
                 from /home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/torch.h:3,
                 from /tmp/pip-install-bhf3anci/neural-renderer-pytorch_8cf9e80a49224bd798a82d088a984e58/neural_renderer/cuda/load_textures_cuda.cpp:1:
/home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/ATen/core/TensorBody.h:303:30: note: declared here
  303 |   DeprecatedTypeProperties & type() const {
      |                              ^~~~
/tmp/pip-install-bhf3anci/neural-renderer-pytorch_8cf9e80a49224bd798a82d088a984e58/neural_renderer/cuda/load_textures_cuda.cpp:15:39: warning: ‘at::DeprecatedTypeProperties& at::Tensor::type() const’ is deprecated: Tensor.type() is deprecated. Instead use Tensor.options(), which in many cases (e.g. in a constructor) is a drop-in replacement. If you were using data from type(), that is now available from Tensor itself, so instead of tensor.type().scalar_type(), use tensor.scalar_type() instead and instead of tensor.type().backend() use tensor.device(). [-Wdeprecated-declarations]
   15 | #define CHECK_CUDA(x) AT_CHECK(x.type().is_cuda(), #x " must be a CUDA tensor")
      |                                       ^
/tmp/pip-install-bhf3anci/neural-renderer-pytorch_8cf9e80a49224bd798a82d088a984e58/neural_renderer/cuda/load_textures_cuda.cpp:17:24: note: in expansion of macro ‘CHECK_CUDA’
   17 | #define CHECK_INPUT(x) CHECK_CUDA(x); CHECK_CONTIGUOUS(x)
      |                        ^~~~~~~~~~
/tmp/pip-install-bhf3anci/neural-renderer-pytorch_8cf9e80a49224bd798a82d088a984e58/neural_renderer/cuda/load_textures_cuda.cpp:30:5: note: in expansion of macro ‘CHECK_INPUT’
   30 |     CHECK_INPUT(is_update);
      |     ^~~~~~~~~~~
In file included from /home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/ATen/Tensor.h:3,
                 from /home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/ATen/Context.h:4,
                 from /home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/ATen/ATen.h:9,
                 from /home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/types.h:3,
                 from /home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader_options.h:4,
                 from /home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/base.h:3,
                 from /home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/stateful.h:3,
                 from /home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader.h:3,
                 from /home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data.h:3,
                 from /home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/all.h:8,
                 from /home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/torch.h:3,
                 from /tmp/pip-install-bhf3anci/neural-renderer-pytorch_8cf9e80a49224bd798a82d088a984e58/neural_renderer/cuda/load_textures_cuda.cpp:1:
/home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/ATen/core/TensorBody.h:303:30: note: declared here
  303 |   DeprecatedTypeProperties & type() const {
      |                              ^~~~
/tmp/pip-install-bhf3anci/neural-renderer-pytorch_8cf9e80a49224bd798a82d088a984e58/neural_renderer/cuda/load_textures_cuda.cpp:15:39: warning: ‘at::DeprecatedTypeProperties& at::Tensor::type() const’ is deprecated: Tensor.type() is deprecated. Instead use Tensor.options(), which in many cases (e.g. in a constructor) is a drop-in replacement. If you were using data from type(), that is now available from Tensor itself, so instead of tensor.type().scalar_type(), use tensor.scalar_type() instead and instead of tensor.type().backend() use tensor.device(). [-Wdeprecated-declarations]
   15 | #define CHECK_CUDA(x) AT_CHECK(x.type().is_cuda(), #x " must be a CUDA tensor")
      |                                       ^
/tmp/pip-install-bhf3anci/neural-renderer-pytorch_8cf9e80a49224bd798a82d088a984e58/neural_renderer/cuda/load_textures_cuda.cpp:17:24: note: in expansion of macro ‘CHECK_CUDA’
   17 | #define CHECK_INPUT(x) CHECK_CUDA(x); CHECK_CONTIGUOUS(x)
      |                        ^~~~~~~~~~
/tmp/pip-install-bhf3anci/neural-renderer-pytorch_8cf9e80a49224bd798a82d088a984e58/neural_renderer/cuda/load_textures_cuda.cpp:31:5: note: in expansion of macro ‘CHECK_INPUT’
   31 |     CHECK_INPUT(textures);
      |     ^~~~~~~~~~~
In file included from /home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/ATen/Tensor.h:3,
                 from /home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/ATen/Context.h:4,
                 from /home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/ATen/ATen.h:9,
                 from /home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/types.h:3,
                 from /home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader_options.h:4,
                 from /home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/base.h:3,
                 from /home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/stateful.h:3,
                 from /home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader.h:3,
                 from /home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data.h:3,
                 from /home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/all.h:8,
                 from /home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/torch.h:3,
                 from /tmp/pip-install-bhf3anci/neural-renderer-pytorch_8cf9e80a49224bd798a82d088a984e58/neural_renderer/cuda/load_textures_cuda.cpp:1:
/home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/include/ATen/core/TensorBody.h:303:30: note: declared here
  303 |   DeprecatedTypeProperties & type() const {
      |                              ^~~~
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
  File "/home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1667, in _run_ninja_build
    subprocess.run(
  File "/usr/lib/python3.8/subprocess.py", line 512, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/pip-install-bhf3anci/neural-renderer-pytorch_8cf9e80a49224bd798a82d088a984e58/setup.py", line 29, in <module>
    setup(
  File "/home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/setuptools/__init__.py", line 153, in setup
    return distutils.core.setup(**attrs)
  File "/usr/lib/python3.8/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/lib/python3.8/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/setuptools/command/install.py", line 61, in run
    return orig.install.run(self)
  File "/usr/lib/python3.8/distutils/command/install.py", line 589, in run
    self.run_command('build')
  File "/usr/lib/python3.8/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/usr/lib/python3.8/distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
  File "/usr/lib/python3.8/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 79, in run
    _build_ext.run(self)
  File "/usr/lib/python3.8/distutils/command/build_ext.py", line 340, in run
    self.build_extensions()
  File "/home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 708, in build_extensions
    build_ext.build_extensions(self)
  File "/usr/lib/python3.8/distutils/command/build_ext.py", line 449, in build_extensions
    self._build_extensions_serial()
  File "/usr/lib/python3.8/distutils/command/build_ext.py", line 474, in _build_extensions_serial
    self.build_extension(ext)
  File "/home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 196, in build_extension
    _build_ext.build_extension(self, ext)
  File "/usr/lib/python3.8/distutils/command/build_ext.py", line 528, in build_extension
    objects = self.compiler.compile(sources,
  File "/home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 529, in unix_wrap_ninja_compile
    _write_ninja_file_and_compile_objects(
  File "/home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1354, in _write_ninja_file_and_compile_objects
    _run_ninja_build(
  File "/home/danyilzozulia/PycharmProjects/test/venv/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1683, in _run_ninja_build
    raise RuntimeError(message) from e
RuntimeError: Error compiling objects for extension

I am new in CUDA and pytorch so can you help, please, resolve this problem.
I`m using CUDA V10.1, pytorch V1.8.0, putorch3d 0.3.0.


title: pytorch batch normalization in distributed train

body: wondering how distributed pytorch handle batch norm, when I add a batch norm layer, will pytorch engine use the same allreduce call to sync the data cross node? or the batch norm only happen on local node.


title: PyTorch Image Classification with custom Dataset

body:
I'm still a beginner in terms of AI and Neuronal Networks and during the time this time of learning I'm trying to do some examples but I have a problem and no idea how can I fix it. If any of you would be able to help me, I would be really grateful.

What I'm trying to do?

A simple image classification with 10 types of animals using PyTorch with some custom Dataset.

My images

Each image is going to be with a shape as (3, 200, 200)

Also I have something like 40 images on each folder (train and test)

How dose it look my data folders?

train

cat
dog
...
rat

test

cat
dog
...
rat

My model

class NetModel(nn.Module):
    def __init__(self):
        super(NetModel, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 5 * 5)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

def train_model(model, train_loader, optimizer, criterion, epochs, save_path):
    for epoch in range(epochs):  # loop over the dataset multiple times
        running_loss = 0.0
        for i, data in enumerate(train_loader, 0):
            # get the inputs; data is a list of [inputs, labels]
            inputs, labels = data
            # zero the parameter gradients
            optimizer.zero_grad()

            # forward + backward + optimize
            outputs = model(inputs)
            loss = criterion(outputs, labels)
            loss.backward()
            optimizer.step()

            # print statistics
            running_loss += loss.item()
            if i % 2000 == 1999:  # print every 2000 mini-batches
                print('[%d, %5d] loss: %.3f' %
                      (epoch + 1, i + 1, running_loss / 2000))
                running_loss = 0.0
    print('Finished Training')
    # PATH = './cifar_net.pth'
    torch.save(model.state_dict(), save_path)
    print('Saved in: {}'.format(save_path))

def test(model, test_loader, classes, saved_path):
    model.load_state_dict(torch.load(saved_path))

    dataiter = iter(test_loader)
    images, labels = dataiter.next()

    print('GroundTruth: ', ' '.join('%5s' % classes[labels[j]] for j in range(4)))

    outputs = model(images)

    _, predicted = torch.max(outputs, 1)

    print('Predicted: ', ' '.join('%5s' % classes[predicted[j]] for j in range(4)))

transform = transforms.Compose(
    [transforms.ToTensor(),
     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

charset = string.ascii_letters + "-' "
trainset = TESNamesDataset('../data/train', charset, 10)
testset = TESNamesDataset('../data/test', charset, 10)

trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,
                                          shuffle=True, num_workers=2)

testloader = torch.utils.data.DataLoader(testset, batch_size=4,
                                         shuffle=False, num_workers=2)

model = NetModel()

criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(model.parameters(), lr=0.001, momentum=0.9)

path = "../check_point/saved_model.pth"
train_model(model, trainloader, optimizer, criterion, 100, path)
test(model, testloader, animals, path)

My custom dataset

class TESNamesDataset(Dataset):
    def __init__(self, data_root, charset, length):
        self.data_root = data_root
        self.charset = charset + '\0'
        self.length = length
        self.samples = []
        self.char_codec = LabelEncoder()
        self._init_dataset()

    def __len__(self):
        return len(self.samples)

    def __getitem__(self, idx):
        name, pixels = self.samples[idx]
        return self.one_hot_sample(name), pixels

    def _init_dataset(self):
        names = set()
        # self.samples = []
        for animal in os.listdir(self.data_root):
            animal_filepath = os.path.join(self.data_root, animal)
            names.add(animal)
            for img_name in os.listdir(animal_filepath):
                img_path = os.path.join(animal_filepath, img_name)
                im = cv2.imread(img_path)
                if len(animal) < self.length:
                    animal += '\0' * (self.length - len(animal))
                else:
                    animal = animal[:self.length - 1] + '\0'
                self.samples.append((animal, im))

        self.char_codec.fit(list(self.charset))

    def to_one_hot(self, codec, values):
        value_idxs = codec.transform(values)
        return torch.eye(len(codec.classes_))[value_idxs]

    def one_hot_sample(self, name):
        t_name = self.to_one_hot(self.char_codec, list(name))
        return t_name

The ERROR message

  RuntimeError: Expected 4-dimensional input for 4-dimensional weight 6
  3 5 5, but got 3-dimensional input of size [4, 10, 56] instead

Can any of you tell me what I'm doing wrong? In case that this question is off-topic just let me know where I can ask it.

Thanks


title: how to create a pytorch NN with 2 hidden layer with nn.Sequential?

body:
how to add an another hidden layer in this 1 layer model? :

model = nn.Sequential(OrderedDict([ ('fc1', nn.Linear(D_in, H)),
                                    ('Tanh', nn.Tanh()),
                                    ('fc2', nn.Linear(H, D_out))]))


title: Is it mandatory in pytorch to add modules to ModuleList to access its parameters

body:
I read some posts about ModuleList and all of them said that adding modules to ModuleList gives access to parameters of the Neural Network but in “Training a classifier” example of 60 mins blitz pytorch tutorial the modules are not added to any ModuleList and still the parameters could be accessed using

optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)

This is confusing. Please clarify how the parameters are accessible even though the modules have not been added to any ModuleList

class mmodel(nn.Module):
    def __init__(self):
        super(mmodel,self).__init__()
        self.lst=[]
        self.lst.append(nn.Linear(1,1))
        self.lst.append(nn.Linear(1,1))
        self.mlist = nn.ModuleList([nn.Linear(2,1,bias=False),nn.Linear(3,1)])
    def forward(self,x):
        for m in self.lst:
            print(type(m))
            x = m(x)
        for m in self.mlist:
            print(type(m))

mm = mmodel()
mm(X)
print(list(mm.parameters()))

<class 'torch.nn.modules.linear.Linear'>
<class 'torch.nn.modules.linear.Linear'>
<class 'torch.nn.modules.linear.Linear'>
<class 'torch.nn.modules.linear.Linear'>
[Parameter containing:
tensor([[0.2302, 0.3712]], requires_grad=True), Parameter containing:
tensor([[-0.3451, -0.0274,  0.3990]], requires_grad=True), Parameter containing:
tensor([0.3258], requires_grad=True)]

As evident from the above outputs the parameters of modules added in a python list are not visible in the forward method. Only the parameters of the modules added in the ModuleList are visible in the forward method


title: Why Does PyTorch Use So Much GPU Memory to Store Tensors?

body:
I made a simple test using PyTorch which involves measuring the current GPU memory use, creating a tensor of a certain size, moving it to the GPU, and measuring the GPU memory again. According to my calculation, around 6.5k bytes were required to store each element of the tensor! Here is the break down:

GPU memory use before creating the tensor as shown by nvidia-smi: 384 MiB.
Create a tensor with 100,000 random elements:

a = torch.rand(100000)

Transfer the tensor to the GPU:

device = torch.device('cuda')
b = a.to(device)

GPU memory use after the transfer: 1020 MiB
Calculate memory change per one element of the tensor

(1020-384)*1024*1024/len(b)
# Answer is 6668.94336

This is weird, to say the least. Why would 6.5 KiB of GPU memory be required to store a single float32 element?

UPDATE: Following Robert Crovella's suggestion in the comment, I created another tensor c and then moved it to the CUDA device on d. The GPU memory usage didn't increase. So, it seems that PyTorch or CUDA requires some 636 MiB for bootstrapping. Why is that? What is this memory used for? It seems a lot to me!


title: Onnxruntime Test Error after Successfully Converting Midas Model to ONNX

body:
Finally, I ask my first question. I've been struggling the past few days with this and couldn't find the same question here.
I converted this pretrained model to ONNX with this following codes:
import torch
from midas import midas_net
import onnx

model_path = "model-f46da743.pt"
model = midas_net.MidasNet(model_path, non_negative=True)
sample = torch.randn(1,3,384,672) # input_shape
torch.onnx.export(model,
                  sample,
                  "midas.onnx",
                  opset_version=11)

onnx_model = onnx.load("midas.onnx")
# Print a human readable representation of the graph
graph_output = onnx.helper.printable_graph(onnx_model.graph)
with open("graph.txt", mode="w") as fout:
    fout.write(graph_output)

I opened the midas.onnx with Netron and it looked good. And then I tried to test it with onnxruntime:
import torch
import utils
import cv2
from torchvision.transforms import Compose
from midas import transforms
import onnxruntime # Tool for scoring ONNX models
import onnx

# select device
device = torch.device("cpu")
# print("device: %s" % device)

model = "/home/ipu/libraries/MiDaS/midas.onnx"
sess = onnxruntime.InferenceSession(model) # Load a ONNX model
# print(sess) # <onnxruntime.capi.session.InferenceSession object at 0x7f07d5e10f10>

input_name = sess.get_inputs()[0].name
# print(input_name) # input.1

input_shape = sess.get_inputs()[0].shape
# print("input shape", input_shape) # [1, 3, 384, 672]

output_name = sess.get_outputs()[0].name
# print(output_name) # 1176

output_shape = sess.get_outputs()[0].shape
# print("output shape", output_shape) # [1, 384, 672]

# Check that the Intermediate Representation (IR) is well formed
# print("check onnx model ", onnx.checker.check_model(onnx.load(model))) # None

transform = Compose(
    [
        transforms.Resize(
            384,
            384,
            resize_target=None,
            keep_aspect_ratio=True,
            ensure_multiple_of=32,
            resize_method="lower_bound",
            image_interpolation_method=cv2.INTER_CUBIC,
        ),
        transforms.NormalizeImage(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
        transforms.PrepareForNet(),
    ]
)

# get input
img = utils.read_image("input/20200629_123103.jpg")
img_input = transform({"image": img})["image"]

sample = torch.from_numpy(img_input).to(device).unsqueeze(0)
# print("sample", sample)
# print("sample shape", sample.shape) # torch.Size([1, 3, 384, 672])
# print(type(sample)) # <class 'torch.Tensor'>

def to_numpy(tensor):
    return tensor.detach().cpu().numpy() if tensor.requires_grad else tensor.cpu().numpy()

# print("to numpy sample", to_numpy(sample))
# print("to numpy sample shape", to_numpy(sample).shape) # (1, 3, 384, 672)
# print(type(to_numpy(sample))) # <class 'numpy.ndarray'>

# Input must be a list of dictionaries or a single numpy array
ort_outs = sess.run([output_name], {input_name: to_numpy(sample)})
print(ort_outs)

But, I got this error:
2020-07-15 10:55:23.402764485 [E:onnxruntime:, sequential_executor.cc:281 Execute] Non-zero status code returned while running Resize node. Name:'' Status Message: /onnxruntime_src/onnxruntime/core/providers/cpu/tensor/upsample.h:283 void onnxruntime::UpsampleBase::ScalesValidation(const std::vector<float>&, onnxruntime::UpsampleMode) const scales.size() == 2 || (scales.size() == 4 && scales[0] == 1 && scales[1] == 1) was false. 'Linear' mode and 'Cubic' mode only support 2-D inputs ('Bilinear', 'Bicubic') or 4-D inputs with the corresponding outermost 2 scale values being 1 in the Resize operator

Traceback (most recent call last):
  File "simple_test.py", line 73, in <module>
    ort_outs = sess.run([output_name], {input_name: to_numpy(sample)})
  File "/home/ipu/anaconda3/envs/midas/lib/python3.7/site-packages/onnxruntime/capi/session.py", line 111, in run
    return self._sess.run(output_names, input_feed, run_options)
onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running Resize node. Name:'' Status Message: /onnxruntime_src/onnxruntime/core/providers/cpu/tensor/upsample.h:283 void onnxruntime::UpsampleBase::ScalesValidation(const std::vector<float>&, onnxruntime::UpsampleMode) const scales.size() == 2 || (scales.size() == 4 && scales[0] == 1 && scales[1] == 1) was false. 'Linear' mode and 'Cubic' mode only support 2-D inputs ('Bilinear', 'Bicubic') or 4-D inputs with the corresponding outermost 2 scale values being 1 in the Resize operator

For more info, I use Pytorch (1.3.1), ONNX (1.7.0) and Onnxruntime (1.3.0)
I would really appreciate any help.


title: "RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn " error BertFoeSequenceClassification

body:
I am trying to build Bert model for Arabic Text classification task using pretrained model from https://github.com/alisafaya/Arabic-BERT
i want to know the exact difference between the two statement:
model_name = 'kuisailab/albert-large-arabic'
model = AutoModel.from_pretrained(model_name)
model = BertForSequenceClassification .from_pretrained(model_name)

I fine-tuned the model by adding the following layers on top of the model:
for param in model.parameters():
    param.requires_grad = False

model.classifier = nn.Sequential(
nn.Dropout(0.5),
nn.ReLU(),
nn.Linear(768,512),
nn.Linear(512,2),
nn.LogSoftmax(dim=1),
nn.Softmax(dim=1)
)
model = model.to(device)

and used the optimizer:
optimizer = AdamW(model.parameters(),
                  lr = 2e-5)

finally this is my training loop:
model.train()

for idx, row in train_data.iterrows():
    text_parts = preprocess_text(str(row['sentence']))
    label = torch.tensor([row['label']]).long().to(device)

    optimizer.zero_grad()

    overall_output = torch.zeros((1, 2)).float().to(device)
    for part in text_parts:
        if len(part) > 0:
            try:
                input = part.reshape(-1)[:512].reshape(1, -1)
                # print(input.shape)
                overall_output += model(input, labels=label)[1].float().to(device)
            except Exception as e:
                print(str(e))

#     overall_output /= len(text_parts)
    overall_output = F.softmax(overall_output[0], dim=-1)

    if label == 0:
        label = torch.tensor([1.0, 0.0]).float().to(device)
    elif label == 1:
        label = torch.tensor([0.0, 1.0]).float().to(device)

    # print(overall_output, label)

    loss = criterion(overall_output, label)
    total_loss += loss.item()

    loss.backward()
    optimizer.step()

and i get the error:
mat1 dim 1 must match mat2 dim 0
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-33-5c2f0fea6c1f> in <module>()
     39     total_loss += loss.item()
     40
---> 41     loss.backward()
     42     optimizer.step()
     43

1 frames
/usr/local/lib/python3.6/dist-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
    130     Variable._execution_engine.run_backward(
    131         tensors, grad_tensors_, retain_graph, create_graph,
--> 132         allow_unreachable=True)  # allow_unreachable flag
    133
    134

RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

any idea how to solve this error


title: Triton inference server serving TorchScript model

body:
I am trying to serve a TorchScript model with the triton (tensorRT) inference server. But every time I start the server it throws the following error:
PytorchStreamReader failed reading zip archive: failed finding central directory

My folder structure is :
<model_repository>
  <model_name>
   config.pbtxt
   <1>
    <model.pt>

My config.pbtxt file is :
name: "model"
platform: "pytorch_libtorch"
max_batch_size: 1

input[
{
name: "INPUT__0"
data_type:  TYPE_FP32
dims: [-1,3,-1,-1]
}
]

output:[
{
name: "OUTPUT__0"
data_type:  TYPE_FP32
dims: [-1,1,-1,-1]
}
]


title: RuntimeError: Could not run 'aten::xxxx', Can't load Pytorch model trained with TPU

body:
Training for MLM was added based on the Japanese model of BERT.
At that time, we used TPU on Google Colab.
I get the following error when loading the created model.
Is there a way to load the model?

code

from transformers import BertJapaneseTokenizer, BertForMaskedLM

# Load pre-trained model
model = BertForMaskedLM.from_pretrained('/content/drive/My Drive/Bert/models/sample/')
model.eval()

output

RuntimeError                              Traceback (most recent call last)
/usr/local/lib/python3.6/dist-packages/transformers/modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    469             try:
--> 470                 state_dict = torch.load(resolved_archive_file, map_location="cpu")
    471             except Exception:

/usr/local/lib/python3.6/dist-packages/torch/serialization.py in load(f, map_location, pickle_module, **pickle_load_args)
    528                 return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
--> 529         return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
    530

/usr/local/lib/python3.6/dist-packages/torch/serialization.py in _legacy_load(f, map_location, pickle_module, **pickle_load_args)
    701     unpickler.persistent_load = persistent_load
--> 702     result = unpickler.load()
    703

/usr/local/lib/python3.6/dist-packages/torch/_utils.py in _rebuild_xla_tensor(data, dtype, device, requires_grad)
    151 def _rebuild_xla_tensor(data, dtype, device, requires_grad):
--> 152     tensor = torch.from_numpy(data).to(dtype=dtype, device=device)
    153     tensor.requires_grad = requires_grad

RuntimeError: Could not run 'aten::empty.memory_format' with arguments from the 'XLATensorId' backend. 'aten::empty.memory_format' is only available for these backends: [CUDATensorId, SparseCPUTensorId, VariableTensorId, CPUTensorId, MkldnnCPUTensorId, SparseCUDATensorId].


title: Binary DenseNet 121 Classifier only predicting positive with probability >0.5

body:
I borrowed code from this github repo for training of a DenseNet-121 [https://github.com/gaetandi/cheXpert/blob/master/cheXpert_final.ipynb][1]

The github code is for 14 class classification on the CheXpert chest X-ray dataset. I've revised it for binary classification.

# initialize and load the model
pathModel = "/ds2/images/model_ones_2epoch_densenet.tar"#"m-epoch0-07032019-213933.pth.tar"

I initialize the 14 class model so I can use the pretrained weights:

model = DenseNet121(nnClassCount).cuda()

model = torch.nn.DataParallel(model).cuda()
modelCheckpoint = torch.load(pathModel)
model.load_state_dict(modelCheckpoint['state_dict'])

And then convert to binary classification:

nnClassCount = 1
model.module.densenet121.classifier = nn.Sequential(
    nn.Linear(1024, nnClassCount),
    nn.Sigmoid()
).cuda()
model = torch.nn.DataParallel(model).cuda()

And then train via:

batch, losst, losse = CheXpertTrainer.train(model, dataLoaderTrain, dataLoaderVal, nnClassCount, 100, timestampLaunch, checkpoint = None, weight_path = weight_path)

My training data is laid out in a 2 column csv with column headers ('Path' and 'Class-Positive'), with path locations in the first column and 0 or 1 in the second column. I used oversampling when compiling the training list so paths in the csv are roughly a 50/50 split between 0's and 1's...shuffled.

I use livelossplot to monitor training/validation loss and accuracy. My loss plots look as expected but accuracy plots are flatlined around 0.5 (which makes sense given the 50/50 data if the net is saying its 100% positive or negative). I'm assuming I'm doing something wrong in how I'm doing predictions, but maybe something in the training is incorrect.

For predictions and probabilities I'm running:

 varOutput = model(varInput)
    _, preds = torch.max(varOutput, 1)
    print('varshape: ',varOutput.shape)
    probs = torch.sigmoid(varOutput)

*My issue: preds are all coming out as 0 and probs all above 0.5 *

Here is the initial code from github:

import os
import numpy as np
import time
import sys
import csv
import cv2
import matplotlib.pyplot as plt

import torch
import torch.nn as nn
import torch.backends.cudnn as cudnn
import torchvision
import torchvision.transforms as transforms
import torch.optim as optim
import torch.nn.functional as tfunc
from torch.utils.data import Dataset
from torch.utils.data.dataset import random_split
from torch.utils.data import DataLoader
from torch.optim.lr_scheduler import ReduceLROnPlateau
from PIL import Image
import torch.nn.functional as func

from sklearn.metrics.ranking import roc_auc_score
import sklearn.metrics as metrics
import random

use_gpu = torch.cuda.is_available()
# Paths to the files with training, and validation sets.
# Each file contains pairs (path to image, output vector)
pathFileTrain = '../CheXpert-v1.0-small/train.csv'
pathFileValid = '../CheXpert-v1.0-small/valid.csv'

# Neural network parameters:
nnIsTrained = False                 #pre-trained using ImageNet
nnClassCount = 14                   #dimension of the output

# Training settings: batch size, maximum number of epochs
trBatchSize = 64
trMaxEpoch = 3

# Parameters related to image transforms: size of the down-scaled image, cropped image
imgtransResize = (320, 320)
imgtransCrop = 224

# Class names
class_names = ['No Finding', 'Enlarged Cardiomediastinum', 'Cardiomegaly', 'Lung Opacity',
               'Lung Lesion', 'Edema', 'Consolidation', 'Pneumonia', 'Atelectasis', 'Pneumothorax',
               'Pleural Effusion', 'Pleural Other', 'Fracture', 'Support Devices']
class CheXpertDataSet(Dataset):
    def __init__(self, image_list_file, transform=None, policy="ones"):
        """
        image_list_file: path to the file containing images with corresponding labels.
        transform: optional transform to be applied on a sample.
        Upolicy: name the policy with regard to the uncertain labels
        """
        image_names = []
        labels = []

        with open(image_list_file, "r") as f:
            csvReader = csv.reader(f)
            next(csvReader, None)
            k=0
            for line in csvReader:
                k+=1
                image_name= line[0]
                label = line[5:]

                for i in range(14):
                    if label[i]:
                        a = float(label[i])
                        if a == 1:
                            label[i] = 1
                        elif a == -1:
                            if policy == "ones":
                                label[i] = 1
                            elif policy == "zeroes":
                                label[i] = 0
                            else:
                                label[i] = 0
                        else:
                            label[i] = 0
                    else:
                        label[i] = 0

                image_names.append('../' + image_name)
                labels.append(label)

        self.image_names = image_names
        self.labels = labels
        self.transform = transform

    def __getitem__(self, index):
        """Take the index of item and returns the image and its labels"""

        image_name = self.image_names[index]
        image = Image.open(image_name).convert('RGB')
        label = self.labels[index]
        if self.transform is not None:
            image = self.transform(image)
        return image, torch.FloatTensor(label)

    def __len__(self):
        return len(self.image_names)
#TRANSFORM DATA

normalize = transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
transformList = []
#transformList.append(transforms.Resize(imgtransCrop))
transformList.append(transforms.RandomResizedCrop(imgtransCrop))
transformList.append(transforms.RandomHorizontalFlip())
transformList.append(transforms.ToTensor())
transformList.append(normalize)
transformSequence=transforms.Compose(transformList)
#LOAD DATASET

dataset = CheXpertDataSet(pathFileTrain ,transformSequence, policy="ones")
datasetTest, datasetTrain = random_split(dataset, [500, len(dataset) - 500])
datasetValid = CheXpertDataSet(pathFileValid, transformSequence)
#Problèmes de l'overlapping de patients et du transform identique ?

dataLoaderTrain = DataLoader(dataset=datasetTrain, batch_size=trBatchSize, shuffle=True,  num_workers=24, pin_memory=True)
dataLoaderVal = DataLoader(dataset=datasetValid, batch_size=trBatchSize, shuffle=False, num_workers=24, pin_memory=True)
dataLoaderTest = DataLoader(dataset=datasetTest, num_workers=24, pin_memory=True)
class CheXpertTrainer():

    def train (model, dataLoaderTrain, dataLoaderVal, nnClassCount, trMaxEpoch, launchTimestamp, checkpoint):

        #SETTINGS: OPTIMIZER & SCHEDULER
        optimizer = optim.Adam (model.parameters(), lr=0.0001, betas=(0.9, 0.999), eps=1e-08, weight_decay=1e-5)

        #SETTINGS: LOSS
        loss = torch.nn.BCELoss(size_average = True)

        #LOAD CHECKPOINT
        if checkpoint != None and use_gpu:
            modelCheckpoint = torch.load(checkpoint)
            model.load_state_dict(modelCheckpoint['state_dict'])
            optimizer.load_state_dict(modelCheckpoint['optimizer'])

        #TRAIN THE NETWORK
        lossMIN = 100000

        for epochID in range(0, trMaxEpoch):

            timestampTime = time.strftime("%H%M%S")
            timestampDate = time.strftime("%d%m%Y")
            timestampSTART = timestampDate + '-' + timestampTime

            batchs, losst, losse = CheXpertTrainer.epochTrain(model, dataLoaderTrain, optimizer, trMaxEpoch, nnClassCount, loss)
            lossVal = CheXpertTrainer.epochVal(model, dataLoaderVal, optimizer, trMaxEpoch, nnClassCount, loss)

            timestampTime = time.strftime("%H%M%S")
            timestampDate = time.strftime("%d%m%Y")
            timestampEND = timestampDate + '-' + timestampTime

            if lossVal < lossMIN:
                lossMIN = lossVal
                torch.save({'epoch': epochID + 1, 'state_dict': model.state_dict(), 'best_loss': lossMIN, 'optimizer' : optimizer.state_dict()}, 'm-epoch'+str(epochID)+'-' + launchTimestamp + '.pth.tar')
                print ('Epoch [' + str(epochID + 1) + '] [save] [' + timestampEND + '] loss= ' + str(lossVal))
            else:
                print ('Epoch [' + str(epochID + 1) + '] [----] [' + timestampEND + '] loss= ' + str(lossVal))

        return batchs, losst, losse
    #--------------------------------------------------------------------------------

    def epochTrain(model, dataLoader, optimizer, epochMax, classCount, loss):

        batch = []
        losstrain = []
        losseval = []

        model.train()

        for batchID, (varInput, target) in enumerate(dataLoaderTrain):

            varTarget = target.cuda(non_blocking = True)

            #varTarget = target.cuda()

            varOutput = model(varInput)
            lossvalue = loss(varOutput, varTarget)

            optimizer.zero_grad()
            lossvalue.backward()
            optimizer.step()

            l = lossvalue.item()
            losstrain.append(l)

            if batchID%35==0:
                print(batchID//35, "% batches computed")
                #Fill three arrays to see the evolution of the loss

                batch.append(batchID)

                le = CheXpertTrainer.epochVal(model, dataLoaderVal, optimizer, trMaxEpoch, nnClassCount, loss).item()
                losseval.append(le)

                print(batchID)
                print(l)
                print(le)

        return batch, losstrain, losseval

    #--------------------------------------------------------------------------------

    def epochVal(model, dataLoader, optimizer, epochMax, classCount, loss):

        model.eval()

        lossVal = 0
        lossValNorm = 0

        with torch.no_grad():
            for i, (varInput, target) in enumerate(dataLoaderVal):

                target = target.cuda(non_blocking = True)
                varOutput = model(varInput)

                losstensor = loss(varOutput, target)
                lossVal += losstensor
                lossValNorm += 1

        outLoss = lossVal / lossValNorm
        return outLoss

    #--------------------------------------------------------------------------------

    #---- Computes area under ROC curve
    #---- dataGT - ground truth data
    #---- dataPRED - predicted data
    #---- classCount - number of classes

    def computeAUROC (dataGT, dataPRED, classCount):

        outAUROC = []

        datanpGT = dataGT.cpu().numpy()
        datanpPRED = dataPRED.cpu().numpy()

        for i in range(classCount):
            try:
                outAUROC.append(roc_auc_score(datanpGT[:, i], datanpPRED[:, i]))
            except ValueError:
                pass
        return outAUROC

    #--------------------------------------------------------------------------------

    def test(model, dataLoaderTest, nnClassCount, checkpoint, class_names):

        cudnn.benchmark = True

        if checkpoint != None and use_gpu:
            modelCheckpoint = torch.load(checkpoint)
            model.load_state_dict(modelCheckpoint['state_dict'])

        if use_gpu:
            outGT = torch.FloatTensor().cuda()
            outPRED = torch.FloatTensor().cuda()
        else:
            outGT = torch.FloatTensor()
            outPRED = torch.FloatTensor()

        model.eval()

        with torch.no_grad():
            for i, (input, target) in enumerate(dataLoaderTest):

                target = target.cuda()
                outGT = torch.cat((outGT, target), 0).cuda()

                bs, c, h, w = input.size()
                varInput = input.view(-1, c, h, w)

                out = model(varInput)
                outPRED = torch.cat((outPRED, out), 0)
        aurocIndividual = CheXpertTrainer.computeAUROC(outGT, outPRED, nnClassCount)
        aurocMean = np.array(aurocIndividual).mean()

        print ('AUROC mean ', aurocMean)

        for i in range (0, len(aurocIndividual)):
            print (class_names[i], ' ', aurocIndividual[i])

        return outGT, outPRED
class DenseNet121(nn.Module):
    """Model modified.
    The architecture of our model is the same as standard DenseNet121
    except the classifier layer which has an additional sigmoid function.
    """
    def __init__(self, out_size):
        super(DenseNet121, self).__init__()
        self.densenet121 = torchvision.models.densenet121(pretrained=True)
        num_ftrs = self.densenet121.classifier.in_features
        self.densenet121.classifier = nn.Sequential(
            nn.Linear(num_ftrs, out_size),
            nn.Sigmoid()
        )

    def forward(self, x):
        x = self.densenet121(x)
        return x
# initialize and load the model
model = DenseNet121(nnClassCount).cuda()
model = torch.nn.DataParallel(model).cuda()
timestampTime = time.strftime("%H%M%S")
timestampDate = time.strftime("%d%m%Y")
timestampLaunch = timestampDate + '-' + timestampTime

batch, losst, losse = CheXpertTrainer.train(model, dataLoaderTrain, dataLoaderVal, nnClassCount, trMaxEpoch, timestampLaunch, checkpoint = None)
print("Model trained")


title: What is the point of multinomial vs argmax evaluation of accuracy?

body:
What is the purpose of evaluating prediction accuracy using multinomial instead of the straight up argmax?

probs_Y = torch.softmax(model(test_batch, feature_1, feature_2), 1)

sampled_Y = torch.multinomial(probs_Y, 1)
argmax_Y = torch.max(probs_Y, 1)[1].view(-1, 1)

print('Accuracy of sampled predictions on the test set: {:.4f}%'.format(
    (test_Y == sampled_Y.float()).sum().item() / len(test_Y) * 100))
print('Accuracy of argmax predictions on the test set: {:4f}%'.format(
    (test_Y == argmax_Y.float()).sum().item() / len(test_Y) * 100))

Result:

Accuracy of sampled predictions on the test set: 88.8889%

Accuracy of argmax predictions on the test set: 97.777778%

Reading the pytorch docs it looks like multinomial is sampling according to some distribution - just not sure how that is relevant in assessing accuracy.

I've noticed that the multinomial is non-deterministic - meaning that it is outputting a different accuracy, presumably by including different samples, each time it runs.


title: ValueError: not enough values to unpack (expected 4, got 1): Pytorch

body:
I am getting this error while running the eval report. I trained my model with bert-base-german-cased for my custom dataset.

The code is as follows:

    from sklearn.metrics import mean_squared_error, matthews_corrcoef, confusion_matrix
    from scipy.stats import pearsonr

    def get_mismatched(labels, preds):
        mismatched = labels != preds
        examples = processor.get_dev_examples(args['data_dir'])
        wrong = [i for (i, v) in zip(examples, mismatched) if v]

        return wrong

    def get_eval_report(labels, preds):
        mcc = matthews_corrcoef(labels, preds)
        tn, fp, fn, tp = confusion_matrix(labels, preds).ravel()
        return {
            "mcc": mcc,
            "tp": tp,
            "tn": tn,
            "fp": fp,
            "fn": fn
        }, get_mismatched(labels, preds)

    def compute_metrics(task_name, preds, labels):
        assert len(preds) == len(labels)
        return get_eval_report(labels, preds)

    def evaluate(model, tokenizer, prefix=""):
        # Loop to handle MNLI double evaluation (matched, mis-matched)
        eval_output_dir = args['output_dir']

        results = {}
        EVAL_TASK = args['task_name']

        eval_dataset = load_and_cache_examples(EVAL_TASK, tokenizer, evaluate=True)
        if not os.path.exists(eval_output_dir):
            os.makedirs(eval_output_dir)

        eval_sampler = SequentialSampler(eval_dataset)
        eval_dataloader = DataLoader(eval_dataset, sampler=eval_sampler, batch_size=args['eval_batch_size'])

        # Eval!
        logger.info("***** Running evaluation {} *****".format(prefix))
        logger.info("  Num examples = %d", len(eval_dataset))
        logger.info("  Batch size = %d", args['eval_batch_size'])
        eval_loss = 0.0
        nb_eval_steps = 0
        preds = None
        out_label_ids = None
        for batch in tqdm_notebook(eval_dataloader, desc="Evaluating"):
            model.eval()
            batch = tuple(t.to(device) for t in batch)

            with torch.no_grad():
                inputs = {'input_ids':      batch[0],
                          'attention_mask': batch[1],
                          'token_type_ids': batch[2] if args['model_type'] in ['bert', 'xlnet'] else None,  # XLM don't use segment_ids
                          'labels':         batch[3]}
                outputs = model(**inputs)
                tmp_eval_loss, logits = outputs[:2]

                eval_loss += tmp_eval_loss.mean().item()
            nb_eval_steps += 1
            if preds is None:
                preds = logits.detach().cpu().numpy()
                out_label_ids = inputs['labels'].detach().cpu().numpy()
            else:
                preds = np.append(preds, logits.detach().cpu().numpy(), axis=0)
                out_label_ids = np.append(out_label_ids, inputs['labels'].detach().cpu().numpy(), axis=0)

        eval_loss = eval_loss / nb_eval_steps
        if args['output_mode'] == "classification":
            preds = np.argmax(preds, axis=1)
        elif args['output_mode'] == "regression":
            preds = np.squeeze(preds)
        result, wrong = compute_metrics(EVAL_TASK, preds, out_label_ids)
        results.update(result)

        output_eval_file = os.path.join(eval_output_dir, "eval_results.txt")
        with open(output_eval_file, "w") as writer:
            logger.info("***** Eval results {} *****".format(prefix))
            for key in sorted(result.keys()):
                logger.info("  %s = %s", key, str(result[key]))
                writer.write("%s = %s\n" % (key, str(result[key])))

        return results, wrong

    ValueError                                Traceback (most recent call last)
    <ipython-input-33-0d7757abd7dd> in <module>
         10         model = model_class.from_pretrained(checkpoint)
         11         model.to(device)
    ---> 12         result, wrong_preds = evaluate(model, tokenizer, prefix=global_step)
         13         result = dict((k + '_{}'.format(global_step), v) for k, v in result.items())
         14         results.update(result)

    <ipython-input-30-c0946f859f89> in evaluate(model, tokenizer, prefix)
         76     elif args['output_mode'] == "regression":
         77         preds = np.squeeze(preds)
    ---> 78     result, wrong = compute_metrics(EVAL_TASK, preds, out_label_ids)
         79     results.update(result)
         80

    <ipython-input-30-c0946f859f89> in compute_metrics(task_name, preds, labels)
         25 def compute_metrics(task_name, preds, labels):
         26     assert len(preds) == len(labels)
    ---> 27     return get_eval_report(labels, preds)
         28
         29 def evaluate(model, tokenizer, prefix=""):

    <ipython-input-30-c0946f859f89> in get_eval_report(labels, preds)
         14 def get_eval_report(labels, preds):
         15     mcc = matthews_corrcoef(labels, preds)
    ---> 16     tn, fp, fn, tp = confusion_matrix(labels, preds).ravel()
         17     return {
         18         "mcc": mcc,

When i run the do_eval function the following error happend. How do I fix this error?


title: Proper way to log things when using Pytorch Lightning DDP

body:
I was wondering what is the proper way of logging metrics when using DDP. I noticed that if I want to print something inside validation_epoch_end it will be printed twice when using 2 GPUs. I was expecting validation_epoch_end to be called only on rank 0 and to receive the outputs from all GPUs, but I am not sure this is correct anymore. Therefore I have several questions:

validation_epoch_end(self, outputs) - When using DDP does every subprocess receive the data processed from the current GPU or data processed from all GPUs, i.e. does the input parameter outputs contains the outputs of the entire validation set, from all GPUs?
If outputs is GPU/process specific what is the proper way to calculate any metric on the entire validation set in validation_epoch_end when using DDP?

I understand that I can solve the printing by checking self.global_rank == 0 and printing/logging only in that case, however I am trying to get a deeper understanding of what I am printing/logging in this case.
Here is a code snippet from my use case. I would like to be able to report f1, precision and recall on the entire validation dataset and I am wondering what is the correct way of doing it when using DDP.
    def _process_epoch_outputs(self,
                               outputs: List[Dict[str, Any]]
                               ) -> Tuple[torch.Tensor, torch.Tensor]:
        """Creates and returns tensors containing all labels and predictions

        Goes over the outputs accumulated from every batch, detaches the
        necessary tensors and stacks them together.

        Args:
            outputs (List[Dict])
        """
        all_labels = []
        all_predictions = []

        for output in outputs:
            for labels in output['labels'].detach():
                all_labels.append(labels)

            for predictions in output['predictions'].detach():
                all_predictions.append(predictions)

        all_labels = torch.stack(all_labels).long().cpu()
        all_predictions = torch.stack(all_predictions).cpu()

        return all_predictions, all_labels

    def validation_epoch_end(self, outputs: List[Dict[str, Any]]) -> None:
        """Logs f1, precision and recall on the validation set."""

        if self.global_rank == 0:
            print(f'Validation Epoch: {self.current_epoch}')

        predictions, labels = self._process_epoch_outputs(outputs)
        for i, name in enumerate(self.label_columns):

            f1, prec, recall, t = metrics.get_f1_prec_recall(predictions[:, i],
                                                             labels[:, i],
                                                             threshold=None)
            self.logger.experiment.add_scalar(f'{name}_f1/Val',
                                              f1,
                                              self.current_epoch)
            self.logger.experiment.add_scalar(f'{name}_Precision/Val',
                                              prec,
                                              self.current_epoch)
            self.logger.experiment.add_scalar(f'{name}_Recall/Val',
                                              recall,
                                              self.current_epoch)

            if self.global_rank == 0:
                print((f'F1: {f1}, Precision: {prec}, '
                       f'Recall: {recall}, Threshold {t}'))


title: How to solve AttributeError when using torch?

body:
I am learning PyTorch and facing a problem. There's always AttributeError when I try to run the code if I use torch.xxx.

Here is my environment:
Windows 10, cuda 10.0.132, Pycharm 2018.3.3, Python 3.7.
I used Anaconda prompt and followed instruction on https://pytorch.org/get-started/locally/ to install PyTorch. Here's the command:
pip install https://download.pytorch.org/whl/cu100/torch-1.0.0-cp37-cp37m-win_amd64.whl
pip install torchvision
Then I set the interpreter in PyCharm as the python.exe in the path where the above environment is.

Here are something I tried after error cames out:
change torch.xx to torch._C.xx
pip install pylint
set the interpreter as system interpreter instead of conda interpreter
None of which helps.

this is the code at first:

    import torch
    import matplotlib.pyplot as plt  # 画图的模块
    x = torch.unsqueeze(torch.linspace(-2, 2, 500), dim=1)

error:

    Traceback (most recent call last):
      File "E:/Python/Winter/torch.py", line 1, in <module>
        import torch
      File "E:\Python\Winter\torch.py", line 6, in <module>
        x = torch.unsqueeze(torch.linspace(-2, 2, 500), dim=1)
    AttributeError: module 'torch' has no attribute 'unsqueeze'

then I tried this:

    import torch
    print(torch.FloatTensor([1, 2, 3], [4, 5, 6]))

same error:

    Traceback (most recent call last):
      File "E:/Python/Winter/torch.py", line 1, in <module>
        import torch
      File "E:\Python\Winter\torch.py", line 3, in <module>
        print(torch.FloatTensor([1, 2, 3], [4, 5, 6]))
    AttributeError: module 'torch' has no attribute 'FloatTensor'

I want to install and use PyTorch correctly.
 Thank you.


title: CIFAR-10 test set classification accuracy different on PyTorch and Keras

body:
I’ve made a custom CNN in PyTorch for classifying 10 classes in the CIFAR-10 dataset. My classification accuracy on the test dataset is 45.739%, this is very low and I thought it’s because my model is not very deep but I implemented the same model in Keras and the classification accuracy come outs to be 78.92% on test dataset. No problem in Keras however I think there's something I'm missing in my PyTorch program.

I have used the same model architecture, strides, padding, dropout rate, optimizer, loss function, learning rate, batch size, number of epochs on both PyTorch and Keras and despite that, the difference in the classification accuracy is still huge thus I’m not able to decide how I should debug my PyTorch program further.

For now I suspect 3 things: in Keras, I use the categorical cross entropy loss function (one hot vector labels) and in PyTorch I use the standard cross entropy loss function (scalar indices labels), can this be a problem?, if not then I suspect either my training loop or the code for calculating classification accuracy in PyTorch. I have attached both my programs below, will be grateful to any suggestions.

My program in Keras:

#================Function that defines the CNN model===========
def CNN_model():
    model = Sequential()

    model.add(Conv2D(32,(3,3),activation='relu',padding='same', input_shape=(size,size,channels))) #SAME PADDING
    model.add(Conv2D(32,(3,3),activation='relu')) #VALID PADDING
    model.add(MaxPooling2D(pool_size=(2,2))) #VALID PADDING
    model.add(Dropout(0.25))

    model.add(Conv2D(64,(3,3),activation='relu', padding='same')) #SAME PADDING
    model.add(Conv2D(64,(3,3),activation='relu')) #VALID PADDING
    model.add(MaxPooling2D(pool_size=(2,2))) #VALID PADDING
    model.add(Dropout(0.25))

    model.add(Conv2D(128,(3,3),activation='relu', padding='same')) #SAME PADDING
    model.add(Conv2D(128,(3,3),activation='relu')) #VALID PADDING
    model.add(MaxPooling2D(pool_size=(2,2),name='feature_extractor_layer')) #VALID PADDING
    model.add(Dropout(0.25))

    model.add(Flatten())

    model.add(Dense(512, activation='relu', name='second_last_layer'))
    model.add(Dropout(0.25))
    model.add(Dense(10, activation='softmax', name='softmax_layer')) #10 nodes in the softmax layer
    model.summary()

    return model

#=====Main program starts here========
#get_train_data() and get_test_data() are my own custom functions to get CIFAR-10 dataset
images_train, labels_train, class_train = get_train_data(0,10)
images_test, labels_test, class_test = get_test_data(0,10)

model = CNN_model()

model.compile(loss='categorical_crossentropy', #loss function of the CNN
             optimizer=Adam(lr=1.0e-4), #Optimizer
              metrics=['accuracy'])#'accuracy' metric is to be evaluated

#images_train and images_test contain images and
#class_train and class_test contains one hot vectors labels
model.fit(images_train,class_train,
          batch_size=128,
          epochs=50,
          validation_data=(images_test,class_test),
          verbose=1)

scores=model.evaluate(images_test,class_test,verbose=0)
print("Accuracy: "+str(scores[1]*100)+"% \n")

My program in PyTorch:

#========DEFINE THE CNN MODEL=====
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 32, 3,1,1)#SAME PADDING
        self.conv2 = nn.Conv2d(32,32,3,1,0)#VALID PADDING
        self.pool1 = nn.MaxPool2d(2,2) #VALID PADDING
        self.drop1 = nn.Dropout2d(0.25) #DROPOUT OF 0.25

        self.conv3 = nn.Conv2d(32,64,3,1,1)#SAME PADDING
        self.conv4 = nn.Conv2d(64,64,3,1,0)#VALID PADDING
        self.pool2 = nn.MaxPool2d(2,2)#VALID PADDING
        self.drop2 = nn.Dropout2d(0.25) #DROPOUT OF 0.25

        self.conv5 = nn.Conv2d(64,128,3,1,1)#SAME PADDING
        self.conv6 = nn.Conv2d(128,128,3,1,0)#VALID PADDING
        self.pool3 = nn.MaxPool2d(2,2)#VALID PADDING
        self.drop3 = nn.Dropout2d(0.25) #DROPOUT OF 0.25

        self.fc1 = nn.Linear(128*2*2, 512)#128*2*2 IS OUTPUT DIMENSION AFTER THE PREVIOUS LAYER
        self.drop4 = nn.Dropout(0.25) #DROPOUT OF 0.25
        self.fc2 = nn.Linear(512,10) #10 output nodes

    def forward(self, x):
        x = F.relu(self.conv1(x))
        x = F.relu(self.conv2(x))
        x = self.pool1(x)
        x = self.drop1(x)

        x = F.relu(self.conv3(x))
        x = F.relu(self.conv4(x))
        x = self.pool2(x)
        x = self.drop2(x)

        x = F.relu(self.conv5(x))
        x = F.relu(self.conv6(x))
        x = self.pool3(x)
        x = self.drop3(x)

        x = x.view(-1,2*2*128) #FLATTENING OPERATION 2*2*128 IS OUTPUT AFTER THE PREVIOUS LAYER
        x = F.relu(self.fc1(x))
        x = self.drop4(x)
        x = self.fc2(x) #LAST LAYER DOES NOT NEED SOFTMAX BECAUSE THE LOSS FUNCTION WILL TAKE CARE OF IT
        return x

#=======FUNCTION TO CONVERT INPUT AND TARGET TO TORCH TENSORS AND LOADING INTO GPU======
def PrepareInputDataAndTargetData(device,images,labels,batch_size):

    #GET MINI BATCH OF TRAINING IMAGES AND RESHAPE THE TORCH TENSOR FOR CNN PROCESSING
    mini_batch_images = torch.tensor(images)
    mini_batch_images = mini_batch_images.view(batch_size,3,32,32)

    #GET MINI BATCH OF TRAINING LABELS, TARGET SHOULD BE IN LONG FORMAT SO CONVERT THAT TOO
    mini_batch_labels = torch.tensor(labels)
    mini_batch_labels = mini_batch_labels.long()

    #FEED THE INPUT DATA AND TARGET LABELS TO GPU
    mini_batch_images = mini_batch_images.to(device)
    mini_batch_labels = mini_batch_labels.to(device)

    return mini_batch_images,mini_batch_labels

#==========MAIN PROGRAM==========
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
#get_train_data() and get_test_data() are my own custom functions to get CIFAR-10 dataset
Images_train, Labels_train, Class_train = get_train_data(0,10)
Images_test, Labels_test, Class_test = get_test_data(0,10)

net = Net()
net = net.double() #https://discuss.pytorch.org/t/runtimeerror-expected-object-of-scalar-type-double-but-got-scalar-type-float-for-argument-2-weight/38961
print(net)

#MAP THE MODEL ONTO THE GPU
net = net.to(device)

#CROSS ENTROPY LOSS FUNCTION AND ADAM OPTIMIZER
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(net.parameters(), lr=1e-4)

#PREPARE THE DATALOADER
#Images_train contains images and Labels_trains contains indices i.e. 0,1,...,9
dataset = TensorDataset( Tensor(Images_train), Tensor(Labels_train) )
trainloader = DataLoader(dataset, batch_size= 128, shuffle=True)

#START TRAINING THE CNN MODEL FOR 50 EPOCHS
for epoch in range(0,50):
    for i, data in enumerate(trainloader, 0):
        inputs, labels = data
        inputs = torch.tensor(inputs).double()
        inputs = inputs.view(len(inputs),3,32,32) #RESHAPE THE IMAGES
        labels = labels.long() #MUST CONVERT LABEL TO LONG FORMAT

        #MAP THE INPUT AND LABELS TO THE GPU
        inputs=inputs.to(device)
        labels=labels.to(device)

        #FORWARD PROP, BACKWARD PROP, PARAMETER UPDATE
        optimizer.zero_grad()
        outputs = net.forward(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

#CALCULATE CLASSIFICATION ACCURACY ON ALL 10 CLASSES
with torch.no_grad():
    Images_class,Labels_class = PrepareInputDataAndTargetData(device,Images_test,Labels_test,len(Images_test))
    network_outputs = net.forward(Images_class)
    correct = (torch.argmax(network_outputs.data,1) == Labels_class.data).float().sum()
    acc = float(100.0*(correct/len(Images_class)))
    print("Accuracy is: "+str(acc)+"\n")

    del Images_class
    del Labels_class
    del network_outputs
    del correct
    del acc
    torch.cuda.empty_cache()

print("Done\n")

I am not fully aware of how the actual core backend works in both libraries however I suppose that the classification accuracy of any model should be almost the same regardless of the library.


title: How can I do return_sequences for a stacked LSTM model with PyTorch?

body:
I have a Tensorflow / Keras model with:

        self.model.add(Bidirectional(LSTM(lstm1_size, input_shape=(
            seq_length, feature_dim), return_sequences=True)))
        self.model.add(BatchNormalization())
        self.model.add(Dropout(0.2))

        self.model.add(Bidirectional(
            LSTM(lstm2_size, return_sequences=True)))
        self.model.add(BatchNormalization())
        self.model.add(Dropout(0.2))

        # BOTTLENECK HERE

        self.model.add(Bidirectional(
            LSTM(lstm3_size, return_sequences=True)))
        self.model.add(BatchNormalization())
        self.model.add(Dropout(0.2))

        self.model.add(Bidirectional(
            LSTM(lstm4_size, return_sequences=True)))
        self.model.add(BatchNormalization())
        self.model.add(Dropout(0.2))

        self.model.add(Bidirectional(
            LSTM(lstm5_size, return_sequences=True)))
        self.model.add(BatchNormalization())
        self.model.add(Dropout(0.2))

        self.model.add(Dense(feature_dim, activation='linear'))

How do I create a stacked PyTorch model with the return_sequences? My understanding of return_sequences is that it returns the "output" of each layer of LSTM's which are then fed into the next layer.

How would I accomplish this with PyToch?


I need your help. Running the code below throws:

RuntimeError: size mismatch, m1: [5 x 10], m2: [5 x 32] at /pytorch/aten/src/TH/generic/THTensorMath.cpp

I looked at similar questions but they are image related and suggest flattening the input, I tried them with no luck.

I'm using Python 3.6.8 and torch 1.1.0

code sample:

state = [[307, 1588204935.0, 1.0869, 1.08708, 1.08659, 1.08662, 1.08708, 1.08724, 1.08674, 1.08677],
          [370, 1588204920.0, 1.08668, 1.08709, 1.08661, 1.08693, 1.08682, 1.08724, 1.08677, 1.08708],
          [243, 1588204905.0, 1.08637, 1.08671, 1.08624, 1.08669, 1.08651, 1.08686, 1.08639, 1.08683],
          [232, 1588204890.0, 1.08614, 1.08656, 1.08609, 1.08636, 1.08628, 1.0867, 1.08626, 1.0865],
          [349, 1588204875.0, 1.086, 1.08623, 1.08585, 1.08614, 1.08614, 1.08638, 1.08597, 1.0863]]

def predict(state, state_dim=5, action_dim=3, hidden_dim=32, lr=0.05):
    """ Compute Q values for all actions using the DQL. """
    criterion = torch.nn.MSELoss()
    model = torch.nn.Sequential(torch.nn.Linear(state_dim, hidden_dim),
                                     torch.nn.LeakyReLU(),
                                     torch.nn.Linear(hidden_dim, hidden_dim*2),
                                     torch.nn.LeakyReLU(),
                                     torch.nn.Linear(hidden_dim*2, action_dim))

    optimizer = torch.optim.Adam(model.parameters(), lr)

    with torch.no_grad():
        return model(torch.Tensor(state)) # Error throw here

predict(state).tolist()


title: PyTorch - RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 0. Got 224 and 244 in dimension 2

body:
I am trying to do semantic segmentation with two classes.
I have 224x224x3 images and 224x224 binary segmentation masks. I am reshaping the masks to be 224x224x1 (I read somewhere that this is the format that I should pass to the model).
When I try to loop through the train data loader it either runs without errors or I get the following error (randomly):
Traceback (most recent call last):
  File "/Users/nikolaykolibarov/Desktop/GATE/rooftop-recognition/rooftop-edge-segmentation-ml/main.py", line 43, in <module>
    for i, sample in enumerate(train_loader):
  File "/Users/nikolaykolibarov/opt/anaconda3/envs/dtcc/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 346, in __next__
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/Users/nikolaykolibarov/opt/anaconda3/envs/dtcc/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
    return self.collate_fn(data)
  File "/Users/nikolaykolibarov/opt/anaconda3/envs/dtcc/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py", line 79, in default_collate
    return [default_collate(samples) for samples in transposed]
  File "/Users/nikolaykolibarov/opt/anaconda3/envs/dtcc/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py", line 79, in <listcomp>
    return [default_collate(samples) for samples in transposed]
  File "/Users/nikolaykolibarov/opt/anaconda3/envs/dtcc/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py", line 55, in default_collate
    return torch.stack(batch, 0, out=out)
RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 0. Got 224 and 244 in dimension 2 at /tmp/pip-req-build-9oilk29k/aten/src/TH/generic/THTensor.cpp:689

Most of the issues with the same exception I found online don’t have matching numbers like I do (Got 224 and 224). I don’t understand, this exception should occur when they aren’t equal (if I am not wrong).
When I run it, occasionally I get this error a few times, and then it doesn’t occur a few times.
I am not sure if I messed something up with the data types and shapes or it is something else.
Here is the code:
roof_edges_dataset.py
import os
import cv2 as cv
from torch.utils.data import Dataset
from torchvision.transforms import transforms

from utils import create_binary_mask, get_labelme_shapes, plot_segmentation_dataset

class RoofEdgesDataset(Dataset):
    def __init__(self, im_path, ann_path, transform=None):
        self.im_path = im_path
        self.ann_path = ann_path
        self.transform = transform

        self.im_fn_list = sorted(os.listdir(im_path), key=lambda x: int(x.split('.')[0]))
        self.ann_fn_list = sorted(os.listdir(ann_path), key=lambda x: int(x.split('.')[0]))

    def __len__(self):
        return len(self.im_fn_list)

    def __getitem__(self, index):
        im_path = os.path.join(self.im_path, self.im_fn_list[index])
        im = cv.imread(im_path)

        ann_path = os.path.join(self.ann_path, self.ann_fn_list[index])
        ann = create_binary_mask(im, get_labelme_shapes(ann_path))
        ann = ann.reshape(ann.shape[0], ann.shape[1], 1)
        ann = transforms.ToTensor()(ann)

        if self.transform:
            im = self.transform(im)

        return im, ann

main.py:
import torch
import torchvision.transforms as transforms
from torch.utils.data import DataLoader

from roof_edges_dataset import RoofEdgesDataset

# Device
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# Hyperparameters
in_im_shape = (3, 224, 224)
num_classes = 2  # Edge / Non-edge
learning_rate = 0.001
batch_size = 4
n_epochs = 10

# Data - 60% Train - 20% Val - 20% Test
transformations = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
])

dataset = RoofEdgesDataset(im_path='data/images', ann_path='data/annotations', transform=transformations)

train_size = int(0.8 * len(dataset))
test_size = len(dataset) - train_size
train_dataset, test_dataset = torch.utils.data.random_split(dataset, [train_size, test_size])

train_size = int(0.8 * len(train_dataset))
val_size = len(train_dataset) - train_size
train_dataset, val_dataset = torch.utils.data.random_split(train_dataset, [train_size, val_size])

train_loader = DataLoader(dataset=train_dataset, batch_size=batch_size, shuffle=True)
val_loader = DataLoader(dataset=val_dataset, batch_size=batch_size, shuffle=True)
test_loader = DataLoader(dataset=test_dataset, batch_size=batch_size, shuffle=True)

# Model

# Loss and Optimizer

# Train
for epoch in range(n_epochs):
    for batch_idx, (image, annotation) in enumerate(train_loader):
        image = image.to(device=device)
        annotation = annotation.to(device=device)
        print(image.shape)
        print(annotation.shape)

    break

# Evaluate

I would be also very thankful if you let me know if something is done in a wrong or stupid way and how to do it better.
Thanks in advance!


title: pytorch seq2seq encoder forward method

body:
I'm following Pytorch seq2seq tutorial and below is how they define the encoder function.

class EncoderRNN(nn.Module):
    def __init__(self, input_size, hidden_size):
        super(EncoderRNN, self).__init__()
        self.hidden_size = hidden_size

        self.embedding = nn.Embedding(input_size, hidden_size)
        self.gru = nn.GRU(hidden_size, hidden_size)

    def forward(self, input, hidden):
        embedded = self.embedding(input).view(1, 1, -1)
        output = embedded
        output, hidden = self.gru(output, hidden)
        return output, hidden

    def initHidden(self):
        return torch.zeros(1, 1, self.hidden_size, device=device)

However, it seems like forward method is never really being called during the training.

Here is how the encoder forward method is being used in the tutorial:

    for ei in range(input_length):
        encoder_output, encoder_hidden = encoder(input_tensor[ei], encoder_hidden)
        encoder_outputs[ei] = encoder_output[0, 0]

isn't it supposed to be encoder.forward instead of just encoder?
Is there some automatic 'forward' mechanism in Pytorch that I am not aware of?


title: Error: cannot assign module before Module.__init__() call, Even after using super(DecoderRNN, self).__init__()

body:
I am receiving following error:

AttributeError: cannot assign module before Module.__init__() call

Although, this error has been resolved in

AttributeError: cannot assign module before Module.__init__() call

through the inclusion of super().__init__() method in the class, I continue receiving the same error even after using the proposed solution.

Following is the class that I have developed:

class DecoderRNN(nn.Module):
def __init__(self, embed_size, hidden_size, vocab_size, num_layers=1):
    super(DecoderRNN, self).__init__()
    self.embed_size = embed_size
    self.hidden_size = hidden_size
    self.vocab_size = vocab_size
    self.num_layers = num_layers
    self.linear = nn.Linear(hidden_size, vocab_size)
    self.embed = nn.Embedding(vocab_size, embed_size)
    self.lstm = nn.LSTM(embed_size, hidden_size, num_layers, batch_first = True)