title: Deep learning docker images (torch-encoding) that build successfully on Catalina fail to build on Big Sur

body:
The image that builds successfully using docker installed on macOS Catalina fails to build using docker installed on macOS Big Sur. I would like to make the image build on macOS BigSur but cannot change any of the files in the image (Dockerfile, conda.yaml etc.) Could somebody help with this ?
Symptoms: images that build and run successfully on Catalina refuse to build on Big Sur with various errors. Seems like the issue is related to python deep learning libraries that are wrapping c++ implementation. For example, I had issues with torch-encoding import on pytorch:1.6.0 base image. From the error attached seems line import encoding in preload_model.py triggers the error when I build the image using docker build . command.
Details:

base docker image: pytorch/pytorch:1.6.0-cuda10.1-cudnn7-devel
docker version 3.2.2
Dockerfile, conda.yaml, preload_model.py and error trace attached

Dockerfile:
FROM pytorch/pytorch:1.6.0-cuda10.1-cudnn7-devel

RUN apt-get -y update && apt-get install -y --no-install-recommends \
         curl \
         ca-certificates \
    && rm -rf /var/lib/apt/lists/* && apt-get clean

ENV PATH="/miniconda/bin:$PATH"
RUN curl https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh >> miniconda.sh \
    && bash ./miniconda.sh -b -p /miniconda; rm ./miniconda.sh

WORKDIR /opt/app

COPY conda.yaml ./

RUN apt-get -y update && apt-get install -y build-essential cmake \
    && conda env update --prefix /miniconda --file conda.yaml \
    && conda clean -tipsy \
    && rm -rf /var/lib/apt/lists/* && apt-get clean \
    && rm -rf ~/.cache/pip

RUN apt-get -y update && apt-get install -y --no-install-recommends \
         libgl1 libglib2.0-0 \
    && rm -rf /var/lib/apt/lists/* && apt-get clean

COPY app/preload_model.py ./

RUN python preload_model.py

preload_model.py
import encoding

encoding.models.get_model('DeepLab_ResNeSt200_ADE', pretrained=True)

conda.yaml
channels:
  - defaults
  - pytorch
dependencies:
  - python=3.8
  - pytorch=1.6.0
  - cudatoolkit=10.1
  - scipy=1.5.2
  - Flask=1.1.2
  - gunicorn=20.0.4
  - torchvision=0.7.0
  - Pillow=7.2.0
  - requests=2.24.0
  - numpy=1.19.1
  - ca-certificates
  - certifi
  - pip=20.2.2
  - pip:
      - brotlipy==0.7.0
      - chardet==3.0.4
      - click==7.1.2
      - future==0.18.2
      - itsdangerous==1.1.0
      - Jinja2==2.11.2
      - nose==1.3.7
      - opencv-python==4.4.0.44
      - portalocker==2.0.0
      - six==1.15.0
      - torch-encoding==1.2.1
      - tqdm==4.50.0
      - Werkzeug==1.0.1
      - imagehash==4.2.0
      - Flask-Caching==1.9.0

Error:
> [10/17] RUN python preload_model.py:
#14 1.680 No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
#14 69.42 Traceback (most recent call last):
#14 69.42   File "/miniconda/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1509, in _run_ninja_build
#14 69.42     subprocess.run(
#14 69.42   File "/miniconda/lib/python3.8/subprocess.py", line 516, in run
#14 69.42     raise CalledProcessError(retcode, process.args,
#14 69.42 subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.
#14 69.42
#14 69.42 During handling of the above exception, another exception occurred:
#14 69.42
#14 69.42 Traceback (most recent call last):
#14 69.42   File "preload_model.py", line 1, in <module>
#14 69.42     import encoding
#14 69.42   File "/miniconda/lib/python3.8/site-packages/encoding/__init__.py", line 13, in <module>
#14 69.42     from . import nn, functions, parallel, utils, models, datasets, transforms
#14 69.42   File "/miniconda/lib/python3.8/site-packages/encoding/nn/__init__.py", line 12, in <module>
#14 69.42     from .encoding import *
#14 69.42   File "/miniconda/lib/python3.8/site-packages/encoding/nn/encoding.py", line 18, in <module>
#14 69.42     from ..functions import scaled_l2, aggregate, pairwise_cosine
#14 69.42   File "/miniconda/lib/python3.8/site-packages/encoding/functions/__init__.py", line 2, in <module>
#14 69.42     from .encoding import *
#14 69.42   File "/miniconda/lib/python3.8/site-packages/encoding/functions/encoding.py", line 14, in <module>
#14 69.42     from .. import lib
#14 69.42   File "/miniconda/lib/python3.8/site-packages/encoding/lib/__init__.py", line 9, in <module>
#14 69.42     cpu = load('enclib_cpu', [
#14 69.42   File "/miniconda/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 963, in load
#14 69.42     return _jit_compile(
#14 69.42   File "/miniconda/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1170, in _jit_compile
#14 69.42     _write_ninja_file_and_build_library(
#14 69.42   File "/miniconda/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1276, in _write_ninja_file_and_build_library
#14 69.42     _run_ninja_build(
#14 69.42   File "/miniconda/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1529, in _run_ninja_build
#14 69.42     raise RuntimeError(message)
#14 69.42 RuntimeError: Error building extension 'enclib_cpu': [1/7] c++ -MMD -MF encoding_cpu.o.d -DTORCH_EXTENSION_NAME=enclib_cpu -DTORCH_API_INCLUDE_EXTENSION_H -isystem /miniconda/lib/python3.8/site-packages/torch/include -isystem /miniconda/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /miniconda/lib/python3.8/site-packages/torch/include/TH -isystem /miniconda/lib/python3.8/site-packages/torch/include/THC -isystem /miniconda/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -c /miniconda/lib/python3.8/site-packages/encoding/lib/cpu/encoding_cpu.cpp -o encoding_cpu.o
#14 69.42 [2/7] c++ -MMD -MF rectify_cpu.o.d -DTORCH_EXTENSION_NAME=enclib_cpu -DTORCH_API_INCLUDE_EXTENSION_H -isystem /miniconda/lib/python3.8/site-packages/torch/include -isystem /miniconda/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /miniconda/lib/python3.8/site-packages/torch/include/TH -isystem /miniconda/lib/python3.8/site-packages/torch/include/THC -isystem /miniconda/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -c /miniconda/lib/python3.8/site-packages/encoding/lib/cpu/rectify_cpu.cpp -o rectify_cpu.o
#14 69.42 FAILED: rectify_cpu.o
#14 69.42 c++ -MMD -MF rectify_cpu.o.d -DTORCH_EXTENSION_NAME=enclib_cpu -DTORCH_API_INCLUDE_EXTENSION_H -isystem /miniconda/lib/python3.8/site-packages/torch/include -isystem /miniconda/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /miniconda/lib/python3.8/site-packages/torch/include/TH -isystem /miniconda/lib/python3.8/site-packages/torch/include/THC -isystem /miniconda/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -c /miniconda/lib/python3.8/site-packages/encoding/lib/cpu/rectify_cpu.cpp -o rectify_cpu.o
#14 69.42 c++: internal compiler error: Killed (program cc1plus)
#14 69.42 Please submit a full bug report,
#14 69.42 with preprocessed source if appropriate.
#14 69.42 See <file:///usr/share/doc/gcc-7/README.Bugs> for instructions.
#14 69.42 [3/7] c++ -MMD -MF nms_cpu.o.d -DTORCH_EXTENSION_NAME=enclib_cpu -DTORCH_API_INCLUDE_EXTENSION_H -isystem /miniconda/lib/python3.8/site-packages/torch/include -isystem /miniconda/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /miniconda/lib/python3.8/site-packages/torch/include/TH -isystem /miniconda/lib/python3.8/site-packages/torch/include/THC -isystem /miniconda/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -c /miniconda/lib/python3.8/site-packages/encoding/lib/cpu/nms_cpu.cpp -o nms_cpu.o
#14 69.42 FAILED: nms_cpu.o
#14 69.42 c++ -MMD -MF nms_cpu.o.d -DTORCH_EXTENSION_NAME=enclib_cpu -DTORCH_API_INCLUDE_EXTENSION_H -isystem /miniconda/lib/python3.8/site-packages/torch/include -isystem /miniconda/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /miniconda/lib/python3.8/site-packages/torch/include/TH -isystem /miniconda/lib/python3.8/site-packages/torch/include/THC -isystem /miniconda/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -c /miniconda/lib/python3.8/site-packages/encoding/lib/cpu/nms_cpu.cpp -o nms_cpu.o
#14 69.42 c++: internal compiler error: Killed (program cc1plus)
#14 69.42 Please submit a full bug report,
#14 69.42 with preprocessed source if appropriate.
#14 69.42 See <file:///usr/share/doc/gcc-7/README.Bugs> for instructions.
#14 69.42 [4/7] c++ -MMD -MF syncbn_cpu.o.d -DTORCH_EXTENSION_NAME=enclib_cpu -DTORCH_API_INCLUDE_EXTENSION_H -isystem /miniconda/lib/python3.8/site-packages/torch/include -isystem /miniconda/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /miniconda/lib/python3.8/site-packages/torch/include/TH -isystem /miniconda/lib/python3.8/site-packages/torch/include/THC -isystem /miniconda/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -c /miniconda/lib/python3.8/site-packages/encoding/lib/cpu/syncbn_cpu.cpp -o syncbn_cpu.o
#14 69.42 FAILED: syncbn_cpu.o
#14 69.42 c++ -MMD -MF syncbn_cpu.o.d -DTORCH_EXTENSION_NAME=enclib_cpu -DTORCH_API_INCLUDE_EXTENSION_H -isystem /miniconda/lib/python3.8/site-packages/torch/include -isystem /miniconda/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /miniconda/lib/python3.8/site-packages/torch/include/TH -isystem /miniconda/lib/python3.8/site-packages/torch/include/THC -isystem /miniconda/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -c /miniconda/lib/python3.8/site-packages/encoding/lib/cpu/syncbn_cpu.cpp -o syncbn_cpu.o
#14 69.42 c++: internal compiler error: Killed (program cc1plus)
#14 69.42 Please submit a full bug report,
#14 69.42 with preprocessed source if appropriate.
#14 69.42 See <file:///usr/share/doc/gcc-7/README.Bugs> for instructions.
#14 69.42 [5/7] c++ -MMD -MF roi_align_cpu.o.d -DTORCH_EXTENSION_NAME=enclib_cpu -DTORCH_API_INCLUDE_EXTENSION_H -isystem /miniconda/lib/python3.8/site-packages/torch/include -isystem /miniconda/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /miniconda/lib/python3.8/site-packages/torch/include/TH -isystem /miniconda/lib/python3.8/site-packages/torch/include/THC -isystem /miniconda/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -c /miniconda/lib/python3.8/site-packages/encoding/lib/cpu/roi_align_cpu.cpp -o roi_align_cpu.o
#14 69.42 In file included from /miniconda/lib/python3.8/site-packages/torch/include/ATen/ATen.h:9:0,
#14 69.42                  from /miniconda/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/types.h:3,
#14 69.42                  from /miniconda/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader_options.h:4,
#14 69.42                  from /miniconda/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/base.h:3,
#14 69.42                  from /miniconda/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/stateful.h:3,
#14 69.42                  from /miniconda/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader.h:3,
#14 69.42                  from /miniconda/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data.h:3,
#14 69.42                  from /miniconda/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/all.h:4,
#14 69.42                  from /miniconda/lib/python3.8/site-packages/torch/include/torch/extension.h:4,
#14 69.42                  from /miniconda/lib/python3.8/site-packages/encoding/lib/cpu/

.....

roi_align_cpu.cpp:1:
#14 69.42 /miniconda/lib/python3.8/site-packages/torch/include/ATen/core/TensorBody.h:354:7: note: declared here
#14 69.42    T * data() const {
#14 69.42        ^~~~
#14 69.42 In file included from /miniconda/lib/python3.8/site-packages/torch/include/ATen/ATen.h:9:0,
#14 69.42                  from /miniconda/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/types.h:3,
#14 69.42                  from /miniconda/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader_options.h:4,
#14 69.42                  from /miniconda/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/base.h:3,
#14 69.42                  from /miniconda/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/stateful.h:3,
#14 69.42                  from /miniconda/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader.h:3,
#14 69.42                  from /miniconda/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data.h:3,
#14 69.42                  from /miniconda/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/all.h:4,
#14 69.42                  from /miniconda/lib/python3.8/site-packages/torch/include/torch/extension.h:4,
#14 69.42                  from /miniconda/lib/python3.8/site-packages/encoding/lib/cpu/roi_align_cpu.cpp:1:
#14 69.42 /miniconda/lib/python3.8/site-packages/encoding/lib/cpu/roi_align_cpu.cpp:472:34: warning: ‘T* at::Tensor::data() const [with T = float]’ is deprecated: Tensor.data<T>() is deprecated. Please use Tensor.data_ptr<T>() instead. [-Wdeprecated-declarations]
#14 69.42        bottom_rois.data<scalar_t>(),
#14 69.42                                   ^
#14 69.42 In file included from /miniconda/lib/python3.8/site-packages/torch/include/ATen/Tensor.h:3:0,
#14 69.42                  from /miniconda/lib/python3.8/site-packages/torch/include/ATen/Context.h:4,
#14 69.42                  from /miniconda/lib/python3.8/site-packages/torch/include/ATen/ATen.h:5,
#14 69.42                  from /miniconda/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/types.h:3,
#14 69.42                  from /miniconda/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader_options.h:4,
#14 69.42                  from /miniconda/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/base.h:3,
#14 69.42                  from /miniconda/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/stateful.h:3,
#14 69.42                  from /miniconda/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader.h:3,
#14 69.42                  from /miniconda/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/data.h:3,
#14 69.42                  from /miniconda/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/all.h:4,
#14 69.42                  from /miniconda/lib/python3.8/site-packages/torch/include/torch/extension.h:4,
#14 69.42                  from /miniconda/lib/python3.8/site-packages/encoding/lib/cpu/roi_align_cpu.cpp:1:
#14 69.42 /miniconda/lib/python3.8/site-packages/torch/include/ATen/core/TensorBody.h:354:7: note: declared here
#14 69.42    T * data() const {
#14 69.42        ^~~~
#14 69.42 [6/7] c++ -MMD -MF operator.o.d -DTORCH_EXTENSION_NAME=enclib_cpu -DTORCH_API_INCLUDE_EXTENSION_H -isystem /miniconda/lib/python3.8/site-packages/torch/include -isystem /miniconda/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /miniconda/lib/python3.8/site-packages/torch/include/TH -isystem /miniconda/lib/python3.8/site-packages/torch/include/THC -isystem /miniconda/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -c /miniconda/lib/python3.8/site-packages/encoding/lib/cpu/operator.cpp -o operator.o
#14 69.42 ninja: build stopped: subcommand failed.
#14 69.42
------
executor failed running [/bin/sh -c python preload_model.py]: exit code: 1


title: Converting Keras to Pytorch

body:
I am trying to convert the following model to pytorch:
def get_model():

    model = keras.models.Sequential()

    model.add(Conv2D(64, kernel_size=(3,3), activation='relu', padding='same', input_shape=(9,9,1)))
    model.add(BatchNormalization())
    model.add(Conv2D(64, kernel_size=(3,3), activation='relu', padding='same'))
    model.add(BatchNormalization())
    model.add(Conv2D(128, kernel_size=(1,1), activation='relu', padding='same'))

    model.add(Flatten())
    model.add(Dense(81*9))
    model.add(Reshape((-1, 9)))
    model.add(Activation('softmax'))

    return model

model = get_model()

adam = keras.optimizers.Adam(lr=.001)
model.compile(loss='sparse_categorical_crossentropy', optimizer=adam)

model.fit(x_train, y_train, batch_size=32, epochs=2)

from https://github.com/shivaverma/Sudoku-Solver/blob/master/model.py
class Conv2dSame(torch.nn.Module):
def __init__(self, in_channels, out_channels, kernel_size, padding, bias=True, padding_layer=torch.nn.ReflectionPad2d):
    """It only support square kernels and stride=1, dilation=1, groups=1."""
    super(Conv2dSame, self).__init__()
    self.net = nn.Sequential(
        nn.Conv2d(in_channels, out_channels, kernel_size, bias=bias, padding=padding),
        nn.ReLU(True),
    )
def forward(self, x):
    return self.net(x)

class CNN(torch.nn.Module):
def __init__(self, hidden_channels, batch_size):

    super(CNN, self).__init__()
    self.hidden_channels = hidden_channels
    self.batch_size = batch_size

    self.embed = nn.Sequential(

        Conv2dSame(1, hidden_channels, 3, 1),
        nn.BatchNorm2d(hidden_channels),
        Conv2dSame(hidden_channels, hidden_channels, 3, 1),
        nn.BatchNorm2d(hidden_channels),
        Conv2dSame(hidden_channels, 2*hidden_channels, 1, 0),

    )

    self.out = nn.Sequential(

        nn.Linear(2*hidden_channels*81, 81*9),

    )

    self.softmax = nn.Softmax(dim=1)

def forward(self, x):
    embed = self.embed(x)
    embed = embed.view(-1, 2*self.hidden_channels*81)
    out = self.out(embed).view(-1,9,81)
    return self.softmax(out)

But it doesn't seem to be working with:
model = CNN(hidden_channels=64,batch_size=32) # GCN(hidden_channels=64)
optimizer = torch.optim.Adam(model.parameters(), lr=0.001) #0.01
criterion = nn.CrossEntropyLoss()

def train():

    model.train()

    for batch_i, data in tqdm(enumerate(train_loader)):  # Iterate in batches over the training dataset.

      optimizer.zero_grad()  # Clear gradients.

      unsolved = data[0]
      solved = data[1]

      out = model(unsolved) # data.x, data.edge_index, data.batch)  # Perform a single forward pass.
      loss = criterion(out, solved)  # Compute the loss.
      if batch_i % 100 == 0:
        print(loss)

      loss.backward()  # Derive gradients.
      optimizer.step()  # Update parameters based on gradients.

batch_size = 32 #64
num_epochs = 2

train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True, num_workers=0)

for _ in range(num_epochs):
  train()

My loss stays very stagnant and my accuracy is bad. I know the keras code works fine. Please let me know if there is any part of the keras code I am misunderstanding. Or if my pytorch implementation is off. I am very new to keras (and pytorch). Thank you!


title: Flatten 3D tensor

body: I have a tensor of the shape T x B x N (training data for a RNN, T is max seq length, B is number of batches, and N number of features) and I'd like to flatten all the features across timesteps, such that I get a tensor of the shape B x TN. Haven't been able to figure out how to do this..


title: Unknown Python syntax in PyTorch: a instance can directly receive a parameter

body:
I have a question about Python syntax when I'm learning PyTorch. The following codes are an example from the PyTorch document.

m = nn.Linear(20, 30)
input = autograd.Variable(torch.randn(128, 20))
output = m(input)
print(output.size())

This first line is to create an instance m, but why this instance m can directly receive a parameter like the line 3? I think it should use method to deal with parameter like m.method(input).


title: Need very different learning rate for manual updates vs. using model

body:
I am currently just trying to write some pedagogical material, in which I borrow from some common examples that have been reworked numerous times on the web.

I have a simple bit of code where I manually create tensors for layers, and update them within a loop.  E.g.:

w1 = torch.randn(D_in, H, dtype=torch.float, requires_grad=True)
w2 = torch.randn(H, D_out, dtype=torch.float, requires_grad=True)

learning_rate = 1e-6
for t in range(501):
    y_pred = x.mm(w1).clamp(min=0).mm(w2)
    loss = (y_pred - y).pow(2).sum()
    loss.backward()
    w1 -= learning_rate * w1.grad
    w2 -= learning_rate * w2.grad
    w1.grad.zero_()
    w2.grad.zero_()

This works great.  Then I construct similar code using actual modules:

model = torch.nn.Sequential(
          torch.nn.Linear(D_in, H),
          torch.nn.ReLU(),
          torch.nn.Linear(H, D_out),
        )
loss_fn = torch.nn.MSELoss(reduction='sum')
learning_rate = 1e-4
for t in range(501):
    y_pred = model(x)
    loss = loss_fn(y_pred, y)
    model.zero_grad()
    loss.backward()
    for param in model.parameters():
        param.data -= learning_rate * param.grad

This also works great.

BUT there is a difference here.  If I use a 1e-4 LR in the manual case, the loss explodes, become large, then inf, then nan.  So that's no good.  If I use a 1e-6 LR in the model case, the loss decreases far too slowly.

Basically I'm just trying to understand why learning rate means something very different in these two snippets which are otherwise equivalent.


title: Does the input of model require gradient?

body: In pytorch, when do we need to set the require_grad of input tensor to be True?


title: updating the input layer of a neural networks in python

body:
I have a network with two input layers, the first one is an image data ( e.g. MNIST) and the second one is an auxiliary vector,

A_vec = np.random.normal(mu, sigma, (1, n_d))
A_vec1=np.tile(A_vec, (size_1, 1))

image_inputs = Input(shape=(original_dim,),name='Image_Inputs')
A_inputs = Input(shape=(n_d,),name='A_Inputs')
inputs = keras.layers.concatenate([image_inputs, A_inputs],name='Inputs')

.......intermediate layers......

Image_Outputs= Dense(
original_dim,
activation='sigmoid',
name='Image_Outputs')(intermediate layers)

A_Outputs= Dense(
    n_d,
    activation='sigmoid',
    name='A_Outputs')(intermediate layers)

outputs = keras.layers.concatenate([Image_Outputs, A_Outputs],name='Outputs')

Here, the A_vecin the initialization for A_inputs. But, I want to update it in each epoch by using the A_Outputs. I mean after the first iteration, I need to use the estimated values for A_Outputs as the updated values of theA_inputs in the second epochs and same prcedure for next epochs. Is there any way to do that in Tensorflow, Keras or pytorch?


title: Something wrong happens when shuffle a tensor

body:
I am trying to shuffle a tensor, but I find there is something wrong when tryting to using random.shuffle(),here is my code:

import numpy as np
import torch
import random

if __name__ == '__main__':

    label_0, label_1 = np.zeros([1, 1000])[0], np.ones([1, 1000])[0]
    test_label = np.hstack((label_0, label_1))

    num_0, num_1 = 0, 0

    for i in range(len(test_label)):
        if test_label[i] == 0:
            num_0 += 1
        elif test_label[i] == 1:
            num_1 += 1

    print('Num1:  ', num_1)
    print('Num0:  ', num_0)

    random.shuffle(test_label)

    num_0, num_1 = 0, 0
    for i in range(len(test_label)):
        if test_label[i] == 0:
            num_0 += 1
        elif test_label[i] == 1:
            num_1 += 1

    print('After Shuffle Num1:  ', num_1)
    print('After Shuffle Num0:  ', num_0)

If test_label is not a tensor, it works perfectly:

Num1:   1000
Num0:   1000
After Shuffle Num1:   1000
After Shuffle Num0:   1000

However when I try to transform the test_label from ndarray to tensor, something wrong would happend:

import numpy as np
import torch
import random

if __name__ == '__main__':

    label_0, label_1 = np.zeros([1, 1000])[0], np.ones([1, 1000])[0]
    test_label = np.hstack((label_0, label_1))

    # now transform to tensor
    test_label = torch.tensor(test_label)

    num_0, num_1 = 0, 0

    for i in range(len(test_label)):
        if test_label[i] == 0:
            num_0 += 1
        elif test_label[i] == 1:
            num_1 += 1

    print('Num1:  ', num_1)
    print('Num0:  ', num_0)

    random.shuffle(test_label)

    num_0, num_1 = 0, 0
    for i in range(len(test_label)):
        if test_label[i] == 0:
            num_0 += 1
        elif test_label[i] == 1:
            num_1 += 1

    print('After Shuffle Num1:  ', num_1)
    print('After Shuffle Num0:  ', num_0)

And the result seems to be kind wrong:

Num1:   1000
Num0:   1000
After Shuffle Num1:   306
After Shuffle Num0:   1694

Anybody can tell me why will make such problem?


title: how to batch dialog dataset in pytorch?

body:
I want to do a task oriented dialog chatbot which is used to book restaurant.Because every dialog has different sequences(eg. some has 5 turns of dialogs which is 10 sentences while another may has 6 turns of dialogs which is 12 sentences totally),I don't know how to batch dataset.

Could you give me some tutorial or github example?


title: Why is the variable `input` often marked special in Python syntax markers?

body:
In many syntax highlighters (e.g. jupyter notebooks) the variable input is highlighted in python. Is there a reason for this? I've avoided using input as a variablename because I thought it did something special internally.

I'm asking because pytorch often uses "input" as a variable in their tutorials.


title: Pytorch Unsigned int 32, 64 tensor

body: It seems that Pytorch tensor does not support unsigned int32 and int64 currently. While Numpy and TensorFlow both support them. I am wondering whether Pytorch has any plan about it? If not, any short cut i could do to make extension from Numpy? Or write C++ code is necessary?


title: autograd differentiation example in PyTorch - should be 9/8?

body:
In the example for the Torch tutorial for Python, they use the following graph:

x = [[1, 1], [1, 1]]
y = x + 2
z = 3y^2
o = mean( z )  # 1/4 * x.sum()

Thus, the forward pass gets us this:

x_i = 1, y_i = 3, z_i = 27, o = 27

In code this looks like:

import torch

# define graph
x = torch.ones(2, 2, requires_grad=True)
y = x + 2
z = y * y * 3
out = z.mean()

# if we don't do this, torch will only retain gradients for leaf nodes, ie: x
y.retain_grad()
z.retain_grad()

# does a forward pass
print(z, out)

however, I get confused at the gradients computed:

# now let's run our backward prop & get gradients
out.backward()
print(f'do/dz = {z.grad[0,0]}')

which outputs:

do/dx = 4.5

By chain rule, do/dx = do/dz * dz/dy * dy/dx, where:

dy/dx = 1
dz/dy = 9/2 given x_i=1
do/dz = 1/4 given x_i=1

which means:

do/dx = 1/4 * 9/2 * 1 = 9/8

However this doesn't match the gradients returned by Torch (9/2 = 4.5). Perhaps I have a math error (something with the do/dz = 1/4 term?), or I don't understand autograd in Torch.

Any pointers?


title: Create a list where the even indexes have a path to a positive category and odd indexes have a path to the negative category

body:
I'm basically sorting my CNN images into a list with even and odd indexing. Even index will have positive images and odd index will have negative images. Here's  my code so far:

from PIL import Image
import matplotlib.pyplot as plt
import os
import glob
import torch
from torch.utils.data import Dataset

def show_data(data_sample, shape = (28, 28)):
    plt.imshow(data_sample[0].numpy().reshape(shape), cmap='gray')
    plt.title('y = ' + data_sample[1])

directory="/resources/data"
negative='Negative'
negative_file_path=os.path.join(directory,negative)
negative_files=[os.path.join(negative_file_path,file) for file in  os.listdir(negative_file_path) if file.endswith(".jpg")]
negative_files.sort()
negative_files[0:3]

positive="Positive"
positive_file_path=os.path.join(directory,positive)
positive_files=[os.path.join(positive_file_path,file) for file in  os.listdir(positive_file_path) if file.endswith(".jpg")]
positive_files.sort()
positive_files[0:3]

n = len(negative_files)
p = len(positive_files)
number_of_samples = n + p
print(number_of_samples)

Y=torch.zeros([number_of_samples])

Y=Y.type(torch.LongTensor)
Y.type()

Y[::2]=1
Y[1::2]=0


title: Tensor Entry Selection Logic Divergence in PyTorch & Numpy

body:
Description

I'm setting up a torch.Tensor for masking purpose. When attempting to select entries by indices, it turns out that behaviors between using numpy.ndarray and torch.Tensor to hold index data are different. I would like to have access to the design in both frameworks and related documents that explain the difference.

Steps to replicate

Environment

Pytorch 1.3 in container from official release: pytorch/pytorch:1.3-cuda10.1-cudnn7-devel

Example

Say I need to set up mask as torch.Tensor object with shape [3,3,3] and set values at entries (0,0,1) & (1,2,0) to 1. The code below explains the difference.

mask = torch.zeros([3,3,3])
indices = torch.tensor([[0, 1],
                        [0, 2],
                        [1, 0]])

mask[indices.numpy()] = 1 # Works
# mask[indices] = 1 # Incorrect result

I noticed that when using mask[indices.numpy()] a new torch.Tensor of shape [2], while mask[indices] returns a new torch.Tensor of shape [3, 2, 3, 3], which suggests difference in tensor slicing logic.


title: How do I use Pytorch's View function correctly?

body:
I have a tensor of size x={4,2,C,H,W}. I need it reshape to y={8,C,H,W}, but I want to make sure the images are stored in the right order so say an image at x[1,0,:,:,:] has to be equal to y[2,C,H,W]. I know I can use the view function for this but I am not sure how to use it correctly.

Currently I am doing it as such

feat_imgs_all = feat_imgs_all.view(
    rgb.shape[0], rgb.shape[1], feat_imgs_all.shape[1],
    feat_imgs_all.shape[2], feat_imgs_all.shape[3])

This seems really hacky, is there a way I can just feed the first two shapes, and pytorch figures out the rest?


title: Why does loss skyrocket during convolutional neural net training?

body: I am training a simple CNN in Pytorch for segmentation on a very small dataset (just a few images as this is just for proof of concept purposes). For some reason, loss skyrockets to as high as 6 and IoU drops to 0 (intersection over union accuracy metric) randomly during training before going back up. I was wondering why this could be happening?


title: py-test coverage covers def, but not method

body:
I am successfully running pytest and getting coverage in my forked PyTorch repository. However, in some of the files covered in the report, I am often getting coverage for the definition, but not the method.

I run this command:

pytest test_indexing.py --cov-report html --cov=torch

And in one of the files covered in the report I have:

There is a stackoverflow post with a very similar issue, but the mentioned paste.fixture library does not work for me.

How do I get the method to run? Any guidance / tips will be greatly appreciated.


title: Question about dimensions when processing lists with a multi layer perceptron

body: I'm quite new to PyTorch and I'm trying to build a net that is composed only of linear layers that will get a list of objects as input and output some score (which is a scalar) for each object. I'm wondering if my input tensor's dimensions should be (batch_size, list_size, object_size) or should I flatten each list and get (batch_size, list_size*object_size)? According to my understanding, in the first option I will have an output dimension of (batch_size, list_size, 1) and in the second (batch_size, list_size), does it matter? I read the documentation but it still wasn't very clear to me.


title: Matrix multiplication dimensions confusing

body:
I am following this tutorial https://pytorch.org/tutorials/beginner/nlp/deep_learning_tutorial.html#example-logistic-regression-bag-of-words-classifier

nn.Linear(vocab_size, num_labels)
means that the matrix shape is num_labels x vocab_size

bow_vector dimensions is 1 x vocab_size and input expected for nn.linear is batch_size x features

Now, we are multiplying num_labels x vocab_size matrix by 1 x vocab_size. Thus, the dimensions don't match for matrix multiplication. What am I missing here? :thinking:

https://discuss.pytorch.org/t/matrix-multiplication-dimentions-confusing/79376?u=abhigenie92


title: How to select 5% of total values from a tensor randomly?

body:
a = torch.rand(2,5,10)
I want to select at most 5% of values from tensor a randomly and then multiply those values with -1? How to do that? kindly, give a generic solution as the shape of the tensor is not fixed


title: Converting two Numpy data sets into a particularr PyTorch data set

body:
I want to play around with a neural network that recognizes handwritten numbers. I found some of these on the web which use PyTorch, however they seem to download the data from the MNIST website in a particular format. My data is, however, available as follows:

with np.load('prediction-challenge-01-data.npz') as fh:
     data_x = fh['data_x']
     data_y = fh['data_y']

Where data_x is the training data and data_y are the labels of the pictures. I want these data sets to be in the same format as trainloader as shown below:

trainset = datasets.MNIST('/data/mnist', download=True, train=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=64, shuffle=True)

Where trainloader already has the training set data_x and labels data_y together in one set.

Is there any way to do this?

Edit: Shapes of data_x and data_y:

In [1]:  data_x.shape
Out[2]: (20000, 1, 28, 28)

In [5]:  data_y.shape
Out[7]: (20000,)


title: Combine tensor matrix and sparse matrix of same dataset for splitting the data

body:
I have an array and sparse matrix in tensor type and both are to be feed into the Neural network classifier. Since the data is too large, I need to split it batch-wise.

How should I club my sparse matrix and array together and then split train data batch-wise?

tensor(indices=tensor([[     0,      0,      0,  ..., 426017, 426017, 426017],
                       [   223,    310,   1045,  ...,     39,     14,    542]]),
       values=tensor([0.0730, 0.0722, 0.0911,  ..., 0.3244, 0.0883, 0.1659]),
       size=(426018, 10000), nnz=19173512, layout=torch.sparse_coo)

torch.Size([426018, 16])
tensor([[0.0154, 0.3296, 0.7500,  ..., 0.7628, 0.1643, 0.0092],
        [0.1041, 0.6322, 1.0000,  ..., 0.7612, 0.1629, 0.1283],
        [0.4987, 0.3387, 0.7500,  ..., 0.8692, 0.1879, 0.0150],
        ...,
        [0.1158, 0.6453, 1.0000,  ..., 0.6511, 0.2130, 0.0590],
        [0.0751, 0.2653, 0.7500,  ..., 0.7232, 0.1644, 0.0208],
        [0.4051, 0.3801, 1.0000,  ..., 0.8479, 0.2427, 0.1179]],
       dtype=torch.float64)

Above are two parts of same dataset in which one are numerical features(16) and second are tfidf features in sparse matrix.So in order to split it batch wise I need to club them together.
I know one function data.utils.data.DataLoader but dont't know how to combine my dataset to feed into it.


title: Accessing Azure DSVMs GPU for running Flask app developed in local machine

body:
I'm very new to ML and cloud services and am wondering how I could run my flask app that was build using Pytorch and consists of restful APIs, in the pycharm IDE in my local machine(Mac) to use GPU from the DSVM-Ubuntu 18.04 which consists of Standard NC6 (6 vcpus, 56 GiB memory).

For now I'm able to connect to the VM via ssh and don't know how to proceed with running the flask app to use this GPU, and not really sure whether this can be done using Docker either. Thererfore would like to know options that are possible to perform this task. Any help is appriciated and suggestions are welcome.


title: Runtime error while running PyTorch model on local machine

body:
I'm running this notebook locally

https://github.com/udacity/deep-learning-v2-pytorch/blob/master/sentiment-rnn/Sentiment_RNN_Solution.ipynb
everything was working just until I started training the model

# training params

epochs = 4 # 3-4 is approx where I noticed the validation loss stop decreasing

counter = 0
print_every = 100
clip=5 # gradient clipping

# move model to GPU, if available
if(train_on_gpu):
    net.cuda()

net.train()
# train for some number of epochs
for e in range(epochs):
    # initialize hidden state
    h = net.init_hidden(batch_size)

    # batch loop
    for inputs, labels in train_loader:
        counter += 1

        if(train_on_gpu):
            inputs, labels = inputs.cuda(), labels.cuda()

        # Creating new variables for the hidden state, otherwise
        # we'd backprop through the entire training history
        h = tuple([each.data for each in h])

        # zero accumulated gradients
        net.zero_grad()

        # get the output from the model
        output, h = net(inputs, h)

        # calculate the loss and perform backprop
        loss = criterion(output.squeeze(), labels.float())
        loss.backward()
        # `clip_grad_norm` helps prevent the exploding gradient problem in RNNs / LSTMs.
        nn.utils.clip_grad_norm_(net.parameters(), clip)
        optimizer.step()

        # loss stats
        if counter % print_every == 0:
            # Get validation loss
            val_h = net.init_hidden(batch_size)
            val_losses = []
            net.eval()
            for inputs, labels in valid_loader:

                # Creating new variables for the hidden state, otherwise
                # we'd backprop through the entire training history
                val_h = tuple([each.data for each in val_h])

                if(train_on_gpu):
                    inputs, labels = inputs.cuda(), labels.cuda()

                output, val_h = net(inputs, val_h)
                val_loss = criterion(output.squeeze(), labels.float())

                val_losses.append(val_loss.item())

            net.train()
            print("Epoch: {}/{}...".format(e+1, epochs),
                  "Step: {}...".format(counter),
                  "Loss: {:.6f}...".format(loss.item()),
                  "Val Loss: {:.6f}".format(np.mean(val_losses)))

the error that occurred:

RuntimeError                              Traceback (most recent call last)
<ipython-input-31-9f7dea11cb7b> in <module>
     32
     33         # get the output from the model
---> 34         output, h = net(inputs, h)
     35
     36         # calculate the loss and perform backprop

c:\users\asus\.conda\envs\pytorch\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--> 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

<ipython-input-16-b99cefc1dc61> in forward(self, x, hidden)
     36
     37         # embeddings and lstm_out
---> 38         embeds = self.embedding(x)
     39         lstm_out, hidden = self.lstm(embeds, hidden)
     40

c:\users\asus\.conda\envs\pytorch\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--> 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

c:\users\asus\.conda\envs\pytorch\lib\site-packages\torch\nn\modules\sparse.py in forward(self, input)
    110
    111     def forward(self, input):
--> 112         return F.embedding(
    113             input, self.weight, self.padding_idx, self.max_norm,
    114             self.norm_type, self.scale_grad_by_freq, self.sparse)

c:\users\asus\.conda\envs\pytorch\lib\site-packages\torch\nn\functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
   1722         # remove once script supports set_grad_enabled
   1723         _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-> 1724     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
   1725
   1726

RuntimeError: Expected tensor for argument #1 'indices' to have scalar type Long; but got torch.cuda.IntTensor instead (while checking arguments for embedding)

I don't understand why it's happening. I tried to find a solution online. which said I need transfer my model and data to GPU. I did but the problem still persists.


title: Why use multiple ReLU objects in Neural Net class definition?

body:
Recently I observed that a lot of times while defining the neural nets we define separate ReLU objects for each layer. Why can't we use the same ReLU object wherever it is needed.
For example instead of writing like this-
def __init__(self):
    self.fc1     = nn.Linear(784, 500)
    self.ReLU_1  = nn.ReLU()
    self.fc2     = nn.Linear(500, 300)
    self.ReLU_2  = nn.ReLU()

def forward(x):
    x = self.fc1(x)
    x = self.ReLU_1(x)
    x = self.fc2(x)
    x = self.ReLU_2(x)

why can't we use
def __init__(self):
    self.fc1    = nn.Linear(784, 500)
    self.ReLU   = nn.ReLU()
    self.fc2    = nn.Linear(500, 300)

def forward(x):
    x = self.fc1(x)
    x = self.ReLU(x)
    x = self.fc2(x)
    x = self.ReLU(x)

Is this something specific to PyTorch?


title: How to aggregate gradients in PyTorch?

body:
I have been using a combination of mpi4py and pytorch to try to combine the gradients of multiple instances of the same model. When each model is done its epoch, it executes this code to try and combine the gradients on one model and then send its updated model back to the other processes:
on client node:

    grads = []
    for param in model.parameters():
        grads.append(param.grad)
    comm.gather(grads, root = 0)
    model = comm.bcast(model, root = 0)

on server node:

    optimizer.zero_grad()
    for param in model.parameters():
        grads.append(param.grad)
    newGrads = comm.gather(grads, root = 0)
    for i in range(1, len(newGrads))
        newGrads += newGrads[i]
    for param, grad, in zip(model.parameters(), result):
        param.grad = grad/comm.Get_size()
    optimizer.step()
    comm.bcast(model, root=0)

When the models run, however, the accuracy stays the same throughout the epochs. I was wondering if there's another way to do what I am trying to do or if the logic of my code is wrong and what I should do to fix it. Thank you all for your time! I really appreciate it!


title: Expected input batch_size (32) to match target batch_size (19840) BERT Classifier

body:
I ran into this error with code:
model = BertForSequenceClassification.from_pretrained("pretrained/", num_labels=ohe_count)
model.to(device)

from IPython.display import clear_output

train_loss_set = []
train_loss = 0

model.train()

for step, batch in enumerate(train_dataloader):
    # добавляем батч для вычисления на GPU
    batch = tuple(t.to(device) for t in batch)
    # Распаковываем данные из dataloader
    b_input_ids, b_input_mask, b_labels = batch

    b_input_ids = b_input_ids.type(torch.LongTensor)
    b_input_mask = b_input_mask.type(torch.LongTensor)
    b_labels = b_labels.type(torch.LongTensor)

    b_input_ids = b_input_ids.to(device)
    b_input_mask = b_input_mask.to(device)
    b_labels = b_labels.to(device)

    optimizer.zero_grad()

    # Forward pass
    loss = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask, labels=b_labels)

    train_loss_set.append(loss[0].item())

    # Backward pass
    loss[0].backward()

    optimizer.step()

    train_loss += loss[0].item()

    clear_output(True)
    plt.plot(train_loss_set)
    plt.title("Training loss")
    plt.xlabel("Batch")
    plt.ylabel("Loss")
    plt.show()

b_input_ids.shape = torch.Size([32, 100])

b_labels.shape = torch.Size([32, 620])

Shapes seem fine, but I am getting the error Expected input batch_size (32) to match target batch_size (19840)


title: Modify PyTorch model for inference - then resume training

body:
I’d like to alternate inference and training in a model in PyTorch, but I need to modify it during inference, and I have some questions about that.
Can I unload the model from the gpu by calling model.to('cpu'), make a modified copy (and run it on the gpu), and then move the original back to gpu by calling model.to('gpu')? In other words, is moving the model gpu->cpu->gpu a lossless operation? What happens to the parameters that were passed to the optimizer? I don’t want to lose the optimizer state
What is the best way to make a copy of an in-memory model? I can save it and then reload a copy, but not sure if that is necessary just to copy.
If I want to run inference in half precision (more than 2x faster in this case), can I change the model to half and then change it back? Is that lossless? (Does the model keep a full-precision copy of everything, or does it replace weights with half-precision in place?)
The model is a ResNet50 lookalike.  Not enough GPU memory for two models :)


title: How to load weights of Weight Standardization model?

body:
using this link : https://github.com/joe-siyuan-qiao/WeightStandardization from  pytorch-classification  folder i was using resnet50 for a classification task,
here  is what i did - copied code from here : https://github.com/joe-siyuan-qiao/pytorch-classification/blob/e6355f829e85ac05a71b8889f4fff77b9ab95d0b/models/imagenet/resnet.py
and did - model = l_resnet50()
now we have resnet50 model that uses gn and ws together but then i was trying to load resnet50 weight file provided by author of that repo
but i get this  error :
RuntimeError: Error(s) in loading state_dict for ResNet:
    Missing key(s) in state_dict: "layer3.6.conv1.weight", "layer3.6.bn1.weight", "layer3.6.bn1.bias", "layer3.6.conv2.weight", "layer3.6.bn2.weight", "layer3.6.bn2.bias", "layer3.6.conv3.weight", "layer3.6.bn3.weight", "layer3.6.bn3.bias", "layer3.7.conv1.weight", "layer3.7.bn1.weight", "layer3.7.bn1.bias", "layer3.7.conv2.weight", "layer3.7.bn2.weight", "layer3.7.bn2.bias", "layer3.7.conv3.weight", "layer3.7.bn3.weight", "layer3.7.bn3.bias", "layer3.8.conv1.weight", "layer3.8.bn1.weight", "layer3.8.bn1.bias", "layer3.8.conv2.weight", "layer3.8.bn2.weight", "layer3.8.bn2.bias", "layer3.8.conv3.weight", "layer3.8.bn3.weight", "layer3.8.bn3.bias", "layer3.9.conv1.weight", "layer3.9.bn1.weight", "layer3.9.bn1.bias", "layer3.9.conv2.weight", "layer3.9.bn2.weight", "layer3.9.bn2.bias", "layer3.9.conv3.weight", "layer3.9.bn3.weight", "layer3.9.bn3.bias", "layer3.10.conv1.weight", "layer3.10.bn1.weight", "layer3.10.bn1.bias", "layer3.10.conv2.weight", "layer3.10.bn2.weight", "layer3.10.bn2.bias", "layer3.10.conv3.weight", "layer3.10.bn3.weight", "layer3.10.bn3.bias", "layer3.11.conv1.weight", "layer3.11.bn1.weight", "layer3.11.bn1.bias", "layer3.11.conv2.weight", "layer3.11.bn2.weight", "layer3.11.bn2.bias", "layer3.11.conv3.weight", "layer3.11.bn3.weight", "layer3.11.bn3.bias", "layer3.12.conv1.weight", "layer3.12.bn1.weight", "layer3.12.bn1.bias", "layer3.12.conv2.weight", "layer3.12.bn2.weight", "layer3.12.bn2.bias", "layer3.12.conv3.weight", "layer3.12.bn3.weight", "layer3.12.bn3.bias", "layer3.13.conv1.weight", "layer3.13.bn1.weight", "layer3.13.bn1.bias", "layer3.13.conv2.weight", "layer3.13.bn2.weight", "layer3.13.bn2.bias", "layer3.13.conv3.weight", "layer3.13.bn3.weight", "layer3.13.bn3.bias", "layer3.14.conv1.weight", "layer3.14.bn1.weight", "layer3.14.bn1.bias", "layer3.14.conv2.weight", "layer3.14.bn2.weight", "layer3.14.bn2.bias", "layer3.14.conv3.weight", "layer3.14.bn3.weight", "layer3.14.bn3.bias", "layer3.15.conv1.weight", "layer3.15.bn1.weight", "layer3.15.bn1.bias", "layer3.15.conv2.weight", "layer3.15.bn2.weight", "layer3.15.bn2.bias", "layer3.15.conv3.weight", "layer3.15.bn3.weight", "layer3.15.bn3.bias", "layer3.16.conv1.weight", "layer3.16.bn1.weight", "layer3.16.bn1.bias", "layer3.16.conv2.weight", "layer3.16.bn2.weight", "layer3.16.bn2.bias", "layer3.16.conv3.weight", "layer3.16.bn3.weight", "layer3.16.bn3.bias", "layer3.17.conv1.weight", "layer3.17.bn1.weight", "layer3.17.bn1.bias", "layer3.17.conv2.weight", "layer3.17.bn2.weight", "layer3.17.bn2.bias", "layer3.17.conv3.weight", "layer3.17.bn3.weight", "layer3.17.bn3.bias", "layer3.18.conv1.weight", "layer3.18.bn1.weight", "layer3.18.bn1.bias", "layer3.18.conv2.weight", "layer3.18.bn2.weight", "layer3.18.bn2.bias", "layer3.18.conv3.weight", "layer3.18.bn3.weight", "layer3.18.bn3.bias", "layer3.19.conv1.weight", "layer3.19.bn1.weight", "layer3.19.bn1.bias", "layer3.19.conv2.weight", "layer3.19.bn2.weight", "layer3.19.bn2.bias", "layer3.19.conv3.weight", "layer3.19.bn3.weight", "layer3.19.bn3.bias", "layer3.20.conv1.weight", "layer3.20.bn1.weight", "layer3.20.bn1.bias", "layer3.20.conv2.weight", "layer3.20.bn2.weight", "layer3.20.bn2.bias", "layer3.20.conv3.weight", "layer3.20.bn3.weight", "layer3.20.bn3.bias", "layer3.21.conv1.weight", "layer3.21.bn1.weight", "layer3.21.bn1.bias", "layer3.21.conv2.weight", "layer3.21.bn2.weight", "layer3.21.bn2.bias", "layer3.21.conv3.weight", "layer3.21.bn3.weight", "layer3.21.bn3.bias", "layer3.22.conv1.weight", "layer3.22.bn1.weight", "layer3.22.bn1.bias", "layer3.22.conv2.weight", "layer3.22.bn2.weight", "layer3.22.bn2.bias", "layer3.22.conv3.weight", "layer3.22.bn3.weight", "layer3.22.bn3.bias".
    Unexpected key(s) in state_dict: "bn1.running_mean", "bn1.running_var", "layer1.0.bn1.running_mean", "layer1.0.bn1.running_var", "layer1.0.bn2.running_mean", "layer1.0.bn2.running_var", "layer1.0.bn3.running_mean", "layer1.0.bn3.running_var", "layer1.0.downsample.1.running_mean", "layer1.0.downsample.1.running_var", "layer1.1.bn1.running_mean", "layer1.1.bn1.running_var", "layer1.1.bn2.running_mean", "layer1.1.bn2.running_var", "layer1.1.bn3.running_mean", "layer1.1.bn3.running_var", "layer1.2.bn1.running_mean", "layer1.2.bn1.running_var", "layer1.2.bn2.running_mean", "layer1.2.bn2.running_var", "layer1.2.bn3.running_mean", "layer1.2.bn3.running_var", "layer2.0.bn1.running_mean", "layer2.0.bn1.running_var", "layer2.0.bn2.running_mean", "layer2.0.bn2.running_var", "layer2.0.bn3.running_mean", "layer2.0.bn3.running_var", "layer2.0.downsample.1.running_mean", "layer2.0.downsample.1.running_var", "layer2.1.bn1.running_mean", "layer2.1.bn1.running_var", "layer2.1.bn2.running_mean", "layer2.1.bn2.running_var", "layer2.1.bn3.running_mean", "layer2.1.bn3.running_var", "layer2.2.bn1.running_mean", "layer2.2.bn1.running_var", "layer2.2.bn2.running_mean", "layer2.2.bn2.running_var", "layer2.2.bn3.running_mean", "layer2.2.bn3.running_var", "layer2.3.bn1.running_mean", "layer2.3.bn1.running_var", "layer2.3.bn2.running_mean", "layer2.3.bn2.running_var", "layer2.3.bn3.running_mean", "layer2.3.bn3.running_var", "layer3.0.bn1.running_mean", "layer3.0.bn1.running_var", "layer3.0.bn2.running_mean", "layer3.0.bn2.running_var", "layer3.0.bn3.running_mean", "layer3.0.bn3.running_var", "layer3.0.downsample.1.running_mean", "layer3.0.downsample.1.running_var", "layer3.1.bn1.running_mean", "layer3.1.bn1.running_var", "layer3.1.bn2.running_mean", "layer3.1.bn2.running_var", "layer3.1.bn3.running_mean", "layer3.1.bn3.running_var", "layer3.2.bn1.running_mean", "layer3.2.bn1.running_var", "layer3.2.bn2.running_mean", "layer3.2.bn2.running_var", "layer3.2.bn3.running_mean", "layer3.2.bn3.running_var", "layer3.3.bn1.running_mean", "layer3.3.bn1.running_var", "layer3.3.bn2.running_mean", "layer3.3.bn2.running_var", "layer3.3.bn3.running_mean", "layer3.3.bn3.running_var", "layer3.4.bn1.running_mean", "layer3.4.bn1.running_var", "layer3.4.bn2.running_mean", "layer3.4.bn2.running_var", "layer3.4.bn3.running_mean", "layer3.4.bn3.running_var", "layer3.5.bn1.running_mean", "layer3.5.bn1.running_var", "layer3.5.bn2.running_mean", "layer3.5.bn2.running_var", "layer3.5.bn3.running_mean", "layer3.5.bn3.running_var", "layer4.0.bn1.running_mean", "layer4.0.bn1.running_var", "layer4.0.bn2.running_mean", "layer4.0.bn2.running_var", "layer4.0.bn3.running_mean", "layer4.0.bn3.running_var", "layer4.0.downsample.1.running_mean", "layer4.0.downsample.1.running_var", "layer4.1.bn1.running_mean", "layer4.1.bn1.running_var", "layer4.1.bn2.running_mean", "layer4.1.bn2.running_var", "layer4.1.bn3.running_mean", "layer4.1.bn3.running_var", "layer4.2.bn1.running_mean", "layer4.2.bn1.running_var", "layer4.2.bn2.running_mean", "layer4.2.bn2.running_var", "layer4.2.bn3.running_mean", "layer4.2.bn3.running_var".

then i tried loading that weight  file into resnet50 model of torchvision and  it works fine there, so it is clear that the error coming from modified resnet50 with gn and  ws but how do i use pretrained weight of author then? the author shared pretrained weight link in his repo and i can't load that in gn,ws resnet50  model that he designed,i can only use his resnet50 with gn,ws and without the resnet50 weight file he provided,that means i am not able to load weights he shared,where  am i making mistakes?
resnet50 weight was collected from here : https://github.com/joe-siyuan-qiao/pytorch-classification/tree/e6355f829e85ac05a71b8889f4fff77b9ab95d0b


title: How is batching normally performed for sequence data for an RNN/LSTM

body:
This Udacity course notebook batches data in a way that is not intuitive to me.
For a long sequence of data, they first truncates the data so it can be evenly divided by batch_size. Next, they .reshape() the data to be (batch_size, -1). Then they creates a sliding window over these batches of sub-sequences. When the sliding window goes out of bounds, they add fake data (via wrap-around to) to the end.
This provided graphic might explain better than I can:

I'm just wondering if this practice is normal, or if there is a different way. It seems strange that the batches are non-consecutive sub-sequences. Wouldn't that make it hard to interpret the output of a single batch?
Is there a better approach? The woman in the video literally said something to the affect "I don't know why it's done this way, but I've seen it before and the network trains fine".


title: How to get occurrence count of item in array in-place

body:
I have a array which I am trying to get weights for. It looks like this:
Target: tensor([ 1, 6, 5, 8, 10, 5, 4, 5, 10, 10, 9, 8, 10, 4, 10, 9])

I need to get the counts of each item in the array. So I would like it to return this:
Class count: [1, 1, 3, 2, 5, 3, 2, 3, 5, 5, 2, 2, 5, 2, 5, 2]

This is my code right now:
class_sample_count = np.unique(y, return_counts=True)[1]
weight = 1. / class_sample_count

print(f'Target: {y}, Class count: {class_sample_count}, Weight: {weight}')

samples_weight = weight[y]
weights = torch.from_numpy(samples_weight)

.. but the class count prints as this:
Class count: [1 2 3 1 2 2 5]

How can I get the counts of each value while keeping the length of the original array?


title: PyTorch Linear MINST model training error

body:
I am creating a binary classifier based on the MINST dataset using PyTorch. I want my classifier to classify between only 0s and 1s, however, when I train it, the error doesn't decrease and the loss becomes negative.
Here's the error and loss at the first few iterations:

I was obviously expecting better results.
Here is the code I am using:
# Loading the MNISR data reduced to  the 0/1 examples

from torchvision import datasets, transforms
from torch.utils.data import DataLoader

mnist_train = datasets.MNIST("./data", train=True, download=True, transform=transforms.ToTensor())
mnist_test = datasets.MNIST("./data", train=False, download=True, transform=transforms.ToTensor())

train_idx = mnist_train.train_labels <= 1
try:
    mnist_train.train_data = mnist_train.train_data[train_idx]
except AttributeError:
    mnist_train._train_data = mnist_train.train_data[train_idx]
try:
    mnist_train.train_labels = mnist_train.train_labels[train_idx]
except AttributeError:
    mnist_train._train_labels = mnist_train.train_labels[train_idx]

test_idx = mnist_test.test_labels <= 1
try:
    mnist_test.test_data = mnist_test.test_data[test_idx]
except AttributeError:
    mnist_test._test_data = mnist_test.test_data[test_idx]
try:
    mnist_test.test_labels = mnist_test.test_labels[test_idx]
except AttributeError:
    mnist_test._test_labels = mnist_test.test_labels[test_idx]

train_loader = DataLoader(mnist_train, batch_size = 100, shuffle=True)
test_loader = DataLoader(mnist_test, batch_size = 100, shuffle=False)

# Creating a simple linear classifier

import torch
import torch.nn as nn
import torch.optim as optim

# do a single pass over the data
def epoch(loader, model, opt=None):
    total_loss, total_err = 0.,0.
    for X,y in loader:
        yp = model(X.view(X.shape[0], -1))[:,0]
        loss = nn.BCEWithLogitsLoss()(yp, y.float())
        if opt:
            opt.zero_grad()
            loss.backward()
            opt.step()

        total_err += ((yp > 0) * (y==0) + (yp < 0) * (y==1)).sum().item()
        total_loss += loss.item() * X.shape[0]
    return total_err / len(loader.dataset), total_loss / len(loader.dataset)

model = nn.Linear(784, 1)
opt = optim.SGD(model.parameters(), lr=1)
print("Train Err", "Train Loss", "Test Err", "Test Loss", sep="\t")
for i in range(10):
    train_err, train_loss = epoch(train_loader, model, opt)
    test_err, test_loss = epoch(test_loader, model)
    print(*("{:.6f}".format(i) for i in (train_err, train_loss, test_err, test_loss)), sep="\t")

I don't know why my error does not decrease nor why my loss keeps getting more negative. Does anyone spot the error?


title: Number of layers vs list(net.parameters())

body:
New to convolutional neural nets so sorry if this doesn't make much sense. I have this code:
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 5 * 5)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

net = Net()

and I think this has 5 layers. However, when I print len((list(net.parameters())) I get 10. Shouldn't this be a list of size 5 with parameters for each layer?


title: Model not training when implementing a custom loss Pytorch

body:
I have a code of a custom loss that I implemented on tensorflow that I would like to pass on Pytorch for technical reasons. However I can't seem to make it work and I don't know why.
The pytorch loss doesn't seem to train the network
The aim is to compare raw linear output of a network (before softmax) with true probabilities.
The wanted loss is similar to the one in this video at 13:10
Tensorflow code:
import numpy as np
from scipy.special import softmax

import tensorflow as tf
from tensorflow.keras import backend as K
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.layers import Dense, Softmax
from tensorflow.keras import Input, Model

delta = tf.Variable([[1.]], trainable=False)

main_input = Input(shape=(10,))
output = Dense(4, activation='linear')(main_input)

def custom_loss(delta):
    def loss(y_true, y_pred):
        y_pred_softmax = Softmax()(y_pred)
        y_pred_softmax_clipped = K.clip(y_pred_softmax, 1e-8, 1 - 1e-8)
        log_likelihood = y_true * K.log(y_pred_softmax_clipped)
        return K.sum(-log_likelihood * delta)
    return loss

model = Model(inputs=[main_input], outputs=output)
model.compile(optimizer=Adam(lr=0.01), loss=custom_loss(delta))

print(model.predict(np.ones((1,10))))
print(softmax(model.predict(np.ones((1,10)))[0]))
delta.assign([[1.0]])
model.fit(np.ones((1000,10), dtype='float'),np.asarray(1000*[[0.7, 0.3, 0.0, 0.0]], dtype='float'))
print(model.predict(np.ones((1,10))))
print(softmax(model.predict(np.ones((1,10)))[0]))

Pytorch code:
import numpy as np

import torch
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc = nn.Linear(10, 4)

    def forward(self, x):
        x = self.fc(x)
        return x

def custom_loss(delta):
    def loss(y_pred, y_true):
        y_pred_softmax = nn.Softmax(dim=1)(y_pred)
        y_pred_softmax_clipped = torch.clamp(y_pred, 1e-8, 1 - 1e-8)
        log_likelihood = y_true * torch.log(y_pred_softmax_clipped)
        return torch.sum(-log_likelihood * delta)
    return loss

delta = 1
batch_size = 32
n_sample = 1000
network = Net()
loss_function = custom_loss(delta)
optimizer = optim.Adam(network.parameters(), lr=0.001)

x = torch.ones((1,10))
print(network(x))
print(nn.Softmax(dim=1)(network(x)), '\n')

target = [0.7, 0.3, 0.0, 0.0]

for i in range(int(n_sample/batch_size)):
    optimizer.zero_grad()

    inputs = torch.ones((batch_size,10))
    targets = torch.FloatTensor(batch_size*[target])
    outputs = network(inputs)

    loss = loss_function(outputs, targets)
    loss.backward()
    optimizer.step()

x = torch.ones((1,10), dtype=torch.float)
print(network(x))
print(nn.Softmax(dim=1)(network(x)))

Two examples of outputs using pytorch:
tensor([[-0.5445,  1.1960, -0.1856,  0.3652]], grad_fn=<AddmmBackward>)
tensor([[0.0942, 0.5370, 0.1349, 0.2340]], grad_fn=<SoftmaxBackward>)

tensor([[-0.5445,  1.1960, -0.1856,  0.3652]], grad_fn=<AddmmBackward>)
tensor([[0.0942, 0.5370, 0.1349, 0.2340]], grad_fn=<SoftmaxBackward>)

tensor([[-0.1185,  0.5827,  0.7683,  0.4973]], grad_fn=<AddmmBackward>)
tensor([[0.1371, 0.2764, 0.3327, 0.2538]], grad_fn=<SoftmaxBackward>)

tensor([[-0.1185,  1.8670,  0.7683,  0.4973]], grad_fn=<AddmmBackward>)
tensor([[0.0796, 0.5798, 0.1932, 0.1474]], grad_fn=<SoftmaxBackward>)

output with the TF code:
[[ 0.0284583   0.35027373  0.4858752  -0.61461353]]
[0.2229509  0.30758977 0.35225958 0.11719974]
Train on 1000 samples
1000/1000 [==============================] - 0s 302us/sample - loss: 25.1664
[[ 2.3004684  1.4706173 -1.8008366 -2.9013252]]
[0.68579024 0.29908288 0.01135056 0.00377643]


title: How to properly preserve image dimensions

body:
i am currently working on the project where i have to train a network to reconstruct corrupted images from input. Each input image is different in size and has cropped out part in it, which my network should 'restore'.
So i trained the autoencoder, time for test restoration. I load each test image into model and save the output:

to_tensor = transforms.ToTensor()
new_test = []

for i in test_data['images']:
    i = to_tensor(i)
    new_test.append(i)

test_loader = torch.utils.data.DataLoader(dataset=new_test, batch_size=1)

def reconstruct(net, loader):
    for i,data in enumerate(loader):
        data = data.to(device)
        output = net(data)
        #save_image(data, f'test_outs/damaged_input_{i}.jpg')
        save_image(output, f'test_outs/recon_output_{i}.jpg')

net.to(device)
reconstruct(net, test_loader)

These reconstructions are then used to crop out the restored part:
from glob import glob
from pathlib import Path

def cropper(image_array, crop_size, crop_center):
    if type(image_array) != np.ndarray:
        raise ValueError('not a numpy array')
    if type(crop_size) != tuple or type(crop_center) != tuple:
        raise ValueError('use tuple as input for crop_size and crop_center')
    if len(crop_size) != 2 or len(crop_center) != 2:
        raise ValueError('invalid amount of values')
    if crop_size[0] % 2 == 0 or crop_size[1] % 2 == 0:
        raise ValueError('crop_size values should be odd!')

    img = image_array.copy()
    crop_array = np.zeros_like(image_array)

    size_x =  round(crop_center[0] - (crop_size[0]-1)/2)
    center_x = round(crop_center[0] + (crop_size[0]+1)/2)
    size_y = round(crop_center[1] - (crop_size[1]-1)/2)
    center_y = round(crop_center[1] + (crop_size[1]+1)/2)

    border_right = size_y
    border_left = size_x
    border_up = image_array.shape[0] - center_y
    border_down = image_array.shape[1] - center_x

    if border_right < 20 or border_left < 20:
        raise ValueError('minimal distance between crop and border is less than 20')
    if border_up < 20 or border_down < 20:
        raise ValueError('minimal distance between crop and border is less than 20')

    target_array = image_array[int(size_x):int(center_x), int(size_y):int(center_y)]
    pad = np.negative(target_array)
    ones_pad = np.ones_like(target_array)
    img[int(size_x):int(center_x), int(size_y):int(center_y)] += pad
    crop_array[int(size_x):int(center_x), int(size_y):int(center_y)] += ones_pad
    image_array = img

    return (image_array, crop_array, target_array)

def rgb2gray(rgb_array: np.ndarray, r=0.2989, g=0.5870, b=0.1140):
    grayscale_array = (rgb_array[..., 0] * r +
                       rgb_array[..., 1] * g +
                       rgb_array[..., 2] * b)
    grayscale_array = np.round(grayscale_array)
    grayscale_array = np.asarray(grayscale_array, dtype=np.uint8)
    return grayscale_array

def load_recons(root = 'test_outs'):
    recons = []
    path = Path(root).glob('**/*.jpg')
    for enum, f in enumerate(path):
        img = Image.open(f)
        arr = np.asarray(img)
        arr = rgb2gray(arr)
        recons.append(arr)
    return recons

recs = load_recons()

def crop_tests(recons, crop_centrs = test_data['crop_centers'], crop_szs = test_data['crop_sizes']):
    for enum, (i,j,arr) in enumerate(zip(crop_centrs, crop_szs, recons)):
            image, crop, target = cropper(arr, j, i)
            Image.fromarray(target).save(f'test_final/{enum}_target.jpg')

crop_tests(recs)

But the problem is when i call crop_tests, i get one of the exceptions:
---------------------------------------------------------------------------

ValueError                                Traceback (most recent call last)

<ipython-input-27-f927403b1f35> in <module>
      6             Image.fromarray(target).save(f'test_final/{enum}_target.jpg')
      7
----> 8 crop_tests(recs)
      9

<ipython-input-27-f927403b1f35> in crop_tests(recons, crop_centrs, crop_szs)
      3 def crop_tests(recons, crop_centrs = test_data['crop_centers'], crop_szs = test_data['crop_sizes']):
      4     for enum, (i,j,arr) in enumerate(zip(crop_centrs, crop_szs, recons)):
----> 5             image, crop, target = cropper(arr, j, i)
      6             Image.fromarray(target).save(f'test_final/{enum}_target.jpg')
      7

~\proj_git\folder_preparator.py in cropper(image_array, crop_size, crop_center)
     52                 raise ValueError('minimal distance between crop and border is less than 20')
     53         if border_up < 20 or border_down < 20:
---> 54                 raise ValueError('minimal distance between crop and border is less than 20')
     55
     56

ValueError: minimal distance between crop and border is less than 20

This means, that the square to crop out is very near the border of reconstructed image (less than 20 pixels). However, it was not a problem with damaged image.
So the shapes of reconstructed images are following:
(70, 93)
(70, 93)
(70, 93)
(70, 93)
(70, 93)
(93, 70)
(70, 93)
(93, 70)
(93, 70)
(70, 93)
(93, 70)
(70, 93)
(70, 93)
(70, 93)
(70, 93)

However, shapes of original images are in a bit different order:
(70, 93)
(70, 93)
(70, 93)
(70, 93)
(70, 93)
(70, 93)
(70, 93)
(70, 93)
(93, 70)
(93, 70)
(70, 93)
(70, 93)
(70, 93)
(100, 70)

So i need my reconstructions to be of exactly same shape as the input, but i can't see the problem in the code myself. Could you please help me find it out? Feel free to ask additional questions.


title: Block_reduce with torch tensors on GPU

body:
I have a 2D tensor and I am trying to reduce it by taking average of every 4 columns. For numpy arrays this can be done by using the block_reduce function from skimage.measure module.
I can use it for CPU torch tensors too, as it converts the input tensor to numpy array in the background. This won't work when my tensors are loaded on the GPU. Is there some efficient way to achieve this in PyTorch?
Here is what I want to do, illustrated by block_reduce:
import numpy as np
import torch
from skimage.measure import block_reduce

t1 = np.array([[0,4,2,4,1,1,1,5],[1,4,2,4,1,1,1,5],[2,4,2,4,1,1,1,5],[3,4,2,4,1,1,1,5]]).astype('float')
t1 = torch.from_numpy(t1)

t2 = block_reduce(t1, block_size=(1,4), func=np.mean)

print(t1)
print(t2)

The output produced:
tensor([[0., 4., 2., 4., 1., 1., 1., 5.],
        [1., 4., 2., 4., 1., 1., 1., 5.],
        [2., 4., 2., 4., 1., 1., 1., 5.],
        [3., 4., 2., 4., 1., 1., 1., 5.]], dtype=torch.float64)
[[2.5  2.  ]
 [2.75 2.  ]
 [3.   2.  ]
 [3.25 2.  ]]


title: When to use detach

body:
If I have two different neural networks (parametrized by model1 and model2) and corresponding two optimizers, would the below operation using model2.parameters without detach() lead to change in its gradients? My requirement is that I want to just compute the mean squared loss between the two model parameters but update the optimizer corresponding to model1, leaving model2 as is.
opt1 = torch.optim.SGD(self.model1.parameters(), lr=1e-3)
opt2 = torch.optim.SGD(self.model2.parameters(), lr=1e-3)

loss = (self.lamb / 2.) * ((torch.nn.utils.parameters_to_vector(self.model1.parameters()) - torch.nn.utils.parameters_to_vector(self.model2.parameters()))**2).sum()

loss.backward()
opt1.step()

How can I decide in general whether to use detach for any operation or not?


title: how does nn.embedding for developing an encoder-decoder model works?

body:
In this tutorial, it teaches how to develop a simple encoder-decoder model with attention using pytorch.
However, in the encoder or decoder, self.embedding = nn.Embedding(input_size, hidden_size) (or similar) is defined. In pytorch documents, nn.Embedding is defined as "A simple lookup table that stores embeddings of a fixed dictionary and size."
So I am confused that, in the initialization, where does this lookup table has come from? Does it initialize some random embeddings for the indices and then they will be trained? Is it really necessary to be in the encoder/decoder part?
Thanks in advance.


title: When training the bilstm+crf model, the emission matrix does not change

body:
I want to build BiLSTM+CRF model to solve NER problem. Build a model based on pytorch.
However, the emission matrix of each epoche has not changed.
I modified the official model code of pytorch. In the bilstm layer, I compressed each batch and input it into the lstm layer, but the output emission matrix remained unchanged.I don't know why the BiLSTM layer doesn't work.
Please help me, thanks.
here is my code.
import utils

#1、先用embedding测试一下
'''random.seed(1)
np.random.seed(1)
torch.manual_seed(1)'''

class BILSTM_CRF(nn.Module):
    def __init__(self,batch_size,vocab_size,tag_to_idx,embed_dim,hidden_dim,word_to_idx):
        super(BILSTM_CRF,self).__init__()
        self.batch_size = batch_size
        self.embed_dim = embed_dim
        self.hidden_dim = hidden_dim
        self.tag_to_idx = tag_to_idx
        self.vocab_size = vocab_size
        self.target_size = len(tag_to_idx) # 要带着start和end
        self.word_to_idx = word_to_idx

        self.embedding = nn.Embedding(vocab_size,embed_dim)

        self.lstm = nn.LSTM(embed_dim,hidden_dim // 2,
                            bidirectional=True,
                            batch_first=True)

        # 发射函数
        self.emission = nn.Linear(hidden_dim,self.target_size)

        # crf 部分，参数的初始化
        # 注意：t[i][j]是指由tag j转移到tag i
        self.transitions = nn.Parameter(torch.randn(self.target_size,self.target_size,requires_grad=True))

        # 其中start不会由任何tag转移
        # end不会转移任何到tag，所以更新这两个的转移值变得很小
        self.transitions.data[tag_to_idx["<start>"],:] = -10000
        self.transitions.data[:,tag_to_idx["<end>"]] = -10000

    # 通过bilstm层获得发射函数
    def _get_lstm_features(self,sents):
        h0 = torch.randn(2,self.batch_size,self.hidden_dim // 2)
        c0 = torch.randn(2, self.batch_size, self.hidden_dim // 2)

        lengths = utils.get_lengths(sents) # 导入的序列长度

        # 下面是修改了官方代码，将整个batch压缩后放进lstm层计算
        input_tensor = utils.pad_sents(sents,self.word_to_idx) # 填充与编码sents
        # print("input",input_tensor)

        emb = self.embedding(input_tensor)
        # print(emb.shape)
        # 压缩
        input_packed = torch.nn.utils.rnn.pack_padded_sequence(emb,lengths,
                                                               batch_first=True)
        output_tensor,_ = self.lstm(input_packed)
        # 解压缩
        output_tensor,_ = torch.nn.utils.rnn.pad_packed_sequence(output_tensor,
                                                                 batch_first=True)
        emission = self.emission(output_tensor) # size[batch,length,tag_size]
        return emission

    def neg_log_likelihood(self,sents,tags): # loss函数
        # sents[batch,length] tags[batch,length]
        feats = self._get_lstm_features(sents) # size[batch,length,tag_size]

        # ！！！！！！For each epoche, the output here is the same
        print("feats",feats)

        forward_score = self._forward_alg(feats) # size[batch,1]
        # print("forward_score",forward_score.shape)
        gold_score = self._score_sentence(feats,tags) # size[batch,1]
        # print("gold_score",gold_score.shape)

        losses = forward_score - gold_score # size[batch,1]
        loss = torch.sum(losses)
        return loss,feats

utils is a package containing some tool functions
show below
# 一些小函数
import torch

# 统计序列长度
def get_lengths(sents):
    lengths = []
    for sent in sents:
        lengths.append(len(sent))
    return lengths

# 填充序列等长
# 放进来的sents是排好序的
# pad是填充内容
# default是不在字典中的默认内容，<unk>
def pad_sents(sents,word_map):
    UNK = word_map.get('<unk>')
    PAD = word_map.get('<pad>')
    max_len = len(sents[0])
    # batch_tensor = torch.ones(len(sents),max_len,dtype=torch.long,requires_grad=True)*pad
    batch_tensor = torch.ones(len(sents),max_len,requires_grad=True).long()*PAD

    for i,l in enumerate(sents):
        for j,e in enumerate(l):
            batch_tensor[i][j] = word_map.get(e,UNK)

    return batch_tensor

# 句子按照长短排序用
def sort_pairs(sents_list,tag_list):
    pairs = list(zip(sents_list,tag_list))
    # print(pairs)
    indices = sorted(range(len(pairs)),
                     key = lambda k:len(pairs[k][0]),
                     reverse = True)
    pairs = [pairs[i] for i in indices]

    sents_list,tag_list = list(zip(*pairs))

    return sents_list,tag_list


title: Loss doesn't decrease when I apply a nn model in Pytorch

body:
I am using a toy model to predict a feature called Probability from a dataset which has almost 30 features. And I apply a toy model and nn.MSELoss(), trying to fit the model with the dataset. However,  I found the loss didn't decrease.
The model is quite simple, which codes are listed below.
class Logistic(nn.Module):
    def __init__(self, input_size, output_size=1):
        super(Logistic, self).__init__()
        self.input_size = input_size
        self.output = 0
        self.layer1 = nn.Linear(input_size, output_size)
        self.activation = nn.Sigmoid()

    def forward(self, x):
        x = self.layer1(x)
        x = self.activation(x)
        return x

And I used Dataset and DataLoader to read the DataFrame format of my dataset. Also, I trained my model in GPU.
criterion = nn.MSELoss()
total_epoch = 50
logi = Logistic(len(valuable_col) - 1, 1)
logi.to(device)
lr = 100

optimizer = torch.optim.Adam(logi.parameters(), lr=lr)
train_loss = []
valid_loss = []

logi.double()

for epoch in range(total_epoch):
    logi.train()
    train_loss_ = 0
    for data_, target_ in tqdm.tqdm(train_loader):
        optimizer.zero_grad()
        data_, target_ = data_.to(device), target_.to(device)

        output = logi.forward(data_)
        loss = criterion(output, (target_).long())
        train_loss_ += loss.item()
        loss.backward()
        optimizer.step()
    train_loss.append(train_loss_ / len(train_loader))

    logi.eval()
    valid_loss_ = 0
    for data_, target_ in tqdm.tqdm(test_loader):
        data_, target_ = data_.to(device), target_.to(device)

        output = logi.forward(data_)
        valid_loss_ += criterion(output, target_)
    valid_loss.append(valid_loss_.item() / len(test_loader))
    print("[Epoch: %d] train_loss: %.6f, valid_loss: %.6f" % ((epoch + 1, train_loss[-1], valid_loss[-1])))

I found the loss is almost the same in each epoch. I can not figure out why. And this make me confused.


title: Cuda out of memory- Matrix multiplication

body:
I am trying to run Stand-Alone-Self-Attention model.
Even with batch size=1, it complains CUDA out of memory because of out=key*query: https://github.com/leaderj1001/Stand-Alone-Self-Attention/blob/a983f0f643632b1f2b7b8b27693182f22e9e574c/attention.py#L48
The tensor of the key is [2,8,8,224,224,49]
The tensor of the query is [2,8,8,224,224,1]
The dim 1 is batch, dim 2 is self.groups, dim 3 is out_channels, dim 4 is height, dim 5 is width, and dim 6 = -1.
A very naive thought is to split these two matrixes into several blocks, like split3D used for self-attention implemented in Keras. But I am not sure how will it work in 6 dimensions.
Thanks!


title: Understanding how to use classes properly in Python

body:
I am trying to understand how to use concrete examples of classes in Python(more particularly in Pytorch).I have the two following classes:
class Context:

    def __init__(self):
        self._saved_tensors = ()
    def save_for_backward(self, *args):
        self._saved_tensors = args
    @property
    def saved_tensors(self):
        return self._saved_tensors

class MSE(Function):

@staticmethod
def forward(ctx, yhat, y):

    ctx.save_for_backward(yhat,y)

    return (1/(yhat.shape[0])*((yhat-y)**2)).sum()
@staticmethod
def backward(ctx, grad_output):

    yhat, y = ctx.saved_tensors
    out_weight =grad_output*(2/(yhat.shape[0])*(y-yhat)).sum()
    out_bias =(2/(yhat.shape[0])*(y-yhat)).sum()
    return out_weight,out_bias

and when i apply it to the following example:
ctx=LinearFunction()
#identity matrix of size 3
yhat=torch.eye(3)
y=torch.ones(3,3)
grad_output=1
loss=MSE(ctx,yhat,y)
loss.backward(ctx,grad_output)

it returns me an error:
4 grad_output=1
5 loss=MSE(ctx,yhat,y)
6 loss.backward(ctx,grad_output)

<ipython-input-38-0c861407599f> in backward(ctx, grad_output)
   13 def backward(ctx, grad_output):
   15 yhat, y = ctx.saved_tensors
   16 out_weight = (2/(yhat.shape[0])*(y-yhat)).sum()
   17 out_bias =(2/(yhat.shape[0])*(y-yhat)).sum()

ValueError: not enough values to unpack (expected 2, got 0)

and the code shows an arrow pointing to lines 6 and 15 indicating the error comes from there.
I understand the code needs yhat and y but I already defined them.So why does the compiler not use them?
Could someone help me out on this? Thank you.


title: Can we reset .requires_grad of all defined tensors in a code to zero at once

body:
I am using PyTorch 1.6.0 to learn a tensor (lets say x) with autograd.
After x is learnt, how can I reset .requires_grad of every tensor that was a node in the autograd comp. graph to zero?
I know about torch.detach() and about setting .requires_grad to False manually. I am searching for an one-shot instruction.
Ps: I want to do that because I still want to use these tensors after the part of my code that learns x is executed. Plus, some are to be converted to numpy.


title: Data format and stride

body:
First of all, What is channel last format? Is it NHWC?
For the following code, how stride is calculated?
import torch
N, C, H, W = 10, 3, 32, 32
x = torch.empty(N, C, H, W)
print(x.stride()) # Ouputs: (3072, 1024, 32, 1)

Output is (3072, 1024, 32, 1)
When converted to channel last format, how stride is calculated here?
x = x.contiguous(memory_format=torch.channels_last)
print(x.shape) # Outputs: (10, 3, 32, 32) as dimensions order preserved
print(x.stride()) # Outputs: (3072, 1, 96, 3)

Output is torch.Size([10, 3, 32, 32])
(3072, 1, 96, 3)


title: torch.matmul gives RuntimeError

body:
I have two tensors
t1=torch.Size([400, 32, 400])
t2= torch.Size([400, 32, 32])

when i excute this
torch.matmul(t1,t2)
i got this error RuntimeError:

Expected tensor to have size 400 at dimension 1, but got size 32 for
argument #2 'batch2' (while checking arguments for bmm)

Any help will be much appreciated


title: Add validation data on pytorch

body:
Below I split my data on train and test and then load into a tensordataset. Which is a straightforward way to add a validation split?
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 0, stratify = y)

train_data = TensorDataset(torch.from_numpy(X_train), torch.from_numpy(y_train))
train_loader = DataLoader(train_data, shuffle=True, batch_size=batch_size)

test_data = TensorDataset(torch.from_numpy(X_test), torch.from_numpy(y_test))
test_loader = DataLoader(test_data, shuffle=True, batch_size=batch_size)


title: Modifying Dataset class for feeding data to a branched CNN in pytorch?

body:
I am trying to make an image classifier to classify images into hierarchical classes (coarse_label1, coarse_label2, fine_label). The data is stored in a csv file with columns image_path, coarse_label1, coarse_label2, fine_label. I am trying to use the Dataset and DataLoader class from torch.data.utils to create the data generator for feeding it to my model.
I went through a few tutorials (Tutorial1 and Tutorial2) and modified the dataset class as follows:
import torch
import numpy as np
import pandas as pd
from torch.utils.data import Dataset, DataLoader
from torchvision import transforms, utils
from PIL import Image
from imgaug import augmenters as iaa

class WDataset(Dataset):
    def __init__(self, img_path, coarse1, coarse2, fine, train_mode, shape):
        """
        Args:
            img_path (list): List of image paths.
            coarse1, coarse2, fine: List of coarse and fine labels.
        """
        self.img_path = img_path
        self.c1_label_list = coarse1
        self.c2_label_list = coarse2
        self.fine_label_list = fine
        self.train_mode = train_mode
        self.shape = shape

    def __len__(self):
        return len(self.fine_label_list)

    def __getitem__(self, idx):
        if torch.is_tensor(idx):
            idx = idx.tolist()

        augmentor = self._get_augmentor()

        image = self.pil_loader(self.img_path[idx])
        image = np.array(image.resize(self.shape))
        output = self.fine_label_list[idx]
        output_c1 = self.c1_label_list[idx]
        output_c2 = self.c2_label_list[idx]
        if self.train_mode:
            image = augmentor.augment_image(image)

        return image, output_c1, output_c2, output

    def pil_loader(self, path: str) -> Image.Image:
        # open path as file to avoid ResourceWarning (https://github.com/python-pillow/Pillow/issues/835)
        with open(path, 'rb') as f:
            img = Image.open(f)
            return img.convert('RGB')

    def _get_augmentor(self):
        arg_list = []
        arg_list.append(iaa.Affine(shear=(-5, 5), name="shear"))
        arg_list.append(iaa.Affine(rotate=(-25, 25), name="rotate"))
        arg_list.append(iaa.Dropout((0, 0.3), name="gomashio"))
        arg_list.append(iaa.contrast.LinearContrast((0.7, 1.3), name="contrast"))
        arg_list.append(iaa.Multiply((0.7, 1.3), name="darken_lighten"))
        seq = iaa.SomeOf((0, 3), arg_list, random_order=True)
        return seq

Using this class, I am able to generate a dataset and access the data using an index from the generated dataset. However, when I create a DataLoader and try to grab a batch of data from it, the task does not terminate. In other words
train_ds = WDataset(X[train_index], y_c1[train_index], y_c2[train_index], y_fine[train_index], train_mode=True, shape=(height, width))
image, coarse1, coarse2, fine = train_ds[index]

works as expected.
But the code below keeps on processing and doen't give any warning, error or output.
# Parameters
params = {'batch_size': 4,
          'shuffle': True,
          'num_workers': 2}
# Generators
train_generator = torch.utils.data.DataLoader(train_ds, **params)
next(iter(train_generator))

I want to know if the way I have written WDataset is correct or not?
How else should I modify it so that I can create a DataLoader?
Is there any other way to feed such (hierarchical/multi-class classification) data to a model in PyTorch?
Edit:
I found the source of the problem. Turns out, this error occurs in macOS with python >= 3.8 when num_workers is set greater than zero in DataLoader. Here's the github issue which describes this problem.
From a comment in the issue:
"It is a fork vs spawn issue here. Thus multiprocessing related, rather than PyTorch.
In Python 3.8 on mac now the default backend for multiprocessing is now spawn, and the default collate_fn could not be found.
This could be resolved by either forcing the start_method to fork; or defining the collate_fn in a separate module, and import it."


title: How to do data visualization from inside vscode devcontainer?

body:
I have worked with various ML frameworks and have used ploty and pyplot for data visualization with no trouble. But of course I'm finding that these frameworks can be less dependable (without workarounds) inside docker containers.
I am trying to set up a good standard environment where I can use either pyplot or plotly (preferably plotly) from inside a standard container (example nvcr.io/nvidia/pytorch).
I have (to my knowledge) solved the issues of enabling X11 in .devcontainer.json in my vscode project. I do not get any runtime exceptions saying that my code cannot connect to display, but still I see nothing on screen after code to create a figure runs.
My guess is that there are some other fundamental hurdles to workaround and I'm trying to understand a good approach to take in order to build on top of a container and still get a good high quality data viz out of it.
My envornment is Linux Mint and vscode.  See uname -a and my .devcontainer.json runargs below.
Linux <hostname> 4.15.0-54-generic #58-Ubuntu SMP x86_64 x86_64 x86_64 GNU/Linux

    "runArgs": ["--gpus", "all",
                "--runtime=nvidia",
                "--net=host",
                "--env", "${env:DISPLAY}",
                "--env", "QT_X11_NO_MITSHM=1",
                "--volume=/home/joey.Xauthority:/root/.Xauthority:rw",
                "--volume=/tmp/.X11-unix:/tmp/.X11-unix:rw",
                "--privileged"
                ]


title: Why am I randomly getting super high perplexities?

body:
A few days ago I finished writing a word prediction program that tests both LSTM and GRU models on a given dataset. I test 4 models - 2 LSTM models and 2 GRU models. I wrote the program on Google Colab.
I use two validation sets to see how it affects their perplexity. I left it quite satisfied with the results I've gotten. Now, a few days later, when I came back to run it I see that I randomly get extremely huge perplexities on the first epoch for at least one of the validation sets, and when I do, those last for all 4 models. However, when I stop the program after the first epoch and run it again immediately, it sometimes fixes the perplexity issues on the first epoch (and consequently on the entire program). This can be done again and again until all 3 datasets give normal results.
Could there be any google colab issue that's causing these random flops in the runs?
I'm asking in case I can prevent digging into my code for no reason, as some times it just works perfectly fine.
Thanks!
EDIT: After testing it for quite a few times I noticed the following ALWAYS happens on the first epoch after I restart runtime:

Training perplexity is normal (as always), both validation sets' is not. I stop and Run All again.
Training perplexity is normal (as always), validation set 1 is normal, validation set 2 is not. I stop and Run All again.
Training perplexity is normal (as always), both validation sets' are normal. I stop and Run All again.
same as 3.


title: Can I calculate loss on specific output value with pytorch binary classifier

body:
I am building a binary classifier.
I want to calculate loss on predicted value 1 only, ignoring 0.
criterion = nn.BCELoss()
preds = model(inputs)
#ignore preds with value 0 and labels with the 0 value index
loss = criterion(preds, labels)

I am not sure, if the network can converge theoretically too.


title: How do I normalise Tensor dataset

body:
At first I have a list of lists of lists.
Each pair corresponds to variables x and y. (For instance in the first list of lists we have x=1 y=2, x=1 y=3 x=1 y=4)
tensor_input=[[[1,2],[1,3],[1,4]],[[2,5],[0,3],[3,5]]]

Now I am transforming it into a numpy array and then into a tensor
tensor_input = np.array(tensor_input)
tensor_input = torch.Tensor(tensor_input)

I am doing the same operation with tensor_output.
My question is how can I normalise the variables x and y so that I have a normalised input and can then feed it to my neural network with Dataloader. Maybe I should do it by excluding a  variable x from the list of lists to get x=[1,1,1,2,0,3] , same with y=[2,3,4,5,3,5] then normalise it and create a list of lists once again from normalised vectors? Or is there a way to do it easier in pytorch?
data_set = TensorDataset(tensor_input, tensor_output)
train, test = torch.utils.data.random_split(data_set, [304, 60])
train_loader = DataLoader(train, batch_size=4, shuffle=True)

And a second question as I get the normalised output from the nerual network, how can I retrieve the actual variable?


title: Error in getting accuracy using test set with PyTorch

body:
I am trying to find the accuracy of my model that I created with PyTorch, but I get an error. Originally I had a different error, which is fixed, but now I get this error.
I use this to get my test set:
testset = torchvision.datasets.FashionMNIST(MNIST_DIR, train=False,
                                       download=True, transform=torchvision.transforms.Compose([
                                            torchvision.transforms.ColorJitter(brightness=0.1, contrast=0.1, saturation=0.1, hue=0.1),
                                            torchvision.transforms.ToTensor(), # image to Tensor
                                            torchvision.transforms.Normalize((0.1307,), (0.3081,)) # image, label
                                            ]))
testloader = torch.utils.data.DataLoader(testset, batch_size=100,
                                         shuffle=False)

When I try to access the test set I created, it tries to retrain the model for some reason, then proceeds to error out.
This is the code that gets the accuracy and calls the test set
correct = 0
total = 0
with torch.no_grad():
    print("entered here")
    for (x, y_gt) in testloader:
        x = x.to(device)
        y_gt = y_gt.to(device)
        outputs = teacher_model(x)
        _, predicted = torch.max(outputs.data, 1)
        total += labels.size(0)
        correct += (predicted == labels).sum().item()

print('Accuracy of the network on the 10000 test images: %d %%' % (100 * correct / total))

This is the error I am getting:
Traceback (most recent call last):
  File "[path]/train_teacher_1.py", line 134, in <module>
    outputs = teacher_model(x)
  File "[path]\anaconda3\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "[path]\models.py", line 17, in forward
    x = F.relu(self.layer1(x))
  File "[path]\anaconda3\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "[path]\anaconda3\lib\site-packages\torch\nn\modules\linear.py", line 93, in forward
    return F.linear(input, self.weight, self.bias)
  File "[path]\anaconda3\lib\site-packages\torch\nn\functional.py", line 1692, in linear
    output = input.matmul(weight.t())
RuntimeError: mat1 dim 1 must match mat2 dim 0

Please let me know if you would like the rest of the code for training the model. I left it out because the post got too long.
I am new to PyTorch and any help is appreciated. Thanks in advance.


title: pytorch crashing in backward() and screen freezing, could you help me?

body:
import torch

def forward(x):
    return x * w

def loss(x, y):
    y_pred = forward(x)
    return (y_pred - y) ** 2

x_data = [1.0, 2.0, 3.0]
y_data = [2.0, 4.0, 6.0]

w = torch.Tensor([1.0])
w.requires_grad = True

print("predict (before training)", 4, forward(4).item())

for epoch in range(100):
    for x, y in zip(x_data, y_data):
        l = loss(x, y)
        l.backward()
        print('\tgrad:', x, y, w.grad.item())
        w.data = w.data - 0.01 * w.grad.data
        w.grad.data.zero_()
        print("progress:", epoch, l.item())
print("predict (after training)", 4, forward(4).item())

Linux distribution is Ubuntu 18.04
GPU: GeForce RTX 2080 SUPER
GPU driver: NVIDIA UNIX x86_64 Kernel Module  450.80.02
CUDA: Cuda compilation tools, release 11.0, V11.0.194
cudNN: 8.0.3
pytorch: 1.7.0, py3.6_cuda11.0.221_cudnn8.0.3_0
python: 3.6
code is very simple, but freezes the whole computer when I run it
screen, keyboard, mouse, not responding at all


title: Calculation of Mean and Std of Phototour dataset

body:
I've got a question about the calculation of Mean and Std of Phototour dataset.
Phototour dataset provides corresponding patches with all shape of (64,64,1) to train a descriptor (usually CNN networks).
These patches are extracted from original images. these original images are captured by tourists from all around the world from specific locations (Notredame, Liberty, Yosemite) with different image resolutions. center of every patch (32, 32) is a keyPoint that detected by SIFT detector.
for example, in these datasets we have nearly 500K patchs that extracted from 715 images for Notredame.
The link to the dataset:
http://phototour.cs.washington.edu/
In Pytorch Phototour dataset they defined those as below:
means = {'notredame': 0.4854, 'yosemite': 0.4844, 'liberty': 0.4437,
         'notredame_harris': 0.4854, 'yosemite_harris': 0.4844, 'liberty_harris': 0.4437}
stds = {'notredame': 0.1864, 'yosemite': 0.1818, 'liberty': 0.2019,
        'notredame_harris': 0.1864, 'yosemite_harris': 0.1818, 'liberty_harris': 0.2019}

How can I reproduce these two parameters?
I must calculate Mean and Std on extracted patches or on original images?

Any help would be appreciated.