title: How is Siamese network realized with Pytorch if it is single input during inference?

body:
I am trying to train one CNN model with Pytorch, so that the output behaves differently for different types of inputs. (i.e. If the input images are human-beings, it outputs pattern A, but if the input is some other animals, it outputs pattern B).

After some online search, it seems Siamese network is related to this. So I have the following 2 questions:

(1)  Is Siamese network really a good way to train such a model?

(2)  From the implementation point of view, how should I implement the code in pytorch?

class SiameseNetwork(nn.Module):
def __init__(self):
    super(SiameseNetwork, self).__init__()
    self.cnn1 = nn.Sequential(
        nn.ReflectionPad2d(1),
        nn.Conv2d(1, 4, kernel_size=3),
        nn.ReLU(inplace=True),
        nn.BatchNorm2d(4),

        nn.ReflectionPad2d(1),
        nn.Conv2d(4, 8, kernel_size=3),
        nn.ReLU(inplace=True),
        nn.BatchNorm2d(8),

        nn.ReflectionPad2d(1),
        nn.Conv2d(8, 8, kernel_size=3),
        nn.ReLU(inplace=True),
        nn.BatchNorm2d(8),

    )

    self.fc1 = nn.Sequential(
        nn.Linear(8*100*100, 500),
        nn.ReLU(inplace=True),

        nn.Linear(500, 500),
        nn.ReLU(inplace=True),

        nn.Linear(500, 5))

def forward_once(self, x):
    output = self.cnn1(x)
    output = output.view(output.size()[0], -1)
    output = self.fc1(output)
    return output

def forward(self, input1, input2):
    output1 = self.forward_once(input1)
    output2 = self.forward_once(input2)
    return output1, output2

Currently, I am trying some existing implementation I found online like the above class definition. It works, but there will always be two inputs and two outputs for this model. I agree that it is convenient for training, but ideally, it should be only one input and one (two is also fine) output during inference.

Could someone provide some guidance on how to modify the code to make it single input?


title: Reading csv.gz file in torchtext

body:
Pandas’s read_csv works for csv.gz as well.

Is there a way to achieve similar with PyTorch?https://torchtext.readthedocs.io/en/latest/data.html#torchtext.data.Dataset doesn’t seem to have such an option.


title: BERT finetuning : training loss decreases but accuracy remains unchanged

body:
I have been training bert on a custom dataset for a while now. While training I observe that while the training loss is decreasing the val accuracy remains fixed at 0.5. I have tried out all the different BERT models available in Huggingface and don't know what to do. Here's the output during training:
step:  0 of total steps:  250
step:  25 of total steps:  250
step:  50 of total steps:  250
step:  75 of total steps:  250
step:  100 of total steps:  250
step:  125 of total steps:  250
step:  150 of total steps:  250
step:  175 of total steps:  250
step:  200 of total steps:  250
step:  225 of total steps:  250
Average loss for epoch:  0.4582762689590454
/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:6: DeprecationWarning: elementwise comparison failed; this will raise an error in the future.

  Accuracy: 0.50
 Eval loss:  0.40
Epoch:  1
step:  0 of total steps:  250
step:  25 of total steps:  250
step:  50 of total steps:  250
step:  75 of total steps:  250
step:  100 of total steps:  250
step:  125 of total steps:  250
step:  150 of total steps:  250
step:  175 of total steps:  250
step:  200 of total steps:  250
step:  225 of total steps:  250
Average loss for epoch:  0.3786401364207268
  Accuracy: 0.50
 Eval loss:  0.39```

Can this be overfitting by any chance?


title: Same padding when kernel size is even

body:
When the kernel size is odd, we can manually calculate the necessary padding to get the output in the same dimension as input such that it creates same padding.
But how can we calculate padding dimensions for kernels with even sizes (ex: (2x2)?


title: Is this the correct way to calculate the covariance matrix for 2D feature map?

body:
Within a neural network, I have some 2D feature maps with values between 0 and 1. For these maps, I want to calculate the covariance matrix based on the values at each coordination. Unfortunately, pytorch has no .cov() function like in numpy. So I wrote the following function instead:
def get_covariance(tensor):
    bn, nk, w, h = tensor.shape
    tensor_reshape = tensor.reshape(bn, nk, 2, -1)
    x = tensor_reshape[:, :, 0, :]
    y = tensor_reshape[:, :, 1, :]
    mean_x = torch.mean(x, dim=2).unsqueeze(-1)
    mean_y = torch.mean(y, dim=2).unsqueeze(-1)

    xx = torch.sum((x - mean_x) * (x - mean_x), dim=2).unsqueeze(-1) / (h * w - 1)
    xy = torch.sum((x - mean_x) * (y - mean_y), dim=2).unsqueeze(-1) / (h * w - 1)
    yx = xy
    yy = torch.sum((y - mean_y) * (y - mean_y), dim=2).unsqueeze(-1) / (h * w - 1)

    cov = torch.cat((xx, xy, yx, yy), dim=2)
    cov = cov.reshape(bn, nk, 2, 2)
    return cov

Is that the correct way to do it?
Edit:
Here is a comparison with the numpy function:
a = torch.randn(1, 1, 64, 64)
a_numpy = a.reshape(1, 1, 2, -1).numpy()

torch_cov = get_covariance(a)
numpy_cov = np.cov(a_numpy[0][0])

torch_cov
tensor([[[[ 0.4964, -0.0053],
          [-0.0053,  0.4926]]]])

numpy_cov
array([[ 0.99295635, -0.01069122],
       [-0.01069122,  0.98539236]])

Apparently, my values are too small by a factor of 2. Why could that be?
Edit2: Ahhh I figured it out. It has to be divided by (h*w/2 - 1) :) Then the values match.


title: Oversampling the dataset with pytorch

body: I'm quite new to PyTorch and python. and I have a binary classification problem where one class have more samples than the other, so I decided to oversample the class that has less number of samples by doing more augmentation on it, so for example I would generate 7 images out of one sample for one class, while for the other class I would generate 3 images out of one sample. I'm using imguag for augmentation with PyTorch, so I'm not sure which is better, to augment my dataset first, then passing it to torch.utils.data.Dataset class, or reading the data and augmenting it inside init function of Dataset class.


title: BERT embeddings in batches

body:
I am following this post to extract embeddings for sentences and for a single sentence the steps are described as follows:
    text = "After stealing money from the bank vault, the bank robber was seen " \
           "fishing on the Mississippi river bank."

    # Add the special tokens.
    marked_text = "[CLS] " + text + " [SEP]"

    # Split the sentence into tokens.
    tokenized_text = tokenizer.tokenize(marked_text)

    # Mark each of the 22 tokens as belonging to sentence "1".
    segments_ids = [1] * len(tokenized_text)

    # Convert inputs to PyTorch tensors
    tokens_tensor = torch.tensor([indexed_tokens])
    segments_tensors = torch.tensor([segments_ids])

    # Load pre-trained model (weights)
    model = BertModel.from_pretrained('bert-base-uncased',
                                 output_hidden_states = True,
                                      )

    # Put the model in "evaluation" mode, meaning feed-forward operation.
    model.eval()

    with torch.no_grad():
        outputs = model(tokens_tensor, segments_tensors)
        hidden_states = outputs[2]

And I want to do this for a batch of sequences. Here is my example code:
seql = ['this is an example', 'today was sunny and', 'today was']
encoded = [tokenizer.encode(seq, max_length=5, pad_to_max_length=True) for seq in seql]

encoded
[[2, 2511, 1840, 3251, 3],
 [2, 1663, 2541, 1957, 3],
 [2, 1663, 2541, 3, 0]]

But since I'm working with batches, sequences need to have same length. So I introduce a padding token (3rd sentence) which confuses me about several points:

What should the segment id for pad_token (0) will be?

Should I use attention masking when feeding the tensors to the model so that padding is ignored? In the example only token and segment tensors are used.
outputs = model(tokens_tensor, segments_tensors)

If I don't work with batches but with individual sentences, then I might not need a padding token. Would it be better to do that compared to batches?


title: How can I feed a batch into LSTM without reordering them by length in PyTorch?

body:
I am new to Pytorch and I have got into some trouble.
I want to build a rank-model to judge the similarity of question and its answers (including right answers and wrong answers). And I use LSTM as an encoder.

There are two LSTMs in my model and they share the weights. So my model’s input is two sequences (question and answer). But if I use batch, reordering will disrupt the correspondence between questions and answers.
What should I do?


title: ImportError when call a pytorch (python) model in matlab

body:
When I try to call a pytorch python model in matlab script, I met an ImportError: "dlopen: cannot load any more object with static TLS".

This is a simple example of the problem:

% matlab script test.m

clear classes
if count(py.sys.path,'') == 0
    insert(py.sys.path,int32(0),'');
end
mod = py.importlib.import_module('test_python');
py.reload(mod);
py.test_python.myfunction;

# python script test_python.py

import torch.optim as optim

def myfunction():
    print 'hello world 2!'

if __name__ == "__main__":
    myfunction()

The  the error message

Error using __init__><module> (line 84)
Python Error: ImportError: dlopen: cannot load any more object with static TLS

Error in test_python><module> (line 3)
import torch.optim as optim

Error in __init__>import_module (line 37)
    __import__(name)

The python script can be run independently, without error.
If do not import torch (delete the line "import torch.optim as optim"), there will be no error.
I am running MATLAB 2017a, Python 2.7 and Pytorch 1.0.1.post2


title: PyTorch: wrapping multiple records in one file?

body:
Is there a standard way of encoding multiple records (in this case, data from multiple .png or .jpeg images) in one file that PyTorch can read? Something similar to TensorFlow's "TFRecord" or MXNet's "RecordIO", but for PyTorch.

I need to download image data from S3 for inference, and it's much slower if my image data is in many small .jpg files rather than fewer files.

Thanks.


title: Is there any tool to profile each layer of a DNN in Pytorch and Tensorflow?

body:
I want to know the execution time of each layer of a DNN.
Is there any tool can profile each layer?
I know I can insert some print in a model, but I am not sure it will be accurate.


title: Beginner PyTorch : RuntimeError: size mismatch, m1: [16 x 2304000], m2: [600 x 120]

body:
I'm a beginner with PyTorch and building NNs in general and I'm kinda stuck.

I have this CNN architecture:

class ConvNet(nn.Module):

    def __init__(self, num_classes=10):
        super(ConvNet, self).__init__()

        self.conv1 = nn.Conv2d(
            in_channels=3,
            out_channels=16,
            kernel_size=3)

        self.conv2 = nn.Conv2d(
            in_channels=16,
            out_channels=24,
            kernel_size=4)

        self.conv3 = nn.Conv2d(
            in_channels=24,
            out_channels=32,
            kernel_size=4)

        self.dropout = nn.Dropout2d(p=0.3)

        self.pool = nn.MaxPool2d(2)

        self.fc1 = nn.Linear(600, 120)
        self.fc2 = nn.Linear(512, 10)

        self.final = nn.Softmax(dim=1)

    def forward(self, x):

        # conv 3 layers

        x = F.max_pool2d(F.relu(self.conv1(x)), 2)  # output of conv layers
        x = self.dropout(x)

        x = F.max_pool2d(F.relu(self.conv2(x)), 2)  # output of conv layers
        x = self.dropout(x)

        x = F.max_pool2d(F.relu(self.conv3(x)), 2)  # output of conv layers
        x = self.dropout(x)

        # linear layer

        x = F.interpolate(x, size=(600, 120))
        x = x.view(x.size(0), -1)

        x = self.fc1(x)

        return x

But when I try to train with my images, it doesn't work and I have this error:

RuntimeError: size mismatch, m1: [16 x 2304000], m2: [600 x 120]

I would like to add a second linear layer (self.fc2) as well as a final SoftMax layer (self.final) but since I'm stuck at the first linear layer I cannot make any progress.


title: LSTM in PyTorch Classifying Names

body:
I am trying the example presented in https://pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial.html but I am using a LSTM model instead of a RNN. The dataset is composed by different names (of different sizes) and their corresponding language (total number of languages is 18), and the objective is to train a model that given a certain name outputs the language it belongs to.

My problems right now are:

How to deal with variable size names, i.e. Hector and Kim, in the LSTM
A whole name (secuence of character) is processed every time in the LSTM so the output of the softmax function has shape (#characters of name, #target classes) but I would like just to obtain (1,#target of classes) in order to decide each name to which class does it correspond to. I have tried to just get the last row but results are very bad.

class LSTM(nn.Module):

    def __init__(self, embedding_dim, hidden_dim, vocab_size, tagset_size):
        super(LSTM, self).__init__()
        self.hidden_dim = hidden_dim

        self.word_embeddings = nn.Embedding(vocab_size, embedding_dim)

        # The LSTM takes word embeddings as inputs, and outputs hidden states
        # with dimensionality hidden_dim.
        self.lstm = nn.LSTM(embedding_dim, hidden_dim)

        # The linear layer that maps from hidden state space to tag space
        self.hidden2tag = nn.Linear(hidden_dim, tagset_size)
        self.softmax = nn.LogSoftmax(dim = 1)

    def forward(self, word):
        embeds = self.word_embeddings(word)
        lstm_out, _ = self.lstm(embeds.view(len(word), 1, -1))
        tag_space = self.hidden2tag(lstm_out.view(len(word), -1))
        tag_scores = self.softmax(tag_space)
        return tag_scores

    def initHidden(self):
        return Variable(torch.zeros(1, self.hidden_dim))

    lstm = LSTM(n_embedding_dim,n_hidden,n_characters,n_categories)
    optimizer = torch.optim.SGD(lstm.parameters(), lr=learning_rate)
    criterion = nn.NLLLoss()
    def train(category_tensor, line_tensor):
        # i.e. line_tensor = tensor([37,  4, 14, 13, 19,  0, 17,  0, 10,  8, 18]) and category_tensor = tensor([11])
        optimizer.zero_grad()
        output = lstm(line_tensor)

        loss = criterion(output[-1:], category_tensor) # VERY BAD
        loss.backward()

        optimizer.step()

        return output, loss.data.item()

Where line_tensor  is of variable size (depending the size of each name) and is a mapping between character and their index in the dictionary


title: How to optimize pip imports for Dockerfile layers caching

body:
I have a Dockerfile for ML/DL stack that needs a lot of requirements that could be logically split into python standard libraries and python ml libraries at least:

Python libraries (requirements.txt):

Cython
python-dateutil==2.8.0
setuptools>=41.0.0
progressbar2
argparse
smart_open
backoff
boto3
botocore
google
protobuf
tornado==5.1.1

Python ml libraries (requirements.lib.txt):

numpy==1.15.1
intel-numpy
matplotlib
pandas
scipy==1.2.1
scikit-learn==0.21.3
torch
tensorflow==1.14.0
keras==2.1.1

Now when I build my Docker image it turns out that the whole image size is ~5-7GB.
The other point is that I have, in terms of size in MB, small layers (< 100MB), big layers (~100MB-500MB) and huge layers (>500MB up to 1-2GB).

Of course ml python libraries cached to layers do not help since torch itself is about 800MB, tensorflow is ~500MB, intel mkl is about 300MB, etc.

Currently to prevent packages version override I do like

COPY $ROOT_APPLICATION/src/requirements.txt /tmp/requirements.txt
RUN cat /tmp/requirements.txt | xargs -n 1 -L 1 pip3 install

COPY $ROOT_APPLICATION/src/requirements.lib.txt /tmp/requirements.lib.txt
RUN cat /tmp/requirements.lib.txt | xargs -n 1 -L 1 pip3 install

where I do a copy to /tmp before RUN for caching reasons.
To keep everything compact I start from the good python slim buster:

FROM python:3.7.4-slim-buster

while the missing part of the Dockerfile is the apt-get of required libraries like:

########################################  BASE SYSTEM
RUN echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections
RUN apt-get update && apt-get install -y --no-install-recommends \
    software-properties-common \
    build-essential \
    pkg-config \
    libjemalloc2 \
    libgmp3-dev \
    libicu-dev \
    python3.7-icu \
    swig \
    curl \
    unzip \
    cron \
    jq

######################################## AUDIO
RUN apt-get update && apt-get install -y \
    libyaml-0-2 \
    libfftw3-3 \
    libtag1v5 \
    libsamplerate0 \
    libasound-dev \
    portaudio19-dev \
    libportaudio2 \
    libportaudiocpp0 \
    ffmpeg \
    espeak

How to optimize layers caching to 1) keep the docker images as small as possible 2) optimize layer's size for bandwidth when using docker push (then avoiding to push to registry huge layers over the network)?


title: Can auto-encoder encode new vector without re-training afresh?

body:
Here is a simple autoencoder to encode 3 vectors of dimension 1x3 : [1,2,3],[1,2,3],[100,200,500] to 1x1 :

epochs = 1000
from pylab import plt
plt.style.use('seaborn')
import torch.utils.data as data_utils
import torch
import torchvision
import torch.nn as nn
from torch.autograd import Variable

cuda = torch.cuda.is_available()
FloatTensor = torch.cuda.FloatTensor if cuda else torch.FloatTensor
import numpy as np
import pandas as pd
import datetime as dt

features = torch.tensor(np.array([ [1,2,3],[1,2,3],[100,200,500] ]))

print(features)

batch = 1
data_loader = torch.utils.data.DataLoader(features, batch_size=2, shuffle=False)

encoder = nn.Sequential(nn.Linear(3,batch), nn.Sigmoid())
decoder = nn.Sequential(nn.Linear(batch,3), nn.Sigmoid())
autoencoder = nn.Sequential(encoder, decoder)

optimizer = torch.optim.Adam(params=autoencoder.parameters(), lr=0.001)

encoded_images = []
for i in range(epochs):
    for j, images in enumerate(data_loader):
    #     images = images.view(images.size(0), -1)
        images = Variable(images).type(FloatTensor)
        optimizer.zero_grad()
        reconstructions = autoencoder(images)
        loss = torch.dist(images, reconstructions)
        loss.backward()
        optimizer.step()

#     encoded_images.append(encoder(images))

# print(decoder(torch.tensor(np.array([1,2,3])).type(FloatTensor)))

encoded_images = []
for j, images in enumerate(data_loader):
    images = images.view(images.size(0), -1)
    images = Variable(images).type(FloatTensor)

    encoded_images.append(encoder(images))

The variable encoded_images is an array of size 3 where each array entry represents the reduced dimensionality of a feature array :

[tensor([[0.9972],
         [0.9972]], grad_fn=<SigmoidBackward>),
 tensor([[1.]], grad_fn=<SigmoidBackward>)]

In order to determine similarity of a new feature, for example [1,1,1] is it required to re-train the network or can the existing trained network configuration/weights be "bootstrapped" such that the new vector can be encoded without require to retrain the network afresh ?


title: How does the Transformer Model Compute Self Attention?

body:
In the transformer model, https://arxiv.org/pdf/1706.03762.pdf there is self-attention which is computed using softmax on Query (Q) and Key (K) vectors:

I am trying to understand the matrix multiplications:

Q = batch_size x seq_length x embed_size

K = batch_size x seq_length x embed_size

QK^T = batch_size x seq_length x seq_length

Softmax QK^T = Softmax (batch_size x seq_length x seq_length)

How is the softmax computed since there are seq_length x seq_length values per batch element?

A reference to Pytorch computation will be very helpful.

Cheers!


title: How to use list as index with pytorch

body:
For example, a 2d tensor:

>>> t = torch.tensor([[1, 2, 3], [4, 5, 6], [7, 8, 9]])

And a list l = [0, 1]

If I execute t[l], then it ends up printing the 0th and first line of t.

But what if I want to use l as an index? I expect to use l to find the element at 0th row and 1th column. In other words, I expect the same result as t[0, 1] or t[0][1].

And I want to use it in more than 2d dimensions as well. Using l with length n as an index to track elements in n dimensions tensor.


title: Bert + Resnet joint learning, pytorch model is empty after instantiation

body:
I'm writing a simple joint model, which has two branches, one branch is a resnet50 another one is a bert. I concatenate the two outputs and pass that to a simple linear layer with 2 output neurons.

I implemented the following model :

import torch
from torch import nn
import torchvision.models as models
import torch.nn as nn
from collections import OrderedDict
from transformers import BertModel

class BertResNet(nn.Module):
    def __init__(self):
        super(BertResNet, self).__init__()
        # resnet
        resnet50 = models.resnet50(pretrained=True)
        n_inputs = resnet50.fc.in_features
        # compressed embedding space
        classifier = nn.Sequential(OrderedDict([
            ('fc1', nn.Linear(n_inputs, 512))
        ]))

        resnet50.fc = classifier # 512 out resnet

        bert = BertModel.from_pretrained('bert-base-uncased')

        # final classification layer

        classification = nn.Linear(512 + 768, 2)
        #print(resnet50)
        #print(bert)

    def forward(self, img, text):
        res_emb = self.resnet50(img)
        bert_emb = self.bert(text)

        combined = torch.cat(res_emb,
                              bet_emb, dim=1)
        out = self.classification(combined)
        return out

But when I instantiate, I get an empty model:

bert_resnet = BertResNet()

print(bert_resnet)

Out:
BertResNet()

list(bert_resnet.parameters()) also returns []


title: SiamMask on Jetson Xavier NX, pytorch, slow FPS

body:
I am running SiamMask (https://github.com/foolwood/SiamMask) on a Jetson Xavier NX, using pytorch.
I currently am getting a frame rate of about 5 FPS, I was hoping for much faster.
I know the pytorch would be slower than tensorRT but I was still expecting a much faster rate. It seems like it may only be using the CPU, not GPU.
Could anyone help me with this or give me some insight?
Thanks!


title: PyTorch arguments not valid on android

body:
I want to use this model in my android app. But when I start the app it falls with an error. The model works fine on my PC.
To Reproduce
Steps to reproduce the behavior:

Clone repository and use instructions in readme to run the model.
Add code below to save the model

            traced_script_module = torch.jit.trace(i2d, data)
            traced_script_module.save("i2d.pt")

I used PyTorch Android DemoApp link to run the model on android.

Error:
 E/AndroidRuntime: FATAL EXCEPTION: ModuleActivity
    Process: com.hypersphere.depthvisor, PID: 4765
    com.facebook.jni.CppException:
    Arguments for call are not valid.
    The following variants are available:

      aten::upsample_bilinear2d(Tensor self, int[2] output_size, bool align_corners) -> (Tensor):
      Expected at most 3 arguments but found 5 positional arguments.

      aten::upsample_bilinear2d.out(Tensor self, int[2] output_size, bool align_corners, *, Tensor(a!) out) -> (Tensor(a!)):
      Argument out not provided.

    The original call is:
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\site-packages\torch\nn\functional.py(3013): interpolate
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\site-packages\torch\nn\functional.py(2797): upsample
    <ipython-input-1-e1d92bec6901>(75): _upsample_add
    <ipython-input-1-e1d92bec6901>(89): forward
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\site-packages\torch\nn\modules\module.py(534): _slow_forward
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\site-packages\torch\nn\modules\module.py(548): __call__
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\site-packages\torch\jit\__init__.py(1027): trace_module
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\site-packages\torch\jit\__init__.py(875): trace
    <ipython-input-12-19d2ccccece4>(16): <module>
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\site-packages\IPython\core\interactiveshell.py(3343): run_code
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\site-packages\IPython\core\interactiveshell.py(3263): run_ast_nodes
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\site-packages\IPython\core\interactiveshell.py(3072): run_cell_async
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\site-packages\IPython\core\async_helpers.py(68): _pseudo_sync_runner
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\site-packages\IPython\core\interactiveshell.py(2895): _run_cell
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\site-packages\IPython\core\interactiveshell.py(2867): run_cell
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\site-packages\ipykernel\zmqshell.py(536): run_cell
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\site-packages\ipykernel\ipkernel.py(300): do_execute
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\site-packages\tornado\gen.py(209): wrapper
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\site-packages\ipykernel\kernelbase.py(545): execute_request
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\site-packages\tornado\gen.py(209): wrapper
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\site-packages\ipykernel\kernelbase.py(268): dispatch_shell
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\site-packages\tornado\gen.py(209): wrapper
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\site-packages\ipykernel\kernelbase.py(365): process_one
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\site-packages\tornado\gen.py(748): run
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\site-packages\tornado\gen.py(787): inner
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\site-packages\tornado\ioloop.py(743): _run_callback
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\site-packages\tornado\ioloop.py(690): <lambda>
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\asyncio\events.py(88): _run
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\asyncio\base_events.py(1786): _run_once
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\asyncio\base_events.py(541): run_forever
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\site-packages\tornado\platform\asyncio.py(149): start
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\site-packages\ipykernel\kernelapp.py(597): start
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\site-packages\traitlets\config\application.py(664): launch_instance
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\site-packages\ipykernel_launcher.py(16): <module>
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\runpy.py(85): _run_code
    D:\ProgramData\Anaconda\envs\ml3 torch\lib\runpy.py(193): _run_module_as_main
    Serialized   File "code/__torch__/___torch_mangle_907.py", line 39
        _17 = ops.prim.NumToTensor(torch.size(_16, 2))
        _18 = ops.prim.NumToTensor(torch.size(_16, 3))
2020-06-29 23:50:09.536 4765-4872/com.hypersphere.depthvisor E/AndroidRuntime:     _19 = torch.upsample_bilinear2d(_15, [int(_17), int(_18)], False, None, None)
              ~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
        input = torch.add(_19, _16, alpha=1)
        _20 = (_6).forward(input, )

        at org.pytorch.NativePeer.initHybrid(Native Method)
        at org.pytorch.NativePeer.<init>(NativePeer.java:18)
        at org.pytorch.Module.load(Module.java:23)
        at com.hypersphere.depthvisor.MainActivity.analyzeImage(MainActivity.java:56)
        at com.hypersphere.depthvisor.MainActivity.analyzeImage(MainActivity.java:21)
        at com.hypersphere.depthvisor.AbstractCameraXActivity.lambda$setupCameraX$2$AbstractCameraXActivity(AbstractCameraXActivity.java:86)
        at com.hypersphere.depthvisor.-$$Lambda$AbstractCameraXActivity$KgCZmrRflavSsq5aSHYb53Fi-P4.analyze(Unknown Source:2)
        at androidx.camera.core.ImageAnalysisAbstractAnalyzer.analyzeImage(ImageAnalysisAbstractAnalyzer.java:57)
        at androidx.camera.core.ImageAnalysisNonBlockingAnalyzer$1.run(ImageAnalysisNonBlockingAnalyzer.java:135)
        at android.os.Handler.handleCallback(Handler.java:873)
        at android.os.Handler.dispatchMessage(Handler.java:99)
        at android.os.Looper.loop(Looper.java:214)
        at android.os.HandlerThread.run(HandlerThread.java:65)

Environment
PyTorch version: 1.5.0
Is debug build: No
CUDA used to build PyTorch: Could not collect

OS:  Windows 10 Pro
GCC version: Could not collect
CMake version: Could not collect

Python version: 3.7
Is CUDA available: No
CUDA runtime version: 10.2.89
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: Could not collect

Versions of relevant libraries:
[pip3] numpy==1.18.5
[pip3] torch==1.5.0
[pip3] torchvision==0.6.0
[conda] _pytorch_select           0.1                       cpu_0
[conda] blas                      1.0                         mkl
[conda] cudatoolkit               10.2.89              h74a9793_1
[conda] libmklml                  2019.0.5                      0
[conda] mkl                       2019.4                      245
[conda] mkl-service               2.3.0            py37hb782905_0
[conda] mkl_fft                   1.1.0            py37h45dec08_0
[conda] mkl_random                1.1.0            py37h675688f_0
[conda] numpy                     1.18.5           py37h6530119_0
[conda] numpy-base                1.18.5           py37hc3f5095_0
[conda] pytorch                   1.5.0           cpu_py37h9f948e0_0
[conda] torchvision               0.6.0                py37_cu102    pytorch

Android Studio 4.0
Device: Samsung s8 plus
Android version: 9


title: PyTorch custom model Android

body:
I am trying to replace the model provided to the PyTorch HelloWorld example as provided here, with my custom trained model. The custom trained model is hosted here. I export the model using the following script:
import torch.nn
import stable_hopenetlite

model = model = stable_hopenetlite.shufflenet_v2_x1_0()
model.load_state_dict(torch.load('model/shuff_epoch_120.pkl', map_location="cpu"), strict=False)
model.eval()
example = torch.randn(1, 3, 224, 224)
traced_script_module = torch.jit.trace(model, example)
traced_script_module.save("model.pt")

I copy the model.pt into the assets folder of the PyTorch HelloWorld example and run it but instead of getting a [1, 66] shaped Tensor, I'm getting a [1, 1000] shaped Tensor. Could anyone please explain where I'm going wrong or what do I need to do to get the correct output?


title: PyTorch linear regression model

body:
I have a multivariate linear regression problem in which each data point looks like this:
y_i = 3                             # Some integer between 0 and 20
X_i = [0.5, 80, 0.004, 0.5, 0.789]  # A 5 dimensional vector

I can train a simple linear model by using sklearn, something like:
from sklearn import linear_model
ols = linear_model.LinearRegression()
model = ols.fit(X, y)

This gets me an accuracy of ~55 % (a linear model is not suitable for the problem, but this is a baseline to demonstrate the feasibility of modelling the problem, and a way for me to learn PyTorch, having use TensorFlow previously).
When I try to train a linear model using PyTorch I am defining the model as:
class TwoLayerNet(torch.nn.Module):
    def __init__(self, D_in, D_out):

        super(TwoLayerNet, self).__init__()
        self.linear1 = torch.nn.Linear(D_in, D_out)

    def forward(self, x):

        y_pred = self.linear1(x)
        return y_pred

D_in, D_out = 5, 1
model = TwoLayerNet(D_in, D_out)

And training as:
epochs = 10
criterion = torch.nn.MSELoss(reduction='sum')
optimizer = torch.optim.SGD(model.parameters(), lr=1e-4)
for epoch in range(epochs):
    for n, batch in enumerate(batches):
        X = []
        y = []
        for values in batch:
            X.append(values[0])
            y.append(values[1])

        X = torch.from_numpy(np.asarray(X))
        y = torch.from_numpy(np.asarray(y))
        # Forward pass: Compute predicted y by passing x to the model
        optimizer.zero_grad()
        y_pred = model(X)
        # Compute and print loss
        loss = criterion(y_pred, y)
        if n % 100 == 99:
            print(n, loss.item())

        # Zero gradients, perform a backward pass, and update the weights.

        loss.backward()
        optimizer.step()

This is just some code from the PyTorch documentation which I have adjusted. The current set up only achieves ~25%, no where near the accuracy that I would expect from the linear model. Am I doing something incorrect in the model training wrt PyTorch?


title: pytorch: how to apply function over all cells of 4-d tensor

body:
I'm trying to apply a function over a 4-D tensor (I think about it as a 2-D matrix with a 2-D matrix in each cell) with the following dimensions: [N x N x N x N].
The apply function returns [1 x N] tensor, so after the apply function I'm expecting a tensor of the following dimensions: [N x N x 1 x N].
Example: let's define [4 x 4 x 4 x 4] tensor:
tensor_4d = torch.tensor([[[[1., 0., 0., 0.],
  [0., 0., 0., 0.],
  [0., 0., 0., 0.],
  [0., 0., 0., 0.]],
 [[1., 0., 0., 0.],
  [1., 0., 0., 0.],
  [0., 0., 0., 0.],
  [0., 0., 0., 0.]],
 [[1., 0., 0., 0.],
  [0., 0., 0., 0.],
  [0., 0., 0., 0.],
  [0., 0., 0., 0.]],
 [[1., 0., 0., 0.],
  [0., 0., 0., 0.],
  [0., 0., 0., 0.],
  [0., 0., 0., 0.]]],
[[[0., 0., 0., 0.],
  [0., 1., 0., 0.],
  [0., 0., 0., 0.],
  [0., 0., 0., 0.]],
 [[0., 0., 0., 0.],
  [0., 1., 0., 0.],
  [0., 0., 0., 0.],
  [0., 0., 0., 0.]],
 [[0., 0., 0., 0.],
  [0., 1., 0., 0.],
  [0., 0., 0., 0.],
  [0., 0., 0., 0.]],
 [[0., 0., 0., 0.],
  [0., 1., 0., 0.],
  [0., 0., 0., 0.],
  [0., 0., 0., 0.]]],
[[[0., 0., 1., 0.],
  [0., 0., 0., 0.],
  [0., 0., 1., 0.],
  [0., 0., 0., 0.]],
 [[0., 0., 0., 0.],
  [0., 0., 1., 0.],
  [0., 0., 1., 0.],
  [0., 0., 0., 0.]],
 [[0., 0., 0., 0.],
  [0., 0., 0., 0.],
  [0., 0., 1., 0.],
  [0., 0., 0., 0.]],
 [[0., 0., 0., 0.],
  [0., 0., 0., 0.],
  [0., 0., 1., 0.],
  [0., 0., 0., 0.]]],
[[[0., 0., 1., 0.],
  [0., 0., 0., 0.],
  [0., 0., 0., 1.],
  [0., 0., 0., 1.]],
 [[0., 0., 0., 0.],
  [0., 0., 1., 0.],
  [0., 0., 0., 1.],
  [0., 0., 0., 1.]],
 [[0., 0., 0., 0.],
  [0., 0., 0., 0.],
  [0., 0., 0., 1.],
  [0., 0., 0., 1.]],
 [[0., 0., 0., 0.],
  [0., 0., 0., 0.],
  [0., 0., 0., 0.],
  [0., 0., 0., 1.]]]], dtype=torch.float64)

lets look on tensor_4d at [3][0]:
tensor_4d[3][0]
tensor([[0., 0., 1., 0.],
        [0., 0., 0., 0.],
        [0., 0., 0., 1.],
        [0., 0., 0., 1.]], dtype=torch.float64)

this my apply function :
def apply_function(tensor_2d):
    eigenvalues, eigenvectors = torch.eig(input=tensor_2d, eigenvectors=True)
    return eigenvector[:, 2]

and this the result of the apply function:
apply_function(tensor_4d[3][0])
tensor([-1.0000e+00,  0.0000e+00, 4.0083e-292,  0.0000e+00],
       dtype=torch.float64)

so the apply_function works for each cell.
Next, I'm trying to use the apply_function with the whole matrix, and expecting each cell will contain the result of activating 'apply_function' for this cell. but, when using the apply function I'm getting the following error:
apply_function(tensor_4d)

Traceback (most recent call last):
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2019.2.3\helpers\pydev\_pydevd_bundle\pydevd_exec2.py", line 3, in Exec
    exec(exp, global_vars, local_vars)
  File "<input>", line 1, in <module>
  File "C:/Users/LiavB/PycharmProjects/RBC/Components/RBC_torch.py", line 41, in apply_function
    eigenvalues, eigenvectors = torch.eig(input=tensor_2d, eigenvectors=True)
RuntimeError: invalid argument 1: A should be 2 dimensional at ..\aten\src\TH/generic/THTensorLapack.cpp:206


title: Pytorch multiprocessing with shared memory causes matmul to be 30x slower (with only two processes)

body:
I am trying to improve the speed of my reinforcement learning algorithm by using multiprocessing to have multiple workers generating experience at the same time.  Each process just runs the forward pass of my neural net, no gradient computation is needed.
As I understand it, when passing Tensors and nn.Modules across process boundaries (using torch.multiprocessing.Queue or torch.multiprocessing.Pool), the tensor data is moved to shared memory, which shouldn't be any slower than non-shared memory.
However, when I run my multiprocess code with 2 processes (on an 8 core machine), I find that my pytorch operations become more than 30x slower, more than counteracting the speedup from running two processes simultaneously.
I profiled my application to find which operations specifically are slowing down.  I found that much of my time was spend in nn.functional.linear(), specifically on this line inside a Tensor.matmul call:
    output = input.matmul(weight.t())

I added a timer just to this specific matmul call, and I found that when one process is running, this operation takes less than 0.3 milliseconds, but when two processes are running, it takes more than 10 milliseconds. Note that in both cases the weight matrix has been put in shared memory and passed across process boundaries to a worker process, the only difference is that in the second case there are two worker processes instead of one.
For reference, the shapes of input and weight tensors are torch.Size([1, 24, 180]) and torch.Size([31, 180]), respectively.
What could be causing this drastic slowdown?  is there some subtlety to using torch multiprocessing or shared memory that is not mentioned in any of the documentation?  I feel like there must be some hidden lock that is causing contention here, because this drastic slowdown makes no sense to me.


title: Run Pytorch stacked model on Colab TPU

body:
I am trying to run this my model on Colab Multi core TPU but I really don't know how to do it. I tried this tutorial notebook but I got some error and I can't fix it but I think there is maybe simpler wait for to do it.
About my model:
class BERTModel(nn.Module):
    def __init__(self,...):
        super().__init__()
        if ...:
            self.bert_model = XLMRobertaModel.from_pretrained(...)   # huggingface XLM-R
        elif ...:
            self.bert_model = others_model.from_pretrained(...)   # huggingface XLM-R

        ... # some other model's parameters

    def forward(self,...):
        bert_input = ...
        output = self.bert_model(bert_input)

        ... # some function that process on output

    def other_function(self,...):
        # just doing some process on output. like concat layers's embedding and return ...

class MAINModel(nn.Module):
    def __init__(self,...):
        super().__init__()

        print('Using model 1')
        self.bert_model_1 = BERTModel(...)

        print('Using model 2')
        self.bert_model_2 = BERTModel(...)

        self.linear = nn.Linear(...)

    def forward(self,...):
        bert_input = ...
        bert_output = self.bert_model(bert_input)
        linear_output = self.linear(bert_output)

        return linear_output

Can you please tell me how to run a model like my model on Colab TPU? I used Colab PRO to make sure Ram memory is not a big problem. Thanks you so so much.


title: How to convert PyTorch graph to ONNX and then inference from OpenCV?

body:
I'm attempting to convert a PyTorch graph to ONNX via the torch.onnx.export function and then use the OpenCV functions blobFromImage, setInput, and forward to inference the converted graph.  I think I'm on the right track but I keep running into errors and there are very few helpful examples of how to do this that I could find.
I realize the general stack overflow policy is to post only relevant portions of code, however with the errors I'm getting it seems this is a case where the devil is in the details so I suspect I'll have to post a full example to make the cause of the errors clear.
Here is my training net (pretty standard for MNIST):
# MnistNet.py

# Net Layout:
# batchSize x 1 x 28 x 28
#     conv1 Conv2d(1, 6, 5)
# batchSize x 6 x 24 x 24
#     relu(x)
#     max_pool2d(x, kernel_size=2)
# batchSize x 6 x 12 x 12
#     conv2 Conv2d(6, 16, 5)
# batchSize x 16 x 8 x 8
#     relu(x)
#     max_pool2d(x, kernel_size=2)
# batchSize x 16 x 4 x 4
#     view(-1, 16 * 4 * 4)    Note: 16 * 4 * 4 = 256
# batchSize x 1 x 256
#     fc1 Linear(256, 120)
#     relu(x)
# batchSize x 1 x 120
#     fc2 Linear(120, 84)
#     relu(x)
# batchSize x 1 x 84
#     fc3 Linear(84, 10)
# batchSize x 1 x 10

import torch
import torch.nn as nn
import torch.nn.functional as F
import torchvision

class MnistNet(nn.Module):

    TRANSFORM = torchvision.transforms.Compose([
        torchvision.transforms.Resize((28, 28)),
        torchvision.transforms.ToTensor(),
        torchvision.transforms.Normalize([0.5], [0.5])
    ])

    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 6, 5)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(256, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)
    # end function

    def forward(self, x):
        x = F.max_pool2d(F.relu(self.conv1(x)), kernel_size=2)
        x = F.max_pool2d(F.relu(self.conv2(x)), kernel_size=2)
        x = x.view(-1, 256)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x
    # end function

# end class

Here is my training script (again, pretty standard for MNIST):
# 1_train.py

from MnistNet import MnistNet

import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import Dataset, DataLoader
import torchvision

from termcolor import colored

BATCH_SIZE = 64
NUM_EPOCHS = 10

GRAPH_NAME = 'MNIST.pt'

def main():
    trainDataset = torchvision.datasets.MNIST('built_in_mnist_download', train=True, transform=MnistNet.TRANSFORM, download=True)

    trainDataLoader = DataLoader(trainDataset, batch_size=BATCH_SIZE, shuffle=True)

    # declare net, loss function, and optimizer
    mnistNet = MnistNet()
    lossFunction = nn.CrossEntropyLoss()
    optimizer = optim.Adam(mnistNet.parameters())

    # get device (cuda or cpu)
    if torch.cuda.is_available():
        device = torch.device('cuda')
        print(colored('using cuda', 'green'))
    else:
        device = torch.device('cpu')
        print(colored('GPU does not seem to be available, using CPU', 'red'))
    # end if

    # set network to device
    mnistNet.to(device)

    # set network to train mode
    mnistNet.train()

    print('beginning training . . .')

    # for each epoch . . .
    for epoch in range(1, NUM_EPOCHS+1):

        # variables to calculate loss and accuracy within the epoch
        epochLosses = []
        epochAccuracies = []

        # for each batch . . .
        for i, element in enumerate(trainDataLoader):
            # break out the input images and labels, note these are Tensors
            inputImages, labels = element

            inputImages = inputImages.to(device)
            labels = labels.to(device)

            # clear gradients from the previous step
            optimizer.zero_grad()

            # get net output
            outputs = mnistNet(inputImages)
            # calculate loss
            loss = lossFunction(outputs, labels)
            # call backward() to compute gradients
            loss.backward()
            # update parameters using gradients
            optimizer.step()

            # append the current classification loss to the list of epoch losses
            epochLosses.append(loss.item())

            # calculate current classification accuracy

            # get the highest scoring classification for each prediction
            _, predictions = torch.max(outputs.data, 1)

            # number of labels and predictions should always be the same, log an error if this is not the case
            if labels.size(0) != predictions.size(0):
                print(colored('ERROR: labels.size(0) != predictions.size(0)', 'red'))
            # end if

            # determine the number of correct predictions for the current batch
            correctPredictions = 0
            for j in range(len(labels)):
                if predictions[j].item() == labels[j].item():
                    correctPredictions += 1
                # end if
            # end for

            # append the current batch accuracy to the list of accuracies
            epochAccuracies.append(correctPredictions / labels.size(0))
        # end for

        # calculate epoch loss and accuracy from the respective lists
        epochLoss = sum(epochLosses) / len(epochLosses)
        epochAccuracy = sum(epochAccuracies) / len(epochAccuracies)

        print('epoch ' + str(epoch) + ', epochLoss = ' + '{:.4f}'.format(epochLoss) +
              ', epochAccuracy = ' + '{:.4f}'.format(epochAccuracy * 100) + '%')
    # end for

    print('finished training')

    # save the model
    torch.save(mnistNet.state_dict(), GRAPH_NAME)

    print('saved graph as ' + str(GRAPH_NAME))

# end function

if __name__ == '__main__':
    main()

Here is my best attempt so far at a script to convert a saved graph from PyTorch to ONNX (I'm not sure if this is correct, I can at least say it runs without error):
# 3_convert_graph_to_onnx.py

from MnistNet import MnistNet

import torch

GRAPH_NAME = 'MNIST.pt'
ONNX_GRAPH_NAME = 'MNIST.onnx'

def main():

    net = MnistNet()
    net.load_state_dict(torch.load(GRAPH_NAME))

    net.eval()

    # make a dummy input with a batch size of 1, 1 channel, 28 x 28
    dummyInput = torch.randn(10, 1, 28, 28)

    torch.onnx.export(net, dummyInput, ONNX_GRAPH_NAME, verbose=True)

# end function

if __name__ == '__main__':
    main()

Here is my attempt to inference the ONNX graph with OpenCV (Note that PyTorch is included, but is only used to load the test MNIST dataset, and the images are converted to OpenCV format before inferencing):
# 4_onnx_opencv_inf.py

from MnistNet import MnistNet

import torchvision

import cv2
import numpy as np
from termcolor import colored

ONNX_GRAPH_NAME = 'MNIST.onnx'

def main():
    testDataset = torchvision.datasets.MNIST('built_in_mnist_download', train=False, transform=MnistNet.TRANSFORM, download=True)

    labels = [ '0', '1', '2', '3', '4', '5', '6', '7', '8', '9' ]

    net = cv2.dnn.readNetFromONNX(ONNX_GRAPH_NAME)

    # test on 3 images
    for i in range(3):
        # get PyTorch tensor image and ground truth index from dataset
        ptImage, gndTrIdx = testDataset[i]
        # convert to PIL image
        pilImage = torchvision.transforms.ToPILImage()(ptImage)
        # convert to OpenCV image, would convert RGB to BGR here if image was color
        openCvImage = np.array(pilImage)

        gndTr = labels[gndTrIdx]

        # can show OpenCV image here if desired
        # cv2.imshow('openCvImage', openCvImage)
        # cv2.waitKey()

        blob = cv2.dnn.blobFromImage(image=openCvImage, scalefactor=1.0/255.0, size=(64, 64))

        net.setInput(blob)
        preds = net.forward()

        predIdx = np.array(preds)[0].argmax()

        prediction = str(predIdx)
        if prediction == gndTr:
            print(colored('i = ' + str(i) + ', predIdx = ' + str(predIdx) + ', gndTrIdx = ' + str(gndTrIdx) + ', correct answer', 'green'))
        else:
            print(colored('i = ' + str(i) + ', predIdx = ' + str(predIdx) + ', gndTrIdx = ' + str(gndTrIdx) + ', incorrect answer', 'red'))
        # end if

    # end for

# end function

if __name__ == '__main__':
    main()

Currently this final script crashes with this error:
$ python3 4_onnx_opencv_inf.py
[ERROR:0] global /tmp/pip-req-build-99ib2vsi/opencv/modules/dnn/src/dnn.cpp (3441) getLayerShapesRecursively OPENCV/DNN: [Reshape]:(18): getMemoryShapes() throws exception. inputs=1 outputs=1/1 blobs=0
[ERROR:0] global /tmp/pip-req-build-99ib2vsi/opencv/modules/dnn/src/dnn.cpp (3447) getLayerShapesRecursively     input[0] = [ 1 16 13 13 ]
[ERROR:0] global /tmp/pip-req-build-99ib2vsi/opencv/modules/dnn/src/dnn.cpp (3451) getLayerShapesRecursively     output[0] = [ 1 256 ]
[ERROR:0] global /tmp/pip-req-build-99ib2vsi/opencv/modules/dnn/src/dnn.cpp (3457) getLayerShapesRecursively Exception message: OpenCV(4.4.0) /tmp/pip-req-build-99ib2vsi/opencv/modules/dnn/src/layers/reshape_layer.cpp:154: error: (-1:Backtrace) Can't infer a dim denoted by -1 in function 'computeShapeByReshapeMask'

Traceback (most recent call last):
  File "4_onnx_opencv_inf.py", line 54, in <module>
    main()
  File "4_onnx_opencv_inf.py", line 38, in main
    preds = net.forward()
cv2.error: OpenCV(4.4.0) /tmp/pip-req-build-99ib2vsi/opencv/modules/dnn/src/layers/reshape_layer.cpp:154: error: (-1:Backtrace) Can't infer a dim denoted by -1 in function 'computeShapeByReshapeMask'

I'm not really sure what to do next based on this error, can anybody please advise on this?  I suspect that I'm at least doing the procedure generally correctly and missing a few small details.


title: Transformer Model Output Nan Values in Pytorch

body:
2021-03-09
I trained my transformer models in pytrorch. In the first few batches, the loss calculation and gradient updates were all performing well. However, the output of the model turned out to be nan values after several iterations. I am confident that there are no flawed data in the dataset. Besides, it's not a classification problem, the labels are float numbers.

2021-03-10
Follow-up:
What an interesting story! When I ran this transformer model with a larger architecture (like 6 encoder layers, 8 heads, etc.). The NAN values disappeared. It seems that the gradient explosion only existed in tiny models.
Solutions:
I searched the Pytorch forum and Stackoverflow and found out the accurate reason for this NAN instance. First, since the NAN loss didn't appear at the very beginning. We can conclude that the model might be well defined. The cause might be the data or the training process. I ran torch.autograd.set_detect_anomaly(True) as told in https://discuss.pytorch.org/t/gradient-value-is-nan/91663/2. It returned that the RuntimeError: Function ‘StdBackward1’ returned nan values in its 0th output.
According to the similar question in https://discuss.pytorch.org/t/gradient-of-standard-deviation-is-nan/14713, I double-checked the output in each layer inside the transformer. Strangely, after dozens of iterations, the positional embedding layer outputted a vector full of zeros. As a result, the LayerNorm that does the normalization job cannot backward the loss well, since it calculated the standard deviations and the standard deviation has no gradient at zero (or you can say it's infinite)! The possible solution is to add x.std(unbiased=False) if you are using pytorch.
This's my encounter with the NAN loss and mse. I hope my experience can give some insights to you when you meet this circumstance!
Relative Questions: Deep-Learning Nan loss reasons


title: I need some explanations about GAN codes

body:
Here is github codes

for epoch in range(num_epoch):
for i, (img, _) in enumerate(dataloader):
    num_img = img.size(0)
    # =================train discriminator
    img = img.view(num_img, -1)
    real_img = Variable(img).cuda()
    real_label = Variable(torch.ones(num_img)).cuda()
    fake_label = Variable(torch.zeros(num_img)).cuda()

I don`t understand whats the torch.ones and torch.zeros in training codes.

Can anyone explain about this?


title: why loss jump up to nan with swa method?

body:
I use SWA methods to train model in pytorch.

SWA: https://pytorch.org/blog/stochastic-weight-averaging-in-pytorch/

my train code's loss jump up to nan immidiately.

model' loss jump up to nan

loss outputs is below.

1.loss: tensor(4.8463, device='cuda:0', grad_fn=)

2.loss: tensor(118317.8516, device='cuda:0', grad_fn=)

3.loss: tensor(5.7568e+22, device='cuda:0', grad_fn=)

4.loss: tensor(nan, device='cuda:0', grad_fn=)

without SWA methods,loss don't jump up.
is there any problems in train model's code with SWA method?

I would appreciate any advice,thank you.

#batch_size
batch_size=5

#DataLoader
train_dataloader=torch.utils.data.DataLoader(train_dataset,batch_size=batch_size,shuffle=True)
val_dataloader=torch.utils.data.DataLoader(val_dataset,batch_size=batch_size,shuffle=False)

#dict
dataloaders_dict={"train":train_dataloader,"val":val_dataloader}

#example outputs
#train :torch.Size([5, 25, 32, 32])
#target : torch.Size([5])

def train_model_withSWA(net,dataloaders_dict,criterion,optimizer,num_epochs):

    loss_list=[]
    acc_list=[]

    #validation list
    val_loss_list=[]
    val_acc_list=[]

    for epoch in tqdm(range(num_epochs)):
        print("Epoch{}/{}".format(epoch+1,num_epochs))
        print("--------------------------")

        for phase in ["train","val"]:
            if phase=="train":
                net.train()
            else:
                net.eval()

            epoch_loss=0.0
            epoch_corrects=0

            #if (epoch==0) and (phase=="train"):
                continue

            for inputs,labels in dataloaders_dict[phase]:
                #optimizerを初期化：
                optimizer.zero_grad()

                with torch.set_grad_enabled(phase=="train"):

                    inputs=inputs.to(device)

                    labels=labels.to(device)

                    outputs=net(inputs)

                    loss=criterion(outputs,labels)
                    print("loss:",loss)

                    _,preds=torch.max(outputs,1)

                    if phase == "train":
                        loss.backward()
                        optimizer.step()

                    epoch_loss += loss.item()*inputs.size(0)
                    epoch_corrects +=torch.sum(preds==labels.data)

            #for swa
            optimizer.swap_swa_sgd()
            epoch_loss=epoch_loss/len(dataloaders_dict[phase].dataset)
            epoch_acc=epoch_corrects.double()/len(dataloaders_dict[phase].dataset)
            print("{} Loss:{:.4f} Acc:{:.4f}".format(phase,epoch_loss,epoch_acc))

            if phase=="train":
                loss_list.append(epoch_loss.detach().numpy())
                acc_list.append(epoch_acc.detach().numpy())
            else:
                val_loss_list.append(epoch_loss.detach().numpy())
                val_acc_list.append(epoch_acc.detach().numpy())

from torchcontrib.optim import SWA

# ignore warning
import warnings
warnings.filterwarnings('ignore') # set to ignore

#criterion
criterion = nn.CrossEntropyLoss()
net=net.to(device)

base_opt = torch.optim.SGD(net.parameters(), lr=0.1)
optimizer = SWA(base_opt, swa_start=10, swa_freq=5, swa_lr=0.05)

### train with SWA
train_model_withSWA(net=net,dataloaders_dict=dataloaders_dict,
                    criterion=criterion,
                    optimizer=optimizer,
                    num_epochs=num_epochs)

#model' loss jump up to nan....

#loss: tensor(4.8463, device='cuda:0', grad_fn=<NllLossBackward>)
#loss: tensor(118317.8516, device='cuda:0', grad_fn=<NllLossBackward>)
#loss: tensor(5.7568e+22, device='cuda:0', grad_fn=<NllLossBackward>)
#loss: tensor(nan, device='cuda:0', grad_fn=<NllLossBackward>)

# In additional.
I use Self Attention and Positional Encoder code

class Self_Attention(nn.Module):
    """ Self-Attention Layer"""

    def __init__(self, in_dim):
        super(Self_Attention, self).__init__()

        #pointwise convolution
        self.query_conv = nn.Conv2d(
            in_channels=in_dim, out_channels=in_dim, kernel_size=1)
        self.key_conv = nn.Conv2d(
            in_channels=in_dim, out_channels=in_dim, kernel_size=1)
        self.value_conv = nn.Conv2d(
            in_channels=in_dim, out_channels=in_dim, kernel_size=1)

        # softmax
        self.softmax = nn.Softmax(dim=-2)

        #output = x +gamma*o
        # first:gamma=0
        self.gamma = nn.Parameter(torch.zeros(1))

    def forward(self, x):

        x=x.to(device)

        X = x

        #B,C',W,H→B,C',N
        proj_query = self.query_conv(X).view(
            X.shape[0], -1, X.shape[2]*X.shape[3])  # size：B,C',N
        proj_query = proj_query.permute(0, 2, 1)  # transpose
        proj_key = self.key_conv(X).view(
            X.shape[0], -1, X.shape[2]*X.shape[3])  # size：B,C',N

        # bmm
        S = torch.bmm(proj_query, proj_key)

        #
        attention_map_T = self.softmax(S)
        attention_map = attention_map_T.permute(0, 2, 1)

        # Self-Attention Map
        proj_value = self.value_conv(X).view(
            X.shape[0], -1, X.shape[2]*X.shape[3])  # size：B,C,N
        o = torch.bmm(proj_value, attention_map.permute(
            0, 2, 1))

        # Self-Attention Map
        o = o.view(X.shape[0], X.shape[1], X.shape[2], X.shape[3])
        out = x+self.gamma*o
        #print("gamma:",self.gamma)
        return out, attention_map

class PositionalEncoder(nn.Module):

    def __init__(self, d_model=300, max_seq_len=256):
        super(PositionalEncoder,self).__init__()

        self.d_model = d_model

        pe = torch.zeros(max_seq_len, d_model)

        # GPU
        device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
        pe = pe.to(device)

        for pos in range(max_seq_len):
            for i in range(0, d_model, 2):
                pe[pos, i] = math.sin(pos / (10000 ** ((2 * i)/d_model)))
                pe[pos, i + 1] = math.cos(pos /
                                          (10000 ** ((2 * (i + 1))/d_model)))

        #
        self.pe = pe.unsqueeze(0)

        #
        self.pe.requires_grad = False

    def forward(self, x):
        x=x.to(device)
        ret = math.sqrt(self.d_model)*x + self.pe
        return ret


title: Ran out of Ram while training LSTM

body:
I am kind of a beginner in RNNs, so I coded a LSTM architecture using Pytorch, but I always run out of RAM whenever I am in the 3rd epoch. I am already using a DataLoader and I tried to detach the gradient from the input tensor but it doesn't solve the problem out.

This is my training loop

writer = SummaryWriter()
criterion = nn.CrossEntropyLoss(reduction='mean', ignore_index = 0)
optimizer = optim.Adam(lstm.parameters(), lr = 1e-5)
gradient_clip = clip_grad_norm_(lstm.parameters(), max_norm = 5)

num_epochs = 20
epoch_loss = -1.0
loss = - 1

t = trange(num_epochs, desc= "Epoch loss",  leave=True)

for epoch in t:
    trainLoader = iter(DataLoader(dataset, batch_size = batch_size))

    tt = trange(len(trainLoader)-1,  desc= "Batch loss",  leave=True)

    for i in tt:

        text, embedding = next(trainLoader)

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize

        y = lstm.forward(embedding.transpose(1,0))

        labels = text.transpose(0,1)[1:].transpose(0,1).flatten()

        loss = criterion(y.reshape(-1, y.shape[-1]), labels)

        tt.set_description("Batch loss : %.4f" % loss)

        tt.refresh()

        loss.backward(retain_graph=True)

        optimizer.step()

        epoch_loss += loss

    epoch_loss = epoch_loss / (len(trainLoader) - 1)

    # Saving model
    save_date = datetime.now().strftime("%d%m%Y-%H:%M:%S")
    PATH = './save/lstm_model_'+save_date
    torch.save(lstm, PATH)

    # Updating progression bar

    t.set_description("Epoch loss : %.4f" % epoch_loss)

    t.refresh()

    # Plotting gradients histograms in Tensorboard

    writer.add_scalar('Text_generation_Loss/train', epoch_loss, epoch)

    for tag, parm in lstm.named_parameters():

        with torch.no_grad():

            writer.add_histogram(tag, parm.grad.data.cpu().numpy(), epoch)

    writer.flush()

print('Finished Training')

writer.close()

And this is the LSTM class that I built:

class LSTM(nn.Module):

    def __init__(self, in_size : int, hidden_size : int):
        super().__init__()
        self.in_size = in_size
        self.hidden_size = hidden_size
        self.W_fi = nn.Linear(in_size,hidden_size)
        self.W_fh = nn.Linear(hidden_size,hidden_size, bias=False)
        self.W_ii = nn.Linear(in_size,hidden_size)
        self.W_ih = nn.Linear(hidden_size,hidden_size, bias=False)
        self.W_Ci = nn.Linear(in_size,hidden_size)
        self.W_Ch = nn.Linear(hidden_size,hidden_size, bias=False)
        self.W_oi = nn.Linear(in_size,hidden_size)
        self.W_oh = nn.Linear(hidden_size,hidden_size, bias=False)
        self.sigmoid = nn.Sigmoid()
        self.tanh = nn.Tanh()

    def one_step(self, x, h, C):
        f_t = self.sigmoid(self.W_fi(x) + self.W_fh(h))
        i_t = self.sigmoid(self.W_ii(x) + self.W_ih(h))
        g_t = self.tanh(self.W_Ci(x) + self.W_Ch(h))
        C_t = torch.mul(f_t, C) + torch.mul(i_t, g_t)
        o_t = self.sigmoid(self.W_oi(x) + self.W_oh(h))
        h_t = torch.mul(o_t, self.tanh(C_t))
        return h_t, C_t

    def forward(self, X):
        h_out = []
        h = - torch.ones(X.shape[1], self.hidden_size)
        C = - torch.ones(X.shape[1], self.hidden_size)
        h_t, C_t = self.one_step(X[0], h, C)
        h_out.append(h_t)

        for i in range(1, X.shape[0]  - 1):
            h_t, C_t = self.one_step(X[i], h_t, C_t)
            h_out.append(h_t)
        h_out = torch.cat(h_out)

        return h_out #h_out.reshape(-1,batch_size,num_embeddings)

I already searched for a similar case but I wasn't able to find a solution


title: PyTorch - what is the reason to resize my image and how do I decide the best size?

body:
I have some malaria infected blood cell images (multiple blood cells per image) with details of:

num_features = 929
num_features_train = 743
num_features_test = 186
depth = 24
channels = 3
width = 1600
height = 1200

In the previous image classification I did on a malaria infected blood cell image datasets (single blood cell per image) the image details were width = 100, height = 101, depth = 24.  So resizing to 50x50 didn't seem an issue.

I need to resize this, obviously, but don't know how to choose the best size for resizing an image so large.  I can't find anything in my online searching talking about this.  Any advise/experience would be helpful and greatly appreciated.  Thanks!!

p.s. I already figured out that if I don't resize something this large I will get a memory error. Yeah, probably. :)
MemoryError: Unable to allocate 6.64 GiB for an array with shape (929, 1600, 1600, 3) and data type uint8

p.p.s. resized to 100x100 and still got memory error
       resized to 50x50 and it was OK

So, I guess my question is, doesn't reducing the size so much reduce the resolution?  So how does the convolutional layer filters do proper filtering if the resolution is reduced so drastically?


title: computation graph of setting weights in pytorch

body:
I need a clarification of code written for some function in FastAI2 library.

this is the code WeightDropout written in FastAI2 library.

 class WeightDropout(Module):
        "A module that warps another layer in which some weights will be replaced by 0 during training."

        def __init__(self, module, weight_p, layer_names='weight_hh_l0'):
            self.module,self.weight_p,self.layer_names = module,weight_p,L(layer_names)
            for layer in self.layer_names:
                #Makes a copy of the weights of the selected layers.
                w = getattr(self.module, layer)
                delattr(self.module, layer)
                self.register_parameter(f'{layer}_raw', nn.Parameter(w.data))
                setattr(self.module, layer, F.dropout(w.data, p=self.weight_p, training=False))
                if isinstance(self.module, (nn.RNNBase, nn.modules.rnn.RNNBase)):
                    self.module.flatten_parameters = self._do_nothing

        def _setweights(self):
            "Apply dropout to the raw weights."
            for layer in self.layer_names:
                raw_w = getattr(self, f'{layer}_raw')
                setattr(self.module, layer, F.dropout(raw_w.data, p=self.weight_p, training=self.training))

        def forward(self, *args):
            self._setweights()
            with warnings.catch_warnings():
                #To avoid the warning that comes because the weights aren't flattened.
                warnings.simplefilter("ignore")
                return self.module.forward(*args)

        def reset(self):
            for layer in self.layer_names:
                raw_w = getattr(self, f'{layer}_raw')
                setattr(self.module, layer,
    F.dropout(raw_w.data, p=self.weight_p, training=False))
            if hasattr(self.module, 'reset'): self.module.reset()

        def _do_nothing(self): pass

where above code randomly drops weights in weight matrix of hidden layers.I am primarily interested in ,

 def _setweights(self):
                "Apply dropout to the raw weights."
                for layer in self.layer_names:
                    raw_w = getattr(self, f'{layer}_raw')
                    setattr(self.module, layer, F.dropout(raw_w.data, p=self.weight_p, training=self.training))

my question is that, does this operation of changing weights is recorded in gradient computation.


title: how to change PyTorch maxpool (ceil mode) in TensorFlow 2.0?

body:
self.pool3 = nn.MaxPool2d(kernel_size=2, stride=2, ceil_mode=True)  # ceiling (not floor) here for even dims

How to change PyTorch's ceil_mode=True in TensorFlow 2.0?


title: How to create ND4J NDArrays on GPU in DeepLearning4J?

body:
Is there a way to create NDArrays in DL4J so that the operations are done on GPU ?
For example in pytorch,
cuda0 = torch.device('cuda:0')
x = torch.tensor([1., 2.], device=cuda0)

I cannot find a way to specify a back-end (cpu or gpu) if my system is equipped with a GPU ?


title: RuntimeError: Trying to backward through the graph a second time, but the saved intermediate results have already been freed

body:
I get the error like the title....and I found some answers so I try retain_graph=True, but it doesn't work. Maybe my code got another problems (it occurs in loss_actor.backward(retain_grah....))
q = torch.zeros(len(reward))
q_target = torch.zeros(len(reward))
for j, r in enumerate(reward):
    q_target[j] = self.critic_network(torch.transpose(next_state[j], 0, 1), self.actor_network(torch.transpose(next_state[j], 0, 1)).view(1, 1))
    q_target[j] = r + (done[j] * gamma * q_target[j]).detach()
    q[j] = self.critic_network(torch.transpose(state[j], 0, 1), action[j].view(1, 1))
loss_critic = F.mse_loss(q, q_target)
self.critic_optimizer.zero_grad()
loss_critic.backward()
self.critic_optimizer.step()

b = torch.zeros(len(reward))
for j, r in enumerate(reward):
    b[j] = self.critic_network(torch.transpose(state[j], 0, 1), self.actor_network(torch.transpose(state[j], 0, 1)).view(1, 1))
loss_actor = -torch.mean(b)
self.actor_optimizer.zero_grad()
loss_actor.backward(retain_graph=True)
self.actor_optimizer.step()


title: Implementing memory monger method for pytorch

body:
I want to implement memory monger method as described in this paper Training Deep Nets with Sublinear Memory Cost. The code that implements that method for mxnet framework can be found here.

My question is, this memory monger method is suitable for symbolic graphs like the ones in Theano and Tensorflow, but the graphs in pytorch are dynamic, so can this method be applied for pytorch ?


title: pytorch adjust only certain weights with missing data

body: I am writing code for a model that encounters missing data often. Is there an option in pytorch (or tensorflow, since I haven't written the code yet so either is fine) such that only gradients and weights of existing data are computed/updated? Effectively, this is taking a smaller neural net from a maximal neural net.


title: Gradient calculation not disabled in no_grad() PyTorch

body:
Why is the gradient calculation of y not disabled in the following piece of code?

x = torch.randn(3, requires_grad=True)
print(x.requires_grad)
print((x ** 2).requires_grad)
y = x**2
print(y.requires_grad)
with torch.no_grad():
    print((x ** 2).requires_grad)
    print(y.requires_grad)

Which gives the following output:

True
True
True
False
True


title: How to use Adam optimizer in code instead of L-BFGS-B for constrained optimization

body:
I was using old source code and was using Scipy's L-BFGS-B algorithm. I have inaccuracies with it, though.

I got the following instructions:
In general, if you're having issues with L-BFGS-B, you also might want to try projected Adam or some other similar algorithm. There is really nothing special about L-BFGS-B for solving that problem, it just was convenient at that time for me. You can start off from the optim.Adam code and after each step, clamp x to be within your bounds.

I was wondering how to change the following code to use ADAM instead of L-BFGS-B. I am unfamiliar with Scipy. Please help because I'm a beginner

def get_sharpness(data_loader, model, criterion, epsilon, manifolds=0):

  # extract current x0
  x0 = None
  for p in model.parameters():
    if x0 is None:
      x0 = p.data.view(-1)
    else:
      x0 = torch.cat((x0, p.data.view(-1)))
  x0 = x0.cpu().numpy()

  # get current f_x
  f_x0, _ = get_minus_cross_entropy(x0, data_loader, model, criterion)
  f_x0 = -f_x0
  logging.info('min loss f_x0 = {loss:.4f}'.format(loss=f_x0))

  # find the minimum
  if 0==manifolds:
    x_min = np.reshape(x0 - epsilon * (np.abs(x0) + 1), (x0.shape[0], 1))
    x_max = np.reshape(x0 + epsilon * (np.abs(x0) + 1), (x0.shape[0], 1))
    bounds = np.concatenate([x_min, x_max], 1)
    func = lambda x: get_minus_cross_entropy(x, data_loader, model, criterion, training=True)
    init_guess = x0
  else:
    warnings.warn("Small manifolds may not be able to explore the space.")
    assert(manifolds<=x0.shape[0])
    #transformer = rp.GaussianRandomProjection(n_components=manifolds)
    #transformer.fit(np.random.rand(manifolds, x0.shape[0]))
    #A_plus = transformer.components_
    #A = np.linalg.pinv(A_plus)
    A_plus = np.random.rand(manifolds, x0.shape[0])*2.-1.
    # normalize each column to unit length
    A_plus_norm = np.linalg.norm(A_plus, axis=1)
    A_plus = A_plus / np.reshape(A_plus_norm, (manifolds,1))
    A = np.linalg.pinv(A_plus)
    abs_bound = epsilon * (np.abs(np.dot(A_plus, x0))+1)
    abs_bound = np.reshape(abs_bound, (abs_bound.shape[0], 1))
    bounds = np.concatenate([-abs_bound, abs_bound], 1)
    def func(y):
      floss, fg = get_minus_cross_entropy(x0 + np.dot(A, y), data_loader, model, criterion, training=True)
      return floss, np.dot(np.transpose(A), fg)
    #func = lambda y: get_minus_cross_entropy(x0+np.dot(A, y), data_loader, model, criterion, training=True)
    init_guess = np.zeros(manifolds)

  #rand_selections = (np.random.rand(bounds.shape[0])+1e-6)*0.99
  #init_guess = np.multiply(1.-rand_selections, bounds[:,0])+np.multiply(rand_selections, bounds[:,1])

  minimum_x, f_x, d = sciopt.fmin_l_bfgs_b(func, init_guess, maxiter=10, bounds=list(bounds), disp=1, iprint=101)

And get_minus_cross_entropy is defined as

def get_minus_cross_entropy(x, data_loader, model, criterion, training=False):
  if type(x).__module__ == np.__name__:
    x = torch.from_numpy(x).float()
    x = x.cuda()
  # switch to evaluate mode
  model.eval()

  # fill vector x of parameters to model
  x_start = 0
  for p in model.parameters():
    psize = p.data.size()
    peltnum = 1
    for s in psize:
      peltnum *= s
    x_part = x[x_start:x_start+peltnum]
    p.data = x_part.view(psize)
    x_start += peltnum

  result, grads = forward(data_loader, model, criterion, 0,
                 training=training, optimizer=None)
  #print ('get_minus_cross_entropy {}!'.format(-result['loss']))
  return (-result['loss'], None if grads is None else grads.cpu().numpy().astype(np.float64))


title: Should I use `torch.nn.Softmax` for huge number of classes?

body:
I want to train Relevance-based Word Embedding. In the model described in the paper, there is a simple 3 layer neural network (input, hidden and output similar to one used in Word2Vec). The weight matrix between input layer and hidden layer is of shape |V| x 300 (where 300 is embedding dimensions), and the weight matrix between hidden layer and output is of 300 x |V|. The problem is with the softmax layer on the top of output layer. It is a well known fact that the complexity of the softmax is O(|V|), where V is the vocabulary size.

So, there are several approximations to the softmax in order to reduce the computation complexity. I'm aware about the negative sampling, and hierarchical softmax.

The author in the paper I linked clearly mentions that we can not use negative sampling to approximate the softmax here, and so I need to use some other method (there are reasons for this, and they are mentioned clearly in the paper)(author uses hierarchical softmax).

Does the builtin torch.nn.softmax function provide any optimisation guaranteees in terms of running time of the softmax? Should I use it assuming that it will run efficiently? If not, what are the other options here? I can not find the well tested hierarchical softmax implementation anywhere.

The vocabulary size is in the order of hundred thousand.


title: Pointer type behavior in pytorch

body:
Is there pointer type assignment in pytorch? What I would like to do is have the following (pseudo-code):
b=8
a->b
print(a) --> 8
b=10
print(a) --> 10


title: How to make my implementation of GRU fast as nn.GRU

body:
I’m trying to build a GRU network using simpler blocks of nn. While using nn.GRU works really fast (with GPU), my "manual" version is slow, even though it should also run on GPU. I’m trying to understand why that is - and if there's a way to optimize it.
Here’s the nn.GRU version:
def __init__(self, in_dim, h_dim, out_dim, n_layers, dropout=0):
        self.rnn = nn.GRU(in_dim, h_dim, n_layers)
        self.decoder = nn.Linear(h_dim, out_dim)
def forward_(self, input: Tensor, hidden_state: Tensor=None):
        output, hidden = self.rnn(input, hidden_state)
        layer_output = self.decoder(output)
        return layer_output, hidden

My "manual" version:
def __init__(self, in_dim, h_dim, out_dim, n_layers, dropout=0):
        params = dict()
        for i in range(n_layers):
            if i != 0:
                in_dim = h_dim
            params[f'L{i}W1'] = nn.Linear(in_dim, h_dim)
            params[f'L{i}W2'] = nn.Linear(h_dim, h_dim, bias=False)
            params[f'L{i}W3'] = nn.Linear(in_dim, h_dim)
            params[f'L{i}W4'] = nn.Linear(h_dim, h_dim, bias=False)
            params[f'L{i}W5'] = nn.Linear(in_dim, h_dim)
            params[f'L{i}W6'] = nn.Linear(h_dim, h_dim, bias=False)
        params['W7'] = nn.Linear(h_dim, out_dim)
        for param in params:
            self.add_module(param, params[param])
        self.params = params
def forward(self, input: Tensor, hidden_state: Tensor=None):
    if hidden_state is None:
            layer_states.append(torch.zeros(batch_size, self.h_dim, device=input.device))
    for i in range(self.n_layers):
        layer_input = layer_middle
        layer_middle = torch.zeros(batch_size, seq_len, self.h_dim, device=input.device)
        params = self.params
        for j in range(seq_len):
            x = layer_input[:, j, :]
            z = F.sigmoid(params[f'L{i}W1'](x) + params[f'L{i}W2'](layer_states[i]))
            r = F.sigmoid(params[f'L{i}W3'](x) + params[f'L{i}W4'](layer_states[i]))
            g = F.tanh(params[f'L{i}W5'](x) + r*params[f'L{i}W6'](layer_states[i]))
            layer_states[i] = z*layer_states[i] + (1-z)*g
            layer_middle[:, j, :] = layer_states[i]
        layer_middle = nn.Dropout(self.dropout)(layer_middle)
    layer_output = torch.zeros((batch_size, seq_len, self.out_dim), device=input.device)
    for j in range(seq_len):
        x = layer_middle[:, j, :]
        layer_output[:, j, :] = params['W7'](x)

Both work, but the 1st is much faster. This (is what I assume) is the implementation of the first. I think I'm doing pretty much the same. I also call .to(device) on my model, so that everything will run on GPU.
Any ideas what might be the cause?


title: PyTorch - Correctly Computing Neural Tangent Kernel (Jacobian per data point)

body: In PyTorch, I have a fixed dataset x of shape (batch size, input dimension) and a feedforward network network. If I call the output of the network yhat = network(x) of shape (batch size, output dimension), how do I compute the gradient of yhat with respect to network's parameters for each data point individually?


title: Iterator doesn't work with DataLoader on GPU

body:
I'm using PyTorch on Google Colab, I'm getting this error when Using GPU,
TypeError                                 Traceback (most recent call last)
<ipython-input-33-41cdbc758ecd> in <module>()
----> 1 dataiter= iter(trainloader)

TypeError: '_SingleProcessDataLoaderIter' object is not callable

but wen using normal CPU there is no Error.
My code:
%matplotlib inline
%config InlineBackend.figure_format = 'retina'

import torch
import numpy as np
from torchvision import datasets, transforms

from collections import  OrderedDict

from torch import nn
from torch import  optim
import torch.nn.functional as F
import helper

transform = transforms.Compose([transforms.ToTensor(),
     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

trainset= datasets.MNIST("MINIST_data/", download= True, train=True, transform=transform)
trainloader= torch.utils.data.DataLoader(trainset, batch_size= 64, shuffle=True)
dataiter= iter(trainloader)

Using enumerate instead of iter works with GPU but I don't know why, can someone explain the error to me and why it is happening !?


title: Parameters of my network on PyTorch are not updated

body:
I want to make an auto calibration system using PyTorch.
I try to deal with a homogeneous transform matrix as weights of neural networks.
I write a code referring to PyTorch tutorials, but my custom parameters are not updated after backward method is called.
When I print a 'grad' attribute of each parameter, it is a None.
My code is below. Is there anything wrong?
Please give any advise to me. Thank you.
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

import numpy as np

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.params = nn.Parameter(torch.rand(6))
        self.rx, self.ry, self.rz = self.params[0], self.params[1], self.params[2]
        self.tx, self.ty, self.tz = self.params[3], self.params[4], self.params[5]

    def forward(self, x):
        tr_mat = torch.tensor([[1, 0, 0, self.params[3]],
                                [0, 1, 0, self.params[4]],
                                [0, 0, 1, self.params[5]],
                                [0, 0, 0, 1]], requires_grad=True)

        rz_mat = torch.tensor([[torch.cos(self.params[2]), -torch.sin(self.params[2]), 0, 0],
                                [torch.sin(self.params[2]), torch.cos(self.params[2]), 0, 0],
                                [0, 0, 1, 0],
                                [0, 0, 0, 1]], requires_grad=True)

        ry_mat = torch.tensor([[torch.cos(self.params[1]), 0, torch.sin(self.params[1]), 0],
                                [0, 1, 0, 0],
                                [-torch.sin(self.params[1]), 0, torch.cos(self.params[1]), 0],
                                [0, 0, 0, 1]], requires_grad=True)

        rx_mat = torch.tensor([[1, 0, 0, 0],
                                [0, torch.cos(self.params[0]), -torch.sin(self.params[0]), 0],
                                [0, torch.sin(self.params[0]), torch.cos(self.params[0]), 0],
                                [0, 0, 0, 1]], requires_grad=True)

        tf1 = torch.matmul(tr_mat, rz_mat)
        tf2 = torch.matmul(tf1, ry_mat)
        tf3 = torch.matmul(tf2, rx_mat)

        tr_local = torch.tensor([[1, 0, 0, x[0]],
                                [0, 1, 0, x[1]],
                                [0, 0, 1, x[2]],
                                [0, 0, 0, 1]])
        tf_output = torch.matmul(tf3, tr_local)
        output = tf_output[:3, 3]
        return output

    def get_loss(self, output):
        pass

model = Net()

input_ex = np.array([[-0.01, 0.05, 0.92],
                    [-0.06, 0.03, 0.94]])

output_ex = np.array([[-0.3, 0.4, 0.09],
                        [-0.5, 0.2, 0.07]])
print(list(model.parameters()))

optimizer = optim.Adam(model.parameters(), 0.001)
criterion = nn.MSELoss()

for input_np, label_np in zip(input_ex, output_ex):
    input_tensor = torch.from_numpy(input_np).float()
    label_tensor = torch.from_numpy(label_np).float()
    output = model(input_tensor)

    optimizer.zero_grad()
    loss = criterion(output, label_tensor)
    loss.backward()
    optimizer.step()

print(list(model.parameters()))


title: Where is the definition of torch.embedding in PyTorch?

body:
I'm trying to understand how PyTorch creates embeddings and read the source code of torch.nn.functional.embedding github link.
The function returns the result of torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse).
Then, I try to understand the definition of torch.embedding but I can't find its source code in the GitHub repository. Where is it?


title: CUDA Assertion error when trying to train YOLOv3 on multiple GPUs (PyTorch)

body:
I am trying to use this YOLOv3 architecture to make it compatible with Fisheye images with rotated bounding boxes. I have written the code which works fine on single GPU but it is giving me 'RuntimeError: Caught RuntimeError in replica 0 on device 0' and 'RuntimeError: CUDA error: device-side assert triggered' when trying to use with Multiple GPUs.
I have 4 'GeForce GTX 1080 Ti' GPUs on my university server which I am trying to use remotely. I wish to access the first 2 GPUs for running this code.
I tried setting the ['CUDA_LAUNCH_BLOCKING'] = "1" but it is taking forever to load the data. I also tried setting num_workers=0 in data loader which I read on one of the PyTorch forum discussion but the problem still persists
I have made the following changes to the code: https://gitlab.informatik.uni-bremen.de/digipath/band-saurabh/-/tree/rotated
I am using following part of the code to run on multiple GPUs:
#os.environ['CUDA_LAUNCH_BLOCKING'] = "1"
os.environ['CUDA_VISIBLE_DEVICES'] = "0,1"
from torch.nn import DataParallel

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

os.makedirs("output", exist_ok=True)
os.makedirs("checkpoints", exist_ok=True)

# Get data configuration
data_config = parse_data_config(opt.data_config)
train_path = data_config["train"]
valid_path = data_config["valid"]
train_annpath = data_config["json_train"]
valid_annpath = data_config["json_val"]
class_names = load_classes(data_config["names"])

if len(class_names) == 80:
    class_80 = True
else:
    class_80 = False

# Initiate model
model = Darknet(opt.model_def).to(device)
model.apply(weights_init_normal)

# If specified we start from checkpoint
if opt.pretrained_weights:
    if opt.pretrained_weights.endswith(".pth"):
        model.load_state_dict(torch.load(opt.pretrained_weights))
    else:
        model.load_darknet_weights(opt.pretrained_weights)

#Use multiple GPUs
#model = DataParallel(model)

#Check if multiple GPUs available
if torch.cuda.device_count() > 1:
    print("Let's use", torch.cuda.device_count(), "GPUs!")
    # dim = 0 [30, xxx] -> [10, ...], [10, ...], [10, ...] on 3 GPUs
    model = DataParallel(model)```

Following is the exact error and its original traceback. I ould really use some help. Thanks in advance.

> /pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [0,0,0], thread: [93,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [0,0,0], thread: [94,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:84: operator(): block: [0,0,0], thread: [95,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
THCudaCheck FAIL file=/pytorch/aten/src/THC/THCCachingHostAllocator.cpp line=278 error=710 : device-side assert triggered
Traceback (most recent call last):
  File "/localdata/saurabh/.local/Python3.8/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/localdata/saurabh/.local/Python3.8/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/sband/.vscode-server/extensions/ms-python.python-2020.11.358366026/pythonFiles/lib/python/debugpy/__main__.py", line 45, in <module>
    cli.main()
  File "/home/sband/.vscode-server/extensions/ms-python.python-2020.11.358366026/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 430, in main
    run()
  File "/home/sband/.vscode-server/extensions/ms-python.python-2020.11.358366026/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 267, in run_file
    runpy.run_path(options.target, run_name=compat.force_str("__main__"))
  File "/localdata/saurabh/.local/Python3.8/lib/python3.8/runpy.py", line 265, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "/localdata/saurabh/.local/Python3.8/lib/python3.8/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/localdata/saurabh/.local/Python3.8/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/localdata/saurabh/yolov3/train.py", line 131, in <module>
    loss, outputs = model(imgs, targets)
  File "/localdata/saurabh/yolov3/yol/lib/python3.8/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/localdata/saurabh/yolov3/yol/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py", line 155, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/localdata/saurabh/yolov3/yol/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py", line 165, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/localdata/saurabh/yolov3/yol/lib/python3.8/site-packages/torch/nn/parallel/parallel_apply.py", line 85, in parallel_apply
    output.reraise()
  File "/localdata/saurabh/yolov3/yol/lib/python3.8/site-packages/torch/_utils.py", line 395, in reraise
    raise self.exc_type(msg)
RuntimeError: Caught RuntimeError in replica 0 on device 0.

Original Traceback (most recent call last):

> File "/localdata/saurabh/yolov3/yol/lib/python3.8/site-packages/torch/nn/parallel/parallel_apply.py",
line 60, in _worker
    output = module(*input, **kwargs)
  File "/localdata/saurabh/yolov3/yol/lib/python3.8/site-packages/torch/nn/modules/module.py", line 722,
 in _call_impl
    result = self.forward(*input, **kwargs)
  File "/localdata/saurabh/yolov3/models.py", line 287, in forward
    x, layer_loss = module[0](x, targets, img_dim)
  File "/localdata/saurabh/yolov3/yol/lib/python3.8/site-packages/torch/nn/modules/module.py", line 722,
 in _call_impl
    result = self.forward(*input, **kwargs)
  File "/localdata/saurabh/yolov3/models.py", line 205, in forward
    iou_scores, class_mask, obj_mask, noobj_mask, tx, ty, tw, th, tangle, tcls, tconf = build_targets(
  File "/localdata/saurabh/yolov3/utils/utils.py", line 405, in build_targets
    noobj_mask[b[i], anchor_ious > ignore_thres, gj[i], gi[i]] = 0
RuntimeError: CUDA error: device-side assert triggered


title: PyTorch alternative for tf.data.experimental.sample_from_datasets

body:
Suppose I have two datasets, dataset one with 100 items and dataset two with 5000 items.
Now I want that during training my model sees as much items from dataset one as from dataset two.
In Tensorflow I can do:
dataset = tf.data.experimental.sample_from_datasets(
    [dataset_one, dataset_two], weights=[50,1], seed=None
)

Is there an alternative in PyTorch that does the same?
I think this is not too difficult to implement by creating a custom dataset (not working example)
from torch.utils.data import Dataset

class SampleDataset(Dataset):
    def __init__(self, datasets, weights):
        self.datasets = datasets
        self.weights = weights

    def __len__(self):
        return sum([len(dataset) for dataset in self.datasets])

    def __getitem__(self, idx):
        # sample a random number and based on that sample an item

        return self.datasets[dataset_idx][sample_idx]

However, this seems quite common. Is there already something like this available?


title: Should I create a PyTorch Dataset to train a model off a pyspark dataframe?

body:
I want to train a PyTorch NLP model over training data in columnar format, and I thought to construct a PyTorch Dataset using as raw data a pyspark dataframe (not sure it's the right approach...).
To preprocess text I'm using a tokenizer provided by the transformers library and a tokenizing_UDF function to apply the tokenization.
The Dataset object is then fed to a DataLoader to train a ML model.
What I currently have is this:
import pandas as pd # ideally I'd like to get rid of pandas here
import torch
from torch.utils.data.dataset import Dataset
from transformers import BertTokenizer
from pyspark.sql import types as T
from pyspark.sql import functions as F

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

text = ["This is a test.", "This is not a test."]*100
label = [1, 0]*100

df = sqlContext.createDataFrame(zip(text, label), schema=['text', 'label'])
tokenizing_UDF = udf(lambda t: tokenizer.encode(t),  T.ArrayType(T.LongType()))
df = df.withColumn("tokenized", tokenizing_UDF(F.col("text"))) # not sure this is the right way
df = df.toPandas() # ugly

class TokenizedDataset(Dataset):
    """needs refactoring..."""
    def __init__(self, df):
        self.data = df

    def __getitem__(self, index):
        text = self.data.loc[index].tokenized
        text = torch.LongTensor(text)
        label = self.data.loc[index].label
        return (text, label)

    def __len__(self):
        count = len(self.data)
        return count

dataset = TokenizedDataset(df) # slow...

I currently invoke .toPandas() so that my TokenizedDataset can deal with a pandas dataframe.
Is this a sensible approach?
If so, how should I modify the TokenizedDataset code to handle pyspark dataframes directly?
If I am off track, should I use https://github.com/uber/petastorm instead?


title: How to solve this "RuntimeError: Failed to run torchinfo" when trying to get the summary of my SegNet architechture?

body:
Can anyone help me with this error?
I want to see the summary of my deeplearning architecture features which is SegNet that using PyTorch. I try the code that've been made by trypag from github (https://github.com/trypag/pytorch-unet-segnet). I use summary() from torchinfo made by TylerYep (https://github.com/TylerYep/torchinfo). It was fine with other architecture like vgg16, resnet50, even with the samples code in the repository. But when I try to use it with codes from trypag I get:
RuntimeError: Failed to run torchinfo. See above stack traces for more details. Executed layers up to:
[Encoder: 2-1, Sequential: 3-1, Conv2d: 4-1, BatchNorm2d: 4-2, ReLU: 4-3, Conv2d: 4-4, BatchNorm2d: 4-5,
ReLU: 4-6, Encoder: 2-2, Sequential: 3-2, Conv2d: 4-7, BatchNorm2d: 4-8, ReLU: 4-9, Conv2d: 4-10, BatchNorm2d: 4-11,
ReLU: 4-12, Encoder: 2-3, Sequential: 3-3, Conv2d: 4-13, BatchNorm2d: 4-14, ReLU: 4-15, Conv2d: 4-16,
BatchNorm2d: 4-17, ReLU: 4-18, Dropout: 4-19, Encoder: 2-4, Sequential: 3-4, Conv2d: 4-20, BatchNorm2d: 4-21,
ReLU: 4-22, Conv2d: 4-23, BatchNorm2d: 4-24, ReLU: 4-25,` Dropout: 4-26, Encoder: 2-5, Sequential: 3-5, Conv2d: 4-27,
BatchNorm2d: 4-28, ReLU: 4-29, Conv2d: 4-30, BatchNorm2d: 4-31, ReLU: 4-32, Dropout: 4-33, Decoder: 2-6,
Sequential: 3-6, Conv2d: 4-34, BatchNorm2d: 4-35, ReLU: 4-36, Conv2d: 4-37, BatchNorm2d: 4-38, ReLU: 4-39,
Dropout: 4-40, Decoder: 2-7, Sequential: 3-7, Conv2d: 4-41, BatchNorm2d: 4-42, ReLU: 4-43, Conv2d: 4-44,
BatchNorm2d: 4-45, ReLU: 4-46, Dropout: 4-47, Decoder: 2-8, Sequential: 3-8, Conv2d: 4-48, BatchNorm2d: 4-49,
ReLU: 4-50, Conv2d: 4-51, BatchNorm2d: 4-52, ReLU: 4-53, Dropout: 4-54, Decoder: 2-9, Sequential: 3-9, Conv2d: 4-55,
BatchNorm2d: 4-56, ReLU: 4-57, Conv2d: 4-58, BatchNorm2d: 4-59, ReLU: 4-60, Conv2d: 4-61]

This is the code that I use:
from torch import nn
import torch.nn.functional as F
import torch

class SegNet(nn.Module):
    def __init__(self, num_classes, in_features = 1, drop_rate = 0.5,
                 filter_config=(64, 128, 256, 512, 512)):
        super(SegNet, self).__init__()

        self.encoders = nn.ModuleList()
        self.decoders = nn.ModuleList()

        encoder_n_layers = (2, 2, 3, 3, 3)
        encoder_filter_config = (in_features,) + filter_config
        decoder_n_layers = (3, 3, 3, 2, 1)
        decoder_filter_config = filter_config[::-1] + (filter_config[0],)

        for i in range(0, 5):
            self.encoders.append(Encoder(encoder_filter_config[i],
                                         encoder_filter_config[i + 1],
                                         encoder_n_layers[i], drop_rate))

            self.decoders.append(Decoder(decoder_filter_config[i],
                                         decoder_filter_config[i + 1],
                                         decoder_n_layers[i], drop_rate))

        self.classifier = nn.Conv2d(filter_config[0],  num_classes, kernel_size = 3,
                                    stride = 1, padding = 1)

    def forward(self, x):
        indices = []
        unpool_sizes = []
        feat = x

        for i in range(0, 5):
            (feat, ind), size = self.encoders[i](feat)
            indices.append(ind)
            unpool_sizes.append(size)

        for i in range(0, 5):
            feat = self.decoders[i](feat, indices[4 - i], unpool_sizes[4 - i])

        return self.classifier(feat)

class Encoder(nn.Module):
    """

    A helper Module that performs ConvoBlock (Convolutional, BN, Activation).
    Maxpooling follows each ConvoBlock.

    """

    def __init__(self,
                 in_features: int,
                 out_features: int,
                 n_blocks: int = 2,
                 drop_rate = 0.5):
        super(Encoder, self).__init__()

        layers = [nn.Conv2d(in_features, out_features, kernel_size = 3,
                            stride = 1, padding = 1, bias = True),
                  nn.BatchNorm2d(out_features),
                  nn.ReLU(inplace = True)]

        if n_blocks > 1:
            layers += [nn.Conv2d(out_features, out_features, kernel_size = 3,
                                 stride = 1, padding = 1, bias = True),
                       nn.BatchNorm2d(out_features),
                       nn.ReLU(inplace = True)]
            if n_blocks == 3:
                layers += [nn.Dropout(drop_rate)]

        self.features = nn.Sequential(*layers)

    def forward(self, x):
        output = self.features(x)
        return F.max_pool2d(output, kernel_size = 2, stride = 2, return_indices =
                            True), output.size()

class Decoder(nn.Module):
    def __init__(self,
                 in_features: int,
                 out_features: int,
                 n_blocks: int = 2,
                 drop_rate = 0.5):

        super(Decoder, self).__init__()

        layers = [nn.Conv2d(in_features, out_features, kernel_size = 3,
                            stride = 1, padding = 1, bias = True),
                  nn.BatchNorm2d(out_features),
                  nn.ReLU(inplace = True)]

        if n_blocks > 1:
             layers += [nn.Conv2d(out_features, out_features, kernel_size = 3,
                                 stride = 1, padding = 1, bias = True),
                       nn.BatchNorm2d(out_features),
                       nn.ReLU(inplace = True)]
            if n_blocks == 3:
                 layers += [nn.Dropout(drop_rate)]

         self.features = nn.Sequential(*layers)

    def forward(self, x, indices, size):
        unpooled = F.max_unpool2d(x, indices = indices, kernel_size = 2,  stride = 2,
                                  padding = 0, output_size = size)
        return self.features(unpooled)

 device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

model = SegNet(num_classes = 2, in_features = 3).to(device)
x = torch.randn(size = (1, 3, 360, 480),
                dtype=torch.float32).to(device)
with torch.no_grad():
    out = model(x)

from torchinfo import summary

summary = summary(model, (32, 3, 360, 480))

This part of code is only for testing to display the summary:
model = SegNet(num_classes = 2, in_features = 3).to(device)
x = torch.randn(size = (1, 3, 360, 480),
                dtype=torch.float32).to(device)
with torch.no_grad():
    out = model(x)

from torchinfo import summary

summary = summary(model, (32, 3, 360, 480))

Should I trace the error by looking the code behind torchinfo summary() then revised my code? Or any other advice for me?
Any advice or solution will help me alot.
Thank You !


title: PRelue is not supperted with mmdnn?

body:
I converted my caffe model to IR successfully, error happened when I tried convert IR to pytorch:

Pytorch Emitter has not supported operator [PRelu]

How should I deal with that please?


title: Error Importing Pyro after source install for Anaconda release of Python

body:
I downloaded Pyro using source install instructions here under the Install from source link because I was always told not to use pip when installing to an Anaconda installation.  When I try import pyro I get the attribute error below which does not make sense

import pyro
Traceback (most recent call last):

  File "<ipython-input-48-05a8cb49cc0d>", line 1, in <module>
    import pyro

  File "C:\Users\bjwil\Anaconda3\Lib\site-packages\pyro\__init__.py", line 6, in <module>
    from pyro.primitives import (clear_param_store, enable_validation, get_param_store, iarange, irange, module, param,

  File "C:\Users\bjwil\Anaconda3\Lib\site-packages\pyro\primitives.py", line 10, in <module>
    import pyro.infer as infer

  File "C:\Users\bjwil\Anaconda3\Lib\site-packages\pyro\infer\__init__.py", line 14, in <module>
    from pyro.infer.traceenum_elbo import JitTraceEnum_ELBO, TraceEnum_ELBO

  File "C:\Users\bjwil\Anaconda3\Lib\site-packages\pyro\infer\traceenum_elbo.py", line 183, in <module>
    class BackwardSampleMessenger(pyro.poutine.messenger.Messenger):

AttributeError: module 'pyro.poutine' has no attribute 'messenger'

This doesn't make much sense because in the poutine folder I have the messenger file and it has the Class Messenger so line 183 class BackwardSampleMessenger(pyro.poutine.messenger.Messenger) in the traceenum_elbo.py file should be working.