title: Pyro - Likelihood function and sampling dimension

body:
I am learning Pyro and I find the dimensions confusing, despite the rich and detailed documentation

This is the sketch of my model:

DATA_SIZE = 1000

simulated_daily_demand = torch.distributions.Beta(torch.tensor(2.0), torch.tensor(2.0)).sample([DATA_SIZE,])

def model(SIMULATION_DAYS = 30):
    alpha = pyro.param("alpha", pyro.distributions.Uniform(0, 10))
    beta = pyro.param("beta", pyro.distributions.Uniform(0, 10))
    total_demand = 0
    for t in range(0, SIMULATION_DAYS):
        daily_demand = pyro.sample("daily_demand", pyro.distributions.Beta(alpha, beta), obs=simulated_daily_demand)
        total_demand = total_demand + daily_demand
    return total_demand

model()

I set here priors on the concentrations (alpha, beta).
My understanding is that calling pyro.sample with observations fits on the data - I suppose that it maximizes the likelihood of the concentrations given the data.

The output I get:

len(model())
C:\ProgramData\Anaconda3\lib\site-packages\pyro\primitives.py:85: RuntimeWarning: trying to observe a value outside of inference at daily_demand
  RuntimeWarning)
1000

The size()

The values I get seem to look good. The mean of simulated_daily_demand is roughly 0.5 and the mean of model() is ~15 which is ~30*0.5.
I don't get the size of the tensor.  I would have expected it to be of .size() torch.Size([1]).

I also noticed the warning. I suppose Pyro is complaining because it would like me to write a guide and run some inference on the parameters (eg SVI) before being able to sample from "daily demand". I am also wondering how I could the run the model after the inference of the latent concentrations.
A little sketch of the code would really help, thanks!

On a hindsight, I think I may have misunderstood the use of plates. Now, if I assume that the observations as independent, I would need to set up a plate.
Something like:

import torch
import pyro

NUM_RUNS = 5
DATA_SIZE = 1000

simulated_daily_demand = torch.distributions.Beta(torch.tensor(2.0), torch.tensor(2.0)).sample([DATA_SIZE,])

def model(hist_demand, START_INVENTORY = torch.tensor(100.0), SIMULATION_DAYS = 30):
    alpha = pyro.param("alpha", pyro.distributions.Uniform(0, 10))
    beta = pyro.param("beta", pyro.distributions.Uniform(0, 10))
    total_demand = 0
    for t in range(0, SIMULATION_DAYS):
        with pyro.plate("obs_loop"):
            daily_demand = pyro.sample("daily_demand", pyro.distributions.Beta(alpha, beta), obs=simulated_daily_demand)
        total_demand = total_demand + daily_demand
    return total_demand

total_demand_runs = []
for r in range(0, NUM_RUNS):
    total_demand_runs.append(model(simulated_daily_demand))

Which returns a nested list of size (NUM_RUNS, SIMULATION_DAYS) that contains a tensor of size DATA_SIZE. The elements (daily_demand) are the same across simulation days. Probably getting closer, but no cigar.

import torch
import pyro
import torch.distributions.constraints as constraints

NUM_RUNS = 5
SIMULATION_DAYS = 30
DATA_SIZE = 1000

simulated_daily_demand = torch.distributions.Beta(torch.tensor(2.0), torch.tensor(2.0)).sample([DATA_SIZE, SIMULATION_DAYS])

def model(hist_demand = None, START_INVENTORY = torch.tensor(100.0), SIMULATION_DAYS = SIMULATION_DAYS):
    alpha = pyro.param("alpha", pyro.distributions.Uniform(0, 10))
    beta = pyro.param("beta", pyro.distributions.Uniform(0, 10))
    with pyro.plate("obs_loop"):
        daily_demand_vector = pyro.sample("daily_demand", pyro.distributions.Beta(
            alpha * torch.ones([SIMULATION_DAYS]),
            beta * torch.ones([SIMULATION_DAYS])),
            obs=hist_demand
        )
    total_demand = 0
    for t in range(0, SIMULATION_DAYS):
        total_demand = total_demand + daily_demand_vector[t]
    return total_demand

def guide(hist_demand):
    alpha = pyro.param(
        "alpha",
        pyro.distributions.Normal(torch.tensor(2.0), torch.tensor(0.1)),
        constraint = constraints.positive
        )
    beta = pyro.param(
        "beta",
        pyro.distributions.Normal(torch.tensor(2.0), torch.tensor(0.1)),
        constraint = constraints.positive
        )
    return alpha, beta

from pyro.optim import Adam
adam_params = {"lr": 0.005, "betas": (0.95, 0.999)}
optimizer = Adam(adam_params)
svi = pyro.infer.SVI(model, guide, optimizer, loss=pyro.infer.Trace_ELBO())

n_steps = 5000
# do gradient steps
for step in range(n_steps):
    svi.step(simulated_daily_demand)

alpha_q = pyro.param("alpha").item()
beta_q = pyro.param("beta").item()

Something like this seems to make sense and seems to converge: the SVI spits out approximately right parameter values. Now, the question still holds - how can I run a simulation using the inferred alpha and beta?


title: How to convert Pytorch model parameters to long datatype?

body:
I was able to convert pytorch model parameters to float or double but not to long.

model = model.long()

gives error, while

model = model.float()

runs.

The error I get is:

'Net' object has no attribute 'long'


title: Can’t build pytorch from source on macOS 10.14 for CUDA support: “no member named ‘out_of_range’ in namespace ‘std’”

body:
I am following the instructions of the get started page of Pytorch site to build pytorch with CUDA support on mac OS 10.14 (Mojave) but I am getting an error:

[ 80%] Building CXX object
caffe2/CMakeFiles/torch.dir/__/torch/csrc/jit/passes/alias_analysis.cpp.o
In file included from /Users/adriantineo/dev/python/pytorch/torch/csrc/jit/passes/alias_analysis.cpp:1:
In file included from /Users/adriantineo/dev/python/pytorch/torch/csrc/jit/passes/alias_analysis.h:3:
/Users/adriantineo/dev/python/pytorch/c10/util/flat_hash_map.h:1367:24: error: no
      member named 'out_of_range' in namespace 'std'
            throw std::out_of_range("Argument passed to at() was not in the map.");
                  ~~~~~^
/Users/adriantineo/dev/python/pytorch/c10/util/flat_hash_map.h:1374:24: error: no
      member named 'out_of_range' in namespace 'std'
            throw std::out_of_range("Argument passed to at() was not in the map.");
                  ~~~~~^
[ 80%] Building CXX object caffe2/CMakeFiles/torch.dir/__/torch/csrc/jit/passes/batch_mm.cpp.o
[ 80%] Building CXX object caffe2/CMakeFiles/torch.dir/__/torch/csrc/jit/passes/bailout_graph.cpp.o
[ 80%] Building CXX object caffe2/CMakeFiles/torch.dir/__/torch/csrc/jit/passes/canonicalize.cpp.o
[ 80%] Building CXX object caffe2/CMakeFiles/torch.dir/__/torch/csrc/jit/passes/constant_propagation.cpp.o
[ 80%] Building CXX object caffe2/CMakeFiles/torch.dir/__/torch/csrc/jit/passes/constant_pooling.cpp.o
[ 81%] Building CXX object caffe2/CMakeFiles/torch.dir/__/torch/csrc/jit/passes/common_subexpression_elimination.cpp.o
2 errors generated.
make[2]: *** [caffe2/CMakeFiles/torch.dir/__/torch/csrc/jit/passes/alias_analysis.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [caffe2/CMakeFiles/torch.dir/all] Error 2
make: *** [all] Error 2
Traceback (most recent call last):
  File "setup.py", line 749, in <module>
    build_deps()
  File "setup.py", line 323, in build_deps
    cmake=cmake)
  File "/Users/adriantineo/dev/python/pytorch/tools/build_pytorch_libs.py", line 64, in build_caffe2
    cmake.build(my_env)
  File "/Users/adriantineo/dev/python/pytorch/tools/setup_helpers/cmake.py", line 340, in build
    self.run(build_args, my_env)
  File "/Users/adriantineo/dev/python/pytorch/tools/setup_helpers/cmake.py", line 93, in run
    check_call(command, cwd=self.build_dir, env=env)
  File "/Users/adriantineo/anaconda3/lib/python3.7/subprocess.py", line 347, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--target', 'install', '--config', 'Release', '--', '-j', '8']' returned non-zero exit status 2.

I have followed the steps for the prerequisites as stated in the documentation. I am using Anaconda 4.7.5, Python 3.7.3, and CUDA Toolkit 10.1.

On top of that, I am using Xcode command line tools for Xcode 8.2, since the ones for Xcode 10 were giving trouble with the error message fatal error: 'string.h' file not found. I found the tip to use an older version of Xcode command line tools here.

More info about the build:

-- ******** Summary ********
--   CMake version         : 3.14.5
--   CMake command         : /Users/adriantineo/anaconda3/bin/cmake
--   System                : Darwin
--   C++ compiler          : /Library/Developer/CommandLineTools/usr/bin/clang++
--   C++ compiler version  : 8.0.0.8000042
--   CXX flags             :   -Wno-deprecated -fvisibility-inlines-hidden -Wno-deprecated-declarations -Wnon-virtual-dtor
--   Build type            : Release
--   Compile definitions   : TH_BLAS_MKL;ONNX_ML=1
--   CMAKE_PREFIX_PATH     : /Users/adriantineo/anaconda3/lib/python3.7/site-packages
--   CMAKE_INSTALL_PREFIX  : /Users/adriantineo/dev/python/pytorch/torch
--   CMAKE_MODULE_PATH     : /Users/adriantineo/dev/python/pytorch/cmake/Modules;/Users/adriantineo/dev/python/pytorch/cmake/public/../Modules_CUDA_fix
--
--   ONNX version          : 1.5.0
--   ONNX NAMESPACE        : onnx_torch
--   ONNX_BUILD_TESTS      : OFF
--   ONNX_BUILD_BENCHMARKS : OFF
--   ONNX_USE_LITE_PROTO   : OFF
--   ONNXIFI_DUMMY_BACKEND : OFF
--   ONNXIFI_ENABLE_EXT    : OFF
--
--   Protobuf compiler     :
--   Protobuf includes     :
--   Protobuf libraries    :
--   BUILD_ONNX_PYTHON     : OFF
--

[...]

--   USE_CUDA              : True
--     CUDA static link    : OFF
--     USE_CUDNN           : OFF
--     CUDA version        : 10.1
--     CUDA root directory : /usr/local/cuda
--     CUDA library        : /usr/local/cuda/lib/libcuda.dylib
--     cudart library      : /usr/local/cuda/lib/libcudart.dylib
--     cublas library      : /usr/local/cuda/lib/libcublas.dylib
--     cufft library       : /usr/local/cuda/lib/libcufft.dylib
--     curand library      : /usr/local/cuda/lib/libcurand.dylib
--     nvrtc               : /usr/local/cuda/lib/libnvrtc.dylib
--     CUDA include path   : /usr/local/cuda/include
--     NVCC executable     : /usr/local/cuda/bin/nvcc
--     CUDA host compiler  : /Library/Developer/CommandLineTools/usr/bin/clang

Any tips to troubleshoot the build?


title: Taking a derivative through torch.ge, or how to explicitly define a derivative in pytorch

body:
I am trying to set up a network in which one layer maps from real numbers to {0, 1} (i.e. makes output binary).

What I tried

While I was able to find that torch.ge provides such functionality, whenever I want to train any parameter occurring before that layer in a network PyTorch breaks.

I have been also trying to find if there is any way in PyTorch/autograd, to override the derivative of a module by hand. More specifically in this cause, I would just like to pass derivative through the torch.ge, without changing it.

Minimal Example

Here is a minimal example I produced, which uses a typical neural network training structure in PyTorch.

import torch
import torch.nn as nn
import torch.optim as optim

class LinearGE(nn.Module):
    def __init__(self, features_in, features_out):
        super().__init__()
        self.fc = nn.Linear(features_in, features_out)

    def forward(self, x):
        return torch.ge(self.fc(x), 0)

x = torch.randn(size=(10, 30))
y = torch.randint(2, size=(10, 10))

# Define Model
m1 = LinearGE(30, 10)

opt = optim.SGD(m1.parameters(), lr=0.01)

crit = nn.MSELoss()

# Train Model
for x_batch, y_batch in zip(x, y):
    # zero the parameter gradients
    opt.zero_grad()

    # forward + backward + optimize
    pred = m1(x_batch)
    loss = crit(pred.float(), y_batch.float())
    loss.backward()
    opt.step()

What I encountered

When I run the above code the following error occurs:

File "__minimal.py", line 33, in <module>
    loss.backward()
...
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

This error makes sense since torch.ge function is not differentiable. However, since MaxPool2D is also not differentiable, I believe that there are ways of mitigating non-differentiability in PyTorch.

It would be great if someone could point me to any source which can help me either implement my own backprop for a custom module, or any way of avoiding this error message.

Thanks!


title: install conda on windows with conda or pip

body:
I am trying to install pytorch on my window.
First, I get command conda install pytorch torchvision cpuonly -c pytorchfrom here
(PyTorch Build:Stable(1.3);Your OS:Windows;Package:Conda;Language:Python3.6;CUDA:None), there are some problems described as followings:
**(python36) C:\Users\li_dan0109>conda install pytorch torchvision cpuonly -c pytorch
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
PackagesNotFoundError: The following packages are not available from current channels:
  - pytorch
Current channels:
  - https://conda.anaconda.org/pytorch/win-32
  - https://conda.anaconda.org/pytorch/noarch
  - https://repo.anaconda.com/pkgs/main/win-32
  - https://repo.anaconda.com/pkgs/main/noarch
  - https://repo.anaconda.com/pkgs/r/win-32
  - https://repo.anaconda.com/pkgs/r/noarch
  - https://repo.anaconda.com/pkgs/msys2/win-32
  - https://repo.anaconda.com/pkgs/msys2/noarch
To search for alternate channels that may provide the conda package you're
looking for, navigate to
    https://anaconda.org
and use the search bar at the top of the page.**

Then I also switch pip install by the command pip3 install torch==1.3.1+cpu torchvision==0.4.2+cpu -f https://download.pytorch.org/whl/torch_stable.html Then I met this error:
**(python36) C:\Users\li_dan0109>pip3 install torch==1.3.1+cpu torchvision==0.4.2+cpu -f https://download.pytorch.org/whl/torch_stable.html
Looking in links: https://download.pytorch.org/whl/torch_stable.html
ERROR: Could not find a version that satisfies the requirement torch==1.3.1+cpu (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)
ERROR: No matching distribution found for torch==1.3.1+cpu**

I really don't know how to fix it, please help me and thank you very much!


title: How to interpret the P numbers that fairseq generate produces?

body:
Using fairseq-generate.py, with the transformer architecture, each translation produces a section like this:

Why is it rare to discover new marine mammal species?
S-0     Why is it rare to discover new marine mam@@ mal species ?
H-0     -0.0643349438905716     Pourquoi est-il rare de découvrir de nouvelles espèces de mammifères marins?
P-0     -0.0763 -0.1849 -0.0956 -0.0946 -0.0735 -0.1150 -0.1301 -0.0042 -0.0321 -0.0171 -0.0052 -0.0062 -0.0015

With this explanation:

  H is the hypothesis along with an average log-likelihood; and P is the positional score per token position, including the end-of-sentence marker

I'm wondering if it is reasonable to say a low (absolute) number in the P row means higher confidence in that particular word? E.g. does -0.07 for "Pourquoi" means it was happier about that than it was (-0.1849) for "est-il"? And the low -0.0015 at the end means it was really confident the sentence should end there.

Background: What I'm trying to work out is if I can use either the H number, or somehow to use the individual P numbers, to get a confidence measure in its translation. I've been analyzing a handful of translations against the H number and didn't notice much correspondence between it and my subjective opinion of translation quality. But I've a couple where I thought it was particularly poor - it had missed a bit of key information - and the final P number was a relatively high -0.6099 and -0.3091 (The final P number is -0.11 or so on most of them.)


title: what does clamp_ does in pytorch and how to change it to the tensorflow 2.0?

body:
prior_boxes = torch.FloatTensor(prior_boxes).to(device)  # (8732, 4)
   prior_boxes.clamp_(0, 1)  # (8732, 4)

what dooes clamp_ do in pytorch and how to change it to the tensorflow 2.0?
I'm not sure what clamp_ do exactly?


title: Pytorch BERT: Misshaped inputs

body:
I am running into issues of evaluating huggingface's BERT model ('bert-base-uncased') on large input sequences.
model = BertModel.from_pretrained('bert-base-uncased', output_hidden_states=True)
token_ids = [101, 1014, 1016, ...] # len(token_ids) == 33286
token_tensors = torch.tensor([token_ids]) # shape == [1, 33286]
segment_tensors = torch.tensor([[1] * len(token_ids)]) # shape == [1, 33286]
model(token_tensors, segment_tensors)

Traceback
self.model(token_tensors, segment_tensors)
  File "/home/.../python3.8/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/.../python3.8/site-packages/transformers/modeling_bert.py", line 824, in forward
    embedding_output = self.embeddings(
  File "/home/.../python3.8/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/.../python3.8/site-packages/transformers/modeling_bert.py", line 211, in forward
    embeddings = inputs_embeds + position_embeddings + token_type_embeddings
RuntimeError: The size of tensor a (33286) must match the size of tensor b (512) at non-singleton dimension 1

I noticed that model.embeddings.positional_embeddings.weight.shape == (512, 768). I.e. when I restrict the input size to model(token_tensors[:, :10], segment_tensors[:, :10]) it works. I am misunderstanding how the the token_tensors and segment_tensors should be shaped. I thought they should be sized (batch_size, sequence_length)
Thanks for the help


title: How to convert pip install to Poetry file?

body:
With hours of research, I still can't figure out a way to convert this pip install cmd to pyproject.toml
file. I'm trying to install PyTorch.
pip install torch==1.7.0+cpu torchvision==0.8.1+cpu torchaudio===0.7.0 -f https://download.pytorch.org/whl/torch_stable.html
This is what I got at the moment (Completely Wrong!)
[tool.poetry]
name = "poetry-test"
version = "0.1.0"
description = ""
authors = ["Your Name <you@example.com>"]

[tool.poetry.dependencies]
python = "^3.8"
torch = "^1.7.0"
torchvision = "^0.8.1"
torchaudio = "^0.7.0"

[tool.poetry.dependencies.torch]
url = "https://download.pytorch.org/whl/torch_stable.html"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"


title: Many-to-Many LSTM PyTorch

body:
I want to build an LSTM model for the FashionMNIST dataset in PyTorch. I will later on need to extend this to a different dataset that contains videos.
It should get a sequence of images (of the FashionMNIST) as the input (let's say 20 images) and the output should tell me how many sneakers are in the sequence (class 6) and where in the sequence they are located.
I was wondering whether this is even possible with a simple LSTM or a simple CNN or whether I need an CNN_LSTM?
I tried to implement an CNN_LSTM in PyTorch. Below you can find my current model (which throws an error right now).
The last line throws the following error: "input must have 3 dimensions, got 4" (I also added the first part of the error message as a picture).
Could someone please provide some help by any chance? Is the way I am doing this correct? I couldn't fix the error and I am not sure whether the rest of my code is correct.  I am quite new to LSTMs.
Also, how can I transform the FashionMNIST dataset such that it always comes in a sequence of 20 images?
Many thanks in advance!
class CNN(nn.Module):
  def __init__(self, K):
    super(CNN, self).__init__()
    self.layer1 = nn.Sequential(
        nn.Conv2d(in_channels=1, out_channels=32, kernel_size=3, padding=1),
        nn.BatchNorm2d(32),
        nn.ReLU(),
        nn.MaxPool2d(kernel_size=2, stride=2))

    self.layer2 = nn.Sequential(
        nn.Conv2d(in_channels=32, out_channels=64, kernel_size=3),
        nn.BatchNorm2d(64),
        nn.ReLU(),
        nn.MaxPool2d(2))

    # three fully connected layer
    self.fc1 = nn.Linear(in_features=64*6*6, out_features=600)
    self.drop = nn.Dropout2d(0.25)
    self.fc2 = nn.Linear(in_features=600, out_features=120)
    self.fc3 = nn.Linear(in_features=120, out_features=10)

  def forward(self, x):
      out = self.layer1(x)
      out = self.layer2(out)
      out = out.view(out.size(0), -1)
      out = self.fc1(out)
      out = self.drop(out)
      out = self.fc2(out)
      out = self.fc3(out)

      return out

class Combine(nn.Module):
  def __init__(self, K):
    super(Combine, self).__init__()
    self.cnn = CNN(K)
    self.D = 10  # n_inputs
    self.M = 128 # n_hidden
    self.K = 2 # n_outputs
    self.L = 10 # n_rnnlayers

    self.rnn = nn.LSTM(
        input_size=self.D,
        hidden_size=self.M,
        num_layers=self.L,
        batch_first=True)
    self.fc =nn.Linear(self.M, self.K)

  def forward(self, X):
    # initial hidden states
    h0 = torch.zeros(self.L, X.size(0), self.M).to(device)
    c0 = torch.zeros(self.L, X.size(0), self.M).to(device)

    # get RNN unit output
    out, _ = self.rnn(X, (h0, c0))
    out = self.fc(out)

    return out

model = Combine(K)

# use GPU in colab if available
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
print(device)
model.to(device)

# Loss and optimizer
learning_rate = 0.001
criterion = nn.CrossEntropyLoss() # because mutli-class classification (includes softmax activation function for multi-class already)
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

# Training and testing the model
def batch_gd(model, criterion, optimizer, train_loader, test_loader, epochs):
  train_losses = np.zeros(epochs)
  test_losses = np.zeros(epochs)

  # iterate over epochs
  for it in range(epochs):
    model.train()
    t0 = datetime.now()
    train_loss = []
    for inputs, targets in train_loader:
      # move data to GPU
      #inputs = inputs.reshape(-1,28,28)
      inputs, targets = inputs.to(device), targets.to(device)

      # zero the parameter gradients (empty gradients) for backward pass
      # Initializing a gradient as 0 so there is no mixing of gradient among the batches
      optimizer.zero_grad()

      # Forward pass
      outputs = model(inputs)
      loss = criterion(outputs, targets)

      # Backward and optimize
      loss.backward() # propagating the error backward
      optimizer.step() # optimizing the parameters

      train_loss.append(loss.item())

    # Get train loss and test loss
    train_loss = np.mean(train_loss) # a little misleading

    # evaluate model
    model.eval()
    test_loss = []
    for inputs, targets in test_loader: # test samples and targets
      # move data to GPU
      inputs, targets = inputs.to(device), targets.to(device)
      outputs = model(inputs)
      loss = criterion(outputs, targets)
      test_loss.append(loss.item())
    test_loss = np.mean(test_loss)

    # Save losses
    train_losses[it] = train_loss
    test_losses[it] = test_loss

    dt = datetime.now() - t0
    print(f'Epoch {it+1}/{epochs}, Train Loss: {train_loss:.4f}, \
      Test Loss: {test_loss:.4f}, Duration: {dt}')

  return train_losses, test_losses

train_losses, test_losses = batch_gd(
    model, criterion, optimizer, train_loader, test_loader, epochs=15)

Part of the Error Message


title: Failing to train SkipGram word embedding in Pytorch

body:
I am training the skipgram word embeddings using the famous model described in https://arxiv.org/abs/1310.4546. I want to train it in PyTorch but I am getting errors and I can't figure out where they are coming from. Below I have provided my model class, training loop, and batching method. Does anyone have any insight into whats going on?

I am getting an error on the output = loss(data, target) line. It is having a problem with <class 'torch.LongTensor'> which is weird because CrossEntropyLoss takes a long tensor. The output shape might be wrong which is: torch.Size([1000, 100, 1000]) after the feedforward.

I have my model defined as:

import torch
import torch.nn as nn

torch.manual_seed(1)

class SkipGram(nn.Module):

    def __init__(self, vocab_size, embedding_dim):
        super(SkipGram, self).__init__()
        self.embeddings = nn.Embedding(vocab_size, embedding_dim)
        self.hidden_layer = nn.Linear(embedding_dim, vocab_size)

        # Loss needs to be input: (minibatch (N), C) target: (minibatch, 1), each label is a class
        # Calculate loss in training

    def forward(self, x):
        embeds = self.embeddings(x)
        x = self.hidden_layer(embeds)
        return x

My training is defined as:

import torch.optim as optim
from torch.autograd import Variable

net = SkipGram(1000, 300)
optimizer = optim.SGD(net.parameters(), lr=0.01)

batch_size = 100
size = len(train_ints)

batches = batch_index_gen(batch_size, size)
inputs, targets = build_tensor_from_batch_index(batches[0], train_ints)

for i in range(100):
    running_loss = 0.0

    for batch_idx, batch in enumerate(batches):
        data, target = build_tensor_from_batch_index(batch, train_ints)
#         if (torch.cuda.is_available()):
#             data, target = data.cuda(), target.cuda()
#             net = net.cuda()
        data, target = Variable(data), Variable(target)
        optimizer.zero_grad()
        output = net.forward(data)
        loss = nn.CrossEntropyLoss()
        output = loss(data, target)
        output.backward()
        optimizer.step()
        running_loss += loss.data[0]
        optimizer.step()

        print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                i, batch_idx * len(batch_size), len(size),
                100. * (batch_idx * len(batch_size)) / len(size), loss.data[0]))

If useful my batching is:

def build_tensor_from_batch_index(index, train_ints):
    minibatch = []
    for i in range(index[0], index[1]):
        input_arr = np.zeros( (1000,1), dtype=np.int )
        target_arr = np.zeros( (1000,1), dtype=np.int )
        input_index, target_index = train_ints[i]
        input_arr[input_index] = 1
        target_arr[input_index] = 1

        input_tensor = torch.from_numpy(input_arr)
        target_tensor = torch.from_numpy(target_arr)

        minibatch.append( (input_tensor, target_tensor) )

    # Concatenate all tensors into a minibatch
    #x = [tensor[0] for tensor in minibatch]
    #print(x)
    input_minibatch = torch.cat([tensor[0] for tensor in minibatch], 1)
    target_minibatch = torch.cat([tensor[1] for tensor in minibatch], 1)

    #target_minibatch = minibatch[0][1]
    return input_minibatch, target_minibatch


title: Does Docker image supporting Nvidia gpu on a cpu machine

body:
I want to run a deep neural network on a Docker container. I can use TensorFlow or PyTorch or CNTK images available publicly on Docker Hub that support a GPU. And I can get an NVIDA GPU in the cloud.

What happens when I run those GPU-based containers on my PC that does not have a GPU (or an Intel GPU)? Will the Docker container based on NVIDIA GPU gracefully run on my a CPU (albeit slower)?

It would be great if I could develop on locally without a GPU and scale up with GPU when I am ready to run at scale.

Will this work? Thoughts?


title: In pytorch data parallel mode, how to use the global tensor?

body:
In this example, I wish the z_proto could be global for different GPUs. However, in the data parallel mode, it is split into different GPUs as well. How to solve such a problem? Thank you.

class SequencePrototypeTokenClassification(nn.Module):
    def __init__(self,seq_model, label_num):
        super(SequencePrototypeTokenClassification, self).__init__()
        self.seq_model = seq_model
        self.label_num = label_num

    def forward(self, input_ids, token_type_ids, attention_mask, labels, z_proto, n_query, target_inds):
        z, _ = self.seq_model(input_ids, token_type_ids, attention_mask, output_all_encoded_layers=False)
        z_dim = z.size(-1)
        zq = z.squeeze().view(-1, z_dim)
        dists = euclidean_dist(zq, z_proto)
        log_p_y = F.log_softmax(-dists, dim=1).view(-1, self.label_num)
        loss_val = -log_p_y.gather(1, self.target_inds).squeeze().view(-1).mean()
        _, y_hat = log_p_y.max(1)

        return loss_val, y_hat


title: Torch sum each row excluding an index

body:
Given a Tensor A of shape (N,C) and an indices Tensor Idx of shape (N,), i'd like to sum all the elements of each row in A excluding the corresponding column index in I. For example:

A = torch.tensor([[1,2,3],
                  [4,5,6]])

Idx = torch.tensor([0,2])

#result:
torch.tensor([[5],
              [9]])

A solution using loops is known.


title: PyTorch Neural Net not learning

body:
I am new to PyTorch and I'm trying to build a simple neural net for classification. The problem is the network doesn't learn at all. I tried various learning rate ranging from 0.3 to 1e-8 and I also tried training it for a longer duration. My data is small with only 120 training examples and the batch size is 16. Here is the code
Define network
model = nn.Sequential(nn.Linear(4999, 1000),
                  nn.ReLU(),
                  nn.Linear(1000,200),
                  nn.ReLU(),
                  nn.Linear(200,1),
                  nn.Sigmoid())

Loss and optimizer
import torch.optim as optim

optimizer = optim.SGD(model.parameters(), lr=0.01)
criterion = nn.BCELoss(reduction="mean")

Training
num_epochs = 100
for epoch in range(num_epochs):
  cumulative_loss = 0
  for i, data in enumerate(batch_gen(X_train, y_train, batch_size=16)):
    inputs, labels = data
    inputs = torch.from_numpy(inputs).float()
    labels = torch.from_numpy(labels).float()

    optimizer.zero_grad()

    outputs = model(inputs)
    loss = criterion(outputs, labels)
    loss.backward()
    optimizer.step()
    cumulative_loss += loss.item()

    if i%5 == 0 and i != 0:
      print(f"epoch {epoch} batch {i} => Loss: {cumulative_loss/5}")

print("Finished Training!!")

Any help is appreciated!


title: deep reinforcement learning parameters and training time for a simple game

body:
I want to learn how deep reinforcement algorithm works and what time it takes to train itself for any given environment.
I came up with a very simple example of environment:

There is a counter which holds an integer between 0 to 100.
counting to 100 is its goal.

there is one parameter direction whose value can be +1 or -1.
it simply show the direction to move.

out neural network takes this direction as input and 2 possible action as output.

Change the direction
Do not change the direction

1st action will simply flip the direction (+1 => -1 or -1 =>+1). 2nd action will keep the direction as it is.

I am using python for backend and javascript for frontend.
It seems to take too much time, and still it is pretty random. i have used 4 layer perceptron. training rate of 0.001 . memory learning with batch of 100. Code is of Udemy tutorial of Artificial Intelligence and is working properly.

My question is, What should be the reward for completion and for each state.? and  how much time it is required to train simple example as that.?


title: Getting an Error in Pytorch: IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

body:
I seem to be having a problem with my code. The error occurs at:

x, predicted = torch.max(net(value).data.squeeze(), 1)

I'm not sure what the issue is, and I've tried everything to fix. From my understanding, there seems to be a problem with the tensor dimension. I'm not sure what else to do. Can anyone give me any suggestions or solutions on how to fix this problem? Thank you in advance.

class Network(nn.Module): #Class for the neural network
def __init__(self):
    super(Network, self).__init__()
    self.layer1 = nn.Linear(6, 10) #First number in the number of inputs(784 since 28x28 is 784.) Second number indicates the number of inputs for the hidden layer(can be any number).
    self.hidden = nn.Softmax() #Activation Function
    self.layer2 = nn.Linear(10, 1) #First number is the hidden layer number(same as first layer), second number is the number of outputs.
    self.layer3 = nn.Sigmoid()

def forward(self, x): #Feed-forward part of the neural network. We will will feed the input through every layer of our network.
    y = self.layer1(x)
    y = self.hidden(y)
    y = self.layer2(y)
    y = self.layer3(y)
    return y #Returns the result

net = Network()
loss_function = nn.BCELoss()
optimizer = optim.SGD(net.parameters(), lr=0.01)

for x in range(1): #Amount of epochs over the dataset
for index, value in enumerate(new_train_loader):
    print(value)#This loop loops over every image in the dataset
    #actual = value[0]
    actual_value = value[5]
    #print(value.size())
    #print(net(value).size())
    print("ACtual", actual_value)
    net(value)
    loss = loss_function(net(value), actual_value.unsqueeze(0)) #Updating our loss function for every image
    #Backpropogation
    optimizer.zero_grad() #Sets gradients to zero.
    loss.backward() #Computes gradients
    optimizer.step() #Updates gradients
    print("Loop #: ", str(x+1), "Index #: ", str(index+1), "Loss: ", loss.item())

right = 0
total = 0
for value in new_test_loader:
actual_value = value[5]
#print(torch.max(net(value).data, 1))
print(net(value).shape)
x, predicted = torch.max(net(value).data.squeeze(), 1)
total += actual_value.size(0)
right += (predicted==actual_value).sum().item()
print("Accuracy: " + str((100*right/total)))

I should also mention that i'm using the latest versions.


title: Change values inside a Pytorch 3D tensor

body:
I have a 224x224 binary image in a tensor (1, 224, 224), with 0-pixels representing background a 1-pixels representing foreground.
I want to reshape it in a tensor (2, 224, 224), such as the first "layer" gt[0] has 1-pixels where there were 0-pixels in the original image and viceversa. This way one layer should show 1s where there is background and the other one will have 1s on the foreground (basically I need to have two complementary binary images in this tensor).

This is my code:

# gt is a tensor (1, 224, 224)
gt = gt.expand((2, 224, 224))
backgr = gt[0]
foregr = gt[1]

backgr[backgr == 0] = 2 # swap all 0s in 1s and viceversa
backgr[backgr == 1] = 0
backgr[backgr == 2] = 1

gt[0] = backgr

print(gt[0])
print(gt[1])

The problem is both layers are modified with this code and I can't figure out how to keep one of the two constant and change only gt[0].


title: Unable to import TorchVision after installation Mac OSX

body:
I've installed Pytorch and Torchvision in the way suggested on their website via pip within a virtual environment (env), and whilst no errors occur during installation when I go to import torchvision in my python code the following error occurs.

Traceback (most recent call last):
  File "demo.py", line 2, in <module>
    import torchvision
  File "/Users/QuinceyBee/env/lib/python3.7/site-packages/torchvision/__init__.py", line 2, in <module>
    from torchvision import datasets
  File "/Users/QuinceyBee/env/lib/python3.7/site-packages/torchvision/datasets/__init__.py", line 9, in <module>
    from .fakedata import FakeData
  File "/Users/QuinceyBee/env/lib/python3.7/site-packages/torchvision/datasets/fakedata.py", line 3, in <module>
    from .. import transforms
  File "/Users/QuinceyBee/env/lib/python3.7/site-packages/torchvision/transforms/__init__.py", line 1, in <module>
    from .transforms import *
  File "/Users/QuinceyBee/env/lib/python3.7/site-packages/torchvision/transforms/transforms.py", line 17, in <module>
    from . import functional as F
  File "/Users/QuinceyBee/env/lib/python3.7/site-packages/torchvision/transforms/functional.py", line 5, in <module>
    from PIL import Image, ImageOps, ImageEnhance, PILLOW_VERSION
ImportError: cannot import name 'PILLOW_VERSION' from 'PIL' (/Users/QuinceyBee/env/lib/python3.7/site-packages/PIL/__init__.py)

I have tried creating new virtual environments to rebuild from scratch, tried to install via conda within a conda environment, however, neither of these resolved this issue.

I apologise for any format issues, this is the first time posting on here and also I'm relatively new to using python.

Any assistance would be greatly appreciated.


title: TypeError: __call__() takes 2 positional arguments but 3 were given. To train Raccoon prediction model using FastRCNN through Transfer Learning

body:
from torchvision.models.detection.faster_rcnn import FastRCNNPredictor
 from engine import train_one_epoch, evaluate
 import utils
 import torchvision.transforms as T

 num_epochs = 10
 for epoch in range(num_epochs):
    train_one_epoch(model, optimizer, data_loader, device, epoch, print_freq=10)
    lr_scheduler.step()
    evaluate(model, data_loader_test, device=device)

I am using the same code as provided in this link Building Raccoon Model but mine is not working.

This is the error message I am getting
TypeError                                 Traceback (most recent call last)
 in ()

  2 for epoch in range(num_epochs):

  3    # train for one epoch, printing every 10 iterations

  4   ----> train_one_epoch(model, optimizer, data_loader, device, epoch, print_freq=10)

  5     # update the learning rate

  6   lr_scheduler.step()

7 frames

 in getitem(self, idx)

 29         target["iscrowd"] = iscrowd

 30         if self.transforms is not None:

 31        ---> img, target = self.transforms(img, target)

 32         return img, target
 33

TypeError: call() takes 2 positional arguments but 3 were given


title: model.parameters() does not produce an iterable of Tensors

body:
I am trying to use torch.nn.utils.clip_grad_norm_() which requires an iterable of Tensors.  See below

for epoch in progress_bar(range(num_epochs)):
    lstm.train()
    outputs = lstm(trainX.to(device))
    optimizer.zero_grad()
    torch.nn.utils.clip_grad_norm_(lstm.parameters(), 1)

My code errors with:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-168-4cd34e6fd44d> in <module>
     28     lstm.train()
     29     outputs = lstm(trainX.to(device))
---> 30     torch.nn.utils.clip_grad_norm_(lstm.parameters(), 1)
     31
     32

/opt/conda/lib/python3.6/site-packages/torch/nn/utils/clip_grad.py in clip_grad_norm_(parameters, max_norm, norm_type)
     28         total_norm = max(p.grad.detach().abs().max() for p in parameters)
     29     else:
---> 30         total_norm = torch.norm(torch.stack([torch.norm(p.grad.detach(), norm_type) for p in parameters]), norm_type)
     31     clip_coef = max_norm / (total_norm + 1e-6)
     32     if clip_coef < 1:

RuntimeError: stack expects a non-empty TensorList

If I example lstm.parameters() I get a list of Parameters, instead of a list of Tensors:

<class 'torch.nn.parameter.Parameter'> torch.Size([2048, 1])
<class 'torch.nn.parameter.Parameter'> torch.Size([2048, 512])
<class 'torch.nn.parameter.Parameter'> torch.Size([2048])
<class 'torch.nn.parameter.Parameter'> torch.Size([2048])
<class 'torch.nn.parameter.Parameter'> torch.Size([2048, 512])
<class 'torch.nn.parameter.Parameter'> torch.Size([2048, 512])
<class 'torch.nn.parameter.Parameter'> torch.Size([2048])
<class 'torch.nn.parameter.Parameter'> torch.Size([2048])
<class 'torch.nn.parameter.Parameter'> torch.Size([1, 512])
<class 'torch.nn.parameter.Parameter'> torch.Size([1])

Looking at the first Parameter, it is a list of Tensors:

<class 'torch.Tensor'> torch.Size([1])
<class 'torch.Tensor'> torch.Size([1])
<class 'torch.Tensor'> torch.Size([1])
<class 'torch.Tensor'> torch.Size([1])
<class 'torch.Tensor'> torch.Size([1])
<class 'torch.Tensor'> torch.Size([1])
.
.
.

Does anyone know what is going on here?


title: pytorch .stack final shape after .squeeze

body:
I had a pandas dataframe 200 columns by 2500 rows which I made it into a tensor

tensor = torch.tensor(df.values)
tensor.size() => ([2500,200])

which i chunked and enumerated

list=[]
for i,chunk in enumerate(tensor.chunk(100,dim=0))
    chunk.size =>([25,200])
    output = hiddenlayer(chunks)
    output.size() => ([25,1])
    list += output

chunks were fed through some layers and outputted as 1 feature tensors. So now I have a list of 100 tensors, each with 25 blocks of 1, 100x25x1

so i

stacked = torch.stack(list, 1).squeeze(2)
stacked.size()=([25,100])

I've played around with the stacking and squeezing but i can't seem to get back to ([2500,1]) which is what I want. Am I missing something? If you could quickly help me understand what stacking and squeezing is doing and why it's not working for me I'd be forever in your debt! Thanks


title: Exhaustive Concatenation between the tensors

body:
I am trying to do the exhaustive concatenation between the tensors. So, for example,
I have tensor:

a = torch.randn(3, 512)

I want to concatenate like
concat(t1,t1),concat(t1,t2), concat(t1,t3), concat(t2,t1), concat(t2,t2)....

As a naive solution,
I have used for loop:

ans = []
result = []
split = torch.split(a, [1, 1, 1], dim=0)

for i in range(len(split)):
    ans.append(split[i])

for t1 in ans:
    for t2 in ans:
        result.append(torch.cat((t1,t2), dim=1))

The issue is that each epoch is taking very long time and the code is slow.
I tried the solution posted in question on PyTorch: How to implement attention for graph attention layer but this gives a memory error.

t1 = a.repeat(1, a.shape[0]).view(a.shape[0] * a.shape[0], -1)
t2 = a.repeat(a.shape[0], 1)
result.append(torch.cat((t1, t2), dim=1))

I am sure there is a faster way, but I was unable to figure it out.


title: Inference time and TFLOPS in pytorch

body:
I am currently looking into the half-precision inference time of different CNN models using the torch.autograd.profiler using two different GPUs:

Nvidia RTX 2080 Ti (26.90 TFLOPS) - done locally (better CPU)
Nvidia T4 (65.13 TFLOPS) - done in the cloud

It took me by surprise that the 2080 Ti is significantly faster (half the time or less), independent of batch size, input resolution, and architecture even though it has less than half the TFLOPS.

Does anyone know why?

import torch
import segmentation_models_pytorch as smp # pip install git+https://github.com/qubvel/segmentation_models.pytorch

runs = 10
res = 512
bs = 8
is_half = True

m = smp.Unet(encoder_name='resnet101', encoder_weights=None)
m.eval()
m.cuda()
m.half()

t = torch.rand((bs, 3, res, res)).cuda()
t = t.half()

if is_half:
    m.half()
    t = t.half()

# warm up
with torch.no_grad():
    m(t)

cpu_time_ms = 0
cuda_time_ms = 0
for i in range(runs):
    with torch.no_grad():
        with torch.autograd.profiler.profile(use_cuda=True) as prof:
            m(t)
        cpu_time_ms += prof.self_cpu_time_total / 1000
        cuda_time_ms += sum([evt.cuda_time_total for evt in prof.key_averages()]) / 1000

cpu_time_ms /= runs * bs
cuda_time_ms /= runs * bs

print('res={}x{} cuda={:.1f}ms cpu={:.1f}ms'.format(res, res, cuda_time_ms, cpu_time_ms))

Example:

unet with resnet101 as backbone and batch size 8

t4
res=128x128 cuda=11.3ms cpu=3.0ms
res=256x256 cuda=14.5ms cpu=2.8ms
res=512x512 cuda=50.4ms cpu=7.3ms

rtx 2080 ti
res=128x128 cuda=7.5ms cpu=1.7ms
res=256x256 cuda=8.6ms cpu=1.8ms
res=512x512 cuda=21.1ms cpu=3.0ms


title: What sort of loss function should I use this multi-class multi-label(?) problem?

body:
In my experiment I am trying to train a neural network to detect if patients exhibit symptom A, B, C, D. My data consists of different angled photos of each patient along with whether or not they have symptom A, B, C, D.

Right now in, pytoch, I am using MSELoss and calculating my test error as the total number of correct classifications out of the total number of classifications. I'm guessing this is too naive and even inappropriate.

An example of a test error computation would be like this:
Suppose we have 2 patients with two images each of them. Then there would be 16 total classifications (1 for whether patient 1 has symptom A, B, C, D in photo 1, etc). And if the model correctly predicted that in photo 1 patient 1 exhibited symptom A then that would add 1 to the total number of correct classifications.


title: Testing and Confidence score of Network trained with nn.CrossEntropyLoss()

body:
I have trained a network with the following structure:

Intent_LSTM(
(attention): Attention()
(embedding): Embedding(34601, 400)
(lstm): LSTM(400, 512, num_layers=2, batch_first=True, dropout=0.5)
(dropout): Dropout(p=0.5, inplace=False)
(fc): Linear(in_features=512, out_features=3, bias=True)
)

Now I want to test this trained network and also get the confidence score of classification.
Here is my current implementation of test function:

output = model_current(inputs)
pred = torch.round(output.squeeze())
pred = pred.argmax(dim=1, keepdim=True)

Now my question isas follows.

Here pred is just the output from a Fully connected layer from my network without softmax (as required by a loss function). Is this(pred = pred.argmax(dim=1, keepdim=True)) the right way to get the predictions? Or should I pass the output from the network to a softmax layer and then do argmax?
How do I get the confidence score? Should I pass the output from the network to a softmax layer and select the argmax as the confidence of the class?


title: Are zero loss tensor also backpropagated?

body:
I was reading the paper Training Region-based Object Detectors with Online Hard Example Mining and one way of implementing online hard example mining would be to erase (multiply by zero) some elements of the loss tensor. However, in this paper the authors claim that this would be inefficient since current deep learning frameworks allocate memory and perform backprop even for 0 loss tensors. Unfortunately, the authors don't mention which frameworks do that and I'd like to know about Tensorflow and PyTorch in this regard.

Furthermore, if instead of multiplying the loss by zero I gathered only the slice I'm interested from the loss would that avoid back-propagating through the non-interesting RoIs? For example:

active_rois = tf.where(tf.not_equal(rois, 0)) # get the rois I care about
truth = tf.gather(truth, active_rois) # filter gt tensor
pred = tf.gather(pred, active_rois) # filter pred tensor
loss = compute_loss(truth, pred)

The excerpt from paper I mentioned earlier:

  The loss layer can compute loss for all RoIs, sort them based on this
  loss to select hard RoIs,  and finally set the loss  of  all non-hard
  RoIs  to 0 .   Though  straightforward, this implementation is
  inefficient as the RoI network still allocates memory and performs
  backward pass for all RoIs, even though most RoIs have 0 loss and
  hence no gradient updates (a limitation of current deep learning
  toolboxes).


title: Pytorch loss function dimensions do not match

body:
I'm trying to run word embeddings using batch training, as shown below.

def forward(self, inputs):
    print(inputs.shape)
    embeds = self.embeddings(inputs)
    print(embeds.shape)
    out = self.linear1(embeds)
    print(out.shape)
    out = self.activation_function1(out)
    print(out.shape)
    out = self.linear2(out).cuda()
    print(out.shape)
    out = self.activation_function2(out)
    print(out.shape)
    return out.cuda()

Here, I'm using context size 4, batch size 32, embedding size 50, hidden layer size 64, vocab size 9927

The output of the "shape" functions is

print(inputs.shape) ----> torch.Size([4, 32])

print(embeds.shape) ----> torch.Size([4, 32, 50])

print(out.shape) ----> torch.Size([4, 32, 64])

print(out.shape) ----> torch.Size([4, 32, 64])

print(out.shape) ----> torch.Size([4, 32, 9927])

print(out.shape) ----> torch.Size([4, 32, 9927])

Are the shapes of these correct? I'm quite confused.

Also, when I train, it returns an error:

def train(epoch):
  model.train()
  for batch_idx, (data, target) in enumerate(train_loader, 0):
    optimizer.zero_grad()
    output = model(torch.stack(data))
    loss = criterion(output, target)
    loss.backward()
    optimizer.step()

I'm getting an error in the line "loss = criterion(output, target)". It says "Expected input batch_size (4) to match target batch_size (32)." Are my shapes for the "forward" function correct? I'm not that familiar with batch training. How do I make the dimensions match?

-------EDIT: Posting init code below -----

  def __init__(self, vocab_size, embedding_dim):
    super(CBOW, self).__init__()
    self.embeddings = nn.Embedding(vocab_size, embedding_dim)
    self.linear1 = nn.Linear(embedding_dim, 64)
    self.activation_function1 = nn.ReLU()
    self.linear2 = nn.Linear(64, vocab_size)
    self.activation_function2 = nn.LogSoftmax(dim = -1)


title: Proper dataloader setup to train fasterrcnn-resnet50 for object detection with pytorch

body:
I am trying to train pytorches torchvision.models.detection.fasterrcnn_resnet50_fpn to detect objects in my own images.

According to the documentation, this model expects a list of images and a list of dictionaries with
'boxes' and 'labels' as keys. So my dataloaders __getitem__() looks like this:

def __getitem__(self, idx):
    # load images
    _, img = self.images[idx].getImage()
    img = Image.fromarray(img, mode='RGB')
    objects = self.images[idx].objects

    boxes = []
    labels = []
    for o in objects:
        # append bbox to boxes
        boxes.append([o.x, o.y, o.x+o.width, o.y+o.height])
        # append the 4th char of class_id, the number of lights (1-4)
        labels.append(int(str(o.class_id)[3]))

    # convert everything into a torch.Tensor
    boxes = torch.as_tensor(boxes, dtype=torch.float32)
    labels = torch.as_tensor(labels, dtype=torch.int64)

    target = {}
    target["boxes"] = boxes
    target["labels"] = labels

    # transforms consists only of transforms.Compose([transforms.ToTensor()]) for the time being
    if self.transforms is not None:
        img = self.transforms(img)

    return img, target

To my best knowledge, it returns exactly what's asked. My dataloader looks like this

data_loader = torch.utils.data.DataLoader(
    dataset, batch_size=4, shuffle=False, num_workers=2)

however, when it get's to this stage:
for images, targets in dataloaders[phase]:
it raises

  RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 0. Got 12 and 7 in dimension 1 at C:\w\1\s\windows\pytorch\aten\src\TH/generic/THTensor.cpp:689

Can someone point me in the right direction?


title: How to define the __len__ method for PyTorch Dataloader when I have separate length datasets?

body:
I'm currently loading in my data with one single dataset class. Within the dataset, I split the train, test, and validation data separately. For example:

class Data():
    def __init__(self):
        self.load()

    def load(self):
        with open(file=file_name, mode='r') as f:
            self.data = f.readlines()

        self.train = self.data[:checkpoint]
        self.valid = self.data[checkpoint:halfway]
        self.test = self.data[halfway:]

Many of the details have been omitted for the sake of readability. Basically, I read in one big dataset and make the splits manually.

My question is arising from how to override the __len__ method when the lengths of my train, valid, and test data all differ?

The reason I want to do this is because I want to keep the split data in one single class, and I also want to create separate Dataloaders for each, and so something like:

def __len__(self):
    return len(self.train)

wouldn't be appropriate for self.test and self.valid.

Perhaps I'm fundamentally misunderstanding the Dataloader, but how should I approach this issue? Thanks in advance.


title: Index out of range error while training dataset

body:
I am trying to train MaskRCNN to detect and segment apples using the dataset from this paper,

github link to code being used

I am simply following the instructions as provided in the ReadMe file..

Here is the output on console

(venv) PS > python train_rcnn.py --data_path 'D:\Research Report\tensorflow\Mask_RCNN-TRIALS\Mask_RCNN-master\datasets\apples-minneapple' --model mrcnn --epochs 50 --output-dir 'D:\Research Report\tensorflow\Mask_RCNN-TRIALS\Mask_RCNN-master\samples\apples'

mrcnn
Namespace(batch_size=2, data_path='D:\\Research Report\\tensorflow\\Mask_RCNN-TRIALS\\Mask_RCNN-master\\datasets\\apples-minneapple', dataset='AppleDataset', device='cuda', epochs=50, lr=0.02, lr_gamma=0.1, lr_step_size=8, lr_steps=[8, 11], model='mrcnn', momentum=0.9, output_dir='D:\\Research Report\\tensorflow\\Mask_RCNN-TRIALS\\Mask_RCNN-master\\samples\\apples', print_freq=20, resume='', weight_decay=0.0001, workers=4)
Loading data
Creating data loaders
Creating model
Start training
Epoch: [0]  [  0/335]  eta: 1:00:28  lr: 0.000080  loss: 2.4100 (2.4100)  loss_classifier: 0.8481 (0.8481)  loss_box_reg: 0.4164 (0.4164)  loss_objectness: 0.9299 (0.9299)  loss_rpn_box_reg: 0.2157 (0.2157)  time: 10.8327  data: 7.9925  max mem: 2733
Epoch: [0]  [ 20/335]  eta: 0:06:18  lr: 0.001276  loss: 1.4465 (1.4728)  loss_classifier: 0.5526 (0.5496)  loss_box_reg: 0.3586 (0.3572)  loss_objectness: 0.2666 (0.3418)  loss_rpn_box_reg: 0.2233 (0.2242)  time: 0.7204  data: 0.0132  max mem: 3247
Epoch: [0]  [ 40/335]  eta: 0:04:48  lr: 0.002473  loss: 0.9622 (1.2287)  loss_classifier: 0.2927 (0.4276)  loss_box_reg: 0.3188 (0.3314)  loss_objectness: 0.1422 (0.2491)  loss_rpn_box_reg: 0.2168 (0.2207)  time: 0.7408  data: 0.0210  max mem: 3282
Epoch: [0]  [ 60/335]  eta: 0:04:05  lr: 0.003669  loss: 0.7924 (1.0887)  loss_classifier: 0.2435 (0.3654)  loss_box_reg: 0.2361 (0.2983)  loss_objectness: 0.1289 (0.2105)  loss_rpn_box_reg: 0.1898 (0.2144)  time: 0.7244  data: 0.0127  max mem: 3432
Epoch: [0]  [ 80/335]  eta: 0:03:37  lr: 0.004865  loss: 0.7438 (1.0117)  loss_classifier: 0.2565 (0.3376)  loss_box_reg: 0.2193 (0.2799)  loss_objectness: 0.0776 (0.1835)  loss_rpn_box_reg: 0.1983 (0.2108)  time: 0.7217  data: 0.0127  max mem: 3432
Epoch: [0]  [100/335]  eta: 0:03:14  lr: 0.006062  loss: 0.7373 (0.9490)  loss_classifier: 0.2274 (0.3156)  loss_box_reg: 0.2193 (0.2654)  loss_objectness: 0.0757 (0.1643)  loss_rpn_box_reg: 0.1867 (0.2037)  time: 0.7291  data: 0.0132  max mem: 3432
Epoch: [0]  [120/335]  eta: 0:02:54  lr: 0.007258  loss: 0.8275 (0.9243)  loss_classifier: 0.2689 (0.3094)  loss_box_reg: 0.2315 (0.2602)  loss_objectness: 0.0867 (0.1539)  loss_rpn_box_reg: 0.1883 (0.2008)  time: 0.7270  data: 0.0134  max mem: 3432
Epoch: [0]  [140/335]  eta: 0:02:35  lr: 0.008455  loss: 0.7886 (0.9057)  loss_classifier: 0.2573 (0.3029)  loss_box_reg: 0.2246 (0.2539)  loss_objectness: 0.0724 (0.1455)  loss_rpn_box_reg: 0.2459 (0.2035)  time: 0.7170  data: 0.0124  max mem: 3432
Epoch: [0]  [160/335]  eta: 0:02:17  lr: 0.009651  loss: 0.7588 (0.8878)  loss_classifier: 0.2341 (0.2948)  loss_box_reg: 0.2226 (0.2486)  loss_objectness: 0.1032 (0.1427)  loss_rpn_box_reg: 0.2020 (0.2016)  time: 0.7139  data: 0.0118  max mem: 3432
Epoch: [0]  [180/335]  eta: 0:02:01  lr: 0.010847  loss: 0.7340 (0.8744)  loss_classifier: 0.2331 (0.2898)  loss_box_reg: 0.2120 (0.2441)  loss_objectness: 0.1086 (0.1392)  loss_rpn_box_reg: 0.1993 (0.2012)  time: 0.7800  data: 0.0584  max mem: 3432
Epoch: [0]  [200/335]  eta: 0:01:45  lr: 0.012044  loss: 0.8106 (0.8694)  loss_classifier: 0.2616 (0.2873)  loss_box_reg: 0.2208 (0.2411)  loss_objectness: 0.1117 (0.1397)  loss_rpn_box_reg: 0.1927 (0.2014)  time: 0.7344  data: 0.0143  max mem: 3432
Epoch: [0]  [220/335]  eta: 0:01:29  lr: 0.013240  loss: 0.8191 (0.8610)  loss_classifier: 0.2581 (0.2848)  loss_box_reg: 0.2140 (0.2382)  loss_objectness: 0.0860 (0.1362)  loss_rpn_box_reg: 0.2177 (0.2018)  time: 0.7213  data: 0.0126  max mem: 3432
Epoch: [0]  [240/335]  eta: 0:01:13  lr: 0.014437  loss: 0.7890 (0.8590)  loss_classifier: 0.2671 (0.2842)  loss_box_reg: 0.2094 (0.2357)  loss_objectness: 0.1175 (0.1360)  loss_rpn_box_reg: 0.2256 (0.2030)  time: 0.7576  data: 0.0564  max mem: 3432
Epoch: [0]  [260/335]  eta: 0:00:57  lr: 0.015633  loss: 0.8631 (0.8587)  loss_classifier: 0.2900 (0.2849)  loss_box_reg: 0.2089 (0.2337)  loss_objectness: 0.0925 (0.1350)  loss_rpn_box_reg: 0.2271 (0.2050)  time: 0.7371  data: 0.0220  max mem: 3432
Epoch: [0]  [280/335]  eta: 0:00:42  lr: 0.016830  loss: 0.8464 (0.8580)  loss_classifier: 0.2679 (0.2840)  loss_box_reg: 0.2156 (0.2321)  loss_objectness: 0.0940 (0.1346)  loss_rpn_box_reg: 0.2345 (0.2073)  time: 0.7379  data: 0.0143  max mem: 3432
Epoch: [0]  [300/335]  eta: 0:00:27  lr: 0.018026  loss: 0.7991 (0.8519)  loss_classifier: 0.2485 (0.2819)  loss_box_reg: 0.2125 (0.2305)  loss_objectness: 0.0819 (0.1315)  loss_rpn_box_reg: 0.2217 (0.2080)  time: 0.8549  data: 0.1419  max mem: 3450
Epoch: [0]  [320/335]  eta: 0:00:11  lr: 0.019222  loss: 0.6906 (0.8432)  loss_classifier: 0.2362 (0.2791)  loss_box_reg: 0.2036 (0.2285)  loss_objectness: 0.0662 (0.1285)  loss_rpn_box_reg: 0.1801 (0.2070)  time: 0.7257  data: 0.0238  max mem: 3450
Epoch: [0]  [334/335]  eta: 0:00:00  lr: 0.020000  loss: 0.7822 (0.8441)  loss_classifier: 0.2501 (0.2785)  loss_box_reg: 0.2224 (0.2285)  loss_objectness: 0.1135 (0.1296)  loss_rpn_box_reg: 0.1948 (0.2075)  time: 0.7249  data: 0.0139  max mem: 3450
Epoch: [0] Total time: 0:04:18 (0.7707 s / it)
Traceback (most recent call last):
  File "train_rcnn.py", line 143, in <module>
    main(args)
  File "train_rcnn.py", line 109, in main
    evaluate(model, data_loader_test, device=device)
  File "C:\Users\___\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\autograd\grad_mode.py", line 49, in decorate_no_grad
    return func(*args, **kwargs)
  File "D:\Research Report\tensorflow\Mask_RCNN-TRIALS\Mask_RCNN-master\samples\apples\utility\engine.py", line 78, in evaluate
    coco = get_coco_api_from_dataset(data_loader.dataset)
  File "D:\Research Report\tensorflow\Mask_RCNN-TRIALS\Mask_RCNN-master\samples\apples\utility\coco_utils.py", line 205, in get_coco_api_from_dataset
    return convert_to_coco_api(dataset)
  File "D:\Research Report\tensorflow\Mask_RCNN-TRIALS\Mask_RCNN-master\samples\apples\utility\coco_utils.py", line 154, in convert_to_coco_api
    img, targets = ds[img_idx]
  File "D:\Research Report\tensorflow\Mask_RCNN-TRIALS\Mask_RCNN-master\samples\apples\data\apple_dataset.py", line 22, in __getitem__
    mask_path = os.path.join(self.root_dir, "masks", self.masks[idx])
IndexError: list index out of range

This is the file that is run in order to train the network

import datetime
import os
import time

import torch
import torch.utils.data
import torchvision
from torchvision.models.detection.faster_rcnn import FastRCNNPredictor
from torchvision.models.detection.mask_rcnn import MaskRCNNPredictor

from data.apple_dataset import AppleDataset
from utility.engine import train_one_epoch, evaluate

import utility.utils as utils
import utility.transforms as T

######################################################
# Train either a Faster-RCNN or Mask-RCNN predictor
# using the MinneApple dataset
######################################################

def get_transform(train):
    transforms = []
    transforms.append(T.ToTensor())
    if train:
        transforms.append(T.RandomHorizontalFlip(0.5))
    return T.Compose(transforms)

def get_maskrcnn_model_instance(num_classes):
    # load an instance segmentation model pre-trained pre-trained on COCO
    model = torchvision.models.detection.maskrcnn_resnet50_fpn(pretrained=True)

    # get number of input features for the classifier
    in_features = model.roi_heads.box_predictor.cls_score.in_features
    # replace the pre-trained head with a new one
    model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)

    # now get the number of input features for the mask classifier
    in_features_mask = model.roi_heads.mask_predictor.conv5_mask.in_channels
    hidden_layer = 256
    # and replace the mask predictor with a new one
    model.roi_heads.mask_predictor = MaskRCNNPredictor(in_features_mask, hidden_layer, num_classes)
    return model

def get_frcnn_model_instance(num_classes):
    # load an instance segmentation model pre-trained pre-trained on COCO
    model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)

    # get number of input features for the classifier
    in_features = model.roi_heads.box_predictor.cls_score.in_features
    # replace the pre-trained head with a new one
    model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)
    return model

def main(args):
    print(args)
    device = args.device

    # Data loading code
    print("Loading data")
    num_classes = 2
    dataset = AppleDataset(os.path.join(args.data_path, 'train'), get_transform(train=True))
    dataset_test = AppleDataset(os.path.join(args.data_path, 'test'), get_transform(train=False))

    print("Creating data loaders")
    data_loader = torch.utils.data.DataLoader(dataset, batch_size=args.batch_size, shuffle=True,
                                              num_workers=args.workers, collate_fn=utils.collate_fn)

    data_loader_test = torch.utils.data.DataLoader(dataset_test, batch_size=1,
                                                   shuffle=False, num_workers=args.workers,
                                                   collate_fn=utils.collate_fn)

    print("Creating model")
    # Create the correct model type
    if args.model == 'maskrcnn':
        model = get_maskrcnn_model_instance(num_classes)
    else:
        model = get_frcnn_model_instance(num_classes)

    # Move model to the right device
    model.to(device)

    params = [p for p in model.parameters() if p.requires_grad]
    optimizer = torch.optim.SGD(params, lr=args.lr, momentum=args.momentum, weight_decay=args.weight_decay)

    #  lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=args.lr_step_size, gamma=args.lr_gamma)
    lr_scheduler = torch.optim.lr_scheduler.MultiStepLR(optimizer, milestones=args.lr_steps, gamma=args.lr_gamma)

    if args.resume:
        checkpoint = torch.load(args.resume, map_location='cpu')
        model.load_state_dict(checkpoint['model'])
        optimizer.load_state_dict(checkpoint['optimizer'])
        lr_scheduler.load_state_dict(checkpoint['lr_scheduler'])

    print("Start training")
    start_time = time.time()
    for epoch in range(args.epochs):
        train_one_epoch(model, optimizer, data_loader, device, epoch, args.print_freq)
        lr_scheduler.step()

        if args.output_dir:
            torch.save(model.state_dict(), os.path.join(args.output_dir, 'model_{}.pth'.format(epoch)))

        # evaluate after every epoch
        evaluate(model, data_loader_test, device=device)

    total_time = time.time() - start_time
    total_time_str = str(datetime.timedelta(seconds=int(total_time)))
    print('Training time {}'.format(total_time_str))

if __name__ == "__main__":
    import argparse
    parser = argparse.ArgumentParser(description='PyTorch Detection Training')
    parser.add_argument('--data_path', default='~~~~', help='dataset')
    parser.add_argument('--dataset', default='AppleDataset', help='dataset')
    parser.add_argument('--model', default='maskrcnn', help='model')
    parser.add_argument('--device', default='cuda', help='device')
    parser.add_argument('-b', '--batch-size', default=2, type=int)
    parser.add_argument('--epochs', default=13, type=int, metavar='N', help='number of total epochs to run')
    parser.add_argument('-j', '--workers', default=4, type=int, metavar='N', help='number of data loading workers (default: 16)')
    parser.add_argument('--lr', default=0.02, type=float, help='initial learning rate')
    parser.add_argument('--momentum', default=0.9, type=float, metavar='M', help='momentum')
    parser.add_argument('--wd', '--weight-decay', default=1e-4, type=float, metavar='W', help='weight decay (default: 1e-4)', dest='weight_decay')
    parser.add_argument('--lr-step-size', default=8, type=int, help='decrease lr every step-size epochs')
    parser.add_argument('--lr-steps', default=[8, 11], nargs='+', type=int, help='decrease lr every step-size epochs')
    parser.add_argument('--lr-gamma', default=0.1, type=float, help='decrease lr by a factor of lr-gamma')
    parser.add_argument('--print-freq', default=20, type=int, help='print frequency')
    parser.add_argument('--output-dir', default='.', help='path where to save')
    parser.add_argument('--resume', default='', help='resume from checkpoint')

    args = parser.parse_args()
    print(args.model)
    assert(args.model in ['mrcnn', 'frcnn'])

    if args.output_dir:
        utils.mkdir(args.output_dir)

    main(args)

Apple_dataset.py is as follows

import os
import numpy as np
import torch
from PIL import Image

#####################################
# Class that takes the input instance masks
# and extracts bounding boxes on the fly
#####################################
class AppleDataset(object):
    def __init__(self, root_dir, transforms):
        self.root_dir = root_dir
        self.transforms = transforms

        # Load all image and mask files, sorting them to ensure they are aligned
        self.imgs = list(sorted(os.listdir(os.path.join(root_dir, "images"))))
        self.masks = list(sorted(os.listdir(os.path.join(root_dir, "masks"))))

    def __getitem__(self, idx):
        # Load images and masks
        img_path = os.path.join(self.root_dir, "images", self.imgs[idx])
        mask_path = os.path.join(self.root_dir, "masks", self.masks[idx])

        img = Image.open(img_path).convert("RGB")
        mask = Image.open(mask_path)     # Each color of mask corresponds to a different instance with 0 being the background

        # Convert the PIL image to np array
        mask = np.array(mask)
        obj_ids = np.unique(mask)

        # Remove background id
        obj_ids = obj_ids[1:]

        # Split the color-encoded masks into a set of binary masks
        masks = mask == obj_ids[:, None, None]

        # Get bbox coordinates for each mask
        num_objs = len(obj_ids)
        boxes = []
        h, w = mask.shape
        for ii in range(num_objs):
            pos = np.where(masks[ii])
            xmin = np.min(pos[1])
            xmax = np.max(pos[1])
            ymin = np.min(pos[0])
            ymax = np.max(pos[0])

            if xmin == xmax or ymin == ymax:
                continue

            xmin = np.clip(xmin, a_min=0, a_max=w)
            xmax = np.clip(xmax, a_min=0, a_max=w)
            ymin = np.clip(ymin, a_min=0, a_max=h)
            ymax = np.clip(ymax, a_min=0, a_max=h)
            boxes.append([xmin, ymin, xmax, ymax])

        # Convert everything into a torch.Tensor
        boxes = torch.as_tensor(boxes, dtype=torch.float32)

        # There is only one class (apples)
        labels = torch.ones((num_objs,), dtype=torch.int64)
        masks = torch.as_tensor(masks, dtype=torch.uint8)

        image_id = torch.tensor([idx])
        area = (boxes[:, 3] - boxes[:, 1]) * (boxes[:, 2] - boxes[:, 0])

        # All instances are not crowd
        iscrowd = torch.zeros((num_objs,), dtype=torch.int64)

        target = {}
        target["boxes"] = boxes
        target["labels"] = labels
        target["masks"] = masks
        target["image_id"] = image_id
        target["area"] = area
        target["iscrowd"] = iscrowd

        if self.transforms is not None:
            img, target = self.transforms(img, target)

        return img, target

    def __len__(self):
        return len(self.imgs)

    def get_img_name(self, idx):
        return self.imgs[idx]

How do i fix the index going out of range??? OR what is the underlying issue that needs to be addressed here??

EDIT1: ok.. so whats happening here is i have two folders "train" and "test".. the training folder has the images and masks, while test folder has only images.. the apple_dataset.py is written such that its looking for masks folder in both train and test folders.. i think i need to change the code such that it looks for masks in only the train folder and not the test set


I covert onnxmodel to pb, and get this problem:
Traceback (most recent call last):
  File "/home/fffan/fffan_files/Experiment/Example/onnx2pb/onnx2pb.py", line 45, in <module>
    onnx2pb_2(onnx_input_path)
  File "/home/fffan/fffan_files/Experiment/Example/onnx2pb/onnx2pb.py", line 14, in onnx2pb_2
    tf_rep = prepare(model,strict=False)
  File "/home/fffan/下载/onnx-tensorflow-tf-1.x/onnx_tf/backend.py", line 65, in prepare
    return cls.onnx_model_to_tensorflow_rep(model, strict)
  File "/home/fffan/下载/onnx-tensorflow-tf-1.x/onnx_tf/backend.py", line 85, in onnx_model_to_tensorflow_rep
    return cls._onnx_graph_to_tensorflow_rep(model.graph, opset_import, strict)
  File "/home/fffan/下载/onnx-tensorflow-tf-1.x/onnx_tf/backend.py", line 146, in _onnx_graph_to_tensorflow_rep
    strict=strict)
  File "/home/fffan/下载/onnx-tensorflow-tf-1.x/onnx_tf/backend.py", line 241, in _onnx_node_to_tensorflow_op
    return handler.handle(node, tensor_dict=tensor_dict, strict=strict)
  File "/home/fffan/下载/onnx-tensorflow-tf-1.x/onnx_tf/handlers/handler.py", line 60, in handle
    cls.args_check(node, **kwargs)
  File "/home/fffan/下载/onnx-tensorflow-tf-1.x/onnx_tf/handlers/backend/resize.py", line 89, in args_check
    "Tensorflow")
  File "/home/fffan/下载/onnx-tensorflow-tf-1.x/onnx_tf/common/exception.py", line 50, in __call__
    raise self._func(self.get_message(op, framework))
RuntimeError: Resize coordinate_transformation_mode=pytorch_half_pixel is not supported in Tensorflow.

The code is :
import onnx
from onnx_tf.backend import prepare
import numpy as np
model = onnx.load(onnx_input_path)
tf_rep = prepare(model,strict=False)

How can I solve this problem?
ps: The algorithm model I covert is PSEnet.


title: fine tune causal language model using transformers and pytorch

body:
I have some questions about fine-tuning causal language model using transformers and PyTorch.
My main goal is to fine-tune XLNet. However, I found the most of posts online was targeting at text classification, like this post. I was wondering, is there any way to fine-tune the model, without using the run_language_model.py from transformers' GitHub?
Here is a piece of my code trying to fine-tune XLNet:
model = XLNetLMHeadModel.from_pretrained("xlnet-base-cased")
tokenizer = XLNetTokenizer.from_pretrained("xlnet-base-cased", do_lower_case=True)
LOSS = torch.nn.CrossEntrypoLoss()
batch_texts = ["this is sentence 1", "i have another sentence like this", "the final sentence"]
encodings = tokenizer.encode_plus(batch_texts, add_special_tokens=True,
                                  return_tensors=True, return_attention_mask=True)
outputs = model(encodings["input_ids"], encodings["attention_mask"])
loss = LOSS(outputs[0], target_ids)
loss.backward()
# ignoring the rest of codes...

I got stuck at the last two lines. At first, when using this LM model, it seems I don't have any labels as the supervised learning usually do; Second, as the language model which is to minimize the loss (cross-entropy here), I need a target_ids to compute the loss and perplexity with input_ids.
Here are my follow-up questions:

How should I deal with this labels during the model fitting?
Should I set something like target_ids=encodings["input_ids"].copy() to compute cross-entropy loss and perplexity?
If not, how should set this target_ids?
From the perplexity page from transformers' documentation, how should I adapt its method for non-fixed length of input text?
I saw another post from the documentation saying that it requires padding text for causal language modeling. However, from the link in 3), there is no such sign for padding text. Which one should I follow?

Any suggestions and advice will be appreciated!


title: Is pytorch 1.7 officially enabled for cuda 10.0?

body:
I had to stay on CUDA 10.0 for personal projects.
Rather than installing Pytorch with versions appropriate for CUDA 10.0, I accidentally installed Pytorch 1.7 supported with CUDA 10.1. In particular, I installed by
pip install torch==1.7.1+cu101 torchvision==0.8.2+cu101 torchaudio==0.7.2 -f https://download.pytorch.org/whl/torch_stable.html

Surprisingly, everything works fine so far although the CUDA versions do not match.
To verify my installation, I've run the code given in collect_env.py, and it was fine.
I am just wondering few things.

Did Pytorch team officially comment Pytorch 1.7 is compatible with CUDA 10.0?
Would there be more rigorous ways to verify my Pytorch installation?


title: Error 403: Forbidden when trying to load pre-trained model in Google colab

body:
Here is my code to load a pre-trained model in a Google Colab notebookL

# Import resources
%matplotlib inline
%config InlineBackend.figure_format = 'retina'
import time
import json
import copy
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np
import PIL
from PIL import Image
from collections import OrderedDict
import torch
from torch import nn, optim, cuda
from torch.optim import lr_scheduler
from torch.autograd import Variable
import torchvision
from torchvision import datasets, models, transforms
from torch.utils.data.sampler import SubsetRandomSampler
import torch.nn as nn
import torch.nn.functional as F
import os

device = torch.device("cuda" if torch.cuda.is_available()
                                  else "cpu")
print(device)

model = models.resnet152(pretrained=True)
num_in_features = 2048
print(model)

I have run this line before with success, however now I get the error:

Downloading: "https://download.pytorch.org/models/resnet152-b121ed2d.pth" to /root/.cache/torch/checkpoints/resnet152-b121ed2d.pth
---------------------------------------------------------------------------
HTTPError                                 Traceback (most recent call last)
<ipython-input-13-361fc480b515> in <module>()
----> 1 model = models.resnet152(pretrained=True)
      2 num_in_features = 2048
      3 print(model)

9 frames
/usr/lib/python3.6/urllib/request.py in http_error_default(self, req, fp, code, msg, hdrs)
    648 class HTTPDefaultErrorHandler(BaseHandler):
    649     def http_error_default(self, req, fp, code, msg, hdrs):
--> 650         raise HTTPError(req.full_url, code, msg, hdrs, fp)
    651
    652 class HTTPRedirectHandler(BaseHandler):

HTTPError: HTTP Error 403: Forbidden

I have tried loading the model in various other ways, starting a new notebook, and !kill -9 -1

I'm new to Google Colab and am not sure what the issue really is here! Thanks in advance


title: How to solve size mismatch error in pytorch?

body:
I am trying to create a logistic model by using CIFAR10 data in PyTorch. After running the model for evaluation I run into an error :

  RuntimeError: size mismatch, m1: [750 x 4096], m2: [1024 x 10] at C:\w\1\s\tmp_conda_3.7_100118\conda\conda-bld\pytorch_1579082551706\work\aten\src\TH/generic/THTensorMath.cpp:136

It seems like input_size is creating a problem, I dont know I am new to this. Please let me know what changes should I make in order to overcome this error.

These are the hyperparameters:

batch_size = 100
learning_rate = 0.001

# Other constants
input_size = 4*4*64
num_classes = 10

This is the cell that downloads and splits the dataset into train, validation and test.

transform = torchvision.transforms.Compose(
    [torchvision.transforms.ToTensor(),
     torchvision.transforms.Normalize((0.5,0.5,0.5), (0.5,0.5,0.5))])

testset = torchvision.datasets.CIFAR10(root='D:\PyTorch\cifar-10-python', train=False,download=False, transform=transform)
trainvalset = torchvision.datasets.CIFAR10(root='D:\PyTorch\cifar-10-python', train=True,download=False, transform=transform)
trainset, valset = torch.utils.data.random_split(trainvalset, [45000, 5000]) # 10% for validation

train_loader = torch.utils.data.DataLoader(trainset, batch_size=50, shuffle=True)
test_loader = torch.utils.data.DataLoader(testset, batch_size=1000, shuffle=False)
val_loader = torch.utils.data.DataLoader(valset, batch_size=1000, shuffle=False)

This is the architecture of my model.

class CifarModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(input_size,  num_classes)
    def forward(self, xb):
        xb = xb.view(-1, 64*8*8)
        #xb = xb.reshape(-1, 784)
        print(xb.shape)
        out = self.linear(xb)
        return out

    def training_step(self, batch):
        images, labels = batch
        out = self(images)                  # Generate predictions
        loss = F.cross_entropy(out, labels) # Calculate loss
        return loss

    def validation_step(self, batch):
        images, labels = batch
        out = self(images)                    # Generate predictions
        loss = F.cross_entropy(out, labels)   # Calculate loss
        acc = accuracy(out, labels)           # Calculate accuracy
        return {'val_loss': loss.detach(), 'val_acc': acc.detach()}

    def validation_epoch_end(self, outputs):
        batch_losses = [x['val_loss'] for x in outputs]
        epoch_loss = torch.stack(batch_losses).mean()   # Combine losses
        batch_accs = [x['val_acc'] for x in outputs]
        epoch_acc = torch.stack(batch_accs).mean()      # Combine accuracies
        return {'val_loss': epoch_loss.item(), 'val_acc': epoch_acc.item()}

    def epoch_end(self, epoch, result):
        print("Epoch [{}], val_loss: {:.4f}, val_acc: {:.4f}".format(epoch, result['val_loss'], result['val_acc']))

model = CifarModel()

def accuracy(outputs, labels):
    _, preds = torch.max(outputs, dim=1)
    return torch.tensor(torch.sum(preds == labels).item() / len(preds))
def evaluate(model, val_loader):
    outputs = [model.validation_step(batch) for batch in val_loader]
    return model.validation_epoch_end(outputs)

def fit(epochs, lr, model, train_loader, val_loader, opt_func=torch.optim.SGD):
    history = []
    optimizer = opt_func(model.parameters(), lr)
    for epoch in range(epochs):
        # Training Phase
        for batch in train_loader:
            loss = model.training_step(batch)
            loss.backward()
            optimizer.step()
            optimizer.zero_grad()
        # Validation phase
        result = evaluate(model, val_loader)
        model.epoch_end(epoch, result)
        history.append(result)
    return history
evaluate(model, val_loader)


title: Warp-ctc can not run on gpu

body:
I have some variable saved from crnn-pytorch, which runs on gpu and uses warp-ctc.

warp-ctc: https://github.com/SeanNaren/warp-ctc

Then I use those variables in my code, I can run it on cpu and the results are correct. But when I run it on gpu, I get Segmentation fault error.
This is my code which runs on gpu:

import torch
from warpctc_pytorch import CTCLoss
import numpy as np
from torch.autograd import Variable

i = 1
criterion = CTCLoss()
criterion = criterion.cuda()
a = np.load('preds_%d.npy' % i)
b = np.load('text_%d.npy' % i)
c = np.load('preds_size_%d.npy' % i)
d = np.load('length_%d.npy' % i)

#a = Variable(torch.from_numpy(a)).cuda()
#b = Variable(torch.from_numpy(b)).cuda()
#c = Variable(torch.from_numpy(c)).cuda()
#d = Variable(torch.from_numpy(d)).cuda()

a = torch.from_numpy(a).cuda()
b = torch.from_numpy(b).cuda()
c = torch.from_numpy(c).cuda()
d = torch.from_numpy(d).cuda()

print(a.dtype)
print(b.dtype)
print(c.dtype)
print(d.dtype)

cost = criterion(a, b, c, d) / 64
print('cost:', cost)

I get Segmentation fault. When delete all .cuda(), I get correct answer.
All cpu and gpu test has passed.
I really hope someone can help.


title: How to convert BatchNorm weight of caffe to pytorch BathNorm?

body:
BathNorm and Scale weight of caffe model can be read from pycaffe, which are three weights in BatchNorm and two weights in Scale. I tried to copy those weights to pytorch BatchNorm with codes like this:

if 'conv3_final_bn' == name:
    assert len(blobs) == 3, '{} layer blob count: {}'.format(name, len(blobs))
    torch_mod['conv3_final_bn.running_mean'] = blobs[0].data
    torch_mod['conv3_final_bn.running_var'] = blobs[1].data
elif 'conv3_final_scale' == name:
    assert len(blobs) == 2, '{} layer blob count: {}'.format(name, len(blobs))
    torch_mod['conv3_final_bn.weight'] = blobs[0].data
    torch_mod['conv3_final_bn.bias'] = blobs[1].data

The two BatchNorm acts differently. I also tried to set conv3_final_bn.weight=1 and conv3_final_bn.bias=0 to verify the BN layer of caffe, the results didn't match either.

How should I deal with the wrong matching?


title: AWS tensorboard Segmentation fault (core dumped)

body:
I am trying to use tensorboardX to debug a pytorch NN that is running in a p2.xlarge instance of AWS.

I followed this tutorial to open the port 6006.

The model is running and tensorboardX is making its writer file. I get the following warning there. I am not sure how relevant it is.

  WARNING:root:tuple appears in op that does not forward tuples
  (VisitNode at /pytorch/torch/csrc/jit/passes/lower_tuples.cpp:117)
  frame #0: std::function::operator()() const + 0x11
  (0x7fbe3dd04441 in
  /home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/lib/libc10.so)
  frame #1: c10::Error::Error(c10::SourceLocation, std::string const&) +
  0x2a (0x7fbe3dd03d7a in
  /home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/lib/libc10.so)
  frame #2:  + 0xaf61f5 (0x7fbe3cdc41f5 in
  /home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/lib/libtorch.so.1)
  frame #3:  + 0xaf6464 (0x7fbe3cdc4464 in
  /home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/lib/libtorch.so.1)
  frame #4:
  torch::jit::LowerAllTuples(std::shared_ptr&) + 0x13
  (0x7fbe3cdc44a3 in
  /home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/lib/libtorch.so.1)
  frame #5:  + 0x3f84b4 (0x7fbe7d2cb4b4 in
  /home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
  frame #6:  + 0x130cfc (0x7fbe7d003cfc in
  /home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
   frame #40: __libc_start_main + 0xf0
  (0x7fbe8d69c830 in /lib/x86_64-linux-gnu/libc.so.6)

  WARNING:root:tuple appears in op that does not forward tuples
  (VisitNode at /pytorch/torch/csrc/jit/passes/lower_tuples.cpp:117)
  frame #0: std::function::operator()() const + 0x11
  (0x7fbe3dd04441 in
  /home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/lib/libc10.so)
  frame #1: c10::Error::Error(c10::SourceLocation, std::string const&) +
  0x2a (0x7fbe3dd03d7a in
  /home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/lib/libc10.so)
  frame #2:  + 0xaf61f5 (0x7fbe3cdc41f5 in
  /home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/lib/libtorch.so.1)
  frame #3:  + 0xaf6464 (0x7fbe3cdc4464 in
  /home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/lib/libtorch.so.1)
  frame #4:
  torch::jit::LowerAllTuples(std::shared_ptr&) + 0x13
  (0x7fbe3cdc44a3 in
  /home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/lib/libtorch.so.1)
  frame #5:  + 0x3f84b4 (0x7fbe7d2cb4b4 in
  /home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
  frame #6:  + 0x130cfc (0x7fbe7d003cfc in
  /home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
   frame #40: __libc_start_main + 0xf0
  (0x7fbe8d69c830 in /lib/x86_64-linux-gnu/libc.so.6)

The problem is that I don't have access to the tensorboard browser user interface. I take the following steps:

$ cd PATH_TO_FOLDER_CONTAINING_runs
$ source activate pytorch_p36
$ tensorboard --logdir=runs

Where I get the error message:

  Segmentation fault (core dumped)

When I check the syslog var/log/syslog I see that following:

  Jun 26 09:06:40 ip-172-xx-xx-xxx kernel: [515315.598917] tensorboard[1446]: segfault at 0 ip           (null) sp 00007ffd64c5f178 error 14 in python2.7[55d8673d1000+1000]

My googling skills were far from enough. How can I access tensorboard through the browser with it running in the ASW instance?

Please let me know if something is unclear or if some info is missing.


title: Vectorizing assignment of a tensor to a slice in PyTorch

body:
I'm trying to vectorize a slice assignment of the form

for i in range(a.shape[1]):
    for j in range(a.shape[2]):
        a[:,i,j,:,i:i+b.shape[2],j:j+b.shape[3]] = b

where b itself is an array. This is because the nested Python loop is too inefficient and is taking up most of the runtime. Is there a way to do this?

For a simpler case, consider the following:

for i in range(a.shape[1]):
    a[:,i,:,i:i+b.shape[2]] = b

This is what b and a might look like:

You can see the diagonal, "sliding" structure of the resulting matrix.


title: How can I implement a checkerboard stride for Conv2d in pytorch?

body: I am trying to create a convnet using pytorch to work on an input of 2d matrices. I am using a 3x5 filter and I want it to have a custom stride as follows - on even line numbers I want the filter to start from the element at position 0 (red in the image), on odd line numbers I want it to start on the element of position 1 (blue in the image), and in both cases have a stride of 2 on the x direction. That means that if I have a matrix as in the image as my input, I want the filter to have only 0s in its center. I know this is very unusual in convnets but this is actually a problem in physics so the exact stride is important.


title: Is there any diffrence between index_select and tensor[sequence] in PyTorch?

body:
everyone. I'm new to PyTorch. Now I'm learning the indexing of a tensor. I notice that we can indexing a tensor by tensor.index_select() and tensor[sequence].

In [1]: x = torch.randn(3, 4)

In [2]: indices = torch.tensor([0, 2])

In [3]: x.index_select(0, indices)
Out[3]:
tensor([[ 0.2760, -0.9543, -1.0499,  0.7828],
        [ 1.3514, -1.1289,  0.5052, -0.0547]])

In [4]: x[[0,2]]
Out[4]:
tensor([[ 0.2760, -0.9543, -1.0499,  0.7828],
        [ 1.3514, -1.1289,  0.5052, -0.0547]])

I am puzzled about these two methods and look for some doc. But I failed. Can anyone can tell me are there some differences between them and what are these difference?


title: Loss on dev set is always increasing unlike training set loss

body:
I designed a network for a text classification problem. To do this, I'm using huggingface transformet's BERT model with a linear layer above that for fine-tuning. My problem is that the loss on the training set is decreasing which is fine, but when it comes to do the evaluation after each epoch on the development set, the loss is increasing with epochs. I'm posting my code to investigate if there's something wrong with it.

for epoch in range(1, args.epochs + 1):
    total_train_loss = 0
    trainer.set_train()

    for step, batch in enumerate(train_dataloader):
        loss = trainer.step(batch)
        total_train_loss += loss

    avg_train_loss = total_train_loss / len(train_dataloader)

    logger.info(('Training loss for epoch %d/%d: %4.2f') % (epoch, args.epochs, avg_train_loss))

    print("\n-------------------------------")
    logger.info('Start validation ...')
    trainer.set_eval()
    y_hat = list()
    y = list()
    total_dev_loss = 0
    for step, batch_val in enumerate(dev_dataloader):
        true_labels_ids, predicted_labels_ids, loss = trainer.validate(batch_val)
        total_dev_loss += loss
        y.extend(true_labels_ids)
        y_hat.extend(predicted_labels_ids)
    avg_dev_loss = total_dev_loss / len(dev_dataloader)
    print(("\n-Total dev loss: %4.2f on epoch %d/%d\n") % (avg_dev_loss, epoch, args.epochs))

print("Training terminated!")

Following is the trainer file, which I use for doing a forward pass on a given batch and then backpropagate accordingly.

class Trainer(object):
    def __init__(self, args, model, device, data_points, is_test=False, train_stats=None):
        self.args = args
        self.model = model
        self.device = device
        self.loss = nn.CrossEntropyLoss(reduction='none')

        if is_test:
            # Should load the model from checkpoint
            self.model.eval()
            self.model.load_state_dict(torch.load(args.saved_model))
            logger.info('Loaded saved model from %s' % args.saved_model)

        else:
            self.model.train()
            self.optim = AdamW(model.parameters(), lr=2e-5, eps=1e-8)
            total_steps = data_points * self.args.epochs
            self.scheduler = get_linear_schedule_with_warmup(self.optim, num_warmup_steps=0,
                                                             num_training_steps=total_steps)

    def step(self, batch):
        batch = tuple(t.to(self.device) for t in batch)
        batch_input_ids, batch_input_masks, batch_labels = batch
        self.model.zero_grad()
        outputs = self.model(batch_input_ids,
                             attention_mask=batch_input_masks,
                             labels=batch_labels)
        loss = self.loss(outputs, batch_labels)
        loss = loss.sum()
        (loss / loss.numel()).backward()
        torch.nn.utils.clip_grad_norm_(self.model.parameters(), 1.0)
        self.optim.step()
        self.scheduler.step()
        return loss

    def validate(self, batch):
        batch = tuple(t.to(self.device) for t in batch)
        batch_input_ids, batch_input_masks, batch_labels = batch
        with torch.no_grad():
            model_output = self.model(batch_input_ids,
                                 attention_mask=batch_input_masks,
                                 labels=batch_labels)

        predicted_label_ids = self._predict(model_output)
        label_ids = batch_labels.to('cpu').numpy()

        loss = self.loss(model_output, batch_labels)
        loss = loss.sum()

        return label_ids, predicted_label_ids, loss

    def _predict(self, logits):
        return np.argmax(logits.to('cpu').numpy(), axis=1)

Finally, the following is my model (i.e., Classifier) class:

import torch.nn as nn
from transformers import BertModel

class Classifier(nn.Module):

    def __init__(self, args, is_eval=False):
        super(Classifier, self).__init__()

        self.bert_model = BertModel.from_pretrained(
            args.init_checkpoint,
            output_attentions=False,
            output_hidden_states=True,
        )
        self.is_eval_mode = is_eval
        self.linear = nn.Linear(768, 2) # binary classification

    def switch_state(self):
        self.is_eval_mode = not self.is_eval_mode

    def forward(self, input_ids, attention_mask=None, labels=None):

        bert_outputs = self.bert_model(input_ids,
                                       token_type_ids=None,
                                       attention_mask=attention_mask)

        # Should give the logits to the the linear layer
        model_output = self.linear(bert_outputs[1])

        return model_output

For visualization the loss throughout the epochs:


title: pytorch tensor stride - how it works

body:
PyTorch doesn't seem to have documentation for tensor.stride().
Can someone confirm my understanding?
My questions are three-fold.

Stride is for accessing an element in the storage. So stride size will be the same as the dimension of the tensor. Correct?

For each dimension, the corresponding element of stride tells how much it takes to move along the 1-dimensional storage. Correct?

For example:
In [15]: x = torch.arange(1,25)

In [16]: x
Out[16]:
tensor([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17, 18
, 19, 20, 21, 22, 23, 24])

In [17]: a = x.view(4,3,2)

In [18]: a
Out[18]:
tensor([[[ 1,  2],
         [ 3,  4],
         [ 5,  6]],

        [[ 7,  8],
         [ 9, 10],
         [11, 12]],

        [[13, 14],
         [15, 16],
         [17, 18]],

        [[19, 20],
         [21, 22],
         [23, 24]]])

In [20]: a.stride()
Out[20]: (6, 2, 1)

How does having this information help perform tensor operations efficiently? Basically this is showing the memory layout. So how does it help?


title: Torch sum subsets of tensor

body:
if the tensor is of shape [20, 5] then I need to take 10 at a time and sum them, so result is [2,5].
eg:
shape[20,5] -> shape[2, 5] (sum 10 at a time)
shape[100, 20] -> shape[10,20] (sum 10 at a time)
Is there any faster/optimal way to do this?
eg:
[[1, 1], [1, 2], [3, 4], [1,2]] i want [[2, 3], [4, 6]] by taking sum of 2 rows.


title: In a Kaggle kernel while having selected the GPU option when checking torch.cuda.is_available(), it says is not available

body:
I created a kernel for a finished Kaggle competition in which i used pytorch. When checking if cuda is available, it returns False.

I checked the GPU option from settings and it says it is on in the bottom bar with resources info. I tried to restart the session without any changes.
What could be the problem? (cpu only pytorch installed maybe?)


title: Combined GRU and CNN network always returns the same value for all inputs

body:
I am trying to train a combined CNN and GRU/LSTM to find out the number of objetcs in a series of pictures that move and the number of objects that do not move. For this reason I am using a CNN to process my images and consequently use a GRU.
My problem is that the GRU always returns the same value for each input set. What could be reasons for that?

I have already tried to use different learning rates and adding linear layers after the GRU.

My network:

    class GRU(nn.Module):
        def __init__(self, **kwargs):
            super(GRU, self).__init__()
            self.n_class = int(kwargs.get("n_class"))
            self.seq_length = int(kwargs.get("seq_length"))
            self.input_shape = int(kwargs.get("input_shape"))
            self.n_channels = int(kwargs.get("n_channels"))
            self.conv1 = nn.Conv2d(in_channels=1 * seq_length, out_channels=4 * seq_length, kernel_size=5)
            self.conv2 = nn.Conv2d(in_channels=4 * seq_length, out_channels=8 * seq_length, kernel_size=5)
            self.conv3 = nn.Conv2d(in_channels=8 * seq_length, out_channels=16 * seq_length, kernel_size=5)
            self.rnn = nn.GRU(
                input_size=self.seq_length,
                hidden_size=64,
                num_layers=1,
                batch_first=True)
            self.linear = nn.Linear(64, 2)

        def forward(self, t):
            t = self.conv1(t)
            t = F.relu(t)
            t = F.max_pool2d(t, kernel_size=2, stride=2)
            # second conv layer
            t = self.conv2(t)
            t = F.relu(t)
            t = F.max_pool2d(t, kernel_size=4, stride=4)
            # third conv layer
            t = self.conv3(t)
            t = F.relu(t)
            t = F.max_pool2d(t, kernel_size=3, stride=3)
            t = t.reshape(-1 , self.seq_length, 16 * 20 ** 2)
            t = t.permute(0,2,1)
            t, (h_n) =self.rnn(t)
            t = self.linear(t[:,-1])
            return t

and this is my training:

for epoch in range(number_epochs):
    for batch in get_batch_generator(batch_size, rootdir, seq_length=seq_length):
        current_batch = batch[0].cuda()
        current_labels = batch[1].cuda()
        pre = nw(current_batch)
        loss_func = torch.nn.MSELoss()
        loss = loss_func(pre, current_labels)
        loss.backward()
        optimizer = optim.Adam(nw.parameters(), lr=learning_rate)
        optimizer.step()

Here is an example of the output, actual labels:

tensor([[ 4.,  5.],
        [10.,  0.],
        [10.,  0.],
        [ 2.,  9.],
        [ 5.,  1.],
        [10.,  0.]], device='cuda:0')

Prediction of my network:

tensor([[2.0280, 1.1517],
        [2.0175, 1.1593],
        [2.0323, 1.1434],
        [2.0333, 1.1557],
        [2.0200, 1.1546],
        [2.0069, 1.1687]], device='cuda:0', grad_fn=<AddmmBackward>)

So for both classes the output is the same for both classes (moving and not moving objects), which should not be the case.


title: Pytorch tensor multiplication with Float tensor giving wrong answer

body:
I am seeing some strange behavior when i multiply two pytorch tensors.

x = torch.tensor([99397544.0])
y = torch.tensor([0.1])
x * y

This outputs

tensor([9939755.])

However, the answer should be 9939754.4


title: How do I create a torch diagonal matrices with different element in each batch?

body:
I want to create a tensor like

 tensor([[[1,0,0],[0,1,0],[0,0,1]],[[2,0,0],[0,2,0],[0,0,2]]]])

That is, when a torch tensor B of size (1,n) is given, I want to create a torch tensor A of size (n,3,3) such that A[i] is an B[i] * (identity matrix of size 3x3).

Without using 'for sentence', how do I create this?


title: how to #include in a .cpp file with Visual Studio?

body:
Now I'm working with a project requiring using Pytorch C++ extension.
I've installed a Pytorch of version 1.4.0 in a python virtual environment:

activate crfasrnn
>>>import torch
>>>print(torch.__version__)
1.4.0

I'm using visual studio as a C++ compiler.
The code is like this:

#include<iostream>
#include<torch/extension.h>

int main()
{
    std::cout << "hello, world!" << std::endl;
    return 0;
}

However, the compiler tells me "can not open 'torch/extension.h'"
How can I solve this problem?