title: Recurrent network (RNN) won't learn a very simple function (plots shown in the question)

body:
So I am trying to train a simple recurrent network to detect a "burst" in an input signal. The following figure shows the input signal (blue) and the desired (classification) output of the RNN, shown in red.

So the output of the network should switch from 1 to 0 whenever the burst is detected and stay like with that output. The only thing that changes between the input sequences used to train the RNN is at which time step the burst occurs.

Following the Tutorial on https://github.com/MorvanZhou/PyTorch-Tutorial/blob/master/tutorial-contents/403_RNN_regressor.py, I cannot get a RNN to learn. The learned RNN always operates in a "memoryless" way, i.e., does not use memory to make its predictions, as shown in the following example behavior:

The green line shows the predicted output of the network. What do I do wrong in this example so that the network cannot be learned correctly? Isn't the network task quite simple?

I'm using:

torch.nn.CrossEntropyLoss as loss function
The Adam Optimizer for learning
A RNN with 16 internal/hidden nodes and 2 output nodes. They use the default activation function of the torch.RNN class.

The experiment has been repeated a couple of times with different random seeds, but there is little difference in the outcomes. I've used the following code:

import torch
import numpy, math
import matplotlib.pyplot as plt

nofSequences = 5
maxLength = 130

# Generate training data
x_np = numpy.zeros((nofSequences,maxLength,1))
y_np = numpy.zeros((nofSequences,maxLength))
numpy.random.seed(1)
for i in range(0,nofSequences):
    startPos = numpy.random.random()*50
    for j in range(0,maxLength):
        if j>=startPos and j<startPos+10:
            x_np[i,j,0] = math.sin((j-startPos)*math.pi/10)
        else:
            x_np[i,j,0] = 0.0
        if j<startPos+10:
            y_np[i,j] = 1
        else:
            y_np[i,j] = 0

# Define the neural network
INPUT_SIZE = 1
class RNN(torch.nn.Module):
    def __init__(self):
        super(RNN, self).__init__()

        self.rnn = torch.nn.RNN(
            input_size=INPUT_SIZE,
            hidden_size=16,     # rnn hidden unit
            num_layers=1,       # number of rnn layer
            batch_first=True,
        )
        self.out = torch.nn.Linear(16, 2)

    def forward(self, x, h_state):
        r_out, h_state = self.rnn(x, h_state)

        outs = []    # save all predictions
        for time_step in range(r_out.size(1)):    # calculate output for each time step
            outs.append(self.out(r_out[:, time_step, :]))
        return torch.stack(outs, dim=1), h_state

# Learn the network
rnn = RNN()
optimizer = torch.optim.Adam(rnn.parameters(), lr=0.01)
h_state = None      # for initial hidden state

x = torch.Tensor(x_np)    # shape (batch, time_step, input_size)
y = torch.Tensor(y_np).long()

torch.manual_seed(2)
numpy.random.seed(2)

for step in range(100):

    prediction, h_state = rnn(x, h_state)   # rnn output

    # !! next step is important !!
    h_state = h_state.data        # repack the hidden state, break the connection from last iteration

    loss = torch.nn.CrossEntropyLoss()(prediction.reshape((-1,2)),torch.autograd.Variable(y.reshape((-1,))))         # calculate loss
    optimizer.zero_grad()                   # clear gradients for this training step
    loss.backward()                         # backpropagation, compute gradients
    optimizer.step()                        # apply gradients

    errTrain = (prediction.max(2)[1].data != y).float().mean()
    print("Error Training:",errTrain.item())

For those who want to reproduce the experiment, the plot is drawn using the following code (using Jupyter Notebook):

steps = range(0,maxLength)
plotChoice = 3

plt.figure(1, figsize=(12, 5))
plt.ion()           # continuously plot

plt.plot(steps, y_np[plotChoice,:].flatten(), 'r-')
plt.plot(steps, numpy.argmax(prediction.detach().numpy()[plotChoice,:,:],axis=1), 'g-')
plt.plot(steps, x_np[plotChoice,:,0].flatten(), 'b-')

plt.ioff()
plt.show()


title: pytorch compute pairwise difference: Incorrect result in NumPy vs PyTorch and different PyTorch versions

body:
Suppose I have two arrays, and I want to calculate row-wise differences between every two rows of two matrices of the same shape as follows. This is how the procedure looks like in numpy, and I want to replicate the same thing in pytorch.

>>> a = np.array([[1,2,3],[4,5,6]])
>>> b = np.array([[3,4,5],[5,3,2]])
>>> c = a[np.newaxis,:,:] - b[:,np.newaxis,:]
>>> print(c)
[[[-2 -2 -2]
  [ 1  1  1]]

 [[-4 -1  1]
  [-1  2  4]]]

BTW, I tried the same thing using pytorch, but it does not work. Is there anyway we could accomplish the same thing in pytorch

>>> import torch
>>> a = torch.from_numpy(a)
>>> b = torch.from_numpy(b)
>>> c1 = a[None,:,:]
>>> c2 = b[:,None,:]
>>> diff = c1 - c2
>>> print(diff.size())
torch.Size([1, 2, 3])

I was actually looking for torch.Size([2,2,3]). (P.S. I also tried unsqueeze from pytorch, but it doesn't work).


title: Pytorch model: drastically different inference results on different machines?

body:
I trained a segmentation model in Pytorch and tested it to give an F1-score of 0.93 on my local computer (Windows, conda, CUDA 10.2, Pytorch 1.2). However, the F1-score dropped to 0.3 when testing on a Linux server (conda, CUDA 9.0, Pytorch 1.1). I double checked that both sets of code, label files, test sets were the same, and there is no "explicit" random sampling in my code (even if so, the effect shouldn't be so drastic).

I tried the following to solve my problem:

I suspected that it might have something to do with Pytorch versions, so installed Pytorch 1.1 on my Windows machine to match the server's, but got the same 0.93 score. I then thought it had to do with CUDA itself, so eliminated the GPU factor altogether and did inference on cpu on server, I still got 0.3 on server.

What could possibly be causing this huge discrepancy?


title: How to set the output size of an RNN?

body:
I want to have an RNN with input size 7, hidden size 10 and output size 2.
So for an input of, say, shape 99x1x7 I expect an output of shape 99x1x2.
For an RNN alone, I get:

model = nn.RNN(input_size=7, hidden_size=10, num_layers=1)

output,hn=model(torch.rand(99,1,7))
print(output.shape) #torch.Size([99, 1, 10])
print(hn.shape)     #torch.Size([ 1, 1, 10])

So I assume I still have to put a Linear behind it:

model = nn.Sequential(nn.RNN(input_size=7, hidden_size=10, num_layers=1),
                      nn.Linear(in_features=10, out_features=2))
model(torch.rand(99,1,7))

Traceback (most recent call last):
  File "train_rnn.py", line 80, in <module>
    main()
  File "train_rnn.py", line 25, in main
    model(torch.rand(99,1,7))
  File "/home/.../virtual-env/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/.../virtual-env/lib/python3.6/site-packages/torch/nn/modules/container.py", line 92, in forward
    input = module(input)
  File "/home/.../virtual-env/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/.../virtual-env/lib/python3.6/site-packages/torch/nn/modules/linear.py", line 92, in forward
    return F.linear(input, self.weight, self.bias)
  File "/home/.../virtual-env/lib/python3.6/site-packages/torch/nn/functional.py", line 1404, in linear
    if input.dim() == 2 and bias is not None:
AttributeError: 'tuple' object has no attribute 'dim'

I guess this is because Linear receives the tuple that RNN.forward yields. But how am I supposed to combine the two?


title: Install specific PyTorch version (pytorch==1.0.1)

body:
I'm trying to install specific PyTorch version under conda env:
Using pip:
pip3 install pytorch==1.0.1
WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.
Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue.
To avoid this problem you can invoke Python with '-m pip' instead of running pip directly.
Defaulting to user installation because normal site-packages is not writeable
ERROR: Could not find a version that satisfies the requirement pytorch==1.0.1 (from versions: 0.1.2, 1.0.2)
ERROR: No matching distribution found for pytorch==1.0.1

Using Conda:
conda install pytorch==1.0.1
    Collecting package metadata (current_repodata.json): done
    Solving environment: failed with initial frozen solve. Retrying with flexible solve.
    Collecting package metadata (repodata.json): done
    Solving environment: failed with initial frozen solve. Retrying with flexible solve.

    PackagesNotFoundError: The following packages are not available from current channels:

      - pytorch==1.0.1

    Current channels:

      - https://repo.anaconda.com/pkgs/main/osx-64
      - https://repo.anaconda.com/pkgs/main/noarch
      - https://repo.anaconda.com/pkgs/r/osx-64
      - https://repo.anaconda.com/pkgs/r/noarch

    To search for alternate channels that may provide the conda package you're
    looking for, navigate to

        https://anaconda.org

    and use the search bar at the top of the page.

I was able to find this version under https://anaconda.org/soumith/pytorch but is there a way to find it and install from console?


title: sklearn pipeline with multiple inputs/outputs

body:
How can I build a sklearn pipeline to do the following?

What I have:

A, B = getAB(X_train)
X_train = transform(X_train)
model(A, B, X_train)

What I want:

pipe = Pipeline([
(‘ab’, getAB),
(‘tranf’, transform),
(‘net’, net)
]
pipe.fit(X_train, y_train)

Please help!


title: PyTorch gives incorrect results due to broadcasting

body:
I want to run some neural net experiments with PyTorch, but a minimal test case is giving wrong answers. The test case sets up a simple neural network with two input variables and an output variable that is just the sum of the inputs, and tries learning it as a regression problem; I expect it to converge on zero mean squared error, but it actually converges on 0.165. It's probably because of the issue alluded to in the warning message; how can I fix it?
Code:
import torch
import torch.nn as nn

# data
Xs = []
ys = []
n = 10
for i in range(n):
    i1 = i / n
    for j in range(n):
        j1 = j / n
        Xs.append([i1, j1])
        ys.append(i1 + j1)

# torch tensors
X_tensor = torch.tensor(Xs)
y_tensor = torch.tensor(ys)

# hyperparameters
in_features = len(Xs[0])
hidden_size = 100
out_features = 1
epochs = 500

# model
class Net(nn.Module):
    def __init__(self, hidden_size):
        super(Net, self).__init__()
        self.L0 = nn.Linear(in_features, hidden_size)
        self.N0 = nn.ReLU()
        self.L1 = nn.Linear(hidden_size, 1)

    def forward(self, x):
        x = self.L0(x)
        x = self.N0(x)
        x = self.L1(x)
        return x

model = Net(hidden_size)
criterion = nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.1)

# train
print("training")
for epoch in range(1, epochs + 1):
    # forward
    output = model(X_tensor)
    cost = criterion(output, y_tensor)

    # backward
    optimizer.zero_grad()
    cost.backward()
    optimizer.step()

    # print progress
    if epoch % (epochs // 10) == 0:
        print(f"{epoch:6d} {cost.item():10f}")
print()

output = model(X_tensor)
cost = criterion(output, y_tensor)
print("mean squared error:", cost.item())

Output:
training
C:\Users\russe\Anaconda3\envs\torch2\lib\site-packages\torch\nn\modules\loss.py:445: UserWarning: Using a target size (torch.Size([100])) that is different to the input size (torch.Size([100, 1])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
  return F.mse_loss(input, target, reduction=self.reduction)
    50   0.167574
   100   0.165108
   150   0.165070
   200   0.165052
   250   0.165039
   300   0.165028
   350   0.165020
   400   0.165013
   450   0.165009
   500   0.165006

mean squared error: 0.1650056540966034

And the message:

UserWarning: Using a target size (torch.Size([100])) that is different to the input size (torch.Size([100, 1])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.


title: Caffe2: Load ONNX model, and inference single threaded on multi-core host / docker

body:
I'm having trouble running inference on a model in docker when the host has several cores. The model is exported via PyTorch 1.0 ONNX exporter:

torch.onnx.export(pytorch_net, dummyseq, ONNX_MODEL_PATH)

Starting the model server (wrapped in Flask) with a single core yields acceptable performance (cpuset pins the process to specific cpus) docker run --rm -p 8081:8080 --cpus 0.5 --cpuset-cpus 0 my_container

response from ab -c 1 -n 1000 http://0.0.0.0:8081/predict\?itemids\=5,100

Percentage of the requests served within a certain time (ms)
  50%      5
  66%      5
  75%      5
  80%      5
  90%      7
  95%     46
  98%     48
  99%     49

But pinning it to four cores gives completely different stats for the same ab-call docker run --rm -p 8081:8080 --cpus 0.5 --cpuset-cpus 0,1,2,3 my_container

Percentage of the requests served within a certain time (ms)
  50%      9
  66%     12
  75%     14
  80%     18
  90%     62
  95%     66
  98%     69
  99%     69
 100%     77 (longest request)

Model inference is done like this, and except this issue it seems to work as expected. (This runs in a completely separate environment from the model export of course)

from caffe2.python import workspace
from caffe2.python.onnx.backend import Caffe2Backend as c2
from onnx import ModelProto

class Model:
    def __init__(self):
        self.predictor = create_caffe2_predictor(path)

    @staticmethod
    def create_caffe2_predictor(onnx_file_path):
        with open(onnx_file_path, 'rb') as onnx_model:
            onnx_model_proto = ModelProto()
            onnx_model_proto.ParseFromString(onnx_model.read())
            init_net, predict_net = c2.onnx_graph_to_caffe2_net(onnx_model_proto)
            predictor = workspace.Predictor(init_net, predict_net)
        return predictor

    def predict(self, numpy_array):
        return self.predictor.run({'0': numpy_array})

** wrapper flask app which calls Model.predict() on calls to /predict **

OMP_NUM_THREADS=1 is also present in the container environment, which had some effect, but it is not the end issue.

The benchmark stats you're seeing here are run on a local machine with 8 hyperthreads, so I shouldn't be saturating my machine and affect the test. These results also show up in my kubernetes environment, and I'm getting a large amount of CFS (Completely Fair Scheduler) throttling there.

I'm running in a kubernetes environment, so there's no way for me to control how many CPUs the host exposes, and doing some sort of pinning there seems a bit hacky as well.

Is there any way to pin caffe2 model inference to a single processor? Am I doing something obviously wrong here? Is the caffe2.Predictor object not suited to this task?

Any help appreciated.

EDIT:

I've added the simplest possible reproducable example I can think of here, with a docker-container and run-script included: https://github.com/NegatioN/Caffe2Struggles


title: Pytorch model training CPU Memory leak issue

body:
When I trained my pytorch model on GPU device,my python script was killed out of blue.Dives into OS log files , and I find script was killed by OOM killer because my CPU ran out of memory.It’s very strange that I trained my model on GPU device but I ran out of my CPU memory.
Snapshot of OOM killer log file

In order to  debug this issue,I install python memory profiler. Viewing log file from memory profiler, I find when column wise -= operation  occurred, my CPU memory gradually increased until OOM killer killed my
program.
Snapshot of Python memory profiler

It’s very strange, I try many ways to solve this issue.Finally, I found before assignment operation,I detach Tensor first.Amazingly,it solves this issue.But I don’t understand clearly why it works.Here is my original function code.
def GeneralizedNabla(self, image):
        pad_size = 2
        affinity = torch.zeros(image.shape[0], self.window_size**2, self.h, self.w).to(self.device)
        h = self.h+pad_size
        w = self.w+pad_size
        #pad = nn.ZeroPad2d(pad_size)
        image_pad = self.pad(image)
        for i in range(0, self.window_size**2):
            affinity[:, i, :, :] = image[:, :, :].detach()  # initialization
            dy = int(i/5)-2
            dx = int(i % 5)-2
            h_start = pad_size+dy
            h_end = h+dy  # if 0 <= dy else h+dy
            w_start = pad_size+dx
            w_end = w+dx  # if 0 <= dx else w+dx
            affinity[:, i, :, :] -= image_pad[:, h_start:h_end, w_start:w_end].detach()
        self.Nabla=affinity
        return

If everyone has any ideas,I will appreciate very much, thank you.


title: Finding memory leak in python by tracemalloc module

body:
I have a python script which uses an opensource pytorch model and this code has a memory leak. I am running this with memory_profiler mprof run --include-children python my_sctipt.py and get the following image:

I am trying to search for the reason of the leak by the system python module tracemalloc:

tracemalloc.start(25)
while True:
    ...
    snap = tracemalloc.take_snapshot()
    domain_filter = tracemalloc.DomainFilter(True, 0)
    snap = snap.filter_traces([domain_filter])
    stats = snap.statistics('lineno', True)
    for stat in stats[:10]:
        print(stat)

If looking only at tracemalloc output, I will not be able to identify the problem. I assume that the problem is in the C extension but, I would like to make sure it is true.
I tried to change the domain by DomainFilter, but I have output only in 0 domain.

Also, I don't understand the meaning of the parameter which tracemalloc.start(frameno) has got, frameno is a number of the most recent frames, but nothing happens when I change it.

What can I do next to find the problematic place in the code which causes the memory leak?

Looking forward to your answer.


title: Espresso ANERuntimeEngine Program Inference overflow

body:
I have two CoreML models. One works fine, and the other generates this error message:

[espresso] [Espresso::ANERuntimeEngine::__forward_segment 0] evaluate[RealTime]WithModel returned 0; code=5 err=Error Domain=com.apple.appleneuralengine Code=5 "processRequest:qos:qIndex:error:: 0x3: Program Inference overflow" UserInfo={NSLocalizedDescription=processRequest:qos:qIndex:error:: 0x3: Program Inference overflow}
[espresso] [Espresso::overflow_error] /var/containers/Bundle/Application/E0DE5E08-D2C6-48AF-91B2-B42BA7877E7E/xxx demoapp.app/mpii-hg128.mlmodelc/model.espresso.net:0

Both models are very similar, (Conv2D models). There are generated with the same scripts and versions of PyTorch, ONNX, and onnx-coreml. The model that works has 1036 layers, and the model that generates the error has 599 layers. They both use standard layers - Conv2D, BatchNorm, ReLU, MaxPool, and Upsample (no custom layers and no Functional or Numpy stuff). They both use relatively the same number of features per layer. They follow essentially the same structure, except the erroring model skips a maxpool layer at the start (hence the higher output resolution).

They both take a 256x256 color image as input, and output 16 channels at (working) 64x64 and (erroring) 128x128 pixels.

The app does not crash, but gives garbage results for the erroring model.

Both models train, evaluate, etc. fine in their native formats (PyTorch).

I have no idea what a Code=5 "processRequest:qos:qIndex:error:: 0x3: Program Inference overflow" error is, and google searches are not yielding anything productive, as I gather "Espresso" and "ANERuntimeEngine" are both private Apple Libraries.

What is this error message telling me? How can I fix it?

Can I avoid this error message by not running the model on the bionic chip but on the CPU/GPU?

Any help is appreciated, thanks.


title: Is torch.as_tensor() the same as torch.from_numpy() for a numpy array on a CPU?

body:
On a CPU, is torch.as_tensor(a) the same as torch.from_numpy(a) for a numpy array, a?  If not, then why not?

From the docs for torch.as_tensor

  if the data is an ndarray of the corresponding dtype and
      the device is the cpu, no copy will be performed.

From the docs for torch.from_numpy:

  The returned tensor and ndarray share the same memory. Modifications to
      the tensor will be reflected in the ndarray and vice versa.

In both cases, any changes the resulting tensor changes the original numpy array.

a = np.array([[1., 2], [3, 4]])
t1 = torch.as_tensor(a)
t2 = torch.from_numpy(a)
t1[0, 0] = 42.
print(a)
# prints [[42., 2.], [3., 4.]]
t2[1, 1] = 55.
print(a)
# prints [[42., 2.], [3., 55.]]

Also, in both cases, attempting to resize_ the tensor results in an error.


title: Trying to pass custom loss but it will not allow me to. AttributeError: 'float' object has no attribute 'backward'

body:
I have a custom loss function that I am trying to use on my model however when i use loss.backward() in Pytorch is not working.
This is my loss function:
class Neg_Pearson(nn.Module):    # Pearson range [-1, 1] so if < 0, abs|loss| ; if >0, 1- loss
    def __init__(self):
        super(Neg_Pearson,self).__init__()
        return
    def forward(self, preds, labels):       # tensor [Batch, Temporal]
        loss = 0
        for i in range(preds.shape[0]):
            sum_x = torch.sum(preds[i])                # x
            sum_y = torch.sum(labels[i])               # y
            sum_xy = torch.sum(preds[i]*labels[i])        # xy
            sum_x2 = torch.sum(torch.pow(preds[i],2))  # x^2
            sum_y2 = torch.sum(torch.pow(labels[i],2)) # y^2
            N = preds.shape[1]
            pearson = (N*sum_xy - sum_x*sum_y)/(torch.sqrt((N*sum_x2 - torch.pow(sum_x,2))*(N*sum_y2 - torch.pow(sum_y,2))))

            loss += 1 - pearson

        loss = loss.tolist()
        loss = loss/preds.shape[0]
        #print(loss)
        return loss

When I try to use it on with model like so:
        yp = (yp-torch.mean(yp)) /torch.std(yp)     # normalize
        yt = (yt-torch.mean(yt)) /torch.std(yt)     # normalize

        loss = neg_pears_loss(yp, yt)
        print(loss)

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

I get the following error: AttributeError: 'float' object has no attribute 'backward' any suggestions on how to fix this issue?


title: How can I extract the feature vector of the last hidden layer of the Alex net in pytorch?

body:
I’ve trained an Alex net in pytorch and I want to extract feature vectors from the layers
What is the function which I should use?


title: Pytorch - Subclasses of torchvision.dataset.ImageFolder - Import Error

body:
Following my last post, I am now trying to implement a subclass of the torchvision.datasets.ImageFolder class. The following code returns an error ("name 'default_loader' is not defined"), and I can't figure out why. Will you please help me?

class ExtendingImageFolder(torchvision.datasets.ImageFolder)
   def __init__(self,root,transform=None, target_transform=None,loader=default_loader):
       super().__init__(root,transform,target_transform,loader)

When I delete the "None" and "default_loader", and write it like this;

    class ExtendingImageFolder(torchvision.datasets.ImageFolder)
   def __init__(self,root,transform, target_transform,loader):
       super().__init__(root,transform,target_transform,loader)

I get an error of missing input arguments when trying to create an instance of this class, like:

JJ=ExtendingImageFolder(root='C:/',transform=transform)

What am I doing wrong here?

Thanks in advance!


title: Pytorch on GPU: variable still on CPU

body:
I am trying to modify some of my existing Pytorch scripts in order to run them on GPU. To do so,
I have read I should add .cuda() when instantiating my models and also on the input tensors.

When modifying a simple vanilla GAN, I get the following error:

Traceback (most recent call last):

    File “”, line 1, in
    runfile(’/home/florian/DeepLearning/GAN/GAN_pytorch.py’, wdir=’/home/florian/DeepLearning/GAN’)

    File “/home/florian/anaconda3/lib/python3.7/site-packages/spyder_kernels/customize/spydercustomize.py”, line 668, in runfile
    execfile(filename, namespace)

    File “/home/florian/anaconda3/lib/python3.7/site-packages/spyder_kernels/customize/spydercustomize.py”, line 108, in execfile
    exec(compile(f.read(), filename, ‘exec’), namespace)

    File “/home/florian/DeepLearning/GAN/GAN_pytorch.py”, line 145, in
    fake_data = generator(Variable(sample_z(batch_size, hidden_size)))

    File “/home/florian/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 489, in call
    result = self.forward(*input, **kwargs)

    File “/home/florian/DeepLearning/GAN/GAN_pytorch.py”, line 33, in forward
    x = F.leaky_relu(self.layer1(z))

    File “/home/florian/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 489, in call
    result = self.forward(*input, **kwargs)

    File “/home/florian/anaconda3/lib/python3.7/site-packages/torch/nn/modules/linear.py”, line 67, in forward
    return F.linear(input, self.weight, self.bias)

    File “/home/florian/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py”, line 1352, in linear
    ret = torch.addmm(torch.jit._unwrap_optional(bias), input, weight.t())

    RuntimeError: Expected object of backend CUDA but got backend CPU for argument #4 ‘mat1’

Thus, it seems a variable is still on CPU while I thought I’ve moved my data on the GPU. Here is the full code:

import torch
from torch.autograd import Variable
import torch.nn as nn
import torch.nn.functional as F
from torchvision import datasets, transforms

import matplotlib.pyplot as plt
import matplotlib.gridspec as gridspec

class Generator(nn.Module):
def init(self, hidden_size):
super(Generator, self).init()
self.layer1 = nn.Linear(hidden_size, 256)
torch.nn.init.xavier_normal(self.layer1.weight)
self.layer2 = nn.Linear(256, 512)
torch.nn.init.xavier_normal(self.layer2.weight)
self.layer3 = nn.Linear(512, 1024)
torch.nn.init.xavier_normal(self.layer3.weight)
self.layer4 = nn.Linear(1024, 28*28)
torch.nn.init.xavier_normal(self.layer4.weight)

def forward(self, z):
    x = F.leaky_relu(self.layer1(z))
    x = F.leaky_relu(self.layer2(x))
    x = F.leaky_relu(self.layer3(x))
    x = F.tanh(self.layer4(x))
    return x

class Discriminator(nn.Module):
def init(self):
super(Discriminator, self).init()
self.layer1 = nn.Linear(784, 1024)
torch.nn.init.xavier_normal(self.layer1.weight)
self.layer2 = nn.Linear(1024, 512)
torch.nn.init.xavier_normal(self.layer2.weight)
self.layer3 = nn.Linear(512, 256)
torch.nn.init.xavier_normal(self.layer3.weight)
self.layer4 = nn.Linear(256, 1)
torch.nn.init.xavier_normal(self.layer4.weight)

def forward(self, x):
    x = F.leaky_relu(self.layer1(x))
    #x = F.dropout(x, p=0.2)
    x = F.leaky_relu(self.layer2(x))
    #x = F.dropout(x, p=0.2)
    x = F.sigmoid(self.layer3(x))
    #x = F.dropout(x, p=0.2)
    x = F.sigmoid(self.layer4(x))

    return x

def sample_z(batch_size, size):
‘’’
Sample the generator input from a uniform (rand) or normal (randn) distribution
‘’’
#return 2 * torch.rand(batch_size, size) - 1
return torch.randn(batch_size, size)

def plot(samples):
fig = plt.figure(figsize=(4, 4))
gs = gridspec.GridSpec(4, 4)
gs.update(wspace=0.05, hspace=0.05)

for i, sample in enumerate(samples):
    ax = plt.subplot(gs[i])
    plt.axis('off')
    ax.set_xticklabels([])
    ax.set_yticklabels([])
    ax.set_aspect('equal')
    plt.imshow(sample.data.numpy().reshape(28, 28), cmap='Greys_r')

return fig

if name == ‘main’:

# parameters
lr = 2e-4
optim_betas = (0.9, 0.999)

batch_size = 128
hidden_size = 100
nb_epochs = 100
train_loader = torch.utils.data.DataLoader(
                               datasets.MNIST('../MNIST-PyTorch',
                               train=True,
                               download=True,
                               transform=transforms.Compose([transforms.ToTensor(), transforms.Normalize(mean=(0.5,), std=(0.5,))])),
                                batch_size=batch_size, shuffle=True,
                                drop_last=True)

ones_label = Variable(torch.ones((batch_size, 1)), requires_grad=False)
ones_label = ones_label.cuda()
zeros_label = Variable(torch.zeros((batch_size, 1)), requires_grad=False)
zeros_label = zeros_label.cuda()

generator = Generator(hidden_size=hidden_size)
generator = generator.cuda()
discriminator = Discriminator()
discriminator = discriminator.cuda()

# binary cross-entropy loss
criterion = nn.BCELoss()

# optimizer
G_optim = torch.optim.Adam(generator.parameters(), lr=lr, betas=optim_betas)
D_optim = torch.optim.Adam(discriminator.parameters(), lr=lr, betas=optim_betas)

for epoch_idx in range(nb_epochs):

    print('------------------------')
    print('epoch %i' % (epoch_idx+1) )

    d_loss_sum = 0
    g_loss_sum = 0

    for batch_idx, (data, labels) in enumerate(train_loader):

        '''
        Train the discriminator (50/50 real/fake) with two different
        mini-batches, one for real images and another one for fake images
        '''
        D_optim.zero_grad()

        # sample real data
        real_data = Variable(data).view(batch_size, -1)
        real_data = real_data.cuda()
        # sample fake data
        fake_data = generator(Variable(sample_z(batch_size, hidden_size)))
        fake_data = fake_data.cuda()

        # discriminator must classify real data as 1's and fake ones as 0's
        real_out = discriminator(real_data)
        real_loss = criterion(real_out, ones_label)
        real_loss.backward()

        fake_out = discriminator(fake_data)
        fake_loss = criterion(fake_out, zeros_label)
        fake_loss.backward()

        D_optim.step()

        d_loss_sum += real_loss.item() + fake_loss.item()

        '''
        Train the generator
        '''
        G_optim.zero_grad()

        # sample fake data
        fake_data = generator(Variable(sample_z(batch_size, hidden_size)))

        fake_out = discriminator(fake_data)

        # generator want the discriminator to classify fake data as 1's

        g_loss = criterion(fake_out, ones_label)

        g_loss_sum += g_loss.item()

        g_loss.backward()

        G_optim.step()

    print('discriminator loss = %f' % d_loss_sum)
    print('generator loss = %f' % g_loss_sum)
    print('------------------------')

    fig = plot(generator(Variable(sample_z(16, hidden_size))))
    plt.savefig('out_vanilla_GAN/%i.png' % epoch_idx)
    plt.close(fig)


title: Pytorch - inference all images and back-propagate batch by batch

body:
I have a special use case that I have to separate inference and back-propagation: I have to inference all images and slice outputs into batches followed by back-propagating batches by batches. I don't need to update my networks' wegiths.

I modified snippets of cifar10_tutorial into the following to simulate my problem: j is a variable to represent the index which returns by my own logic and I want the gradient of some variables.

for epoch in range(2):  # loop over the dataset multiple times

    for i, data in enumerate(trainloader, 0):
        # get the inputs
        inputs, labels = data
        inputs.requires_grad = True

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = net(inputs)

        for j in range(4): # j is given by external logic in my own case

            loss = criterion(outputs[j, :].unsqueeze(0), labels[j].unsqueeze(0))

            loss.backward()

            print(inputs.grad.data[j, :]) # what I really want

I got the following errors:

  RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.

My questions are:

According to my understanding, the problem rises because the first back-propagate backwards the whole outputs and outputs[1,:].unsqueeze(0) was released so second back-propagate failed. Am I right?
In my case, if I set retain_graph=True, will the codes run slower and slower according to this post?
Is there better way to achieve my goal?


title: Explicit slicing across a particular dimension

body:
I've got a 3D tensor x (e.g 4x4x100). I want to obtain a subset of this by explicitly choosing elements across the last dimension. This would have been easy if I was choosing the same elements across last dimension (e.g. x[:,:,30:50] but I want to target different elements across that dimension using the 2D tensor indices which specifies the idx across third dimension. Is there an easy way to do this in numpy?

A simpler 2D example:

x = [[1,2,3,4,5,6],[10,20,30,40,50,60]]
indices = [1,3]

Let's say I want to grab two elements across third dimension of x starting from points specified by indices. So my desired output is:

[[2,3],[40,50]]

Update: I think I could use a combination of take() and ravel_multi_index() but some of the platforms that are inspired by numpy (like PyTorch) don't seem to have ravel_multi_index so I'm looking for alternative solutions


title: How to convert tensor to image array?

body:
I would like to convert a tensor to image array and use tensor.data() method.

But it doesn't work.

#include <torch/script.h> // One-stop header.

#include <iostream>
#include <memory>
#include <sstream>
#include <string>
#include <vector>

#include "itkImage.h"
#include "itkImageFileReader.h"
#include "itkImageFileWriter.h"
#include "itkImageRegionIterator.h"

//////////////////////////////////////////////////////
//Goal: load jit script model and segment myocardium
//Step: 1. load jit script model
//      2. load input image
//      3. predict by model
//      4. save the result to file
//////////////////////////////////////////////////////
typedef short                                               PixelType;
const unsigned int Dimension = 3;
typedef itk::Image<PixelType, Dimension>                    ImageType;
typedef itk::ImageFileReader<ImageType>                     ReaderType;
typedef itk::ImageRegionIterator<ImageType>                 IteratorType;

bool itk2tensor(ImageType::Pointer itk_img, torch::Tensor &tensor_img) {

    typename ImageType::RegionType region = itk_img->GetLargestPossibleRegion();
    const typename ImageType::SizeType size = region.GetSize();
    std::cout << "Input size: " << size[0] << ", " << size[1]<< ", " << size[2] << std::endl;

    int len = size[0] * size[1] * size[2];
    short rowdata[len];
    int count = 0;
    IteratorType iter(itk_img, itk_img->GetRequestedRegion());

    // convert itk to array
    for (iter.GoToBegin(); !iter.IsAtEnd(); ++iter) {
        rowdata[count] = iter.Get();
        count++;
    }
    std::cout << "Convert itk to array DONE!" << std::endl;

    // convert array to tensor
    tensor_img = torch::from_blob(rowdata, {1, 1, (int)size[0], (int)size[1], (int)size[2]}, torch::kShort).clone();
    tensor_img = tensor_img.toType(torch::kFloat);
    tensor_img = tensor_img.to(torch::kCUDA);
    tensor_img.set_requires_grad(0);

    return true;
}

bool tensor2itk(torch::Tensor &t, ImageType::Pointer itk_img) {

    std::cout << "tensor dtype = " << t.dtype() << std::endl;
    std::cout << "tensor size = " << t.sizes() << std::endl;
    t = t.toType(torch::kShort);
    short * array = t.data<short>();

    ImageType::IndexType start;
    start[0] = 0;  // first index on X
    start[1] = 0;  // first index on Y
    start[2] = 0;  // first index on Z

    ImageType::SizeType  size;
    size[0] = t.size(2);
    size[1] = t.size(3);
    size[2] = t.size(4);

    ImageType::RegionType region;
    region.SetSize( size );
    region.SetIndex( start );

    itk_img->SetRegions( region );
    itk_img->Allocate();

    int len = size[0] * size[1] * size[2];

    IteratorType iter(itk_img, itk_img->GetRequestedRegion());
    int count = 0;
    // convert array to itk
    std::cout << "start!" << std::endl;
    for (iter.GoToBegin(); !iter.IsAtEnd(); ++iter) {
        short temp = *array++;    //  ERROR!
        std::cout << temp << " ";
        iter.Set(temp);
        count++;
    }
    std::cout << "end!" << std::endl;

    return true;
}

int main(int argc, const char* argv[]) {
    int a, b, c;
    if (argc != 4) {
        std::cerr << "usage: automyo input jitmodel output\n";
        return -1;
    }

    std::cout << "=========  jit start  =========\n";
    // 1. load jit script model
    std::cout << "Load script module: " << argv[2] << std::endl;
    std::shared_ptr<torch::jit::script::Module> module = torch::jit::load(argv[2]);
    module->to(at::kCUDA);

    // assert(module != nullptr);
    std::cout << "Load script module DONE" << std::endl;

    // 2. load input image
    const char* img_path = argv[1];
    std::cout << "Load image: " << img_path << std::endl;

    ReaderType::Pointer reader = ReaderType::New();

    if (!img_path) {
        std::cout << "Load input file error!" << std::endl;
        return false;
    }

    reader->SetFileName(img_path);
    reader->Update();

    std::cout << "Load image DONE!" << std::endl;

    ImageType::Pointer itk_img = reader->GetOutput();

    torch::Tensor tensor_img;
    if (!itk2tensor(itk_img, tensor_img)) {
        std::cerr << "itk2tensor ERROR!" << std::endl;
    }
    else {
        std::cout << "Convert array to tensor DONE!" << std::endl;
    }

    std::vector<torch::jit::IValue> inputs;
    inputs.push_back(tensor_img);

    // 3. predict by model
    torch::Tensor y = module->forward(inputs).toTensor();
    std::cout << "Inference DONE!" << std::endl;

    // 4. save the result to file
    torch::Tensor seg = y.gt(0.5);
    // std::cout << seg << std::endl;

    ImageType::Pointer out_itk_img = ImageType::New();
    if (!tensor2itk(seg, out_itk_img)) {
        std::cerr << "tensor2itk ERROR!" << std::endl;
    }
    else {
        std::cout << "Convert tensor to itk DONE!" << std::endl;
    }

    std::cout << out_itk_img << std::endl;

    return true;
}

The runtime log is showed below:

  Load script module:model_myo_jit.pt

  Load script module DONE Load

  image: patch_6.nii.gz

  Load image DONE!

  Input size: 128, 128, 128

  Convert itk to array DONE!

  Convert array to tensor DONE!

  Inference DONE!

  tensor dtype = unsigned char

  tensor size = [1, 1, 96, 96, 96]

  start!

  Segmentation fault (core dumped)

Why and how to convert?


title: Conda environment : several environment files - specify cpu-only version of Pytorch

body:
I am using conda 4.8.3 with Python 3.7, I am writing environment files to specify the dependencies of my project. I would like to write several files to be able to install several environments:

main.yml : containing the dependencies of my project, and pytorch CPU-only version
dev.yml : containing dev tools (mypy, flake8, pytest ..)
gpu.yml : containing pytorch-GPU (with a specified version of CUDA)

To get a basic (CPU) installation one would write : conda env update --file main.yml
To get an installation with GPU compatibility would then add conda env update --file gpu.yml
Here is my question : at the moment I cannot find the right way to specify the 'CPU-only' criteria for pytorch in an environment file, does anyone know if it is doable?
The command usually used for that purpose is conda install pytorch torchvision cpuonly -c pytorch, but I cannot find a way to specify it in the yml file.
On the pytorch channel site , there is a pytorch-cpu package, but its version is quite outdated (1.1.0, while the current main is 1.6.0)
Here is my main.yml environment file:
name: my_env
channels:
  - intel
  - conda-forge
  - pytorch

dependencies:

  - numpy
  - scipy
  - scikit-image
  - matplotlib
  - wxpython
  - colorama
  - dill
  - protobuf
  - pytorch   # How to specify the 'cpu' criteria here??
  - torchvision
  - pip:
      - -r env/requirements.txt


title: Run-time error for (allegedly) empty batches in Pytorch

body:
(Updated with further insights on the problem)

I have a dataset of 3000 images which gets into a DataLoader with the following lines:

training_left_eyes = torch.utils.data.DataLoader(train_dataset, batch_size=2,shuffle=True, drop_last=True)
print(len(training_left_eyes)) #Outputs 1500

My training loop looks like this:

for i,(data,output) in enumerate(training_left_eyes):
      data,output = data.to(device),output.to(device)
      prediction = net(data)

      loss = costFunc(prediction,output)
      closs = loss.item()
      optimizer.zero_grad()
      loss.backward()
      optimizer.step()

      print("batch #{}".format(i))
      if i%100 == 0:
        print('[%d %d] loss: %.8f' % (epoch+1,i+1,closs/1000))
        closs = 0

The information inside the tensors "data" and "output" (labels) are correct and the system works fine until it reaches the batch number 1500. All my batches are full as 3000/2=1500 with no remainder. Once it reaches that last batch there is a RunTimeError stating that there is a 0-dimensional input size. But I don't know why this is supposed to happen as enumerate(training_left_eyes) should be iterating over the values of the DataLoader which are full.

I searched on the web how to solve this and some people were mentioning the "drop_last=True" attribute on the DataLoader, althought this was done in order to prevent semi-empty batches to enter the model I tried it anyway without no avail.

I am starting to tunnel vision too much and can't seem to solve the problem on my own. I could simply insert an if statement but I think that would be very bad practice and I want to learn the proper solution.

If it helps, here is my custom DataSet:

class LeftEyeDataset(torch.utils.data.Dataset):
  """Left eye retinography dataset. Normal/Not-normal"""

  def __init__(self, csv_file, root_dir, transform=None):
    """
    Args:
        csv_file (string): Path to the csv file with annotations.
        root_dir (string): Directory with all the images.
        transform (callable, optional): Optional transform to be applied
            on a sample.
    """
    self.labels  = label_mapping(csv_file)
    self.root_dir = root_dir
    self.transform = transform
    self.names = name_mapping(csv_file)

  def __len__(self):
    return len(self.labels)

  def __getitem__(self, idx):
    if torch.is_tensor(idx):
        idx = idx.tolist()

    img_name = self.root_dir +'/'+ self.names[idx]
    image = io.imread(img_name)
    label = self.labels[idx]

    if self.transform:
        image = self.transform(image)

    return image,label

def label_mapping(csv_file) -> np.array:
  df = read_excel(excel_file, 'Sheet1')
  x= []
  for key,value in df['Left-Diagnostic Keywords'].iteritems():
    if value=='normal fundus':
      x.append(1)
    else:
      x.append(0)
  x_tensor = torch.LongTensor(x)
  return x_tensor

def name_mapping(csv_file) -> list:
  #Reads the names of the excel file
  df = read_excel(excel_file, 'Sheet1')
  names= list()
  serie = df['Left-Fundus']
  for i in range(serie.size):
    names.append(df['Left-Fundus'][i])
  return names

I can provide any additional code if needed.

Update: after some time trying to solve the problem I manage to pin point what is happening. For some reason on the last batch, the data entering the network is fine but right before the first layer something happens and it dissapears. On this next image you can see a print I do before entering the forward(self,x) and one right after. The sizes are aligned until the batch number 61 (I reduced it from 1500 for this example) in which it somehow goes through the print twice. After that line, the error mentioned above appears.


title: pytorch math with exponents less than 1 return nan 's

body:
torch.pow() returns nan when it's given the exponent argument between ranges -1 and 1

a = torch.randn(1,3)
a

>> tensor([[-1.7871, -0.5375, -0.1164]])

torch.pow(a, 2)      #or a**2

>> tensor([[3.1938, 0.2889, 0.0136]])

torch.pow(a,0.5)     #or a**0.5

>> tensor([[nan, nan, nan]])

expect result:

tensor([[-1.3368, -0.7331, -0.3412]])

Edit: turns out that ** works the same way as well. a**2 does the same thing with the tensor as torch.pow(a,2).  a**0.5 returns nan's like torch.pow does.


title: Pytorch autograd raising the error: "Expected isFloatingType(grads[i].type().scalarType()) to be true, but got false" when loss.backward()

body:
I am working on the implementation of an adversarial training. The following code does not work:

    for i, data in tqdm(enumerate(train_loader), total=len(train_loader), smoothing=0.9):

        pc1, pc2 = data
        pc1 = pc1.to(device).transpose(2, 1).contiguous().float()
        pc2 = pc2.to(device).transpose(2, 1).contiguous().float()

        # train the discriminator
        net_gen.eval()
        net_disc.train()

        i_odds = torch.arange(start=1, end=pc1.shape[2], step=2)
        ...

        # train the flow extractor
        net_gen.train()
        net_disc.eval()

        flow_pred = net_gen(pc1, pc2)
        pc_pred = pc1 + flow_pred

        z_pred = net_disc(pc1[:, :, i_odds], pc_pred[:, :, i_odds])
        loss_flow = - z_pred.mean()

        opt_gen.zero_grad()

        loss_flow.backward()

I get the following error on last line: Expected isFloatingType(grads[i].type().scalarType()) to be true, but got false.

What could that be? For all I could check, the model and all the tensors are all type cuda.Float.

That is the traceback:
Traceback (most recent call last):
  File "/home/zuanazzi/anaconda3/envs/tomtom2/lib/python3.6/pdb.py", line 1667, in main
    pdb._runscript(mainpyfile)
  File "/home/zuanazzi/anaconda3/envs/tomtom2/lib/python3.6/pdb.py", line 1548, in _runscript
    self.run(statement)
  File "/home/zuanazzi/anaconda3/envs/tomtom2/lib/python3.6/bdb.py", line 434, in run
    exec(cmd, globals, locals)
  File "<string>", line 1, in <module>
  File "/home/zuanazzi/PointCloudFlow/flownet3d_pytorch/self_train_adversarial.py", line 452, in <module>
    main()
  File "/home/zuanazzi/PointCloudFlow/flownet3d_pytorch/self_train_adversarial.py", line 445, in main
    train(args, net_flow, net_disc, train_loader, test_loader, boardio, textio)
  File "/home/zuanazzi/PointCloudFlow/flownet3d_pytorch/self_train_adversarial.py", line 309, in train
    train_stats = train_one_epoch(args, net_flow, net_disc, train_loader, opt_flow, opt_disc, writer, epoch)
  File "/home/zuanazzi/PointCloudFlow/flownet3d_pytorch/self_train_adversarial.py", line 227, in train_one_epoch
    loss_flow.backward()
  File "/home/zuanazzi/anaconda3/envs/tomtom2/lib/python3.6/site-packages/torch/tensor.py", line 195, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/zuanazzi/anaconda3/envs/tomtom2/lib/python3.6/site-packages/torch/autograd/__init__.py", line 99, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: Expected isFloatingType(grads[i].type().scalarType()) to be true, but got false.  (Could this error message be improved?  If so, please report an enhancement request to PyTorch.)

Question originally asked here: https://discuss.pytorch.org/t/expected-isfloatingtype-grads-i-type-scalartype-to-be-true-but-got-false/67840

Edit: It seems that architectures based on PointNet++ layers cannot be used as Discriminator because they won't return gradients. That is currently more a theory than a claim.


title: in Pytorch, restore the model parameters but the same initial loss

body:
I am training a dnn (CRNN) with Pytorch, but some abnormal things happened in terms of loss val.

The program can print avg_loss for every 20 batches and save the model_parameters every 100 batches. And the initial loss is about 20-30. Some problems happened in my program, so the training process is interrupted. After loading the parameters from the saved model, I continue training but find the initial loss still start from 20-30. By the way, I have a dataset about 10 million pictures and I have trained about 3 million of them.

I want to figure about where the problem is, pytorch mechanism or program bugs.

Here is more detailed:

1. CRNN structure:

    CRNN (

(cnn): Sequential (
    (conv0): Conv2d(1, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (relu0): ReLU (inplace)
    (pooling0): MaxPool2d (size=(2, 2), stride=(2, 2), dilation=(1, 1))
    (conv1): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (relu1): ReLU (inplace)
    (pooling1): MaxPool2d (size=(2, 2), stride=(2, 2), dilation=(1, 1))
    (conv2): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (batchnorm2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True)
    (relu2): ReLU (inplace)
    (conv3): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (relu3): ReLU (inplace)
    (pooling2): MaxPool2d (size=(2, 2), stride=(2, 1), dilation=(1, 1))
    (conv4): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (batchnorm4): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True)
    (relu4): ReLU (inplace)
    (conv5): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (relu5): ReLU (inplace)
    (pooling3): MaxPool2d (size=(2, 2), stride=(2, 1), dilation=(1, 1))
    (conv6): Conv2d(512, 512, kernel_size=(2, 2), stride=(1, 1))
    (batchnorm6): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True)
    (relu6): ReLU (inplace)
  )
  (rnn): Sequential (
    (0): BidirectionalLSTM (
      (rnn): LSTM(512, 256, bidirectional=True)
      (embedding): Linear (512 -> 256)
    )
    (1): BidirectionalLSTM (
      (rnn): LSTM(256, 256, bidirectional=True)
      (embedding): Linear (512 -> 5530)
    )
  )
)

2. model init and parameters loading.

def crnnSource():
    alphabet = keys.alphabet
    converter = util.strLabelConverter(alphabet)
    model = crnn.CRNN(32, 1 ,len(alphabet)+1, 256, 1) #need 1?
    model.apply(weights_init)
    path = './models/crnn_OCR.pkl'
    model.load_state_dict(torch.load(path))
    return model, converter

3. training code

def trainProc(net ,trainset, converter):
    print ("--------------------------------")
    print ("Start to Train.")

    criterion = CTCLoss().cuda()
    loss_avg = util.averager()
    optimizer = optim.RMSprop(net.parameters(), lr = 0.001)

    image = torch.FloatTensor(BATCH_SIZE, 3, 32, 100) #opt.imgH
    text = torch.IntTensor(BATCH_SIZE * 5)
    length = torch.IntTensor(BATCH_SIZE)
    image = image.cuda()

    image = Variable(image)
    text = Variable(text)
    length = Variable(length)

    sav_inv = 0
    for epoch in range(TRAIN_EPOCHS):
        sav_inv = 0
        timer = time.time()
        for i,data in enumerate(trainset, 0):
            img, txt = data
            img = ConvtFileToTensor(img)
            batch_size = img.size(0)
            util.loadData(image, img)
            t, l = converter.encode(txt)
            util.loadData(text,t)
            util.loadData(length,l)

            preds  = net(image)
            preds_size = Variable(torch.IntTensor([preds.size(0)] * batch_size))
            cost = criterion(preds, text, preds_size, length) / batch_size

            net.zero_grad()
            cost.backward()
            optimizer.step()

            loss_avg.add(cost)
            #running_loss += loss.data[0]
            if i % 20 == 19:
                time2 = time.time()
                print ("[%d, %5d] loss: %.6f  TIME: %.6f" %(epoch+1, i+1, loss_avg.val(),time2 - timer))
                print (cost)
                loss_avg.reset()
                timer = time.time()
            if sav_inv == SAV_INV-1:
                torch.save(net.state_dict(),'./models/crnn_OCR.pkl')
                sav_inv = 0
            else:
                sav_inv += 1
    torch.save(net.state_dict(),'./models/crnn_OCR.pkl')
    print ("Finished Training.")
    return net


title: Pre-trained model in Pytorch

body:
I just loaded the pre-trained model 3D resnet
https://github.com/kenshohara/3D-ResNets-PyTorch       .

I loaded this model in this way after i download from the link above the file 'resnext-101-kinetics.pth' :
way i defined the model

this model work on a big dataset of videos (unfortunately i don't have enough space on my pc and i didn't find a link that i can download part of the data set). I just want to know if i load the model correctly and to know that this net is trained.
There is an option to be sure that the model loaded well?
I printed the state_dict and it prints the weights..
Thanks for any help.


title: * (tuple of Tensors tensors, name dim, Tensor out)

body:
Suppose I have the memory list list_of_tensors = [tensor1, tensor2, tensor3, tensor4]. Each element is a pytorch tensor of shape (1, 1, 84, 84).

I want to concatenate that list of tensors to get a tensor of shape (4, 1, 84, 84). torch.cat(TT, dim=0) might surely allow me to do that. TT must be a tuple of tensor, so torch.cat(*list_of_tensors, dim=0) or torch.cat((*list_of_tensors), dim=0) won't work.

How can I use list_of_tensors and torch.cat(???, dim=0) to create a new tensor of shape (4, 1, 84, 84)


title: How to resolve eta 0:00:01Killed?

body:
I was downloading torch and after 99% completion of the download, I am getting this error.
I am using Ubuntu, on an 8gb ram system.

the memory allocation

ankita@ankita-HP-Laptop-15-bs0xx:~$ free -h
              total        used        free      shared  buff/cache   available
Mem:           7.7G        2.0G        1.6G        495M        4.1G        5.0G
Swap:            0B          0B          0B


title: cffi error when compiling faster rcnn pytorch

body:
I'm trying to compile faster_rcnn_pytorch using the instructions given here: https://github.com/longcw/faster_rcnn_pytorch

I get this error:

(p27) [$USER@compute-1-5 faster_rcnn]$ ./make.sh
Traceback (most recent call last):
  File "setup.py", line 59, in <module>
    CUDA = locate_cuda()
  File "setup.py", line 45, in locate_cuda
    raise EnvironmentError('The nvcc binary could not be '
EnvironmentError: The nvcc binary could not be located in your $PATH. Either add it to your path, or set $CUDAHOME
Compiling roi pooling kernels by nvcc...
./make.sh: line 10: nvcc: command not found
/home/$USER/play/hw2/gitdir/hw2-release/code/faster_rcnn/roi_pooling
generating /tmp/tmpP8mucv/_roi_pooling.c
setting the current directory to '/tmp/tmpP8mucv'
running build_ext
building '_roi_pooling' extension
creating home
creating home/$USER
creating home/$USER/play
creating home/$USER/play/hw2
creating home/$USER/play/hw2/gitdir
creating home/$USER/play/hw2/gitdir/hw2-release
creating home/$USER/play/hw2/gitdir/hw2-release/code
creating home/$USER/play/hw2/gitdir/hw2-release/code/faster_rcnn
creating home/$USER/play/hw2/gitdir/hw2-release/code/faster_rcnn/roi_pooling
creating home/$USER/play/hw2/gitdir/hw2-release/code/faster_rcnn/roi_pooling/src
gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/data2/$USER/anaconda2/envs/p27/lib/python2.7/site-packages/torch/utils/ffi/../../lib/include -I/data2/$USER/anaconda2/envs/p27/lib/python2.7/site-packages/torch/utils/ffi/../../lib/include/TH -I/data2/$USER/anaconda2/envs/p27/include/python2.7 -c _roi_pooling.c -o ./_roi_pooling.o
gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/data2/$USER/anaconda2/envs/p27/lib/python2.7/site-packages/torch/utils/ffi/../../lib/include -I/data2/$USER/anaconda2/envs/p27/lib/python2.7/site-packages/torch/utils/ffi/../../lib/include/TH -I/data2/$USER/anaconda2/envs/p27/include/python2.7 -c /home/$USER/play/hw2/gitdir/hw2-release/code/faster_rcnn/roi_pooling/src/roi_pooling.c -o ./home/$USER/play/hw2/gitdir/hw2-release/code/faster_rcnn/roi_pooling/src/roi_pooling.o
gcc -pthread -shared -L/data2/$USER/anaconda2/envs/p27/lib -Wl,-rpath=/data2/$USER/anaconda2/envs/p27/lib,--no-as-needed ./_roi_pooling.o ./home/$USER/play/hw2/gitdir/hw2-release/code/faster_rcnn/roi_pooling/src/roi_pooling.o /home/$USER/play/hw2/gitdir/hw2-release/code/faster_rcnn/roi_pooling/src/cuda/roi_pooling.cu.o -L/data2/$USER/anaconda2/envs/p27/lib -lpython2.7 -o ./_roi_pooling.so
gcc: error: /home/$USER/play/hw2/gitdir/hw2-release/code/faster_rcnn/roi_pooling/src/cuda/roi_pooling.cu.o: No such file or directory
Traceback (most recent call last):
  File "build.py", line 34, in <module>
    ffi.build()
  File "/data2/$USER/anaconda2/envs/p27/lib/python2.7/site-packages/torch/utils/ffi/__init__.py", line 167, in build
    _build_extension(ffi, cffi_wrapper_name, target_dir, verbose)
  File "/data2/$USER/anaconda2/envs/p27/lib/python2.7/site-packages/torch/utils/ffi/__init__.py", line 103, in _build_extension
    ffi.compile(tmpdir=tmpdir, verbose=verbose, target=libname)
  File "/data2/$USER/anaconda2/envs/p27/lib/python2.7/site-packages/cffi/api.py", line 697, in compile
    compiler_verbose=verbose, debug=debug, **kwds)
  File "/data2/$USER/anaconda2/envs/p27/lib/python2.7/site-packages/cffi/recompiler.py", line 1520, in recompile
    compiler_verbose, debug)
  File "/data2/$USER/anaconda2/envs/p27/lib/python2.7/site-packages/cffi/ffiplatform.py", line 22, in compile
    outputfilename = _build(tmpdir, ext, compiler_verbose, debug)
  File "/data2/$USER/anaconda2/envs/p27/lib/python2.7/site-packages/cffi/ffiplatform.py", line 58, in _build
    raise VerificationError('%s: %s' % (e.__class__.__name__, e))
cffi.error.VerificationError: LinkError: command 'gcc' failed with exit status 1

As far as I can tell, there are 2 problems:

nvcc is not found
cffi error

Regarding nvcc, I'm not sure how to add it to my path. It looks like I have both CUDA 8 and CUDA 9 so maybe I need to add it to only CUDA 9?

(p27) [$USER@compute-1-5 faster_rcnn]$ which nvcc
/usr/bin/which: no nvcc in (/data2/$USER/anaconda2/envs/p27/bin:/data2/$USER/anaconda2/bin:/opt/afni:/data2/$USER/anaconda2/bin:/opt/afni:/data2/$USER/anaconda2/bin:/opt/fsl/bin:/opt/openmpi/bin:/usr/lib64/qt-3.3/bin:/opt/afni:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/ganglia/bin:/opt/ganglia/sbin:/usr/java/latest/bin:/opt/maui/bin:/opt/torque/bin:/opt/torque/sbin:/opt/pdsh/bin:/opt/rocks/bin:/opt/rocks/sbina:/home/$USER/bin:/opt/maui/bin:/opt/torque/bin:/opt/torque/sbina:/opt/maui/bin:/opt/torque/bin:/opt/torque/sbina)

(p27) [$USER@compute-1-5 faster_rcnn]$ find /opt -name nvcc
/opt/cuda-8.0/bin/nvcc
/opt/freesurfer/lib/cuda/bin/nvcc

(p27) [$USER@compute-1-5 faster_rcnn]$ echo $LD_LIBRARY_PATH
/home/$USER/lib/cuDNN/v7/cuda/lib64:/home/$USER/cuda-9.0/lib64:/home/$USER/cuda-9.0/extras/CUPTI/lib64:/home/$USER/lib/cuDNN/v7/cuda/lib64:/home/$USER/cuda-9.0/lib64:/home/$USER/cuda-9.0/extras/CUPTI/lib64:/opt/fsl/lib:/opt/openmpi/lib:/home/$USER/cuda/lib64:/home/$USER/cuda/lib64

Regarding cffi error, I have installed both gcc and cffi using conda but I'm not sure why it still complains.

(p27) [$USER@compute-1-5 faster_rcnn]$ conda list | grep cffi
cffi                      1.11.5           py27h9745a5d_0    anaconda
(p27) [$USER@compute-1-5 faster_rcnn]$ conda list | grep gcc
gcc                       4.8.5                         7    anaconda
libgcc-ng                 7.2.0                hdf63c60_3    anaconda

Thank you for helping!!


title: Why is PIL used so often with Pytorch?

body:
I noticed that a lot of dataloaders use PIL to load and transform images, e.g. the dataset builders in torchvision.datasets.folder.

My question is: why use PIL? You would need to do an np.asarray operation before turning it into a tensor. OpenCV seems to load it directly as a numpy array, and is faster too.

One reason I can think of is because PIL has a rich transforms library, but I feel like several of those transforms can be quickly implemented.


title: PyTorch Sparse Tensors number of dimensions must be nDimI + nDimV

body:
I'm trying to insert the value in gd to coordinate [1,0]. Below are the matrices. When I try this, I get a RuntimeError.

>>> import torch
>>> cd = [[1, 0]]
>>> gd = [0.39613232016563416]
>>> i = torch.LongTensor(cd)
>>> v = torch.FloatTensor(gd)
>>> p = torch.rand(2)
>>> i

 1  0
[torch.LongTensor of size 1x2]

>>> v

 0.3961
[torch.FloatTensor of size 1]

>>> p

 0.4678
 0.0996
[torch.FloatTensor of size 2]

>>> torch.sparse.FloatTensor(i.t(), v, torch.Size(list(p.size()))).to_dense()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: invalid argument 2: number of dimensions must be nDimI + nDimV at /Users/soumith/code/builder/wheel/pytorch-src/torch/lib/THS/generic/THSTensor.c:169


title: pytorch (numpy) calculation about the closest pixels to points

body:
I am trying to solve a complicated problem.

For example, I have a batch of 2D predicted images (softmax output, value between 0 and 1) with size: Batch x H x W and ground truth Batch x H x W

The light gray color pixels are the background with value 0, and the dark gray color pixels are the foreground with value 1. I try to compute the mass center coordinates using scipy.ndimage.center_of_mass on each ground truth image. Then I get the center location point C (red color) for each ground truth. The C points set is Batch x 1.

Now, for each pixel A (yellow color) in the predicted images, I want to get three pixels B1, B2, B3 (blue color) which are the closest to A on the line AC (here C is corresponding location of mass center in ground truth).

I used following code to get the three closest points B1, B2, B3.

def connect(ends, m=3):
    d0, d1 = np.abs(np.diff(ends, axis=0))[0]
    if d0 > d1:
        return np.c_[np.linspace(ends[0, 0], ends[1, 0], m + 1, dtype=np.int32),
                 np.round(np.linspace(ends[0, 1], ends[1, 1], m + 1))
                     .astype(np.int32)]
    else:
        return np.c_[np.round(np.linspace(ends[0, 0], ends[1, 0], m + 1))
                     .astype(np.int32),
                 np.linspace(ends[0, 1], ends[1, 1], m + 1, dtype=np.int32)]

So the B points set is Batch x 3 x H x W.

Then, I want to compute like this: |Value(A)-Value(B1)|+|Value(A)-Value(B2)|+|Value(A)-Value(B3)|. The size of the result should be Batch x H x W.

Is there any numpy vectorization tricks that can be used to update the value of each pixel in predicted images? Or can this be solved using pytorch functions? I need to find a method to update the whole image. The predicted image is the softmax output. I cannot use for loop to compute each single value since it will become non-differentiable. Thanks a lot.


title: Pytorch Image Segmentation Problems BCELoss

body:
Currently I am creating an Unet in order to do Image Segmentation.
In my Unet, the last output channel is 183 which is the classes size.
When I was doing the train, It shows the following error.
I tried the BCEWithLogitsLoss function which works for me, but it does not shows me the right output images and it slowly train to a black image, instead of GrayScale Classes Image.
Expected imageOutPut Image

    def Append_files():

        train_folder = "train/"

        train_label_folder = "train_label/"

        root_dir = "data/Train_Data/"

        ext_cond = lambda filename:filename.endswith('.png')
        train_data = os.path.join(root_dir,train_folder)
        train_data_path = lambda filename: True
        train_data_label = os.path.join(root_dir,train_label_folder)
        train_data_label_path = lambda filename: True
        filtered_train_data = []
        filtered_train_label_data = []

        for path, _, files in os.walk(train_data):
              files.sort()
              for file in files:
                  if train_data_path(file) and ext_cond(file):
                        full_file_path = os.path.join(path,file)
                        filtered_train_data.append(full_file_path)

        for path,_,files in os.walk(train_data_label):
              files.sort()
              for file in files:
                    if train_data_label_path(file) and ext_cond(file):
                            full_file_path = os.path.join(path,file)
                            filtered_train_label_data.append(full_file_path)
    return filtered_train_data, filtered_train_label_data

def open_image(train_data,train_data_label):

    data = Image.open(train_data)
    data = np.array(data)
    label = Image.open(train_data_label)
    label = np.array(label)

    return data,label

def load_dataset(train_data,train_label_data):

    train_set_loader = transforms.Compose([transforms.ToPILImage(),transforms.CenterCrop(size=(572,572)),transforms.ToTensor()])
    train_label_set_loader = transforms.Compose([transforms.ToPILImage(),transforms.CenterCrop(size=(388,388)),transforms.ToTensor()])

    train_set = train_set_loader(train_data)
    train_label_set = train_label_set_loader(train_label_data)

    return train_set, train_label_set

### Main
if __name__ == '__main__':

    epochs = 100
    loss = 0
    train_data,train_label_data = Append_files()
    data_len = len(train_data)

    model = UNet(3,183).to(device)

    #loss_func = nn.BCEWithLogitsLoss()
    loss_func = nn.BCELoss()
    get_optimizer = torch.optim.Adam(model.parameters(),0.001,betas=(0.9,0.999),eps=0.1,weight_decay = 0,amsgrad=False)

    for epoch in range(epochs):
        print ('starting epoch{}/{}.'.format(epoch+1,epochs))
        for i in range(data_len):

            data, label = open_image(train_data[i],train_label_data[i])

            train, label = load_dataset(data,label)

            train = Variable(train).to(device)
            label = Variable(label).to(device)

            train = train.unsqueeze(0)
            label = label.unsqueeze(0)

            get_optimizer.zero_grad()                #train.shape = 1,1,388,388
            train_real_image = model.forward(train)  #train_real_image.shape = 1,1,388,388
                                                     #I can change it to 1,183,388 where 183 is the class size

        #Loss Function
            #Label.shape = 1,1,388,388
            loss = loss_func(train_real_image,label)
            print (loss)
            loss.backward()
            get_optimizer.step()

/opt/conda/conda-bld/pytorch_1556653183467/work/aten/src/THCUNN/BCECriterion.cu:42: Acctype bce_functor<Dtype, Acctype>::operator()(Tuple) [with Tuple = thrust::detail::tuple_of_iterator_references<thrust::device_reference<float>, thrust::device_reference<float>, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type>, Dtype = float, Acctype = float]: block: [23,0,0], thread: [28,0,0] Assertion `input >= 0. && input <= 1.` failed.
/opt/conda/conda-bld/pytorch_1556653183467/work/aten/src/THCUNN/BCECriterion.cu:42: Acctype bce_functor<Dtype, Acctype>::operator()(Tuple) [with Tuple = thrust::detail::tuple_of_iterator_references<thrust::device_reference<float>, thrust::device_reference<float>, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type>, Dtype = float, Acctype = float]: block: [23,0,0], thread: [29,0,0] Assertion `input >= 0. && input <= 1.` failed.
/opt/conda/conda-bld/pytorch_1556653183467/work/aten/src/THCUNN/BCECriterion.cu:42: Acctype bce_functor<Dtype, Acctype>::operator()(Tuple) [with Tuple = thrust::detail::tuple_of_iterator_references<thrust::device_reference<float>, thrust::device_reference<float>, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type>, Dtype = float, Acctype = float]: block: [23,0,0], thread: [31,0,0] Assertion `input >= 0. && input <= 1.` failed.

RuntimeError: reduce failed to synchronize: device-side assert triggered


title: Converting a darknet model to a tensorflow model

body:
I converted a pytorch(.pt) model  to a darknet model (.weights)
then when I tried to convert the darknet mode to tensorflow model (.tf) I got this error

assert len(wf.read()) == 0, 'failed to read all data'

AssertionError: failed to read all data


title: Unable install Pytorch from source for older GPU card

body:
Good day,

I am going fast.ai courses for deep learning. I want to set up locally fastai environment. However, when I try running the first part of the tutorial, I receive an error message stating:

Found GPU0 GeForce GTX 860M which is of cuda capability 5.0.
PyTorch no longer supports this GPU because it is too old.

I found that the problem is related to Pytorch version, which does not support older GPU cards. I found a solution to install older Pytorch from source: http://forums.fast.ai/t/pytorch-not-working-with-an-old-nvidia-card/14632/7

However, when I clone pytorch git and try to install with install.py, I receive a message error.

python setup.py install
running install
running build_deps
error: [WinError 2] The system cannot find the file specified

I have navigated to the correct folder “~\Fast_AI\Github material\fastai\pytorch”, when I list the files I can see setup.py, but it does not allow to run.

Could anyone help solve this problem? I could use CPU, however, it works very slowly when working with deep learning, therefore I would prefer to use GPU.


title: RuntimeError occurs in PyTorch backward function

body:
I am trying to calculate the grad of a variable in PyTorch. However, there was a RuntimeError which tells me that the shape of output and grad must be the same. However, in my case, the shape of output and grad cannot be the same. Here is my code to reproduce:

import numpy as np
import torch
from torch.autograd import Variable as V

ne = 3
m, n = 79, 164
G = np.random.rand(m, n).astype(np.float64)
w = np.random.rand(n, n).astype(np.float64)
z = -np.random.rand(n).astype(np.float64)

G = V(torch.from_numpy(G))
w = V(torch.from_numpy(w))
z = V(torch.from_numpy(z), requires_grad=True)
e, v = torch.symeig(torch.diag(2 * z - torch.sum(w, dim=1)) + w, eigenvectors=True, upper=False)
ssev = torch.sum(torch.pow(e[-ne:] * v[:, -ne:], 2), dim=1)
out = torch.sum(torch.matmul(G, ssev.reshape((n, 1))))
out.backward(z)
print(z.grad)

The error message is: RuntimeError: Mismatch in shape: grad_output[0] has a shape of torch.Size([164]) and output[0] has a shape of torch.Size([])

Similar calculation is allowed in TensorFlow and I can successfully get the gradient I want:

import numpy as np
import tensorflow as tf

m, n = 79, 164
G = np.random.rand(m, n).astype(np.float64)
w = np.random.rand(n, n).astype(np.float64)
z = -np.random.rand(n).astype(np.float64)

def tf_function(z, G, w, ne=3):
    e, v = tf.linalg.eigh(tf.linalg.diag(2 * z - tf.reduce_sum(w, 1)) + w)
    ssev = tf.reduce_sum(tf.square(e[-ne:] * v[:, -ne:]), 1)
    return tf.reduce_sum(tf.matmul(G, tf.expand_dims(ssev, 1)))

z, G, w = [tf.convert_to_tensor(_, dtype=tf.float64) for _ in (z, G, w)]
z = tf.Variable(z)
with tf.GradientTape() as g:
    g.watch(z)
    out = tf_function(z, G, w)
print(g.gradient(out, z).numpy())

My tensorflow version is 2.0 and my PyTorch version is 1.14.0. I am using Python3.6.9. In my opinion, calculating the gradients when the output and the variables have different shapes is very reasonable and I don't think I made any mistake.Can anyone help me with this problem? I really appreciate it!


title: RuntimeError: Expected object of device type cuda but got device type cpu for argument #3 'index' in call to _th_index_select site:stackoverflow.com

body:
I am using the bert on the SMILE dataset. I have written following code can you guide me where I am getting wrong.
I have written training code which is evaluating correctly but the when I try to run evaluate code for validation it is giving error. I tried to pass the parameters directly to cuda. still I am facing the issue

'''

def evaluate(dataloader_val):

  print("in evaluate")
  model.eval()

  loss_val_total = 0
  predictions, true_value = [],[]

  for batch in dataloader_val:

    print("in for loop of dataloader")
    barch = tuple(b.to(device) for b in batch)

    inputs = {
               'input_ids':  batch[0],
                'attention_mask': batch[1],
                 'labels' : batch[2],
    }

    with torch.no_grad():
      outputs = model(**inputs)

    loss = outputs[0]
    logits = outputs[1]
    loss_val_total += loss.item()

    print("before logit")

    logits = logits.to(device)
    print("in the for batch evaluate: ",logits)
    label_ids = inputs['labels'].to(device)
    true_vals.append(label_ids)

  loss_val_avg = loss_val_total/len(dataloader_val)

  predictions = np.concatenate(predictions, axis = 0)
  true_vals = np.concatenate(true_vals,axis = 0)

  return loss_val_avg, predictions, true_vals
'''

and another function is

'''
for epoch in tqdm(range(1, epochs+1)):
  model.train()

  loss_train_total = 0

  progress_bar = tqdm(dataloader_train,
                      desc = 'Epoch {:1d}'.format(epoch),
                      leave = False,
                      disable = False)
  for batch in progress_bar:

    model.zero_grad()

    batch = tuple(b.to(device) for b in batch)

    inputs = {

            'input_ids'      : batch[0],
            'attention_mask' : batch[1],
            'labels'         : batch[2]
    }

    outputs = model(**inputs)

    loss = outputs[0]
    loss_train_total += loss.item()
    loss.backward()

    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)

    optimizer.step()
    scheduler.step()

    progress_bar.set_postfix({'training_loss' : '{:.3f}'.format(loss.item()/len(batch))})

  torch.save(model.state_dict(), f'/content/drive/My Drive/Bert/Coursera/SMILE/Bert_ft_epoch{epoch}.model')

  tqdm.write(f'\n Epoch {epoch}')

  loss_train_avg = loss_train_total / len(dataloader_train)

  tqdm.write(f'Training Loss: {loss_train_avg}')

  val_loss, predictions, true_vals = evaluate(dataloader_val)
  val_f1 = f1_score_func(predictions, true_vals)
  tqdm.write(f'Validation loss : {val_loss}')
  tqdm.write(f'F1 score(weighted): {val_f1}')
'''


title: How can I extract the weight and bias of Linear layers in PyTorch?

body:
In model.state_dict(), model.parameters() and model.named_parameters() weights and biases of nn.Linear() modules are contained separately, e.q. fc1.weight and fc1.bias. Is there a simple pythonic way to get both of them?
Expected example looks similar to this:
layer = model['fc1']
print(layer.weight)
print(layer.bias)


title: Binary One Hot Encoding For Dimension Reduction

body:
Introduction:

I've been studying machine learning this last couple of months, more specifically neural networks. I found out that categorical data, such as cities, could be transformed to integer data, but shouldn't because there's no linearity. What I mean by this is: there is no reason why New York should be 1 or 100. A better solution to this is one hot encoding. However, it greatly increases the dimensionality of the problem (Picture a network input with the number of cities as input nodes).

Example:

New York = [1, 0, 0]

Buenos Aires = [0, 1, 0]

Tokio = [0, 0, 1]

You need N inputs, N being the number of cities.

Possible Solution:

I don't really know if this already exists or if it could work, I haven't tested it yet. That being said, in the example above, I'm using decimal inputs. Could you use binary inputs, in order to reduce the dimensionality of the problem?

Example using binary inputs

New York = [0, 1]

Buenos Aires = [1, 0]

Tokio = [1, 1]

In this way, you only need ≈ log(N) (base 2) number of inputs which greatly increases, specially if there are a great number of features. For example: if you have 1000 categorical data inputs, it would only result in 10 inputs for a neural network.

Thank you in advance. Remember I'm only learning.


title: Bidirectional RNN Implementation pytorch

body:
Hi I am trying to understand bidirectional RNN.
> class RNN(nn.Module):
>

>
>   def __init__(self,n_vocab,n_embed,hidden_size,output_size):
>
>     super().__init__()
>
>     self.hidden_size = hidden_size
>
>     self.embedding = nn.Embedding(n_vocab+1,n_embed) ## n_vocab is unique words in dictionary ## n_embed is hyperparameter

>     self.rnn = nn.RNN(n_embed, hidden_size, num_layers = 1, batch_first = True,bidirectional = True) #
>
>     self.fc =  nn.Linear(hidden_size,output_size)
>
>   def forward(self,x):>
>
>     x = x # input batch_size * seq_length
>
>     batch_size = x.size(0)
>
>     #print('Batch Size is',batch_size)
>
>     x = self.embedding(x) # batch-size x seq_length x embedding_dimension
>
>     x,hidden =self.rnn(x)   #batch-size x seq_length x hidden_size
>
>
>
>     return x,hidden

I am returning both hidden state and output while going through tutorials some says that I need to concatenate hidden state (torch.cat((hidden[-2,:,:], hidden[-1,:,:]), dim = 1)) and in some tutorials take output state (x[:,-1,:]) but both of results come difference.
What is the correct way of doing Bidirectional RNN.


title: Is there an equivalent of pytorch.nn.functional.unfold() in keras or tensorflow?

body:
I want to perform a similar operation in keras. However, I am unable to do the unfold operation in keras. I tried it with conv1D layer, but unable to figure out. Any help would be appreciated
'''
import numpy as np
import torch

x = torch.tensor(np.random.rand(25,100,24))   # tensor of shape (batch_size, seq_length,feature_dim)
x = x.unsqueeze(1) # shape=(25,1,100,24)

import torch.nn.functional as F
x = F.unfold(x,(5, 24), stride=(1,24),dilation=(1,1)) #shape (25,120,96)

'''


title: Can anyone tell me how to checking if Pytorch model exists, and if it does, delete it and replace it with a new one?

body: So I save a lot of torch models for training and with different batchsize and epochs, and the models are saves with strings of epoch and batchsize. Basically I sometimes change some layers hyperparamters and some augmentation to check the prediction results, but if the torch model is there, I want to delete it and replace it with the new one.


title: PyTorch loss decreases even if requires_grad = False for all variables

body:
When I create a neural network with PyTorch, using the torch.nn.Sequential method for defining layers, it seems that the parameters have requires_grad = False by default. However, when I train this network, the loss decreases. How is this possible if the layers are not being updated via gradients?

For example, this is the code for defining my network:

class Network(torch.nn.Module):

def __init__(self):
    super(Network, self).__init__()
    self.layers = torch.nn.Sequential(
        torch.nn.Linear(10, 5),
        torch.nn.Linear(5, 2)
    )
    print('Network Parameters:')
    model_dict = self.state_dict()
    for param_name in model_dict:
        param = model_dict[param_name]
        print('Name: ' + str(param_name))
        print('\tRequires Grad: ' + str(param.requires_grad))

def forward(self, input):
    prediction = self.layers(input)
    return prediction

And this prints out:

Network Parameters:
Name: layers.0.weight
    Requires Grad: False
Name: layers.0.bias
    Requires Grad: False
Name: layers.1.weight
    Requires Grad: False
Name: layers.1.bias
    Requires Grad: False

Then this is the code to train my network:

network = Network()
network.train()
optimiser = torch.optim.SGD(network.parameters(), lr=0.001)
criterion = torch.nn.MSELoss()
inputs = np.random.random([100, 10]).astype(np.float32)
inputs = torch.from_numpy(inputs)
labels = np.random.random([100, 2]).astype(np.float32)
labels = torch.from_numpy(labels)

while True:
    prediction = network.forward(inputs)
    loss = criterion(prediction, labels)
    print('loss = ' + str(loss.item()))
    optimiser.zero_grad()
    loss.backward()
    optimiser.step()

And this prints out:

loss = 0.284633219242
loss = 0.278225809336
loss = 0.271959483624
loss = 0.265835255384
loss = 0.259853869677
loss = 0.254015892744
loss = 0.248321473598
loss = 0.242770522833
loss = 0.237362638116
loss = 0.232097044587
loss = 0.226972639561
loss = 0.221987977624
loss = 0.217141270638
loss = 0.212430402637
loss = 0.207852959633
loss = 0.203406244516
loss = 0.199087426066
loss = 0.19489350915
loss = 0.190821439028
loss = 0.186868071556
loss = 0.183030322194
loss = 0.179305106401
loss = 0.175689414144
loss = 0.172180294991
loss = 0.168774917722
loss = 0.165470585227
loss = 0.162264674902
loss = 0.159154698253

Why is the loss decreasing if all of the parameters have requires_grad = False?


title: Pytorch model.train() and a separte train() function written in a tutorial

body:
I am new to PyTorch and I was wondering if you could explain to me some of the key differences between the default model.train() in PyTorch and the train() function here.

The other train() function is on the official PyTorch tutorial on text classification and was confused as to whether the model weights are being stored at the end of training.

https://pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial.html

learning_rate = 0.005

criterion = nn.NLLLoss()

def train(category_tensor, line_tensor):
    hidden = rnn.initHidden()
    rnn.zero_grad()
    for i in range(line_tensor.size()[0]):
        output, hidden = rnn(line_tensor[i], hidden)
    loss = criterion(output, category_tensor)
    loss.backward()
    # Add parameters' gradients to their values, multiplied by learning rate
    for p in rnn.parameters():
        p.data.add_(-learning_rate, p.grad.data)
    return output, loss.item()

This is the function. This function is then called multiple times in this form:

n_iters = 100000
print_every = 5000
plot_every = 1000
record_every = 500

# Keep track of losses for plotting
current_loss = 0
all_losses = []
predictions = []
true_vals = []

def timeSince(since):
    now = time.time()
    s = now - since
    m = math.floor(s / 60)
    s -= m * 60
    return '%dm %ds' % (m, s)

start = time.time()

for iter in range(1, n_iters + 1):
    category, line, category_tensor, line_tensor = randomTrainingExample()
    output, loss = train(category_tensor, line_tensor)
    current_loss += loss

    if iter % print_every == 0:
        guess, guess_i = categoryFromOutput(output)
        correct = 'O' if guess == category else 'X (%s)' % category
        print('%d %d%% (%s) %.4f %s / %s %s' % (iter, iter / n_iters * 100, timeSince(start), loss, line, guess, correct))

    if iter % plot_every == 0:
        all_losses.append(current_loss / plot_every)
        current_loss = 0

    if iter % record_every == 0:
        guess, guess_i = categoryFromOutput(output)
        predictions.append(guess)
        true_vals.append(category)

To me it seems that the model weights are not being saved or updated but rather being overridden at each iteration when written like this. Is this correct? Or does the model appear to be training correctly?

Additionally, if I were to use the default function model.train(), what is the main advantage and does model.train() perform more or less the same functionality as the train() function above?


title: Use SHAP Values for PyTorch RNN / LSTM

body:
Is there a way to do the above?
The SHAP Package is very helpful and works pretty well for PyTorch Neural Nets. For PyTorch RNNs i get the error message below (for LSTMs its the same):

Seems like it doesn't work but is there a workaround or something? Does anyone have experience with PyTorch and SHAP?


title: How to use GPUs with Ray in Pytorch? Should I specify the num_gpus for the remote class?

body:
When I use the Ray with pytorch, I do not set any num_gpus flag for the remote class.

I get the following error:

RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False.

The main process is: I create a remote class and transfer a pytorch model state_dict()(created in main function) to it. In the main function, the torch.cuda.is_available() is True, but In the remote function, torch.cuda.is_available() is False. Thanks

I try to set the num_gpus=1 and got a new issue: the program just got stuck. Below is the minimal example code for reproducing this issue. Thanks.

import ray

@ray.remote(num_gpus=1)
class Worker(object):
    def __init__(self, args):
        self.args = args
        self.gen_frames = 0

    def set_gen_frames(self, value):
        self.gen_frames = value
        return self.gen_frames

    def get_gen_num(self):
        return self.gen_frames

class Parameters:
    def __init__(self):
        self.is_cuda = False;
        self.is_memory_cuda = True
        self.pop_size = 10

if __name__ == "__main__":
    ray.init()
    args = Parameters()
    workers = [Worker.remote(args) for _ in range(args.pop_size)]
    get_num_ids = [worker.get_gen_num.remote() for worker in workers]
    gen_nums = ray.get(get_num_ids)
    print(gen_nums)


title: Why must use DataParallel when testing？

body:
Train on the GPU, num_gpus is set to 1：

device_ids = list(range(num_gpus))
model = NestedUNet(opt.num_channel, 2).to(device)
model = nn.DataParallel(model, device_ids=device_ids)

Test on the CPU：

model = NestedUNet_Purn2(opt.num_channel, 2).to(dev)
device_ids = list(range(num_gpus))
model = torch.nn.DataParallel(model, device_ids=device_ids)
model_old = torch.load(path, map_location=dev)
pretrained_dict = model_old.state_dict()
model_dict = model.state_dict()
pretrained_dict = {k: v for k, v in pretrained_dict.items() if k in model_dict}
model_dict.update(pretrained_dict)
model.load_state_dict(model_dict)

This will get the correct result, but when I delete：

device_ids = list(range(num_gpus))
model = torch.nn.DataParallel(model, device_ids=device_ids)

the result is wrong.


title: How to Find Confusion Matrix and Plot it for Image Classifier in PyTorch

body:
Basically this is the VGG-16 Model, I have performed Transfer Learning and Fine Tuned the model, I have trained this model 2 weeks ago and found both the test and train accuracy but now I need Class wise accuracy of the model too, I am trying to find out the Confusion matrix and wanna plot the matrix too. Training Code:
# Training the model again from the last CNN Block to The End of the Network
dataset = 'C:\\Users\\Sara Latif Khan\\OneDrive\\Desktop\\FYP_\\Scene15\\15-Scene'
model = model.to(device)
optimizer = Adam(filter(lambda p: p.requires_grad, model.parameters()))

#Training Fixed Feature Extractor for 15 epochs
num_epochs = 5
batch_loss = 0
cum_epoch_loss = 0 #cumulative loss for each batch

for e in range(num_epochs):
    cum_epoch_loss = 0

    for batch, (images, labels) in enumerate(trainloader,1):
        images = images.to(device)
        labels = labels.to(device)

        optimizer.zero_grad()
        logps = model(images)
        loss = criterion(logps, labels)
        loss.backward()
        optimizer.step()

        batch_loss += loss.item()
        print(f'Epoch({e}/{num_epochs} : Batch number({batch}/{len(trainloader)}) : Batch loss : {loss.item()}')
        torch.save(model, dataset+'_model_'+str(e)+'.pt')

print(f'Training loss : {batch_loss/len(trainloader)}')

This is the code I am using to check the accuracy of my model based on data from the test loader.
model. to('cpu')

model.eval()
with torch.no_grad():
    num_correct = 0
    total = 0

    #set_trace ()
    for batch, (images,labels) in enumerate(testloader,1):

        logps = model(images)
        output = torch.exp(logps)

        pred = torch.argmax(output,1)
        total += labels.size(0)

        num_correct += (pred==labels).sum().item()
        print(f'Batch ({batch} / {len(testloader)})')

        # to check the accuracy of model on 5 batches
        # if batch == 5:
            # break

    print(f'Accuracy of the model on {total} test images: {num_correct * 100 / total }% ')

Next, I need to find the class-wise accuracy of the model. I am working on the Jupyter Notebook. Should I reload a saved model and find the cm or what will the appropriate way of doing it.


title: How to iterate over a group of tensor and pass the elements from each group to a function?

body:
Suppose you have 3 tensors of the same size:

a = torch.randn(3,3)
a = ([[ 0.1945,  0.8583,  2.6479],
    [-0.1000,  1.2136, -0.3706],
    [-0.0094,  0.4279, -0.6840]])

b = torch.randn(3, 3)
b = ([[-1.1155,  0.2106, -0.2183],
    [ 1.6610, -0.6953,  0.0052],
    [-0.8955,  0.0953, -0.7737]])

c = torch.randn(3, 3)
c = ([[-0.2303, -0.3427, -0.4990],
    [-1.1254,  0.4432,  0.3999],
    [ 0.2489, -0.9459, -0.5576]])

In Lua (torch7), they have this function:

[self] map2(tensor1, tensor2, function(x, xt1, xt2))

which applies the given function to all elements of self.

My questions are:

Is there any similar function in python (pytorch)?
Is there any pythonic method to iterate over the 3 tensors and get the respective elements of each tensor without using for loop and indices?

For example:

0.1945 -1.1155 -0.2303
0.8583 0.2106 -0.3427
2.6479 -0.2183 -0.4990
-0.1000 1.6610 -1.1254
...

Edit_1: I have also tried itertools.zip_longest and zip, but the results are not as I expected as mentioned above


title: OpenAI gym 0.10.9 'module' object has no attribute 'benchmark_spec'

body:
benchmark = gym.benchmark_spec('Atari40M')

AttributeError: 'module' object has no attribute 'benchmark_spec'

I just got this error for gym-0.10.9. Any idea? Thx


title: PyTorch Data Augmentation is taking too long

body:
For the task that involves regression, I need to train my models to generate density maps from RGB images. To augment my dataset I have decided to flip all the images horizontally. For that matter, I also have to flip my ground truth images and I did so.

dataset_for_augmentation.listDataset(train_list,
                        shuffle=True,
                        transform=transforms.Compose([
                            transforms.RandomHorizontalFlip(p=1),
                            transforms.ToTensor(),
                            transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
                        ]),
                        target_transform=transforms.Compose([
                            transforms.RandomHorizontalFlip(p=1),
                            transforms.ToTensor()
                        ]),
                        train=True,
                        resize=4,
                        batch_size=args.batch_size,
                        num_workers=args.workers),

But here is the problem : For some reason, PyTorch transforms.RandomHorizontalFlip function takes only PIL images (numpy is not allowed) as input. So I decided to convert the type to PIL Image.

img_path = self.lines[index]

img, target = load_data(img_path, self.train, resize=self.resize)

if type(target[0][0]) is np.float64:
    target = np.float32(target)

img = Image.fromarray(img)
target = Image.fromarray(target)

if self.transform is not None:
    img = self.transform(img)
    target = self.target_transform(target)

return img, target

And yes, this operation need enormous amount of time. Considering I need this operation to be carried out for thousands of images, 23 seconds (should have been under half a second at most) per batch is not tolerable.

2019-11-01 16:29:02,497 - INFO - Epoch: [0][0/152]  Time 27.095 (27.095)    Data 23.150 (23.150)    Loss 93.7401 (93.7401)

I would appreciate any suggestions to speed up my augmentation process


title: Why return self.head(x.view(x.size(0), -1)) in the nn.Module for pyTorch reinforcement learning example

body:
I understand that the balancing the pole example requires 2 outputs. Reinforcement Learning (DQN) Tutorial
Here is the output for self.head
  print ('x',self.head)
  x = Linear(in_features=512, out_features=2, bias=True)

When I run the epochs below is the outputs:
print (self.head(x.view(x.size(0), -1)))
return self.head(x.view(x.size(0), -1))

tensor([[-0.6945, -0.1930]])
tensor([[-0.0195, -0.1452]])
tensor([[-0.0906, -0.1816]])
tensor([[ 0.0631, -0.9051]])
tensor([[-0.0982, -0.5109]])
...

The size of x is:
x = torch.Size([121, 32, 2, 8])

So I am trying to understand what x.view(x.size(0), -1) is doing?
I understand from the comment in the code that it's returning:
Returns tensor([[left0exp,right0exp]...]).
But how does x which is torch.Size([121, 32, 2, 8]) being reduced to a tensor of size 2?
Is there an alternative way of writing that makes more sense?  What if I had 4 outputs.  How would I represent that?  Why x.size(0).  Why -1?
So appears to take self.head with 4 outputs to 2 outputs.  Is that correct?
At the bottom is that class I am referring:
class DQN(nn.Module):

    def __init__(self, h, w, outputs):
        super(DQN, self).__init__()
        self.conv1 = nn.Conv2d(3, 16, kernel_size=5, stride=2)
        self.bn1 = nn.BatchNorm2d(16)
        self.conv2 = nn.Conv2d(16, 32, kernel_size=5, stride=2)
        self.bn2 = nn.BatchNorm2d(32)
        self.conv3 = nn.Conv2d(32, 32, kernel_size=5, stride=2)
        self.bn3 = nn.BatchNorm2d(32)

        # Number of Linear input connections depends on output of conv2d layers
        # and therefore the input image size, so compute it.
        def conv2d_size_out(size, kernel_size = 5, stride = 2):
            return (size - (kernel_size - 1) - 1) // stride  + 1
        convw = conv2d_size_out(conv2d_size_out(conv2d_size_out(w)))
        convh = conv2d_size_out(conv2d_size_out(conv2d_size_out(h)))
        linear_input_size = convw * convh * 32
        self.head = nn.Linear(linear_input_size, outputs)

    # Called with either one element to determine next action, or a batch
    # during optimization. Returns tensor([[left0exp,right0exp]...]).
    def forward(self, x):
        x = F.relu(self.bn1(self.conv1(x)))
        x = F.relu(self.bn2(self.conv2(x)))
        x = F.relu(self.bn3(self.conv3(x)))
        return self.head(x.view(x.size(0), -1))


title: Why the loss function can be apply on different size tensors

body:
For example, I have a net that take tensor [N, 7](N is the samples num) as input and tensor [N, 4] as output, the “4” represents the different classes’ probabilities.

And the training data’s labels are the form of tensor [N], from range 0 to 3(represent the ground-truth class).

Here’s my question, I’ve seen some demos, they directly apply the loss function on the output tensor and label tensor. I wonder why this can work, since they have different size, and there sizes seems don’t fit the “broadcasting semantics”.

Here’s the minimal demo.

import torch
import torch.nn as nn
import torch.optim as optim

if __name__ == '__main__':
    features = torch.randn(2, 7)
    gt = torch.tensor([1, 1])
    model = nn.Sequential(
        nn.Linear(7, 4),
        nn.ReLU(),
        nn.Linear(4, 4)
    )
    optimizer = optim.SGD(model.parameters(), lr=0.005)
    f = nn.CrossEntropyLoss()

    for epoch in range(1000):
        optimizer.zero_grad()
        output = model(features)
        loss = f(output, gt)
        loss.backward()
        optimizer.step()