title: How to check if pytorch is using the GPU?

body:
I would like to know if pytorch is using my GPU. It's possible to detect with nvidia-smi if there is any activity from the GPU during the process, but I want something written in a python script.

Is there a way to do so?


title: How to initialize weights in PyTorch?

body: How to initialize the weights and biases (for example, with He or Xavier initialization) in a network in PyTorch?


title: Best way to save a trained model in PyTorch?

body:
I was looking for alternative ways to save a trained model in PyTorch. So far, I have found two alternatives.

torch.save() to save a model and torch.load() to load a model.
model.state_dict() to save a trained model and model.load_state_dict() to load the saved model.

I have come across to this discussion where approach 2 is recommended over approach 1.

My question is, why the second approach is preferred? Is it only because torch.nn modules have those two function and we are encouraged to use them?


title: How does the "view" method work in PyTorch?

body:
I am confused about the method view() in the following code snippet.

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool  = nn.MaxPool2d(2,2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1   = nn.Linear(16*5*5, 120)
        self.fc2   = nn.Linear(120, 84)
        self.fc3   = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16*5*5)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

net = Net()

My confusion is regarding the following line.

x = x.view(-1, 16*5*5)

What does tensor.view() function do? I have seen its usage in many places, but I can't understand how it interprets its parameters.

What happens if I give negative values as parameters to the view() function? For example, what happens if I call, tensor_variable.view(1, 1, -1)?

Can anyone explain the main principle of view() function with some examples?


title: PyTorch: How to get the shape of a Tensor as a list of int

body:
In numpy, V.shape gives a tuple of ints of dimensions of V.

In tensorflow V.get_shape().as_list() gives a list of integers of the dimensions of V.

In pytorch, V.size() gives a size object, but how do I convert it to ints?


title: Pytorch tensor to numpy array

body:
I have a pytorch Tensor of size torch.Size([4, 3, 966, 1296])

I want to convert it to numpy array using the following code:

imgs = imgs.numpy()[:, ::-1, :, :]

Can anyone please explain what this code is doing ?


title: Model summary in pytorch

body:
How do I print the summary of a model in PyTorch like the model.summary() method does in Keras:
Model Summary:
____________________________________________________________________________________________________
Layer (type)                     Output Shape          Param #     Connected to
====================================================================================================
input_1 (InputLayer)             (None, 1, 15, 27)     0
____________________________________________________________________________________________________
convolution2d_1 (Convolution2D)  (None, 8, 15, 27)     872         input_1[0][0]
____________________________________________________________________________________________________
maxpooling2d_1 (MaxPooling2D)    (None, 8, 7, 27)      0           convolution2d_1[0][0]
____________________________________________________________________________________________________
flatten_1 (Flatten)              (None, 1512)          0           maxpooling2d_1[0][0]
____________________________________________________________________________________________________
dense_1 (Dense)                  (None, 1)             1513        flatten_1[0][0]
====================================================================================================
Total params: 2,385
Trainable params: 2,385
Non-trainable params: 0


title: Pytorch reshape tensor dimension

body:
For example, I have 1D vector with dimension (5). I would like to reshape it into 2D matrix (1,5).

Here is how I do it with numpy

>>> import numpy as np
>>> a = np.array([1,2,3,4,5])
>>> a.shape
(5,)
>>> a = np.reshape(a, (1,5))
>>> a.shape
(1, 5)
>>> a
array([[1, 2, 3, 4, 5]])
>>>

But how can I do that with Pytorch Tensor (and Variable). I don't want to switch back to numpy and switch to Torch variable again, because it will loss backpropagation information.

Here is what I have in Pytorch

>>> import torch
>>> from torch.autograd import Variable
>>> a = torch.Tensor([1,2,3,4,5])
>>> a

 1
 2
 3
 4
 5
[torch.FloatTensor of size 5]

>>> a.size()
(5L,)
>>> a_var = variable(a)
>>> a_var = Variable(a)
>>> a_var.size()
(5L,)
.....do some calculation in forward function
>>> a_var.size()
(5L,)

Now I want it size to be (1, 5).
How can I resize or reshape the dimension of pytorch tensor in Variable without loss grad information. (because I will feed into another model before backward)


title: No module named "Torch"

body:
I installed pytorch via
conda install pytorch-cpu torchvision-cpu -c pytorch
And I also tried
pip3 install https://download.pytorch.org/whl/cpu/torch-1.0.1-cp36-cp36m-win_amd64.whl

pip3 install torchvision

Both installed successfully!
But, it only works in jupiter notebook. Whenever I try to execute a script from the console, I get the error message:
No module named "torch"
How can I fix this?


title: How to install pytorch in Anaconda with conda or pip?

body:
I am trying to install pytorch in Anaconda to work with Python 3.5 in Windows. Following the instructions in pytorch.org I introduced the following code in Anaconda:

pip3 install torch torchvision

But the following error came in:

Command "python setup.py egg_info" failed with error code 1 in C:\Users\sluis\AppData\Local\Temp\pip-install-qmrvz7b9\torch\

By searching on the web I found out that it may be because of setuptools being out of date but I checked and have it updated. I also tried:

conda install -c peterjc123 pytorch cuda80

But the following error arise:

The following specifications were found to be in conflict:
  - pytorch
Use "conda info <package>" to see the dependencies for each package.

I also tried to load the pytorch's tar.bz2 file which I download in the following website:

anaconda.org/peterjc123/pytorch/files

And then just do:

$ conda install filename.tar.bz2

But I got the following error:

Error: HTTPError: 404 Client Error: None for url: file:///C|/Users/sluis/pytorch-0.3.1-py36_cuda80_cudnn6he774522_2.tar.bz2: file:///C|/Users/sluis/pytorch-0.3.1-py36_cuda80_cudnn6he774522_2.tar.bz2

I am quite new to this programming world so I don't really know how to dig more on the errors. Anyone knows how to get pytorch installed?

Edit: As suggested in the comments I tried:

conda install pytorch torchivsion -c pytorch

And I got the following error:

Error: Packages missing in current win-64 channels:
 - pytorch
 - torchvision

I did:

anaconda search -t conda torchvision

And tried to install dericlk/torchvision using the following command:

conda install -c derickl torchvision

But I am getting the same error:

Error: Package missing in current win-64 channels:
  - torchvision

I couldn't find any torchvisionpackages for win-64.

conda list is giving me the following:

# packages in environment at C:\Users\aaaa\AppData\Local\Continuum\Anaconda3\envs\torchenv2:
#
mkl-include               2018.0.2                      1    anaconda
certifi                   2016.2.28                py35_0
cffi                      1.10.0                   py35_0
cmake                     3.6.3                    vc14_0  [vc14]
openmp                    2018.0.0                intel_8    intel
mkl                       2017.0.3                      0
numpy                     1.13.1                   py35_0
pip                       10.0.0                    <pip>
pip                       9.0.1                    py35_1
pycparser                 2.18                     py35_0
python                    3.5.4                         0
pyyaml                    3.12                     py35_0
setuptools                36.4.0                   py35_1
typing                    3.6.2                    py35_0
vc                        14                            0
vs2015_runtime            14.0.25420                    0
wheel                     0.29.0                   py35_0
wincertstore              0.2                      py35_0
zlib                      1.2.11                   vc14_0  [vc14]

=======


title: Why do we need to call zero_grad() in PyTorch?

body:
The method zero_grad() needs to be called during training. But the documentation is not very helpful

|  zero_grad(self)
|      Sets gradients of all model parameters to zero.

Why do we need to call this method?


title: How to do product of matrices in PyTorch

body:
In numpy I can do a simple matrix multiplication like this:

a = numpy.arange(2*3).reshape(3,2)
b = numpy.arange(2).reshape(2,1)
print(a)
print(b)
print(a.dot(b))

However, when I am trying this with PyTorch Tensors, this does not work:

a = torch.Tensor([[1, 2, 3], [1, 2, 3]]).view(-1, 2)
b = torch.Tensor([[2, 1]]).view(2, -1)
print(a)
print(a.size())

print(b)
print(b.size())

print(torch.dot(a, b))

This code throws the following error:

  RuntimeError: inconsistent tensor size at
  /Users/soumith/code/builder/wheel/pytorch-src/torch/lib/TH/generic/THTensorMath.c:503

Any ideas how matrix multiplication can be conducted in PyTorch?


title: How to fix RuntimeError "Expected object of scalar type Float but got scalar type Double for argument"?

body:
I'm trying to train a classifier via PyTorch. However, I am experiencing problems with training when I feed the model with training data.
I get this error on y_pred = model(X_trainTensor):

  RuntimeError: Expected object of scalar type Float but got scalar type Double for argument #4 'mat1'

Here are key parts of my code:

# Hyper-parameters
D_in = 47  # there are 47 parameters I investigate
H = 33
D_out = 2  # output should be either 1 or 0

# Format and load the data
y = np.array( df['target'] )
X = np.array( df.drop(columns = ['target'], axis = 1) )
X_train, X_test, y_train, y_test = train_test_split(X, y, train_size = 0.8)  # split training/test data

X_trainTensor = torch.from_numpy(X_train) # convert to tensors
y_trainTensor = torch.from_numpy(y_train)
X_testTensor = torch.from_numpy(X_test)
y_testTensor = torch.from_numpy(y_test)

# Define the model
model = torch.nn.Sequential(
    torch.nn.Linear(D_in, H),
    torch.nn.ReLU(),
    torch.nn.Linear(H, D_out),
    nn.LogSoftmax(dim = 1)
)

# Define the loss function
loss_fn = torch.nn.NLLLoss()

for i in range(50):
    y_pred = model(X_trainTensor)
    loss = loss_fn(y_pred, y_trainTensor)
    model.zero_grad()
    loss.backward()
    with torch.no_grad():
        for param in model.parameters():
            param -= learning_rate * param.grad


title: Adding L1/L2 regularization in PyTorch?

body: Is there any way, I can add simple L1/L2 regularization in PyTorch? We can probably compute the regularized loss by simply adding the data_loss with the reg_loss but is there any explicit way, any support from PyTorch library to do it more easily without doing it manually?


title: How can l uninstall PyTorch?

body:
I can't find any command to uninstall and remove all PyTorch dependencies. Even on the pytorch.org website.

I installed PyTorch with

conda install pytorch torchvision cuda80 -c soumith


title: How to fix this strange error: "RuntimeError: CUDA error: out of memory"

body:
I ran a code about the deep learning network,first I trained the network,and it works well,but this error occurs when running to the validate network.

I have five epoch,every epoch has a process of training and validation. I met the error when validate in the first epoch. So I don not run the validate code, I found that code can run to the second epoch and have no error.

My code:

for epoch in range(10,15): # epoch: 10~15
    if(options["training"]["train"]):
        trainer.epoch(model, epoch)

    if(options["validation"]["validate"]):
    #if(epoch == 14):
        validator.epoch(model)

I feel the code of validation may have some bugs. But I can not find that.


I am trying to import a package "torch".
For same, I tried to install it using pip command as below, installation even started but after few seconds it got error

below is the command that I executed

pip install torch

Error that I got:

from tools.nnwrap import generate_wrappers as generate_nn_wrappers
    ModuleNotFoundError: No module named 'tools.nnwrap'

OS: Windows.
IDE : pyCharm

I got the only link related to this issue but I was not able to interpret it.

https://www.gitmemory.com/torch


title: How to install pytorch in windows?

body:
I am trying to install pytorch on windows and there is one which is available for it but shows an error.

conda install -c peterjc123 pytorch=0.1.12


title: How do I split a custom dataset into training and test datasets?

body:
import pandas as pd
import numpy as np
import cv2
from torch.utils.data.dataset import Dataset

class CustomDatasetFromCSV(Dataset):
    def __init__(self, csv_path, transform=None):
        self.data = pd.read_csv(csv_path)
        self.labels = pd.get_dummies(self.data['emotion']).as_matrix()
        self.height = 48
        self.width = 48
        self.transform = transform

    def __getitem__(self, index):
        pixels = self.data['pixels'].tolist()
        faces = []
        for pixel_sequence in pixels:
            face = [int(pixel) for pixel in pixel_sequence.split(' ')]
            # print(np.asarray(face).shape)
            face = np.asarray(face).reshape(self.width, self.height)
            face = cv2.resize(face.astype('uint8'), (self.width, self.height))
            faces.append(face.astype('float32'))
        faces = np.asarray(faces)
        faces = np.expand_dims(faces, -1)
        return faces, self.labels

    def __len__(self):
        return len(self.data)

This is what I could manage to do by using references from other repositories.
However, I want to split this dataset into train and test.

How can I do that inside this class? Or do I need to make a separate class to do that?


title: Torch sum a tensor along an axis

body:
ipdb> outputs.size()
torch.Size([10, 100])
ipdb> print sum(outputs,0).size(),sum(outputs,1).size(),sum(outputs,2).size()
(100L,) (100L,) (100L,)

How do I sum over the columns instead?


title: Convert PyTorch tensor to python list

body:
How do I convert a PyTorch Tensor into a python list?

My current use case is to convert a tensor of size [1, 2048, 1, 1] into a list of 2048 elements.

My tensor has floating point values. Is there a solution which also accounts for int and possibly other data types?


title: What does model.train() do in PyTorch?

body:
Does it call forward() in nn.Module? I thought when we call the model, forward method is being used.
Why do we need to specify train()?


title: PyTorch - What does contiguous() do?

body:
I was going through this example of a LSTM language model on github (link).
What it does in general is pretty clear to me. But I'm still struggling to understand what calling contiguous() does, which occurs several times in the code.
For example in line 74/75 of the code input and target sequences of the LSTM are created.
Data (stored in ids) is 2-dimensional where first dimension is the batch size.

for i in range(0, ids.size(1) - seq_length, seq_length):
    # Get batch inputs and targets
    inputs = Variable(ids[:, i:i+seq_length])
    targets = Variable(ids[:, (i+1):(i+1)+seq_length].contiguous())

So as a simple example, when using batch size 1 and seq_length 10 inputs and targets looks like this:
inputs Variable containing:
0     1     2     3     4     5     6     7     8     9
[torch.LongTensor of size 1x10]

targets Variable containing:
1     2     3     4     5     6     7     8     9    10
[torch.LongTensor of size 1x10]

So in general my question is, what does contiguous() do and why do I need it?
Further I don't understand why the method is called for the target sequence and but not the input sequence as both variables are comprised of the same data.
How could targets be non-contiguous and inputs still be contiguous?

EDIT:
I tried to leave out calling contiguous(), but this leads to an  error message when computing the loss.
RuntimeError: invalid argument 1: input is not contiguous at .../src/torch/lib/TH/generic/THTensor.c:231

So obviously calling contiguous() in this example is necessary.


title: How Pytorch Tensor get the index of specific value

body:
In python list, we can use list.index(somevalue). How can pytorch do this?
For example:

    a=[1,2,3]
    print(a.index(2))

Then, 1 will be output. How can a pytorch tensor do this without converting it to a python list?


title: Using Dropout in Pytorch: nn.Dropout vs. F.dropout

body:
By using pyTorch there is two ways to dropout
 torch.nn.Dropout and torch.nn.functional.Dropout.

I struggle to see the difference between the use of them:

When to use what?
Does it make a difference?

I don't see any performance difference when I switched them around.


title: CUDA runtime error (59) : device-side assert triggered

body:
I have access to Tesla K20c, I am running ResNet50 on CIFAR10 dataset...
Then I get the error as: THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1524584710464/work/aten/src/THC/generated/../generic/THCTensorMathPointwise.cu line=265 error=59 : device-side assert triggered
Traceback (most recent call last):
File "main.py", line 109, in <module>
train(loader_train, model, criterion, optimizer)
File "main.py", line 54, in train
optimizer.step()
 File "/usr/local/anaconda35/lib/python3.6/site-packages/torch/optim/sgd.py", line 93, in step
d_p.add_(weight_decay, p.data)
 RuntimeError: cuda runtime error (59) : device-side assert triggered at /opt/conda/conda-bld/pytorch_1524584710464/work/aten/src/THC/generated/../generic/THCTensorMathPointwise.cu:265
 How to resolve this error


title: Embedding in pytorch

body:
I have checked the PyTorch tutorial and questions similar to this one on Stackoverflow.

I get confused; does the embedding in pytorch (Embedding) make the similar words closer to each other? And do I just need to give to it all the sentences? Or it is just a lookup table and I need to code the model?


title: Convert Pandas dataframe to PyTorch tensor?

body:
I want to train a simple neural network with PyTorch on a pandas dataframe df.
One of the columns is named "Target", and it is the target variable of the network. How can I use this dataframe as input to the PyTorch network?
I tried this, but it doesn't work:
import pandas as pd
import torch.utils.data as data_utils

target = pd.DataFrame(df['Target'])
train = data_utils.TensorDataset(df, target)
train_loader = data_utils.DataLoader(train, batch_size=10, shuffle=True)


title: Understanding torch.nn.Parameter

body:
I am new to pytorch and I have difficulty in understanding how torch.nn.Parameter() works.

I have gone through the documentation in https://pytorch.org/docs/stable/nn.html but could make a very little sense out of it.

Can someone please help?

The code snippet that I am working on:

def __init__(self, weight):
    super(Net, self).__init__()
    # initializes the weights of the convolutional layer to be the weights of the 4 defined filters
    k_height, k_width = weight.shape[2:]
    # assumes there are 4 grayscale filters
    self.conv = nn.Conv2d(1, 4, kernel_size=(k_height, k_width), bias=False)
    self.conv.weight = torch.nn.Parameter(weight)


I am trying to train the following CNN as follows, but I keep getting the same error regarding .cuda() and I am not sure how to fix it. Here is a chunk of my code so far.

import matplotlib.pyplot as plt
import numpy as np
import torch
from torch import nn
from torch import optim
import torch.nn.functional as F
import torchvision
from torchvision import datasets, transforms, models
from torch.utils.data.sampler import SubsetRandomSampler

data_dir = "/home/ubuntu/ML2/ExamII/train2/"
valid_size = .2

# Normalize the test and train sets with torchvision
train_transforms = transforms.Compose([transforms.Resize(224),
                                           transforms.ToTensor(),
                                           ])

test_transforms = transforms.Compose([transforms.Resize(224),
                                          transforms.ToTensor(),
                                          ])

# ImageFolder class to load the train and test images
train_data = datasets.ImageFolder(data_dir, transform=train_transforms)
test_data = datasets.ImageFolder(data_dir, transform=test_transforms)

# Number of train images
num_train = len(train_data)
indices = list(range(num_train))
# Split = 20% of train images
split = int(np.floor(valid_size * num_train))
# Shuffle indices of train images
np.random.shuffle(indices)
# Subset indices for test and train
train_idx, test_idx = indices[split:], indices[:split]
# Samples elements randomly from a given list of indices
train_sampler = SubsetRandomSampler(train_idx)
test_sampler = SubsetRandomSampler(test_idx)
# Batch and load the images
trainloader = torch.utils.data.DataLoader(train_data, sampler=train_sampler, batch_size=1)
testloader = torch.utils.data.DataLoader(test_data, sampler=test_sampler, batch_size=1)

#print(trainloader.dataset.classes)

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = models.resnet50(pretrained=True)

model.fc = nn.Sequential(nn.Linear(2048, 512),
                                 nn.ReLU(),
                                 nn.Dropout(0.2),
                                 nn.Linear(512, 10),
                                 nn.LogSigmoid())
                                 # nn.LogSoftmax(dim=1))
# criterion = nn.NLLLoss()
criterion = nn.BCELoss()
optimizer = optim.Adam(model.fc.parameters(), lr=0.003)
model.to(device)

#Train the network
for epoch in range(2):  # loop over the dataset multiple times

    running_loss = 0.0
    for i, data in enumerate(trainloader, 0):
        # get the inputs; data is a list of [inputs, labels]
        inputs, labels = data

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = model(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()
        if i % 2000 == 1999:    # print every 2000 mini-batches
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 2000))
            running_loss = 0.0

print('Finished Training')

However, I keep getting this error in the console:

  RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same`

Any thoughts on how to fix it? I read that maybe the model hasn't been pushed into my GPU, but not sure how to fix it. Thanks!


title: How to do gradient clipping in pytorch?

body:
What is the correct way to perform gradient clipping in pytorch?

I have an exploding gradients problem, and I need to program my way around it.


title: pytorch how to set .requires_grad False

body:
I want to set some of my model frozen. Following the official docs:

with torch.no_grad():
    linear = nn.Linear(1, 1)
    linear.eval()
    print(linear.weight.requires_grad)

But it prints True instead of False. If I want to set the model in eval mode, what should I do?


title: What's the difference between reshape and view in pytorch?

body:
In numpy, we use ndarray.reshape() for reshaping an array.

I noticed that in pytorch, people use torch.view(...) for the same purpose, but at the same time, there is also a torch.reshape(...) existing.

So I am wondering what the differences are between them and when I should use either of them?


title: How do I flatten a tensor in pytorch?

body:
Given a tensor of multiple dimensions, how do I flatten it so that it has a single dimension?

Eg:

>>> t = torch.rand([2, 3, 5])
>>> t.shape
torch.Size([2, 3, 5])

How do I flatten it to have shape:

torch.Size([30])


title: What does model.eval() do in pytorch?

body:
I am using this code, and saw model.eval() in some cases.
I understand it is supposed to allow me to "evaluate my model", but I don't understand when I should and shouldn't use it, or how to turn if off.
I would like to run the above code to train the network, and also be able to run validation every epoch. I wasn't able to do it still.


title: How to get mini-batches in pytorch in a clean and efficient way?

body:
I was trying to do a simple thing which was train a linear model with Stochastic Gradient Descent (SGD) using torch:

import numpy as np

import torch
from torch.autograd import Variable

import pdb

def get_batch2(X,Y,M,dtype):
    X,Y = X.data.numpy(), Y.data.numpy()
    N = len(Y)
    valid_indices = np.array( range(N) )
    batch_indices = np.random.choice(valid_indices,size=M,replace=False)
    batch_xs = torch.FloatTensor(X[batch_indices,:]).type(dtype)
    batch_ys = torch.FloatTensor(Y[batch_indices]).type(dtype)
    return Variable(batch_xs, requires_grad=False), Variable(batch_ys, requires_grad=False)

def poly_kernel_matrix( x,D ):
    N = len(x)
    Kern = np.zeros( (N,D+1) )
    for n in range(N):
        for d in range(D+1):
            Kern[n,d] = x[n]**d;
    return Kern

## data params
N=5 # data set size
Degree=4 # number dimensions/features
D_sgd = Degree+1
##
x_true = np.linspace(0,1,N) # the real data points
y = np.sin(2*np.pi*x_true)
y.shape = (N,1)
## TORCH
dtype = torch.FloatTensor
# dtype = torch.cuda.FloatTensor # Uncomment this to run on GPU
X_mdl = poly_kernel_matrix( x_true,Degree )
X_mdl = Variable(torch.FloatTensor(X_mdl).type(dtype), requires_grad=False)
y = Variable(torch.FloatTensor(y).type(dtype), requires_grad=False)
## SGD mdl
w_init = torch.zeros(D_sgd,1).type(dtype)
W = Variable(w_init, requires_grad=True)
M = 5 # mini-batch size
eta = 0.1 # step size
for i in range(500):
    batch_xs, batch_ys = get_batch2(X_mdl,y,M,dtype)
    # Forward pass: compute predicted y using operations on Variables
    y_pred = batch_xs.mm(W)
    # Compute and print loss using operations on Variables. Now loss is a Variable of shape (1,) and loss.data is a Tensor of shape (1,); loss.data[0] is a scalar value holding the loss.
    loss = (1/N)*(y_pred - batch_ys).pow(2).sum()
    # Use autograd to compute the backward pass. Now w will have gradients
    loss.backward()
    # Update weights using gradient descent; w1.data are Tensors,
    # w.grad are Variables and w.grad.data are Tensors.
    W.data -= eta * W.grad.data
    # Manually zero the gradients after updating weights
    W.grad.data.zero_()

#
c_sgd = W.data.numpy()
X_mdl = X_mdl.data.numpy()
y = y.data.numpy()
#
Xc_pinv = np.dot(X_mdl,c_sgd)
print('J(c_sgd) = ', (1/N)*(np.linalg.norm(y-Xc_pinv)**2) )
print('loss = ',loss.data[0])

the code runs fine and all though my get_batch2 method seems really dum/naive, its probably because I am new to pytorch but I have not found a good place where they discuss how to retrieve data batches. I went through their tutorials (http://pytorch.org/tutorials/beginner/pytorch_with_examples.html) and through the data set (http://pytorch.org/tutorials/beginner/data_loading_tutorial.html) with no luck. The tutorials all seem to assume that one already has the batch and batch-size at the beginning and then proceeds to train with that data without changing it (specifically look at http://pytorch.org/tutorials/beginner/pytorch_with_examples.html#pytorch-variables-and-autograd).

So my question is do I really need to turn my data back into numpy so that I can fetch some random sample of it and then turn it back to pytorch with Variable to be able to train in memory? Is there no way to get mini-batches with torch?

I looked at a few functions torch provides but with no luck:

#pdb.set_trace()
#valid_indices = torch.arange(0,N).numpy()
#valid_indices = np.array( range(N) )
#batch_indices = np.random.choice(valid_indices,size=M,replace=False)
#indices = torch.LongTensor(batch_indices)
#batch_xs, batch_ys = torch.index_select(X_mdl, 0, indices), torch.index_select(y, 0, indices)
#batch_xs,batch_ys = torch.index_select(X_mdl, 0, indices), torch.index_select(y, 0, indices)

even though the code I provided works fine I am worried that its not an efficient implementation AND that if I were to use GPUs that there would be a considerable further slow down (because my guess it putting things in memory and then fetching them back to put them GPU like that is silly).

I implemented a new one based on the answer that suggested to use torch.index_select():

def get_batch2(X,Y,M):
    '''
    get batch for pytorch model
    '''
    # TODO fix and make it nicer, there is pytorch forum question
    #X,Y = X.data.numpy(), Y.data.numpy()
    X,Y = X, Y
    N = X.size()[0]
    batch_indices = torch.LongTensor( np.random.randint(0,N+1,size=M) )
    pdb.set_trace()
    batch_xs = torch.index_select(X,0,batch_indices)
    batch_ys = torch.index_select(Y,0,batch_indices)
    return Variable(batch_xs, requires_grad=False), Variable(batch_ys, requires_grad=False)

however, this seems to have issues because it does not work if X,Y are NOT variables...which is really odd. I added this to the pytorch forum: https://discuss.pytorch.org/t/how-to-get-mini-batches-in-pytorch-in-a-clean-and-efficient-way/10322

Right now what I am struggling with is making this work for gpu. My most current version:

def get_batch2(X,Y,M,dtype):
    '''
    get batch for pytorch model
    '''
    # TODO fix and make it nicer, there is pytorch forum question
    #X,Y = X.data.numpy(), Y.data.numpy()
    X,Y = X, Y
    N = X.size()[0]
    if dtype ==  torch.cuda.FloatTensor:
        batch_indices = torch.cuda.LongTensor( np.random.randint(0,N,size=M) )# without replacement
    else:
        batch_indices = torch.LongTensor( np.random.randint(0,N,size=M) ).type(dtype)  # without replacement
    pdb.set_trace()
    batch_xs = torch.index_select(X,0,batch_indices)
    batch_ys = torch.index_select(Y,0,batch_indices)
    return Variable(batch_xs, requires_grad=False), Variable(batch_ys, requires_grad=False)

the error:

RuntimeError: tried to construct a tensor from a int sequence, but found an item of type numpy.int64 at index (0)

I don't get it, do I really have to do:

ints = [ random.randint(0,N) for i i range(M)]

to get the integers?

It would also be ideal if the data could be a variable. It seems that it torch.index_select does not work for Variable type data.

this list of integers thing still doesn't work:

TypeError: torch.addmm received an invalid combination of arguments - got (int, torch.cuda.FloatTensor, int, torch.cuda.FloatTensor, torch.FloatTensor, out=torch.cuda.FloatTensor), but expected one of:
 * (torch.cuda.FloatTensor source, torch.cuda.FloatTensor mat1, torch.cuda.FloatTensor mat2, *, torch.cuda.FloatTensor out)
 * (torch.cuda.FloatTensor source, torch.cuda.sparse.FloatTensor mat1, torch.cuda.FloatTensor mat2, *, torch.cuda.FloatTensor out)
 * (float beta, torch.cuda.FloatTensor source, torch.cuda.FloatTensor mat1, torch.cuda.FloatTensor mat2, *, torch.cuda.FloatTensor out)
 * (torch.cuda.FloatTensor source, float alpha, torch.cuda.FloatTensor mat1, torch.cuda.FloatTensor mat2, *, torch.cuda.FloatTensor out)
 * (float beta, torch.cuda.FloatTensor source, torch.cuda.sparse.FloatTensor mat1, torch.cuda.FloatTensor mat2, *, torch.cuda.FloatTensor out)
 * (torch.cuda.FloatTensor source, float alpha, torch.cuda.sparse.FloatTensor mat1, torch.cuda.FloatTensor mat2, *, torch.cuda.FloatTensor out)
 * (float beta, torch.cuda.FloatTensor source, float alpha, torch.cuda.FloatTensor mat1, torch.cuda.FloatTensor mat2, *, torch.cuda.FloatTensor out)
      didn't match because some of the arguments have invalid types: (int, torch.cuda.FloatTensor, int, torch.cuda.FloatTensor, torch.FloatTensor, out=torch.cuda.FloatTensor)
 * (float beta, torch.cuda.FloatTensor source, float alpha, torch.cuda.sparse.FloatTensor mat1, torch.cuda.FloatTensor mat2, *, torch.cuda.FloatTensor out)
      didn't match because some of the arguments have invalid types: (int, torch.cuda.FloatTensor, int, torch.cuda.FloatTensor, torch.FloatTensor, out=torch.cuda.FloatTensor)


title: How do I display a single image in PyTorch?

body:
I want to display a single image loaded using ImageLoader and stored in a PyTorch Tensor. When I try to display it via plt.imshow(image) I get:
TypeError: Invalid dimensions for image data

The .shape of the tensor is:
torch.Size([3, 244, 244])

How do I display a PyTorch tensor as an image?


title: Cannot convert list to array: ValueError: only one element tensors can be converted to Python scalars

body:
I'm currently working with the PyTorch framework and trying to understand foreign code. I got an indices issue and wanted to print the shape of a list.
The only way of doing so (as far as Google tells me) is to convert the list into a numpy array and then getting the shape with numpy.ndarray.shape().

But trying to convert my list into an array, I got a ValueError: only one element tensors can be converted to Python scalars.

My List is a converted PyTorch Tensor (list(pytorchTensor)) and looks somewhat like this:

  [tensor([[-0.2781, -0.2567, -0.2353,  ..., -0.9640, -0.9855, -1.0069],
          [-0.2781, -0.2567, -0.2353,  ..., -1.0069, -1.0283, -1.0927],
          [-0.2567, -0.2567, -0.2138,  ..., -1.0712, -1.1141, -1.1784],
          ...,
          [-0.6640, -0.6425, -0.6211,  ..., -1.0712, -1.1141, -1.0927],
          [-0.6640, -0.6425, -0.5997,  ..., -0.9426, -0.9640, -0.9640],
          [-0.6640, -0.6425, -0.5997,  ..., -0.9640, -0.9426, -0.9426]]),   tensor([[-0.0769, -0.0980, -0.076                          9,  ..., -0.9388, -0.9598, -0.9808],
          [-0.0559, -0.0769, -0.0980,  ..., -0.9598, -1.0018, -1.0228],
          [-0.0559, -0.0769, -0.0769,  ..., -1.0228, -1.0439, -1.0859],
          ...,
          [-0.4973, -0.4973, -0.4973,  ..., -1.0018, -1.0439, -1.0228],
          [-0.4973, -0.4973, -0.4973,  ..., -0.8757, -0.9177, -0.9177],
          [-0.4973, -0.4973, -0.4973,  ..., -0.9177, -0.8967, -0.8967]]),   tensor([[-0.1313, -0.1313, -0.110                          0,  ..., -0.8115, -0.8328, -0.8753],
          [-0.1313, -0.1525, -0.1313,  ..., -0.8541, -0.8966, -0.9391],
          [-0.1100, -0.1313, -0.1100,  ..., -0.9391, -0.9816, -1.0666],
          ...,
          [-0.4502, -0.4714, -0.4502,  ..., -0.8966, -0.8966, -0.8966],
          [-0.4502, -0.4714, -0.4502,  ..., -0.8115, -0.8115, -0.7903],
          [-0.4502, -0.4714, -0.4502,  ..., -0.8115, -0.7690, -0.7690]])]

Is there a way of getting the shape of that list without converting it into a numpy array?


title: Pytorch preferred way to copy a tensor

body:
There seems to be several ways to create a copy of a tensor in Pytorch, including

y = tensor.new_tensor(x) #a

y = x.clone().detach() #b

y = torch.empty_like(x).copy_(x) #c

y = torch.tensor(x) #d

b is explicitly preferred over a and d according to a UserWarning I get if I execute either a or d. Why is it preferred? Performance? I'd argue it's less readable.

Any reasons for/against using c?


title: Pytorch softmax: What dimension to use?

body:
The function torch.nn.functional.softmax takes two parameters: input and dim.  According to its documentation, the softmax operation is applied to all slices of input along the specified dim, and will rescale them so that the elements lie in the range (0, 1) and sum to 1.

Let input be:

input = torch.randn((3, 4, 5, 6))

Suppose I want the following, so that every entry in that array is 1:

sum = torch.sum(input, dim = 3) # sum's size is (3, 4, 5, 1)

How should I apply softmax?

softmax(input, dim = 0) # Way Number 0
softmax(input, dim = 1) # Way Number 1
softmax(input, dim = 2) # Way Number 2
softmax(input, dim = 3) # Way Number 3

My intuition tells me that is the last one, but I am not sure. English is not my first language and the use of the word along seemed confusing to me because of that.

I am not very clear on what "along" means, so I will use an example that could clarify things.  Suppose we have a tensor of size (s1, s2, s3, s4), and I want this to happen


title: CMake on Linux CentOS 7, how to force the system to use cmake3?

body:
I tried to install PyTorch on my Linux CentOS 7.3. I downloaded its package, ran this command and got this error:

sudo python setup.py install

running install
running build_deps
CMake Error at CMakeLists.txt:1 (cmake_minimum_required):
  CMake 3.0 or higher is required.  You are running version 2.8.12.2

-- Configuring incomplete, errors occurred!

So I tried to install CMake 3 by using the command

sudo yum -y install cmake3

The installation went alright, but the system still uses cmake2.8 as default.
If I type the yum info comnmand, I get this:

sudo yum info cmake

Installed Packages
Name        : cmake
Arch        : x86_64
Version     : 2.8.12.2
Release     : 2.el7
Size        : 27 M
Repo        : installed
From repo   : base
Summary     : Cross-platform make system
URL         : http://www.cmake.org
License     : BSD and MIT and zlib
Description : CMake is used to control the software compilation process using simple
            : platform and compiler independent configuration files. CMake generates
            : native makefiles and workspaces that can be used in the compiler
            : environment of your choice. CMake is quite sophisticated: it is possible
            : to support complex environments requiring system configuration, preprocessor
            : generation, code generation, and template instantiation.

So, the problem is clear: the system still sees cmake2.8 as default, and therefore Python does not use cmake3 for its PyTorch installation.
How can I solve this problem?

Thanks


title: How to load a list of numpy arrays to pytorch dataset loader?

body:
I have a huge list of numpy arrays, where each array represents an image and I want to load it using torch.utils.data.Dataloader object. But the documentation of torch.utils.data.Dataloader mentions that it loads data directly from a folder. How do I modify it for my cause? I am new to pytorch and any help would be greatly appreciated.
my numpy array for a single image looks something like this. The image is RBG image.
[[[ 70  82  94]
  [ 67  81  93]
  [ 66  82  94]
  ...,
  [182 182 188]
  [183 183 189]
  [188 186 192]]

 [[ 66  80  92]
  [ 62  78  91]
  [ 64  79  95]
  ...,
  [176 176 182]
  [178 178 184]
  [180 180 186]]

 [[ 62  82  93]
  [ 62  81  96]
  [ 65  80  99]
  ...,
  [169 172 177]
  [173 173 179]
  [172 172 178]]

 ...,


title: Using CUDA with pytorch?

body:
I have searched on here but I found only outdated posts.
I want to run the training on my GPU. I found on some forums that I need to apply .cuda() on anything I want to use CUDA with (I've applied it to everything I could without making the program crash). Surprisingly, this makes the training even slower.
Then, I found that you could use this torch.set_default_tensor_type('torch.cuda.FloatTensor') to use CUDA. With both enabled, nothing changes. What is happening?
Is there a way to reliably enable CUDA on the whole model?

Also, what does MyModel() mean? I need more tangible examples, like code examples.  (This is the post I am referring to)


title: What does "unsqueeze" do in Pytorch?

body:
I cannot understand how the example in the PyTorch documentation corresponds to the explanation:

Returns a new tensor with a dimension of size one inserted at the specified position. [...]
>>> x = torch.tensor([1, 2, 3, 4])
>>> torch.unsqueeze(x, 0)
tensor([[ 1,  2,  3,  4]])
>>> torch.unsqueeze(x, 1)
tensor([[ 1],
        [ 2],
        [ 3],
        [ 4]])


title: Check the total number of parameters in a PyTorch model

body: How to count the total number of parameters in a PyTorch model? Something similar to model.count_params() in Keras.


title: Cross Entropy in PyTorch

body:
I'm a bit confused by the cross entropy loss in PyTorch.

Considering this example:

import torch
import torch.nn as nn
from torch.autograd import Variable

output = Variable(torch.FloatTensor([0,0,0,1])).view(1, -1)
target = Variable(torch.LongTensor([3]))

criterion = nn.CrossEntropyLoss()
loss = criterion(output, target)
print(loss)

I would expect the loss to be 0. But I get:

Variable containing:
 0.7437
[torch.FloatTensor of size 1]

As far as I know cross entropy can be calculated like this:

But shouldn't be the result then 1*log(1) = 0 ?

I tried different inputs like one-hot encodings, but this doesn't work at all, so it seems the input shape of the loss function is okay.

I would be really grateful if someone could help me out and tell me where my mistake is.

Thanks in advance!


title: How to clear Cuda memory in PyTorch

body:
I am trying to get the output of a neural network which I have already trained. The input is an image of the size 300x300. I am using a batch size of 1, but I still get a CUDA error: out of memory error after I have successfully got the output for 25 images.

I searched for some solutions online and came across torch.cuda.empty_cache(). But this still doesn't seem to solve the problem.

This is the code I am using.

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

train_x = torch.tensor(train_x, dtype=torch.float32).view(-1, 1, 300, 300)
train_x = train_x.to(device)
dataloader = torch.utils.data.DataLoader(train_x, batch_size=1, shuffle=False)

right = []
for i, left in enumerate(dataloader):
    print(i)
    temp = model(left).view(-1, 1, 300, 300)
    right.append(temp.to('cpu'))
    del temp
    torch.cuda.empty_cache()

This for loop runs for 25 times every time before giving the memory error.

Every time, I am sending a new image in the network for computation. So, I don't really need to store the previous computation results in the GPU after every iteration in the loop. Is there any way to achieve this?

Any help will be appreciated. Thanks.


title: Data Augmentation in PyTorch

body:
I am a little bit confused about the data augmentation performed in PyTorch. Now, as far as I know, when we are performing data augmentation, we are KEEPING our original dataset, and then adding other versions of it (Flipping, Cropping...etc). But that doesn't seem like happening in PyTorch. As far as I understood from the references, when we use data.transforms in PyTorch, then it applies them one by one. So for example:

data_transforms = {
    'train': transforms.Compose([
        transforms.RandomResizedCrop(224),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
    'val': transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
}

Here , for the training, we are first randomly cropping the image and resizing it to shape (224,224). Then we are taking these (224,224) images and horizontally flipping them. Therefore, our dataset is now containing ONLY the horizontally flipped images, so our original images are lost in this case.

Am I right? Is this understanding correct? If not, then where do we tell PyTorch in this code above (taken from Official Documentation) to keep the original images and resize them to the expected shape (224,224)?

Thanks


I am using a modified predict.py for testing a pruned SqueezeNet Model
[phung@archlinux SqueezeNet-Pruning]$ python predict.py --image 3_100.jpg --model model_prunned --num_class 2
prediction in progress
Traceback (most recent call last):
File “predict.py”, line 66, in
prediction = predict_image(imagepath)
File “predict.py”, line 52, in predict_image
index = output.data.numpy().argmax()
TypeError: can’t convert CUDA tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.
[phung@archlinux SqueezeNet-Pruning]$

I understand that numpy does not support GPU yet.
How shall I modify the code to get away from this error without invoking tensor copy data operation, tensor.cpu() ?


title: How do I visualize a net in Pytorch?

body:
import torch
import torch.nn as nn
import torch.optim as optim
import torch.utils.data as data
import torchvision.models as models
import torchvision.datasets as dset
import torchvision.transforms as transforms
from torch.autograd import Variable
from torchvision.models.vgg import model_urls
from torchviz import make_dot

batch_size = 3
learning_rate =0.0002
epoch = 50

resnet = models.resnet50(pretrained=True)
print resnet
make_dot(resnet)

I want to visualize resnet from the pytorch models. How can I do it? I tried to use torchviz but it gives an error:

'ResNet' object has no attribute 'grad_fn'


title: Calculate the accuracy every epoch in PyTorch

body:
I am working on a Neural Network problem, to classify data as 1 or 0. I am using Binary cross entropy loss to do this. The loss is fine, however, the accuracy is very low and isn't improving. I am assuming I did a mistake in the accuracy calculation. After every epoch, I am calculating the correct predictions after thresholding the output, and dividing that number by the total number of the dataset. Is there any thing wrong I did in the accuracy calculation? And why isn't it improving, but getting more worse?
This is my code:

net = Model()
criterion = torch.nn.BCELoss(size_average=True)
optimizer = torch.optim.SGD(net.parameters(), lr=0.1)

num_epochs = 100
for epoch in range(num_epochs):
    for i, (inputs,labels) in enumerate (train_loader):
        inputs = Variable(inputs.float())
        labels = Variable(labels.float())
        output = net(inputs)
        optimizer.zero_grad()
        loss = criterion(output, labels)
        loss.backward()
        optimizer.step()

    #Accuracy
    output = (output>0.5).float()
    correct = (output == labels).float().sum()
    print("Epoch {}/{}, Loss: {:.3f}, Accuracy: {:.3f}".format(epoch+1,num_epochs, loss.data[0], correct/x.shape[0]))

And this is the strange output I get:

Epoch 1/100, Loss: 0.389, Accuracy: 0.035
Epoch 2/100, Loss: 0.370, Accuracy: 0.036
Epoch 3/100, Loss: 0.514, Accuracy: 0.030
Epoch 4/100, Loss: 0.539, Accuracy: 0.030
Epoch 5/100, Loss: 0.583, Accuracy: 0.029
Epoch 6/100, Loss: 0.439, Accuracy: 0.031
Epoch 7/100, Loss: 0.429, Accuracy: 0.034
Epoch 8/100, Loss: 0.408, Accuracy: 0.035
Epoch 9/100, Loss: 0.316, Accuracy: 0.035
Epoch 10/100, Loss: 0.436, Accuracy: 0.035
Epoch 11/100, Loss: 0.365, Accuracy: 0.034
Epoch 12/100, Loss: 0.485, Accuracy: 0.031
Epoch 13/100, Loss: 0.392, Accuracy: 0.033
Epoch 14/100, Loss: 0.494, Accuracy: 0.030
Epoch 15/100, Loss: 0.369, Accuracy: 0.035
Epoch 16/100, Loss: 0.495, Accuracy: 0.029
Epoch 17/100, Loss: 0.415, Accuracy: 0.034
Epoch 18/100, Loss: 0.410, Accuracy: 0.035
Epoch 19/100, Loss: 0.282, Accuracy: 0.038
Epoch 20/100, Loss: 0.499, Accuracy: 0.031
Epoch 21/100, Loss: 0.446, Accuracy: 0.030
Epoch 22/100, Loss: 0.585, Accuracy: 0.026
Epoch 23/100, Loss: 0.419, Accuracy: 0.035
Epoch 24/100, Loss: 0.492, Accuracy: 0.031
Epoch 25/100, Loss: 0.537, Accuracy: 0.031
Epoch 26/100, Loss: 0.439, Accuracy: 0.033
Epoch 27/100, Loss: 0.421, Accuracy: 0.035
Epoch 28/100, Loss: 0.532, Accuracy: 0.034
Epoch 29/100, Loss: 0.234, Accuracy: 0.038
Epoch 30/100, Loss: 0.492, Accuracy: 0.027
Epoch 31/100, Loss: 0.407, Accuracy: 0.035
Epoch 32/100, Loss: 0.305, Accuracy: 0.038
Epoch 33/100, Loss: 0.663, Accuracy: 0.025
Epoch 34/100, Loss: 0.588, Accuracy: 0.031
Epoch 35/100, Loss: 0.329, Accuracy: 0.035
Epoch 36/100, Loss: 0.474, Accuracy: 0.033
Epoch 37/100, Loss: 0.535, Accuracy: 0.031
Epoch 38/100, Loss: 0.406, Accuracy: 0.033
Epoch 39/100, Loss: 0.513, Accuracy: 0.030
Epoch 40/100, Loss: 0.593, Accuracy: 0.030
Epoch 41/100, Loss: 0.265, Accuracy: 0.036
Epoch 42/100, Loss: 0.576, Accuracy: 0.031
Epoch 43/100, Loss: 0.565, Accuracy: 0.027
Epoch 44/100, Loss: 0.576, Accuracy: 0.030
Epoch 45/100, Loss: 0.396, Accuracy: 0.035
Epoch 46/100, Loss: 0.423, Accuracy: 0.034
Epoch 47/100, Loss: 0.489, Accuracy: 0.033
Epoch 48/100, Loss: 0.591, Accuracy: 0.029
Epoch 49/100, Loss: 0.415, Accuracy: 0.034
Epoch 50/100, Loss: 0.291, Accuracy: 0.039
Epoch 51/100, Loss: 0.395, Accuracy: 0.033
Epoch 52/100, Loss: 0.540, Accuracy: 0.026
Epoch 53/100, Loss: 0.436, Accuracy: 0.033
Epoch 54/100, Loss: 0.346, Accuracy: 0.036
Epoch 55/100, Loss: 0.519, Accuracy: 0.029
Epoch 56/100, Loss: 0.456, Accuracy: 0.031
Epoch 57/100, Loss: 0.425, Accuracy: 0.035
Epoch 58/100, Loss: 0.311, Accuracy: 0.039
Epoch 59/100, Loss: 0.406, Accuracy: 0.034
Epoch 60/100, Loss: 0.360, Accuracy: 0.035
Epoch 61/100, Loss: 0.476, Accuracy: 0.030
Epoch 62/100, Loss: 0.404, Accuracy: 0.034
Epoch 63/100, Loss: 0.382, Accuracy: 0.036
Epoch 64/100, Loss: 0.538, Accuracy: 0.031
Epoch 65/100, Loss: 0.392, Accuracy: 0.034
Epoch 66/100, Loss: 0.434, Accuracy: 0.033
Epoch 67/100, Loss: 0.479, Accuracy: 0.031
Epoch 68/100, Loss: 0.494, Accuracy: 0.031
Epoch 69/100, Loss: 0.415, Accuracy: 0.034
Epoch 70/100, Loss: 0.390, Accuracy: 0.036
Epoch 71/100, Loss: 0.330, Accuracy: 0.038
Epoch 72/100, Loss: 0.449, Accuracy: 0.030
Epoch 73/100, Loss: 0.315, Accuracy: 0.039
Epoch 74/100, Loss: 0.450, Accuracy: 0.031
Epoch 75/100, Loss: 0.562, Accuracy: 0.030
Epoch 76/100, Loss: 0.447, Accuracy: 0.031
Epoch 77/100, Loss: 0.408, Accuracy: 0.038
Epoch 78/100, Loss: 0.359, Accuracy: 0.034
Epoch 79/100, Loss: 0.372, Accuracy: 0.035
Epoch 80/100, Loss: 0.452, Accuracy: 0.034
Epoch 81/100, Loss: 0.360, Accuracy: 0.035
Epoch 82/100, Loss: 0.453, Accuracy: 0.031
Epoch 83/100, Loss: 0.578, Accuracy: 0.030
Epoch 84/100, Loss: 0.537, Accuracy: 0.030
Epoch 85/100, Loss: 0.483, Accuracy: 0.035
Epoch 86/100, Loss: 0.343, Accuracy: 0.036
Epoch 87/100, Loss: 0.439, Accuracy: 0.034
Epoch 88/100, Loss: 0.686, Accuracy: 0.023
Epoch 89/100, Loss: 0.265, Accuracy: 0.039
Epoch 90/100, Loss: 0.369, Accuracy: 0.035
Epoch 91/100, Loss: 0.521, Accuracy: 0.027
Epoch 92/100, Loss: 0.662, Accuracy: 0.027
Epoch 93/100, Loss: 0.581, Accuracy: 0.029
Epoch 94/100, Loss: 0.322, Accuracy: 0.034
Epoch 95/100, Loss: 0.375, Accuracy: 0.035
Epoch 96/100, Loss: 0.575, Accuracy: 0.031
Epoch 97/100, Loss: 0.489, Accuracy: 0.030
Epoch 98/100, Loss: 0.435, Accuracy: 0.033
Epoch 99/100, Loss: 0.440, Accuracy: 0.031
Epoch 100/100, Loss: 0.444, Accuracy: 0.033


title: How do you load MNIST images into Pytorch DataLoader?

body:
The pytorch tutorial for data loading and processing is quite specific to one example, could someone help me with what the function should look like for a more generic simple loading of images?

Tutorial: http://pytorch.org/tutorials/beginner/data_loading_tutorial.html

My Data:

I have the MINST dataset as jpg's in the following folder structure. (I know I can just use the dataset class, but this is purely to see how to load simple images into pytorch without csv's or complex features).

The folder name is the label and the images are 28x28 png's in greyscale, no transformations required.

data
    train
        0
            3.png
            5.png
            13.png
            23.png
            ...
        1
            3.png
            10.png
            11.png
            ...
        2
            4.png
            13.png
            ...
        3
            8.png
            ...
        4
            ...
        5
            ...
        6
            ...
        7
            ...
        8
            ...
        9
            ...


title: PyTorch: How to use DataLoaders for custom Datasets

body:
How to make use of the torch.utils.data.Dataset and torch.utils.data.DataLoader on your own data (not just the torchvision.datasets)?

Is there a way to use the inbuilt DataLoaders which they use on TorchVisionDatasets to be used on any dataset?


title: What does the gather function do in pytorch in layman terms?

body:
I have been through the official doc and this but it is hard to understand what is going on.

I am trying to understand a DQN source code and it uses the gather function on line 197.

Could someone explain in simple terms what the gather function does? What is the purpose of that function?


title: Pytorch Operation to detect NaNs

body:
Is there a Pytorch-internal procedure to detect NaNs in Tensors? Tensorflow has the tf.is_nan and the tf.check_numerics operations ... Does Pytorch have something similar, somewhere? I could not find something like this in the docs...

I am looking specifically for a Pytorch internal routine, since I would like this to happen on the GPU as well as on the CPU. This excludes numpy - based solutions (like np.isnan(sometensor.numpy()).any()) ...


title: Concatenate Two Tensors in Pytorch

body:
RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 2. Got 32 and 71 in dimension 0 at /pytorch/aten/src/THC/generic/THCTensorMath.cu:87

I have a tensor of shape [71 32 1].

I want to make it of shape [100 32 1] by padding zero vectors.

I tried by concatenating a padding vector of zeros of shape [29 32 1]. I get the error above.

I try with a padding vector of zeros of shape [29 32 1], I still get an error.

How could I create the required tensor?


title: How to get a uniform distribution in a range [r1,r2] in PyTorch?

body: I want to get a 2-D torch.Tensor with size [a,b] filled with values from a uniform distribution (in range [r1,r2]) in PyTorch.


title: How do I get the value of a tensor in PyTorch?

body:
Printing the tensor gives:
>>> x = torch.tensor([3])
>>> print(x)
tensor([3])

Likewise indexing its .data gives:
>>> x.data[0]
tensor(3)

How do I get just the value 3?


title: Why do we "pack" the sequences in PyTorch?

body:
I was trying to replicate How to use packing for variable-length sequence inputs for rnn but I guess I first need to understand why we need to "pack" the sequence.
I understand why we need to "pad" them but why is "packing" (through pack_padded_sequence) necessary?
Any high-level explanation would be appreciated!


title: reshaping a tensor with padding in pytorch

body:
I have a tensor with dimensions (30, 35, 49). I want to reshape it to (30, 35, 512) in order to be able to multiply with another tensor which has also the shape (30, 35, 512).

I want to do padding on the tensor with (30, 35, 49) dimension in order to make it (30, 35, 512) dimensional.

How can this be done?


title: What's the difference between torch.stack() and torch.cat() functions?

body:
OpenAI's REINFORCE and actor-critic example for reinforcement learning has the following code:

REINFORCE:

policy_loss = torch.cat(policy_loss).sum()

actor-critic:

loss = torch.stack(policy_losses).sum() + torch.stack(value_losses).sum()

One is using torch.cat, the other uses torch.stack.

As far as my understanding goes, the doc doesn't give any clear distinction between them.

I would be happy to know the differences between the functions.


title: Issues installing PyTorch 1.4 - "No matching distribution found for torch===1.4.0"

body:
Used the install guide on pytorch.org on how to install it and the command I'm using is
pip install torch===1.4.0 torchvision===0.5.0 -f https://download.pytorch.org/whl/torch_stable.html

But it's coming up with this error;

ERROR: Could not find a version that satisfies the requirement torch===1.4.0 (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)
ERROR: No matching distribution found for torch===1.4.0

Is this even a me-related issue? Can other people use this command?
Pip is installed and works for other modules, Python 3.8, CUDA version 10.1, Windows 10 Home 2004


title: AttributeError: 'collections.OrderedDict' object has no attribute 'eval'

body:
I have a model file which looks like this

OrderedDict([('inp.conv1.conv.weight',
          (0 ,0 ,0 ,.,.) =
           -1.5073e-01  6.4760e-02  1.9156e-01
            1.2175e-01  3.5886e-02  1.3992e-01
           -1.5903e-01  8.2055e-02  1.7820e-01

          (0 ,0 ,1 ,.,.) =
            1.0604e-01 -1.3653e-01  1.4803e-01
            6.0276e-02 -1.4674e-02  2.3059e-06
           -6.2192e-02 -5.1061e-03 -7.4145e-03

          (0 ,0 ,2 ,.,.) =
           -5.5632e-02  3.5326e-02  6.5108e-02
            1.1411e-01 -4.4160e-02  8.2610e-02
            8.9979e-02 -3.5454e-02  4.2549e-02

          (1 ,0 ,0 ,.,.) =
            4.8523e-02 -4.3961e-02  5.3614e-02
           -1.2644e-01  1.2777e-01  8.9547e-02
            3.8392e-02  2.7016e-02 -1.4552e-01

          (1 ,0 ,1 ,.,.) =
            9.5537e-02  2.8748e-02  3.9772e-02
           -6.2410e-02  1.1264e-01  7.8663e-02
           -2.6374e-02  1.4401e-01 -1.7109e-01

          (1 ,0 ,2 ,.,.) =
            5.1791e-02 -1.6388e-01 -1.7605e-01
            3.5028e-02  7.7164e-02 -1.4499e-01
           -2.9189e-02  2.7064e-03 -2.3228e-02

          (2 ,0 ,0 ,.,.) =
           -7.4446e-03 -9.7202e-02 -1.4704e-01
           -1.0019e-02  8.1780e-02 -5.3530e-02
           -1.8412e-01  1.5988e-01 -1.3450e-01

          (2 ,0 ,1 ,.,.) =
           -1.1075e-01 -5.2478e-02  6.0658e-02
            1.6739e-01 -2.9360e-02  1.2621e-01
            2.0686e-02  1.1468e-01  1.2282e-01

I want to do inference on this model, but when i do model.eval() i get,

AttributeError: 'collections.OrderedDict' object has no attribute 'eval
Not really sure how to go about this, any suggestions on how i can get this fixed will be really helpful,Thanks in advance


title: How to write a PyTorch sequential model?

body:
So far, I wrote my MLP, RNN and CNN in Keras, but now PyTorch is gaining popularity inside deep learning communities, and so I also started to learn this framework. I am a big fan of sequential models in Keras, which allow us to make simple models very fast. I also saw that PyTorch has this functionality, but I don't know how to code one. I tried this way

import torch
import torch.nn as nn

net = nn.Sequential()
net.add(nn.Linear(3, 4))
net.add(nn.Sigmoid())
net.add(nn.Linear(4, 1))
net.add(nn.Sigmoid())
net.float()

print(net)

but it is giving this error

  AttributeError: 'Sequential' object has no attribute 'add'

Also, if possible, can you give simple examples for RNN and CNN models in PyTorch sequential model?


title: How to create a normal distribution in pytorch

body: I want to create a random normal distribution in pytorch and mean and std are 4, 0.5 respectively. I didn't find a API for it. Anyone knows? Thanks very much.


title: What is the class definition of nn.Linear in PyTorch?

body:
I have the following code for PyTorch:
import torch.nn as nn
import torch.nn.functional as F

class Network(nn.Module):
    def __init__(self):
        super().__init__()
        self.hidden = nn.Linear(784, 256)
        self.output = nn.Linear(256, 10)

    def forward(self, x):
        x = F.sigmoid(self.hidden(x))
        x = F.softmax(self.output(x), dim=1)

        return x

My question: What is this self.hidden?
It returns from nn.Linear and it can take x as argument. What exactly is the purpose of self.hidden?


title: pytorch, AttributeError: module 'torch' has no attribute 'Tensor'

body:
I'm working with Python 3.5.1 on a computer having CentOS Linux 7.3.1611 (Core) operating system.

I'm trying to use PyTorch and I'm getting started with this tutorial.

Unfortunately, the #4 line of the example creates troubles:

>>> torch.Tensor(5, 3)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'torch' has no attribute 'Tensor'

I cannot understand this error... of course in Torch the 'torch' does have an attribute 'Tensor'. The same command works in Torch.

How can I solve this problem?


title: How to use torch.stack function

body:
I have a question about torch.stack

I have 2 tensors, a.shape=(2, 3, 4) and b.shape=(2, 3).
How to stack them without in-place operation?


title: PyTorch : predict single example

body:
Following the example from:

https://github.com/jcjohnson/pytorch-examples

This code trains successfully:

# Code in file tensor/two_layer_net_tensor.py
import torch

device = torch.device('cpu')
# device = torch.device('cuda') # Uncomment this to run on GPU

# N is batch size; D_in is input dimension;
# H is hidden dimension; D_out is output dimension.
N, D_in, H, D_out = 64, 1000, 100, 10

# Create random input and output data
x = torch.randn(N, D_in, device=device)
y = torch.randn(N, D_out, device=device)

# Randomly initialize weights
w1 = torch.randn(D_in, H, device=device)
w2 = torch.randn(H, D_out, device=device)

learning_rate = 1e-6
for t in range(500):
  # Forward pass: compute predicted y
  h = x.mm(w1)
  h_relu = h.clamp(min=0)
  y_pred = h_relu.mm(w2)

  # Compute and print loss; loss is a scalar, and is stored in a PyTorch Tensor
  # of shape (); we can get its value as a Python number with loss.item().
  loss = (y_pred - y).pow(2).sum()
  print(t, loss.item())

  # Backprop to compute gradients of w1 and w2 with respect to loss
  grad_y_pred = 2.0 * (y_pred - y)
  grad_w2 = h_relu.t().mm(grad_y_pred)
  grad_h_relu = grad_y_pred.mm(w2.t())
  grad_h = grad_h_relu.clone()
  grad_h[h < 0] = 0
  grad_w1 = x.t().mm(grad_h)

  # Update weights using gradient descent
  w1 -= learning_rate * grad_w1
  w2 -= learning_rate * grad_w2

How can I predict a single example ? My experience thus far is utilising feedforward networks using just numpy. After training a model I utilise forward propagation but for a single example :

numpy code snippet where new is the output value I'm attempting to predict:

new = np.asarray(toclassify)
Z1 = np.dot(weight_layer_1, new.T) + bias_1
sigmoid_activation_1 = sigmoid(Z1)
Z2 = np.dot(weight_layer_2, sigmoid_activation_1) + bias_2
sigmoid_activation_2 = sigmoid(Z2)

sigmoid_activation_2 contains the predicted vector attributes

Is the idiomatic PyTorch way same?  Use forward propagation in order to make a single prediction?


title: PyTorch memory model: "torch.from_numpy()" vs "torch.Tensor()"

body:
I'm trying to have an in-depth understanding of how PyTorch Tensor memory model works.

# input numpy array
In [91]: arr = np.arange(10, dtype=float32).reshape(5, 2)

# input tensors in two different ways
In [92]: t1, t2 = torch.Tensor(arr), torch.from_numpy(arr)

# their types
In [93]: type(arr), type(t1), type(t2)
Out[93]: (numpy.ndarray, torch.FloatTensor, torch.FloatTensor)

# ndarray
In [94]: arr
Out[94]:
array([[ 0.,  1.],
       [ 2.,  3.],
       [ 4.,  5.],
       [ 6.,  7.],
       [ 8.,  9.]], dtype=float32)

I know that PyTorch tensors share the memory buffer of NumPy ndarrays. Thus, changing one will be reflected in the other. So, here I'm slicing and updating some values in the Tensor t2

In [98]: t2[:, 1] = 23.0

And as expected, it's updated in t2 and arr since they share the same memory buffer.

In [99]: t2
Out[99]:

  0  23
  2  23
  4  23
  6  23
  8  23
[torch.FloatTensor of size 5x2]

In [101]: arr
Out[101]:
array([[  0.,  23.],
       [  2.,  23.],
       [  4.,  23.],
       [  6.,  23.],
       [  8.,  23.]], dtype=float32)

But, t1 is also updated. Remember that t1 was constructed using torch.Tensor() whereas t2 was constructed using torch.from_numpy()

In [100]: t1
Out[100]:

  0  23
  2  23
  4  23
  6  23
  8  23
[torch.FloatTensor of size 5x2]

So, no matter whether we use torch.from_numpy() or torch.Tensor() to construct a tensor from an ndarray, all such tensors and ndarrays share the same memory buffer.

Based on this understanding, my question is why does a dedicated function torch.from_numpy() exists when simply torch.Tensor() can do the job?

I looked at the PyTorch documentation but it doesn't mention anything about this? Any ideas/suggestions?


Trying to train a pytorch model(begginner)
Im using a unet model to which i am feeding in a image as input and along with that i am feeding the label as the input image mask and traning the dataset on it.
The unet model i have picked up from somewhere else, and i am using the cross-entropy loss as a loss function but i get this dimension out of range error,

    RuntimeError
Traceback (most recent call last)
<ipython-input-358-fa0ef49a43ae> in <module>()
     16 for epoch in range(0, num_epochs):
     17     # train for one epoch
---> 18     curr_loss = train(train_loader, model, criterion, epoch, num_epochs)
     19
     20     # store best loss and save a model checkpoint

<ipython-input-356-1bd6c6c281fb> in train(train_loader, model, criterion, epoch, num_epochs)
     16         # measure loss
     17         print (outputs.size(),labels.size())
---> 18         loss = criterion(outputs, labels)
     19         losses.update(loss.data[0], images.size(0))
     20

/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py in     _ _call__(self, *input, **kwargs)
    323         for hook in self._forward_pre_hooks.values():
    324             hook(self, input)
--> 325         result = self.forward(*input, **kwargs)
    326         for hook in self._forward_hooks.values():
    327             hook_result = hook(self, input, result)

<ipython-input-355-db66abcdb074> in forward(self, logits, targets)
      9         probs_flat = probs.view(-1)
     10         targets_flat = targets.view(-1)
---> 11         return self.crossEntropy_loss(probs_flat, targets_flat)

/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py in     __call__(self, *input, **kwargs)
    323         for hook in self._forward_pre_hooks.values():
    324             hook(self, input)
  --> 325         result = self.forward(*input, **kwargs)
    326         for hook in self._forward_hooks.values():
    327             hook_result = hook(self, input, result)

/usr/local/lib/python3.5/dist-packages/torch/nn/modules/loss.py in f orward(self, input, target)
    599         _assert_no_grad(target)
    600         return F.cross_entropy(input, target, self.weight, self.size_average,
--> 601                                self.ignore_index, self.reduce)
    602
    603

/usr/local/lib/python3.5/dist-packages/torch/nn/functional.py in     cross_entropy(input, target, weight, size_average, ignore_index, reduce)
   1138         >>> loss.backward()
   1139     """
-> 1140     return nll_loss(log_softmax(input, 1), target, weight, size_average, ignore_index, reduce)
   1141
   1142

/usr/local/lib/python3.5/dist-packages/torch/nn/functional.py in     log_softmax(input, dim, _stacklevel)
    784     if dim is None:
    785         dim = _get_softmax_dim('log_softmax', input.dim(),      _stacklevel)
--> 786     return torch._C._nn.log_softmax(input, dim)
    787
    788

RuntimeError: dimension out of range (expected to be in range of [-1, 0], but got 1)

Part of my code looks like this

class crossEntropy(nn.Module):
    def __init__(self, weight = None, size_average = True):
        super(crossEntropy, self).__init__()
        self.crossEntropy_loss = nn.CrossEntropyLoss(weight, size_average)

    def forward(self, logits, targets):
        probs = F.sigmoid(logits)
        probs_flat = probs.view(-1)
        targets_flat = targets.view(-1)
        return self.crossEntropy_loss(probs_flat, targets_flat)

class UNet(nn.Module):
    def __init__(self, imsize):
        super(UNet, self).__init__()
        self.imsize = imsize

        self.activation = F.relu

        self.pool1 = nn.MaxPool2d(2)
        self.pool2 = nn.MaxPool2d(2)
        self.pool3 = nn.MaxPool2d(2)
        self.pool4 = nn.MaxPool2d(2)
        self.conv_block1_64 = UNetConvBlock(4, 64)
        self.conv_block64_128 = UNetConvBlock(64, 128)
        self.conv_block128_256 = UNetConvBlock(128, 256)
        self.conv_block256_512 = UNetConvBlock(256, 512)
        self.conv_block512_1024 = UNetConvBlock(512, 1024)

        self.up_block1024_512 = UNetUpBlock(1024, 512)
        self.up_block512_256 = UNetUpBlock(512, 256)
        self.up_block256_128 = UNetUpBlock(256, 128)
        self.up_block128_64 = UNetUpBlock(128, 64)

        self.last = nn.Conv2d(64, 2, 1)

    def forward(self, x):
        block1 = self.conv_block1_64(x)
        pool1 = self.pool1(block1)

        block2 = self.conv_block64_128(pool1)
        pool2 = self.pool2(block2)

        block3 = self.conv_block128_256(pool2)
        pool3 = self.pool3(block3)

        block4 = self.conv_block256_512(pool3)
        pool4 = self.pool4(block4)

        block5 = self.conv_block512_1024(pool4)

        up1 = self.up_block1024_512(block5, block4)

        up2 = self.up_block512_256(up1, block3)

        up3 = self.up_block256_128(up2, block2)

        up4 = self.up_block128_64(up3, block1)

        return F.log_softmax(self.last(up4))

Any suggestion,tips would be really helpful

Thanks in advance.
Do let me know if you need more code,


title: Pytorch RuntimeError: The size of tensor a (4) must match the size of tensor b (3) at non-singleton dimension 0

body:
I use code from here to train a model to predict printed style number from 0 to 9:

idx_to_class = {0: "0", 1: "1", 2: "2", 3: "3", 4: "4", 5: "5", 6: "6", 7:"7", 8: "8", 9:"9"}
def predict(model, test_image_name):

    transform = image_transforms['test']

    test_image = Image.open(test_image_name)
    plt.imshow(test_image)

    test_image_tensor = transform(test_image)

    if torch.cuda.is_available():
        test_image_tensor = test_image_tensor.view(1, 3, 224, 224).cuda()
    else:
        test_image_tensor = test_image_tensor.view(1, 3, 224, 224)

    with torch.no_grad():
        model.eval()
        # Model outputs log probabilities
        out = model(test_image_tensor)
        ps = torch.exp(out)
        topk, topclass = ps.topk(1, dim=1)
        # print(topclass.cpu().numpy()[0][0])
        print("Image class:  ", idx_to_class[topclass.cpu().numpy()[0][0]])

predict(model, "path_of_test_image")

But I get an error when try to use predict:

Traceback (most recent call last):

  File "<ipython-input-12-f8636d3ba083>", line 26, in <module>
    predict(model, "/home/x/文档/Deep_Learning/pytorch/MNIST/test/2/QQ截图20191022093955.png")

  File "<ipython-input-12-f8636d3ba083>", line 9, in predict
    test_image_tensor = transform(test_image)

  File "/home/x/.local/lib/python3.6/site-packages/torchvision/transforms/transforms.py", line 61, in __call__
    img = t(img)

  File "/home/x/.local/lib/python3.6/site-packages/torchvision/transforms/transforms.py", line 166, in __call__
    return F.normalize(tensor, self.mean, self.std, self.inplace)

  File "/home/x/.local/lib/python3.6/site-packages/torchvision/transforms/functional.py", line 217, in normalize
    tensor.sub_(mean[:, None, None]).div_(std[:, None, None])

RuntimeError: The size of tensor a (4) must match the size of tensor b (3) at non-singleton dimension 0

How could I fix it? Thanks.


title: How to tell PyTorch to not use the GPU?

body: I want to do some timing comparisons between CPU & GPU as well as some profiling and would like to know if there's a way to tell pytorch to not use the GPU and instead use the CPU only? I realize I could install another CPU-only pytorch, but hoping there's an easier way.


title: PyTorch: How to change the learning rate of an optimizer at any given moment (no LR schedule)

body:
Is it possible in PyTorch to change the learning rate of the optimizer in the middle of training dynamically (I don't want to define a learning rate schedule beforehand)?

So let's say I have an optimizer:

optim = torch.optim.SGD(model.parameters(), lr=0.01)

Now due to some tests which I perform during training, I realize my learning rate is too high so I want to change it to say 0.001. There doesn't seem to be a method optim.set_lr(0.001) but is there some way to do this?


title: PyTorch / Gensim - How to load pre-trained word embeddings

body:
I want to load a pre-trained word2vec embedding with gensim into a PyTorch embedding layer.

So my question is, how do I get the embedding weights loaded by gensim into the PyTorch embedding layer.

Thanks in Advance!


title: How to cast a 1-d IntTensor to int in Pytorch

body:
I get a 1-D IntTensor,but i want to convert it to a integer.
I try it by this method:

print(dictionary[IntTensor.int()])

but got an error:

KeyError: Variable containing:
 423
[torch.IntTensor of size 1]

Thanks~


title: Pytorch: Can't call numpy() on Variable that requires grad. Use var.detach().numpy() instead

body:
I have an error in my code which is not getting fixed any which way I try.

The Error is simple, I return a value:

torch.exp(-LL_total/T_total)

and get the error later in the pipeline:

RuntimeError: Can't call numpy() on Variable that requires grad. Use var.detach().numpy() instead.

Solutions such as cpu().detach().numpy() give the same error.

How could I fix it? Thanks.


title: How to use PyTorch multiprocessing?

body:
I'm trying to use python's multiprocessing Pool method in pytorch to process a image. Here's the code:

from multiprocessing import Process, Pool
from torch.autograd import Variable
import numpy as np
from scipy.ndimage import zoom

def get_pred(args):

  img = args[0]
  scale = args[1]
  scales = args[2]
  img_scale = zoom(img.numpy(),
                     (1., 1., scale, scale),
                     order=1,
                     prefilter=False,
                     mode='nearest')

  # feed input data
  input_img = Variable(torch.from_numpy(img_scale),
                     volatile=True).cuda()
  return input_img

scales = [1,2,3,4,5]
scale_list = []
for scale in scales:
    scale_list.append([img,scale,scales])
multi_pool = Pool(processes=5)
predictions = multi_pool.map(get_pred,scale_list)
multi_pool.close()
multi_pool.join()

I'm getting this error:

`RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method

`
In this line:

predictions = multi_pool.map(get_pred,scale_list)

Can anyone tell me what I'm doing wrong ?


title: "RuntimeError: Expected 4-dimensional input for 4-dimensional weight 32 3 3, but got 3-dimensional input of size [3, 224, 224] instead"?

body:
I am trying to use a pre-trained model. Here's where the problem occurs

Isn't the model supposed to take in a simple colored image? Why is it expecting a 4-dimensional input?

RuntimeError                              Traceback (most recent call last)
<ipython-input-51-d7abe3ef1355> in <module>()
     33
     34 # Forward pass the data through the model
---> 35 output = model(data)
     36 init_pred = output.max(1, keepdim=True)[1] # get the index of the max log-probability
     37

5 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py in forward(self, input)
    336                             _pair(0), self.dilation, self.groups)
    337         return F.conv2d(input, self.weight, self.bias, self.stride,
--> 338                         self.padding, self.dilation, self.groups)
    339
    340

RuntimeError: Expected 4-dimensional input for 4-dimensional weight 32 3 3, but got 3-dimensional input of size [3, 224, 224] instead

Where

inception = models.inception_v3()
model = inception.to(device)


title: How to Display Custom Images in Tensorboard (e.g. Matplotlib Plots)?

body:
The Image Dashboard section of the Tensorboard ReadMe says:

  Since the image dashboard supports arbitrary pngs, you can use this to embed custom visualizations (e.g. matplotlib scatterplots) into TensorBoard.

I see how a pyplot image could be written to file, read back in as a tensor, and then used with tf.image_summary() to write it to TensorBoard, but this statement from the readme suggests there is a more direct way. Is there? If so, is there any further documentation and/or examples of how to do this efficiently?


My input tensor is torch.DoubleTensor type. But I got the RuntimeError below:

RuntimeError: Expected object of type torch.DoubleTensor but found type torch.FloatTensor for argument #2 'weight'

I didn't specify the type of the weight explicitly(i.e. I did not init my weight by myself. The weight is created by pytorch). What will influence the type of weight in the forward process?

Thanks a lot!!


title: How to convert a pytorch tensor into a numpy array?

body:
I have a torch tensor

a = torch.randn(1, 2, 3, 4, 5)

How can I get it in numpy?

Something like

b = a.tonumpy()

output should be the same as if I did

b = np.random.randn(1, 2, 3, 4, 5)


title: What does the parameter retain_graph mean in the Variable's backward() method?

body:
I'm going through the neural transfer pytorch tutorial and am confused about the use of retain_variable(deprecated, now referred to as retain_graph). The code example show:

class ContentLoss(nn.Module):

    def __init__(self, target, weight):
        super(ContentLoss, self).__init__()
        self.target = target.detach() * weight
        self.weight = weight
        self.criterion = nn.MSELoss()

    def forward(self, input):
        self.loss = self.criterion(input * self.weight, self.target)
        self.output = input
        return self.output

    def backward(self, retain_variables=True):
        #Why is retain_variables True??
        self.loss.backward(retain_variables=retain_variables)
        return self.loss

From the documentation

  retain_graph (bool, optional) – If False, the graph used to compute
  the grad will be freed. Note that in nearly all cases setting this
  option to True is not needed and often can be worked around in a much
  more efficient way. Defaults to the value of create_graph.

So by setting retain_graph= True, we're not freeing the memory allocated for the graph on the backward pass. What is the advantage of keeping this memory around, why do we need it?


title: converting list of tensors to tensors pytorch

body:
I have list of tensor each tensor has different size how can I convert this list of tensors into a tensor using pytroch

for more info my list contains tensors each tensor have different size
for example the first tensor size is torch.Size([76080, 38])

the shape of the other tensors would differ in the second element for example the second tensor in the list is torch.Size([76080, 36])

when I use
        torch.tensor(x)
I get an error
      ValueError: only one element tensors can be converted to Python scalars


title: What is the relationship between PyTorch and Torch?

body:
There are two PyTorch repositories :

https://github.com/hughperkins/pytorch
https://github.com/pytorch/pytorch

The first clearly requires Torch and lua and is a wrapper, but the second doesn't make any reference to the Torch project except with its name.
How is it related to the Lua Torch?


title: Difference between "detach()" and "with torch.nograd()" in PyTorch?

body:
I know about two ways to exclude elements of a computation from the gradient calculation backward

Method 1: using with torch.no_grad()

with torch.no_grad():
    y = reward + gamma * torch.max(net.forward(x))
loss = criterion(net.forward(torch.from_numpy(o)), y)
loss.backward();

Method 2: using .detach()

y = reward + gamma * torch.max(net.forward(x))
loss = criterion(net.forward(torch.from_numpy(o)), y.detach())
loss.backward();

Is there a difference between these two? Are there benefits/downsides to either?


title: Pytorch: Convert FloatTensor into DoubleTensor

body:
I have 2 numpy arrays, which I convert into tensors to use the TensorDataset object.

import torch.utils.data as data_utils

X = np.zeros((100,30))
Y = np.zeros((100,30))

train = data_utils.TensorDataset(torch.from_numpy(X).double(), torch.from_numpy(Y))
train_loader = data_utils.DataLoader(train, batch_size=50, shuffle=True)

when I do:

for batch_idx, (data, target) in enumerate(train_loader):
    data, target = Variable(data), Variable(target)
    optimizer.zero_grad()
    output = model(data)               # error occurs here

I get the fallowing error:

  TypeError: addmm_ received an invalid combination of arguments - got (int, int, torch.DoubleTensor, torch.FloatTensor), but expected one of:
   [...]
   * (float beta, float alpha, torch.DoubleTensor mat1, torch.DoubleTensor mat2)
        didn't match because some of the arguments have invalid types: (int, int, torch.DoubleTensor, torch.FloatTensor)
   * (float beta, float alpha, torch.SparseDoubleTensor mat1, torch.DoubleTensor mat2)
        didn't match because some of the arguments have invalid types: (int, int, torch.DoubleTensor, torch.FloatTensor)

The last error comes from:

  output.addmm_(0, 1, input, weight.t())

As you see in my code I tried converting the tensor by using .double() - but this did not work. Why is he casting one array into a FloatTensor object and the other into a DoubleTensor?
Any ideas?


title: Understanding a simple LSTM pytorch

body:
import torch,ipdb
import torch.autograd as autograd
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.autograd import Variable

rnn = nn.LSTM(input_size=10, hidden_size=20, num_layers=2)
input = Variable(torch.randn(5, 3, 10))
h0 = Variable(torch.randn(2, 3, 20))
c0 = Variable(torch.randn(2, 3, 20))
output, hn = rnn(input, (h0, c0))

This is the LSTM example from the docs. I don't know understand the following things:

What is output-size and why is it not specified anywhere?
Why does the input have 3 dimensions. What does 5 and 3 represent?
What are 2 and 3 in h0 and c0, what do those represent?

Edit:

import torch,ipdb
import torch.autograd as autograd
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.autograd import Variable
import torch.nn.functional as F

num_layers=3
num_hyperparams=4
batch = 1
hidden_size = 20
rnn = nn.LSTM(input_size=num_hyperparams, hidden_size=hidden_size, num_layers=num_layers)

input = Variable(torch.randn(1, batch, num_hyperparams)) # (seq_len, batch, input_size)
h0 = Variable(torch.randn(num_layers, batch, hidden_size)) # (num_layers, batch, hidden_size)
c0 = Variable(torch.randn(num_layers, batch, hidden_size))
output, hn = rnn(input, (h0, c0))
affine1 = nn.Linear(hidden_size, num_hyperparams)

ipdb.set_trace()
print output.size()
print h0.size()

  *** RuntimeError: matrices expected, got 3D, 2D tensors at


title: What are transforms in PyTorch used for?

body:
I am new with Pytorch and not very expert in CNN.
I have done a successful classifier with the tutorial that they provide Tutorial Pytorch, but I don't really understand what I am doing when loading the data.

They do some data augmentation and normalisation for training, but when I try to modify the parameters, the code does not work.

# Data augmentation and normalization for training
# Just normalization for validation
data_transforms = {
    'train': transforms.Compose([
        transforms.RandomResizedCrop(224),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
    'val': transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
}

Am I extending my training dataset? I don't see the data augmentation.

Why if I modify the value of transforms.RandomResizedCrop(224) the data loading stop working?

Do I need to transform as well the test dataset?

I am a bit confused with this data transformation that they do.


title: PyTorch: Testing with torchvision.datasets.ImageFolder and DataLoader

body:
I'm a newbie trying to make this PyTorch CNN work with the Cats&Dogs dataset from kaggle. As there are no targets for the test images, I manually classified some of the test images and put the class in the filename, to be able to test (maybe should have just used some of the train images).

I used the torchvision.datasets.ImageFolder class to load the train and test images. The training seems to work.

But what do I need to do to make the test-routine work? I don't know, how to connect my test_data_loader with the test loop at the bottom, via test_x and test_y.

The Code is based on this MNIST example CNN. There, something like this is used right after the loaders are created. But I failed to rewrite it for my dataset:

test_x = Variable(torch.unsqueeze(test_data.test_data, dim=1), volatile=True).type(torch.FloatTensor)[:2000]/255.   # shape from (2000, 28, 28) to (2000, 1, 28, 28), value in range(0,1)
test_y = test_data.test_labels[:2000]

The Code:

import os
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.autograd import Variable
import torch.utils.data as data
import torchvision
from torchvision import transforms

EPOCHS = 2
BATCH_SIZE = 10
LEARNING_RATE = 0.003
TRAIN_DATA_PATH = "./train_cl/"
TEST_DATA_PATH = "./test_named_cl/"
TRANSFORM_IMG = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(256),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406],
                         std=[0.229, 0.224, 0.225] )
    ])

train_data = torchvision.datasets.ImageFolder(root=TRAIN_DATA_PATH, transform=TRANSFORM_IMG)
train_data_loader = data.DataLoader(train_data, batch_size=BATCH_SIZE, shuffle=True,  num_workers=4)
test_data = torchvision.datasets.ImageFolder(root=TEST_DATA_PATH, transform=TRANSFORM_IMG)
test_data_loader  = data.DataLoader(test_data, batch_size=BATCH_SIZE, shuffle=True, num_workers=4)

class CNN(nn.Module):
    # omitted...

if __name__ == '__main__':

    print("Number of train samples: ", len(train_data))
    print("Number of test samples: ", len(test_data))
    print("Detected Classes are: ", train_data.class_to_idx) # classes are detected by folder structure

    model = CNN()
    optimizer = torch.optim.Adam(model.parameters(), lr=LEARNING_RATE)
    loss_func = nn.CrossEntropyLoss()

    # Training and Testing
    for epoch in range(EPOCHS):
        for step, (x, y) in enumerate(train_data_loader):
            b_x = Variable(x)   # batch x (image)
            b_y = Variable(y)   # batch y (target)
            output = model(b_x)[0]
            loss = loss_func(output, b_y)
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

            # Test -> this is where I have no clue
            if step % 50 == 0:
                test_x = Variable(test_data_loader)
                test_output, last_layer = model(test_x)
                pred_y = torch.max(test_output, 1)[1].data.squeeze()
                accuracy = sum(pred_y == test_y) / float(test_y.size(0))
                print('Epoch: ', epoch, '| train loss: %.4f' % loss.data[0], '| test accuracy: %.2f' % accuracy)


title: Pytorch: how to convert data into tensor

body:
I am a beginner for Pytorch.
I was trying to write CNN code referring Pytorch tutorial.
Below is a part of the code, but it shows error "RuntimeError: Variable data has to be a tensor, but got list". I tried to cast input data to tensor but didn't work well. If anybody know the solution, please help me out...

    def read_labels(file):
      dic = {}
      with open(file) as f:
        reader = f
        for row in reader:
            dic[row.split(",")[0]]  = row.split(",")[1].rstrip() #rstrip(): eliminate "\n"
      return dic

    image_names= os.listdir("./train_mini")
    label_dic = read_labels("labels.csv")

    names =[]
    labels = []
    images =[]

    for name in image_names[1:]:
        images.append(cv2.imread("./train_mini/"+name))
        labels.append(label_dic[os.path.splitext(name)[0]])

    """
    Data distribution
    """
    N = len(images)
    N_train = int(N * 0.7)
    N_test = int(N*0.2)

    X_train, X_tmp, Y_train, Y_tmp = train_test_split(images, labels, train_size=N_train)
    X_validation, X_test, Y_validation, Y_test = train_test_split(X_tmp, Y_tmp, test_size=N_test)

    """
    Model Definition
    """

    class CNN(nn.Module):
        def __init__(self):
            super(CNN, self).__init__()
            self.head = nn.Sequential(
                nn.Conv2d(in_channels=1, out_channels=10,
                          kernel_size=5, stride=1),
                nn.MaxPool2d(kernel_size=2),
                nn.ReLU(),
                nn.Conv2d(10, 20, kernel_size=5),
                nn.MaxPool2d(kernel_size=2),
                nn.ReLU())
            self.tail = nn.Sequential(
                nn.Linear(320, 50),
                nn.ReLU(),
                nn.Linear(50, 10))

        def forward(self, x):
            x = self.head(x)
            x = x.view(-1, 320)
            x = self.tail(x)
            return F.log_softmax(x)

    CNN = CNN()
    criterion = nn.CrossEntropyLoss()
    optimizer = optim.SGD(CNN.parameters(), lr=0.001, momentum=0.9)

    """
    Training
    """
    batch_size = 50
    for epoch in range(2):  # loop over the dataset multiple times
        running_loss = 0.0
        for i in range(N / batch_size):
        #for i, data in enumerate(trainloader, 0):
            batch = batch_size * i

            # get the inputs
            images_batch = X_train[batch:batch + batch_size]
            labels_batch = Y_train[batch:batch + batch_size]

            # wrap them in Variable
            images_batch, labels_batch = Variable(images_batch), Variable(labels_batch)

            # zero the parameter gradients
            optimizer.zero_grad()

            # forward + backward + optimize
            outputs = CNN(images_batch)
            loss = criterion(outputs, labels_batch)
            loss.backward()
            optimizer.step()

            # print statistics
            running_loss += loss.data[0]
            if i % 2000 == 1999:    # print every 2000 mini-batches
                print('[%d, %5d] loss: %.3f' %
                      (epoch + 1, i + 1, running_loss / 2000))
                running_loss = 0.0

    print('Finished Training')

And error is happening here

# wrap them in Variable
images_batch, labels_batch = Variable(images_batch), Variable(labels_batch)


title: TypeError: can't convert np.ndarray of type numpy.object_

body:
How to convert a numpy ndarry to torch tensor?

Here is my data:

array([array([-0.4287  , -1.193   , -2.156   , -0.2264  , -1.978   , -1.101   ,   -3.395   ,  0.2974  ], dtype=float16),
   array([-0.3386 ,  1.398  , -1.083  ,  0.2961 , -0.7354 , -1.326  , -4.33   ,  0.6284 ], dtype=float16)],
   dtype=object)


I got the following error when I ran my pytorch deep learning model in colab

/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in linear(input, weight, bias)
   1370         ret = torch.addmm(bias, input, weight.t())
   1371     else:
-> 1372         output = input.matmul(weight.t())
   1373         if bias is not None:
   1374             output += bias

RuntimeError: CUDA error: CUBLAS_STATUS_ALLOC_FAILED when calling `cublasCreate(handle)`

I even reduced batch size from 128 to 64 i.e., reduced to half,  but still, I got this error. Earlier, I ran the same code with a batch size of 128 but didn't get any error like this.


title: Taking subsets of a pytorch dataset

body:
I have a network which I want to train on some dataset (as an example, say CIFAR10). I can create data loader object via

trainset = torchvision.datasets.CIFAR10(root='./data', train=True,
                                        download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,
                                          shuffle=True, num_workers=2)

My question is as follows: Suppose I want to make several different training iterations. Let's say I want at first to train the network on all images in odd positions, then on all images in even positions and so on. In order to do that, I need to be able to access to those images. Unfortunately, it seems that trainset does not allow such access. That is, trying to do trainset[:1000] or more generally trainset[mask] will throw an error.

I could do instead

trainset.train_data=trainset.train_data[mask]
trainset.train_labels=trainset.train_labels[mask]

and then

trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,
                                              shuffle=True, num_workers=2)

However, that will force me to create a new copy of the full dataset in each iteration (as I already changed trainset.train_data so I will need to redefine trainset). Is there some way to avoid it?

Ideally, I would like to have something "equivalent" to

trainloader = torch.utils.data.DataLoader(trainset[mask], batch_size=4,
                                              shuffle=True, num_workers=2)


title: Why `torch.cuda.is_available()` returns False even after installing pytorch with cuda?

body:
On a Windows 10 PC with an NVidia GeForce 820M
I installed CUDA 9.2 and cudnn 7.1 successfully,
and then installed PyTorch using the instructions at pytorch.org.

Specifically I used the command

pip install torch==1.4.0+cu92 torchvision==0.5.0+cu92 -f https://download.pytorch.org/whl/torch_stable.html

because i use pip and not Anaconda.

Yet I get the following

>>> import torch
>>> torch.cuda.is_available()
False

Why is this happening?


I can't get the dtypes to match, either the loss wants long or the model wants float if I change my tensors to long. The shape of the tensors are 42000, 1, 28, 28 and 42000. I'm not sure where I can change what dtypes are required for the model or loss.

I'm not sure if dataloader is required, using Variable didn't work either.

dataloaders_train = torch.utils.data.DataLoader(Xt_train, batch_size=64)

dataloaders_test = torch.utils.data.DataLoader(Yt_train, batch_size=64)

class Network(nn.Module):
    def __init__(self):
        super().__init__()

        self.hidden = nn.Linear(42000, 256)

        self.output = nn.Linear(256, 10)

        self.sigmoid = nn.Sigmoid()
        self.softmax = nn.Softmax(dim=1)

    def forward(self, x):

        x = self.hidden(x)
        x = self.sigmoid(x)
        x = self.output(x)
        x = self.softmax(x)

        return x

model = Network()

input_size = 784
hidden_sizes = [28, 64]
output_size = 10
model = nn.Sequential(nn.Linear(input_size, hidden_sizes[0]),
                      nn.ReLU(),
                      nn.Linear(hidden_sizes[0], hidden_sizes[1]),
                      nn.ReLU(),
                      nn.Linear(hidden_sizes[1], output_size),
                      nn.Softmax(dim=1))
print(model)

criterion = nn.NLLLoss()
optimizer = optim.SGD(model.parameters(), lr=0.003)

epochs = 5

for e in range(epochs):
    running_loss = 0
    for images, labels in zip(dataloaders_train, dataloaders_test):

        images = images.view(images.shape[0], -1)
        #images, labels = Variable(images), Variable(labels)
        print(images.dtype)
        print(labels.dtype)

        optimizer.zero_grad()

        output = model(images)
        loss = criterion(output, labels)
        loss.backward()
        optimizer.step()

        running_loss += loss.item()
    else:
        print(f"Training loss: {running_loss}")

Which gives

RuntimeError                              Traceback (most recent call last)
<ipython-input-128-68109c274f8f> in <module>
     11
     12         output = model(images)
---> 13         loss = criterion(output, labels)
     14         loss.backward()
     15         optimizer.step()

/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--> 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

/opt/conda/lib/python3.6/site-packages/torch/nn/modules/loss.py in forward(self, input, target)
    202
    203     def forward(self, input, target):
--> 204         return F.nll_loss(input, target, weight=self.weight, ignore_index=self.ignore_index, reduction=self.reduction)
    205
    206

/opt/conda/lib/python3.6/site-packages/torch/nn/functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)
   1836                          .format(input.size(0), target.size(0)))
   1837     if dim == 2:
-> 1838         ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
   1839     elif dim == 4:
   1840         ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)

RuntimeError: expected scalar type Long but found Float