title: Efficiently find centroid of labelled image regions

body:
I have a segmented image as a 2 dimensional matrix of unique labels 1 ... k. For example:

img =
    [1 1 2 2 2 2 2 3 3]
    [1 1 1 2 2 2 2 3 3]
    [1 1 2 2 2 2 3 3 3]
    [1 4 4 4 2 2 2 2 3]
    [4 4 4 5 5 5 2 3 3]
    [4 4 4 5 5 6 6 6 6]
    [4 4 5 5 5 5 6 6 6]

I am trying to determine the region centroids. That is, per label, what is the X,Y coordinate of the center of mass? For example, the centroid of label 1 is (1.25, 0.625). Just average up the row numbers ((0 + 0 + 1 + 1 + 1 + 2 + 2 + 3) / 8 = 1.25) and the column numbers ((0 + 0 + 0 + 0 + 1 + 1 + 1 + 2) / 8 = 0.625)

The only way I know how to do this is to use a for loop from 1 to k (or in the case of my example, 1 through 6), find the indices of the points for each label, and average their coordinates by indexing a meshgrid of the image.

However, I am looking to do this in a way optimized for GPU computations. Hence, the use of a for loop is less than ideal (takes about 1 sec per image on a nice GPU for a few hundred labels). I am using PyTorch, but really any numpy solution should suffice.

Is there a GPU-efficient solution for this task?


title: How to index/slice the last dimension of a PyTorch tensor/numpy array of unknown dimensions

body:
For example, if I have a 2D tensor X, I can do slicing X[:,1:]; if I have a 3D tensor Y, then I can do similar slicing for the last dimension like Y[:,:,1:].

What is the right way to do the slicing when given a tensor Z of unknown dimension?
How about a numpy array?

Thanks!


title: Pytorch is Throwing out of bounds Error? Expects a scalar

body:
This is the code below, Not sure what the error being thrown is. Please can someone explain what is wrong and the fix. I'm new to pytorch and decided to try learn it using the house prices data set but ran into this error.

It is apprently something to do with scalar value or somethings but not sure that the problem is because the y value given is a scalar not a vector.

import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.utils.data import TensorDataset, DataLoader
import torch.optim as optim
import pandas as pd
import numpy as np
df = pd.read_csv('housepricedata.csv')
dataset = df.values
X = dataset[:,0:10]
y = dataset[:, 10]
from sklearn import preprocessing
min_max = preprocessing.MinMaxScaler()
x_scale = min_max.fit_transform(X)
y_scale = min_max.fit_transform(y_scale)
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(x_scale, y, test_size=0.3)

X_train = torch.FloatTensor(X_train)
X_test = torch.FloatTensor(X_test)
y_train = torch.LongTensor(y_train)
y_test = torch.LongTensor(y_test)

trainD = TensorDataset(X_train, y_train)
testD = TensorDataset(X_test, y_test)
class Model(nn.Module):
    def __init__(self, inp1=10, out=1):
        super().__init__()
        self.Dense1 = nn.Linear(inp1, 32)
        self.Dense2 = nn.Linear(32, 32)
        self.out = nn.Linear(32, out)

    def forward(self, x):
        x = F.relu(self.Dense1(x))
        x = F.relu(self.Dense2(x))
        x = self.out(x)
        return x
model = Model()
trainloader = DataLoader(trainD, batch_size=64, shuffle=False)
testloader  = DataLoader(testloader, batch_size=64, shuffle=False)
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
epochs1 = 500
losses = []

epochs1 = 500
losses = []
for i in range(epochs1):
    for data in trainloader:
        X, y = data
        optimizer.zero_grad()
        output = model(X)
        loss = criterion(output, y)
        losses.append(loss)
        loss.backward()
        optimizer.step()

Error thrown:

IndexError                                Traceback (most recent call last)
 in
      5     i =+1
      6     y_pred = model.forward(X_train)
----> 7     loss = criterion(y_pred, y_train)
      8     losses.append(loss)
      9

~\Anaconda3\envs\ml1\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--> 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

~\Anaconda3\envs\ml1\lib\site-packages\torch\nn\modules\loss.py in forward(self, input, target)
    914     def forward(self, input, target):
    915         return F.cross_entropy(input, target, weight=self.weight,
--> 916                                ignore_index=self.ignore_index, reduction=self.reduction)
    917
    918

~\Anaconda3\envs\ml1\lib\site-packages\torch\nn\functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction)
   2019     if size_average is not None or reduce is not None:
   2020         reduction = _Reduction.legacy_get_string(size_average, reduce)
-> 2021     return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
   2022
   2023

~\Anaconda3\envs\ml1\lib\site-packages\torch\nn\functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)
   1836                          .format(input.size(0), target.size(0)))
   1837     if dim == 2:
-> 1838         ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
   1839     elif dim == 4:
   1840         ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)

IndexError: Target 1 is out of bounds.


title: difference between python2 and python3 in terms of multiprocessing?

body:
So I had a paragraph of code "train.py" that goes like

    do something
    print('log something...')
    do something else

and I used pytorch's multiprocessing toolbox

    import torch.multiprocessing as mp

to execute multithreads of "train.py", however, the print function works well under python3, but not python2. Why?


title: Stacking up of LSTM outputs in pytorch

body:
I was going through some tutorial about the sentiment analysis using lstm network.
The below code said that its stacks up the lstm output. I Don't know how it works.

lstm_out = lstm_out.contiguous().view(-1, self.hidden_dim)


title: Pytorch saving and loading a VGG16 with knowledge transfer

body:
I am saving a VGG16 with knowledge transfer by using the following statement:

torch.save(model.state_dict(), 'checkpoint.pth')

and reloading by using the following statement:

state_dict = torch.load('checkpoint.pth')
model.load_state_dict(state_dict)

That works as long as I reload the VGG16 model and give it the same settings as before with the following code:

model = models.vgg16(pretrained=True)
model.cuda()
for param in model.parameters(): param.requires_grad = False

class Network(nn.Module):
    def __init__(self, input_size, output_size, hidden_layers, drop_p=0.5):

#             input_size: integer, size of the input
#             output_size: integer, size of the output layer
#             hidden_layers: list of integers, the sizes of the hidden layers
#             drop_p: float between 0 and 1, dropout probability

        super().__init__()
        # Add the first layer, input to a hidden layer
        self.hidden_layers = nn.ModuleList([nn.Linear(input_size, hidden_layers[0])])

        # Add a variable number of more hidden layers
        layer_sizes = zip(hidden_layers[:-1], hidden_layers[1:])
        self.hidden_layers.extend([nn.Linear(h1, h2) for h1, h2 in layer_sizes])
        self.output = nn.Linear(hidden_layers[-1], output_size)
        self.dropout = nn.Dropout(p=drop_p)

    def forward(self, x):
        ''' Forward pass through the network, returns the output logits '''

        # Forward through each layer in `hidden_layers`, with ReLU activation and dropout
        for linear in self.hidden_layers:
            x = F.relu(linear(x))
            x = self.dropout(x)

        x = self.output(x)
        return F.log_softmax(x, dim=1)

classifier = Network(25088, 102, [4096], drop_p=0.5)
model.classifier = classifier

How to avoid this?
How can I reload the model without having to reload a VGG16 and redefine a classifier?


title: Pytorch select tensor

body: I want to know if Pytorch have a slice function (same as tf). In particular, I want to select the orange color rows.


title: Problems installing and importing pytorch

body:
It's taken me hours trying to install pytorch using an anaconda environment. It appears that I've finally managed it using conda install filename.tar.bz2:

(pytorch) C:\Users\conner\aipND\img_classifier_project>conda list
# packages in environment at C:\Users\conner\Anaconda3\envs\pytorch:
#
blas                      1.1                    openblas    conda-forge
certifi                   2018.4.16                py35_0    conda-forge
cffi                      1.11.5                   py35_0    conda-forge
cmake                     3.11.1                        0    conda-forge
libflang                  5.0.0             vc14_20180208  [vc14]  conda-forge
llvm-meta                 5.0.0                         0    conda-forge
mkl                       2017.0.3                      0
numpy                     1.14.3          py35_blas_openblas_200  [blas_openblas]  conda-forge
openblas                  0.2.20                   vc14_7  [vc14]  conda-forge
openmp                    5.0.0                    vc14_1  [vc14]  conda-forge
pip                       9.0.3                    py35_0    conda-forge
pycparser                 2.18                     py35_0    conda-forge
python                    3.5.1                         1    conda-forge
pytorch                   0.3.1           py35_cuda90_cudnn7he774522_2  [cuda90]  file:///C:/Users/User 1/Downloads
pyyaml                    3.12                     py35_1    conda-forge
setuptools                39.1.0                   py35_0    conda-forge
typing                    3.6.4                    py35_0    conda-forge
vc                        14                            0    conda-forge
vs2015_runtime            14.0.25420                    0    conda-forge
wheel                     0.31.0                   py35_0    conda-forge
wincertstore              0.2                      py35_0    conda-forge
yaml                      0.1.7                    vc14_0  [vc14]  conda-forge

But when trying to import the package using import pytorch I get:

>>> import pytorch
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named 'pytorch'

And when I try import torch:

>>> import torch
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\conner\Anaconda3\envs\pytorch\lib\site-packages\torch\__init__.py", line 76, in <module>
    from torch._C import *
ImportError: DLL load failed: The specified module could not be found.

The difference in errors is actually encouraging, but I don't know how to interpret it. Why am I not able to import pytorch if it shows up on the list of packages in the environment?

EDIT

After continued attempts, I'll ask a much more general question: Has anyone had success at installing and importing Pytorch for Windows 10 on a 64-bit machine?


title: Using Pytorch's Multiprocessing along with Distributed

body:
I am trying to spawn a couple of process using pytorch's multiprocessing module within a openmpi distributed back-end. What I have is the following code:

def run(rank_local, rank, world_size, maingp):
    print("I WAS SPAWNED ", rank_local, " OF ", rank)

    tensor = torch.zeros(1)
    tensor += 1

    if rank == 0:
        tensor += 100
        dist.send(tensor, dst=1)
    else:
        print("I am spawn: ", rank, "and my tensor value before receive: ", tensor[0])
        dist.recv(tensor, src=0)
        print("I am spawn: ", rank, "and my tensor value after  receive: ", tensor[0])

if __name__ == '__main__':

    # Initialize Process Group
    dist.init_process_group(backend="mpi", group_name="main")
    maingp = None #torch.distributed.new_group([0,1])
    mp.set_start_method('spawn')

    # get current process information
    world_size = dist.get_world_size()
    rank = dist.get_rank()

    # Establish Local Rank and set device on this node
    mp.spawn(run, args=(rank, world_size, maingp), nprocs=1)

I run this code using the openmpi as follows:

mpirun -n 2 python code.py

So my understanding is that mpirun creates two process with ranks [0, 1], each of these process spawn new process with their local rank as 0. Now if I want to communicate between these two sub-processes of the main process I get some Traceback and following error:

-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/home/usama/anaconda3/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
    fn(i, *args)
  File "/home/usama/code/test/code.py", line 19, in run
    dist.send(tensor, dst=1)
  File "/home/usama/anaconda3/lib/python3.6/site-packages/torch/distributed/distributed_c10d.py", line 666, in send
    _check_default_pg()
  File "/home/usama/anaconda3/lib/python3.6/site-packages/torch/distributed/distributed_c10d.py", line 191, in _check_default_pg
    "Default process group is not initialized"
AssertionError: Default process group is not initialized

My question is how do I make these sub-processes to be able to communicate i.e the [0, 0] process sending something to [1, 0] process. Any ideas?


title: Pytorch Forward Pass Changes Each Time?

body:
I am learning pytorch and running a toy regression problem. I am baffled by the fact that it appears that each time I run a tensor through a model, the prediction changes. Clearly this is cant be the case but what am I missing?

Pytorch version: 0.4.0

I am running here without GPU to eliminate that potential issue.

Code:

from __future__ import division
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import torch
import torch.utils.data as utils_data
from torch.autograd import Variable
from torch import optim, nn
from torch.utils.data import Dataset
import torch.nn.functional as F
from torch.nn.init import xavier_uniform_, xavier_normal_,uniform_
from sklearn.datasets import load_boston
from sklearn.metrics import mean_squared_error

cuda=False #set to true uses GPU

#load boston data from scikit
boston = load_boston()
x=boston.data
y=boston.target
y=y.reshape(y.shape[0],1)

#change to tensors
x = torch.from_numpy(x)
y = torch.from_numpy(y)

#create dataset and use data loader
training_samples = utils_data.TensorDataset(x, y)
data_loader = utils_data.DataLoader(training_samples, batch_size=64)

#simple model
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()

        #all the layers
        self.fc1   = nn.Linear(x.shape[1], 20)
        xavier_uniform_(self.fc1.weight.data) #this is how you can change the weight init
        self.drop = nn.Dropout(p=0.5)
        self.fc2   = nn.Linear(20, 1)

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x=  self.drop(x)
        x = self.fc2(x)
        return x

net=Net()

if cuda:
    net.cuda()

# create a stochastic gradient descent optimizer
optimizer = optim.Adam(net.parameters())
# create a loss function (mse)
loss = nn.MSELoss(size_average=True)

# run the main training loop
epochs =50
hold_loss=[]

for epoch in range(epochs):
    cum_loss=0.
    for batch_idx, (data, target) in enumerate(data_loader):
        tr_x, tr_y = data.float(), target.float()
        if cuda:
            tr_x, tr_y = tr_x.cuda(), tr_y.cuda()

        # Reset gradient
        optimizer.zero_grad()

        # Forward pass
        fx = net(tr_x)
        output = loss(fx, tr_y) #loss for this batch

        cum_loss += output.item() #accumulate the loss

        # Backward
        output.backward()

        # Update parameters based on backprop
        optimizer.step()
    hold_loss.append(cum_loss/len(training_samples))

#training loss
plt.plot(np.array(hold_loss))

This part, if re-ran will return different predictions each time, the actuals dont change so the order of the data is not changing!

#score the training set
for batch_idx, (data, target) in enumerate(data_loader):
        tr_x, tr_y = data.float(), target.float()
        if batch_idx ==0:
           hold_pred=net(tr_x).data.numpy()
           hold_actual=tr_y.data.numpy().reshape(tr_y.data.numpy().shape[0],1)
        else:
            hold_pred =np.row_stack([hold_pred,net(tr_x).data.numpy()])
hold_actual=np.row_stack([hold_actual,tr_y.data.numpy().reshape(tr_y.data.numpy().shape[0],1)])

#view the first few predictions
print(hold_pred[0:10])
print(hold_actual[0:10])


title: Log-likelihood function in NumPy

body:
I followed this tutorial
and I was confused with the part where the author defines the negative-loglikelihood lost function.

def nll(input, target):
    return -input[range(target.shape[0]), target].mean()

loss_func = nll

Here, target.shape[0] is 64 and target is a vector with length 64

tensor([5, 0, 4, 1, 9, 2, 1, 3, 1, 4, 3, 5, 3, 6, 1, 7, 2, 8, 6, 9, 4, 0, 9, 1, 1, 2, 4, 3, 2, 7, 3, 8, 6, 9, 0, 5, 6, 0, 7, 6, 1, 8, 7, 9, 3, 9, 8, 5, 9, 3, 3, 0, 7, 4, 9, 8, 0, 9, 4, 1, 4, 4, 6, 0]).

How does that numpy indexing result in the loss function? Moreover, what should the output of a numpy array be when there are range() and another array inside the square bracket?


title: How to use Distributed Data Parallel properly in pytorch

body:
I’m relatively new to PyTorch, but I have good experience with Keras & TensorFlow. I’ve followed this article to use DDP on my own training script. However, for some reason, I always end up getting:

  process 0 terminated with exit status 1.

I've tried running the same code on multiple GPU platforms (Google Colab, Kaggle, Floyd Hub), but all of them pretty much threw me the same error.

I've also tried disabling the join=True option, but then the training process doesn't even start.

DDP related code:

def setup(rank, world_size):
    os.environ['MASTER_ADDR'] = 'localhost'
    os.environ['MASTER_PORT'] = '12355'

    # initialize the process group
    dist.init_process_group("gloo", rank=rank, world_size=world_size)

    # Explicitly setting seed to make sure that models created in two processes
    # start from same random weights and biases.
    torch.manual_seed(42)

def cleanup():
    dist.destroy_process_group()

def run_demo(fn, *args):
    mp.spawn(
        fn,
        args = (args[0], args[1], args[2], args[3], args[4]),
        nprocs = 1, # Also tried 2 , but no difference
        join = True
    )

And my training code:

def train(model, X, batch_size = 32, epochs = 75, gradient_acc = 0):
    setup(1, 2)
    device = model.get_default_device()
    model = model.to(device, non_blocking = True)
    ddp_model = DDP(model, device_ids = [0]) # Only one GPU

    # ...

    ddp_model.hidden_enc = ddp_model.init_hidden_enc()
    ddp_model.hidden_dec = ddp_model.init_hidden_dec()

    ddp_model.train()

    for ep in range(epochs):
        loss_br = 0; nb_batch_steps = 0
        for step, batch in enumerate( data_loader ):
            batch = batch.to(device, non_blocking = True)
            nb_batch_steps += 1
            loss = ddp_model(batch)
            # ...
    cleanup()

Call to training code:

if __name__ == "__main__":
    run_demo(
        train,
        model,
        holder[:], # X
        32,
        75,
        3
    )

I want the model to be running on multiple processes using distributed parallel package. Interestingly, sometimes I get Out of Memory exception for CUDA when I run it without using DDP. I understand that spawn.py terminates all the processes if any of the available processes exist with status code > 1, but I can't seem to figure out yet how to avoid this issue. Any help is highly appreciated.


title: torch in-place operations to save memory (softmax)

body:
Some operations in torch are executed in-place. Shorthand operators like += for example.

Is it possible to get in-place execution for other operations, such as softmax?

I'm currently working with language processing. The model produces a long sequence of probability distributions over a large vocabulary. This final output tensor is responsible for ca 60% of allocated memory. Which is a huge problem, since I need to calculate a softmax over it and that doubles the required memory.

Here is an example of the problem. I am not interested in the tensor t, only in its softmax:

import numpy as np
import torch
import torch.nn.functional as F

t = torch.tensor(np.zeros((30000,30000))).cuda()  #allocates 6.71 GB of GPU
softmax = F.softmax(t, 1)  #out of memory error
del t  #too late, program crashed

Even the following doesn't work:

F.softmax(torch.tensor(np.zeros((30000,30000))).cuda(), 1)


title: How to apply exponential moving average decay for variables in pytorch?

body:
I am reading following paper. And it uses EMA decay for variables.
Bidirectional Attention Flow for Machine Comprehension

  During training, the moving averages of all weights of the model are
  maintained with the exponential decay rate of 0.999.

They use TensorFlow and I found the related code of EMA.
https://github.com/allenai/bi-att-flow/blob/master/basic/model.py#L229

In PyTorch, how do I apply EMA to Variables?


title: 'package not found error' while installing pytorch on Windows Anaconda python

body: I have installed anaconda python 3.5.5 and willing to install pytorch. I tried many commands supporting this but landed into the ‘package not found error’.,yes i had make sure that i’m running the prompt as adminstrator. can i have any better solutions


title: What's the difference between sum and torch.sum for a torch Tensor?

body: I get the same results when using either the python sum or torch.sum so why did torch implement a sum function? Is there a difference between them?


title: PyTorch datasets: ImageFolder and subfolder filtering

body:
I would like to use ImageFolder to create an Image Dataset.

My current image directory structure looks like this:

/root
-- train/
---- 001.jpg
---- 002.jpg
---- ....
-- test/
---- 001.jpg
---- 002.jpg
---- ....

I would like to have a dataset dedicated to training data, and a dataset dedicated to test data.

As I understand, doing so:

dataset = ImageFolder(root='root/train')

does not find any images.

Doing

dataset = ImageFolder(root='root')

find images but train and test images are just scrambled together.

ImageFolder has argument loader but I did not manage to find any use-case for it.

How can I discriminate images in the root folder according to the subfolder they belong to?


title: How to use pytorch's grid_sample()?

body:
I am having some trouble getting torch.nn.functional working as I would like, illustrated by the below example:

import torch
import torch.nn.functional as F
import numpy as np

sz = 5
input_arr = torch.from_numpy(np.arange(sz*sz).reshape(1,1,sz,sz)).float()
indices = torch.from_numpy(np.array([-1,-1, -0.5,-0.5, 0,0, 0.5,0.5, 1,1]).reshape(1, 1, 5, 2)).float()

out = F.grid_sample(input_arr, indices)
print(input_arr)
print(out)

Since the indices are just the diagonals of the input, I'd expect to get something like tensor([[[[0., 6., 12., 18., 24.]]]]) (since (-1,-1) should give the top left and (1,1) should give the bottom right, according to the docs). However, I am getting this as output to the console:

tensor([[[[ 0.,  1.,  2.,  3.,  4.],
          [ 5.,  6.,  7.,  8.,  9.],
          [10., 11., 12., 13., 14.],
          [15., 16., 17., 18., 19.],
          [20., 21., 22., 23., 24.]]]])
tensor([[[[ 0.0000,  4.5000, 12.0000, 19.5000,  6.0000]]]])

What am I doing wrong? Many thanks!


title: Implementing Adam in Pytorch

body:
I’m trying to implement Adam by myself for a learning purpose.

Here is my Adam implementation:

class ADAMOptimizer(Optimizer):
    """
    implements ADAM Algorithm, as a preceding step.
    """
    def __init__(self, params, lr=1e-3, betas=(0.9, 0.99), eps=1e-8, weight_decay=0):
        defaults = dict(lr=lr, betas=betas, eps=eps, weight_decay=weight_decay)
        super(ADAMOptimizer, self).__init__(params, defaults)

    def step(self):
        """
        Performs a single optimization step.
        """
        loss = None
        for group in self.param_groups:
            #print(group.keys())
            #print (self.param_groups[0]['params'][0].size()), First param (W) size: torch.Size([10, 784])
            #print (self.param_groups[0]['params'][1].size()), Second param(b) size: torch.Size([10])
            for p in group['params']:
                grad = p.grad.data
                state = self.state[p]

                # State initialization
                if len(state) == 0:
                    state['step'] = 0
                    # Momentum (Exponential MA of gradients)
                    state['exp_avg'] = torch.zeros_like(p.data)
                    #print(p.data.size())
                    # RMS Prop componenet. (Exponential MA of squared gradients). Denominator.
                    state['exp_avg_sq'] = torch.zeros_like(p.data)

                exp_avg, exp_avg_sq = state['exp_avg'], state['exp_avg_sq']

                b1, b2 = group['betas']
                state['step'] += 1

                # L2 penalty. Gotta add to Gradient as well.
                if group['weight_decay'] != 0:
                    grad = grad.add(group['weight_decay'], p.data)

                # Momentum
                exp_avg = torch.mul(exp_avg, b1) + (1 - b1)*grad
                # RMS
                exp_avg_sq = torch.mul(exp_avg_sq, b2) + (1-b2)*(grad*grad)

                denom = exp_avg_sq.sqrt() + group['eps']

                bias_correction1 = 1 / (1 - b1 ** state['step'])
                bias_correction2 = 1 / (1 - b2 ** state['step'])

                adapted_learning_rate = group['lr'] * bias_correction1 / math.sqrt(bias_correction2)

                p.data = p.data - adapted_learning_rate * exp_avg / denom

                if state['step']  % 10000 ==0:
                    print ("group:", group)
                    print("p: ",p)
                    print("p.data: ", p.data) # W = p.data

        return loss

I think I implemented everything correct however the loss graph of my implementation is very spiky compared to that of torch.optim.Adam.

My ADAM implementation loss graph (below)

torch.optim.Adam loss graph (below)

If someone could tell me what I am doing wrong, I’ll be very grateful.

For the full code including data, graph (super easy to run): https://github.com/byorxyz/AMS_pytorch/blob/master/AdamFails_1dConvex.ipynb


Trying to train the mnist 28x28x1 images

my model is

   def __init__(self):
        super(CNN_mnist, self).__init__()
        self.conv = nn.Sequential(
            # 3 x 128 x 128
            nn.Conv2d(1, 32, 3, 1, 1),
            nn.BatchNorm2d(32),
            nn.LeakyReLU(0.2),

            # 32 x 128 x 128
            nn.Conv2d(32, 64, 3, 1, 1),
            nn.BatchNorm2d(64),
            nn.LeakyReLU(0.2),

            # 64 x 128 x 128
            nn.MaxPool2d(2, 2),

            # 64 x 64 x 64
            nn.Conv2d(64, 128, 3, 1, 1),
            nn.BatchNorm2d(128),
            nn.LeakyReLU(0.2),

            # 128 x 64 x 64
            nn.Conv2d(128, 256, 3, 1, 1),
            nn.BatchNorm2d(256),
            nn.LeakyReLU(0.2),

            # 256 x 64 x 64
            nn.MaxPool2d(2, 2),

            # 256 x 32 x 32
            nn.Conv2d(256, 10, 3, 1, 1),
            nn.BatchNorm2d(10),
            nn.LeakyReLU(0.2)
        )
        # 256 x 32 x 32
        self.avg_pool = nn.AvgPool2d(32)
        # 256 x 1 x 1
        self.classifier = nn.Linear(10, 10)

    def forward(self, x):
        features = self.conv(x)
        flatten = self.avg_pool(features).view(features.size(0), -1)
        output = self.classifier(flatten)
        return output, features

and I got following error

  RuntimeError: Given input size: (10x7x7). Calculated output size:
  (10x0x0). Output size is too small

not sure what this error mean and where should I fix it?


title: Pinning memory is actually slower in PyTorch?

body:
I'm wondering why would pinning memory in PyTorch make things even slower. By reading the code of torch.utils.data.dataloader, I found the pin_memory=True option of DataLoader simply calls .pin_memory() on each batch before returning them. The returned tensor is still on CPU, and I have to call .cuda(non_blocking=True) manually after this. Therefore, the whole process would be

for x in some_iter:
    yield x.pin_memory().cuda(non_blocking=True)

I compared the performance of this with

for x in some_iter:
    yield x.cuda()

Here is the actual code

a = torch.rand(1024, 655360)

%%time
for i in a:
    i.pin_memory().cuda(non_blocking=True)
# CPU times: user 1.35 s, sys: 55.8 ms, total: 1.41 s
# Wall time: 396 ms

%%time
for i in a:
    i.pin_memory().cuda()
# CPU times: user 1.6 s, sys: 12.2 ms, total: 1.62 s
# Wall time: 404 ms

%%time
for i in a:
    i.cuda(non_blocking=True)
# CPU times: user 855 ms, sys: 3.87 ms, total: 859 ms
# Wall time: 274 ms

%%time
for i in a:
    i.cuda()
# CPU times: user 314 ms, sys: 12 µs, total: 314 ms
# Wall time: 313 ms

As a result, not pinning memory both uses less CPU time, and is faster in terms of actual time. Shouldn't pinning memory make data transfer asynchronous and therefore be faster? If that's not the case, why would we do pin memory?

PS. I thought about the possibility of pinning a whole TensorDataset in advance (rather than pinning batches each time). But this cannot pin a tensor that is bigger than GPU memory:

a = np.memmap('../dat/R/train.3,31,31B', '3,31,31B', 'r')
a.nbytes // 2**30
## 68

torch.from_numpy(a).pin_memory()
## ---------------------------------------------------------------------------
## RuntimeError                              Traceback (most recent call last)
## <ipython-input-36-d6f2d74da8e7> in <module>
## ----> 1 torch.from_numpy(a).pin_memory()
##
## RuntimeError: cuda runtime error (2) : out of memory at /tmp/pip-req-build-58y_cjjl/aten/src/THC/THCCachingHostAllocator.cpp:296

And if I do want to pin a small tensor, why don't I directly move the whole tensor into GPU memory in advance?


title: How to save model architecture in PyTorch?

body:
I know I can save a model by torch.save(model.state_dict(), FILE) or torch.save(model, FILE). But both of them don't save the architecture of model.

So how can we save the architecture of a model in PyTorch like creating a .pb file in Tensorflow ? I want to apply different tweaks to my model. Do I have any better way than copying the whole class definition every time and creating a new class if I can't save the architecture of a model?


title: Segmentation fault (Core dumped) on importing torch with python3.5 in virtualenv

body:
I have installed torch=0.4.1 without cuda in a virtualenv. I am using python3.5 on ubuntu 16.04. Whenever I import torch in an interactive python shell, it quits the python program by showing Segmentation fault (core dumped).

While, surprisingly I had earlier initialized a jupyter notebook and tried importing torch there and it was running fine. Can someone please help? I could not find a solution on official `PyTorch github' discussions.


title: Torch allocates zero GPU memory on PyTorch

body:
I am trying to use GPU to train my model but it seems that torch fails to allocate GPU memory.

My model is a RNN built on PyTorch

device = torch.device('cuda: 0' if torch.cuda.is_available() else "cpu")

rnn = RNN(n_letters, n_hidden, n_categories_train)
rnn.to(device)
criterion = nn.NLLLoss()
criterion.to(device)
optimizer = torch.optim.SGD(rnn.parameters(), lr=learning_rate, weight_decay=.9)

class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(RNN, self).__init__()

        self.input_size = input_size
        self.hidden_size = hidden_size
        self.output_size = output_size

        self.i2h = nn.Linear(input_size + hidden_size, hidden_size)
        self.i2o = nn.Linear(input_size + hidden_size, output_size)

        self.softmax = nn.LogSoftmax(dim=1)

    def forward(self, input, hidden):
        input = input.cuda()
        hidden = hidden.cuda()

        combined = torch.cat((input, hidden), 1)
        hidden = self.i2h(combined)
        output = self.i2o(combined)
        output = self.softmax(output)

        output = output.cuda()
        hidden = hidden.cuda()

        return output, hidden

    def init_hidden(self):
        return Variable(torch.zeros(1, self.hidden_size).cuda())

Training function:

def train(category_tensor, line_tensor, rnn, optimizer, criterion):
    rnn.zero_grad()
    hidden = rnn.init_hidden()

    for i in range(line_tensor.size()[0]):
        output, hidden = rnn(line_tensor[i], hidden)

    loss = criterion(output, category_tensor)
    loss.backward()

    optimizer.step()

    return output, loss.item()

The function to get category_tensor and line_tensor:

def random_training_pair(category_lines, n_letters, all_letters):
    category = random.choice(all_categories_train)
    line = random.choice(category_lines[category])
    category_tensor = Variable(torch.LongTensor([all_categories_train.index(category)]).cuda())
    line_tensor = Variable(process_data.line_to_tensor(line, n_letters, all_letters)).cuda()

    return category, line, category_tensor, line_tensor

I ran the following the code:

 print(torch.cuda.get_device_name(0))
 print('Memory Usage:')
 print('Allocated:', round(torch.cuda.memory_allocated(0) / 1024 ** 3, 1), 'GB')
 print('Cached:   ', round(torch.cuda.memory_cached(0) / 1024 ** 3, 1), 'GB')

and I got:

GeForce GTX 1060
Memory Usage:
Allocated: 0.0 GB
Cached:    0.0 GB

I did not get any errors but GPU usage is just 1% while CPU usage is around 31%.

I am using Windows 10 and Anaconda, where my PyTorch is installed. CUDA and cuDNN is installed from .exe file downloaded from Nvidia website.


title: h5py randomly unable to open object (component not found)

body:
I'm trying to load hdf5 datasets into a pytorch training for loop.

Regardless of num_workers in dataloader, this randomly throws "KeyError: 'Unable to open object (component not found)' " (traceback below).

I'm able to start the training loop, but not able to get through 1/4 of one epoch without this error which happens for random 'datasets' (which are 2darrays each). I'm able to separately load these arrays in the console using the regular f['group/subroup'][()] so it doesn't appear like the hdf file is corrupted or that there's anything wrong with the datasets/array.

I've tried:

adjusting num_workers as per various other issues that people have had with pytorch - still happens with 0 num_workers.
upgrading /downgrading, torch, numpy and python versions.
using f.close() at the end of data loader getitem
using a fresh conda env and installing dependencies.
calling parent groups first, then initialising array eg:
X = f[ID] then X = X[()]
using double slashes in hdf path

Because this recurs with num_workers=0, I figure it's not a multithreading issue although the traceback seems to point to lines from /torch/utils/data/dataloader that prep the next batch.

I just can't figure out why h5py can't see the odd individual dataset, randomly.

IDs are strings to match hdf paths eg:
ID = "ID_12345//Ep_-1//AN_67891011//ABC"

excerpt from dataloader:

def __getitem__(self, index):

    ID = self.list_IDs[index]

    # Start hdf file in read mode:
    f = h5py.File(self.hdf_file, 'r', libver='latest', swmr=True)

    X = f[ID][()]

    X = X[:, :, np.newaxis] # torchvision 0.2.1 needs (H x W x C) for transforms

    y = self.y_list[index]

    if self.transform:
        X = self.transform(X)

    return ID, X, y

`

Expected: training for loop

Actual: IDs / datasets / examples are loaded fine initially, then after between 20 and 200 steps...

Traceback (most recent call last):

  File "Documents/BSSA-loc/mamdl/models/main_v3.py", line 287, in

      main()    File "Documents/BSSA-loc/mamdl/models/main_v3.py", line 203, in main
      for i, (IDs, images, labels) in enumerate(train_loader):    File "/home/james/anaconda3/envs/jc/lib/python3.7/site-packages/torch/utils/data/dataloader.py",
  line 615, in next
      batch = self.collate_fn([self.dataset[i] for i in indices])    File
  "/home/james/anaconda3/envs/jc/lib/python3.7/site-packages/torch/utils/data/dataloader.py",
  line 615, in
      batch = self.collate_fn([self.dataset[i] for i in indices])    File
  "/home/james/Documents/BSSA-loc/mamdl/src/data_loading/Data_loader_v3.py",
  line 59, in getitem
      X = f[ID][()]    File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper    File "h5py/_objects.pyx", line 55,
  in h5py._objects.with_phil.wrapper    File
  "/home/james/anaconda3/envs/jc/lib/python3.7/site-packages/h5py/_hl/group.py",
  line 262, in getitem
      oid = h5o.open(self.id, self._e(name), lapl=self._lapl)    File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5o.pyx", line 190, in h5py.h5o.open

  KeyError: 'Unable to open object (component not found)'


title: How do I load multiple grayscale images as a single tensor in pytorch?

body:
I'm currently trying to use a stack a set of images as a single entity for each label to train a CNN on using cross-validation. Given a dataset of 224x224x1 grayscale images sorted by:

Root/
    Class0/image0_view0.png
    Class0/image0_view1.png
    Class0/image0_view2.png
    ...
    Class1/image0_view0.png
    Class1/image0_view1.png
    Class1/image0_view2.png

How would I go about flowing 3 images (view 0, 1, and 2) as a single tensor with dimensions 224x224x3 (3 grayscale images)? In other words, how would I create a dataset of image stacks in pytorch using ImageFolder/DatasetFolder and DataLoader? Would I have to re-organize my folders and classes, or would it be easier to make the stacks when I make the splits for cross-validation?

Thank you for your time and help! Let me know if I can provide any more info.


title: How to make a PyTorch Distribution on GPU

body:
Is it possible to make the PyTorch distributions create their samples directly on GPU.

If I do

from torch.distributions import Uniform, Normal
normal = Normal(3, 1)
sample = normal.sample()

Then sample will be on CPU. Of course it is possible to do sample = sample.to(torch.device("cuda")) to make it on GPU. But is there a way to have the sample go directly to GPU without first creating it on CPU?

PyTorch distributions inherit from Object, not nn.Module so it does not have a to method the put the distribution instance on GPU.

Any ideas?


title: Elegant way to compare to torch.FloatTensor on GPU

body:
I try to compare two torch.FloatTensor (with only one entry) lying on GPU like this:

  if (FloatTensor_A > FloatTensor_B): do something

The problem is, that (FloatTensor_A > FloatTensor_B) gives ByteTensor back. Is there a way to do boolean comparison between these two scalar FloatTensors, without loading the tensors on CPU and converting them back to numpy or conventional floats?


title: Can't save PIL image in python

body:
I'm trying to run code from here https://github.com/pytorch/examples/tree/master/fast_neural_style, I run the following command from the terminal python neural_style/neural_style.py eval --content-image ./images/content-images/amber.jpg --model ./saved_models/mosaic.pth --output-image a.jpg --cuda 0, where a.jpg is output image, but every time I get an error:

Traceback (most recent call last):
  File "neural_style/neural_style.py", line 240, in <module>
    main()
  File "neural_style/neural_style.py", line 236, in main
    stylize(args)
  File "neural_style/neural_style.py", line 150, in stylize
    utils.save_image(args.output_image, output[0])
  File "C:\Users\Lenovo\Documents\fast_neural_style\neural_style\utils.py", line 19, in save_image
    img.save(filename)
  File "C:\Users\Lenovo\Anaconda3\lib\site-packages\PIL\Image.py", line 1991, in save
    fp = builtins.open(filename, "w+b")
FileNotFoundError: [Errno 2] No such file or directory: 'a.jpg'

I tried to prescribe before a.jpg full path with slashes and backslashes, but every time this error still appears.

I launched the following script with a command python test.py from the terminal:

from PIL import Image, ImageDraw
text = "test image"
color = (0, 0, 120)
img = Image.new('RGB', (100, 50), color)
imgDrawer = ImageDraw.Draw(img)
imgDrawer.text((10, 20), text)
img.save("pil-example.png")

There was again an error

Traceback (most recent call last):
  File "test.py", line 7, in <module>
    img.save("pil-example.png")
  File "C:\Users\Lenovo\Anaconda3\lib\site-packages\PIL\Image.py", line 1966, in save
    fp = builtins.open(filename, "w+b")
FileNotFoundError: [Errno 2] No such file or directory: 'pil-example.png'

But when I launched this script in a jupiter-notebook, it worked without errors.

I use: Windows 10, Python 3.7.1, Pillow 5.3.0.

I would be grateful for any advice in solving this problem!


title: Image is in JPEG but Torchvision shows image extension is unsupported

body:
I just resized the image dataset with Pillow and exported to JPEG with following steps:
Dataset is 1000K Celeb but I have taken few for testing propose.

img = img.crop([10, 10, 118, 118])
img = img.resize((image_size, image_size), Image.BILINEAR)
img.save(save_dir + img_list[i], 'JPEG')

Then I used torchvision's Imagefolder function to read data from folder

 mydata = torchvision.datasets.ImageFolder(data_path_here, transform=transform)

Image is exported in JPG format as expected but when tried to read the dataset from torchvision, it shows that file is not in JPEG format.
RuntimeError: Found 0 files in subfolders of: Data/celeb_data/resized_celeb/
Supported extensions are: .jpg,.jpeg,.png,.ppm,.bmp,.pgm,.tif,.tiff,.webp

OS: Ubuntu 18.04
IDE: Pycharm 19


title: Cannot import torch module

body:
I cannot seem to properly install pytorch on my computer, so here is the background of what I have done:
I had already installed python on my computer and it worked. I used it in Eclipse, using pyDev, so I don't know if that could be the problem. Now I want to install pytorch, so I installed anaconda and entered the command for installing pytorch. To get the right command, I use https://pytorch.org/get-started/locally/, where I tried the options both with and without cuda. In both cases I get an error when I type "import torch".
I have also installed miniconda and tried the same with that without succes. I also tried to work in IDLE in stead of Eclipse, but I keep getting the "no module named 'torch'" error. Each time I run a command in anaconda it appears that the installation is succesfull, but I still can't import 'torch'.
Any idea what the problem could be or what I could try?


title: Converting state-parameters of Pytorch LSTM to Keras LSTM

body:
I was trying to port an existing trained PyTorch model into Keras.

During the porting, I got stuck at LSTM layer.

Keras implementation of LSTM network seems to have three state kind of state matrices while Pytorch implementation have four.

For eg, for an Bidirectional LSTM with hidden_layers=64, input_size=512 & output size=128 state parameters where as follows

State params of Keras LSTM

[<tf.Variable 'bidirectional_1/forward_lstm_1/kernel:0' shape=(512, 256) dtype=float32_ref>,
 <tf.Variable 'bidirectional_1/forward_lstm_1/recurrent_kernel:0' shape=(64, 256) dtype=float32_ref>,
 <tf.Variable 'bidirectional_1/forward_lstm_1/bias:0' shape=(256,) dtype=float32_ref>,
 <tf.Variable 'bidirectional_1/backward_lstm_1/kernel:0' shape=(512, 256) dtype=float32_ref>,
 <tf.Variable 'bidirectional_1/backward_lstm_1/recurrent_kernel:0' shape=(64, 256) dtype=float32_ref>,
 <tf.Variable 'bidirectional_1/backward_lstm_1/bias:0' shape=(256,) dtype=float32_ref>]

State params of PyTorch LSTM

 ['rnn.0.rnn.weight_ih_l0', torch.Size([256, 512])],
 ['rnn.0.rnn.weight_hh_l0', torch.Size([256, 64])],
 ['rnn.0.rnn.bias_ih_l0', torch.Size([256])],
 ['rnn.0.rnn.bias_hh_l0', torch.Size([256])],
 ['rnn.0.rnn.weight_ih_l0_reverse', torch.Size([256, 512])],
 ['rnn.0.rnn.weight_hh_l0_reverse', torch.Size([256, 64])],
 ['rnn.0.rnn.bias_ih_l0_reverse', torch.Size([256])],
 ['rnn.0.rnn.bias_hh_l0_reverse', torch.Size([256])],

I tried to look in to the code of both implementation but not able to understand much.

Can someone please help me to transform 4-set of state params from PyTorch  into 3-set of state params in Keras


title: tf.concat operation in PyTorch

body:
torch.stack is not what I am searching.

I am looking for the Tensorflow's concat operation for Pytorch.

I have searched the doc http://pytorch.org/docs/0.3.0/


title: weighted mse loss in pytorch

body:
def weighted_mse_loss(input_tensor, target_tensor, weight = 1):
    observation_dim = input_tensor.size()[-1]
    streched_tensor = ((input_tensor - target_tensor) ** 2).view(-1, observation_dim)
    entry_num = float(streched_tensor.size())[0]
    non_zero_entry_num = torch.sum(streched_tensor[:,0] != 0).float()
    weighted_tensor = torch.mm(
        ((input_tensor - target_tensor)**2).view(-1, observation_dim),
        (torch.diag(weight.float().view(-1)))
    )
    return torch.mean(weighted_tensor) * weight.nelement() * entry_num / non_zero_entry_num

I can't understand how the code gives weighted Mean Square Error loss.
I get that observation_dim is the final output dimension, (the class number I guess), and after that line, I don't get it. Could someone help me figure out how the code calculates the loss?

Thanks a lot.


title: Wasserstein GAN implemtation in pytorch. How to implement the loss?

body:
I'm currently working on a project in pytorch on Wasserstein GAN (https://arxiv.org/pdf/1701.07875.pdf).

In Wasserstain GAN a new objective function is defined using the wasserstein distance as :

Which leads to the following algorithms for training the GAN:

My question is :

When implementing line 5 and 6 of the algorithm in pytorch should I be multiplying my loss -1 ? As in my code (I use RMSprop as my optimizer for both the generator and critic):

############################
  # (1) Update D network: maximize (D(x)) + (D(G(x)))
  ###########################
  for n in range(n_critic):

    D.zero_grad()

    real_cpu = data[0].to(device)
    b_size = real_cpu.size(0)

    output = D(real_cpu)

    #errD_real = -criterion(output, label) #DCGAN
    errD_real = torch.mean(output)

    # Calculate gradients for D in backward pass
    errD_real.backward()
    D_x = output.mean().item()

    ## Train with all-fake batch
    # Generate batch of latent vectors
    noise = torch.randn(b_size, 100, device=device) #Careful here we changed shape of input (original : torch.randn(4, 100, 1, 1, device=device))

    # Generate fake image batch with G
    fake = G(noise)

    # Classify all fake batch with D
    output = D(fake.detach())

    # Calculate D's loss on the all-fake batch
    errD_fake = torch.mean(output)

    # Calculate the gradients for this batch
    errD_fake.backward()
    D_G_z1 = output.mean().item()

    # Add the gradients from the all-real and all-fake batches
    errD = -(errD_real - errD_fake)

    # Update D
    optimizerD.step()

    #Clipping weights
    for p in D.parameters():
      p.data.clamp_(-0.01, 0.01)

As you can see, I do the operation errD = -(errD_real - errD_fake), with errD_real and errD_fake being respectively the mean of the predictions of the critic on real and fake samples.

To my understanding RMSprop should optimize the weights of the critic the following way :

w <- w - alpha*gradient(w)

(alpha being the learning rate divided by the square root of the weighted moving average of the squared gradient)

Since the optimization problem requires to "go" in the same direction as the gradient it should be required to multiply gradient(w) by -1 before optimizing the weights.

Do you think that my reasoning is right ?

The program runs but my results are quiet poor.

I follow the same logic for the generator's weights but this time in order to go in the opposite direction of the gradient:

      ############################
  # (2) Update G network: minimize -D(G(x))
  ###########################
  G.zero_grad()

  noise = torch.randn(b_size, 100, device=device)
  fake = G(noise)

  #label.fill_(fake_label)  # fake labels are real for generator cost

  # Since we just updated D, perform another forward pass of all-fake batch through D
  output = D(fake).view(-1)

  # Calculate G's loss based on this output

  #errG = criterion(output, label) #DCGAN
  errG = -torch.mean(output)

  # Calculate gradients for G
  errG.backward()
  D_G_z2 = output.mean().item()

  # Update G
  optimizerG.step()

Sorry for the long question, I tried to explain my doubt as clear as possible. Thank you everyone.


title: Load pickle saved GPU tensor with CPU?

body:
I save the last hidden layer of Bert for my following process using pickle on GPU.
# output is the last hidden layer of bert, transformed on GPU
with open(filename, 'wb') as f:
    pk.dump(output, f)

Is it possible to load it on my person laptop without GPU? I tried following code, but all failed.
# 1st try
with open(filename, 'rb') as f:
    torch.load(f, map_location='cpu')

# 2nd
torch.load(filename, map_location=torch.device('cpu'))

All get the following error
RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.

Is it possible to load the file on my laptop?


title: Cannot install pytorch in a virtualenv on windows

body:
I know there are a few topics about that on this website but still, I can't find the solution.

So here is what I did :
I created a projecton Visual Studio 19 for python.
I added an virtual environment with Python 3.7 using the file requirements.txt

It contains

mypy==0.750
pylint==2.4.4
pytest==5.3.1
matplotlib==3.1.1
torch==1.3.1
tensorflow==2.1.0rc0

This requirement file works perfectly well on linux. But on vs19 I get the following :

ERROR: Could not find a version that satisfies the requirement torch==1.3.1 (from -r E:\Documents\Blub\Granolar\requirements.txt (line 5)) (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)`

Since I don't want to downgrade to 0.1.2 (obviously) how can I fix it?


title: Python Jupyter notebook how to display matrix in vertical

body:
When I try to index array, I use this code for printing column part with Numpy or Pytorch.

import numpy as np
a = np.random.randn(5,3)
a[:,1]

or

import torch
a = torch.Tensor(5,3)
a[:,1]

The output is displayed like this.

array([-0.07478094, -1.87787326,  0.50407517,  1.13335836,  0.23140931])

But I want to display output as column.(because i indexed column)

array([-0.07478094,
       -1.87787326,
        0.50407517,
        1.13335836,
        0.23140931])

Furthermore, When I make tensor with torch.ones(5), the result is

tensor([1., 1., 1., 1., 1.])

but I want to see the type of output on the buttom like this

tensor([1., 1., 1., 1., 1.])
[torch.FloatTensor of size 5]

The reason why i want to display this is that i can't distinguish tensor and numpy

Can anyone tell me how to do this? Thanks.


title: Multilabel classification with class imbalance in Pytorch

body:
I have a multilabel classification problem, which I am trying to solve with CNNs in Pytorch. I have 80,000 training examples and 7900 classes; every example can belong to multiple classes at the same time, mean number of classes per example is 130.

The problem is that my dataset is very imbalance. For some classes, I have only ~900 examples, which is around 1%. For “overrepresented” classes I have ~12000 examples (15%). When I train the model I use BCEWithLogitsLoss from pytorch with a positive weights parameter. I calculate the weights the same way as described in the documentation: the number of negative examples divided by the number of positives.

As a result, my model overestimates almost every class… Mor minor and major classes I get almost twice as many predictions as true labels. And my AUPRC is just 0.18. Even though it’s much better than no weighting at all, since in this case the model predicts everything as zero.

So my question is, how do I improve the performance? Is there anything else I can do? I tried different batch sampling techniques (to oversample minority class), but they don’t seem to work.


title: NameError: "linear regression" is not defined

body:
Here is a code snippet where I am applying Linear regression using Pytorch. I face a NameError, that says name "linear regression" not defined. Kindly help in rectifying it.

import torch
from torch.autograd import Variable
import torch.nn as nn
import torch.nn.functional as F

x_data=Variable(torch.Tensor([[10.0],[9.0],[3.0],[2.0]]))
y_data=Variable(torch.Tensor([[90.0],[80.0],[50.0],[30.0]]))

class LinearRegression(torch.nn.Module):

  def __init__(self):
    super(LinearRegression,self). __init__ ()
    self.linear = torch.nn.Linear(1,1)

  def forward(self, x):
    y_pred = self.linear(x)
    return y_pred

  model = LinearRegression()


title: Pytorch 1.6.0 - RuntimeError: CUDA error: device-side assert triggered where (predicted==labels)

body:
When I run this function:
def evaluate(model, dataloader, calc_loss=False):
'''
Helper function to get classification accuracy and (optional) loss for a model over the items in dataloader
This function can just return the accuracy, or calculate the losses if "calc_loss" is set to "True"
'''
correct = 0
total = 0
losses = []

# don't compute gradients
with torch.no_grad():
    for batch in dataloader:

        inputs, labels = batch
        # forward pass through the model
        outputs = model(inputs)
        print('pass 1')
        # calculate loss if parameter set to True
        if (calc_loss==True):
            loss = F.cross_entropy(outputs, labels)   # Calculate loss
            losses.append(loss.detach)

        # Get the prediction of the net on the images
        print('pass 2')
        _, predicted = torch.max(outputs.data, dim=1)
        total += labels.size(0)

        # Count those we got correct
        right = torch.sum(predicted==labels).item()

        print('pass 3', right)
        break

    # calculate total correct cases
    acc = 100 * correct / total
return losses, acc

Here is what i plug in and get back:
evaluate(model_1, dataloader=val_dl, calc_loss=True)

Results:

pass 1
pass 2
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-22-881e9c4f2402> in <module>
      1 os.environ['CUDA_LAUNCH_BLOCKING'] = '1'
----> 2 evaluate(model_1, dataloader=val_dl, calc_loss=True)

<ipython-input-15-edcb180f1931> in evaluate(model, dataloader, calc_loss)
     29
     30             # Count those we got correct
---> 31             right = torch.sum(predicted==labels).item()
     32
     33             print('pass 3', right)

RuntimeError: CUDA error: device-side assert triggered

It seems to be getting this error where the predict==labels are at, but when I run that code in isolation (outside of the for loop / function) it runs everything perfectly. Any ideas?


title: Running conv2d on tensor [batch, channel, sequence, H,W] in Pytorch

body:
I am working on a video frame data where I am getting input data as tensor of the form [batch,channel,frame_sequence,height, weight] (let denote it by [B,C,S,H,W] for clarity. So each batch basically consists of a consecutive sequence of frame. What I basically want to do is run an encoder (consisting of several conv2d) on each frame ie each [C,H,W] and get it back as [B,C_output,S,H_output,W_output]. Now conv2d expects input as (N,C_in,H_in,W_in) form. I am wondering what's the best way to do this without messing up the order within the 5D tensor.
So far I am considering following way of thoughts:

>>> # B,C,seq,h,w
# 4,2, 5,  3,3

>>> x = Variable(torch.rand(4,2,5,3,3))
>>> x.size()
#torch.Size([4, 2, 5, 3, 3])
>>> x = x.permute(0,2,1,3,4)
>>> x.size() #expected = 4,5,2,3,3 B,seq,C,h,w
#torch.Size([4, 5, 2, 3, 3])
>>> x = x.contiguous().view(-1,2,3,3)
>>> x.size()
#torch.Size([20, 2, 3, 3])

And then run conv2d (encoder) on the updated x and reshape it. But I think It wouldn't preserve the original order of tensor. So, how can I achieve the goal?


title: How to implement low-dimensional embedding layer in pytorch

body:
I recently read a paper about embedding.

In Eq. (3), the f is a 4096X1 vector. the author try to compress the vector in to theta (a 20X1 vector) by using an embedding matrix E.

The equation is simple  theta = E*f

I was wondering if it can using pytorch to achieve this goal, then in the training, the E can be learned automatically.

How to finish the rest? thanks so much.

The demo code is follow:

import torch
from torch import nn

f = torch.randn(4096,1)


title: GPU usage shows zero when CUDA with PyTorch using on Windows

body:
I have pytorch script.

import torch

torch.cuda.is_available()
# True

device=torch.device('cuda:0')
# I moved my tensors to device

But Windows Task Manager shows zero GPU (NVIDIA GTX 1050TI) usage when pytorch script running
Speed of my script is fine and if I had changing torch.device to CPU instead GPU a speed become slower, therefore cuda (GPU) is working. Why Windows Task Manager doesn't show GPU usage?

Sample of my code:

device=torch.device("cuda:0")
model=torch.load('mymodel.pth', map_location=torch.device(device))
image=Image.open('picture.png').convert('RGB')
transform=transforms.Compose([
            transforms.Resize(224),
            transforms.CenterCrop(224),
            transforms.ToTensor(),
            transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
        ])
input=transform(image)
input=torch.unsqueeze(input, 0)
input=input.to(device)
output=model(input)


title: Understanding the code in pyTorch

body:
I am having problems with understanding the following part of the code from ResNet architecture. The full code is available at https://github.com/yunjey/pytorch-tutorial/blob/master/tutorials/02-intermediate/deep_residual_network/main-gpu.py . I am not very familiar with Python.

# Residual Block
class ResidualBlock(nn.Module):
    def __init__(self, in_channels, out_channels, stride=1, downsample=None):
        super(ResidualBlock, self).__init__()
        self.conv1 = conv3x3(in_channels, out_channels, stride)
        self.bn1 = nn.BatchNorm2d(out_channels)
        self.relu = nn.ReLU(inplace=True)
        self.conv2 = conv3x3(out_channels, out_channels)
        self.bn2 = nn.BatchNorm2d(out_channels)
        self.downsample = downsample

    def forward(self, x):
        residual = x
        out = self.conv1(x)
        out = self.bn1(out)
        out = self.relu(out)
        out = self.conv2(out)
        out = self.bn2(out)
        if self.downsample:
            residual = self.downsample(x)
        out += residual
        out = self.relu(out)
        return out

# ResNet Module
class ResNet(nn.Module):
    def __init__(self, block, layers, num_classes=10):
        super(ResNet, self).__init__()
        self.in_channels = 16
        self.conv = conv3x3(3, 16)
        self.bn = nn.BatchNorm2d(16)
        self.relu = nn.ReLU(inplace=True)
        self.layer1 = self.make_layer(block, 16, layers[0])
        self.layer2 = self.make_layer(block, 32, layers[0], 2)
        self.layer3 = self.make_layer(block, 64, layers[1], 2)
        self.avg_pool = nn.AvgPool2d(8)
        self.fc = nn.Linear(64, num_classes)

    def make_layer(self, block, out_channels, blocks, stride=1):
        downsample = None
        if (stride != 1) or (self.in_channels != out_channels):
            downsample = nn.Sequential(
                conv3x3(self.in_channels, out_channels, stride=stride),
                nn.BatchNorm2d(out_channels))
        layers = []
        layers.append(block(self.in_channels, out_channels, stride, downsample))
        self.in_channels = out_channels
        for i in range(1, blocks):
            layers.append(block(out_channels, out_channels))
        return nn.Sequential(*layers)

    def forward(self, x):
        out = self.conv(x)
        out = self.bn(out)
        out = self.relu(out)
        out = self.layer1(out)
        out = self.layer2(out)
        out = self.layer3(out)
        out = self.avg_pool(out)
        out = out.view(out.size(0), -1)
        out = self.fc(out)
        return out

resnet = ResNet(ResidualBlock, [3, 3, 3])

My main question is why should we pass 'block' every time? In the function

def make_layer(self, block, out_channels, blocks, stride=1):

instead of passing 'block' why cant we create an instance of 'ResidualBlock' and append it with layers as follows?

   block = ResidualBlock(self.in_channels, out_channels, stride, downsample)
   layers.append(block)


title: Pytorch: "Model Weights not Changing"

body:
Can someone help me understand why the weights are not updating?

    unet = Unet()
    optimizer = torch.optim.Adam(unet.parameters(), lr=0.001)
    loss_fn = torch.nn.MSELoss()
    input =  Variable(torch.randn(32, 1, 64, 64, 64 ), requires_grad=True)
    target = Variable(torch.randn(32, 1, 64, 64, 64), requires_grad=False)

    optimizer.zero_grad()
    y_pred = unet(input)
    y = target[: , : , 20:44, 20:44, 20:44]

    loss = loss_fn(y_pred, y)
    print(unet.conv1.weight.data[0][0]) # weights of the first layer in the unet
    loss.backward()
    optimizer.step()
    print(unet.conv1.weight.data[0][0]) # weights havent changed

The model is defined like:

class Unet(nn.Module):

def __init__(self):
  super(Unet, self).__init__()

  # Down hill1
  self.conv1 = nn.Conv3d(1, 2, kernel_size=3,  stride=1)
  self.conv2 = nn.Conv3d(2, 2, kernel_size=3,  stride=1)

  # Down hill2
  self.conv3 = nn.Conv3d(2, 4, kernel_size=3,  stride=1)
  self.conv4 = nn.Conv3d(4, 4, kernel_size=3,  stride=1)

  #bottom
  self.convbottom1 = nn.Conv3d(4, 8, kernel_size=3,  stride=1)
  self.convbottom2 = nn.Conv3d(8, 8, kernel_size=3,  stride=1)

  #up hill1
  self.upConv0 = nn.Conv3d(8, 4, kernel_size=3,  stride=1)
  self.upConv1 = nn.Conv3d(4, 4, kernel_size=3,  stride=1)
  self.upConv2 = nn.Conv3d(4, 2, kernel_size=3,  stride=1)

  #up hill2
  self.upConv3 = nn.Conv3d(2, 2, kernel_size=3, stride=1)
  self.upConv4 = nn.Conv3d(2, 1, kernel_size=1, stride=1)

  self.mp = nn.MaxPool3d(kernel_size=3, stride=2, padding=1)
  # some more irrelevant properties...

The forward function looks like:

def forward(self, input):
    # Use U-net Theory to Update the filters.
    # Example Approach...
    input = F.relu(self.conv1(input))
    input = F.relu(self.conv2(input))

    input = self.mp(input)

    input = F.relu(self.conv3(input))
    input = F.relu(self.conv4(input))

    input = self.mp(input)

    input = F.relu(self.convbottom1(input))
    input = F.relu(self.convbottom2(input))

    input = F.interpolate(input, scale_factor=2, mode='trilinear')

    input = F.relu(self.upConv0(input))
    input = F.relu(self.upConv1(input))

    input = F.interpolate(input, scale_factor=2, mode='trilinear')

    input = F.relu(self.upConv2(input))
    input = F.relu(self.upConv3(input))

    input = F.relu(self.upConv4(input))

    return input

I have followed the approach of any example and documentation i could find and it is beyound me why that doesn't work?

I can figure out as much that y_pred.grad after the backward call is none which it shouldn't be. If we have no gradient then ofcourse the optimizer can't change the weights in any direction but why is there no gradient?


title: Speed up SVD in PyTorch

body:
I am doing some classification task for CIFAR10 with Pytorch and for each iteration I have to do some preprocessing on each batch before I would be able to feed forward to the model.
Below is the code for the preprocessing part on each batch:

S = torch.zeros((batch_size, C, H, W))
for i in range(batch_size):
    img = batch[i, :, :, :]
    for c in range(C):
        U, _, V = torch.svd(img[c])
        S[i, c] = U[:, 0].view(-1, 1).matmul(V[:, 0].view(1, -1))

However, this calculation is very slow. Is there any way that I could speed up this code?


title: python 3.6 stopped unexpectedly with segmentation fault (core dumped)

body:
I am using Python3.6 to run PyTorch for a model evaluation and after the evaluation is done and results are saved I get the "segmentation fault (core dumped)" error. I can not see any interruption in the evaluation as everything is done and results are saved but this error is making me uncomfortable. Any idea for what the cause might be ?
The environment has the following packages:
dataclasses==0.7 future==0.18.2 numpy==1.19.0 Pillow==7.2.0 pkg-
resources==0.0.0 six==1.15.0 tabulate==0.8.7 torch @~/torch-1.5.0-
cp36-cp36m-linux_aarch64.whl torchvision==0.2.2.post3

Screen shot 1  Screen shot 2  Screen shot 3     Here is
my evaluation code:

import argparse
import json
import os
import shutil
import sys
from pathlib import Path

import numpy as np
import torch
from PIL import Image
from torchvision.transforms import Compose, ToTensor, Normalize

from countmacs import MAC_Counter
from nets import SQUEEZENAS_NETWORKS

DATA_DIR = Path('./data')
os.environ['CITYSCAPES_DATASET'] = str(DATA_DIR.absolute())
print(os.environ['CITYSCAPES_DATASET'])

sys.path.insert(0, 'cityscapesScripts')  # add subdir to path

#RESULTS_DIR = Path('./results')
RESULTS_DIR = Path('/media/cdleml/16 GB Vol./results')
DATASET_DIR = DATA_DIR / 'leftImg8bit/'
INPUT_DATA = list((DATASET_DIR / 'val').glob('**/*.png'))

normalize = Compose([ToTensor(), Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])])

def map_back_to_35(gt):
    from cityscapesScripts.cityscapesscripts.helpers.labels import labels
    """ Maps neural net predictions back to 0-35 range, from 0-18 range """
    tmp = []
    for label in labels:
        if label.trainId != 255:
            tmp.append(label.id)  # creates a len 20 list [7,8,11,...]

    while len(tmp) <= 255:
        tmp.append(0)

    mapping = np.array(tmp)
    return mapping[gt]

def main(verbose: bool, only_macs: bool, save_output: bool = True, net_name: str = "squeezenas_mac_small",
         use_cpu: bool = True):
    result_dir = Path(RESULTS_DIR) / net_name
    result_dir.mkdir(parents=True, exist_ok=True)

    preds_dir = result_dir / 'predictions'
    preds_dir.mkdir(exist_ok=True)
    os.environ['CITYSCAPES_RESULTS'] = str(preds_dir)

    from cityscapesScripts.cityscapesscripts.evaluation.evalPixelLevelSemanticLabeling import \
        main as metrics  # noqa: E402

    net_constructor = SQUEEZENAS_NETWORKS[net_name]
    model = net_constructor()
    if not use_cpu:
        model = model.cuda()

    print('-' * 54)
    print(f'Evaluating {net_name}')
    print('-' * 54)

    print("Counting MACs")
    counter = MAC_Counter(model, [1, 3, 1024, 2048])
    if verbose:
        counter.print_layers()
    macs = counter.print_summary()

    print('-' * 54)

    if only_macs:
        return

    model.eval()

    print("Evaluating Model on the Validation Dataset")
    for idx, fname in enumerate(INPUT_DATA):  # run inference and save predictions
        print(f'\rSaving prediction {idx} out of {len(INPUT_DATA)}', end='')
        data = normalize(Image.open(fname))[None]
        if not use_cpu:
            data = data.cuda()
        output = model(data)
        pred = output['preds']
        pred = torch.argmax(pred[0], dim=0)
        pred = pred.cpu().data.numpy()
        pred = map_back_to_35(pred).astype(np.uint8)
        assert pred.shape == (1024, 2048), pred.shape
        preds_pil = Image.fromarray(pred, mode='L')
        preds_pil.save(preds_dir / fname.name, format='PNG')

    print('\n' + '-' * 54)
    sys.argv = [sys.argv[0]]
    metrics()  # run evaluation using Cityscapes metrics

    export_dir = result_dir / 'evaluationResults'

    if export_dir.exists():
        shutil.rmtree(export_dir)

    shutil.move(DATA_DIR / 'evaluationResults/', export_dir)
    with (result_dir / 'results.txt').open('w') as results_file:
        with open(export_dir / 'resultPixelLevelSemanticLabeling.json') as f:
            scores = json.load(f)  # get the net's performance
            classIOU = scores['averageScoreClasses']
            categoryIOU = scores['averageScoreCategories']

            line = f'Name: {net_name} \tclassIOU: {classIOU} \tcategoryIOU: {categoryIOU} \tGigaMACs: {macs["total_gmacs"]}\n'
            results_file.write(line)
            print(line)

if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('-v', '--verbose', dest='verbose', action='store_true',
                        help='if this option is supplied, a full layer'
                             ' by layer GigaMAC summary of the model will be printed. If this option is not supplied,'
                             ' only the total GigaMACs will be printed.')

    parser.add_argument('-m', '--only_macs', dest='only_macs', action='store_true',
                        help='if this option is supplied, no inference is run, only MAC count is printed.')

    parser.add_argument('-c', '--use_cpu', action='store_true',
                        help='If this option supplied, the network will be evaluated using the cpu.'
                             ' Otherwise the gpu will be used to run evaluation.')

    parser.add_argument('--net', type=str, choices=sorted(SQUEEZENAS_NETWORKS.keys()), default="squeezenas_lat_small")

    args, unknown = parser.parse_known_args()
    print(args)
    main(verbose=args.verbose, only_macs=args.only_macs, net_name=args.net, use_cpu=args.use_cpu)


title: How to extract feature vector from single image in Pytorch?

body:
I am attempting to understand more about computer vision models, and I'm trying to do some exploring of how they work. In an attempt to understand how to interpret feature vectors more I'm trying to use Pytorch to extract a feature vector. Below is my code that I've pieced together from various places.
import torch
import torch.nn as nn
import torchvision.models as models
import torchvision.transforms as transforms
from torch.autograd import Variable
from PIL import Image

img=Image.open("Documents/01235.png")

# Load the pretrained model
model = models.resnet18(pretrained=True)

# Use the model object to select the desired layer
layer = model._modules.get('avgpool')

# Set model to evaluation mode
model.eval()

transforms = torchvision.transforms.Compose([
        torchvision.transforms.Resize(256),
        torchvision.transforms.CenterCrop(224),
        torchvision.transforms.ToTensor(),
        torchvision.transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
    ])

def get_vector(image_name):
    # Load the image with Pillow library
    img = Image.open("Documents/Documents/Driven Data Competitions/Hateful Memes Identification/data/01235.png")
    # Create a PyTorch Variable with the transformed image
    t_img = transforms(img)
    # Create a vector of zeros that will hold our feature vector
    # The 'avgpool' layer has an output size of 512
    my_embedding = torch.zeros(512)
    # Define a function that will copy the output of a layer
    def copy_data(m, i, o):
        my_embedding.copy_(o.data)
    # Attach that function to our selected layer
    h = layer.register_forward_hook(copy_data)
    # Run the model on our transformed image
    model(t_img)
    # Detach our copy function from the layer
    h.remove()
    # Return the feature vector
    return my_embedding

pic_vector = get_vector(img)

When I do this I get the following error:
RuntimeError: Expected 4-dimensional input for 4-dimensional weight [64, 3, 7, 7], but got 3-dimensional input of size [3, 224, 224] instead

I'm sure this is an elementary error, but I can't seem to figure out how to fix this. It was my impression that the "totensor" transformation would make my data 4-d, but it seems it's either not working correctly or I'm misunderstanding it. Appreciate any help or resources I can use to learn more about this!


title: Loss doesn't decrease in training the pytorch RNN

body:
Here is the RNN network I designed for a sentiment.

class rnn(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super().__init__()
        self.hidden_size = hidden_size
        self.i2h = nn.Linear(input_size, hidden_size)
        self.h2o = nn.Linear(hidden_size, output_size)
        self.h2h = nn.Linear(hidden_size , hidden_size)
        self.relu = nn.Tanh()
        self.sigmoid = nn.LogSigmoid()

    def forward(self, input, hidden):
        hidden_new = self.relu(self.i2h(input)+self.h2h(hidden))
        output = self.h2o(hidden)
        output = self.sigmoid(output)
        return output, hidden_new

    def init_hidden(self):
        return Variable(torch.zeros(1, self.hidden_size))

Then, I create and train the network as :

RNN = rnn(50, 50, 1)
learning_rate = 0.0005
criteria = nn.MSELoss()
optimizer = optim.Adam(RNN.parameters(), lr=learning_rate)
hidden = RNN.init_hidden()
epochs = 2
for epoch in range(epochs):
    for i in range(len(train['Phrase'])):
        input = convert_to_vectors(train['Phrase'][i])
        for j in range(len(input)):
            temp_input = Variable(torch.FloatTensor(input[j]))
            output, hidden = RNN(temp_input, hidden)
        temp_output = torch.FloatTensor([np.float64(train['Sentiment'][i])/4])
        loss = criteria( output, Variable(temp_output))
        loss.backward(retain_graph = True)
        if (i%20 == 0):
            print('Current loss is ', loss)

The problem is that the loss of the network isn't decreasing. It increases, then decreases and so on. It isn't stable at all. I tried using a smaller learning rate but it doesn't seem to help.

Why is this happening and how can I rectify this?


title: Why doesn't setting `random.seed(42)` give me identical results in pytorch?

body:
I am setting random seed for both random and numpy.random at the beginning of my main file:

import random
import numpy as np
np.random.seed(42)
random.seed(42)
import torch

Nevertheless, when I create a Net() object with randomly initialized parameters, it gives a completely different result every time:

net=neuralnet.Net()
print ("initialized params: ", net.fc1.weight)

Note that neuralnet.Net() is in a different file, and is a class that extends torch.nn.Module. it is torch.nn.Module that is randomly initializing net.fc1.weight, not my own code.

How is it possible that when I create a Net() object with randomly initialized parameters, it gives a completely different result every time?


title: Why do my kernel dies and restart while training my neural network with PyTorch?

body:
I'm a beginner with python and PyTorch and I've got to implement a neural network for image recognition.

Everything works fine except that my kernel dies and restart automatically at the 78th iteration. At each iteration, we are training the network on a batch of 64 images (the full set has 5000 images).

I am using a Macbook Pro (10.14), and I installed Pytorch using pip. This is the very first time I encounter this issue.

I dig a bit on the internet and it might be a memory issue?

Thanks for reading!

 epochs = 10
 steps = 0
 print_every = 20
 running_loss = 0
 for e in range(epochs):
 start = time.time()

 for i, (images, labels) in enumerate(iter(valloader)):
    steps += 1

    images.resize_(64, 3072)
    optimizer.zero_grad()
    output = net.forward(images)
    loss = loss_func(output, labels)
    loss.backward()
    optimizer.step()

    running_loss += loss.item()


I'm trying to load pre-trained model weights using this line :

state_dict = torch.load('models/seq_to_txt_state_7.tar')

and I'm getting:

       KeyError                                  Traceback (most recent call last)
<ipython-input-30-3f7b5be8fc72> in <module>()
----> 1 state_dict = torch.load('models/seq_to_txt_state_7.tar')

/home/arash/venvs/marzieh_env/local/lib/python2.7/site-packages/torch/serialization.pyc in load(f, map_location, pickle_module)
    365         f = open(f, 'rb')
    366     try:
--> 367         return _load(f, map_location, pickle_module)
    368     finally:
    369         if new_fd:

/home/arash/venvs/marzieh_env/local/lib/python2.7/site-packages/torch/serialization.pyc in _load(f, map_location, pickle_module)
    521         # only if offset is zero we can attempt the legacy tar file loader
    522         try:
--> 523             return legacy_load(f)
    524         except tarfile.TarError:
    525             # if not a tarfile, reset file offset and proceed

/home/arash/venvs/marzieh_env/local/lib/python2.7/site-packages/torch/serialization.pyc in legacy_load(f)
    448                 mkdtemp() as tmpdir:
    449
--> 450             tar.extract('storages', path=tmpdir)
    451             with open(os.path.join(tmpdir, 'storages'), 'rb', 0) as f:
    452                 num_storages = pickle_module.load(f)

/usr/lib/python2.7/tarfile.pyc in extract(self, member, path)
   2107
   2108         if isinstance(member, basestring):
-> 2109             tarinfo = self.getmember(member)
   2110         else:
   2111             tarinfo = member

/usr/lib/python2.7/tarfile.pyc in getmember(self, name)
   1827         tarinfo = self._getmember(name)
   1828         if tarinfo is None:
-> 1829             raise KeyError("filename %r not found" % name)
   1830         return tarinfo
   1831

KeyError: "filename 'storages' not found"

I'm using python 2.7 on Ubuntu 18.

In addition the model is saved using this function in first place:

def save_state(enc, dec, enc_optim, dec_optim, dec_idx_to_word, dec_word_to_idx, epoch):
state = {'enc':enc.state_dict(), 'dec':dec.state_dict(),
         'enc_optim':enc_optim.state_dict(), 'dec_optim':dec_optim.state_dict(),
        'dec_idx_to_word':dec_idx_to_word, 'dec_word_to_idx':dec_word_to_idx}
torch.save(state, epoch_to_save_path(epoch))


title: Can not find

body:
I create a pytorch extention, followed by this link
but it throws out the error that

  fatal error: torch/extension.h: No such file or directory.

ubuntu 18.04
code::blocks 17.04
gcc 7.4.0

#include <torch/extension.h>

I expect the code to run.
Is there any document to fix this problem?


title: How to save grayscale image in Pytorch?

body:
I want to save grayscale image in Pytorch, each image has four gray values, 0, 60, 120 and 180. I try the following way to save images, but the saved image is not I expected.

for i, (inputs) in enumerate(test_generator):
    pred = modelPl(inputs.float()).detach()
    fig,ax = plt.subplots(1,1,figsize = (5,5))
    ax.imshow(pred[0,:,:], cmap = "gray")
    print(pred.shape)
    torchvision.utils.save_image(pred, saveTestPath + 'img_{0}.png'.format(i))

Output:
torch.Size([400, 400])

Expected image:

However, the saved picture is not correct as follows:


title: Do all variables in the loss function have to be tensor with grads in pytorch?

body:
I have the following function

def msfe(ys, ts):
    ys=ys.detach().numpy() #output from the network
    ts=ts.detach().numpy() #Target (true labels)
    pred_class = (ys>=0.5)
    n_0 = sum(ts==0) #Number of true negatives
    n_1 = sum(ts==1) #Number of true positives
    FPE = sum((ts==0)[[bool(p) for p in (pred_class==1)]])/n_0 #False positive error
    FNE = sum((ts==1)[[bool(p) for p in (pred_class==0)]])/n_1 #False negative error
    loss= FPE**2+FNE**2

    loss=torch.tensor(loss,dtype=torch.float64,requires_grad=True)

    return loss

and I wonder, if the autograd in Pytorch works properly, since ys and ts does not have the grad flag.

So my question is: do all the variables (FPE,FNE,ys,ts,n_1,n_0) have to be tensors, before optimizer.step() works, or is it okay that it is only the final function (loss) which is ?


title: Does a `DataLoader` created from `ConcatDataset` create a batch from a different files, or a single file?

body:
I am working with multiple files, and multiple training samples in each file. I will use ConcatDataset as described here:

https://discuss.pytorch.org/t/dataloaders-multiple-files-and-multiple-rows-per-column-with-lazy-evaluation/11769/7

I need to have negative samples in addition to my true samples, and I need my negative samples to be randomly selected from all the training data files. So, I am wondering, would the returned batch samples just be a random consecutive chuck from a single file, or would be batch span across multiple random indexes across all the datafiles?

If there are more details needed about what I am trying to do exactly, it's because I am trying to train over a TPU with Pytorch XLA.

Normally for negative samples, I would just use a 2nd DataSet and DataLoader, however, I am trying to train over TPUs with Pytorch XLA (alpha was just released a few days ago https://github.com/pytorch/xla ), and to do that I need to send my DataLoader to a torch_xla.distributed.data_parallel.DataParallel object, like model_parallel(train_loop_fn, train_loader) which can be seen in these example notebooks

https://github.com/pytorch/xla/blob/master/contrib/colab/resnet18-training-xrt-1-15.ipynb

https://github.com/pytorch/xla/blob/master/contrib/colab/mnist-training-xrt-1-15.ipynb

So, I am now limited to a single DataLoader, which will need to handle both the true samples, and negative samples that need to be randomly selected from all my files.


title: Pytorch: Dropout Layers and Packed Sequences

body:
(PyTorch 0.4.0)

How does one apply a manual dropout layer to a packed sequence (specifically in an LSTM on a GPU)? Passing the packed sequence (which comes from the lstm layer) directly does not work, as the dropout layer doesn’t know quite what to do with it and returns something not a packed sequence. Passing the data of the packed sequence seems like it should work, but results in the attribute error shown below the code sample.

    def __init__ (self, ....):
        super(Model1, self).__init__()
        ....
        self.drop = torch.nn.Dropout(p=0.5, inplace=False)

    def forward(self, inputs, lengths):
        pack1 = nn.utils.rnn.pack_padded_sequence(inputs, lengths, batch_first=True)
        out1, self.hidden1 = self.lstm1(pack1, (self.hidden1[0].detach(), self.hidden1[1].detach()))
        out1.data = self.drop(out1.data)

This results in:

 AttributeError: can't set attribute

Perversely, I can make this an inplace operation (again, on the data directly, not the full packed sequence) and it technically works (i.e., it runs) on the CPU, but gives a warning on the GPU that the inplace operation is modifying a needed gradient.  This gives me no confidence that the CPU version works correctly (does it?  Is it missing a warning?  Would not be the first time that I caught PyTorch silently chugging along doing something it should flag a warning for) and in any case GPU support is vital

So:

Are the different behaviors between CPU and GPU expected?
What is the overall correct way to do this on a GPU?
What is the overall correct way to do this on a CPU?


title: Pytorch - Can not slice torchvision MNIST dataset

body:
In Pytorch, when using torchvision's MNIST dataset, we can get a digit as follows:
from torchvision import datasets, transforms
from torch.utils.data import DataLoader, Dataset, TensorDataset

tsfm = transforms.Compose([transforms.Resize((16, 16)),
                           transforms.ToTensor(),
                           transforms.Normalize((0.1307,), (0.3081,))])

mnist_ds = datasets.MNIST(root='../../../_data/mnist',train=True,download=True,
                          transform=tsfm)

digit_12 = mnist_ds[12]

Though it is possible to slice on many datasets, we cannot slice on this one:
>>> digit_12_to_14 = mnist_ds[12:15]
ValueError: Too many dimensions: 3 > 2.

This is due to a Image.fromarray() in the getItem().
Is it possible to use MNIST dataset without using a Dataloader?

PS: The reason why I would like to avoid using Dataloader is that sending batches one at a time to the GPU slows down the training. I prefer to send the entire dataset to the GPU at once. For this I need to have access to the whole transformed dataset.


title: NCCL Connection Failed Using PyTorch Distributed

body:
I am trying to send a PyTorch tensor from one machine to another with torch.distributed. The dist.init_process_group function works properly. However, there is a connection failure in the dist.broadcast function. Here is my code on node 0:

import torch
from torch import distributed as dist
import numpy as np
import os

master_addr = '47.xxx.xxx.xx'
master_port = 10000
world_size = 2
rank = 0
backend = 'nccl'

os.environ['MASTER_ADDR'] = master_addr
os.environ['MASTER_PORT'] = str(master_port)
os.environ['WORLD_SIZE'] = str(world_size)
os.environ['RANK'] = str(rank)
dist.init_process_group(backend, init_method='tcp://47.xxx.xxx.xx:10000', timeout=datetime.timedelta(0, 10), rank=rank, world_size=world_size)
print("Finished initializing process group; backend: %s, rank: %d, "
"world_size: %d" % (backend, rank, world_size))

a = torch.from_numpy(np.random.rand(3, 3)).cuda()
dist.broadcast(tensor=a, src=0)

Here is my code on node 1:

import torch
from torch import distributed as dist
import numpy as np
import os

master_addr = '47.xxx.xxx.xx'
master_port = 10000
world_size = 2
rank = 1
backend = 'nccl'

os.environ['MASTER_ADDR'] = master_addr
os.environ['MASTER_PORT'] = str(master_port)
os.environ['WORLD_SIZE'] = str(world_size)
os.environ['RANK'] = str(rank)
dist.init_process_group(backend, init_method='tcp://47.xxx.xxx.xx:10000', timeout=datetime.timedelta(0, 10), rank=rank, world_size=world_size)
print("Finished initializing process group; backend: %s, rank: %d, "
"world_size: %d" % (backend, rank, world_size))

a = torch.zeros((3,3)).cuda()
dist.broadcast(tensor=a, src=0)

I set NCCL_DEBUG=INFO before running the code. Here is the information I got on Node 1:

iZbp11ufz31riqnssil53cZ:13530:13530 [0] NCCL INFO Bootstrap : Using [0]eth0:192.168.0.181<0>
iZbp11ufz31riqnssil53cZ:13530:13530 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).
iZbp11ufz31riqnssil53cZ:13530:13530 [0] NCCL INFO NET/IB : No device found.
iZbp11ufz31riqnssil53cZ:13530:13530 [0] NCCL INFO NET/Socket : Using [0]eth0:192.168.0.181<0>
iZbp11ufz31riqnssil53cZ:13530:13553 [0] NCCL INFO Setting affinity for GPU 0 to ffff
iZbp11ufz31riqnssil53cZ:13530:13553 [0] NCCL INFO Call to connect returned Connection timed out, retrying
iZbp11ufz31riqnssil53cZ:13530:13553 [0] NCCL INFO Call to connect returned Connection timed out, retrying

iZbp11ufz31riqnssil53cZ:13530:13553 [0] include/socket.h:395 NCCL WARN Connect to 192.168.0.143<59811> failed : Connection timed out
iZbp11ufz31riqnssil53cZ:13530:13553 [0] NCCL INFO bootstrap.cc:100 -> 2
iZbp11ufz31riqnssil53cZ:13530:13553 [0] NCCL INFO bootstrap.cc:326 -> 2
iZbp11ufz31riqnssil53cZ:13530:13553 [0] NCCL INFO init.cc:695 -> 2
iZbp11ufz31riqnssil53cZ:13530:13553 [0] NCCL INFO init.cc:951 -> 2
iZbp11ufz31riqnssil53cZ:13530:13553 [0] NCCL INFO misc/group.cc:69 -> 2 [Async thread]
Traceback (most recent call last):
  File "test_dist_1.py", line 25, in <module>
    dist.broadcast(tensor=a, src=0)
  File "/root/anaconda3/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 806, in broadcast
    work = _default_pg.broadcast([tensor], opts)
RuntimeError: NCCL error in: /tmp/pip-req-build-58y_cjjl/torch/lib/c10d/ProcessGroupNCCL.cpp:290, unhandled system error

And Node 0 seems to stuck in function dist.broadcast:

iZuf6cu11ru7evq9ybagdjZ:13530:13530 [0] NCCL INFO Bootstrap : Using [0]eth0:192.168.0.143<0>
iZuf6cu11ru7evq9ybagdjZ:13530:13530 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).
iZuf6cu11ru7evq9ybagdjZ:13530:13530 [0] NCCL INFO NET/IB : No device found.
iZuf6cu11ru7evq9ybagdjZ:13530:13530 [0] NCCL INFO NET/Socket : Using [0]eth0:192.168.0.143<0>
iZuf6cu11ru7evq9ybagdjZ:13530:13553 [0] NCCL INFO Setting affinity for GPU 0 to ffff

Can anyone help me with this? How can I send the tensor from node 0 to node 1? I would really appreciate any help!


title: Implementing same convolution in Pytorch

body:
I was trying to implement same convolution. It seems (surprisingly) that such a standard layer isn't available out of the box in Pytorch. I hope this helps make sure a layer available. I was needing help generalizing it to use ALL hyper parameters. Right now it only uses kernel size (i.e. variable stride, dialation or whatever else that might affect things isn't part of this. Anyone know how to generalize it with all hyper params of a Convolution (in Conv Nets)?

Code:

def get_cnn(C, out_filters=[14,13,12,3], kernels=[(9,7),(7,7),(5,3),(3,3)]):
    N = len(out_filters)
    print()
    print(f'C = {C}')
    model_layers = OrderedDict()
    in_channels = C
    x = torch.randn(1,C,10,10)
    out = x
    print(f'initial_size = {x.size()}')
    for i in range(1,N):
        print(f'--- building layer = {i}')
        ## create conv layer
        kernel_size = kernels[i]
        out_channels = out_filters[i]
        # to make sure its a same convolution (ignoring stride)
        print(f'in_channels = {in_channels}')
        print(f'out_channels = {out_channels}')
        padding = ((kernels[i][0]-1)//2,(kernels[i][1]-1)//2)
        conv = torch.nn.Conv2d(in_channels, out_channels, kernel_size, padding=padding)
        #conv = torch.nn.Conv2d(in_channels, out_channels, kernel_size)
        model_layers[f'Conv{i}'] = conv
        ## compute dummy data
        out = conv(out)
        print(f'x.size() = {x.size()}')
        print(f'out.size() = {out.size()}')
        ## add activation
        model_layers[f'ReLU{i}'] = torch.nn.ReLU()
        ## so that next layer works
        in_channels = out_channels
    ## make sequential model
    mdl = torch.nn.Sequential(model_layers)
    y = mdl(x)
    return mdl

References:

Does padding ensure that the sizes of input and output of a CNN match?
: https://stats.stackexchange.com/questions/419902/does-padding-ensure-that-the-sizes-of-input-and-output-of-a-cnn-match
arithmetic of convolutions: https://arxiv.org/abs/1603.07285
https://discuss.pytorch.org/t/same-convolution-in-pytorch/19937
https://discuss.pytorch.org/t/how-do-i-make-sure-the-output-of-a-cnn-never-decrease-in-size-using-padding/52103

Full running code:

import torch

from collections import OrderedDict

import random

def do_conv():
    x = torch.randn(1,3,5,6)
    conv = torch.nn.Conv2d(in_channels=3, out_channels=5, kernel_size=(3,3))
    y = conv(x)
    print(y)

def get_cnn(C, out_filters=[14,13,12,3], kernels=[(9,7),(7,7),(5,3),(3,3)]):
    N = len(out_filters)
    print()
    print(f'C = {C}')
    model_layers = OrderedDict()
    in_channels = C
    x = torch.randn(1,C,10,10)
    out = x
    print(f'initial_size = {x.size()}')
    for i in range(1,N):
        print(f'--- building layer = {i}')
        ## create conv layer
        kernel_size = kernels[i]
        out_channels = out_filters[i]
        # to make sure its a same convolution (ignoring stride)
        print(f'in_channels = {in_channels}')
        print(f'out_channels = {out_channels}')
        padding = ((kernels[i][0]-1)//2,(kernels[i][1]-1)//2)
        conv = torch.nn.Conv2d(in_channels, out_channels, kernel_size, padding=padding)
        #conv = torch.nn.Conv2d(in_channels, out_channels, kernel_size)
        model_layers[f'Conv{i}'] = conv
        ## compute dummy data
        out = conv(out)
        print(f'x.size() = {x.size()}')
        print(f'out.size() = {out.size()}')
        ## add activation
        model_layers[f'ReLU{i}'] = torch.nn.ReLU()
        ## so that next layer works
        in_channels = out_channels
    ## make sequential model
    mdl = torch.nn.Sequential(model_layers)
    y = mdl(x)
    return mdl

def get_hardcoded_variable_size_input():
    C = 3
    variable_size_input = [torch.randn(1,C,15,15), torch.randn(1,C,12,16), torch.randn(1,C,15,11)]
    return variable_size_input

def get_variable_size_input(N=3):
    variable_size_input = []
    for i in range(N):
        C,H,W = 3, random.randint(11,20), random.randint(14,19)
        x = torch.randn(C,H,W)
        variable_size_input.append(x)
    return variable_size_input

def any_input_output_equals_input():
    '''
    test that no matter what input, the output of the network is same as the input
    '''
    X_list = get_hardcoded_variable_size_input()
    for i in range(len(X_list)):
        print(f'data point i = {i}')
        ## get data
        x = X_list[i]
        print(x.size())
        ## create cnn
        _,C,_,_ = x.size()
        cnn = get_cnn(C)
        # pass data
        y = cnn(x)
        ## make sure input and output have same size
        assert x.size() == y.size(), f'Error: {x.size()} != {y.size()}'
    ##

if __name__ == '__main__':
    #do_conv()
    print()
    print('start main')
    any_input_output_equals_input()
    print('Done \a')


title: OSError: [WinError 127] The specified procedure could not be found. pytorch

body:
I get this error OSError: [WinError 127] The specified procedure could not be found.
OSError                                   Traceback (most recent call last)
 in
      5 import matplotlib.pyplot as plt
      6
----> 7 import torch
      8 import torch.nn as nn
      9 import torch.optim as optim

~\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\__init__.py in
     79     dlls = glob.glob(os.path.join(th_dll_path, '*.dll'))
     80     for dll in dlls:
---> 81         ctypes.CDLL(dll)
     82
     83

~\AppData\Local\Programs\Python\Python38\lib\ctypes\__init__.py in __init__(self, name, mode, handle, use_errno, use_last_error, winmode)
    371
    372         if handle is None:
--> 373             self._handle = _dlopen(self._name, mode)
    374         else:
    375             self._handle = handle

OSError: [WinError 127] La procédure spécifiée est introuvable

I installed pytorch a few days ago and hit this error for the first time today ... so I don't think it's an installation problem...
I also tried to reinstall pytorch and still get this error
I am using torch 1.5.1+cu101 and python 3.8.5 on windows 10
I have only one version of python installed.
if someone could please help me on this
thanks a lot


title: PyTorch torch.no_grad() versus requires_grad=False

body:
I'm following a PyTorch tutorial which uses the BERT NLP model (feature extractor) from the Huggingface Transformers library. There are two pieces of interrelated code for gradient updates that I don't understand.
(1) torch.no_grad()
The tutorial has a class where the forward() function creates a torch.no_grad() block around a call to the BERT feature extractor, like this:
bert = BertModel.from_pretrained('bert-base-uncased')

class BERTGRUSentiment(nn.Module):

    def __init__(self, bert):
        super().__init__()
        self.bert = bert

    def forward(self, text):
        with torch.no_grad():
            embedded = self.bert(text)[0]

(2) param.requires_grad = False
There is another portion in the same tutorial where the BERT parameters are frozen.
for name, param in model.named_parameters():
    if name.startswith('bert'):
        param.requires_grad = False

When would I need (1) and/or (2)?

If I want to train with a frozen BERT, would I need to enable both?
If I want to train to let BERT be updated, would I need to disable both?

Additionaly, I ran all four combinations and found:
   with torch.no_grad   requires_grad = False  Parameters  Ran
   ------------------   ---------------------  ----------  ---
a. Yes                  Yes                      3M        Successfully
b. Yes                  No                     112M        Successfully
c. No                   Yes                      3M        Successfully
d. No                   No                     112M        CUDA out of memory

Can someone please explain what's going on? Why am I getting CUDA out of memory for (d) but not (b)? Both have 112M learnable parameters.


title: BrokenPipeError: [Errno 32] Broken pipe on Windows and CUDA 10

body:
I am able to run my program on system with CUDA 9.2, Quadro M5000 (5), and python 3.7.0 (anaconda) while the same program giving me an error on another system with the CUDA 10.0, GeForce GTX 1080 Ti (6), and python 3.7.1

here is the error

 ForkingPickler(file, protocol).dump(obj)

 BrokenPipeError: [Errno 32] Broken pipe

the whole code wrap in a function named train_valid_model() and just add the main guard in the script:

def train_valid_model():
# complete code here

 if __name__=='__main__':
     train_valid_model()

It only works when num_workers=0 and if I set it to 1 or 2 I will get this error. Any idea why I am getting this error while the program is same just the CUDA version and python version are different?


title: pytorch custom dataset: DataLoader returns a list of tensors rather than tensor of a list

body:
import torch

class Custom_Dataset(torch.utils.data.dataset.Dataset):
    def __init__(self, _dataset):
        self.dataset = _dataset

    def __getitem__(self, index):
        example, target = self.dataset[index]
        return example, target

    def __len__(self):
        return len(self.dataset)

train_data = [([1, 3, 5], 0),
              ([2, 4, 6], 1)]
train_loader = torch.utils.data.DataLoader(dataset=Custom_Dataset(train_data),
                                           batch_size=1,
                                           shuffle=False)

for inputs, targets in train_loader:
    print(inputs)
    print(targets)

I'm defining my training data as [([1, 3, 5], 0), ([2, 4, 6], 1)]: input([1, 3, 5]) paired target (0).

But when I fetch data from data loader, it becomes:

[tensor([1]), tensor([3]), tensor([5])]
tensor([0])

How do I get instead:

tensor([[1],
        [3],
        [5]])
tensor([0])

?

I know torch.stack can do the trick, but can I convert it in my custom dataset class?