title: Pytorch says that CUDA is not available

body:
I'm trying to run Pytorch on a laptop that I have. It's an older model but it does have an Nvidia graphics card. I realize it is probably not going to be sufficient for real machine learning but I am trying to do it so I can learn the process of getting CUDA installed.

I have followed the steps on the installation guide for Ubuntu 18.04 (my specific distribution is Xubuntu).

My graphics card is a GeForce 845M, verified by lspci | grep nvidia:

01:00.0 3D controller: NVIDIA Corporation GM107M [GeForce 845M] (rev a2)
01:00.1 Audio device: NVIDIA Corporation Device 0fbc (rev a1)

I also have gcc 7.5 installed, verified by gcc --version

gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

And I have the correct headers installed, verified by trying to install them with sudo apt-get install linux-headers-$(uname -r):

Reading package lists... Done
Building dependency tree
Reading state information... Done
linux-headers-4.15.0-106-generic is already the newest version (4.15.0-106.107).

I then followed the installation instructions using a local .deb for version 10.1.

Npw, when I run nvidia-smi, I get:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.87.00    Driver Version: 418.87.00    CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce 845M        On   | 00000000:01:00.0 Off |                  N/A |
| N/A   40C    P0    N/A /  N/A |     88MiB /  2004MiB |      1%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0       982      G   /usr/lib/xorg/Xorg                            87MiB |
+-----------------------------------------------------------------------------+

and I run nvcc -V I get:

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243

I then performed the post-installation instructions from section 6.1, and so as a result, echo $PATH looks like this:

/home/isaek/anaconda3/envs/stylegan2_pytorch/bin:/home/isaek/anaconda3/bin:/home/isaek/anaconda3/condabin:/usr/local/cuda-10.1/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin

echo $LD_LIBRARY_PATH looks like this:

/usr/local/cuda-10.1/lib64

and my /etc/udev/rules.d/40-vm-hotadd.rules file looks like this:

# On Hyper-V and Xen Virtual Machines we want to add memory and cpus as soon as they appear
ATTR{[dmi/id]sys_vendor}=="Microsoft Corporation", ATTR{[dmi/id]product_name}=="Virtual Machine", GOTO="vm_hotadd_apply"
ATTR{[dmi/id]sys_vendor}=="Xen", GOTO="vm_hotadd_apply"
GOTO="vm_hotadd_end"

LABEL="vm_hotadd_apply"

# Memory hotadd request

# CPU hotadd request
SUBSYSTEM=="cpu", ACTION=="add", DEVPATH=="/devices/system/cpu/cpu[0-9]*", TEST=="online", ATTR{online}="1"

LABEL="vm_hotadd_end"

After all of this, I even compiled and ran the samples. ./deviceQuery returns:

./deviceQuery Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 1 CUDA Capable device(s)

Device 0: "GeForce 845M"
  CUDA Driver Version / Runtime Version          10.1 / 10.1
  CUDA Capability Major/Minor version number:    5.0
  Total amount of global memory:                 2004 MBytes (2101870592 bytes)
  ( 4) Multiprocessors, (128) CUDA Cores/MP:     512 CUDA Cores
  GPU Max Clock rate:                            863 MHz (0.86 GHz)
  Memory Clock rate:                             1001 Mhz
  Memory Bus Width:                              64-bit
  L2 Cache Size:                                 1048576 bytes
  Maximum Texture Dimension Size (x,y,z)         1D=(65536), 2D=(65536, 65536), 3D=(4096, 4096, 4096)
  Maximum Layered 1D Texture Size, (num) layers  1D=(16384), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(16384, 16384), 2048 layers
  Total amount of constant memory:               65536 bytes
  Total amount of shared memory per block:       49152 bytes
  Total number of registers available per block: 65536
  Warp size:                                     32
  Maximum number of threads per multiprocessor:  2048
  Maximum number of threads per block:           1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
  Maximum memory pitch:                          2147483647 bytes
  Texture alignment:                             512 bytes
  Concurrent copy and kernel execution:          Yes with 1 copy engine(s)
  Run time limit on kernels:                     Yes
  Integrated GPU sharing Host Memory:            No
  Support host page-locked memory mapping:       Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Disabled
  Device supports Unified Addressing (UVA):      Yes
  Device supports Compute Preemption:            No
  Supports Cooperative Kernel Launch:            No
  Supports MultiDevice Co-op Kernel Launch:      No
  Device PCI Domain ID / Bus ID / location ID:   0 / 1 / 0
  Compute Mode:
     < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 10.1, CUDA Runtime Version = 10.1, NumDevs = 1
Result = PASS

and ./bandwidthTest returns:

[CUDA Bandwidth Test] - Starting...
Running on...

 Device 0: GeForce 845M
 Quick Mode

 Host to Device Bandwidth, 1 Device(s)
 PINNED Memory Transfers
   Transfer Size (Bytes)    Bandwidth(GB/s)
   32000000         11.7

 Device to Host Bandwidth, 1 Device(s)
 PINNED Memory Transfers
   Transfer Size (Bytes)    Bandwidth(GB/s)
   32000000         11.8

 Device to Device Bandwidth, 1 Device(s)
 PINNED Memory Transfers
   Transfer Size (Bytes)    Bandwidth(GB/s)
   32000000         14.5

Result = PASS

NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is enabled.

But after all of this, this Python snippet (in a conda environment with all dependencies installed):

import torch
torch.cuda.is_available()

returns False

Does anybody have any idea about how to resolve this? I've tried to add /usr/local/cuda-10.1/bin to etc/environment like this:

PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games"
PATH=$PATH:/usr/local/cuda-10.1/bin

And restarting the terminal, but that didn't fix it. I really don't know what else to try.

EDIT - Results of collect_env for @kHarshit

Collecting environment information...
PyTorch version: 1.5.0
Is debug build: No
CUDA used to build PyTorch: 10.2

OS: Ubuntu 18.04.4 LTS
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
CMake version: Could not collect

Python version: 3.6
Is CUDA available: No
CUDA runtime version: 10.1.243
GPU models and configuration: GPU 0: GeForce 845M
Nvidia driver version: 418.87.00
cuDNN version: Could not collect

Versions of relevant libraries:
[pip] numpy==1.18.5
[pip] pytorch-ranger==0.1.1
[pip] stylegan2-pytorch==0.12.0
[pip] torch==1.5.0
[pip] torch-optimizer==0.0.1a12
[pip] torchvision==0.6.0
[pip] vector-quantize-pytorch==0.0.2
[conda] numpy                     1.18.5                   pypi_0    pypi
[conda] pytorch-ranger            0.1.1                    pypi_0    pypi
[conda] stylegan2-pytorch         0.12.0                   pypi_0    pypi
[conda] torch                     1.5.0                    pypi_0    pypi
[conda] torch-optimizer           0.0.1a12                 pypi_0    pypi
[conda] torchvision               0.6.0                    pypi_0    pypi
[conda] vector-quantize-pytorch   0.0.2                    pypi_0    pypi


title: Resize PyTorch Tensor

body:
I am currently using the tensor.resize() function to resize a tensor to a new shape t = t.resize(1, 2, 3).

This gives me a deprecation warning:

  non-inplace resize is deprecated

Hence, I wanted to switch over to the tensor.resize_() function, which seems to be the appropriate in-place replacement. However, this leaves me with an

  cannot resize variables that require grad

error.
I can fall back to

from torch.autograd._functions import Resize
Resize.apply(t, (1, 2, 3))

which is what tensor.resize() does in order to avoid the deprecation warning.
This doesn't seem like an appropriate solution but rather a hack to me.
How do I correctly make use of tensor.resize_() in this case?


title: Pytorch - RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed

body:
I keep running into this error:

  RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.

I had searched in Pytorch forum, but still can’t find out what I have done wrong in my custom loss function. My model is nn.GRU, and here is my custom loss function:

def _loss(outputs, session, items):  # `items` is a dict() contains embedding of all items
    def f(output, target):
        pos = torch.from_numpy(np.array([items[target["click"]]])).float()
        neg = torch.from_numpy(np.array([items[idx] for idx in target["suggest_list"] if idx != target["click"]])).float()
        if USE_CUDA:
            pos, neg = pos.cuda(), neg.cuda()
        pos, neg = Variable(pos), Variable(neg)

        pos = F.cosine_similarity(output, pos)
        if neg.size()[0] == 0:
            return torch.mean(F.logsigmoid(pos))
        neg = F.cosine_similarity(output.expand_as(neg), neg)

        return torch.mean(F.logsigmoid(pos - neg))

    loss = map(f, outputs, session)
return -torch.mean(torch.cat(loss))

Training code:

    # zero the parameter gradients
    model.zero_grad()

    # forward + backward + optimize
    outputs, hidden = model(inputs, hidden)
    loss = _loss(outputs, session, items)
    acc_loss += loss.data[0]

    loss.backward()
    # Add parameters' gradients to their values, multiplied by learning rate
    for p in model.parameters():
        p.data.add_(-learning_rate, p.grad.data)


title: How to use multiple GPUs in pytorch?

body:
I am learning pytorch and follow this tutorial: https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html

I use this command to use a GPU.

device = torch.device("**cuda:0**" if torch.cuda.is_available() else "cpu")

But, I want to use two GPUs in jupyter, like this:

device = torch.device("**cuda:0,1**" if torch.cuda.is_available() else "cpu")

Of course, this is wrong.
So, How can I do this?


title: What's the difference between "hidden" and "output" in PyTorch LSTM?

body:
I'm having trouble understanding the documentation for PyTorch's LSTM module (and also RNN and GRU, which are similar). Regarding the outputs, it says:

  Outputs: output, (h_n, c_n)

  output (seq_len, batch, hidden_size * num_directions): tensor containing the output features (h_t) from the last layer of the RNN, for each t. If a torch.nn.utils.rnn.PackedSequence has been given as the input, the output will also be a packed sequence.
  h_n (num_layers * num_directions, batch, hidden_size): tensor containing the hidden state for t=seq_len
  c_n (num_layers * num_directions, batch, hidden_size): tensor containing the cell state for t=seq_len

It seems that the variables output and h_n both give the values of the hidden state. Does h_n just redundantly provide the last time step that's already included in output, or is there something more to it than that?


title: Can't install pytorch with pip on Windows

body:
I'm trying to install Pytorch with Windows and I'm using the commands of the official site
https://pytorch.org/get-started/locally/

pip3 install torch==1.2.0 torchvision==0.4.0 -f https://download.pytorch.org/whl/torch_stable.html

This is the command if I choose Windows, Cuda 10.0, and Python 3.7
But if I run this I get the error message:

ERROR: Could not find a version that satisfies the requirement torch==1.2.0 (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)
ERROR: No matching distribution found for torch==1.2.0

So why does this happen?
My pip is version 19.2 and I am in a newly installed python 3.7 environment


title: How to use the BCELoss in PyTorch?

body: I want to write a simple autoencoder in PyTorch and use BCELoss, however, I get NaN out, since it expects the targets to be between 0 and 1. Could someone post a simple use case of BCELoss?


title: PyTorch : error message "torch has no [...] member"

body:
Good evening,
I have just installed PyTorch 0.4.0 and I'm trying to carry out the first tutorial "What is PyTorch?"
I have written a Tutorial.py file which I try to execute with Visual Studio Code

Here is the code :

from __future__ import print_function
import torch

print (torch.__version__)

x = x = torch.rand(5, 3)
print(x)

Unfortunately, when I try to debug it, i have an error message :
"torch has no rand member"

This is true with any member function of torch I may try

Can anybody help me please?


title: Pytorch AssertionError: Torch not compiled with CUDA enabled

body:
I am trying to run code from this repo. I have disabled cuda by changing lines 39/40 in main.py from

parser.add_argument('--type', default='torch.cuda.FloatTensor', help='type of tensor - e.g torch.cuda.HalfTensor')

to

parser.add_argument('--type', default='torch.FloatTensor', help='type of tensor - e.g torch.HalfTensor')

Despite this, running the code gives me the following exception:

Traceback (most recent call last):
  File "main.py", line 190, in <module>
    main()
  File "main.py", line 178, in main
    model, train_data, training=True, optimizer=optimizer)
  File "main.py", line 135, in forward
    for i, (imgs, (captions, lengths)) in enumerate(data):
  File "/Users/lakshay/anaconda/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 201, in __next__
    return self._process_next_batch(batch)
  File "/Users/lakshay/anaconda/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 221, in _process_next_batch
    raise batch.exc_type(batch.exc_msg)
AssertionError: Traceback (most recent call last):
  File "/Users/lakshay/anaconda/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 62, in _pin_memory_loop
    batch = pin_memory_batch(batch)
  File "/Users/lakshay/anaconda/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 123, in pin_memory_batch
    return [pin_memory_batch(sample) for sample in batch]
  File "/Users/lakshay/anaconda/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 123, in <listcomp>
    return [pin_memory_batch(sample) for sample in batch]
  File "/Users/lakshay/anaconda/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 117, in pin_memory_batch
    return batch.pin_memory()
  File "/Users/lakshay/anaconda/lib/python3.6/site-packages/torch/tensor.py", line 82, in pin_memory
    return type(self)().set_(storage.pin_memory()).view_as(self)
  File "/Users/lakshay/anaconda/lib/python3.6/site-packages/torch/storage.py", line 83, in pin_memory
    allocator = torch.cuda._host_allocator()
  File "/Users/lakshay/anaconda/lib/python3.6/site-packages/torch/cuda/__init__.py", line 220, in _host_allocator
    _lazy_init()
  File "/Users/lakshay/anaconda/lib/python3.6/site-packages/torch/cuda/__init__.py", line 84, in _lazy_init
    _check_driver()
  File "/Users/lakshay/anaconda/lib/python3.6/site-packages/torch/cuda/__init__.py", line 51, in _check_driver
    raise AssertionError("Torch not compiled with CUDA enabled")
AssertionError: Torch not compiled with CUDA enabled

Spent some time looking through the issues in the Pytorch github, to no avail. Help, please?


title: version `GLIBC_2.28' not found

body:
I'm trying to install PyTorch on ARMv7(32-bit) architecture but PyTorch doesn’t have official ARMv7 builds so i tried this unofficial build.

It installed successfully but when I import torch I get the following error

>>import torch
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/site-packages/torch/__init__.py", line 81, in <module>
    from torch._C import *
ImportError: /lib/arm-linux-gnueabihf/libc.so.6: version `GLIBC_2.28' not found (required by /usr/local/lib/python3.7/site-packages/torch/lib/libtorch_python.so)

I tried the following

sudo apt-get update
sudo apt-get install libc6

but it seams like that i have the newest version of libc6

Reading package lists... Done
Building dependency tree
Reading state information... Done
libc6 is already the newest version (2.23-0ubuntu11).
The following packages were automatically installed and are no longer required:
  busybox-initramfs cpio initramfs-tools initramfs-tools-bin initramfs-tools-core klibc-utils libdbusmenu-gtk4 libklibc
  libllvm3.8 libmircommon5 linux-base
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 10 not upgraded.

Here is my GLIBCXX and GLIBC versions that i have:

strings /usr/lib/arm-linux-gnueabihf/libstdc++.so.6 | grep GLIBC
GLIBCXX_3.4
GLIBCXX_3.4.1
GLIBCXX_3.4.2
GLIBCXX_3.4.3
GLIBCXX_3.4.4
GLIBCXX_3.4.5
GLIBCXX_3.4.6
GLIBCXX_3.4.7
GLIBCXX_3.4.8
GLIBCXX_3.4.9
GLIBCXX_3.4.10
GLIBCXX_3.4.11
GLIBCXX_3.4.12
GLIBCXX_3.4.13
GLIBCXX_3.4.14
GLIBCXX_3.4.15
GLIBCXX_3.4.16
GLIBCXX_3.4.17
GLIBCXX_3.4.18
GLIBCXX_3.4.19
GLIBCXX_3.4.20
GLIBCXX_3.4.21
GLIBCXX_3.4.22
GLIBCXX_3.4.23
GLIBCXX_3.4.24
GLIBCXX_3.4.25
GLIBCXX_3.4.26
GLIBCXX_3.4.27
GLIBCXX_3.4.28
GLIBC_2.4
GLIBC_2.6
GLIBC_2.18
GLIBC_2.16
GLIBC_2.17

Ldd version:

ldd --version

ldd (Ubuntu GLIBC 2.23-0ubuntu11) 2.23
Copyright (C) 2016 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.

My OS:

cat /etc/os-release
NAME="Ubuntu"
VERSION="16.04.6 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.6 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

So is it possible to install GLIBC_2.28 on my machine?


title: Pytorch: how to add L1 regularizer to activations?

body:
I would like to add the L1 regularizer to the activations output from a ReLU.
More generally, how does one add a regularizer only to a particular layer in the network?

Related material:

This similar post refers to adding L2 regularization, but it appears to add the regularization penalty to all layers of the network.

nn.modules.loss.L1Loss() seems relevant, but I do not yet understand how to use this.

The legacy module L1Penalty seems relevant also, but why has it been deprecated?


title: Pytorch doesn't support one-hot vector?

body:
I am very confused by how Pytorch deals with one-hot vectors. In this tutorial, the neural network will generate a one-hot vector as its output. As far as I understand, the schematic structure of the neural network in the tutorial should be like:

However, the labels are not in one-hot vector format. I get the following size

print(labels.size())
print(outputs.size())

output>>> torch.Size([4])
output>>> torch.Size([4, 10])

Miraculously, I they pass the outputs and labels to criterion=CrossEntropyLoss(), there's no error at all.

loss = criterion(outputs, labels) # How come it has no error?

My hypothesis:

Maybe pytorch automatically convert the labels to one-hot vector form. So, I try to convert labels to one-hot vector before passing it to the loss function.

def to_one_hot_vector(num_class, label):
    b = np.zeros((label.shape[0], num_class))
    b[np.arange(label.shape[0]), label] = 1

    return b

labels_one_hot = to_one_hot_vector(10,labels)
labels_one_hot = torch.Tensor(labels_one_hot)
labels_one_hot = labels_one_hot.type(torch.LongTensor)

loss = criterion(outputs, labels_one_hot) # Now it gives me error

However, I got the following error

  RuntimeError: multi-target not supported at
  /opt/pytorch/pytorch/aten/src/THCUNN/generic/ClassNLLCriterion.cu:15

So, one-hot vectors are not supported in Pytorch? How does Pytorch calculates the cross entropy for the two tensor outputs = [1,0,0],[0,0,1] and labels = [0,2] ? It doesn't make sense to me at all at the moment.


title: What does -1 mean in pytorch view?

body:
As the question says, what does -1 do in pytorch view?
>>> a = torch.arange(1, 17)
>>> a
tensor([  1.,   2.,   3.,   4.,   5.,   6.,   7.,   8.,   9.,  10.,
         11.,  12.,  13.,  14.,  15.,  16.])

>>> a.view(1,-1)
tensor([[  1.,   2.,   3.,   4.,   5.,   6.,   7.,   8.,   9.,  10.,
          11.,  12.,  13.,  14.,  15.,  16.]])

>>> a.view(-1,1)
tensor([[  1.],
        [  2.],
        [  3.],
        [  4.],
        [  5.],
        [  6.],
        [  7.],
        [  8.],
        [  9.],
        [ 10.],
        [ 11.],
        [ 12.],
        [ 13.],
        [ 14.],
        [ 15.],
        [ 16.]])

Does it (-1) generate additional dimension?
Does it behave the same as numpy reshape -1?


title: Can't import torch in jupyter notebook

body:
System: macOS 10.13.6
Python: 3.7
Anaconda3

I have trouble when import torch in jupyter notebook.

ModuleNotFoundError: No module named 'torch'

Here is how I install pytorch:

conda install pytorch torchvision -c pytorch

I've checked PyTorch is installed in my anaconda environment:

When I command python3 in my terminal and import torch, it works. But not work in jupyter notebook

I've tried:

conda update conda
conda install mkl=2018

But still the same error.

Can anyone help me with this? Thanks!!


title: How does the "number of workers" parameter in PyTorch dataloader actually work?

body:
If num_workers is 2, Does that mean that it will put 2 batches in the RAM and send 1 of them to the GPU or Does it put 3 batches in the RAM then sends 1 of them to the GPU?
What does actually happen when the number of workers is higher than the number of CPU cores? I tried it and it worked fine but How does it work? (I thought that the maximum number of workers I can choose is the number of cores).
If I set num_workers to 3 and during the training there were no batches in the memory for the GPU, Does the main process waits for its workers to read the batches or Does it read a single batch (without waiting for the workers)?


title: How to convert a list of strings into a tensor in pytorch?

body:
I am working on classification problem in which I have a list of strings as class labels and I want to convert them into a tensor. So far I have tried converting the list of strings into a numpy array using the np.array function provided by the numpy module.

truth = torch.from_numpy(np.array(truths))

but I am getting the following error.

RuntimeError: can't convert a given np.ndarray to a tensor - it has an invalid type. The only supported types are: double, float, int64, int32, and uint8.

Can anybody suggest an alternative approach? Thanks


title: RuntimeError: Expected object of backend CUDA but got backend CPU for argument: ret = torch.addmm(torch.jit._unwrap_optional(bias), input, weight.t())

body:
When the forward function of my neural network (after the training phase is completed) is being executed, I'm experiencing RuntimeError: Expected object of backend CUDA but got backend CPU for argument #4 'mat1'. The error trace indicates the error happens due to the call of output = self.layer1(x) command. I have tried to move all the data of the tensors to my GPU. It seems I miss something to be moved as well.

Here is the code I have tried:

use_cuda = torch.cuda.is_available()
device = torch.device('cuda:0' if use_cuda else 'cpu')

class NeuralNet(nn.Module):

    def __init__(self, input_size, hidden_size, output_size):
        super(NeuralNet, self).__init__()
        self.layer1 = nn.Linear(input_size, hidden_size).cuda(device)
        self.layer2 = nn.Linear(hidden_size, output_size).cuda(device)
        self.relu = nn.ReLU().cuda(device)

    def forward(self, x):
        x.cuda(device)
        output = self.layer1(x)  # throws the error
        output = self.relu(output)
        output = self.layer2(output)
        return output

def main():
    transform = transforms.Compose([
        transforms.ToTensor()
    ])

    mnist_trainset = datasets.MNIST(root='D:\\MNIST', train=True, download=False, transform=transform)
    mnist_testset = datasets.MNIST(root='D:\\MNIST', train=False, download=False, transform=transform)

    train_loader = DataLoader(dataset=mnist_trainset, batch_size=100, shuffle=True)
    test_loader = DataLoader(dataset=mnist_testset, batch_size=100, shuffle=False)

    input_size = 784
    hidden_size = 500
    output_size = 10
    num_epochs = 5

    learning_rate = 0.001

    model = NeuralNet(input_size, hidden_size, output_size)
    model.cuda(device)

    lossFunction = nn.CrossEntropyLoss()
    lossFunction.cuda(device)
    optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

    losses_in_epochs = []
    total_step = len(train_loader)
    for epoch in range(num_epochs):
        for i, (images, labels) in enumerate(train_loader):
            images = images.to(device)
            labels = labels.to(device)
            images = images.reshape(-1, 28 * 28)

            out = model(images)
            loss = lossFunction(out, labels)

            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

            if (i + 1) % 100 == 0:
                print('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}'.format(epoch + 1, num_epochs, i + 1, total_step,
                                                                         loss.item()))

            if (i % 600) == 0:
                losses_in_epochs.append(loss.item())

    with torch.no_grad():
        correct = 0
        total = 0
        for images, labels in test_loader:
            images = images.reshape(-1, 28 * 28)
            out = model(images)
            _, predicted = torch.max(out.data, 1)
            total += labels.size(0)
            correct += (predicted == labels).sum().item()
            print('Accuracy of the network on the 10000 test images: {} %'.format(100 * correct / total))

if __name__ == '__main__':
    main()

The software stack:

Python 3.7.1
torch 1.0.1 (with Cuda 9.0)
Windows 10 64-bit


title: How to resolve runtime error due to size mismatch in PyTorch?

body:
I am trying to implement a simple autoencoder using PyTorch. My dataset consists of 256 x 256 x 3 images. I have built a torch.utils.data.dataloader.DataLoader object which has the image stored as tensor. When I run the autoencoder, I get a runtime error:

  size mismatch, m1: [76800 x 256], m2: [784 x 128] at
  /Users/soumith/minicondabuild3/conda-bld/pytorch_1518371252923/work/torch/lib/TH/generic/THTensorMath.c:1434

These are my hyperparameters:

batch_size=100,
learning_rate = 1e-3,
num_epochs = 100

Following is the architecture of my auto-encoder:

class autoencoder(nn.Module):
    def __init__(self):
        super(autoencoder, self).__init__()
        self.encoder = nn.Sequential(
            nn.Linear(3*256*256, 128),
            nn.ReLU(),
            nn.Linear(128, 64),
            nn.ReLU(True),
            nn.Linear(64, 12),
            nn.ReLU(True),
            nn.Linear(12, 3))

        self.decoder = nn.Sequential(
            nn.Linear(3, 12),
            nn.ReLU(True),
            nn.Linear(12, 64),
            nn.ReLU(True),
            nn.Linear(64, 128),
            nn.Linear(128, 3*256*256),
            nn.ReLU())

def forward(self, x):
    x = self.encoder(x)
    #x = self.decoder(x)
    return x

This is the code I used to run the model:

for epoch in range(num_epochs):
for data in dataloader:
    img = data['image']
    img = Variable(img)
    # ===================forward=====================
    output = model(img)
    loss = criterion(output, img)
    # ===================backward====================
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
# ===================log========================
print('epoch [{}/{}], loss:{:.4f}'
      .format(epoch+1, num_epochs, loss.data[0]))
if epoch % 10 == 0:
    pic = show_img(output.cpu().data)
    save_image(pic, './dc_img/image_{}.jpg'.format(epoch))


title: How to remove the last FC layer from a ResNet model in PyTorch?

body:
I am using a ResNet152 model from PyTorch. I'd like to strip off the last FC layer from the model. Here's my code:

from torchvision import datasets, transforms, models
model = models.resnet152(pretrained=True)
print(model)

When I print the model, the last few lines look like this:

    (2):  Bottleneck(
      (conv1):  Conv2d(2048,  512,  kernel_size=(1,  1),  stride=(1,  1),  bias=False)
      (bn1):  BatchNorm2d(512,  eps=1e-05,  momentum=0.1,  affine=True,  track_running_stats=True)
      (conv2):  Conv2d(512,  512,  kernel_size=(3,  3),  stride=(1,  1),  padding=(1,  1),  bias=False)
      (bn2):  BatchNorm2d(512,  eps=1e-05,  momentum=0.1,  affine=True,  track_running_stats=True)
      (conv3):  Conv2d(512,  2048,  kernel_size=(1,  1),  stride=(1,  1),  bias=False)
      (bn3):  BatchNorm2d(2048,  eps=1e-05,  momentum=0.1,  affine=True,  track_running_stats=True)
      (relu):  ReLU(inplace)
    )
  )
  (avgpool):  AvgPool2d(kernel_size=7,  stride=1,  padding=0)
  (fc):  Linear(in_features=2048,  out_features=1000,  bias=True)
)

I want to remove that last fc layer from the model.

I found an answer here on SO (How to convert pretrained FC layers to CONV layers in Pytorch), where mexmex seems to provide the answer I'm looking for:

list(model.modules()) # to inspect the modules of your model
my_model = nn.Sequential(*list(model.modules())[:-1]) # strips off last linear layer

So I added those lines to my code like this:

model = models.resnet152(pretrained=True)
list(model.modules()) # to inspect the modules of your model
my_model = nn.Sequential(*list(model.modules())[:-1]) # strips off last linear layer
print(my_model)

But this code doesn't work as advertised -- as least not for me. The rest of this post is a detailed explanation of why that answer doesn't work so this question doesn't get closed as a duplicate.

First, the printed model is nearly 5x larger than before. I see the same model as before, but followed by what appears to be a repeat of the model, but perhaps flattened.

    (2): Bottleneck(
      (conv1): Conv2d(2048, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv3): Conv2d(512, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn3): BatchNorm2d(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace)
    )
  )
  (avgpool): AvgPool2d(kernel_size=7, stride=1, padding=0)
  (fc): Linear(in_features=2048, out_features=1000, bias=True)
)
(1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
(2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(3): ReLU(inplace)
(4): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)
(5): Sequential(
  . . . this goes on for ~1600 more lines . . .
  (415): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (416): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
  (417): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (418): Conv2d(512, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False)
  (419): BatchNorm2d(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (420): ReLU(inplace)
  (421): AvgPool2d(kernel_size=7, stride=1, padding=0)
)

Second, the fc layer is still there -- and the Conv2D layer after it looks just like the first layer of ResNet152.

Third, if I try to invoke my_model.forward(), pytorch complains about a size mismatch. It expects size [1, 3, 224, 224], but the input was [1, 1000]. So it looks like a copy of the entire model (minus the fc layer) is getting appended to the original model.

Bottom line, the only answer I found on SO doesn't actually work.


title: Get single random example from PyTorch DataLoader

body:
How do I get a single random example from a PyTorch DataLoader?
If my DataLoader gives minbatches of multiple images and labels, how do I get a single random image and label?
Note that I don't want a single image and label per minibatch, I want a total of one example.


title: How can I install torchtext?

body:
I have PyTorch installed in my machine but whenever I try to do the following-

from torchtext import data
from torchtext import datasets

I get the following error.

ImportError: No module named 'torchtext'

How can I install torchtext?


title: Can I slice tensors with logical indexing or lists of indices?

body:
I'm trying to slice a PyTorch tensor using a logical index on the columns. I want the columns that correspond to a 1 value in the index vector. Both slicing and logical indexing are possible, but are they possible together? If so, how? My attempt keeps throwing the unhelpful error

TypeError: indexing a tensor with an object of type ByteTensor. The
only supported types are integers, slices, numpy scalars and
torch.LongTensor or torch.ByteTensor as the only argument.

MCVE
Desired Output
import torch

C = torch.LongTensor([[1, 3], [4, 6]])
# 1 3
# 4 6

Logical indexing on the columns only:
A_log = torch.ByteTensor([1, 0, 1]) # the logical index
B = torch.LongTensor([[1, 2, 3], [4, 5, 6]])
C = B[:, A_log] # Throws error

If the vectors are the same size, logical indexing works:
B_truncated = torch.LongTensor([1, 2, 3])
C = B_truncated[A_log]

And I can get the desired result by repeating the logical index so that it has the same size as the tensor I am indexing, but then I also have to reshape the output.
C = B[A_log.repeat(2, 1)] # [torch.LongTensor of size 4]
C = C.resize_(2, 2)

I also tried using a list of indices:
A_idx = torch.LongTensor([0, 2]) # the index vector
C = B[:, A_idx] # Throws error

If I want contiguous ranges of indices, slicing works:
C = B[:, 1:2]


title: How to use AMD GPU for fastai/pytorch?

body:
I'm using a laptop which has Intel Corporation HD Graphics 5500 (rev 09), and AMD Radeon r5 m255 graphics card.
Does anyone know how to it set up for Deep Learning, specifically fastai/Pytorch?


title: PyTorch - How to use "toPILImage" correctly

body:
I would like to know, whether I used toPILImage from torchvision correctly. I want to use it, to see how the images look after initial image transformations are applied to the dataset.

When I use it like in the code below, the image that comes up has weird colors like this one. The original image is a regular RGB image.

This is my code:

import os
import torch
from PIL import Image, ImageFont, ImageDraw
import torch.utils.data as data
import torchvision
from torchvision import transforms
import matplotlib.pyplot as plt

# Image transformations
normalize = transforms.Normalize(
    mean=[0.485, 0.456, 0.406],
    std=[0.229, 0.224, 0.225]
    )
transform_img = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(256),
    transforms.ToTensor(),
    normalize ])

train_data = torchvision.datasets.ImageFolder(
    root='./train_cl/',
    transform=transform_img
    )
test_data = torchvision.datasets.ImageFolder(
    root='./test_named_cl/',
    transform=transform_img
    )

train_data_loader = data.DataLoader(train_data,
    batch_size=4,
    shuffle=True,
    num_workers=4) #num_workers=args.nThreads)

test_data_loader = data.DataLoader(test_data,
    batch_size=32,
    shuffle=False,
    num_workers=4)

# Open Image from dataset:
to_pil_image = transforms.ToPILImage()
my_img, _ = train_data[248]
results = to_pil_image(my_img)
results.show()

Edit:

I had to use .data on the Torch Variable to get the tensor.
Also I needed to rescale the numpy array before transposing. I found a working solution here, but it doesn't always work well. How can I do this better?

for i, data in enumerate(train_data_loader, 0):
    img, labels = data
    img = Variable(img)
    break

image = img.data.cpu().numpy()[0]

# This worked for rescaling:
image = (1/(2*2.25)) * image + 0.5

# Both of these didn't work:
# image /= (image.max()/255.0)
# image *= (255.0/image.max())

image = np.transpose(image, (1,2,0))
plt.imshow(image)
plt.show()


title: KL Divergence for two probability distributions in PyTorch

body: I have two probability distributions. How should I find the KL-divergence between them in PyTorch? The regular cross entropy only accepts integer labels.


title: Pytorch inputs for nn.CrossEntropyLoss()

body:
I am trying to perform a Logistic Regression in PyTorch on a simple 0,1 labelled dataset. The criterion or loss is defined as: criterion = nn.CrossEntropyLoss(). The model is: model = LogisticRegression(1,2)

I have a data point which is a pair: dat = (-3.5, 0), the first element is the datapoint and the second is the corresponding label.
Then I convert the first element of the input to a tensor: tensor_input = torch.Tensor([dat[0]]).
Then I apply the model to the tensor_input: outputs = model(tensor_input).
Then I convert the label to a tensor: tensor_label = torch.Tensor([dat[1]]).
Now, when I try to do this, the thing breaks: loss = criterion(outputs, tensor_label). It gives and error: RuntimeError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

import torch
import torch.nn as nn

class LogisticRegression(nn.Module):
    def __init__(self, input_size, num_classes):
        super(LogisticRegression, self).__init__()
        self.linear = nn.Linear(input_size, num_classes)

    def forward(self, x):
        out = self.linear(x)
        return out

model = LogisticRegression(1,2)
criterion = nn.CrossEntropyLoss()
dat = (-3.5,0)
tensor_input = torch.Tensor([dat[0]])
outputs = binary_model(tensor_input)
tensor_label = torch.Tensor([dat[1]])
loss = criterion(outputs, tensor_label)

I can't for the life of me figure it out.


title: Meaning of parameters in torch.nn.conv2d

body:
In the fastai cutting edge deep learning for coders course lecture 7.

 self.conv1 = nn.Conv2d(3,10,kernel_size = 5,stride=1,padding=2)

Does 10 there mean the number of filters or the number activations the filter will give?


title: How can I generate and display a grid of images in PyTorch with plt.imshow and torchvision.utils.make_grid?

body:
I am trying to understand how torchvision interacts with mathplotlib to produce a grid of images. It's easy to generate images and display them iteratively:

import torch
import torchvision
import matplotlib.pyplot as plt

w = torch.randn(10,3,640,640)
for i in range (0,10):
    z = w[i]
    plt.imshow(z.permute(1,2,0))
    plt.show()

However, displaying these images in a grid does not seem to be as straightforward.

w = torch.randn(10,3,640,640)
grid = torchvision.utils.make_grid(w, nrow=5)
plt.imshow(grid)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-61-1601915e10f3> in <module>()
      1 w = torch.randn(10,3,640,640)
      2 grid = torchvision.utils.make_grid(w, nrow=5)
----> 3 plt.imshow(grid)

/anaconda3/lib/python3.6/site-packages/matplotlib/pyplot.py in imshow(X, cmap, norm, aspect, interpolation, alpha, vmin, vmax, origin, extent, shape, filternorm, filterrad, imlim, resample, url, hold, data, **kwargs)
   3203                         filternorm=filternorm, filterrad=filterrad,
   3204                         imlim=imlim, resample=resample, url=url, data=data,
-> 3205                         **kwargs)
   3206     finally:
   3207         ax._hold = washold

/anaconda3/lib/python3.6/site-packages/matplotlib/__init__.py in inner(ax, *args, **kwargs)
   1853                         "the Matplotlib list!)" % (label_namer, func.__name__),
   1854                         RuntimeWarning, stacklevel=2)
-> 1855             return func(ax, *args, **kwargs)
   1856
   1857         inner.__doc__ = _add_data_doc(inner.__doc__,

/anaconda3/lib/python3.6/site-packages/matplotlib/axes/_axes.py in imshow(self, X, cmap, norm, aspect, interpolation, alpha, vmin, vmax, origin, extent, shape, filternorm, filterrad, imlim, resample, url, **kwargs)
   5485                               resample=resample, **kwargs)
   5486
-> 5487         im.set_data(X)
   5488         im.set_alpha(alpha)
   5489         if im.get_clip_path() is None:

/anaconda3/lib/python3.6/site-packages/matplotlib/image.py in set_data(self, A)
    651         if not (self._A.ndim == 2
    652                 or self._A.ndim == 3 and self._A.shape[-1] in [3, 4]):
--> 653             raise TypeError("Invalid dimensions for image data")
    654
    655         if self._A.ndim == 3:

TypeError: Invalid dimensions for image data

Even though PyTorch's documentation indicates that w is the correct shape, Python says that it isn't. So I tried to permute the indices of my tensor:

    w = torch.randn(10,3,640,640)
    grid = torchvision.utils.make_grid(w.permute(0,2,3,1), nrow=5)
    plt.imshow(grid)
---------------------------------------------------------------------------
    RuntimeError                              Traceback (most recent call last)
    <ipython-input-62-6f2dc6313e29> in <module>()
          1 w = torch.randn(10,3,640,640)
    ----> 2 grid = torchvision.utils.make_grid(w.permute(0,2,3,1), nrow=5)
          3 plt.imshow(grid)

    /anaconda3/lib/python3.6/site-packages/torchvision-0.2.1-py3.6.egg/torchvision/utils.py in make_grid(tensor, nrow, padding, normalize, range, scale_each, pad_value)
         83             grid.narrow(1, y * height + padding, height - padding)\
         84                 .narrow(2, x * width + padding, width - padding)\
    ---> 85                 .copy_(tensor[k])
         86             k = k + 1
         87     return grid

    RuntimeError: The expanded size of the tensor (3) must match the existing size (640) at non-singleton dimension 0

What's happening here? How can I place a bunch of randomly generated images into a grid and display them?


title: PyTorch: Extract learned weights correctly

body:
I am trying to extract the weights from a linear layer, but they do not appear to change, although error is dropping monotonously (i.e. training is happening). Printing the weights' sum, nothing happens because it stays constant:

  np.sum(model.fc2.weight.data.numpy())

Here are the code snippets:

def train(epochs):
    model.train()
    for epoch in range(1, epochs+1):
        # Train on train set
        print(np.sum(model.fc2.weight.data.numpy()))
        for batch_idx, (data, target) in enumerate(train_loader):
            data, target = Variable(data), Variable(data)
            optimizer.zero_grad()
            output = model(data)
            loss = criterion(output, target)
            loss.backward()
            optimizer.step()

and

# Define model
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        # an affine operation: y = Wx + b
        self.fc1 = nn.Linear(100, 80, bias=False)
        init.normal(self.fc1.weight, mean=0, std=1)
        self.fc2 = nn.Linear(80, 87)
        self.fc3 = nn.Linear(87, 94)
        self.fc4 = nn.Linear(94, 100)

    def forward(self, x):
        x = self.fc1(x)
        x = F.relu(self.fc2(x))
        x = F.relu(self.fc3(x))
        x = F.relu(self.fc4(x))
        return x

Maybe I am looking on the wrong parameters, although I checked the docs. Thanks for your help!


title: Hyperparameter optimization for Pytorch model

body: What is the best way to perform hyperparameter optimization for a Pytorch model? Implement e.g. Random Search myself? Use Skicit Learn? Or is there anything else I am not aware of?


title: How to convert a list or numpy array to a 1d torch tensor?

body:
I have a list (or, a numpy array) of float values. I want to create a 1d torch tensor that will contain all those values. I can create the torch tensor and run a loop to store the values.

But I want to know is there any way, I can create a torch tensor with initial values from a list or array? Also suggest me if there is any pythonic way to achieve this as I am working in pytorch.


title: Pytorch: Create an boolean tensor (type: torch.ByteTensor)?

body:
I want to create a tensor only containing boolean values.
In Matlab that would be

a = false(10,1)


title: Differences in SciKit Learn, Keras, or Pytorch

body:
Are these libraries fairly interchangeable?

Looking here, https://stackshare.io/stackups/keras-vs-pytorch-vs-scikit-learn, it seems the major difference is the underlying framework (at least for PyTorch).


title: Pytorch, what are the gradient arguments

body:
I am reading through the documentation of PyTorch and found an example where they write

gradients = torch.FloatTensor([0.1, 1.0, 0.0001])
y.backward(gradients)
print(x.grad)

where x was an initial variable, from which y was constructed (a 3-vector). The question is, what are the 0.1, 1.0 and 0.0001 arguments of the gradients tensor ? The documentation is not very clear on that.


title: How to efficiently retrieve the indices of maximum values in a Torch tensor?

body:
Assume to have a torch tensor, for example of the following shape:

x = torch.rand(20, 1, 120, 120)

What I would like now, is to get the indices of the maximum values of each 120x120 matrix. To simplify the problem I would first x.squeeze() to work with shape [20, 120, 120]. I would then like to get torch tensor which is a list of indices with shape [20, 2].

How can I do this fast?


title: Expected object of device type cuda but got device type cpu in Pytorch

body:
I have the following code which calculates a loss function:

class MSE_loss(nn.Module):
    """
    : metric: L1, L2 norms or cosine similarity
    : mode: training or evaluation mode
    """

    def __init__(self,metric, mode, weighted_sum = False):
        super(MSE_loss, self).__init__()
        self.metric = metric.lower()
        self.loss_function = nn.MSELoss()
        self.mode = mode.lower()
        self.weighted_sum = weighted_sum

    def forward(self, output1, output2, labels):
        self.labels = labels
        self.linear = nn.Linear(output1.size()[0],1)

        if self.metric == 'cos':
            self.d= F.cosine_similarity(output1, output2)
        elif self.metric == 'l1':
            self.d = torch.abs(output1-output2)
        elif self.metric == 'l2':
            self.d = torch.sqrt((output1-output2)**2)

        def dimensional_reduction(forward):
            if self.weighted_sum:
                distance = self.linear(self.d)
            else:
                distance = torch.mean(self.d,1)
            return distance

        def estimate_loss(forward):
            distance = dimensional_reduction(self.d)
            pred = torch.exp(-distance)
            pred = torch.round(pred)
            loss = self.loss_function(pred, self.labels)
            return pred, loss

        pred, loss = estimate_loss(self.d)

        if self.mode == 'training':
            return loss
        else:
            return pred, loss

Given

criterion = MSE_loss('l1','training', weighted_sum = True)

I would like to get the distance after having gone through the self.linear neuron when implementing criterion. I am, however, prompted with the error 'Expected object of device type cuda but got device type cpu for argument #1 'self' in call to _th_addmm' indicating that something is wrong. I have omitted the first part of the code, but I provide the whole error message, so that you can get an idea of what is going on.

RuntimeError                              Traceback (most recent call last)
<ipython-input-253-781ed4791260> in <module>()
      7 criterion = MSE_loss('l1','training', weighted_sum = True)
      8
----> 9 train(test_net, train_loader, 10, batch_size, optimiser, clip, criterion)

<ipython-input-207-02fecbfe3b1c> in train(SNN, dataloader, epochs, batch_size, optimiser, clip, criterion)
     57
     58             # calculate the loss and perform backprop
---> 59             loss = criterion(output1, output2, labels)
     60             a = [[n,p, p.grad] for n,p in SNN.named_parameters()]
     61

~/.conda/envs/dalkeCourse/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    539             result = self._slow_forward(*input, **kwargs)
    540         else:
--> 541             result = self.forward(*input, **kwargs)
    542         for hook in self._forward_hooks.values():
    543             hook_result = hook(self, input, result)

<ipython-input-248-fb88b987ce71> in forward(self, output1, output2, labels)
     49             return pred, loss
     50
---> 51         pred, loss = estimate_loss(self.d)
     52
     53         if self.mode == 'training':

<ipython-input-248-fb88b987ce71> in estimate_loss(forward)
     43
     44         def estimate_loss(forward):
---> 45             distance = dimensional_reduction(self.d)
     46             pred = torch.exp(-distance)
     47             pred = torch.round(pred)

<ipython-input-248-fb88b987ce71> in dimensional_reduction(forward)
     36             else:
     37                 if self.weighted_sum:
---> 38                     self.d = self.linear(self.d)
     39                 else:
     40                     self.d = torch.mean(self.d,1)

~/.conda/envs/dalkeCourse/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    539             result = self._slow_forward(*input, **kwargs)
    540         else:
--> 541             result = self.forward(*input, **kwargs)
    542         for hook in self._forward_hooks.values():
    543             hook_result = hook(self, input, result)

~/.conda/envs/dalkeCourse/lib/python3.6/site-packages/torch/nn/modules/linear.py in forward(self, input)
     85
     86     def forward(self, input):
---> 87         return F.linear(input, self.weight, self.bias)
     88
     89     def extra_repr(self):

~/.conda/envs/dalkeCourse/lib/python3.6/site-packages/torch/nn/functional.py in linear(input, weight, bias)
   1368     if input.dim() == 2 and bias is not None:
   1369         # fused op is marginally faster
-> 1370         ret = torch.addmm(bias, input, weight.t())
   1371     else:
   1372         output = input.matmul(weight.t())

RuntimeError: Expected object of device type cuda but got device type cpu for argument #1 'self' in call to _th_addmm

self.d is however a tensor, but this has already been passed into GPU, as shown below:

self.d =
tensor([[3.7307e-04, 8.4476e-04, 4.0426e-04,  ..., 4.2015e-04, 1.7830e-04,
         1.2833e-04],
        [3.9271e-04, 4.8325e-04, 9.5238e-04,  ..., 1.5126e-04, 1.3420e-04,
         3.9260e-04],
        [1.9278e-04, 2.6530e-04, 8.6903e-04,  ..., 1.6985e-05, 9.5103e-05,
         1.9610e-04],
        ...,
        [1.8257e-05, 3.1304e-04, 4.6398e-04,  ..., 2.7327e-04, 1.1909e-04,
         1.5069e-04],
        [1.7577e-04, 3.4820e-05, 9.4168e-04,  ..., 3.2848e-04, 2.2514e-04,
         5.4275e-05],
        [4.2916e-04, 1.6155e-04, 9.3186e-04,  ..., 1.0950e-04, 2.5083e-04,
         3.7374e-06]], device='cuda:0', grad_fn=<AbsBackward>)


I'm trying to visualize my GRU model using PyTorchViz but every time I run this code it gives me an error.

I want something like this in the

import torch
from torch import nn
from torchviz import make_dot, make_dot_from_trace

model = IC_V6(f.tokens)
x = torch.randn(1,8)

make_dot(model(x), params=dict(model.named_parameters()))

here is my Class for holding the data

class Flickr8KImageCaptionDataset:

    def __init__(self):

        all_data = json.load(open('caption_datasets/dataset_flickr8k.json', 'r'))
        all_data=all_data['images']

        self.training_data = []
        self.test_data = []
        self.w2i = {ENDWORD: 0, STARTWORD: 1}
        self.word_frequency = {ENDWORD: 0, STARTWORD: 0}
        self.i2w = {0: ENDWORD, 1: STARTWORD}
        self.tokens = 2 #END is default
        self.batch_index = 0

        for data in all_data:
            if(data['split']=='train'):
                self.training_data.append(data)
            else:
                self.test_data.append(data)

            for sentence in data['sentences']:
                for token in sentence['tokens']:
                    if(token not in self.w2i.keys()):
                        self.w2i[token] = self.tokens
                        self.i2w[self.tokens] = token
                        self.tokens +=1
                        self.word_frequency[token] = 1
                    else:
                        self.word_frequency[token] += 1

    def image_to_tensor(self,filename):
        image = Image.open(filename)
        image = TF.resize(img=image, size=(HEIGHT,WIDTH))

        image = TF.to_tensor(pic=image)
        image = TF.normalize(image, mean=[0.485, 0.456, 0.406],
                                     std=[0.229, 0.224, 0.225])

        return torch.unsqueeze(image,0)

    def return_train_batch(self): #size of 1 always

        #np.random.shuffle(self.training_data)
        for index in range(len(self.training_data)):

        #index = np.random.randint(len(self.training_data))
            sentence_index = np.random.randint(len(self.training_data[index]['sentences']))
            output_sentence_tokens = deepcopy(self.training_data[index]['sentences'][sentence_index]['tokens'])
            output_sentence_tokens.append(ENDWORD) #corresponds to end word

            image = self.image_to_tensor('/home/vincent/Documents/Final Code/Flicker8k_Dataset/'+self.training_data[index]['filename'])

            yield image, list(map(lambda x: self.w2i[x], output_sentence_tokens)), output_sentence_tokens, index

    def convert_tensor_to_word(self, output_tensor):
        output = F.log_softmax(output_tensor.detach().squeeze(), dim=0).numpy()
        return self.i2w[np.argmax(output)]

    def convert_sentence_to_tokens(self, sentence):

        tokens = sentence.split(" ")
        converted_tokens= list(map(lambda x: self.w2i[x], tokens))
        converted_tokens.append(self.w2i[ENDWORD])
        return converted_tokens

    def caption_image_greedy(self, net, image_filename, max_words=15): #non beam search, no temperature implemented
        net.eval()
        inception.eval()
        image_tensor = self.image_to_tensor(image_filename)
        hidden=None
        embedding=None
        words = []

        input_token = STARTWORD
        input_tensor = torch.tensor(self.w2i[input_token]).type(torch.LongTensor)

        for i in range(max_words):

            if(i==0):
                out, hidden=net(input_tensor, hidden=image_tensor, process_image=True)
            else:
                out, hidden=net(input_tensor, hidden)

            word = self.convert_tensor_to_word(out)

            input_token = self.w2i[word]
            input_tensor = torch.tensor(input_token).type(torch.LongTensor)

            if(word==ENDWORD):
                break
            else:
                words.append(word)

        return ' '.join(words)

    def forward_beam(self, net, hidden, process_image, partial_sentences, sentences, topn_words=5, max_sentences=10):

        max_words = 50

        hidden_index = {}

        while(sentences<max_sentences):

            #print("Sentences: ",sentences)

            new_partial_sentences = []
            new_partial_sentences_logp = []
            new_partial_avg_logp= []

            if(len(partial_sentences[-1][0])>max_words):
                break

            for partial_sentence in partial_sentences:

                input_token = partial_sentence[0][-1]

                input_tensor = torch.tensor(self.w2i[input_token]).type(torch.FloatTensor)

                if(partial_sentence[0][-1]==STARTWORD):
                    out, hidden=net(input_tensor, hidden, process_image=True)
                else:
                    out, hidden=net(input_tensor, torch.tensor(hidden_index[input_token]))

                #take first topn words and add as children to root

                out = F.log_softmax(out.detach().squeeze(), dim=0).numpy()
                out_indexes = np.argsort(out)[::-1][:topn_words]

                for out_index in out_indexes:

                    if(self.i2w[out_index]==ENDWORD):
                        sentences=sentences+1

                    else:

                        total_logp = float(out[out_index]) + partial_sentence[1]
                        new_partial_sentences_logp.append(total_logp)
                        new_partial_sentences.append([np.concatenate((partial_sentence[0], [self.i2w[out_index]])),total_logp])

                        len_words = len(new_partial_sentences[-1][0])

                        new_partial_avg_logp.append(total_logp/len_words)

                        #print(self.i2w[out_index])

                        hidden_index[self.i2w[out_index]] = deepcopy(hidden.detach().numpy())

            #select topn partial sentences

            top_indexes = np.argsort(new_partial_sentences_logp)[::-1][:topn_words]

            new_partial_sentences = np.array(new_partial_sentences)[top_indexes]

            #print("New partial sentences (topn):", new_partial_sentences)

            partial_sentences = new_partial_sentences

        return partial_sentences

    def caption_image_beam_search(self, net, image_filename, topn_words=10, max_sentences=10):

        net.eval()
        inception.eval()
        image_tensor = self.image_to_tensor(image_filename)
        hidden=None
        embedding=None
        words = []

        sentences = 0
        partial_sentences = [[[STARTWORD], 0.0]]
        #root_id = hash(input_token) #for start word
        #nodes = {}

        #nodes[root_id] = Node(root_id, [STARTWORD, 0], None)

        partial_sentences = self.forward_beam(net, image_tensor, True, partial_sentences, sentences,  topn_words, max_sentences)

        logp = []

        joined_sentences = []

        for partial_sentence in partial_sentences:

            joined_sentences.append([' '.join(partial_sentence[0][1:]),partial_sentence[1]])

        return joined_sentences

    def print_beam_caption(self, net, train_filename,num_captions=0):
        beam_sentences = f.caption_image_beam_search(net,train_filename)
        if(num_captions==0):
            num_captions=len(beam_sentences)
        for sentence in beam_sentences[:num_captions]:
            print(sentence[0]+" [",sentence[1], "]")

and here is my GRU model

class IC_V6(nn.Module):

    #V2: Fed image vector directly as hidden and fed words generated as iputs back to LSTM
    #V3: Added an embedding layer between words input and GRU/LSTM

    def __init__(self, token_dict_size):
        super(IC_V6, self).__init__()

        #Input is an image of height 500, and width 500

        self.embedding_size = INPUT_EMBEDDING
        self.hidden_state_size = HIDDEN_SIZE
        self.token_dict_size = token_dict_size
        self.output_size = OUTPUT_EMBEDDING

        self.batchnorm = nn.BatchNorm1d(self.embedding_size)

        self.input_embedding = nn.Embedding(self.token_dict_size, self.embedding_size)

        self.embedding_dropout = nn.Dropout(p=0.22)
        self.gru_layers = 3

        self.gru = nn.GRU(input_size=self.embedding_size, hidden_size=self.hidden_state_size, num_layers=self.gru_layers, dropout=0.22)
        self.linear = nn.Linear(self.hidden_state_size, self.output_size)
        self.out = nn.Linear(self.output_size, token_dict_size)

    def forward(self, input_tokens, hidden, process_image=False, use_inception=True):

        if(USE_GPU):
            device = torch.device('cuda')
        else:
            device = torch.device('cpu')

        if(process_image):
            if(use_inception):
                inp=self.embedding_dropout(inception(hidden))
            else:
                inp=hidden
            #inp=self.batchnorm(inp)
            hidden=torch.zeros((self.gru_layers,1, self.hidden_state_size))
        else:
            inp=self.embedding_dropout(self.input_embedding(input_tokens.view(1).type(torch.LongTensor).to(device)))
            #inp=self.batchnorm(inp)

        hidden = hidden.view(self.gru_layers,1,-1)
        inp = inp.view(1,1,-1)

        out, hidden = self.gru(inp, hidden)
        out = self.out(self.linear(out))

        return out, hidden

this is how I called them:

f = Flickr8KImageCaptionDataset()
net = IC_V6(f.tokens)

the error is:

TypeError Traceback (most recent call last)
<ipython-input-42-7993fc1a032f> in <module>
      6 x = torch.randn(1,8)
      7
----> 8 make_dot(model(x), params=dict(model.named_parameters()))

~/anaconda3/envs/Thesis/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    487             result = self._slow_forward(*input, **kwargs)
    488         else:
--> 489             result = self.forward(*input, **kwargs)
    490         for hook in self._forward_hooks.values():
    491             hook_result = hook(self, input, result)

  TypeError: forward() missing 1 required positional argument: 'hidden'

What should I do to solve this problem? Any help will be much appreciated.


title: Where do I get a CPU-only version of PyTorch?

body:
I'm trying to get a basic app running with Flask + PyTorch, and host it on Heroku. However, I run into the issue that the maximum slug size is 500mb on the free version, and PyTorch itself is ~500mb.

After some google searching, someone wrote about finding a cpu-only version of PyTorch, and using that, which is much smaller here.

However, I'm pretty lost as to how this is done, and the person didn't document this at all. Any advice is appreciated, thanks.

EDIT:

To be more specific about my problem, I tried installing torch by (as far as I understand), including a requirements.txt which listed torch as a dependency. Current I have: torch==0.4.1. However this doesn't work bc of size.

My question is, do you know what I could write in the requirements file to get the cpu-only version of torch that is smaller, or alternatively, if the requirements.txt doesn't work for this, what I would do instead, to get the cpu version.


title: Parsing CSV into Pytorch tensors

body:
I have a CSV files with all numeric values except the header row. When trying to build tensors, I get the following exception:

Traceback (most recent call last):
  File "pytorch.py", line 14, in <module>
    test_tensor = torch.tensor(test)
ValueError: could not determine the shape of object type 'DataFrame'

This is my code:

import torch
import dask.dataframe as dd

device = torch.device("cuda:0")

print("Loading CSV...")
test = dd.read_csv("test.csv", encoding = "UTF-8")
train = dd.read_csv("train.csv", encoding = "UTF-8")

print("Converting to Tensor...")
test_tensor = torch.tensor(test)
train_tensor = torch.tensor(train)

Using pandas instead of Dask for CSV parsing produced the same error. I also tried to specify dtype=torch.float64 inside the call to torch.tensor(data), but got the same error again.


title: How to get the device type of a pytorch module conveniently?

body:
I have to stack some my own layers on different kinds of pytorch models with different devices.

E.g. A is a cuda model and B is a cpu model (but I don't know it before I get the device type). Then the new models are C and D respectively, where

class NewModule(torch.nn.Module):
    def __init__(self, base):
        super(NewModule, self).__init__()
        self.base = base
        self.extra = my_layer() # e.g. torch.nn.Linear()

    def forward(self,x):
        y = self.base(x)
        z = self.extra(y)
        return z

...

C = NewModule(A) # cuda
D = NewModule(B) # cpu

However I must move base and extra to the same device, i.e. base and extra of C are cuda models and D's are cpu models. So I tried this __inin__:

def __init__(self, base):
    super(NewModule, self).__init__()
    self.base = base
    self.extra = my_layer().to(base.device)

Unfortunately, there's no attribute device in torch.nn.Module(raise AttributeError).

What should I do to get the device type of base? Or any other method to make base and extra to be on the same device automaticly even the structure of base is unspecific?


title: Confusion matrix and test accuracy for PyTorch Transfer Learning tutorial

body:
Following the Pytorch Transfer learning tutorial, I am interested in reporting only train and test accuracy as well as confusion matrix (say using sklearn confusionmatrix). How can I do that? The current tutorial only reports train/val accuracy and I am having hard time figuring how to incorporate the sklearn confusionmatrix code there. Link to original tutorial here: https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html

%matplotlib inline
from graphviz import Digraph
import torch
from torch.autograd import Variable
# Author: Sasank Chilamkurthy

from __future__ import print_function, division

import torch
import torch.nn as nn
import torch.optim as optim
from torch.optim import lr_scheduler
import numpy as np
import torchvision
from torchvision import datasets, models, transforms
import matplotlib.pyplot as plt
import time
import os
import copy

plt.ion()
data_transforms = {
    'train': transforms.Compose([
        transforms.RandomResizedCrop(224),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
    'val': transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
}

data_dir = "images"
image_datasets = {x: datasets.ImageFolder(os.path.join(data_dir, x),
                                          data_transforms[x])
                  for x in ['train', 'val']}
dataloaders = {x: torch.utils.data.DataLoader(image_datasets[x], batch_size=4,
                                             shuffle=True, num_workers=4)
              for x in ['train', 'val']}
dataset_sizes = {x: len(image_datasets[x]) for x in ['train', 'val']}
class_names = image_datasets['train'].classes

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

def imshow(inp, title=None):
    """Imshow for Tensor."""
    inp = inp.numpy().transpose((1, 2, 0))
    mean = np.array([0.485, 0.456, 0.406])
    std = np.array([0.229, 0.224, 0.225])
    inp = std * inp + mean
    inp = np.clip(inp, 0, 1)
    plt.imshow(inp)
    if title is not None:
        plt.title(title)
    plt.pause(0.001)  # pause a bit so that plots are updated

# Get a batch of training data
inputs, classes = next(iter(dataloaders['train']))

# Make a grid from batch
out = torchvision.utils.make_grid(inputs)

imshow(out, title=[class_names[x] for x in classes])

def train_model(model, criterion, optimizer, scheduler, num_epochs=25):
    since = time.time()

    best_model_wts = copy.deepcopy(model.state_dict())
    best_acc = 0.0

    for epoch in range(num_epochs):
        print('Epoch {}/{}'.format(epoch, num_epochs - 1))
        print('-' * 10)

        # Each epoch has a training and validation phase
        for phase in ['train', 'val']:
            if phase == 'train':
                scheduler.step()
                model.train()  # Set model to training mode
            else:
                model.eval()   # Set model to evaluate mode

            running_loss = 0.0
            running_corrects = 0

            # Iterate over data.
            for inputs, labels in dataloaders[phase]:
                inputs = inputs.to(device)
                labels = labels.to(device)

                # zero the parameter gradients
                optimizer.zero_grad()

                # forward
                # track history if only in train
                with torch.set_grad_enabled(phase == 'train'):
                    outputs = model(inputs)
                    _, preds = torch.max(outputs, 1)
                    loss = criterion(outputs, labels)

                    # backward + optimize only if in training phase
                    if phase == 'train':
                        loss.backward()
                        optimizer.step()

                # statistics
                running_loss += loss.item() * inputs.size(0)
                running_corrects += torch.sum(preds == labels.data)

            epoch_loss = running_loss / dataset_sizes[phase]
            epoch_acc = running_corrects.double() / dataset_sizes[phase]

            print('{} Loss: {:.4f} Acc: {:.4f}'.format(
                phase, epoch_loss, epoch_acc))

            # deep copy the model
            if phase == 'val' and epoch_acc > best_acc:
                best_acc = epoch_acc
                best_model_wts = copy.deepcopy(model.state_dict())

        print()

    time_elapsed = time.time() - since
    print('Training complete in {:.0f}m {:.0f}s'.format(
        time_elapsed // 60, time_elapsed % 60))
    print('Best val Acc: {:4f}'.format(best_acc))

    # load best model weights
    model.load_state_dict(best_model_wts)
    return model

def visualize_model(model, num_images=6):
    was_training = model.training
    model.eval()
    images_so_far = 0
    fig = plt.figure()

    with torch.no_grad():
        for i, (inputs, labels) in enumerate(dataloaders['val']):
            inputs = inputs.to(device)
            labels = labels.to(device)

            outputs = model(inputs)
            _, preds = torch.max(outputs, 1)

            for j in range(inputs.size()[0]):
                images_so_far += 1
                ax = plt.subplot(num_images//2, 2, images_so_far)
                ax.axis('off')
                ax.set_title('predicted: {}'.format(class_names[preds[j]]))
                imshow(inputs.cpu().data[j])

                if images_so_far == num_images:
                    model.train(mode=was_training)
                    return
        model.train(mode=was_training)

model_ft = models.resnet18(pretrained=True)
num_ftrs = model_ft.fc.in_features
model_ft.fc = nn.Linear(num_ftrs, 9)

model_ft = model_ft.to(device)

criterion = nn.CrossEntropyLoss()

# Observe that all parameters are being optimized
optimizer_ft = optim.SGD(model_ft.parameters(), lr=0.001, momentum=0.9)

# Decay LR by a factor of 0.1 every 7 epochs
exp_lr_scheduler = lr_scheduler.StepLR(optimizer_ft, step_size=7, gamma=0.1)

model_ft = train_model(model_ft, criterion, optimizer_ft, exp_lr_scheduler,
                       num_epochs=25)

visualize_model(model_ft)


title: Calculating input and output size for Conv2d in PyTorch for image classification

body:
I'm trying to run the PyTorch tutorial on CIFAR10 image classification here - http://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#sphx-glr-beginner-blitz-cifar10-tutorial-py

I've made a small change and I'm using a different dataset. I have images from the Wikiart dataset that I want to classify by artist (label = artist name).

Here is the code for the Net -

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16*5*5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16*5*5)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

Then there is this section of the code where I start training the Net.

for epoch in range(2):
     running_loss = 0.0

     for i, data in enumerate(wiki_train_dataloader, 0):
        inputs, labels = data['image'], data['class']
        print(inputs.shape)
        inputs, labels = Variable(inputs), Variable(labels)

        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = net(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.data[0]
        if i % 2000 == 1999:  # print every 2000 mini-batches
            print('[%d, %5d] loss: %.3f' %
              (epoch + 1, i + 1, running_loss / 2000))
            running_loss = 0.0

This line print(inputs.shape) gives me torch.Size([4, 32, 32, 3]) with my Wikiart dataset whereas in the original example with CIFAR10, it prints torch.Size([4, 3, 32, 32]).

Now, I'm not sure how to change the Conv2d in my Net to be compatible with torch.Size([4, 32, 32, 3]).

I get this error:

RuntimeError: Given input size: (3 x 32 x 3). Calculated output size: (6 x 28 x -1). Output size is too small at /opt/conda/conda-bld/pytorch_1503965122592/work/torch/lib/THNN/generic/SpatialConvolutionMM.c:45

While reading the images for the Wikiart dataset, I resize them to (32, 32) and these are 3-channel images.

Things I tried:

1) The CIFAR10 tutorial uses a transform which I am not using. I could not incorporate the same into my code.

2) Changing self.conv2 = nn.Conv2d(6, 16, 5) to self.conv2 = nn.Conv2d(3, 6, 5). This gave me the same error as above. I was only changing this to see if the error message changes.

Any resources on how to calculate input & output sizes in PyTorch or automatically reshape Tensors would be really appreciated. I just started learning Torch & I find the size calculations complicated.


title: How can I install python modules in a docker image?

body: I have an image called: Image and a running container called: container. I want to install pytorch and anacoda. what's the easiest way to do this? Do I have to change the dockerfile and build a new image? Thanks a lot.


title: How to convert a pytorch tensor of ints to a tensor of booleans?

body:
I would like to cast a tensor of ints to a tensor of booleans.

Specifically I would like to be able to have a function which transforms tensor([0,10,0,16]) to tensor([0,1,0,1])

This is trivial in Tensorflow by just using tf.cast(x,tf.bool).

I want the cast to change all ints greater than 0 to a 1 and all ints equal to 0 to a 0. This is the equivalent of !! in most languages.

Since pytorch does not seem to have a dedicated boolean type to cast to, what is the best approach here?

Edit: I am looking for a vectorized solution opposed to looping through each element.


title: Multi label classification in pytorch

body:
I have a multi-label classification problem. I have 11 classes, around 4k examples. Each example can have from 1 to 4-5 label. At the moment, i'm training a classifier separately for each class with log_loss. As you can expect, it is taking quite some time to train 11 classifier, and i would like to try another approach and to train only 1 classifier. The idea is that the last layer of this classifer would have 11 nodes, and would output a real number by classes which would be converted to a proba by a sigmoid. The loss I want to optimize is the mean of the log_loss on all classes.

Unfortunately, i'm some kind of noob with pytorch, and even by reading the source code of the losses, i can't figure out if one of the already existing losses does exactly what i want, or if I should create a new loss, and if that's the case, i don't really know how to do it.

To be very specific, i want to give for each element of the batch one vector of size 11(which contains a real number for each label (the closer to infinity, the closer this class is predicted to be 1), and 1 vector of size 11 (which contains a 1 at every true label), and be able to compute the mean log_loss on all 11 labels, and optimize my classifier based on that loss.

Any help would be greatly appreciated :)


title: Pytorch Change the learning rate based on number of epochs

body:
When I set the learning rate and find the accuracy cannot increase after training few epochs

optimizer = optim.Adam(model.parameters(), lr = 1e-4)

n_epochs = 10
for i in range(n_epochs):

    // some training here

If I want to use a step decay: reduce the learning rate by a factor of 10 every 5 epochs, how can I do so?

Thanks


title: Get total amount of free GPU memory and available using pytorch

body: I'm using google colab free Gpu's for experimentation and wanted to know how much GPU Memory available to play around, torch.cuda.memory_allocated() returns the current GPU memory occupied, but how do we determine total available memory using PyTorch.


title: how to flatten input in `nn.Sequential` in Pytorch

body:
how to flatten input inside the nn.Sequential

Model = nn.Sequential(x.view(x.shape[0],-1),
                     nn.Linear(784,256),
                     nn.ReLU(),
                     nn.Linear(256,128),
                     nn.ReLU(),
                     nn.Linear(128,64),
                     nn.ReLU(),
                     nn.Linear(64,10),
                     nn.LogSoftmax(dim=1))


title: Pytorch install with anaconda error

body:
I get this error:

C:\Users>conda install pytorch torchvision -c soumith
Fetching package metadata .............

PackageNotFoundError: Package missing in current win-64 channels:
  - pytorch

I got conda install pytorch torchvision -c soumith from Pytorch official website and I have OSX/conda/3.6/none for settings on Pytorch site(should be correct). I am new to conda, any tips how to solve this?


title: How to do fully connected batch norm in PyTorch?

body: torch.nn has classes BatchNorm1d, BatchNorm2d, BatchNorm3d, but it doesn't have a fully connected BatchNorm class? What is the standard way of doing normal Batch Norm in PyTorch?


title: How to multiply a matrix by a vector in PyTorch

body:
I'm playing around with PyTorch with the aim of learning it, and I have a very dumb question: how can I multiply a matrix by a single vector?

Here's what I've tried:

>>> import torch
>>> a = torch.rand(4,4)
>>> a

 0.3162  0.4434  0.9318  0.8752
 0.0129  0.8609  0.6402  0.2396
 0.5720  0.7262  0.7443  0.0425
 0.4561  0.1725  0.4390  0.8770
[torch.FloatTensor of size 4x4]

>>> b = torch.rand(4)
>>> b

 0.1813
 0.7090
 0.0329
 0.7591
[torch.FloatTensor of size 4]

>>> a.mm(b)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: invalid argument 2: dimension 1 out of range of 1D tensor at /Users/soumith/code/builder/wheel/pytorch-src/torch/lib/TH/generic/THTensor.c:24
>>> a.mm(b.t())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: t() expects a 2D tensor, but self is 1D
>>> b.mm(a)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: matrices expected, got 1D, 2D tensors at /Users/soumith/code/builder/wheel/pytorch-src/torch/lib/TH/generic/THTensorMath.c:1288
>>> b.t().mm(a)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: t() expects a 2D tensor, but self is 1D

On the other hand, if I do

>>> b = torch.rand(4,2)

then my first attempt, a.mm(b), works fine. So the problem is just that I'm multiplying a vector rather than a matrix --- but how can I do this?


I have a very large tensor of shape (512,3,224,224). I input it to model in batches of 32 and I then save the scores corresponding to the target label which is 2. in each iteration, after every slice, the shape of scores changes. Which leads to the following error. What am I doing wrong and how to fix it.
label = torch.ones(1)*2

def sub_forward(self, x):
    x = self.vgg16(x)
    x = self.bn1(x)
    x = self.linear1(x)
    x = self.linear2(x)
    return x

def get_scores(self, imgs, targets):
    b, _, _, _ = imgs.shape
    batch_size = 32
    total_scores = []
    for i in range(0, b, batch_size):
      scores = self.sub_forward(imgs[i:i+batch_size,:,:,:])
      scores = F.softmax(scores)
      labels = targets[i:i+batch_size]
      labels = labels.long()
      scores = scores[:,labels]
      print(i," scores: ", scores)
      total_scores.append(scores)
      print(i," total_socres: ", total_scores)
    total_scores = torch.stack(total_scores)
    return scores

0  scores:  tensor([[0.0811],
        [0.0918],
        [0.0716],
        [0.1680],
        [0.1689],
        [0.1319],
        [0.1556],
        [0.2966],
        [0.0913],
        [0.1238],
        [0.1480],
        [0.1215],
        [0.2524],
        [0.1283],
        [0.1603],
        [0.1282],
        [0.2668],
        [0.1146],
        [0.2043],
        [0.2475],
        [0.0865],
        [0.1869],
        [0.0860],
        [0.1979],
        [0.1677],
        [0.1983],
        [0.2623],
        [0.1975],
        [0.1894],
        [0.3299],
        [0.1970],
        [0.1094]], device='cuda:0')
0  total_socres:  [tensor([[0.0811],
        [0.0918],
        [0.0716],
        [0.1680],
        [0.1689],
        [0.1319],
        [0.1556],
        [0.2966],
        [0.0913],
        [0.1238],
        [0.1480],
        [0.1215],
        [0.2524],
        [0.1283],
        [0.1603],
        [0.1282],
        [0.2668],
        [0.1146],
        [0.2043],
        [0.2475],
        [0.0865],
        [0.1869],
        [0.0860],
        [0.1979],
        [0.1677],
        [0.1983],
        [0.2623],
        [0.1975],
        [0.1894],
        [0.3299],
        [0.1970],
        [0.1094]], device='cuda:0')]
32  scores:  tensor([], device='cuda:0', size=(32, 0))
32  total_socres:  [tensor([[0.0811],
        [0.0918],
        [0.0716],
        [0.1680],
        [0.1689],
        [0.1319],
        [0.1556],
        [0.2966],
        [0.0913],
        [0.1238],
        [0.1480],
        [0.1215],
        [0.2524],
        [0.1283],
        [0.1603],
        [0.1282],
        [0.2668],
        [0.1146],
        [0.2043],
        [0.2475],
        [0.0865],
        [0.1869],
        [0.0860],
        [0.1979],
        [0.1677],
        [0.1983],
        [0.2623],
        [0.1975],
        [0.1894],
        [0.3299],
        [0.1970],
        [0.1094]], device='cuda:0'), tensor([], device='cuda:0', size=(32, 0))]

> RuntimeError: stack expects each tensor to be equal size, but got [32, 1] at entry 0 and [32, 0] at entry 1


title: Cannot import Pytorch [WinError 126] The specified module could not be found

body:
I'm trying to do a basic install and import of Pytorch/Torchvision on Windows 10. I installed a Anaconda and created a new virtual environment named photo. I opened Anaconda prompt, activated the environment, and I ran:

(photo) C:\Users\<user>\anaconda3\envs>conda install pytorch torchvision cudatoolkit=10.2 -c pytorch**

This installed pytorch successfully. Running conda list I see:

  pytorch            pytorch/win-64::pytorch-1.5.0-py3.7_cuda102_cudnn7_0

  torchvision        pytorch/win-64::torchvision-0.6.0-py37_cu102

Then I open a python command prompt while in the virtual environment, and type:

import torch

The following error is printed:

  Traceback (most recent call last):
    File "", line 1, in
    File "C:\Users\njord\anaconda3\envs\photo\lib\site-packages\torch__init__.py", line 81, in
      ctypes.CDLL(dll)
    File "C:\Users\njord\anaconda3\envs\photo\lib\ctypes__init__.py", line 364, in init
      self._handle = _dlopen(self._name, mode)
  OSError: [WinError 126] The specified module could not be found

I have uninstalled/reinstalled python and anaconda but still run into the same issue. Advice appreciated.


title: What is volatile variable in Pytorch

body:
What is volatile attribute of a Variable in Pytorch? Here's a sample code for defining a variable in PyTorch.

datatensor = Variable(data, volatile=True)


title: How does adaptive pooling in pytorch work?

body:
Adaptive pooling is a great function, but how does it work?  It seems to be inserting pads or shrinking/expanding kernel sizes in what seems like a pattered but fairly arbitrary way.  The pytorch documentation I can find is not more descriptive than "put desired output size here."  Does anyone know how this works or can point to where it's explained?

Some test code on a 1x1x6 tensor, (1,2,3,4,5,6), with an adaptive output of size 8:

import torch
import torch.nn as nn

class TestNet(nn.Module):
    def __init__(self):
        super(TestNet, self).__init__()
        self.avgpool = nn.AdaptiveAvgPool1d(8)

    def forward(self,x):
        print(x)
        x = self.avgpool(x)
        print(x)
        return x

def test():
    x = torch.Tensor([[[1,2,3,4,5,6]]])
    net = TestNet()
    y = net(x)
    return y

test()

Output:

tensor([[[ 1.,  2.,  3.,  4.,  5.,  6.]]])
tensor([[[ 1.0000,  1.5000,  2.5000,  3.0000,  4.0000,  4.5000,  5.5000,
       6.0000]]])

If it mirror pads by on the left and right (operating on (1,1,2,3,4,5,6,6)), and has a kernel of 2, then the outputs for all positions except for 4 and 5 make sense, except of course the output isn't the right size.  Is it also padding the 3 and 4 internally?  If so, it's operating on (1,1,2,3,3,4,4,5,6,6), which, if using a size 2 kernel, produces the wrong output size and would also miss a 3.5 output.  Is it changing the size of the kernel?

Am I missing something obvious about the way this works?


title: Pytorch RuntimeError: Expected tensor for argument #1 'indices' to have scalar type Long; but got CUDAType instead

body:
I am trying to re-execute a GitHub project on my computer for recommendation using embedding, the goal is to first embed the user and item present in the movieLens dataset, and then use the inner product to predict a rating, when I finished the integration of all components, I got an error in the training.

Code:

from lightfm.datasets import fetch_movielens
movielens = fetch_movielens()
ratings_train, ratings_test = movielens['train'], movielens['test']
def _binarize(dataset):

    dataset = dataset.copy()

    dataset.data = (dataset.data >= 0.0).astype(np.float32)
    dataset = dataset.tocsr()
    dataset.eliminate_zeros()

    return dataset.tocoo()
train, test = _binarize(movielens['train']), _binarize(movielens['test'])
class ScaledEmbedding(nn.Embedding):
    """ Change the scale from normal to [0,1/embedding_dim] """
    def reset_parameters(self):
        self.weight.data.normal_(0, 1.0 / self.embedding_dim)
        if self.padding_idx is not None:
            self.weight.data[self.padding_idx].fill_(0)

class ZeroEmbedding(nn.Embedding):

    def reset_parameters(self):
        self.weight.data.zero_()
        if self.padding_idx is not None:
            self.weight.data[self.padding_idx].fill_(0)
class BilinearNet(nn.Module):

    def __init__(self, num_users, num_items, embedding_dim, sparse=False):
        super().__init__()

        self.embedding_dim = embedding_dim

        self.user_embeddings = ScaledEmbedding(num_users, embedding_dim,
                                               sparse=sparse)
        self.item_embeddings = ScaledEmbedding(num_items, embedding_dim,
                                               sparse=sparse)
        self.user_biases = ZeroEmbedding(num_users, 1, sparse=sparse)
        self.item_biases = ZeroEmbedding(num_items, 1, sparse=sparse)

    def forward(self, user_ids, item_ids):

        user_embedding = self.user_embeddings(user_ids)
        item_embedding = self.item_embeddings(item_ids)

        user_embedding = user_embedding.view(-1, self.embedding_dim)
        item_embedding = item_embedding.view(-1, self.embedding_dim)

        user_bias = self.user_biases(user_ids).view(-1, 1)
        item_bias = self.item_biases(item_ids).view(-1, 1)

        dot = (user_embedding * item_embedding).sum(1)

        return dot + user_bias + item_bias

def pointwise_loss(net,users, items, ratings, num_items):

    negatives = Variable(
            torch.from_numpy(np.random.randint(0,
                                               num_items,
                                                  len(users))).cuda()
    )

    positives_loss = (1.0 - torch.sigmoid(net(users, items)))
    negatives_loss = torch.sigmoid(net(users, negatives))

    return torch.cat([positives_loss, negatives_loss]).mean()

embedding_dim = 128
minibatch_size = 1024
n_iter = 10
l2=0.0
sparse = True

num_users, num_items = train.shape
net = BilinearNet(num_users,
                            num_items,
                            embedding_dim,
                            sparse=sparse).cuda()

optimizer = optim.Adagrad(net.parameters(),
                              weight_decay=l2)
for epoch_num in range(n_iter):

    users, items, ratings = shuffle(train)

    user_ids_tensor = torch.from_numpy(users).cuda()
    item_ids_tensor = torch.from_numpy(items).cuda()
    ratings_tensor = torch.from_numpy(ratings).cuda()

    epoch_loss = 0.0

    for (batch_user,
         batch_item,
         batch_ratings) in zip(_minibatch(user_ids_tensor,
                                          minibatch_size),
                               _minibatch(item_ids_tensor,
                                          minibatch_size),
                               _minibatch(ratings_tensor,
                                          minibatch_size)):

        user_var = Variable(batch_user)
        item_var = Variable(batch_item)
        ratings_var = Variable(batch_ratings)
        optimizer.zero_grad()
        loss = pointwise_loss(net,user_var, item_var, ratings_var, num_items)
        epoch_loss += loss.data[0]
        loss.backward()
        optimizer.step()
        print('Epoch {}: loss {}'.format(epoch_num, epoch_loss))

Error:

RuntimeError Traceback (most recent call last) <ipython-input-87-dcd04440363f> in <module>()
             22         ratings_var = Variable(batch_ratings)
             23         optimizer.zero_grad()
        ---> 24         loss = pointwise_loss(net,user_var, item_var, ratings_var, num_items)
             25         epoch_loss += loss.data[0]
             26         loss.backward()

        <ipython-input-86-679e10f637a5> in pointwise_loss(net, users, items, ratings, num_items)
              8
              9     positives_loss = (1.0 - torch.sigmoid(net(users, items)))
        ---> 10     negatives_loss = torch.sigmoid(net(users, negatives))
             11
             12     return torch.cat([positives_loss, negatives_loss]).mean()

        ~\Anaconda3\lib\site-packages\torch\nn\modules\module.py in
        __call__(self, *input, **kwargs)
            491             result = self._slow_forward(*input, **kwargs)
            492         else:
        --> 493             result = self.forward(*input, **kwargs)
            494         for hook in self._forward_hooks.values():
            495             hook_result = hook(self, input, result)

        <ipython-input-58-3946abf81d81> in forward(self, user_ids, item_ids)
             16
             17         user_embedding = self.user_embeddings(user_ids)
        ---> 18         item_embedding = self.item_embeddings(item_ids)
             19
             20         user_embedding = user_embedding.view(-1, self.embedding_dim)

        ~\Anaconda3\lib\site-packages\torch\nn\modules\module.py in
        __call__(self, *input, **kwargs)
            491             result = self._slow_forward(*input, **kwargs)
            492         else:
        --> 493             result = self.forward(*input, **kwargs)
            494         for hook in self._forward_hooks.values():
            495             hook_result = hook(self, input, result)

        ~\Anaconda3\lib\site-packages\torch\nn\modules\sparse.py in forward(self, input)
            115         return F.embedding(
            116             input, self.weight, self.padding_idx, self.max_norm,
        --> 117             self.norm_type, self.scale_grad_by_freq, self.sparse)
            118
            119     def extra_repr(self):

        ~\Anaconda3\lib\site-packages\torch\nn\functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)    1504         # remove once script supports set_grad_enabled    1505
        _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
        -> 1506     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)    1507     1508

        RuntimeError: Expected tensor for argument #1 'indices' to have scalar type Long; but got CUDAType instead (while checking arguments for embedding)

can anyone help me please ?


title: PyTorch DataLoader - "IndexError: too many indices for tensor of dimension 0"

body:
I am trying to implement a CNN to identify digits in the MNIST dataset and my code comes up with the error during the data loading process. I don't understand why this is happening.

import torch
import torchvision
import torchvision.transforms as transforms

transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((0.5), (0.5))
])

trainset = torchvision.datasets.MNIST(root='./data', train=True, download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=20, shuffle=True, num_workers=2)

testset = torchvision.datasets.MNIST(root='./data', train=False, download=True, transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=20, shuffle=False, num_workers=2)

for i, data in enumerate(trainloader, 0):
    inputs, labels = data[0], data[1]

Error:

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-6-b37c638b6114> in <module>
      2
----> 3     for i, data in enumerate(trainloader, 0):
      4         inputs, labels = data[0], data[1]

# ...

IndexError: Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 99, in _worker_loop
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "/opt/conda/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 99, in <listcomp>
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "/opt/conda/lib/python3.6/site-packages/torchvision/datasets/mnist.py", line 95, in __getitem__
    img = self.transform(img)
  File "/opt/conda/lib/python3.6/site-packages/torchvision/transforms/transforms.py", line 61, in __call__
    img = t(img)
  File "/opt/conda/lib/python3.6/site-packages/torchvision/transforms/transforms.py", line 164, in __call__
    return F.normalize(tensor, self.mean, self.std, self.inplace)
  File "/opt/conda/lib/python3.6/site-packages/torchvision/transforms/functional.py", line 208, in normalize
    tensor.sub_(mean[:, None, None]).div_(std[:, None, None])
IndexError: too many indices for tensor of dimension 0