title: Multiple PyTorch networks running in parallel on different CPUs

body:
I’m trying to have different PyTorch neural networks run in parallel on different CPUs but am finding that it isn’t leading to any sort of speed up compared to running them sequentially.

Below is my code that replicates the issue exactly. If you run this code it shows that with 2 processes it takes roughly twice as long as running it with 1 process but really it should take the same amount of time.

import time
import torch.multiprocessing as mp
import gym
import numpy as np
import copy
import torch.nn as nn
import torch

class NN(nn.Module):
    def __init__(self, output_dim):
        nn.Module.__init__(self)
        self.fc1 = nn.Linear(4, 50)
        self.fc2 = nn.Linear(50, 500)
        self.fc3 = nn.Linear(500, 5000)
        self.fc4 = nn.Linear(5000, output_dim)
        self.relu = nn.ReLU()

    def forward(self, x):
        x = self.relu(self.fc1(x))
        x = self.relu(self.fc2(x))
        x = self.relu(self.fc3(x))
        x = self.fc4(x)
        return x

def Worker(ix):
  print("Starting training for worker ", ix)
  env = gym.make('CartPole-v0')
  model = NN(2)
  for _ in range(2000):
    model(torch.Tensor(env.reset()))
  print("Finishing training for worker ", ix)

def overall_process(num_workers):
  workers = []
  for ix in range(num_workers):
    worker = mp.Process(target=Worker, args=(ix, ))
    workers.append(worker)
  [w.start() for w in workers]
  for worker in workers:
    worker.join()

  print("Finished Training")
  print(" ")

start = time.time()
overall_process(1)
print("Time taken: ", time.time() - start)
print(" ")

start = time.time()
overall_process(2)
print("Time taken: ", time.time() - start)

Does anyone know why this might be happening and how to fix it?

I thought that it is maybe because PyTorch networks automatically implement CPU parallelism in the background and so I tried adding the below 2 lines but it doesn’t always resolve the issue:

torch.set_num_threads(1)
torch.set_num_interop_threads(1)


title: cuda runtime error (3) : initialization error at /pytorch/aten/src/THC/THCGeneral.cpp:51

body:
I start to run TensorFlow application on terminal. At the same time, when I start Pytorch application on another terminal I get error

THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=51  error=3 : initialization error
terminate called after throwing an instance of 'std::runtime_error'
what():  cuda runtime error (3) : initialization error at /pytorch /aten/src/THC/THCGeneral.cpp:51

Both Pytorch TensorFlow and running on different virtualenvs

My Environment
-Ubuntu 18.04

-GPU GeForce GTX 1060

-Pytorch env (torch==1.1.0, torchvision==0.2.0)

-Tensorflow env (tensorflow-gpu==1.15.0)

Pytorch application was running smoothly before starting TensorFlow application

I stop the TensorFlow application and check

>>> torch.cuda.device_count()
0
>>> torch.cuda.is_available()
False

But error not goes back


title: How to avoid "RuntimeError: error in LoadLibraryA" for torch.cat?

body:
I am running a  pytorch solution for wireframe detection. I am receiving a "RuntimeError: error in LoadLibraryA" when the solution executes "forward return torch.cat(outputs, 1)"

I am not able to provide a minimal re-producable example. Therefore the quesion: Is it possible to produce just type of error in a microsoft library by python programming errors, or is this most likely a version (of python, pytorch, CUDA,...) problem or a bug in my installation?

I am using windows 10, python 3.8.1 and pytorch 1.4.0.

File "main.py", line 144, in <module>
  main()
File "main.py", line 137, in main
  trainer.train(train_loader, val_loader=None)
File "D:\Dev\Python\Projects\wireframe\wireframe\junc\trainer\balance_junction_trainer.py", line 75, in train
  self.step(epoch, train_loader)
File "D:\Dev\Python\Projects\wireframe\wireframe\junc\trainer\balance_junction_trainer.py", line 176, in step
  ) = self.model(input_var, junc_conf, junc_res, bin_conf, bin_res)
File "D:\Dev\Python\Environment\Environments\pytorch\lib\site-packages\torch\nn\modules\module.py", line 532, in __call__
  result = self.forward(*input, **kwargs)
File "D:\Dev\Python\Projects\wireframe\wireframe\junc\model\inception.py", line 41, in forward
  base_feat = self.base_net(im_data)
File "D:\Dev\Python\Environment\Environments\pytorch\lib\site-packages\torch\nn\modules\module.py", line 532, in __call__
  result = self.forward(*input, **kwargs)
File "D:\Dev\Python\Projects\wireframe\wireframe\junc\model\networks\inception_v2.py", line 63, in forward
  x = self.Mixed_3b(x)
File "D:\Dev\Python\Environment\Environments\pytorch\lib\site-packages\torch\nn\modules\module.py", line 532, in __call__
  result = self.forward(*input, **kwargs)
File "D:\Dev\Python\Projects\wireframe\wireframe\junc\model\networks\inception_v2.py", line 97, in forward
  return torch.cat(outputs, 1)
RuntimeError: error in LoadLibraryA


title: Can't import sklearn ([WinError 126] The specified module could not be found)

body:
from sklearn import datasets
I can't import sklearn
here is the error:
OSError                                   Traceback (most recent call last)

<ipython-input-13-f9e6334b9a20> in <module>
  1 import torch
  2 import numpy as np
   ----> 3 from sklearn import datasets
  4
  5 X_numpy, y_numpy = datasets.make_regression(n_samples=100, n_features=1, noise=20,
  random_state=1)

 c:\users\kadiem alqazzaz\appdata\local\programs\python\python37\lib\site-
packages\sklearn\__init__.py in <module>
 78     from . import _distributor_init  # noqa: F401
 79     from . import __check_build  # noqa: F401
---> 80     from .base import clone
 81     from .utils._show_versions import show_versions
 82

c:\users\kadiem alqazzaz\appdata\local\programs\python\python37\lib\site-packages\sklearn\base.py in <module>
 19 from . import __version__
 20 from ._config import get_config
---> 21 from .utils import _IS_32BIT
 22 from .utils.validation import check_X_y
 23 from .utils.validation import check_array

c:\users\kadiem alqazzaz\appdata\local\programs\python\python37\lib\site-packages\sklearn\utils\__init__.py in <module>
 18 import warnings
 19 import numpy as np
---> 20 from scipy.sparse import issparse
 21
 22 from .murmurhash import murmurhash3_32

c:\users\kadiem alqazzaz\appdata\local\programs\python\python37\lib\site-packages\scipy\__init__.py in <module>
102
103 # Allow distributors to run custom init code
--> 104 from . import _distributor_init
105
106 __all__ += _num.__all__

c:\users\kadiem alqazzaz\appdata\local\programs\python\python37\lib\site-packages\scipy\_distributor_init.py in <module>
 59             os.chdir(libs_path)
 60             for filename in glob.glob(os.path.join(libs_path, '*dll')):
 ---> 61                 WinDLL(os.path.abspath(filename))
 62         finally:
 63             os.chdir(owd)

 c:\users\kadiem alqazzaz\appdata\local\programs\python\python37\lib\ctypes\__init__.py in __init__(self, name, mode, handle, use_errno, use_last_error)
362
363         if handle is None:
--> 364             self._handle = _dlopen(self._name, mode)
365         else:
366             self._handle = handle

OSError: [WinError 126] The specified module could not be found

when I type pip install sklearn I get Requirement already satisfied:
Requirement already satisfied: sklearn in c:\users\kadiem alqazzaz\appdata\local\programs\python\python37\lib\site-packages (0.0) Requirement already satisfied: scikit-learn in c:\users\kadiem alqazzaz\appdata\local\programs\python\python37\lib\site-packages (from sklearn) (0.23.1) Requirement already satisfied: joblib>=0.11 in c:\users\kadiem alqazzaz\appdata\local\programs\python\python37\lib\site-packages (from scikit-learn->sklearn) (0.15.1) Requirement already satisfied: threadpoolctl>=2.0.0 in c:\users\kadiem alqazzaz\appdata\local\programs\python\python37\lib\site-packages (from scikit-learn->sklearn) (2.1.0) Requirement already satisfied: numpy>=1.13.3 in c:\users\kadiem alqazzaz\appdata\local\programs\python\python37\lib\site-packages (from scikit-learn->sklearn) (1.18.5) Requirement already satisfied: scipy>=0.19.1 in c:\users\kadiem alqazzaz\appdata\local\programs\python\python37\lib\site-packages (from scikit-learn->sklearn) (1.5.0)
so why can't I use sklearn, any help is appreciated thanks.
EDIT: when I import sklearn in cmd it works with no issues, however, when I run using Jupyter Notebook I get the error above.


title: Float16 (HalfTensor) in pytorch + cuda

body:
Can I set torch.HalfTensor as default and use it with CUDA?

I can't even create usual Conv2D:

In [1]: import torch
In [2]: torch.__version__
Out[2]: '0.2.0_3'

In [3]: from torch import nn

In [4]: torch.set_default_tensor_type('torch.HalfTensor')

In [5]: conv1 = nn.Conv2d(1, 64, kernel_size=3, stride=1, padding=1, bias=False)
terminate called after throwing an instance of 'std::invalid_argument'
  what():  Unsupported tensor type
Aborted (core dumped)

UPD: it doesn't work without CUDA too but I'm interested in case with CUDA support.


title: Highlighting important words in a sentence using Deep Learning

body:
I am trying to highlight important words in imdb dataset which contributed finally to the sentiment analysis prediction .

The dataset is like :

X_train - A review as string .

Y_train - 0 or 1

Now after using Glove embeddings for embedding the X_train value I can feed it to a neural net .

Now my question is , how can I highlight the most important words probability wise ? just like deepmoji.mit.edu ?

What have I tried :

I tried splitting the input sentences into bi-grams and using a 1D CNN to train it . Later when we want to find the important words of X_test , we just split the X_test in bigrams and find their probabilities . It works but not accurate .
I tried using prebuilt Hierarchical Attention Networks and succeeded . I got what I wanted but I couldn't figure out every line and concepts from the code .It's like a black box to me .

I know how a neural net works and I can code it using numpy with manual back propagation from scratch . I have detailed knowledge of how a lstm works and what forget , update , and output gates actually outputs . But I couldn't still figure out how to extract attention weights and how to make the data as a 3D array ( what is the timestep in our 2D data ? )

So , any type of guidance is welcome


title: PyTorch: What does @weak_script_method decorator do?

body:
In the torch.nn.Linear class (and other classes too), the forward method includes a @weak_script_method decorator as follows:

@weak_script_method
def forward(self, input):
    return F.linear(input, self.weight, self.bias)

What does this decorator do? Should I include it if I'm overriding the forward method in my own subclass of the Linear module?


title: Add multiple tensors inplace in PyTorch

body:
I can add two tensors x and y inplace like this

x = x.add(y)

Is there a way of doing the same with three or more tensors given all tensors have same dimensions?


title: Pylint with pytorch: Is there a way to tell pylint to look for module in different place?

body:
I am using pytorch and pylint does not recognize few functions for ex: torch.stack however, if I do import torch._C as torch it seems to work fine.

If I do above, actual modules that exist inside torch package like torch.cuda or torch.nn need to imported individually as simply doing torch.cuda would point to torch._C.cuda and hence won't work.

Is there a way to tell pylint to look at both torch and torch._C when I do import torch or even whenever it sees torch? I don't think I would use torch to reference any other thing in my code.


title: Error when converting PyTorch model to TorchScript

body:
I'm trying to follow the PyTorch guide to load models in C++.

The following sample code works:

import torch
import torchvision

# An instance of your model.
model = torchvision.models.resnet18()

# An example input you would normally provide to your model's forward() method.
example = torch.rand(1, 3, 224, 224)

# Use torch.jit.trace to generate a torch.jit.ScriptModule via tracing.
traced_script_module = torch.jit.trace(model, example)

However, when trying other networks, such as squeezenet (or alexnet), my code fails:

sq = torchvision.models.squeezenet1_0(pretrained=True)
traced_script_module = torch.jit.trace(sq, example)

>> traced_script_module = torch.jit.trace(sq, example)
/home/fabio/.local/lib/python3.6/site-packages/torch/jit/__init__.py:642: TracerWarning: Output nr 1. of the traced function does not match the corresponding output of the Python function.
 Detailed error:
Not within tolerance rtol=1e-05 atol=1e-05 at input[0, 785] (3.1476082801818848 vs. 3.945478677749634) and 999 other locations (100.00%)
  _check_trace([example_inputs], func, executor_options, module, check_tolerance, _force_outplace)


title: Can you reverse a PyTorch neural network and activate the inputs from the outputs?

body:
Can we activate the outputs of a NN to gain insight into how the neurons are connected to input features?

If I take a basic NN example from the PyTorch tutorials. Here is an example of a f(x,y) training example.

import torch

N, D_in, H, D_out = 64, 1000, 100, 10

x = torch.randn(N, D_in)
y = torch.randn(N, D_out)

model = torch.nn.Sequential(
    torch.nn.Linear(D_in, H),
    torch.nn.ReLU(),
    torch.nn.Linear(H, D_out),
)

loss_fn = torch.nn.MSELoss(reduction='sum')

learning_rate = 1e-4
for t in range(500):
    y_pred = model(x)
    loss = loss_fn(y_pred, y)
    model.zero_grad()
    loss.backward()
    with torch.no_grad():
        for param in model.parameters():
            param -= learning_rate * param.grad

After I've finished training the network to predict y from x inputs. Is it possible to reverse the trained NN so that it can now predict x from y inputs?

I don't expect y to match the original inputs that trained the y outputs. So I expect to see what features the model activates on to match x and y.

If it is possible, then how do I rearrange the Sequential model without breaking all the weights and connections?


title: How to normalize images in PyTorch

body:
transform = transforms.Compose([
    transforms.ToTensor()
])

trainset = torchvision.datasets.ImageFolder(root='C:/Users/beomseokpark/Desktop/CNN/train_data', transform = transform)
data_loader = DataLoader(dataset = trainset, batch_size = 8, shuffle = True, num_workers=2)

with torch.no_grad():
    for num, data in enumerate(trainset):
        imgs, label = data

I loaded images with ImageFolder in torchvision library, and how can I get mean and std from each channel of my images?

Can anyone please help me out?


title: How to find which variable is causing RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation

body:
I'm trying to create an extremely simple network with a GRUcell layer to perform the following task: a cue is given in one of two locations. After T timesteps, the agent must learn to take a particular action at the opposite location.

I get the following error when trying to compute backward gradients:

  RuntimeError: one of the variables needed for gradient computation has
  been modified by an inplace operation.

One problem is that I don't fully understand which piece of my code is performing the inplace operation.

I have read other posts on stackoverflow and on the pytorch forum, which all suggest using the .clone() operation. I've peppered it all over my code anywhere I think it could conceivably make a difference, but I haven't had success.

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.gru    = nn.GRUCell(2,50) # GRU layer taking 2 inputs (L or R), has 50 units
        self.actor  = nn.Linear(50,2)  # Linear Actor layer with 2 outputs, takes GRU as input
        self.critic = nn.Linear(50,1)  # Linear Critic layer with 1 output, takes GRU as input

    def forward(self, s, h):
        h  = self.gru(s,h)  # give the input and previous hidden state to the GRU layer
        c  = self.critic(h) # estimate the value of the current state
        pi = F.softmax(self.actor(h),dim=1) # calculate the policy
        return (h,c,pi)

    def backward_rollout(self, gamma, R_t, c_t, t):
        R_t[0,t] = gamma*R_t[0,t+1].clone()
        # calculate the reward prediction error
        Delta_t[0,t] = c_t[0,t].clone() - R_t[0,t].clone()

        #calculate the loss for the critic
        crit = c_t[0,t].clone()
        ret  = R_t[0,t].clone()
        Value_l[0,t] = F.smooth_l1_loss(crit,ret)

###################################
# Run a trial

# parameters
N      = 1   # number of trials to run
T      = 10   # number of time-steps in a trial
gamma  = 0.98 # temporal discount factor

# for each trial
for n in range(N):
    sample  = np.random.choice([0,1],1)[0] # pick the sample input for this trial
    s_t     = torch.zeros((1,2,T))   # state at each time step

    h_0     = torch.zeros((1,50))    # initial hidden state
    h_t     = torch.zeros((1,50,T))  # hidden state at each time step

    c_t     = torch.zeros((1,T))    # critic at each time step
    pi_t    = torch.zeros((1,2,T))  # policy at each time step

    R_t     = torch.zeros((1,T))  # return at each time step
    Delta_t = torch.zeros((1,T))  # difference between critic and true return at each step
    Value_l = torch.zeros((1,T))  # value loss

    # set the input (state) vector/tensor
    s_t[0,sample,0] = 1.0 # set first time-step stimulus
    s_t[0,0,-1]     = 1.0 # set last time-step stimulus
    s_t[0,1,-1]     = 1.0 # set last time-step stimulus

    # step through the trial
    for t in range(T):
        # run a forward step
        state = s_t[:,:,t].clone()
        if t is 0:
            (hidden_state, critic, policy) = net(state, h_0)

        else:
            (hidden_state, critic, policy) = net(state, h_t[:,:,t-1])

        h_t[:,:,t]  = hidden_state.clone()
        c_t[:,t]    = critic.clone()
        pi_t[:,:,t] = policy.clone()

    # select an action using the policy
    action = np.random.choice([0,1], 1, p = policy[0,:].detach().numpy())
    #action = int(np.random.uniform() < pi[0,1])

    # compare the action to the sample
    if action is sample:
        r = 0
        print("WRONG!")
    else:
        r = 1
        print("RIGHT!")

    #h_t_old = h_t
    #s_t_old = s_t

    # step backwards through the trial to calculate gradients
    R_t[0,-1]     = r
    Delta_t[0,-1] = c_t[0,-1].clone() - r
    Value_l[0,-1] = F.smooth_l1_loss(c_t[0,-1],R_t[0,-1]).clone()

    for t in np.arange(T-2,-1,-1): #backwards rollout
        net.backward_rollout(gamma, R_t, c_t, t)

    Vl = Value_l.clone().sum()#calculate total loss

    Vl.backward() #calculate the derivatives
    opt.step() #update the weights
    opt.zero_grad() #zero gradients before next trial


title: tensorflow equivalent of torch.gather

body:
I have a tensor of shape (16, 4096, 3). I have another tensor of indices of shape (16, 32768, 3). I am trying to collect the values along dim=1. This was initially done in pytorch using gather function as shown below-

# a.shape (16L, 4096L, 3L)
# idx.shape (16L, 32768L, 3L)
b = a.gather(1, idx)
# b.shape (16L, 32768L, 3L)

Please note that the size of output b is the same as that of idx. However, when I apply gather function of tensorflow, I get a completely different output. The output dimension was found mismatching as shown below-

b = tf.gather(a, idx, axis=1)
# b.shape (16, 16, 32768, 3, 3)

I also tried using tf.gather_nd but got in vain. See below-

b = tf.gather_nd(a, idx)
# b.shape (16, 32768)

Why am I getting different shapes of tensors? I want to get the tensor of the same shape as calculated by pytorch.

In other words, I want to know the tensorflow equivalent of torch.gather.


title: Set higher shared memory to avoid RuntimeError with PyTorch on Google Colab

body:
Using pytorch 1.0 Preview with fastai v1.0 in Colab.

I often get RuntimeError: DataLoader worker (pid 13) is killed by signal: Bus error. for more memory intensive tasks (nothing huge).

Looks like a shared memory issue:
https://github.com/pytorch/pytorch/issues/5040#issue-294274594

Fix looks like it is to change shared memory in the docker container:

https://github.com/pytorch/pytorch/issues/2244#issuecomment-318864552

  Looks like the shared memory of the docker container wasn't set high enough. Setting a higher amount by adding --shm-size 8G to the docker run command seems to be the trick as mentioned here.

How can I increase the shared memory of the docker container running in Colab or otherwise avoid this error?


title: Pytorch loss function with ignore_index cause that model doesn't converge

body:
I was going through Pytorch official tutorial regarding the implementation of seq2eq Chatbot with Attention. To gain better understanding I used different methods to achieve similar results. One of as I thought improvements was the implementation of simple nn.NLLLoss function with attribute ignore_index = padding_token. Previous solution that used binary mask tensor describing the padding of the target tensor seemed to be too complicated, I wanted a simpler solution, but as it tuned out adding few lines of code made everything worse. Now model doesn't converge at all. I tried to change learning rate, but it didn’t help.

Pytorch chatbot tutirial: https://pytorch.org/tutorials/beginner/chatbot_tutorial.html

criterion = nn.NLLLoss(ignore_index=PAD_token, reduction='mean')

In the definition of train function I only made small change in the loop when we forward propagate through decoder one step at a time:

if use_teacher_forcing:
    for t in range(max_target_len):
        decoder_output, decoder_hidden = decoder(
            decoder_input, decoder_hidden, encoder_outputs
        )
        # Teacher forcing: next input is current target
        decoder_input = target_variable[t].view(1, -1)

        # Calculate and accumulate loss

        loss = criterion(decoder_output, target_variable[t])
        total_loss += loss
        print_losses.append(loss.item())

# Perform backpropatation
total_loss.backward()

# Clip gradients: gradients are modified in place
_ = nn.utils.clip_grad_norm_(encoder.parameters(), clip)
_ = nn.utils.clip_grad_norm_(decoder.parameters(), clip)

# Adjust model weights
encoder_optimizer.step()
decoder_optimizer.step()

return sum(print_losses)/max_target_len

Output:
Iteration: 1; Percent complete: 0.0%; Average loss: 8.5246
Iteration: 2; Percent complete: 0.1%; Average loss: 8.5403
Iteration: 3; Percent complete: 0.1%; Average loss: 8.5663
Iteration: 4; Percent complete: 0.1%; Average loss: 8.5691
Iteration: 5; Percent complete: 0.1%; Average loss: 8.4637
Iteration: 6; Percent complete: 0.1%; Average loss: 8.5378
Iteration: 7; Percent complete: 0.2%; Average loss: 8.5575
Iteration: 8; Percent complete: 0.2%; Average loss: 8.5145
Iteration: 9; Percent complete: 0.2%; Average loss: 8.5717
Iteration: 10; Percent complete: 0.2%; Average loss: 8.5122
Iteration: 11; Percent complete: 0.3%; Average loss: 8.5587
Iteration: 12; Percent complete: 0.3%; Average loss: 8.5491
Iteration: 13; Percent complete: 0.3%; Average loss: 8.5933
Iteration: 14; Percent complete: 0.4%; Average loss: 8.5231
Iteration: 15; Percent complete: 0.4%; Average loss: 8.5239

… and so on

Total loss increase or fluctuate ate the same value, but it doesn’t want to decrease. Did anybody try to use any kind of loss functions with ignore_index in seq2seq model with variate length sequences and has some positive results? nn.CrossEntropyLoss behaves similarly. I am starting to think that the loss with ignore_index parameter isn't calculated properly...

I would like to emphasize that tensors passed to loss function have the following dimensions:

loss = criterion(decoder_output, target_variable[t])
# decoder_output: (batch_size, vocab_size)
# target_variable[t]: (batch_size)

I think that these are proper dimensions, but maybe I should reshape these tensors? In original code they calculate negative log manually passing to it “concatenated” tensors with final shape:

    crossEntropy = -torch.log(torch.gather(inp, 1, target.view(-1, 1)).squeeze(1)) # shape (batch_size)

One additional question. In my version I used:

packed = nn.utils.rnn.pack_padded_sequence(embedded, input_lengths, batch_first=True, enforce_sorted=False)

thus I didn’t sort batch in descending order according to sequence lengths. I noticed that my code process slower than original, could be that cause by using unsorted batch, or it should work well anyway?

Any suggestion would be very appreciated.

Best regards.


title: Pytorch - Porting @ Operator

body:
I have the following line of code I want to port to Torch Matmul

rotMat = xmat @ ymat @ zmat

Can I know if this is the correct ordering:

rotMat = torch.matmul(xmat, torch.matmul(ymat, zmat))


title: Is maxpooling on odd number possible?

body:
I am going through the Udacity DeepLearning Nanodegree and working on the autoencoder mini project.  I do not understand the solution, nor how to check it myself.  So this is 2 questions.

We start with 28*28 images.  These are fed through 3 convolutional layers, each with padding of 1, and each with a maxpooling to half the original dimensions.  What I don't understand is the last element?  Surely 2 rounds of maxpooling (28/2)/2 gives 7 and therefore a further maxpooling shouldn't be possible as it results in an odd number.  Can someone explain why this is the case to me?  The code to replicate is here:
'''

import torch
import numpy as np
from torchvision import datasets
import torchvision.transforms as transforms

# convert data to torch.FloatTensor
transform = transforms.ToTensor()

# load the training and test datasets
train_data = datasets.MNIST(root='data', train=True,
                                   download=True, transform=transform)
test_data = datasets.MNIST(root='data', train=False,
                                  download=True, transform=transform)

# Create training and test dataloaders
num_workers = 0
# how many samples per batch to load
batch_size = 20

# prepare data loaders
train_loader = torch.utils.data.DataLoader(train_data, batch_size=batch_size, num_workers=num_workers)
test_loader = torch.utils.data.DataLoader(test_data, batch_size=batch_size, num_workers=num_workers)

import torch.nn as nn
import torch.nn.functional as F

# define the NN architecture
class ConvDenoiser(nn.Module):
    def __init__(self):
        super(ConvDenoiser, self).__init__()
        ## encoder layers ##
        # conv layer (depth from 1 --> 32), 3x3 kernels
        self.conv1 = nn.Conv2d(1, 32, 3, padding=1)
        # conv layer (depth from 32 --> 16), 3x3 kernels
        self.conv2 = nn.Conv2d(32, 16, 3, padding=1)
        # conv layer (depth from 16 --> 8), 3x3 kernels
        self.conv3 = nn.Conv2d(16, 8, 3, padding=1)
        # pooling layer to reduce x-y dims by two; kernel and stride of 2
        self.pool = nn.MaxPool2d(2, 2)

        ## decoder layers ##
        # transpose layer, a kernel of 2 and a stride of 2 will increase the spatial dims by 2
        self.t_conv1 = nn.ConvTranspose2d(8, 8, 3, stride=2)  # kernel_size=3 to get to a 7x7 image output
        # two more transpose layers with a kernel of 2
        self.t_conv2 = nn.ConvTranspose2d(8, 16, 2, stride=2)
        self.t_conv3 = nn.ConvTranspose2d(16, 32, 2, stride=2)
        # one, final, normal conv layer to decrease the depth
        self.conv_out = nn.Conv2d(32, 1, 3, padding=1)

    def forward(self, x):
        ## encode ##
        # add hidden layers with relu activation function
        # and maxpooling after
        x = F.relu(self.conv1(x))
        x = self.pool(x)
        # add second hidden layer
        x = F.relu(self.conv2(x))
        x = self.pool(x)
        # add third hidden layer
        x = F.relu(self.conv3(x))
        x = self.pool(x)  # compressed representation

        ## decode ##
        # add transpose conv layers, with relu activation function
        x = F.relu(self.t_conv1(x))
        x = F.relu(self.t_conv2(x))
        x = F.relu(self.t_conv3(x))
        # transpose again, output should have a sigmoid applied
        x = F.sigmoid(self.conv_out(x))

        return x

# initialize the NN
model = ConvDenoiser()
print(model)

I wanted to try to understand this by passing a single image through the layers manually and see what the result was but this resulted in an error.  Can someone explain to me how I can see the shapes that pass through the layers?  Code is a bit messy but I left it there so you can see what I tried.

dataiter = iter(train_loader)
images, labels = dataiter.next()
# images = images.numpy()

# get one image from the batch
# img = np.squeeze(images[0])
img=images[0]

#create hidden layer
conv1 = nn.Conv2d(1, 32, 3, padding=1)

# z=torch.from_numpy(images[0])
z1=conv1(img)

Appreciate any insights you can give me.
Thanks,
J


title: How to check which pytorch version fit torchvision

body:
I am trying to clone and run this repository:
https://github.com/switchablenorms/CelebAMask-HQ
The demo runs pytorh 0.4.1
I am trying to find the corresponding version, how can I find it?


title: Pytorch: Softmax on each row of a 2d Tensor

body: I need my neural net to output N distributions over A actions. each distribution should go through softmax. The sum of each row should then obviously be 1 and the sum of the whole layer should be N. Is there such functionality in PyTorch?


title: How can I disable gradient updates for some modules in autograd backpropagation?

body:
I'm building a multi-model neural network for reinforcement learning to include an action network, a world model network, and a critic. The idea is train the world model to emulate whatever simulation you are trying to master based on input from the action network and the previous state, to train the critic to maximize the Bellman equation (total reinforcement over time) based on the world model output, and then backpropagate the critic value through the world model to provide gradient targets for training the actions. So - from some state, the action network outputs an action which is fed into the model to generate the next state, and that state feeds into the critic network for evaluation against some goal state.

For all this to work, I must use 3 separate loss functions, one for each network, and they all add something to the gradients in one or more networks but they can be in conflict. For example - to train the world model I use a target from an environmental simulation and for the critic I use a target of the current state reward + discount * next state forecast value. However, to train the a actor I just use the negative critic value as a loss and backpropagate all the way through all three models to calibrate the best action.

I can make this work without any batching by zeroing out gradients incrementally, but that is inefficient and doesn't let me accumulate gradients for any kind of "time-series batching" optimizer update step. Each model has it's own trainable parameters, but the execution graph flows through all three networks. So inside the calibration loop after firing the networks in sequence:
...

        if self.actor.calibrating:
            self.actor.optimizer.zero_grad()
            #Pick loss For maximizing the value of all actions
            loss = -self.critic.value
            #Backpropagate through all three networks to train actor output
            #How do I stop the critic and model networks from incrementing their gradient values?
            loss.backward(retain_graph=True)
            self.actor.optimizer.step()
        if self.model.calibrating:
            self.model.optimizer.zero_grad()
            #Reduce loss for ambiguous actions
            loss = self.model.get_loss() * self.actor.get_confidence()**2
            #How can I block this from backpropagating through action network?
            loss.backward(retain_graph=True)
            self.model.optimizer.step()
        if self.critic.calibrating:
            self.critic.optimizer.zero_grad()
            #Reduce loss for ambiguous actions
            loss = self.critic.get_loss(self.goal) * self.actor.get_confidence()**2
            #How do I stop this from backpropagating through the model and action networks?
            loss.backward(retain_graph=True)
            self.critic.optimizer.step()

...

Finally - my question is in two parts:

How can I temporarily stop loss.backward() at a given layer without detaching it forever?
How can I block loss.backward() from updating some gradients where I'm just flowing through a model to get gradients for another model?


title: pytorch affine_grid: what is the theta input?

body:
When trying to use torch.nn.functional.affine_grid, it requires a theta affine matrix of size (N x 3 x 4) according to the documentation. I thought a general affine matrix is (N x 4 x 4). What is the supposed affine matrix format in pytorch?

An example of 3D rotation affine input would be ideal. Appreciate your help.


title: How to work with large dataset in pytorch

body:
I have a huge dataset that does not fit in memory (150G) and I'm looking for the best way to work with it in pytorch. The dataset is composed of several .npz files of 10k samples each. I tried to build a Dataset class

class MyDataset(Dataset):
    def __init__(self, path):
        self.path = path
        self.files = os.listdir(self.path)
        self.file_length = {}
        for f in self.files:
            # Load file in as a nmap
            d = np.load(os.path.join(self.path, f), mmap_mode='r')
            self.file_length[f] = len(d['y'])

    def __len__(self):
        raise NotImplementedException()

    def __getitem__(self, idx):
        # Find the file where idx belongs to
        count = 0
        f_key = ''
        local_idx = 0
        for k in self.file_length:
            if count < idx < count + self.file_length[k]:
                f_key = k
                local_idx = idx - count
                break
            else:
                count += self.file_length[k]
        # Open file as numpy.memmap
        d = np.load(os.path.join(self.path, f_key), mmap_mode='r')
        # Actually fetch the data
        X = np.expand_dims(d['X'][local_idx], axis=1)
        y = np.expand_dims((d['y'][local_idx] == 2).astype(np.float32), axis=1)
        return X, y

but when a sample is actually fetched, it takes more than 30s. It looks like the entire .npz is opened, stocked in RAM and it accessed the right index.
How to be more efficient ?

EDIT

It appears to be a misunderstading of .npz files see post, but is there a better approach ?

SOLUTION PROPOSAL

As proposed by @covariantmonkey, lmdb can be a good choice. For now, as the problem comes from .npz files and not memmap, I remodelled my dataset by splitting .npz packages files into several .npy files. I can now use the same logic where memmap makes all sense and is really fast (several ms to load a sample).


I have following code for train function for training an A3C.
I am stuck with following error.
RuntimeError("grad can be implicitly created only for scalar outputs")

at line (policy_loss + 0.5 * value_loss).backward()
Here is my code, can someone please help to check what is wrong with this code.
def train(rank, params, shared_model, optimizer,ticker):
torch.manual_seed(params.seed + rank) # shifting the seed with rank to asynchronize each training agent
print(ticker)
try:
    ohlcv = pd.read_csv(ticker + '.csv')
    data = ohlcv.copy()
    data['rsi'] = ab.RSI(data['Close'],14)
    data['adx'] = ab.ADX(data,20)
    data = ab.BollBnd(data,20)
    data['BBr'] = data['Close']/data['BB_dn']
    data['macd_signal'] = ab.MACD(data)
    data['macd_r'] = data['macd_signal']/data['Close']
    data['ema20'] = ab.EMA(np.asarray(data['Close']), 20)
    data['ema20_r'] = data['ema20']/data['Close']
    data['Close'] = data['Close']/data['Close'].max()
    data = data.iloc[:,[4,7,8,13,15,17]]
    data = data.dropna()
    data = torch.DoubleTensor(np.asarray(data))
    env = ENV(state_dim, action_dim, data)

    model = ActorCritic(env.observation_space, env.action_space)
    state = env.reset()
    done = True
    episode_length = 0
    while True:
        episode_length += 1
        model.load_state_dict(shared_model.state_dict())
        if done:
            cx = Variable(torch.zeros(1, state_dim)) # the cell states of the LSTM are reinitialized to zero
            hx = Variable(torch.zeros(1, state_dim)) # the hidden states of the LSTM are reinitialized to zero
        else:
            cx = Variable(cx.data)
            hx = Variable(hx.data)
        values = []
        log_probs = []
        rewards = []
        entropies = []
        for step in range(params.num_steps):
            value, action_values, (hx, cx) = model((Variable(state.unsqueeze(0)), (hx, cx)))
            prob = F.softmax(action_values,-1)
            log_prob = F.log_softmax(action_values,-1)
            entropy = -(log_prob * prob).sum(1)
            entropies.append(entropy)
            action = prob.multinomial(num_samples = action_dim).data
            log_prob = log_prob.gather(1, Variable(action))
            values.append(value)
            log_probs.append(log_prob)
            state, reward, done = env.step(action)
            done = (done or episode_length >= params.max_episode_length)
            reward = max(min(reward, 1), -1) # clamping the reward between -1 and +1
            if done:
                episode_length = 0
                state = env.reset()
            rewards.append(reward)
            if done:
                break
        R = torch.zeros(1, 1)
        if not done: # if we are not done:
            value, _, _ = model((Variable(state.unsqueeze(0)), (hx, cx)))
            R = value.data
        values.append(Variable(R))
        policy_loss = torch.zeros(1, 1)
        value_loss = torch.zeros(1, 1)
        R = Variable(R)
        gae = torch.zeros(1, 1)
        for i in reversed(range(len(rewards))):
            R = params.gamma * R + rewards[i]
            advantage = R - values[i]
            print("advantage:",advantage)
            value_loss = value_loss + 0.5 * advantage.pow(2) # computing the value loss
            TD = rewards[i] + params.gamma * values[i + 1].data - values[i].data # computing the temporal difference
            gae = gae * params.gamma * params.tau + TD # gae = sum_i (gamma*tau)^i * TD(i) with gae_i = gae_(i+1)*gamma*tau + (r_i + gamma*V(state_i+1) - V(state_i))
            print("gae:",gae)
            policy_loss = policy_loss - log_probs[i] * Variable(gae) - 0.01 * entropies[i] # computing the policy loss
            print("policy_loss:",policy_loss)
        optimizer.zero_grad() # initializing the optimizer
        los = policy_loss + 0.5 * value_loss
        print("los",los.shape)
        (policy_loss + 0.5 * value_loss).backward()
        torch.nn.utils.clip_grad_norm(model.parameters(), 40) # clamping the values
        ensure_shared_grads(model, shared_model)
        optimizer.step() # running the optimization step
except Exception as e:
    print(e)
    traceback.print_exc()
    var = traceback.format_exc()

Below are the outputs:-
advantage: tensor([[-1.0750]], grad_fn=<ThSubBackward>)
gae: tensor([[-1.0750]])
policy_loss: tensor([[-25.8590, -26.1414, -25.9023, -25.2628]], grad_fn=<ThSubBackward>)
los torch.Size([1, 4])

RuntimeError: grad can be implicitly created only for scalar outputs
PS E:\ML\Breakout_a3c\Code_With_Comments>


title: Update step in PyTorch implementation of Newton's method

body:
I'm trying to get some insight into how PyTorch works by implementing Newton's method for solving x = cos(x). Here's a version that works:

x =  Variable(DoubleTensor([1]), requires_grad=True)

for i in range(5):
    y = x - torch.cos(x)
    y.backward()
    x = Variable(x.data - y.data/x.grad.data, requires_grad=True)

print(x.data) # tensor([0.7390851332151607], dtype=torch.float64) (correct)

This code seems inelegant (inefficient?) to me since it's recreating the entire computational graph during each step of the for loop (right?). I tried to avoid this by simply updating the data held by each of the variables instead of recreating them:

x =  Variable(DoubleTensor([1]), requires_grad=True)
y = x - torch.cos(x)
y.backward(retain_graph=True)

for i in range(5):
    x.data = x.data - y.data/x.grad.data
    y.data = x.data - torch.cos(x.data)
    y.backward(retain_graph=True)

print(x.data) # tensor([0.7417889255761136], dtype=torch.float64) (wrong)

Seems like, with DoubleTensors, I'm carrying enough digits of precision to rule out round-off error. So where's the error coming from?

Possibly related: The above snippet breaks without the retain_graph=True flag set at every step if the for loop. The error message I get if I omit it within the loop --- but retain it on line 3 --- is:
RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time. This seems like evidence that I'm misunderstanding something...


title: Compute gradients w.r.t. the values of embedding vectors in PyTorch

body:
I am trying to train a dual encoder LSTM model for a chatbot using PyTorch.

I defined two classes: the Encoder class defines the LSTM itself and the Dual_Encoder class applies the Encoder to both context and response utterances that I am trying to train on:

class Encoder(nn.Module):

    def __init__(self,
                 input_size,
                 hidden_size,
                 vocab_size,
                 num_layers = 1,
                 num_directions = 1,
                 dropout = 0,
                 bidirectional = False,
                 rnn_type = 'lstm'):

                 super(Encoder, self).__init__()

                 self.input_size = input_size
                 self.hidden_size = hidden_size
                 self.vocab_size = vocab_size
                 self.num_layers = 1
                 self.num_directions = 1
                 self.dropout = 0,
                 self.bidirectional = False

                 self.embedding = nn.Embedding(vocab_size, input_size, sparse = False, padding_idx = 0)
                 self.lstm = nn.LSTM(self.input_size, self.hidden_size, self.num_layers, batch_first=False, dropout = dropout, bidirectional=False).cuda()

                 self.init_weights()

    def init_weights(self):
        init.orthogonal(self.lstm.weight_ih_l0)

        init.uniform(self.lstm.weight_hh_l0, a=-0.01, b=0.01)

        embedding_weights = torch.FloatTensor(self.vocab_size, self.input_size).cuda()
        init.uniform(embedding_weights, a = -0.25, b= 0.25)

        id_to_vec, emb_dim = create_id_to_vec('/data/train_shuffled_onethousand.csv','/data/glove.6B.100d.txt')

        for id, vec in id_to_vec.items():
            embedding_weights[id] = vec

        del self.embedding.weight
        self.embedding.weight = nn.Parameter(embedding_weights)
        self.embedding.weight.requires_grad = True

        #self.embedding.weight.data.copy_(torch.from_numpy(self.embedding_weights))

    def forward(self, inputs):
        embeddings = self.embedding(inputs)
        outputs, hiddens = self.lstm(embeddings)
        return outputs, hiddens

#%%

class DualEncoder(nn.Module):

    def __init__(self, encoder):
         super(DualEncoder, self).__init__()
         self.encoder = encoder
         self.number_of_layers = 1
         #h_0 (num_layers * num_directions, batch, hidden_size):
         #tensor containing the initial hidden state for each element in the batch.
         #dual_hidden_size = self.encoder.hidden_size * self.encoder.num_directions

         M = torch.FloatTensor(self.encoder.hidden_size, self.encoder.hidden_size).cuda()

         init.normal(M)

         self.M = nn.Parameter(M, requires_grad = True)

    def forward(self, contexts, responses):
        #output (seq_len, batch, hidden_size * num_directions):
        #tensor containing the output features (h_t) from the last layer
        #of the RNN, for each t.

        #h_n (num_layers * num_directions, batch, hidden_size):
        #tensor containing the hidden state for t=seq_len
        context_out, context_hn = self.encoder(contexts)

        response_out, response_hn = self.encoder(responses)

        scores_list = []

        y_preds = None

        for e in range(999):
            context_h = context_out[e][-1].view(1, self.encoder.hidden_size)
            response_h = response_out[e][-1].view(self.encoder.hidden_size,1)

            dot_var = torch.mm(torch.mm(context_h, self.M), response_h)[0][0]

            dot_tensor = dot_var.data
            dot_tensor.cuda()

            score = torch.sigmoid(dot_tensor)
            scores_list.append(score)

        y_preds_tensor = torch.stack(scores_list).cuda()
        y_preds = autograd.Variable(y_preds_tensor).cuda()

        return y_preds

#%% TRAINING

torch.backends.cudnn.enabled = False
#%%
vocab = create_vocab('/data/train_shuffled_onethousand.csv')
vocab_len = len(vocab)
emb_dim = get_emb_dim('/data/glove.6B.100d.txt')
#%%

encoder_model = Encoder(
        input_size = emb_dim,
        hidden_size = 300,
        vocab_size = vocab_len)

encoder_model.cuda()
#%%
dual_encoder = DualEncoder(encoder_model)

dual_encoder.cuda()
#%%
loss_func = torch.nn.BCELoss()

loss_func.cuda()

learning_rate = 0.001
epochs = 100
#batch_size = 50

optimizer = optim.Adam(dual_encoder.parameters(),
                       lr = learning_rate)
#%%
for i in range(epochs):

    context_matrix, response_matrix, y = make_matrices('/data/train_shuffled_onethousand.csv')

    context_matrix = autograd.Variable(context_matrix, requires_grad=True).cuda()

    response_matrix = autograd.Variable(response_matrix, requires_grad=True).cuda()

    y_label = y.cuda()

    y_preds = dual_encoder(context_matrix, response_matrix)

    loss = loss_func(y_preds, y_label)

    if i % 10 == 0:
        print("Epoch: ", i, ", Loss: ", loss.data[0])

    #evaluation metrics...

    dual_encoder.zero_grad()
    loss.backward()
    torch.nn.utils.clip_grad_norm(dual_encoder.parameters(), 10)

    optimizer.step()

The following error occurs:

2018-01-06 06:07:02,148 INFO - result = self.forward(*input, **kwargs)
2018-01-06 06:07:02,148 INFO - File "all_scripts.py", line 258, in forward
2018-01-06 06:07:02,148 INFO - context_out, context_hn = self.encoder(contexts)
2018-01-06 06:07:02,149 INFO - File "/usr/local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 325, in __call__
2018-01-06 06:07:02,149 INFO - result = self.forward(*input, **kwargs)
2018-01-06 06:07:02,149 INFO - File "all_scripts.py", line 229, in forward
2018-01-06 06:07:02,149 INFO - embeddings = self.embedding(inputs)
2018-01-06 06:07:02,150 INFO - File "/usr/local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 325, in __call__
2018-01-06 06:07:02,150 INFO - result = self.forward(*input, **kwargs)
2018-01-06 06:07:02,150 INFO - File "/usr/local/lib/python3.6/site-packages/torch/nn/modules/sparse.py", line 103, in forward
2018-01-06 06:07:02,150 INFO - self.scale_grad_by_freq, self.sparse
2018-01-06 06:07:02,150 INFO - File "/usr/local/lib/python3.6/site-packages/torch/nn/_functions/thnn/sparse.py", line 40, in forward
2018-01-06 06:07:02,151 INFO - assert not ctx.needs_input_grad[0], "Embedding doesn't " \
2018-01-06 06:07:02,151 INFO - AssertionError: Embedding doesn't compute the gradient w.r.t. the indices

I do understand why the problem occurs (surely it makes no sense to compute the gradient w.r.t. the indices).
But I do not understand how to adjust the code so that it computes the gradients w.r.t. the content values of the embedding vectors.

All help highly appreciated!

(Also see the thread in the PyTorch forum)


title: Pytorch: Missing key(s) in state_dict

body:
I tried converting a pytorch model into deployment (ios) so I'm converting a model to either cafffe or Onnx (https://discuss.pytorch.org/t/typeerror-forward-missing-2-required-positional-arguments-cap-lens-and-hidden/20010/8). My initial error was only having 'one layer" inside an lstm yet I encountered another problem.

I tried implementing two layers (nlayers=2) by instantiating a new rnn object called text_encoder

enter image description here

Yet I'm given an error of some key(s) inside the state_dict is missing. This error doesn't occur for having one layer yet but I do get an error during the conversion (https://discuss.pytorch.org/t/typeerror-forward-missing-2-required-positional-arguments-cap-lens-and-hidden/20010/8)

I'm not sure if this is happening because the model I loaded only had one layer or another problem. How can I recover missing key's while adding a new layer? Or is that impossible? Original post (https://discuss.pytorch.org/t/missing-key-s-in-state-dict/20154) and (https://discuss.pytorch.org/t/typeerror-forward-missing-2-required-positional-arguments-cap-lens-and-hidden/20010/11)

enter image description here


title: Error relating to conversion from list to tensor in Pytorch

body:
There is a variable 'tmp' (3 dimension).

tmp = [torch.tensor([1]),torch.tensor([2,3])]
type(tmp) -> <class 'list'>
type(tmp[0]) -> <class 'torch.Tensor'>
type(tmp[0][0]) -> <class 'torch.Tensor'>

I want to convert 'tmp' into torch.Tensor type.
But, when I run this code below, an error occurs.

torch.Tensor(tmp)
>> ValueError: only one element tensors can be converted to Python scalars

How can I fix this?
torch.stack cannot be effective in this case because tensors in 'tmp' are not the same shape.


title: Hugginface transformers module not recognized by anaconda

body:
I am using Anaconda, python 3.7, windows 10.
I tried to install transformers by https://huggingface.co/transformers/ on my env.
I am aware that I must have either pytorch or TF installed, I have pytorch installed - as seen in anaconda navigator environments.
I would get many kinds of errors, depending on where (anaconda / prompt) I uninstalled and reinstalled pytorch and transformers. Last attempt using
conda install pytorch torchvision cpuonly -c pytorch and
conda install -c conda-forge transformers
I get an error:
from transformers import BertTokenizer
bert_tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True)

def tok(dataset):
    input_ids = []
    attention_masks = []
    sentences = dataset.Answer2EN.values
    labels = dataset.Class.values
    for sent in sentences:
        encoded_sent = bert_tokenizer.encode(sent,
                                             add_special_tokens=True,
                                             max_length = 64,
                                             pad_to_max_length =True)

TypeError: _tokenize() got an unexpected keyword argument
'pad_to_max_length'

Does anyone know a secure installation of transformers using Anaconda?
Thank you


title: Pytorch Hardware Requirement

body:
What is the minimum Computation Capability required by the latest PyTorch version?
I have Nvidia Geforce 820M with computation capability 2.1. How can I run PyTorch models on my GPU (if it doesn't support naturally)


title: Efficient way to implement matrix multiplication when one matrix is extremely wide?

body:
I need to multiply 3 matrices, A: 3000x100, B: 100x100, C: 100x3.6MM.  I currently am just using normal matrix multiplication in PyTorch

A_gpu = torch.from_numpy(A)
B_gpu = torch.from_numpy(B)
C_gpu = torch.from_numpy(C)
D_gpu = (A_gpu @ B_gpu @ C_gpu.t()).t()

C is very wide so the data reuse on gpu is limited but are there other ways to speed this up? I have a machine with 4x GPUs.


title: Modified PyTorch loss function BCEWithLogitsLoss returns NaNs

body:
I'm trying to solve a binary classification problem (target=0 and target=1) with an exception:
Some of my labels are classified as target=0.5 on purpose, and I wish to have zero loss for either classifying it as 0 or 1 (i.e both classes are "correct").
I tried to implement a custom loss from scratch, based on PyTorch's BCEWithLogitsLoss:
class myLoss(torch.nn.Module):

def __init__(self, pos_weight=1):
    super().__init__()
    self.pos_weight = pos_weight

def forward(self, input, target):
    epsilon = 10 ** -44
    my_bce_loss = -1 * (self.pos_weight * target * F.logsigmoid(input + epsilon)
                        + (1 - target) * log(1 - sigmoid(input) + epsilon))
    add_loss = (target - 0.5) ** 2 * 4
    mean_loss = (my_bce_loss * add_loss).mean()
    return mean_loss

epsilon was chosen so the log will be bounded to -100, as suggested in BCE loss.
However I'm still getting NaN errors, after several epochs:
Function 'LogBackward' returned nan values in its 0th output.

or
Function 'SigmoidBackward' returned nan values in its 0th output.

Any suggestions how can I correct my loss function? maybe by somehow inherit and modify forward function?

Update:
The way I call my custom loss function:
y = batch[:, -1, :].to(self.device, dtype=torch.float32)
y_pred_batch = self.model(x)

LossFun = myLoss(self.pos_weight)
batch_result.loss = LossFun.forward(y_pred_batch, y)

I use Temporal Convolutional Network model, implemented as follows:
out = self.conv1(x)
out = self.chomp1(out)
out = self.elu(out)
out = self.dropout1(out)
res = x if self.downsample is None else self.downsample(x)
return self.tanh(out + res)


I keep getting the error message below. I cannot seem to pinpoint to the tensor mentioned. Below you'll find the trainer.py and main.py modules. The model I am developing is GAN on CelebA dataset. I am running the code on a remote server so have spent a handful amount of time debugging my model.

This is the full error message:

Traceback (most recent call last):
  File "main.py", line 52, in <module>
    main(opt)
  File "main.py", line 47, in main
    trainer.train(train_loader)
  File "/home/path/trainer.py", line 45, in train
    d_loss_cls = F.binary_cross_entropy_with_logits(out_cls, label_org, size_average=False) / out_cls.size(0)
  File "/home/path/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py", line 2077, in binary_cross_entropy_with_logits
    return torch.binary_cross_entropy_with_logits(input, target, weight, pos_weight, reduction_enum)
RuntimeError: expected type torch.cuda.FloatTensor but got torch.FloatTensor

trainer.py

from tqdm import tqdm
import torch
import torch.nn.functional as F
from model import Discriminator, Generator
from tensorboardX import SummaryWriter

class Trainer():
    def __init__(self, opt):
        # Generator
        self.G = Generator(64, 5, 6)
        # Discriminator
        self.D = Discriminator(128, 64, 5, 6)
        # Generator optimizer
        self.g_optimizer = torch.optim.Adam(self.G.parameters(), opt.lr)
        self.d_optimizer = torch.optim.Adam(self.D.parameters(), opt.lr)
        self.opt = opt
        if self.opt.cuda:
            self.G = self.G.cuda()
            self.D = self.D.cuda()

    def train(self, data_loader):
        """Function to train the model
        """
        print('Training model')
        writer_d = SummaryWriter('runs/disc') # discriminator writer
        writer_g = SummaryWriter('runs/gen') # generator writer
        print('Start training...')
        for epoch in tqdm(range(self.opt.epochs)):
            for x_real, label_org in tqdm(data_loader):
                pass
                # Generate target domain labels randomly.
                rand_idx = torch.randperm(label_org.size(0))
                label_trg = label_org[rand_idx]
                c_org = label_org.clone()
                c_trg = label_org.clone()
                if self.opt.cuda:
                    x_real = x_real.cuda()           # Input images
                    c_org = c_org.cuda()             # Original domain labels
                    c_trg = c_trg.cuda()             # Target domain labels
                    label_org = label_org.cuda()     # Labels for computing classification loss
                    label_trg = label_trg.cuda()     # Labels for computing classification loss

                out_src, out_cls = self.D(x_real)
                d_loss_real = - torch.mean(out_src)
                d_loss_cls = F.binary_cross_entropy_with_logits(out_cls, label_org, size_average=False) / out_cls.size(0)

                # Compute loss with fake images
                x_fake = self.G(x_real, c_trg)
                out_src, out_cls = self.D(x_fake.detach())
                d_loss_fake = torch.mean(out_src)

                # Compute loss for gradient penalty
                alpha = torch.rand(x_real.size(0), 1, 1, 1).cuda()
                x_hat = (alpha * x_real.data + (1 - alpha) * x_fake.data).requires_grad_(True)
                out_src, _ = self.D(x_hat)

                # Backward and optimize
                d_loss = d_loss_real + d_loss_fake + d_loss_cls
                self.g_optimizer.zero_grad()
                self.d_optimizer.zero_grad()
                d_loss.backward()
                self.d_optimizer.step()

                if (i + 1) % 2 == 0:
                    # Original-to-target domain
                    x_fake = self.G(x_real, c_trg)
                    out_src, out_cls = self.D(x_fake)
                    g_loss_fake = - torch.mean(out_src)
                    g_loss_cls = F.binary_cross_entropy_with_logits(out_cls, label_trg, size_average=False) / out_cls.size(0)

                    # Target-to-original domain
                    x_reconst = self.G(x_fake, c_org)
                    g_loss_rec = torch.mean(torch.abs(x_real - x_reconst))

                    # Backward and optimize
                    g_loss = g_loss_fake + g_loss_rec
                    self.g_optimizer.zero_grad()
                    self.d_optimizer.zero_grad()
                    g_loss.backward()
                    self.g_optimizer.step()

                # write loss to tensorboard
                writer_d.add_scalar('data/loss', d_loss, epoch)
                writer_d.add_scalar('data/loss', g_loss, epoch)
            print('Finished Training')

    def test(self, data_loader):
        with torch.no_grad():
            for i, (x_real, c_org) in enumerate(data_loader):

                # Prepare input images and target domain labels.
                if self.opt.cuda:
                    x_real = x_real.cuda()

                # Translate images.
                x_fake_list = [x_real]
                for c_trg in c_trg_list:
                    x_fake_list.append(self.G(x_real, c_trg))

main.py

import argparse
import random
import torch
from torch import nn
import torch.optim as optim
from torch.utils.data import DataLoader
import torchvision.transforms as transforms
from preprocess import pre_process
from celeb_dataset import CelebDataset
from trainer import Trainer

# Setting up the argument parser
parser = argparse.ArgumentParser()
parser.add_argument('--workers', type=int, help='number of data loading workers', default=4)
parser.add_argument('--batchSize', type=int, default=8, help='input batch size')
parser.add_argument('--epochs', type=int, default=20, help='number of epochs to train')
parser.add_argument('--lr', type=float, default=0.0002, help='learning rate')
parser.add_argument('--cuda', action='store_true', help='enables cuda')
parser.add_argument('--manualSeed', type=int, help='manual seed')
parser.add_argument('--dataset_path', type=str, default='./data/celeba', help='dataset path')

opt = parser.parse_args()
print(opt)

if opt.manualSeed is None:
    opt.manualSeed = random.randint(1, 10000)
print("Random Seed: ", opt.manualSeed)

def main(opt):
    # Setup  the parameters for the training/testing
    params = {
        'batch_size': opt.batchSize,
        'shuffle': True,
        'num_workers': opt.workers
    }

    # preprocess and setup dataset and datalader
    processed_data = pre_process(opt.dataset_path)
    train_dataset = CelebDataset(processed_data[:-2000])
    test_dataset = CelebDataset(processed_data[2000:])
    train_loader = DataLoader(train_dataset, **params)
    test_loader = DataLoader(test_dataset, **params)

    trainer = Trainer(opt)
    trainer.train(train_loader)
    trainer.test(test_loader)

if __name__ == "__main__":
    main(opt)


title: How can I add new layers on pre-trained model with PyTorch? (Keras example given.)

body:
I am working with Keras and trying to analyze the effects on accuracy that models which are built with some layers with meaningful weights, and some layers with random initializations.
Keras:
I load VGG19 pre-trained model with include_top = False parameter on load method.
model = keras.applications.VGG19(include_top=False, weights="imagenet", input_shape=(img_width, img_height, 3))

PyTorch:
I load VGG19 pre-trained model until the same layer with the previous model which loaded with Keras.
model = torch.hub.load('pytorch/vision:v0.6.0', 'vgg19', pretrained=True)
new_base =  (list(model.children())[:-2])[0]

After loaded models following images shows summary of them. (Pytorch, Keras)

So far there is no problem. After that, I want to add a Flatten layer and a Fully connected layer on these pre-trained models. I did it with Keras but I couldn't with PyTorch.

The output of new_model.summary() is that:

My question is, How can I do add a new layer in PyTorch?


title: Why is PyTorch called PyTorch?

body:
I have been looking into deep learning frameworks lately and have been wondering about the origin of the name of PyTorch.

With Keras, their home page nicely explains the name's origin, and with something like TensorFlow, the reasoning behind the name seems rather clear. For PyTorch, however, I cannot seem to come across why it is so named.

Of course, I understand the "Py-" prefix and also know that PyTorch is a successor in some sense of Torch. But I am still wondering: what is the original idea behind the "-Torch" part? Is it known what the origin of the name is?

Thanks for any help!


title: Custom weight initialisation causing error - pytorch

body:
%reset -f

import torch
import torch.nn as nn
import torchvision
import torchvision.transforms as transforms
import numpy as np
import matplotlib.pyplot as plt
import torch.utils.data as data_utils
import torch.nn as nn
import torch.nn.functional as F

num_epochs = 20

x1 = np.array([0,0])
x2 = np.array([0,1])
x3 = np.array([1,0])
x4 = np.array([1,1])

num_epochs = 200

x = torch.tensor([x1,x2,x3,x4]).float()
y = torch.tensor([0,1,1,0]).long()

train = data_utils.TensorDataset(x,y)
train_loader = data_utils.DataLoader(train , batch_size=2 , shuffle=True)

device = 'cpu'

input_size = 2
hidden_size = 100
num_classes = 2

learning_rate = .0001

torch.manual_seed(24)

def weights_init(m):
    m.weight.data.normal_(0.0, 1)

class NeuralNet(nn.Module) :
    def __init__(self, input_size, hidden_size, num_classes) :
        super(NeuralNet, self).__init__()
        self.fc1 = nn.Linear(input_size , hidden_size)
        self.relu = nn.ReLU()
        self.fc2 = nn.Linear(hidden_size , num_classes)

    def forward(self, x) :
        out = self.fc1(x)
        out = self.relu(out)
        out = self.fc2(out)
        return out

model = NeuralNet(input_size, hidden_size, num_classes).to(device)
model.apply(weights_init)

criterionCE = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

for i in range(0 , 1) :

        total_step = len(train_loader)
        for epoch in range(num_epochs) :
            for i,(images , labels) in enumerate(train_loader) :
                images = images.to(device)
                labels = labels.to(device)

                outputs = model(images)
                loss = criterionCE(outputs , labels)

                optimizer.zero_grad()
                loss.backward()
                optimizer.step()

        outputs = model(x)

        print(outputs.data.max(1)[1])

I'm using to initialize the weights :

def weights_init(m):
    m.weight.data.normal_(0.0, 1)

But following error is thrown :

~/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py in __getattr__(self, name)
    533                 return modules[name]
    534         raise AttributeError("'{}' object has no attribute '{}'".format(
--> 535             type(self).__name__, name))
    536
    537     def __setattr__(self, name, value):

AttributeError: 'ReLU' object has no attribute 'weight'

Is this the correct method to initialize the weights ?

Also, should object be of type nn.Module , not Relu ?


title: TypeError: 'int' object is not callable in loss.backward()

body:
While trying to set up a pytorch model, I am getting the error that the loss object is not callable when trying to do Pytorch autograd. (Relevant code shown below)

optimizer = torch.optim.Adam(model.parameters(), lr=lr,
  betas(0.0,0.9))

def train(epoch, shuffle, wisdom_model, optim, loss):
    print('train')
    accuracy = 0
    batch_num = 0
    wisdom_model.train()
    for batch in data.train_dl:

        optim.zero_grad()

        result = model(batch[0])
        loss = nn.CrossEntropyLoss()(result, batch[1].long())

        loss.backward()

        accuracy += accuracy(result, batch[1])
        print(accuracy)
        pdb.set_trace()
        batch_num += 1

    return accuracy / batch_num

TypeError                                 Traceback (most recent call last)
<ipython-input-28-5b9c9fe3b320> in <module>
----> 1 run(1, False)

<ipython-input-27-d0d67dbf6eb2> in run(num_models, dropout)
     71
     72     for epoch in range(10):
---> 73         train_accuracy = train(epoch, False, model, optimizer, loss)
     74         accuracy.append(validate(epoch, model))
     75

<ipython-input-27-d0d67dbf6eb2> in train(epoch, shuffle, model, optim, loss)
     24         pdb.set_trace()
     25
---> 26         loss.backward()
     27         optim.step()
     28

TypeError: 'int' object is not callable


title: TypeError: unhashable type: 'list' when calling .iloc()

body:
I'm currently doing some AI research for a project and for that I have to get used to a framework called "Pytorch". That's fine and all but following the official tutorial (found here) the code doesn't run properly.

The idea is that I analyse a set of facial features from a prepared dataset and then do something with it (haven't gotten to that part yet). But when I run this piece of code:

img_name = os.path.join(self.root_dir, self.landmarks_frame.iloc([index, 0]))  # At this point 'index' is 0

The dataset is initialized like this:

face_dataset = fDataset(csv_file='faces/face_landmarks.csv', root_dir='faces/')

And here is where the error pops up:

for i in range(len(face_dataset)):
    sample = face_dataset[i]  # <-- right there

That leads to the getter function:

def __getitem__(self, index):
    img_name = os.path.join(self.root_dir, self.landmarks_frame.iloc([index, 0]))
    image = io.imread(img_name)
    landmarks = self.landmarks_frame.iloc[index, 1:].as_matrix()
    landmarks = landmarks.astype('float').reshape(-1, 2)
    sample = {'image': image, 'landmarks': landmarks}

Found in my FaceLandmarksDataset(Dataset): class I simply get the error of the title. This is strange I find, because I can read the dataset just fine as a frame in PyCharm:

Where the first picture is clearly visible. I have checked as well that it is in the folder that I'm looking in.

Can anyone help out? :)


title: constants in Pytorch Linear Module Class Definition

body:
What is __constants__ in pytorch class Linear(Module): defined in https://pytorch.org/docs/stable/_modules/torch/nn/modules/linear.html?

What is its functionality and why is it used?

I have been searching around, but did not find any documentation. Please note that this does not mean the __constants__ in torch script.


title: Install Pytorch GPU with pre-installed CUDA and cudnn

body:
As the title suggests, I have pre-installed CUDA and cudnn (my Tensorflow is using them).

The version of CUDA is 10.0 from nvcc --version.

The versiuon of cudnn is 7.4.

I am trying to install pytorch in a conda environment using conda install pytorch torchvision cudatoolkit=10.0 -c pytorch.

However, the installed pytorch does not detect my GPU successfully.

Does anyone know if there is a way to install GPU-version pytorch with a specific CUDA and cudnn version? I do not want to change CUDA and cudnn version because my Tensorflow is using them.

Any ideas would be appreciated!


title: what the difference between att_mask and key_padding_mask in MultiHeadAttnetion

body:
What the difference between att_mask and key_padding_mask in MultiHeadAttnetion of pytorch:

key_padding_mask – if provided, specified padding elements in the key will be ignored by the attention. When given a binary mask and a value is True, the corresponding value on the attention layer will be ignored. When given a byte mask and a value is non-zero, the corresponding value on the attention layer will be ignored

attn_mask – 2D or 3D mask that prevents attention to certain positions. A 2D mask will be broadcasted for all the batches while a 3D mask allows to specify a different mask for the entries of each batch.

Thanks in advance.


title: Why torch.sum() before doing .backward()?

body:
I can see what this code below from this video is trying to do. But the sum from y=torch.sum(x**2) confuses me. With sum operation, y becomes a tensor with one single value. As I understand .backward() as calculating derivatives, why would we want to use sum and reduce y to one value?

import pytorch
import matplotlib.pyplot as plt
x = torch.linspace(-10.0,10.0,10, requires_grad=True)
Y = x**2
y = torch.sum(x**2)
y.backward()

plt.plot(x.detach().numpy(), Y.detach().numpy(), label="Y")
plt.plot(x.detach().numpy(), x.grad.detach().numpy(), label="derivatives")
plt.legend()


title: Trouble with PyTorch's 'ToPILImage'

body:
Why does this not work?
import torchvision.transforms.functional as tf
from torchvision import transforms
pic = np.random.randint(0, 255+1, size=28*28).reshape(28, 28)
pic = pic.astype(int)
plt.imshow(pic)
t = transforms.ToPILImage()
t(pic.reshape(28, 28, 1))
# tf.to_pil_image(pic.reshape(28, 28, 1))

A beautiful random picture is plotted by matplotlib, but no matter what datatype I chose for my NumPy ndarray, neither to_pil_image or ToPILImage work as expected.
The docs have this to say:

Converts a tensor ... or a numpy ndarray of shape H x W x C to a PIL Image while preserving the value range.
...
If the input has 1 channel, the  mode  is determined by the data type (i.e  int ,  float ,  short ).

None of these datatypes work except for "short".
Everything else results in:
TypeError: Input type int64/float64 is not supported

thrown from torchvision/transforms/functional.py in to_pil_image().
Further, even though the short datatype will work for the stand alone code snippet I provided first, it breaks down when used inside a transform.Compose() called from a Dataset object's __getitem__:
choices = transforms.RandomChoice([transforms.RandomAffine(30),
                                   transforms.RandomPerspective()])

transform = transforms.Compose([transforms.ToPILImage(),
                                transforms.RandomApply([choices], 0.5),
                                transforms.ToTensor(),
                                transforms.Normalize((0.5,), (0.5,))])

trainset = MNIST('data/train.csv', transform=transform)
trainloader = DataLoader(trainset, batch_size=32, shuffle=True, num_workers=4)

RuntimeError: DataLoader worker (pid 12917) is killed by signal: Floating point exception.
RuntimeError: DataLoader worker (pid(s) 12917) exited unexpectedly


title: Pytorch dataloader with iterable dataset stops after one epoch in multiprocessing mode

body:
I have a dataloader that is initialised with a iterable dataset. I found that when I use multiprocessing (i.e. num_workers>0 in DataLoader) in dataloader, once the dataloader is exhausted after one epoch, it doesn't get reset automatically when I iterate it again in the second epoch. Below is a small reproducible example.
I am aware that "Workers are shut down once the end of the iteration is reached," according to the documentation. However, I would like to know how to achieve my expected behaviour of "resetting automatically". Thanks for any help in advance!
import torch
class MyIterableDataset(torch.utils.data.IterableDataset):
    def __init__(self, start, end):
        super().__init__()
        self.start = start
        self.end = end

    def __iter__(self):
        return iter(range(self.start, self.end))

dataset = MyIterableDataset(0, 4)
dataloader = torch.utils.data.DataLoader(dataset, batch_size=2, shuffle=False, num_workers=1, drop_last=False)

for epoch in range(2):
    for i, data in enumerate(dataloader):
        print(i, data)

"""
stdout:
0 tensor([0, 1])
1 tensor([2, 3])
2 _IterableDatasetStopIteration(worker_id=0)
"""

While my expectation of stdout is
"""
0 tensor([0, 1])
1 tensor([2, 3])
0 tensor([0, 1])
1 tensor([2, 3])
"""

I am using the latest pytorch version (1.6.0)


title: pytorch inplace operation error. How to get around it?

body:
Using PyTorch in python, I'm feeding-back one of my CNN's layers into input-space by using an inverse network that I'm training.

However, I'm interested in the representation of only one channel. I get this by setting all values to 0 except from those of the channel that I'm interested in.

image_representation is a [torch.FloatTensor of size 1x64x56x56]

image_representation[0, 1:, :, :] = 0

Inserting image_representation into the inverse net yields an image. I'm feeding this image into the original CNN results into another representation that I can compare with the representation resulting from the original image. This comparison makes my loss function

However, when I run it I get the error

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation

I understand that the gradient cannot be fully computed because of the zeros, but how can i design my code in such a way that it will omit this error, and do the computations only for one of the 64 channels?


title: Unable to Learn XOR Representation using 2 layers of Multi-Layered Perceptron (MLP)

body:
Using PyTorch nn.Sequential model, I'm unable to learn all four representation of the XOR booleans:

import numpy as np

import torch
from torch import nn
from torch.autograd import Variable
from torch import FloatTensor
from torch import optim

use_cuda = torch.cuda.is_available()

X = xor_input = np.array([[0,0], [0,1], [1,0], [1,1]])
Y = xor_output = np.array([[0,1,1,0]]).T

# Converting the X to PyTorch-able data structure.
X_pt = Variable(FloatTensor(X))
X_pt = X_pt.cuda() if use_cuda else X_pt
# Converting the Y to PyTorch-able data structure.
Y_pt = Variable(FloatTensor(Y), requires_grad=False)
Y_pt = Y_pt.cuda() if use_cuda else Y_pt

hidden_dim = 5

model = nn.Sequential(nn.Linear(input_dim, hidden_dim),
                      nn.Linear(hidden_dim, output_dim),
                      nn.Sigmoid())
criterion = nn.L1Loss()
learning_rate = 0.03
optimizer = optim.SGD(model.parameters(), lr=learning_rate)
num_epochs = 10000

for _ in range(num_epochs):
    predictions = model(X_pt)
    loss_this_epoch = criterion(predictions, Y_pt)
    loss_this_epoch.backward()
    optimizer.step()
    print([int(_pred > 0.5) for _pred in predictions], list(map(int, Y_pt)), loss_this_epoch.data[0])

After learning:

for _x, _y in zip(X_pt, Y_pt):
    prediction = model(_x)
    print('Input:\t', list(map(int, _x)))
    print('Pred:\t', int(prediction))
    print('Ouput:\t', int(_y))
    print('######')

[out]:

Input:   [0, 0]
Pred:    0
Ouput:   0
######
Input:   [0, 1]
Pred:    1
Ouput:   1
######
Input:   [1, 0]
Pred:    0
Ouput:   1
######
Input:   [1, 1]
Pred:    0
Ouput:   0
######

I've tried running the same code over a couple of random seeds but it didn't manage to learn all for XOR representation.

Without PyTorch, I could easily train a model with self-defined derivative functions and manually perform the backpropagation, see https://www.kaggle.io/svf/2342536/635025ecf1de59b71ea4fa03eb84f9f9/results.html#After-some-enlightenment

Why is it that the 2-layered MLP using PyTorch didn't learn the XOR representation?

How is the model in PyTorch:

hidden_dim = 5

model = nn.Sequential(nn.Linear(input_dim, hidden_dim),
                      nn.Linear(hidden_dim, output_dim),
                      nn.Sigmoid())

different from the one that is hand-written with the derivatives and the manually written backpropagation and optimizer step from https://www.kaggle.com/alvations/xor-with-mlp ?

Are the same the one hidden layered perceptron network?

Updated

Strangely, adding a nn.Sigmoid() between the nn.Linear layers didn't work:

hidden_dim = 5

model = nn.Sequential(nn.Linear(input_dim, hidden_dim),
                      nn.Sigmoid(),
                      nn.Linear(hidden_dim, output_dim),
                      nn.Sigmoid())
criterion = nn.L1Loss()
learning_rate = 0.03
optimizer = optim.SGD(model.parameters(), lr=learning_rate)
num_epochs = 10000

for _ in range(num_epochs):
    predictions = model(X_pt)
    loss_this_epoch = criterion(predictions, Y_pt)
    loss_this_epoch.backward()
    optimizer.step()

for _x, _y in zip(X_pt, Y_pt):
    prediction = model(_x)
    print('Input:\t', list(map(int, _x)))
    print('Pred:\t', int(prediction))
    print('Ouput:\t', int(_y))
    print('######')

[out]:

Input:   [0, 0]
Pred:    0
Ouput:   0
######
Input:   [0, 1]
Pred:    1
Ouput:   1
######
Input:   [1, 0]
Pred:    1
Ouput:   1
######
Input:   [1, 1]
Pred:    1
Ouput:   0
######

But adding nn.ReLU() did:

model = nn.Sequential(nn.Linear(input_dim, hidden_dim),
                      nn.ReLU(),
                      nn.Linear(hidden_dim, output_dim),
                      nn.Sigmoid())

...
for _x, _y in zip(X_pt, Y_pt):
prediction = model(_x)
print('Input:\t', list(map(int, _x)))
print('Pred:\t', int(prediction))
print('Ouput:\t', int(_y))
print('######')

[out]:

Input:   [0, 0]
Pred:    0
Ouput:   0
######
Input:   [0, 1]
Pred:    1
Ouput:   1
######
Input:   [1, 0]
Pred:    1
Ouput:   1
######
Input:   [1, 1]
Pred:    1
Ouput:   0
######

Isn't a sigmoid enough for the non-linear activation?

I understand that the ReLU fits the task of boolean output but shouldn't the Sigmoid function produce the same/similar effect?

UPDATED 2

Running the same training 100 times:

from collections import Counter
import random
random.seed(100)

import torch
from torch import nn
from torch.autograd import Variable
from torch import FloatTensor
from torch import optim
use_cuda = torch.cuda.is_available()

all_results=[]

for _ in range(100):
    hidden_dim = 2

    model = nn.Sequential(nn.Linear(input_dim, hidden_dim),
                          nn.ReLU(), # Does the sigmoid has a build in biased?
                          nn.Linear(hidden_dim, output_dim),
                          nn.Sigmoid())

    criterion = nn.MSELoss()
    learning_rate = 0.03
    optimizer = optim.SGD(model.parameters(), lr=learning_rate)
    num_epochs = 3000

    for _ in range(num_epochs):
        predictions = model(X_pt)
        loss_this_epoch = criterion(predictions, Y_pt)
        loss_this_epoch.backward()
        optimizer.step()
        ##print([float(_pred) for _pred in predictions], list(map(int, Y_pt)), loss_this_epoch.data[0])

    x_pred = [int(model(_x)) for _x in X_pt]
    y_truth = list([int(_y[0]) for _y in Y_pt])
    all_results.append([x_pred == y_truth, x_pred, loss_this_epoch.data[0]])

tf, outputsss, losses__ = zip(*all_results)
print(Counter(tf))

It only managed to learn the XOR representation 18 out of 100 times... -_-|||


title: Can both the GPU and CPU versions of PyTorch be installed in the same Conda environment?

body:
The PyTorch installation web page shows how to install the GPU and CPU versions of PyTorch:

conda install pytorch torchvision cudatoolkit=10.1 -c pytorch

and

conda install pytorch torchvision cpuonly -c pytorch

Can both version be installed in the same Conda environment?

In case you might ask why would this be needed, it's because I would like a single Conda environment which I can use on computers which have a GPU and those which don't.


title: torch find indices of matching rows in 2 2D tensors

body:
I have two 2D tensors, in different length, both are different subsets of the same original 2d tensor and I would like to find all the matching "rows"
e.g

A = [[1,2,3],[4,5,6],[7,8,9],[3,3,3]
B = [[1,2,3],[7,8,9],[4,4,4]]
torch.2dintersect(A,B) -> [0,2] (the indecies of A that B also have)

I've only see numpy solutions, that use dtype as dicts, and does not work for pytorch.

Here is how I do it in numpy

arr1 = edge_index_dense.numpy().view(np.int32)
arr2 = edge_index2_dense.numpy().view(np.int32)
arr1_view = arr1.view([('', arr1.dtype)] * arr1.shape[1])
arr2_view = arr2.view([('', arr2.dtype)] * arr2.shape[1])
intersected = np.intersect1d(arr1_view, arr2_view, return_indices=True)


title: How can I monitor both training and eval loss when finetuning BERT on a GLUE task?

body:
I am running https://github.com/huggingface/transformers/blob/master/examples/run_glue.py to perform finetuning on a binary classification task (CoLA).
I'd like to monitor both the training and evaluation losses to prevent overfitting.

Currently the library is at 2.8.0, and I did the install from source.

When I run the example with

python run_glue.py --model_name_or_path bert-base-uncased
                   --task_name CoLA
                   --do_train
                   --do_eval
                   --data_dir my_dir
                   --max_seq_length 128
                   --per_gpu_train_batch_size 8
                   --per_gpu_eval_batch_size 8
                   --learning_rate 2e-5
                   --num_train_epochs 3.0
                   --output_dir ./outputs
                   --logging_steps 5

In the stdout logs I see lines with one single value for the loss, such as

  {"learning_rate": 3.3333333333333333e-06, "loss": 0.47537623047828675,
  "step": 25}

By peeking in https://github.com/huggingface/transformers/blob/master/src/transformers/trainer.py I see that training and evaluation losses are computed there (looks to me that code was recently refactored).

I have thus replaced https://github.com/huggingface/transformers/blob/abb1fa3f374811ea09d0bc3440d820c50735008d/src/transformers/trainer.py#L314 with

 cr_loss = self._training_step(model, inputs, optimizer)
 tr_loss += cr_loss

and added after line https://github.com/huggingface/transformers/blob/abb1fa3f374811ea09d0bc3440d820c50735008d/src/transformers/trainer.py#L345

logs["training loss"] = cr_loss

with this I get:

0502 14:12:18.644119 23632 summary.py:47] Summary name training loss is illegal; using training_loss instead.
                          | 4/10  [00:02<00:04,  1.49it/s]
{"learning_rate":  3.3333333333333333e-06, "loss": 0.47537623047828675, "training loss": 0.5451719760894775, "step": 25}

Is this OK, or am I doing anything wrong here?

What's the best way to monitor in stdout both the averaged training and evaluation loss for a given logging interval during finetuning?


title: Pad torch tensors of different sizes to be equal

body:
I am looking for a way to take an image/target batch for segmentation and return the batch where the image dimensions have been changed to be equal for the whole batch.  I have tried this using the code below:

def collate_fn_padd(batch):
    '''
    Padds batch of variable length

    note: it converts things ToTensor manually here since the ToTensor transform
    assume it takes in images rather than arbitrary tensors.
    '''
    # separate the image and masks
    image_batch,mask_batch = zip(*batch)

    # pad the images and masks
    image_batch = torch.nn.utils.rnn.pad_sequence(image_batch, batch_first=True)
    mask_batch = torch.nn.utils.rnn.pad_sequence(mask_batch, batch_first=True)

    # rezip the batch
    batch = list(zip(image_batch, mask_batch))

    return batch

However, I get this error:

RuntimeError: The expanded size of the tensor (650) must match the existing size (439) at non-singleton dimension 2.  Target sizes: [3, 650, 650].  Tensor sizes: [3, 406, 439]

How do I efficiently pad the tensors to be of equal dimensions and avoid this issue?


title: How to build an autograd-compatible Pytorch module that resizes tensors like images?

body:
I was wondering if I can build an image resize module in Pytorch that takes a torch.tensor of 3*H*W as the input and return a tensor as the resized image.

I know it is possible to convert tensor to PIL Image and use torchvision,
but I also hope to back propagate gradients from the resized image to the original image, and the following example will return such error (in PyTorch 0.4.0 on Windows 10):

import numpy as np
from torchvision import transforms

t2i = transforms.ToPILImage()
i2t = transforms.ToTensor()

trans = transforms.Compose(
    t2i, transforms.Resize(size=200), i2t]
)

test = np.random.normal(size=[3, 300, 300])
test = torch.tensor(test, requires_grad=True)
resized = trans(test)
resized.backward()

print(test.grad)

Traceback (most recent call last):
  File "D:/Projects/Python/PyTorch/test.py", line 41, in <module>
    main()
  File "D:/Projects/Python/PyTorch/test.py", line 33, in main
    resized = trans(test)
  File "D:\Anaconda3\envs\pytorch\lib\site-packages\torchvision\transforms\transforms.py", line 42, in __call__
    img = t(img)
  File "D:\Anaconda3\envs\pytorch\lib\site-packages\torchvision\transforms\transforms.py", line 103, in __call__
    return F.to_pil_image(pic, self.mode)
  File "D:\Anaconda3\envs\pytorch\lib\site-packages\torchvision\transforms\functional.py", line 102, in to_pil_image
    npimg = np.transpose(pic.numpy(), (1, 2, 0))
RuntimeError: Can't call numpy() on Variable that requires grad. Use var.detach().numpy() instead.

It seems like I cannot "imresize" a tensor without detaching it from autograd first, but detaching it prevents me from computing gradients.

Is there a way to build a torch function/module that does the same thing as torchvision.transforms.Resize that is autograd compatiable? Any help is much appreciated!


title: PyTorch: _thnn_nll_loss_forward is not implemented for type torch.LongTensor

body:
When trying to create a model using PyTorch, when I am trying to implement the loss function nll_loss, it is throwing the following error

RuntimeError: _thnn_nll_loss_forward is not implemented for type torch.LongTensor

The fit function I have created is:

for epoch in tqdm_notebook(range(1, epochs+1)):
    for batch_idx, (data, targets) in enumerate(train_loader):
        optimizer.zero_grad()
        net.float()
        output = net(data)
        output_x = output.argmax(dim=2) #to convert (64,50,43) -> (64, 50)
        loss = F.nll_loss(output_x, targets)
        loss.backward()
        optimizer.step()
        if batch_idx % 100 == 0:
            print('Train epochs: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                epoch, batch_idx*len(data), len(ds.data),
                100.*batch_idx / len(ds), loss.item()
            ))

Where the output and targets' shape is (64, 50) and the dtypes are torch.int64 for both.


title: Pytorch Cuda Runtime Error (with Properly installed Drivers)

body:
When I run

import torch
torch.cuda.FloatTensor([1.])

I seem to be getting the error: RuntimeError: cuda runtime error (38) : no CUDA-capable device is detected at /pytorch/aten/src/THC/THCGeneral.cpp:50.

I'm using Debian (Stretch). Checking the nvidia driveron the console:

/sbin/modinfo nvidia_current

returns:

filename:       /lib/modules/4.9.0-11-amd64/updates/dkms/nvidia-current.ko
alias:          char-major-195-*
version:        418.74
supported:      external
license:        NVIDIA
srcversion:     AB4044DE27C9CA55579A110

when I do nvcc --version, I get:

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2018 NVIDIA Corporation
Built on Tue_Jun_12_23:07:04_CDT_2018
Cuda compilation tools, release 9.2, V9.2.148

On python, when I do

import torch
print(torch.version.cuda)
print(torch.cuda.device_count())
print(torch.cuda.is_available())

I get:

9.2.148
0
False

I've tried doing os.environ["CUDA_VISIBLE_DEVICES"] = '0, but it still doesn't work.

Any help would be much appreciated!


title: TypeError: can’t convert CUDA tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first (fastai)

body:
I am following the code here:

https://www.kaggle.com/tanlikesmath/diabetic-retinopathy-with-resnet50-oversampling

However, during the metrics calculation, I am getting the following error:

File "main.py", line 50, in <module>
 learn.fit_one_cycle(4,max_lr = 2e-3)
...
File "main.py", line 39, in quadratic_kappa
    return torch.tensor(cohen_kappa_score(torch.argmax(y_hat,1), y, weights='quadratic'),device='cuda:0')
...
File "/pfs/work7/workspace/scratch/ul_dco32-conda-0/conda/envs/resnet50/lib/python3.8/site-packages/torch/tensor.py", line 486, in __array__
    return self.numpy()
TypeError: can't convert CUDA tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.

Here are the metrics and the model:

def quadratic_kappa(y_hat, y):
    return torch.tensor(cohen_kappa_score(torch.argmax(y_hat,1), y, weights='quadratic'),device='cuda:0')

learn = cnn_learner(data, models.resnet50, metrics = [accuracy,quadratic_kappa])
learn.fit_one_cycle(4,max_lr = 2e-3)

As it is being said in the discussion https://discuss.pytorch.org/t/typeerror-can-t-convert-cuda-tensor-to-numpy-use-tensor-cpu-to-copy-the-tensor-to-host-memory-first/32850/6, I have to bring the data back to cpu. But I am slightly lost how to do it.

I tried to add .cpu() all over the metrics but could not solve it so far.


title: Using Precision and Recall in training of skewed dataset

body:
I have a skewed dataset (5,000,000 positive examples and only 8000 negative [binary classified]) and thus, I know, accuracy is not a useful model evaluation metric. I know how to calculate precision and recall mathematically but I am unsure how to implement them in python code.

When I train the model on all the data I get 99% accuracy overall but 0% accuracy on the negative examples (ie. classifying everything as positive).

I have built my current model in Pytorch with the criterion = nn.CrossEntropyLoss() and optimiser = optim.Adam().

So, my question is, how do I implement precision and recall into my training to produce the best model possible?

Thanks in advance


title: Pytorch tensor get the index of the element with specific values?

body:
I have two tensores, tensor a and tensor b.
I want to get all indexes of values in tensor b.
For example.
a = torch.Tensor([1,2,2,3,4,4,4,5])

b = torch.Tensor([1,2,4])

I want the index of 1, 2, 4 in tensor a. I can do this by the following code.
a = torch.Tensor([1,2,2,3,4,4,4,5])
b = torch.Tensor([1,2,4])
mask = torch.zeros(a.shape).type(torch.bool)
print(mask)
for e in b:
    mask = mask + (a == e)
    print(mask)

How can I do it without for?


title: How to do CIFAR-10 with PyTorch on CUDA?

body:
I'm following the CIFAR-10 PyTorch tutorial at this pytorch page , and can't get PyTorch running on the GPU. The code is exactly as in the tutorial.

The error I get is

Traceback (most recent call last):
  File "(file path)/CIFAR10_tutorial.py", line 116, in <module>
   outputs = net(images)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 491, in __call__
result = self.forward(*input, **kwargs)
  File "(file path)/CIFAR10_tutorial.py", line 65, in forward
x = self.pool(F.relu(self.conv1(x)).cuda())
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 491, in __call__
result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/conv.py", line 301, in forward
self.padding, self.dilation, self.groups)

My CUDA version is 9.0, Pytorch 0.4.0. I have used tensorflow-gpu on the machine, so I know CUDA is set up correctly. Where exactly must I use .cuda() and .to(device) as suggested in the tutorial?


title: coding patterns for efficient minibatch loop using custom pytorch dataset

body:
Are there any general recommendations to handle data efficiently in a custom Dataset so that it plays nicely with the minibatch eval/train loop? To illustrate what I mean more concretely, let's say I define this synthetic toy dataset that maps x to x+1:

import torch.utils.data as data

class Dataset(data.Dataset):
    def __init__(self):
        super(Dataset, self).__init__()
        # list of [x, y]
        self.dataset = [
            [1, 2],
            [2, 3],
            [3, 4],
            [4, 5]
        ]

    def __getitem__(self, index):
        item = self.dataset[index]
        return item[0], item[1]

    def __len__(self):
        return len(self.dataset)

In practice, this would be wrapped in a DataLoader and accessed inside an eval/train loop, something like this:

dataset = Dataset()
data_loader = data.DataLoader(dataset=dataset, batch_size=2, shuffle=True)
epochs = 100
for i_epoch in range(epochs):
    for i_minibatch, minibatch in enumerate(data_loader):
        x, y = minibatch
        # predict and train

The dataset object might return primitive Python objects like numbers or lists, like in my example implementation, but in the "predict and train" part of the last code snippet, we need some specific data types to compute stuff, like a torch.FloatTensor (it seems the data loader can do this implicitly), probably even wrapped as a torch.autograd.Variable, and some calls to .cuda() might also be necessary. My question is about general advice for when to make these data transformations and function calls.

For example, one option would be to have everything already saved as a torch.FloatTensor inside the dataset, and in the data_loader loop we could add the Variable wrapper and call .cuda(). We could also have all or part of the data on the GPU already by calling .cuda() in the Dataset constructor or in the getitem method. I think there might be pros and cons to all of these approaches. If I'm training a model for several epochs, I don't want to introduce unnecessary overhead each epoch or minibatch iteration that could have been avoided by precomputing stuff in the dataset. Probably someone with more knowledge about the internals of pytorch (maybe related to some caching or jit compilation happening under the hood) might be able to point to more specific reasons to choose one approach over another.


title: What is a fused kernel (or fused layer) in deep learning?

body:
I am reading the Apex AMP documentation:

  A Python-only build omits:

  Fused kernels required to use apex.optimizers.FusedAdam.
  Fused kernels
  required to use apex.normalization.FusedLayerNorm.
  Fused kernels that
  improve the performance and numerical stability of
  apex.parallel.SyncBatchNorm.
  Fused kernels that improve the
  performance of apex.parallel.DistributedDataParallel and apex.amp.
  DistributedDataParallel, amp, and SyncBatchNorm will still be usable,
  but they may be slower.

There also seems to be a "FusedAdam" optimizer:

  The Adam optimizer in Pytorch (like all Pytorch optimizers) carries
  out optimizer.step() by looping over parameters, and launching a
  series of kernels for each parameter. This can require hundreds of
  small launches that are mostly bound by CPU-side Python looping and
  kernel launch overhead, resulting in poor device utilization.
  Currently, the FusedAdam implementation in Apex flattens the
  parameters for the optimization step, then carries out the
  optimization step itself via a fused kernel that combines all the Adam
  operations. In this way, the loop over parameters as well as the
  internal series of Adam operations for each parameter are fused such
  that optimizer.step() requires only a few kernel launches.

  The current implementation (in Apex master) is brittle and only works
  with Amp opt_level O2. I’ve got a WIP branch to make it work for any
  opt_level (https://github.com/NVIDIA/apex/pull/351). I recommend
  waiting until this is merged then trying it.

This partially explains it. I'm left with more questions:

What is meant by kernel? A layer or an optimizer?

Is the idea of fused layer the same as a fused optimizer?


title: How to use a bert pretrained model somewhere else?

body:
I followed this course https://www.coursera.org/learn/sentiment-analysis-bert about building a pretrained model for sentiment analysis. During the trining, at each epoch they saved the model using torch.save(model.state_dict(), f'BERT_ft_epoch{epoch}.model'). Now I want to use one of these models (the best one obviously) elsewhere, for example where a user can paste a tweet as an input and get the emotion of the writer. But I don't know how to load the model and predict, here's what I tried:
import torchvision.models as models
import torch

model = models.resnet101(pretrained=False)
model.load_state_dict(torch.load('Models/BERT_ft_epoch15.model'), strict=False)
model_ft.eval()
output = model_ft(input) #input is a tweets list

I get this error: TypeError: conv2d(): argument 'input' (position 1) must be Tensor, not list


Hi I got an error while trying to install pytorch:
PS C:\windows\system32> pip install torch===1.7.0+cu110 torchvision===0.8.1+cu110 torchaudio===0.7.0 -f https://download.pytorch.org/whl/torch_stable.html
Looking in links: https://download.pytorch.org/whl/torch_stable.html
ERROR: Could not find a version that satisfies the requirement torch===1.7.0+cu110 (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)
ERROR: No matching distribution found for torch===1.7.0+cu110

I have seen this question, and I am using python 3.8.6
PS C:\windows\system32> py --version
Python 3.8.6